Tao Dong
email: dongtao@umich.edu

Mark S Ackerman
email: ackerm@umich.edu

Mark W Newman
email: mwnewman@umich.edu

Gaurav Paruthi
email: gparuthi@umich.edu

Social Overlays: Collectively Making Websites More Usable

Keywords: Usability, community, peer production, social computing

Many small organizations lack the expertise and resources to conduct usability evaluations of their websites. Social Overlays, presented here, is a new system that allows a community of users to collectively improve their website. Social Overlays enables end-users to identify and repair common user interface problems through creating "overlays" on web pages as part of their regular use, thereby improving usability while reducing the need for professional services. In short, Social Overlays harnesses the diversity of experience and ideas within a community to "crowd source" usability.

To evaluate Social Overlays, we examined whether a group of community members without any usability training could use Social Overlays to identify and repair UI problems on their medium-sized community's website. We found that they could. Community users were able to uncover a large number of UI problems and formulate reasonable solutions to the problems they identified. In addition, we compared Social Overlays to two standard ways of assessing website usability: expert inspection and usability testing. We found that Social Overlays users identified more problems, and their reported problems differed in useful ways from those found by the experts and the usability testing team.

Introduction

Many small organizations' websites need usability improvement. For organizations like charities, museums, and schools, their constraints on technical resources and usability expertise keep them from sufficiently making their websites easy to use. For example:

• TriCounty GoodDeeds is rolling out its website. But, its web designer and webmaster are volunteers, and there are no spare resources for usability tests. How can they create a web site that is usable and helpful for the charity's constituents? • The Tree City municipal government's financial services group created a new website for travel reimbursement. It is unusable, since it is full of jargon and appears to be meant for accounting professionals. How can staff in departments help the financial group with their usability? • The History Department at the University of the Midwest recently created a new website using a content management system. The department has a well-meaning web developer, who is an ex-computer science major. The students have to use the website to get information for courses, requirements, and school events, though they often feel the developer doesn't know how they actually use the website. How can the department create a usable, useful website? At a high level, there are two standard approaches to evaluating usability on websites: usability testing and expert evaluation [START_REF] Nielsen | Usability Engineering[END_REF]. While these approaches have many variants, they all share the characteristic that usability experts play a critical role in determining the existence and nature of usability problems. An alternative approach is to solicit problem reports from users. Post-deployment usability approaches focus on collecting feedback from users in the field at the time the problem is encountered [START_REF] Hartson | Remote evaluation for post-deployment usability improvement[END_REF][START_REF] Nichols | Participatory usability: supporting proactive users[END_REF]. However Chilana et al. found that few usability practitioners analyze or respond to such feedback [START_REF] Chilana | Post-deployment usability: a survey of current practices[END_REF]. Participatory techniques have been proposed for involving users in usability evaluation during the formative stages of system development as well [START_REF] Bias | The pluralistic usability walkthrough: coordinated empathies[END_REF][START_REF] Muller | Methods & tools: participatory heuristic evaluation[END_REF], though these techniques still depend on the participation of usability experts and product developers. In this paper, we investigate a novel approach to creating usable interactive systems: enabling community members themselves to collectively improve the system as part of their everyday interactions with it. Our approach is aimed primarily at websites built to serve small-to-medium sized organizations or communities-precisely the sites that often do not have the resources to hire usability professionals or implement frequent changes to a site. Our question, then, is how can we leverage the user community of a website to uncover and service usability problems?

We have designed and built a system called Social Overlays (SO). By harnessing the "wisdom of the crowd," SO allows a community to collectively evaluate and improve their website, without the need for formal usability methods or professional usability expertise. Using SO, community members can create "overlays," which effectively rewrite particular page elements (e.g., text, links, and tooltips), thereby improving the site's usability for subsequent visitors. SO also provides lightweight mechanisms for different community members to nominate potential problems, propose alternate fixes for the identified problems, and vote for the best solution. In short, SO harnesses the diversity of experience and ideas within a community to "crowd source" usability.

While, at a technical level, SO could be deployed on a wide range of sites, our initial focus is on a type of site that is particularly in need of and well-suited to the SO approach. Those are the sites that serve communities or organizations with a few dozen to several hundred members. While such communities are the ones most in need of a low-cost approach to usability improvement, we also expect them to be relatively cohesive and possessing members who have sufficient common ground in vocabulary, practices, and expectations [START_REF] Mcgrath | The Academic Crisis of the Community College[END_REF]. In these communities, members often know one another, creating the grounds for altruism and self-policing [START_REF] Sproull | Connections: New Ways of Working in the Networked Organization[END_REF]. In a word, those communities possess the desirable social properties that would allow the SO approach to be adopted most effectively.

To examine the viability of the SO approach, we conducted a study with thirteen members of a medium-sized academic community. We found that they were able to find and repair a large number of usability problems on the community's website. Moreover, compared with usability experts whom we asked to evaluate the same site and an external usability team who conducted a conventional usability test, the community members reported more problems. The problems they found differed in systematic but useful ways from those found using standard usability methods. Thus, our study results argue for the feasibility of the SO approach, at least for small-to-medium sized communities and organizations.

The contribution of this paper, then, is twofold: 1. We present a novel approach to collectively improving website usability by enabling the site's community to not only identify but also repair usability problems. 2. We provide evidence that our approach leads to usability improvements for important types of communities that are comparable to standard but more costly methods such as expert inspection and usability testing. In addition, we discuss how this approach can be extended for larger or less socially cohesive communities and how this approach is situated in the literature.

System Description

The goal of Social Overlays (SO) is to allow users of a website to identify and repair usability problems during the course of their regular use of the site. We identified, through pre-studies, an initial set of requirements that SO must meet in order to accomplish this goal.

1. It should be easy for non-technical users to report and repair problems. 2. When a user is aware of a problem but does not know how to repair it, he/she should be able to request help from other community members or the webmaster. 3. Community members must be able to see the repairs made by other users and decide whether or not they prefer a peer-modified version to the original design of a UI element. 4. Community members and the webmaster need to be able to review and address requests made by other users. In the rest of this section, we first walk through a scenario that illustrates the use of SO, and then describe the main features of SO.

Scenario

Chelsea, a master's student at the aforementioned history department, is looking up a class she is considering for next semester. On the page that allows her to search for courses, she finds herself annoyed by the weirdly labeled "Apply" button next to the course search field. She wonders, "Why couldn't they just put the word 'Search' on that search button?" Coming from a literature background, Chelsea often jokes that she is a "language snob." Fortunately, Chelsea has a new browser extension called Social Overlays that allows her to revise the button's label. She does so, and sees the change immediately.

In more detail, Chelsea chooses the Text tool in the SO panel (see Fig. 1a). Now when her mouse hovers over a page element, it is highlighted with an orange dashed outline (see Fig. 1b). She clicks the "Apply" button that annoyed her on the page to invoke SO's element editing dialog box, where she enters the text string "Search" to change the button's label (see Fig. 1c). Clicking the "Save" button instantly applies the change for her.

Within an hour, Chelsea finds that her overlay (the alternate label for the button) has received thumbs-up from 5 other SO users in her department. After several days, her department's IT person notified her that the website has implemented her change permanently, because of strong user preference shown on SO.

Overlays

In the above scenario, Chelsea repairs an unconventionally labeled button by creating an overlay using the Text tool provided by SO. Overlays are user-generated in-place modifications to existing web page elements. There are three types of overlays supported in the current version of SO: Text, Links, and Tooltips. Each type of overlay can be created and edited using tools provided by the SO extension panel (see Fig. 2). As the scenario above has shown, the Text tool allows the user to revise inaccurate or unintuitive terms or languages used on buttons, links, or headers.

The Link tool allows the user to add a link to any element of a web page by entering a target URL. This can be useful for creating navigation shortcuts or pointing to additional information that might be helpful for a given task. In addition, the Link tool allows the user to edit the URL of a broken or outdated hyperlink.

The Tooltip tool allows users to create or rewrite tooltips (i.e., short messages that appear when an element is hovered over) that are attached to any page element. This tool allows a user to explain or clarify what an element does (or does not do) and how to make use of a feature on the site to his/her fellow users.

The Link tool and the Tooltip tool follow the same workflow as the Text tool to make overlays. A user can combine different types of overlays if needed. For example, the user can modify the text of a hyperlink, and also install a tooltip for it.

Help Requests

Although the changes that SO currently supports are relatively simple, some users might not feel confident enough to make a change. Additionally, some users might want a more sophisticated change that SO does not yet support. In such situations, a andc) specify the modification. For example, the user in this scenario renames a mislabeled button from "Apply" to "Search."

user can request help from the community or the webmaster using the Help button (see Fig. 2). The Help button allows the user to attach a message to the element that needs to be fixed or improved through a process similar to editing a button's label. If the request is not related to a particular element, the user has the option to submit a general request for the page by selecting the "bucket" at the bottom-right corner.

Indicators

When a page loads, SO shows page modifications made by the user community on the current page. A modified element will flash for half a second to help a user distinguish it from the original element, unless a particular overlay has been previously approved by the current user or by a certain number of other community members.

Inspired by Edit Wear [START_REF] Hill | Edit wear and read wear[END_REF], SO uses the side margin of the web browser to provide lightweight visual indicators of prior community activity on the current page (see Fig. can also toggle between the original page and the community-enhanced page using the "Show Overlays" switch on the SO panel.

Selection Rules

When multiple users have defined overlays for an element, one of them must be chosen to display. We implemented a voting mechanism that allows each user to indicate whether an overlay works for him or not. The most approved overlay for an element is automatically displayed unless the current user has approved a different one. In addition, if an overlay's number of votes reaches a community-specific threshold, it will no longer flash to draw the user's attention when the page loads, since it is likely to be working for her as well.

Implementation Details

We implemented SO as an extension for the Chrome web browser1 that is coupled to a server-side application. Every time the user visits a new web page, the SO extension injects a set of JavaScript files into the current page to access and manipulate the page's HTML Document Object Model (DOM). With the full control of the target page's DOM, the SO extension enables a set of augmented capabilities within the browser, allowing the user to identify usability problems, suggest localized changes, or make quick modifications to the current page. The SO extension sends the changes made by the user to the SO server's repository of overlays. Upon a subsequent page load, the SO extension retrieves the overlays and requests made by all the community members associated with the current page from the server and applies these changes by modifying the DOM of the rendered web page. This implementation allows us to gain complete control over the rendering of each web page. It also allows easy porting to extensions for other browsers or a bookmarklet application.

Study Design, Site, and Data Collection

After creating Social Overlays, we wanted to know whether it was a feasible approach. To evaluate the feasibility of Social Overlays, we conducted a three-phase study. In the first phase, we determined whether community members could use SO to identify and repair usability problems. We learned that people could use SO effectively, but we could not tell how good SO's results were compared to those generated by standard usability methods. Thus, in the next two phases we compared SO to two standard usability methods: expert inspection and usability testing. The results of all three phases show that SO is a feasible approach to website usability.

SO Evaluation

First, we needed to know whether community members without training in usability could identify and repair usability deficiencies on their website using SO. Therefore, in the initial phase of our study (which we will call the "SO evaluation"), we asked 13 information science students, none of whom had any formal usability education or experience, to use SO in the process of completing four tasks on the website of their school (and ours), the University of X School of X (UXSX). We did not choose the website and users as a matter of convenience. UXSX's website had been recently overhauled and had many flaws and problems, thus providing a good testbed. In addition, by using UXSX and its users, we could emulate website users coming to the webpages they use everyday and finding problems based on their normal tasks. Moreover, UXSX is an organization similar to those foreseen as our target communities. It consists of approximately 400 people, including students, staff, and faculty, and is reasonably cohesive socially.

We selected four common information-seeking tasks (see Appendix I) and asked participants to carry out the tasks, using SO to identify and, if possible, repair any issues they encountered as they went along. Each task had known usability issues associated with the website. While the four tasks selected covered only a small subset of the site's possible interactions, they were enough to observe the community process of discovering, noting, fixing and voting on changes to usability problems within a tractable timeframe. We anticipate that, if SO were deployed over a longer time period, a larger number of community members would visit many more pages, ultimately covering all of the most common interaction paths.

To emulate people coming to a web site over time, we ran our participants consecutively. Thus, the overlays (UI modifications) and requests made by a participant were available to all participants after him/her. Participants were first given a demonstration of using SO to fix four example usability problems, and then they were instructed to identify usability problems while solving the four tasks. The participants were asked to state their ideal solution to a problem, and then solve the issue using SO. If a problem could not be repaired by SO's Text, Link, and Tooltip tools, the participant had the option of submitting a request in SO or not doing anything. Each session lasted about an hour and each participant received a $10 coffee shop gift card.

Expert Inspection and Usability Test

As mentioned, to address whether SO worked as well as standard usability methods, we conducted two additional evaluations. In the second phase (which we will term the "expert inspection"), we asked four usability specialists with at least four years of professional experience to conduct an expert walkthrough (as described in [START_REF] Oppermann | Software evaluation using the 9241 evaluator[END_REF]) of the same four tasks that were used in the SO evaluation. The experts were given the same demonstration of SO in the beginning of each session. They were asked to identify as many problems as they could while walking through the four tasks, and envision how SO could be used for implementing their suggested solutions. Each expert was compensated $50 for participating an hour-long session.

In the third and final phase (the usability test), we commissioned a team of external usability evaluators to conduct a conventional lab-based usability test on the UXSX site, and asked them to report the problems they found along with recommendations to address those problems. The usability team consisted of two graduate students with formal training in usability testing and one usability professional with formal training and three years of professional experience.

The usability test followed the standard protocol described in Rubin's widely-used textbook [START_REF] Rubin | Handbook of Usability Testing: Howto Plan, Design, and Conduct Effective Tests[END_REF], which consists of pre-test and post-test questionnaires, task observation, and debriefing. Eight additional UXSX students participated in the test and tackled the same four tasks used in the SO evaluation and expert inspection. Each session in the usability test lasted about an hour and each participant received $20.

Evaluation Results

In this section, we answer the following questions:

• Within our evaluation study, could community members use SO to report a substantial number of usability problems on their website? • In addition to identifying problems, were community members able to use SO to repair at least some of them in a helpful way? • How well did the SO approach work, in comparison with expert inspection and usability testing? • How did community members collaborate informally in using SO to improve their site and benefit from one another's efforts? In short, is the SO approach likely to be viable?

Community-based Usability Improvements

The results of our evaluation show that community members without training in usability can identify a large quantity of usability deficiencies on their website, at least in this community.

In our data analysis, we tallied all problems for which the SO evaluation participants either made an overlay or submitted a help request. We then manually verified these issues on the UXSX website, confirming the existence of identified problems and eliminating duplicates. As a group, they documented 47 unique problems in the process of solving the 4 evaluation tasks. These included issues that could and could not be fixed with SO, but excluded issues that were verbally reported only as well as issues that were similar to those used as examples in the SO demo. In the rest of this subsection, we describe the problems identified by community members in the SO evaluation, and how they as a collective addressed those problems.

Types and Characteristics of Overlays. Using the Text, Link, and Tooltip tools provided by SO, the participants in the SO evaluation made 50 overlays (i.e. page modifications) to address 27 (57.4%) of the total 47 problems they documented.

Among the 50 overlays, 10 were alternative text or labels, 11 were hyperlinks attached to existing elements, and 29 were tooltips.

As expected, the Text tool was often used to correct or clarify a link's label. For example, P3 changed a link's label from "Course schedule" to "Course schedule by term," as she thought people might expect the linked schedule to be organized by week. Other uses of the Text tool included replacing an unfamiliar term, correcting typos, and appending a commonly-used acronym.

The Link tool was generally used to shorten navigational paths. For example, five different participants linked 5 static course titles on a degree requirement page to their respective course information pages, after they found it took too many clicks to check course information from the requirement page.

Usage of the Tooltip tool was more varied and interesting. Most of the tooltips were added to links in order to help users decide whether or not to click on them. First, tooltips were created for a link to hint what information could be expected in the linked page. For example, P7 attached the following tooltip to the "guest speakers" link on the Media page: "job candidate talks are accessible through this link." Second, tooltips were used to clarify community-specific jargon. For example, P3 added the following tooltip to explain "faculty guest lectures" as "another way of saying 'job talks'." Lastly, P1 and P10 used tooltips to give specific directions to aid navigation. Fig. 3 shows a tooltip P10 created to direct users to other course schedule viewing options that were hard to find due to the poor information architecture.

Help Requests. Though the remaining 20 problems were not "overlaid" with page modifications, they were identified via 33 help requests from the SO evaluation participants. When did participants choose to submit a request rather than create an overlay? We found that 10 of those unresolved problems fell out of the scope of tools like SO. They were related to backend issues, missing content, or missing a feature. Among the remaining 10 unresolved problems, 3 of them could possibly be repaired with the current version of SO, while the other 7 issues could potentially become repairable using an improved version of SO outlined in the Discussion section.

We also examined those requests' quality, placements on the page, and intended audiences. First, close to 90% of help requests included specific recommendations for improving the website, which suggests that participants generally had pretty clear ideas about how the problems they identified could be resolved.

Second, participants appear to be able to find an appropriate element to associate their requests to. Among the 33 requests submitted, 21 requests were attached to a specific page element, while 12 went into the "bucket" as page-wide issues or general comments. Most requests were attached to an appropriate element that helped us understand what the issues were.

Third and finally, many requests appeared to be added in the hope that webmasters would fix them from the server. However, there were also 4 requests intended to go to other community members, asking whether a change was accurate or dividing up the work of making a series of similar changes.

Quality of Community-Generated Usability Enhancements. Did the participants in the SO evaluation make helpful changes? Our analysis suggests that at least some user-generated overlays made subsequent participants more efficient in solving tasks. For example, one of these helpful overlays was P1's tooltip added to the "Guest speaker series" link on the "Events and News" page, as Fig. 4 shows. The tooltip explained what was in the linked page, which was linked to the faculty candidate talks that participants were asked to find in Task 1. The mouse click counts in Fig. 5 shows that Task 1 became substantially easier after P1 created that tooltip. The participants made helpful overlays to simplify other tasks as well. For example, after completing task 3, P7 linked a course title to a page that provided additional information about the course, and then P12 came across it and said, "So someone added a link. Oh man, it doesn't tell me if it has PEP credits or not. So someone helpfully probably put this link in. [Clicking the link]. That was helpful, that was totally helpful."

However, not all overlays were well crafted. For example, there were tooltips added to clarify the difference between the course list and the course catalog, but apparently these tooltips still lacked detail that would help P12 pick one course page over another. Although the quality of the user-generated usability enhancements varied, they rarely made the website harder to use.

In summary, we found that Social Overlays could lead to a substantially improved website. However, this improvement was not uniform, and much of the improvement showed satisficing behavior.

Differences between Social Overlays and Standard Usability Methods

To assess and contextualize the problems identified by participants in the SO evaluation, we first compare them with the results of the expert inspection and then with those of the usability test. Specifically, we went through the list of issues reported in the SO evaluation and checked if each of them was covered by the expert inspection or the usability test.

To our surprise, participants in the SO evaluation documented 52% more problems than the 31 problems reported by the expert inspectors. Only 12 out of the 47 problems documented in the SO evaluation were also identified by the experts (see Fig. 6). How could the participants in the SO evaluation find these 35 additional problems? As members of the community, they appear to have advantages in three aspects, based on our analysis:

First, the community members in the SO evaluation leveraged their lived experiences in the organization during problem identification. For example, P3 pointed out Fig. 6. The above bar charts show that community members using Social Overlays reported not only more but also different usability issues than the expert inspectors and the usability testing team. We describe those differences in section 4.2.

that the Tracking and Planning Sheet, a useful PDF file that she could print out and use as a resource in completing task 4, was buried too deeply in the site. She was aware of this problem, because of her prior experience in planning courses as a master's student at UXSX.

Second, community members uncovered similar problems that occurred in different places in the website, since as a whole, they were exposed to more pages and explored different paths than the group of experts.

Third, the community members were able to project the needs and preferences of the sub-community to which they belonged. For example, two masters' students, P4 and P11, believed that other students would want to find student club events in the school's official Events page.

Nevertheless, the SO evaluation participants did not identify 19 problems that were found in the expert inspection. It appears that the experts were able to spot these additional issues by following general principles and heuristics. For example, the experts were sensitive to inconsistencies on the website such as showing Prof. A's profile on the same page of a course taught by Prof. B. The experts also seemed to evaluate the site following best practices in usability. For example, E1, E2, and E3 mentioned that the website failed to visually separate groups of content on several pages.

Next, we compare the results of the SO evaluation with those of the usability test. The usability testing team reported 31 problems, some of which were broad statements and covered multiple narrower problems documented by the participants in the SO evaluation. In total, the usability test covered 22 of the 47 problems documented by the participants in the SO evaluation (see Fig. 6), representing a larger overlap than that between the SO evaluation and the expert inspection. This increased similarity is not surprising, since usability testing could also access community members' lived experience and local knowledge about the site.

But why did SO users report 25 problems that were not found by the usability testing team? From our analysis, there were two main reasons. First, participants showed empathy with their peers in the UXSX community, pointing out 12 issues that did not prevent themselves from solving tasks but they believed could be frustrating or confusing to less experienced peers. For example, P3 found it a potential problem that the acronym "PEP" was not linked to its full description page, though she already knew what PEP stood for ("Practical Engagement Program"). Second, participants in the SO evaluation pointed out 7 more issues related to missing information or features that they expected to see on the site.

In short, we found that community members reported more problems compared with usability experts and the external usability testing team, and that their reported problems differed in systematic but useful ways from the results generated by the expert inspection and the usability test.

Community Processes in SO

We also wanted to assess whether SO would show preliminary evidence of community activity. The most important community activity, as mentioned in section 4.1, was getting assistance from the webpage improvements made by other users. We also observed community members giving feedback on the overlays made by others. Six of our participants used the voting feature of SO, and they gave 15 thumbs-up and 6 thumbs-down to overlays made by prior participants. We did not observe conflicting votes on any of these changes. Other users verbally commented on the helpfulness of other users' overlays, but they did not click the voting button in SO. Four of the 6 thumbs-down were given to point out errors made by P8 and P9, while the other 2 thumbs-down were cast by P7 and P9, when they disliked existing overlays and sought to replace them with one of their own. We also observed some evidence of social dysfunctionalities that might become more critical in larger communities or with more use. P4 showed self-serving tendencies, declaring, "I would make that a link, but it would be motivated by my convenience and not out of altruism for other users." Another participant did not like all of the changes, but did not revert or modify those changes. (This might also be tacit approval of the change's sufficiency, however.) As well, participants did not vote as much as we hoped. We will return to these issues in the Discussion section.

Subjective Perception of the Utility of SO

Participants in the SO evaluation commented on what aspects of SO they found particularly helpful. First of all, participants liked the ability to immediately see the change they made using SO. Many of them were delightedly surprised when they saw problems repaired instantly. Second, participants liked that SO provides peripheral awareness of community activities through the indicators on the side margin (see Fig. 2), as P5 mentioned, "I really like the scroll bar [indicators] that change colors. That's the first thing I looked to, besides the blinking." Third, participants appreciated that the changes they made were shared with other community members. P3 enthusiastically commented, "It can only be in the long term a very big asset to the community, especially the social element ... where people vote up and down changes."

In summary, the above results of the SO evaluation and our comparisons with expert inspection and usability test show that:

1. With Social Overlays, community users identified a substantial number of problems as they interacted with the site. They were able to repair many of them by creating overlays helpful to others and address the rest by submitting constructive requests. 2. With Social Overlays, community users uncovered problems that existed along multiple paths of browsing and brought to the problem identification and solving process their lived experience and ideas that only members of the community would have. 3. With Social Overlays, community users seemed to take on a more active role in reporting UI problems than the participants in the usability test. Our findings argue that Social Overlays is useful and feasible for communities, as least those similar to UXSX in size and cohesiveness, to collectively identify and address usability problems on their website. Moreover, the results of using SO were comparable to standard methods that are more expensive, such as expert inspection and usability testing, though each of the three approaches showed different strengths and priorities in our study.

Discussion

To summarize, this paper:

• Introduced a new community-based approach for addressing web usability problems without requiring expertise in usability evaluation or interface design. • Presented a new system called Social Overlays that embodies this approach of collaborative usability improvement and demonstrated its technical feasibility. • Provided an evaluation that showed that Social Overlays and the approach it enables are useful and feasible for an important set of communities and websites. Despite SO's success, we found five issues with it and the approach in our evaluation. First, our evaluation showed that community users grounded problems in their lived experience inside their community, covered a larger number of web pages, and made quick responses to UI issues. This, we have argued, is critical for many organizations without access to usability professionals. However, we also saw that users using SO might need help formulating holistic and broad redesign recommendations. An interesting question we would like to explore in future work is how SO can be used with some lightweight expert involvement, e.g. with an expert synthesizing or gatekeeping modifications proposed by users.

Second, although our evaluation argued that SO was likely to be helpful for communities with a few dozen to several hundred members, there are many other types of communities with many more members, less social cohesion, and potentially less trust. Addressing these communities will require additional functionality. Some public-facing websites are subject to spam, problematic content, and attacks, requiring more security mechanisms. Larger or more public websites and communities may suffer more motivation issues, although they also potentially have more users for SO. All of these issues remain to be addressed in future work, and we plan to do so in subsequent efforts. Nonetheless, we were pleased that SO appears to be able to work in at least an important subset of small-to-medium communities. The same type of community exists in many educational settings, community organizations, charities, and the like. Nonetheless, we will not know all the issues until this tool is released into the wild and used by other sites and communities.

Third, the current version of SO provided only three simple modification tools. Even with these limited capabilities, our participants were able to address common and important usability problems with ease: confusing labels and vocabulary issues, broken and cumbersome navigational paths, and unclear site functionality. We believe, based on our evaluation, that additional tools would make SO even more helpful. We are currently investigating capabilities to insert new links into a page, convert the highlighted part of text into a hyperlink, and support the navigational use of tooltips with better hyperlinking, and address higher-level issues of page and site organization. We are also examining how to address potential information overload problems that would arise if SO were to scale, including new awareness visualizations and mechanisms for issue consolidation.

While it is technically possible to include more powerful and sophisticated tools such as a full-fledged style editor, we believe this would run the risk of having naïve users freeze or make too many mistakes. We also plan to examine this risk in future studies.

Fourth, based on even our limited study, we suspect there will be resulting problems as SO use grows in a site. As the number of users grows, it will increasingly be a concern that different subcommunities inhabit different language worlds, and individuals may wish to place markers for themselves that would be inappropriate for the entire community. We are currently investigating mechanisms to allow both subcommunities and individual personalization.

Finally, as mentioned, we had less usage of the voting feature than we had hoped. In our observations, non-voting behavior often implied agreement, and there were very few overlays or requests that were perceived negatively by participants. We expect both the use and the utility of the voting feature to increase as more users participate in an SO deployment; however, we are investigating new user interface designs to make voting easier and more meaningful.

We must also acknowledge the many limitations in the current study. Our tasks were artificial and scoped, as with any lab-based study. As well, the study was shortterm and used selected pages. As mentioned, we will not know all the issues until SO is released into the wild and used by other communities-work that is presently ongoing. The current system and evaluation reported here, however, has led us to believe the usefulness and stability of SO are such that such explorations can fruitfully occur.

6

Related Work

The SO system and the approach it enables, as presented above, aim to tackle the challenge of post-deployment usability which is important but has been undervalued in the user-centered design practice [START_REF] Chilana | Post-deployment usability: a survey of current practices[END_REF][START_REF] Wagner | Moving beyond user participation to achieve successful IS design[END_REF]. A number of commercial offerings have sprung up that facilitate the collection of user feedback on deployed websites, either through web forms (e.g., Feedback Army 2 and UserVoice 3 ) or message boards (e.g., SuggestionBox 4 ). Additionally, web annotation systems (e.g., Diigo 5 and AnnotateIt 6 ) could potentially be employed to obtain user feedback, though most of them are focused on learning and information management. It is evident that, while being reasonable solutions for collecting user feedback, all of these tools require users to depend on website supporting staff to assess the severity of the problem, design a solution, and implement the change. In contrast, SO addresses usability by enabling a website's user community to not only identify problems but also repair many of them immediately. To design a system like SO, we draw on three bodies of technical work: webrewriting systems, collaborative web accessibility tools, and community-based help systems. We will describe each of them in turn.

First, we build upon web-rewriting systems that allow users to alter the design or behavior of a webpage at runtime. General-purpose web-rewriting systems like Chickenfoot [START_REF] Bolin | Automation and customization of rendered web pages[END_REF] and GreaseMonkey [START_REF] Mcfarlane | Fixing Web Sites with GreaseMonkey[END_REF] support such rewriting, and they have online repositories whereby users can share, discover, and make use of scripts created by others. However, the versatile capability of these systems creates difficulties for most users, since many users do not know how or want to inspect, edit, and debug scripts. SO leverages some of the same technologies used in web rewriting systems (e.g., JavaScript injection), but SO nonetheless has been designed specially to allow naive users to contribute.

Second, collaborative web accessibility tools (e.g. AccessMonkey [START_REF] Bigham | Accessmonkey: a collaborative scripting framework for web users and developers[END_REF] and Social Accessibility [START_REF] Takagi | Social accessibility: achieving accessibility through collaborative metadata authoring[END_REF]), represent a specialized class of web-rewriting systems-one that shares SO's high-level goal to make websites easier to navigate and use. Accessibility systems, though, are focused on helping particular subsets of users who share particular disabilities. For example, the Social Accessibility system [START_REF] Takagi | Social accessibility: achieving accessibility through collaborative metadata authoring[END_REF] allows volunteers to enter missing metadata that can then be consumed by visually impaired users who subsequently visit the augmented sites using screen readers. Though the underlying technologies that enable SO and Social Accessibility are similar, the sociality afforded by the SO approach is fundamentally different, because in SO both producers and consumers of collaborative usability enhancements belong to the same group. Our study has shown that many usability problems can best be identified and addressed by users who share lived experience relevant to the community of the site.

Therefore, the sociality supported by SO is closer to that studied in the third body of work we draw upon: community-based help systems. These systems assist users by providing information generated by other members of the community. Communitybased help systems, such as Answer Garden [START_REF] Ackerman | Augmenting the organizational memory: a field study of answer garden[END_REF], QuME [START_REF] Zhang | QuME: a mechanism to support expertise finding in online help-seeking communities[END_REF], IP-QAT [START_REF] Matejka | IP-QAT: in-product questions, answers, & tips[END_REF], and Lem-onAid [START_REF] Chilana | LemonAid: selection-based crowdsourced contextual help for web applications[END_REF], have focused on making it easier for users to ask questions and find answers, and these systems have shown resoundingly that users can collectively create useful information. Answer Garden builds a store of questions and answers, but accessing it requires using a separate system, resulting in a potential distraction from the core tasks users are trying to accomplish. IP-QAT and LemonAid associate questions with related UI elements, and display answers as part of the system for which help is sought. IP-QAT and LemonAid, however, do not seek to actually improve the user experience of the system by effecting design changes, as SO does, which has the additional benefit of reducing the need for seeking help in the future.

In short, none of the above systems are focused on post-deployment usability assessment and correction, though we draw inspirations and learn lessons from them to inform different aspects of SO design. By specifically designing for the collaborative work around post-deployment usability, Social Overlays provides new capabilities of community interaction and assistance.

Conclusions

In this paper, we presented the Social Overlays system (SO) and its evaluation. SO enables end-users to identify and repair common user interface problems on a website by making "overlays" that rewrite specific page elements. Moreover, SO displays page modifications to others who also have this extension installed.

We also reported findings from an evaluation of Social Overlays. In our study, we found that a group of community members without any usability training could use SO to identify and fix UI problems on their medium-sized community's website. We also compared the results from those community members to the assessments of usability experts as well as the results of a usability test, and found that SO produced a larger number of issues, and that these issues partially overlapped with those found through standard methods. To summarize, we found that Social Overlays can be a viable approach. By having users improve a website's usability as a part of their use of that site, a community can collectively make their website more usable.

Fig. 1 .

 1 Fig. 1. Social Overlays has a three-step process of making modifications to an existing web page: a) select a modification type, b) select a page element,and c) specify the modification. For example, the user in this scenario renames a mislabeled button from "Apply" to "Search."

  2). Each indicator corresponds to an element that has community-generated overlays or requests, and they are visually aligned on the same line. When the user hovers over an indicator, its associated element will be highlighted. The color of the indicator shows the status of the element. A green indicator signals that there are existing overlays for the element, while a red indicator signals that there are active requests to fix this element. If an element has both overlays and unresolved requests, a yellow indicator is displayed.If the user wants to see the original version of the element, clicking on the indicator will toggle between the original version and the community-enhanced version. He

Fig. 2 .

 2 Fig. 2. The Social Overlays browser extension can be accessed by clicking its icon on the browser's toolbar. The icons from left to right represent the Text tool, the Link tool, Tooltip tool, and the Help button. The green, red, and yellow indicators on the right margin are used to indicate overlays, help requests, and both overlays and requests, respectively, made on a page element on the same horizontal line. The bucket at the right-bottom corner is a repository for page-wide issues.

Fig. 3 .

 3 Fig. 3. P10 added a tooltip to indicate that there are other options to view the class schedule (e.g. by week) on the page linked to the "Courses" header.

Fig. 4 .

 4 Fig. 4. P1 added a tooltip to the "Guest speaker series" link to indicate that users can find faculty candidate talks in the linked page. This tooltip significantly reduced the number of clicks taken by subsequent participants once they reached this page.

Fig. 5 .

 5 Fig. 5. The number of mouse clicks users made decreased dramatically after P1 added a tooltip to clarify a critical link on the Events & News page. Participants who did not take this route were omitted from the figure.

http://developer.chrome.com/extensions/

http://www.feedbackarmy.com

http://www.uservoice.com/

http://www.suggestionbox.com/

http://www.diigo.com/

http://annotateit.org/

Acknowledgements

This work was supported in part by the National Science Foundation (IIS 0905460), the University of Michigan Rackham Graduate School, and kynamatrix Research Network. We also thank Sonali Mishra, Pei-Yao Hung, and Chuan-Che Huang for their help with this study.

Appendix I: Evaluation Tasks


Sandy J J Gould
email: s.gould@cs.ucl.ac.uk

Anna L Cox
email: anna.cox@ucl.ac.uk

Duncan P Brumby
email: brumby@cs.ucl.ac.uk

Using Graphical Representations to Support the Calculation of Infusion Parameters

Keywords: Graphical reasoning, infusion pumps, re-representation, calculation

A variety of medical procedures require arithmetic calculations to be performed. These calculations can be complex and induce errors that can have serious consequences on the ward. In this paper, we consider whether a graphical representation might make these calculations easier. The results of a laboratory experiment are reported in which participants were asked to solve a number of infusion parameter problems that were represented either graphically or textually. Results show that participants were faster but no more accurate in solving graphical problems than they were textual problems. We discuss the need for situated work to be conducted that builds on these initial findings to determine whether the advantages of graphical representations transfer to actual workplace settings.

Introduction

Many tasks on hospital wards require nurses to perform calculations involving rates, volumes and times (e.g., setting up syringe drivers and infusion pumps). Unfortunately, there is a large body of evidence showing that nurses and trainee-nurses have poor arithmetic skills [START_REF] Bindler | Medication Calculation Ability of Registered Nurses[END_REF][START_REF] Mcmullan | Patient safety: numerical skills and drug calculation abilities of nursing students and Registered Nurses[END_REF][START_REF] Harvey | Diagnosing the problem: Using a tool to identify pre-registration nursing students' mathematical ability[END_REF]; as a result, calculation errors are occasionally made [START_REF] Schulmeister | Chemotherapy medication errors: descriptions, severity, and contributing factors[END_REF]. This has implications for the quality of care patients receive.

A number of approaches have been taken to improving calculation ability. These have ranged from traditional pedagogic efforts [START_REF] Harne-Britner | Improving medication calculation skills of practicing nurses and senior nursing students: a pilot study[END_REF] more teachingto the development of interactive learning environments [START_REF] Hutton | Numeracy for nursing, report of a pilot study to compare outcomes of two practical simulation tools -An online medication dosage assessment and practical assessment in the style of objective structured clinical examination[END_REF]. Although educational programmes do much to address arithmetical shortcomings, they provide little insight into whether the traditional textual layout of information used during calculations is best suited to the task that nurses face.

An alternative approach that we explore here is to radically change the presentation of the calculation information from textual to graphical. It has been argued that graphs might make the calculation of infusion parameters easier, thus reducing the likelihood of error [START_REF] Nemeth | The Infusion Device as a Source of Healthcare Resilience[END_REF]. However, there has been little work to date that has evaluated whether people are better at solving these kinds of problems when presented with graphical representations than they are with textual representations.

The aim of this paper is to determine whether a graphical representation makes it easier for people to make calculations about infusion parameters. We report the results of a lab-based experiment in which participants were given a set of calculation problems, represented either graphically or textually, along with some questions to answer. We wanted to know whether the graphical representations would allow participants to achieve the same or better levels of accuracy in less time than was possible with textual representations.

Related Work

Performing calculations is a major component of many nurses' routines. For example, infusion pumps, which are used to deliver medication intravenously to patients, need to be programmed with parameter values, usually infusion duration, volume and rate. Sometimes all of these values are given on a prescription or medication bag so they can be easily programmed into the device. However, sometimes there are missing values that must be calculated before they can be programmed [START_REF] Furniss | Unremarkable errors: low-level disturbances in infusion pump use[END_REF]. Even when all of the values are provided, infusions can occasionally be disrupted midway through (e.g. because of a battery failure [START_REF] Furniss | Unremarkable errors: low-level disturbances in infusion pump use[END_REF]). Such failures require new values to be calculated and then programmed into the pump so that the infusion can be resumed. Reports show that many nurses find these calculations difficult and more worryingly often overestimate their ability to perform them [START_REF] Eastwood | Numeracy skills of nursing students[END_REF]. Hence there is an opportunity to see whether modifications to environmental artefacts might facilitate accurate calculations.

Previous work [START_REF] Thimbleby | Using nomograms to reduce harm from clinical calculations[END_REF] has experimented with nomogramsone dimensional slide rule-like representationsto make such calculations easier. Empirical results indicate that these representations can significantly reduce the number of errors made in calculation tasks. Although nomograms ease calculations, they do not provide a representation of the complete time course of the infusion, or its current state, which may cause difficulties when restarting disrupted infusions (e.g., because of a battery failure). It has been proposed [START_REF] Nemeth | The Infusion Device as a Source of Healthcare Resilience[END_REF] that representing infusions on two dimensional charts might make it easier for nurses to reason about the setup of infusions, but this proposal has not been tested empirically.

The advantages conferred by re-representing information have been of interest to the areas of cognitive science and human-computer interaction for some time. Larkin and Simon [START_REF] Larkin | Why a Diagram is (Sometimes) Worth Ten Thousand Words[END_REF] describe the properties of informational and computational equivalence of representations: representations are informationally equivalent if they are constructed from the same underlying data; they are computationally equivalent if inferences can be drawn with equal speed and ease. Zhang and Norman [START_REF] Zhang | Representations in distributed cognitive tasks[END_REF] extend these definitions from a distributed cognition perspective, finding that multiple representations of the same task can afford different degrees of external representation, which in turn affects problem-solving efficacy.

External representations are not necessarily diagrammatic and there is a subset of research concerned specifically with reasoning about representations that use Cartesian co-ordinates (i.e. a chart or graph). Peebles and Cheng [START_REF] Peebles | Modeling the Effect of Task and Graphical Representation on Response Latency in a Graph Reading Task[END_REF] provide an empirical validation of the equivalencies described previously by Larkin and Simon [START_REF] Larkin | Why a Diagram is (Sometimes) Worth Ten Thousand Words[END_REF] in the context of chart-based reasoning. In their study, they compared informationally equivalent graphs and showed that constructing graphs according to particular psychovisual properties can result in reasoning efficiencies that can mitigate participant unfamiliarity with a particular representation. However, this does not seem to apply universally: in some cases where participants were required to reason about equivalent representations, familiarity both with the topic and with graphs is a significant factor in interpretation performance [START_REF] Shah | Bar and Line Graph Comprehension: An Interaction of Top-Down and Bottom-Up Processes[END_REF]. To reduce the effect of familiarity on reasoning performance, this study uses simple two-point graphs, and participants are not required to perform interpretation; instead, they only need to retrieve calculation parameters from the graph. Despite the extensive work in the area of graphical representations, we are unaware of any comparative experimental studies of the performance characteristics of graphical and textual representations of the same information.

We describe an experiment that investigates whether people are able to perform better in resolving arithmetical problems similar to those that nurses perform for infusions when given graphical representations compared to traditional textual representations. The representations used are informationally equivalent: each problem is represented twice, once in graphical form and once in textual form. However, we hypothesise that the representations are not computationally equivalent: using them requires different degrees of effort. Building on the suggestion that graphical representations might make infusion problems easier to deal with [START_REF] Nemeth | The Infusion Device as a Source of Healthcare Resilience[END_REF], we predict that graphical representations will be less computationally intensive (i.e. easier to use) than textual representations. If graphical representations are indeed less computationally intensive, participants should be able to complete the problems more quickly and more accurately than they can with textual representations.

Method

Participants

Twenty-one participants (11 female) with a mean age of 24 years (SD=4 years) took part in the study. Participants were recruited from the psychology subject pool at University College London and were paid £10 for approximately one and a half hours of their time.

Materials and Design

The experiment was made up of 22 infusion problems. These problems specified the total volume to be infused (i.e. the starting volume), the total duration of the infusion, the time elapsed so far and the total volume remaining. Each of these problems was presented twice during the course of the experiment, once textually and once graphically (see Fig. 1). Participants were not informed that the same problems could appear more than once in different forms.

The textual representations (shown on the left of Fig. 1) give the rate of infusion, the total volume (VTBI), the volume remaining and the time elapsed. This representation is typical of the interface of many infusion pumps that are currently manufactured. The graphical representations use the same information, only plotted on a chart. Although the information differs in representation, the total information content is identical. All graphical representations were displayed on the same scale from 0-1000ml/h and 0-120 minutes. In order to encourage calculation, the graphs were not drawn accurately enough to compute answers directly from the graphs by reading values from the axes.

Underneath the representation of the problem, participants were given either one or three questions on the problem, depending on the phase of the experiment. These problems required participants to retrieve information from the representations and then perform a calculation. For example, for the problem shown in Fig. 1, participants might be asked "How many minutes remain in this infusion?" To come to the correct answer -42participants would need to subtract the time elapsed from the total infusion duration. More complex questions might ask, "If the rate of infusion was halved at this moment, how much longer would the infusion take?" Participants answered the questions associated with the problem and then clicked a button to proceed to the next problem. Participants were not given any feedback on their answers.

In addition to the calculation task, participants were also required to complete a cognitively demanding secondary interrupting task. This was a modification of the nback task [START_REF] Kirchner | Age differences in short-term retention of rapidly changing information[END_REF]. Participants were shown a two-by-two grid. Every three seconds, one of the cells was filled in either blue or red. This was a two-back task so participants had to decide whether the current arrangement matched the arrangement of colour and position they were shown two arrangements prior. The interruptions lasted for 30 seconds after which participants were returned to the primary task.

Finally, an eight item post-experiment questionnaire was also devised. This asked participants about the extent to which they found different aspects of the experiment difficult. Participants indicated their agreement with a statement on a five-point Likert scale. This study used a counterbalanced within-subjects design with a single independent variable, type of representation, which had two levels, graphical and textual. The dependent variables were answer accuracy; time to complete each problem; and time to answer the first question in a problem after it was presented on the screen.

Procedure

After reading an information sheet, watching an introductory video and giving consent, participants started working through the training phase. Participants were presented with eight training problems, four each for the graphical and textual representations. Each problem came with three questions, the majority of which required participants to find the appropriate information in the representation and transcribe it into a text field. For example, for the problem in Fig. 1, participants might be asked what volume remained to be infused or how much time was left. Participants completed all of the problems with the assistance of the experimenter. If participants demonstrated satisfactory competence in answering these questions they continued to the experimental trials, which consisted of two phases. The first experimental phase presented participants with a single question per problem. Participants were given the same problem twice, once with the graphical representation and once with the textual representation. There were a total of nine problems in this first phase, giving a total of 18 trials, two for each problem. The order of presentation was counter-balanced.

The second experimental phase used a different set of nine problems. In order to accommodate interruptions, problems in this phase comprised three questions. Interruptions occurred after the participant had started working on the problem. More specifically, interruptions could occur in one of two places: either before the second question or before the third question. There were a total of 18 trials in this phase, nine of each type of representation. Participants alternated between the two types of representation (i.e. a graphical problem followed by a textual problem) and this order was counterbalanced across participants. Participants were given access to a pen, paper and calculator for the duration of the study and told that they were free to do as they wished with them.

Results

Four participants were unable to complete all questions in the time available, so their data were excluded from the analysis. Participants answered a total of 1224 questions, 306 in the first phase and 918 in the second. Of the 1224 questions, 745 were answered correctly giving an answer accuracy rate of 61%. Of the correct answers, 388 came from graphical problems (per participant, M=23, SD=8) and 357 came from textual problems (M=21, SD=8). Representation type did not have a significant main effect on the number of correct answers t( 16)=2.06, p=0.06.

As well as accuracy, we were also interested in the speed at which participants were able to complete the problems. Problems from the first phase were used for this purpose because there was only one question associated with each problem. This allowed us to eliminate any effects of guessing, by examining the time spent on a problem for correct answers only. On average, participants took longer to complete a problem using the textual representation (M=69s, SD=33s) than with the graphical representation (M=48s, SD=20s). This difference was significant1 , t(15)=2.24, p<.05.

We were unable to meaningfully evaluate the time spent on problems in the second phase, because only accepting problems with three correct answers made for too small a sample. Therefore, the final measure we were interested in was how long it took participants to start answering the first question in a problem. To do this, we measured the time between the presentation of the representation and the first keypress in the answer field. (First click was not used because participants would often click the answer field and then work out the answer.) Including only correct answers, we found that participants took significantly longer to calculate their first answer when given textual representations (M=44s, SD=23s) compared to when they were given graphical representations (M=27s, SD=11s), t( 16)=3.43, p<.01. We do not consider the effect that interruptions had on performance as this is beyond the scope of this article.

The post-experiment questionnaire asked participants to rate their agreement with eight statements about the task. The first three questions in particular were of interest as they asked participants about their subjective sense of the difficulty of the experiment as a whole and the two different kinds of representation. The modal response to the first question "The task I was given was difficult" was 'Agree'. Participants responded 'Neutral' to the second question: "Problems with text representations were harder than problems with graphical representations." There was no single modal response for the third question. "Problems with graphical representations were harder than problems with text representations." The mode was split between 'Disagree' and 'Neutral'.

Discussion

The results of the experiment suggest that graphical representations may confer some performance advantages over textual representations when calculating infusion parameters, giving support to previous proposals (e.g. [START_REF] Nemeth | The Infusion Device as a Source of Healthcare Resilience[END_REF]) and confirming that rerepresenting calculation problems can improve performance (e.g. [START_REF] Thimbleby | Using nomograms to reduce harm from clinical calculations[END_REF]). Although the effect of representation on answer accuracy was not significant, there was a trend toward superior accuracy in graphical problems. When interpreting this trend it should be noted that participants completed each problem twiceonce for each representationmeaning that the difficulty of the problems could not have affected how accurate participants were in their answers.

Stronger evidence for the performance benefits of graphical representations is provided by response time data. These show that participants completed single-answer problems more quickly when presented with graphical representations and answered the first question more quickly in multiple-question problems. Even if participants eventually came to the same answer with both representations, they did so more quickly using the graphical representation. These differences in response time support the idea that these representations, while informationally equivalent, were not computationally equivalent; participants' accuracy may have been knowledge-constrained (i.e. by their arithmetic skills), but graphical representations allowed them to operate within these constraints more effectively. In a time-pressured environment, small time savings that have no negative effects on standards of care might prove valuable.

Our results indicate that as well as providing a small increase in accuracy, graphical representations deliver a significant increase in efficiency. Participants' feedback in post-experiment questioning revealed that they felt that graphical representations were somewhat less difficult to use than textual representations; this is despite the fact that they still had to calculate the answer themselves (i.e., they could not just read the answer from the graph). Overall the results of this study demonstrate that graphical representations may confer some advantages to support the calculation of infusion parameters over traditional textual representations of these problems.

In some ways the experiment underutilises the features of graphical representations. For instance, some medications need to be delivered with a loading dose. This requires a high rate for a short period at the beginning of an infusion, followed by a slower infusion for the remainder of the prescription. These kinds of infusion could easily be represented on a graphical representation: the gradient of the line would change to reflect the changing rate of infusion. Conversely, the simple textual representation in this study provides no mechanism for presenting such an infusion in an easily digestable manner. Moreover, graphs were drawn inaccurately so that participants could not compute answers directly from the graphaccurate graphs might have provided more scope for participants to reason about the problems visually. Graphical representations could be embedded into future infusion devices, where they could provide dynamic indication of infusion progress (e.g. [START_REF] Nemeth | The Infusion Device as a Source of Healthcare Resilience[END_REF]), or their benefits could be realised by printing them on medication bags and prescriptions alongside (or instead of) textual representations.

The extent to which graphical representations would be useful on wards hinges on generalisability of the results of this study. We used a lab-based experimental investigation because we wanted to see if representational differences would affect performance in a group of participants who were unfamiliar with either kind of representation; testing the interfaces on nurses will be difficult because they will likely be accustomed to the traditional textual representation. Future work should investigate whether calculation performance improves with graphical representations of infusion by introducing graphical representations to nurse training programmes. Our results, together with previous work which demonstrates that visual representations have advantages over textual representations in pedagogic situations [START_REF] Ainsworth | The effects of self-explaining when learning with text or diagrams[END_REF], suggest that graphical representations could have positive effects for trainees, where textual representations do not have the familiarity advantage they currently have in hospital wards.

Fig. 1 .

 1 Fig. 1. Examples of the textual (left) and graphical representations used in the experiment.

One participant was excluded from this test because they produced no correct answers to Phase 1 textual problems.

Acknowledgements. Gould, Cox and Brumby are funded by the CHI+MED project, EPSRC grant EP/G059063/1. Thanks to Sarah Wiseman for her comments on this work.


Andreas Sonnleitner

Marvin Pawlowski

Timm Kässer

Matthias Peissner

Experimentally Manipulating Positive User Experience Based on the Fulfilment of User Needs

Keywords: User Experience, UX, User Needs, User-Driven Innovation

), two variations of a neutrally designed tool were developed to address two selected needs (popularity, competition) by slightly changing functionalities or design elements. The manipulation of UX is validated with realtime and retrospective subjective evaluation of UX, and objective data of user behaviour. Participants rated significantly higher positive UX and showed more active behaviour for designs of the prototype addressing the user needs competition and popularity compared to the neutral design. These findings show the importance of considering elements of UX in the early development process of technological products.

Introduction

User Experience (UX) is considered a key factor for the success of interactive products and systems. Looking at the definition of UX, it is "all aspects of the user's experience when interacting with the product ... it includes all aspects of usability and desirability of a product ... from the user's perspective" [START_REF]Ergonomics of human-system interaction -Part 210: Humancentred design for interactive systems[END_REF]. Next to this definition, various different interpretations exist due to the multidisciplinary nature of UX, ranging from a more psychological to a business perspective [START_REF] Roto | User Experience White Paper -Bringing clarity to the concept of user experience[END_REF].

Our approach is based on the fulfilment of user needs during interaction with technological products derived from models describing general human needs [START_REF] Sheldon | What's satisfying about satisfying events? Comparing candidate psychological needs[END_REF][START_REF] Reiss | Multifaceted Nature of Intrinsic Motivation: The Theory of 16 Basic Desires[END_REF]. The emphasis is placed on positive UX by meeting basic needs which is critical to the long-term success of technological products. Products that generated a lot of positive experiences will be used more frequently and will therefore be preferred [START_REF] Kahneman | Objective Happiness[END_REF]. In the context of work environments, UX can have a positive impact on skills and mental health and can also cause for motivated action, increased productivity and job satisfaction [START_REF] Harbich | Using behavioral patterns to assess the interaction of users and product[END_REF]. The developed model in Fig. 1 describes a holistic view of positive UX and will be taken as a basis for generating positive UX for interfaces. In addition to the final evaluation of a product regarding attitude and behaviour of users, also the experience itself and the cause for the use of the product are included in the detection of UX. Usability, directness and comfort are integrated as moderators.

The aim of this study is to specifically address user needs and to experimentally manipulate UX in a work environment, simulating both a more creative phase of idea generation, as well as a blunt execution of tasks. For this, an idea management tool is provided with differently shaped functionalities and design elements illustrating personal performance measures, idea evaluation and received achievements. In a first step, we decided to investigate two user needs with small overlap: Competition, i.e. to be better and outplay others, and popularity, i.e. to act dutiful and be liked by others. The influence of various functionalities and design elements on user behaviour as well as on real-time and retrospective subjective evaluation of UX was measured. The research questions investigated in this study are:

 Does the specific fulfilment of user needs lead to a higher UX?  Can experimentally crafted functionalities and design elements satisfy the addressed user needs competition and popularity?  Do experimentally crafted functionalities and design elements have an influence on user behaviour?

Methods

Participants

In the pilot study nine participants were involved to test first conceptual designs (18-74 years, mean: 35.9 years, 4 females). The sample of the main study consisted of 20 participants (18-70 years, mean: 38.5 years, 11 females, mean office working experience = 15.0 years). No subject participated in both studies.

Subjects were recruited from an in-house database in which volunteers for experiments are listed. Participation was voluntary, data were collected anonymously. Informed consent was obtained after the task had been explained. Participants were informed they had the option to end participation at any time without any type of penalty. After the study, they received 30€ for participation.

Pilot study

As a result of wireframes and user centered paper prototyping, clickable prototypes of the fictional idea management tool were used to get information about elements most perceived by the participants and about user needs addressed particularly. With a combination of Valence method [START_REF] Burmester | Valence Method for Formative Evaluation of User Experience[END_REF] and Laddering technique [START_REF] Reynolds | Laddering Theory, Method, Analysis and Interpretation[END_REF], emotions and needs were associated with functionalities and design elements. These results were used to improve the final design variations by focusing on salient elements with a high effect on user's perception (see Tab. 1).

Table 1. Key differences for designs of the prototype (neutral, popularity, competition).

Main study

Setting. To show coherences between design, functionality and UX, three prototypes of an idea management tool were developed based on Adobe Flash. Since the basic prototype is designed as a neutral one, the other two designs focus on fulfilling selected user needs (popularity, competition). Tab. 1 exemplarily shows some of the key differences of specific design and functionality characteristics. The aim of the study is to validate the assumption that slightly changing design and functionality of elements has an impact on the fulfilment of both user needs, and therefore has an impact on UX. The task is to brainstorm ideas to different topics in a team with four other simulated users that give input by a predefined script identical for all participants. Besides continuously brainstorming ideas to a total of nine topics (three per design), participants had to answer knowledge questions appearing on their screen.

On the main part on the right side of the screen, the current topic of the idea management tool is presented. Entered ideas are automatically arranged around the topic resembling the structure of a mind map. On the left side of the screen, personal information is presented including current performance measures, idea evaluation and received achievements. For entered ideas and correctly answered knowledge questions, the participants receive performance points. For idea evaluation, participants rate entered ideas by the team members positive and/or negative, depending on the functionality of the respective design (details, see Fig. 2 and Tab. 1). 2a 2b 2c 2d

Fig. 2. 2a and 2b exemplarily show a complete screen of the neutral design. On the main part on the right side of the screen the ideas were arranged resembling the structure of a mind map (2b). The key differences on the left side of the screen are drafted for the neutral design (2a) and the designs addressing popularity (2c) and competition (2d). See also Table 1.

UX Questionnaire. Based on the model of positive UX, a questionnaire was developed to measure the UX of a technological product that assesses the amount to which a need is fulfilled during interaction with a product. The User Needs Questionnaire (UNeeQ) consists of five items regarding overall positive and negative UX and 30 items regarding the ten user needs shown in Fig. 1. As a result, two overall scores for positive and negative UX, as well as ten scores for user needs are calculated.

Test Procedure. After filling out a small socio-demographic questionnaire, participants had to rank the three screens presented in Fig. 2 in terms of aesthetics without any additional information about the designs. For the main study, they started to pass through every design of the prototype in a randomly given order balanced over all subjects. At the beginning of each design variation, participants were given a standardized instruction how to use the respective prototype. After each design of the pro-totype, they were instructed to fill out the UNeeQ. After the main study, participants were again asked to rank the three screens presented in Fig. 2 in terms of aesthetics.

To control the influence of individual competitiveness, participants had to fill out a German version of the Competitiveness Index (CI) at the end of the study [START_REF] Smither | The Nature of Competitiveness: The Development and Validation of the Competitiveness Index[END_REF].

Experimental Design. The study implied one independent variable, design of prototype (neutral, popularity, competition), and a covariate, user's competitiveness (low, high). The dependent variables were subjective ratings of UX (UNeeQ), subjective design-rankings (preferences of designs), and objective user behaviour (frequency of likes/dislikes, number of ideas).

A repeated-measures analysis of variance (ANOVA) was used for all withinsubject comparisons to identify the effect of design of prototype and for betweensubject comparisons to identify the effect of user's competitiveness on subjective ratings of UX and on objective user behaviour. Subjective design-rankings were investigated with Wilcoxon signed rank test. Only significant results are reported.

Hypotheses

 Participants report a higher positive overall UX in the UNeeQ for the designs addressing user needs popularity and competition compared to the neutral design.  Participants report a higher fulfilment of the scales popularity and competition in the UNeeQ for the designs addressing user needs popularity respective competition compared to the neutral design.  Participants show a more active behaviour for designs addressing user needs popularity and competition compared to the neutral design.

Results

Pilot study

During Valence Method, participants marked elements regarding the idea evaluation the most (58 markers: 41 positive, 17 negative). Performance elements were marked 22 times (11 pos., 11 neg.), elements regarding the topic 17 times (9 pos., 8 neg.) and elements regarding the ideas 13 times (6 pos., 7 neg.). The user needs addressed most with the clickable prototype were popularity (31), competition (21) and security (20). The other user needs mentioned were competence [START_REF] Burmester | Valence Method for Formative Evaluation of User Experience[END_REF], relatedness [START_REF] Burmester | Valence Method for Formative Evaluation of User Experience[END_REF], stimulation [START_REF] Kahneman | Objective Happiness[END_REF], influence (3), idealism (2) and self-expression (1).

3.2

Competitiveness

To control the effect of user's competitiveness, participants were divided into two groups based on the median of the total score in the CI of all 20 participants. ANOVA showed no effects for any of the dependent variables. Also, correlation between total score of CI and overall positive and negative UX of UNeeQ did not show significant correlations.

Subjective ratings of UX

In Tab. 2, results of the ANOVAs are summarized. Overall positive and negative UX showed a significant difference for the variable design of prototype. Compared to the neutral design, overall positive UX was significantly higher and overall negative UX was reported significantly lower for the designs addressing user needs popularity and competition.

Single scales of UNeeQ showed a significant difference for user needs popularity and competition, as well as for the scale influence. All other scales of UNeeQ showed no significant difference for the variable design of prototype.

Subjective design-rankings

At the end of the study, participants preferred the design addressing competition (mean rank = 1.40) in the subjective ranking over the neutral design (mean rank = 2.45) and the design addressing popularity (mean rank = 2.15). Interestingly, the design addressing competition is ranked significantly better after the study compared to rankings before the study (Z = -2.762, p=.006). For the design addressing popularity, rankings were significantly worse after the study (Z = -2.233, p=.026). Rankings for the neutral design stayed equal.

Objective user behaviour

To objectively compare user behaviour, frequencies of idea evaluation and number of ideas are compared. ANOVA showed a trend for the number of likes for the variable design of prototype (F (2,38) =3.084, p=.057). Post-hoc analysis showed no significant differences between designs (mean number of likes: neutral = 6.05, popularity = 8.05, competition = 8.20). For entered ideas, participants showed no significant differences (mean number of ideas: neutral = 6.20, popularity = 6.70, competition = 6.45).

Discussion

In line with our hypotheses, overall positive UX, as well as single UNeeQ-scales for competition and popularity were significantly higher for designs addressing user needs popularity and competition compared to the neutral design. Also, UNeeQ-score for influence showed significant difference. A valid explanation is the obviously higher influence in the process of idea evaluation due to an rating compared to an isolated rating in the neutral design.

Generally, user needs can only be separated to a certain degree. A small overlap already came up in the beginning of the development of our prototype when asking experts to attribute functionalities addressing special user needs. Nevertheless, all other scales showed no significant differences for the variable design of prototype. Therefore, the developed prototypes only addressed the expected user needs and showed a very good differentiation between single scales of user needs, which was an important requirement in our consideration.

Next to subjective ratings of UX, the two variations addressing user needs stated better spontaneous design-rankings compared to the neutral design. Interestingly, they preferred the design addressing user need popularity before the study, whereas after the study, they preferred the design addressing competition most. We assume that the slightly higher UX led participants to rank the design addressing competition over the design addressing popularity after interaction with the prototype.

In the pilot study, expected user needs popularity and competition were already most prominent in the clickable prototype. Besides, user need security was often mentioned when asking participants using Laddering Technique. It was mostly used for negative aspects of the usability of the clickable prototypes. Suggestions for improvement were considered and integrated for the prototype in the main study.

Regarding objective user behaviour, the design of the three prototypes had no significant influence on the brainstorming of ideas, but showed a tendency towards a more active idea evaluation for designs addressing selected user needs. We assume that the higher fulfilment of user needs, indicated by significantly higher UNeeQscores, led to this more active behaviour. Getting personal feedback due to interactive idea evaluation motivated participants to rate entered ideas by the team members.

Future work

Future work aims at investigating all other user needs mentioned in the model of positive UX (Fig. 1). Variations of the described prototype will be developed to address the whole spectrum of user needs to define guidelines or specific parameters of elements to further improve UX for new products [START_REF] Sproll | From product concept to user experience. Exploring UX potentials at early product stages[END_REF]. Another intended application of this prototype is to further investigate UX and emotions during interaction. Subjective surveys are usually conducted in retrospect by using questionnaires and interviews.

The personal experience cannot be directly measured and is also shaped by personal attitudes and prejudices [START_REF] Amelang | Psychologische Diagnostik und Intervention[END_REF]. To measure UX during interaction with a technical product, neurophysiological parameters are promising to objectively indicate positive and negative emotions during experiencing [START_REF] Roto | User Experience White Paper -Bringing clarity to the concept of user experience[END_REF]. They should help to explore the subconscious processes of implicit cognitions, and to draw conclusions about human behaviour [START_REF] Bargh | Beyond Behaviorism: On the Automaticity of Higher Mental Processes[END_REF]. Neurophysiological methods allow for a real-time measurement of brain activity during the current experience without significantly affecting the situation [START_REF] Spath | Methods from neuroscience for measuring user experience in work environments[END_REF]. The approach chosen here is to identify and apply methods that provide insight into the subjective emotional experience in the interaction with technology and predict certain aspects of user behaviour.

Fig. 1 .

 1 Fig. 1. Model of Positive User Experience.

Table 2 .

 2 Statistical results (ANOVA for repeated measures): Overall positive and negative UX and single scales of UNeeQ for the variable design of prototype.

		Measure	Main effect
			F(2,38)	p	η²
	Overall	Positive UX 10.415	<.001	.354
		Negative UX 7.534	.002	.284
	User	Competition 11.637	<.001	.380
	needs	Popularity	4.546	.017	.193
		Influence	4.469	.018	.190