Aditya Vashistha
email: adityav@cs.washington.edu

Rajan Vaish
email: rvaish@cs.ucsc.edu

Edward Cutrell
email: cutrell@microsoft.com

William Thies
email: thies@microsoft.com

The Whodunit Challenge: Mobilizing the Crowd in India

Keywords: Crowdsourcing, crowd mobilization, HCI4D, ICT4D, India

While there has been a surge of interest in mobilizing the crowd to solve large-scale time-critical challenges, to date such work has focused on high-income countries and Internet-based solutions. In developing countries, approaches for crowd mobilization are often broader and more diverse, utilizing not only the Internet but also face-to-face and mobile communications. In this paper, we describe the Whodunit Challenge, the first social mobilization contest to be launched in India. The contest enabled participation via basic mobile phones and required rapid formation of large teams in order to solve a fictional mystery case. The challenge encompassed 7,700 participants in a single day and was won by a university team in about 5 hours. To understand teams' strategies and experiences, we conducted 84 phone interviews. While the Internet was an important tool for most teams, in contrast to prior challenges we also found heavy reliance on personal networks and offline communication channels. We synthesize these findings and offer recommendations for future crowd mobilization challenges targeting low-income environments in developing countries.

Introduction

Recent years have witnessed the power of cro wdsourcing as a tool for solving i mportant societal challenges [START_REF] Von Ahn | Duolingo: Learn a Language for Free While Helping to Translate the Web[END_REF][START_REF] Bigham | VizWiz: Nearly Real-time Answers to Visual Questions[END_REF][START_REF] Chen | Shreddr: Pipelined Paper Digitization for Low-resource Organizations[END_REF][START_REF] Hara | Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems[END_REF]. Of particu lar note are instances of crowd mobilization, where large groups of people work together in service of a co mmon goal. A landmark demonstration of crowd mobilizat ion is the DA RPA Network Challenge, where teams competed to find 10 red balloons that were hidden across the United States [START_REF]DARPA Network Challenge[END_REF]. The winning team found all the balloons in less than nine hours, utilizing a recursive incentive structure that rewarded participants both for jo ining the search as well as for growing the team [START_REF] Pickard | Time-Critical Social M obilization[END_REF]. Since then, mobilizat ion exercises such as the Tag Challenge have shown that teams can locate people of interest across North America and Europe [START_REF] Rahwan | Global M anhunt Pushes the Limits of Social M obilization[END_REF]. The MyHeartMap Challenge mapped over 1,500 defibrillators in Philadelphia County [8]. Authorities have also turned to crowd mob ilization for help gathering intelligence surrounding the London riots [START_REF] Wilcock | Police "Facewatch" app targets London riot suspects[END_REF] and the Boston Marathon bombings [START_REF]Crowdsourced videos, photos could aid Boston blast investigations[END_REF], though the results have not been without pitfalls [START_REF] Kaufman | Bombings Trip Up Reddit in Its Turn in Spotlight[END_REF] and controversy [START_REF] Wemple | Young men, please sue the New York Post[END_REF]. One limitat ion of prio r crowd mob ilizat ion studies is that they have focused exclusively on North America and Europe, where Internet penetration is so high that most teams pursue purely online strategies. However, in other areas of the world, the Inte rnet remains only one of several co mplementary channels for effective mob ilization of crowd. For example, in India, 1.2% of households have broadband Internet access [13], but there are 929 million mob ile subscribers, over 550 million v iewers of telev ision, and over 160 million listeners to radio [13,[START_REF]Census of India -M ode of Communication[END_REF]. An SMS-based social network called SMS GupShup has 66 million subscribers in India [START_REF]GupShup Re-launches GupShup M essenger[END_REF]. Moreover, there is a rich oral t radition of conveying stories and information face-to-face. Environ ments such as the Indian railwaysserving 175 million passengers every week [16]provide fert ile grounds for mobilizing crowds. India also has a unique social milieu, with its own social h ierarch ies, attitudes towards privacy [START_REF] Kumaraguru | Privacy in India: Attitudes and Awareness[END_REF], and trust in / responsiveness to various incentive schemes . In light of all these characteristics, it stands to reason that effective crowd mobilizat ion in India would require broader and more inclusive techniques than in Western contexts.

To further explore the landscape of crowd mobilization in India, this pape r reports on a new mobilizat ion contest that was designed specifically for the Ind ian context. Dubbed the "Whodunit Challenge", the contest enabled participation through mobile phones instead of via the Internet. The contest offered a Rs. 100,000 (USD 1,66 7) prize 1 for solving a fict ional mystery case, in which teams were asked to gather five pieces of in formation: Who, What, Where, When, and Why. To participate, an individual had to send a missed call 2 to the contest phone number, which returned via SMS one of five phrases, each provid ing one of the pieces of information. Because some phrases were returned with low probability, and only one phrase was sent to each phone number irrespective of the number of missed calls received, participants needed to form teams of several hundred people in order to have a chance of winning.

The Whodunit Challenge attracted over 7,700 participants within the first day, and was won by a university team in just over five hours. To understand teams' exper iences and strategies, we conducted 84 phone interviews, covering most individuals who submitted 3 or mo re phrases or who received phrases sent with low probability. While many of the winning teams did utilize the Internet to mob ilize the crowd for finding phrases, we also uncovered interesting cases that relied main ly on face-to-face or mobile co mmunication. Unlike prev ious crowd mobilization challenges, many successful teams relied only on personal networks, rather than trying to incentivize strangers to help them search for phrases. Members of these teams were usually unaware of (or unmotivated by) the cash award.

In the remainder of this paper, we describe the design rationale, execution strategy, and detailed evaluation of the Whodunit Challenge. To the best of our knowledg e, this is the first paper to describe a large-scale crowd mob ilizat ion contest in a developing-country context, exp loring the portfolio of online and offline commun ication strat egies that teams employed. We also offer reco mmendations to inform the design of future crowd mob ilization challenges targeting low-inco me environ ments in developing countries.

Related Work

There is a v ibrant conversation in the research community surrounding the future of crowd work [START_REF] Kittur | The Future of Crowd Work[END_REF]. Research that is most closely related to our work falls in t wo areas: crowd mobilization challenges and crowdsourcing in developing regions.

One of the most high-profile experiments in crowd mob ilization was DA RPA's Network Challenge, launched in 2009. By asking teams to find ten red balloons that were hidden across the United States, the challenge aimed to exp lore the power o f the Internet and social networks in mob ilizing large groups to solve difficu lt, t ime-crit ical problems [START_REF]DARPA Network Challenge[END_REF]. The winning team, fro m MIT, located all of the balloons within nine hours [START_REF] Tang | Beaker": Reflecting on the DARPA Red Balloon Challenge[END_REF] using a recursive incentive mechanis m that rewarded people for reporting balloons and for recruiting others to look for balloons [START_REF] Pickard | Time-Critical Social M obilization[END_REF]. Th is approach was inspired by the work of Dodds et al. [START_REF] Dodds | An experimental study of search in global social networks[END_REF], wh ich emphasizes the importance of individual f inancial incentives [START_REF] Watts | Financial Incentives and the "Performance of Crowds[END_REF]. Cebrian and colleagues proved that MIT's incentive scheme is optimal in terms of minimizing the investment to recover informat ion [START_REF] Cebrian | Finding Red Balloons with Split Contracts: Robustness to Individuals' Selfishness[END_REF], and that it is robust to misinformation [START_REF] Naroditskiy | Verification in Referral-Based Crowdsourcing[END_REF].

The DARPA Netwo rk Challenge seeded broad interest in the role of social ne tworks in homeland security [START_REF] Ford | Twitter, Facebook, and Ten Red Balloons Social Network Problem Solving and Homeland Security[END_REF]. Th is led to a follow-up contest called the Tag Challenge fro m the U.S. Depart ment of State [START_REF] Rahwan | Global M anhunt Pushes the Limits of Social M obilization[END_REF], in which the task was to find five people across five cit ies and two continents within t welve hours [START_REF] Rutherford | Limits of social mobilization[END_REF]. The winning team found three of the five people and used an incentive scheme similar to the one that won the Network Challenge. Private firms and universities have also exp lored the potential of crowd mob ilization. In 2009, Wired Magazine launched the Vanish Challenge [START_REF]Wired M agazine's Vanish Challenge[END_REF] and in 2012, the University of Pennsylvania launched the MyHeartMap Challenge. The latter challenge saw over 300 participants who found and catalogued over 1,500 defibrillators in Ph iladelph ia County [8]. However, to the best of our knowledge, there has not yet been any social mob ilization contest with a focus on a developing country. There is a need to explore the landscape of crowd mob ilization in developing countries and to identify the differences fro m cro wd mobilization strategies observed in the developed world.

Researchers have also studied the potential and limitat ions of crowdsourcing in d eveloping regions. Platforms such as txt Eagle [START_REF] Eagle | Txteagle: M obile Crowdsourcing[END_REF] and mClerk [START_REF] Gupta | mClerk: enabling mobile crowdsourcing in developing regions[END_REF] aim to enable workers to earn supplemental income on low-end mobile phones. Others have exa mined the usage [START_REF] Ipeirotis | Demographics of M echanical Turk[END_REF][START_REF] Ross | Who Are the Crowdworkers?: Shifting Demographics in M echanical Turk[END_REF] and non-usage [START_REF] Khanna | Evaluating and Improving the Usability of M echanical Turk for Low-income Workers in India[END_REF] of Mechanical Turk in India, where approximately one third of Turkers reside. Effo rts such as Ushahidi [START_REF] Gao | Harnessing the Crowdsourcing Power of Social M edia for Disaster Relief[END_REF] and Mission 4636 in Hait i [START_REF] Hester | Scalable Cr isis Relief: Crowdsourced SM S Translation and Categorization with M ission 4636[END_REF] have leveraged crowd workers to respond to crises in developing countries. Researchers have also explored the role of social networks such as Fac ebook [START_REF] Wyche | Hustling Online: Understanding Consolidated Facebook Use in an Informal Settlement in Nairobi[END_REF] and SMS GupShup [START_REF] Rangaswamy | Re-Sourceful Networks: Notes from a M obile Social Networking Platform in India[END_REF] in low-income environments.

The Whodunit Challenge

The Whodunit Challenge was an India-wide social mobilization contest that awarded 100,000 Rupees (USD 1,667) to the winner. The objective of the challenge was to understand mechanisms, incentives and mediu ms people in India use to mobilize large groups of people for a time-bounded task.

Design Principles

The Whodunit Challenge embodied three design principles to make it broadly acce ssible throughout India. In India, 72% of the adult population is illiterate in English [START_REF] Desai | Human Deve lopment in India: Challenges For a Society in Transition[END_REF]. Thus, we localized the SMS messages by translating them into ten regional languages of India, making them mo re accessible than contests based on English alone.

To ensure that the messages were not distorted in the translation, the translations were done by native speakers of local languages who were highly skilled in English. A majority of the Indian population has constrained access to modern devices and networks: the smartphone penetration is only 10% [START_REF] Wu | Internet Trends[END_REF] and Internet penetration is 20% [START_REF]Internet Live Stats[END_REF]. Thus, we aimed to enable part icipation by owners of basic mobile phones, thereby ruling out any dependence on computers, smart phones, or Internet conne ctions (broadband or mobile). While Internet access could still offer advantages to participants, it was not strictly necessary to compete and win. A round 60% of the Indian population earns less than US$2 per day [START_REF]Poverty headcount ratio at $2 a day (PPP) (% of population)[END_REF]. Thus, we aimed to min imize the costs of participation. To part icipate in the contest, users needed to send a missed call fro m a mob ile phone (wh ich incurs no cost to them). To su bmit a phrase, they needed to send an SMS; this costs at most US$0.015, though is free under many mobile subscription plans. Our design did not require users to initiate any voice calls, as this expense could have thwarted participation from cost-sensitive groups.

Contest Mechanics

The challenge required participants to reconstruct a secret sentence consisting of five pieces of info rmation -Who, What, Where, When and Why (see Figure 1). Each piece of informat ion was referred to as a phrase and represented a part of the secret sentence.

To receive a phrase, participants simply sent a missed call to the contest phone number. On receiv ing the call, our server responded with an SMS containing one of the five phrases. Each phrase was sent in two languages: English and the predominant local language in the teleco m circle fro m wh ich the call was made. The first person to forward all five phrases (i.e., the secret sentence) to our server via SMS was declared the winner. User responses were passed through a transliterat ion API, provid ing robustness to any minor typos incurred in re-typing phrases.

Figure 1. Graphical illustration of the Whodunit Challenge

What made the challenge difficu lt is that some phrases were very rare, thereby requiring participants to form large teams to gather all the phrases. Also, we made it difficult for any one person to receive many phrases by sending only a single phrase to each phone number even if we received mu ltiple missed calls fro m the same number. Regulations in India make it difficult fo r a person to obtain many phone numbers; for examp le, VoIP DID numbers are not available for sale (and our server ignored Vo IP calls anyway). A lso, teleco m operators offer a limited nu mber of SIMs per customer, and each requires several pages of paperwork and supporting documents (pe rsonal identification, proof of address, etc.). While we advised participants that a very large team wou ld be necessary to win, the award itself was made to an individual. Thus, any sharing of the award within a team would need to be managed by a team leader.

While the Whodunit Challenge was framed in lighthearted terms, we intended for the search for phrases to closely mirror the search for serious time-sensitive informat ion, such as missing persons, suspicious containers, counterfeit currencies, etc. By using electronic phrases instead of physical artifacts, we were able to monitor and control each step of the contest.

Chance of Winning

How large of a team was needed in o rder to win the challenge? We did not publicize this informat ion broadly, though during one Q&A session, we indicated that co mpet itive teams would contain several hundred members. In response to each missed call, the server responded according to a weighted random function, returning Who, What, Where, When and Why with probability 89.4%, 10%, 0.2%, 0.2%, and 0.2%, respec- tively. Given these probabilities, the chance of winning as a function of team size is illustrated in Figure 2. To have a 50% chance of winning, a team needed 789 people. However, depending on their luck, s maller or larger teams could also win. To have a 5% chance of winning, a team needed about 230 people; for a 95% chance of winning, a tea m needed about 2040 people. The probability of winning did not depend on participants' location, time of sending a missed call, or other factors, as each phrase was returned independently at random.

Publicity and Outreach

We publicized the challenge widely in order to seed participation. A d istinguished speaker announced the challenge to a live audience of 2,500 undergraduate enginee ring students about one week p rior to the contest launch [START_REF] Lee | From Computing Research to Surprising Inventions[END_REF]. We conducted a large email and social med ia campaign targeting engineering colleges, MBA colleges, and student volunteers connected with Microsoft Research India. We also presented posters at two academic conferences in the month preceding the contest to create awareness among computer scientists. While the audiences for these activities were primar ily co mposed of Internet users, we advised team leaders that outreach to non-Internet users would be highly advantageous for growing a large team and winning the cha llenge. Also, to seed visibility among non-Internet users, we met with a group of cab drivers and called ten group owners on SMS GupShup. Our outreach activities led to med ia coverage by both domestic and international outlets [41,[START_REF]Social whodunnit competition launches in India -tech -31 January 2013 -New Scientist[END_REF]. The basic rules for the contest were explained in the dig ital pro motional material and personal co nversations. Internet users could also visit the contest website [START_REF]The Whodunit? Challenge by M icrosoft Research India[END_REF] for more detailed examples.

N=2

Analysis Methodology

To understand the results of the challenge, we emp loyed a mix of quantitative and qualitative methods. We kept electronic logs of all calls and SMS's submitted to our server, and analyzed the appro ximate geographic orig in of calls using the prefix of the telephone number [START_REF] Wu | Internet Trends[END_REF]. On the qualitative side, we conducted structured phone interviews with 84 part icipants, probing themes such as how they came to learn about the challenge, who they told and how they communicated about it, and what was their strategy (if any) to win. The interviews were conducted in English and Hindi by the first author (male, age 28 At the end of the challenge, we also invited participants to complete a brief online survey. We publicized the survey via SM S and also on the contest website, and received about 300 responses in one day. Many questions in the survey were optional and thus, different questions were answered by a different number of users. There were 167 male and 46 female respondents. The average age of the respondents was 21.4 years (s.d.= 6.28). The respondents were fro m 42 universit ies and 5 organizations. Respondents included 174 students, 14 salaried emp loyees, 2 professors, and 1 homemaker. The majority of the users had a feature phone or basic phone. Fifty-nine respondents heard about the challenge through an email sent by a friend, college a uthorities or professors, 58 heard through offline conversations with friends, relatives, professors and colleagues, 47 got the information through Facebook and websites, and the remainder heard about the challenge through text messages, offline promotional events, advertisements, and tasks on Amazon Mechanical Turk. Most respondents, 192, received Who, 27 received What, 4 received Where, 2 received Why and none received When. Sixty-one respondents reported discovering one phrase while 65, 24, 11 and 36 part icipants reported discovering two, three, four and five phrases respectively. Eleven respondents could not even begin their campaign as the challenge finished much earlier than they expected. On an average, each person reported sharing their phrase with 33 people (s.d.=120) and receiving a phrase fro m 30 people (s.d.=93).

Results

The Whodunit Challenge was launched on February 1, 2013 at 9:00 AM local t ime. The challenge drew 7,739 part icipants in less than 15 hours (see Figure 3). The first winning submission was made in just over 5 hours. However, we delayed announcing that the contest was over until the evening, as we also wanted to rank and recognize the runner-up teams. Participants sent a total of 10,577 missed calls to the system. Of the unique callers, 6,980 received the phrase for "Who"; 740 received "What"; 18 received "Where", 17 received "When" and 17 received "Why".

There were 185 people who submitted at least one phrase. The first person to submit two phrases did so within 26 minutes; 3 phrases, within 57 minutes; 4 phrases, within 3 hours and 19 minutes; and five phrases (winning the contest) after 5 hours and 7 minutes. Geographically, participation spanned across all of India, as illustrated in Figure 4.

Winning Strategies

The winning teams are listed in The winning team was based at the Indraprastha Institute of Informat ion Technology Delhi (IIIT Delhi), led by 2 Ph.D. students and 6 undergraduates. In advance of the contest launch, this team set up a website3 and a Facebook group 4 that attracted 474 members. The website publicized the following finan cial incentives. If the team won, they would award Rs. 10,000 (USD 167) to anyone who sent them a new phrase; Rs. 2,000 (USD 33) to anyone who d irectly referred someone who sent a new phrase, and a mobile top-up worth Rs. 50 (USD 0.83) to the first 200 people who sent any phrase. They set up an SMS server to which people could forward phrases. They recruited team members using a variety of methods, spanning phone calls, SMS, WhatsApp and social media platforms.

The second-place team was based at the Indian Institute of Technology Delhi (IIT Delh i), led by eight second year Co mputer Science undergraduates. This team finished just 7 minutes behind the leader. Yet they used a very different strategy: they set up a small call center, relying mostly on direct calls and SMS to reach out to family and friends who live in smaller towns and villages across the country. In turn, they asked these contacts to gather team members fro m the local co mmunity. One team member also set up a Facebook group and utilized Facebook group chat.

The third-place team was also based at IIT Delhi, led by six undergraduate students. This team found 4 phrases in advance of 4:30p m, and claims to have found the fifth phrase (working independently) by 5:00p m. Unlike other teams, this team relied solely on social media and email to recru it members. They in vited over 4,000 people to a Facebook event 5 , of whom 329 rep lied with "Going" and 63 replied with "Maybe". The group page was lin ked to another website where team members could reg ister and receive a unique ID, which could be used to refer others to the t eam. Part icipation by those referred led to modest payments to the referrer (Rs. 100, or USD 1.67, for 20 referrals).

The fourth-place team, based at the Jansons Institute of Technology in Coimbatore, was led by a single undergraduate student. She estimated that she reached out to 250-300 people, half via SMS and voice calls, and half v ia Facebook. She submitted the fourth phrase at 3:30p m and the fifth at 7:00p m. While she expressed interest in sharing the prize money with team members, she did not have any incentive structure in place and the terms were not discussed with the team members; her team members helped her as a personal favor rather than for a monetary incentive.

The fifth-place team was based at Paavai Engineering Co llege in Tamil Nadu, led by two cousins. They managed seven sub-teams with 15-20 people per team and used face-to-face interactions, phone calls, SMS, and social networks to coordinate. Inte restingly, they also contacted a relative who worked at a mobile shop; the shop asked customers to g ive a missed call on the contest number and forward phrases to him, which he then shared with the team leaders. They did not have a formal incentive strategy, though as they got closer to winn ing, they offered to share a prize with those who helped them.

Emergent Themes

Rather than describe additional teams in detail, we present three high -level themes that emerged across the remainder of our analysis. This draws fro m our interviews with teams, our interviews with recipients of rare phrases, and the web-based followup survey.

Internet but also S MS, Voice, Face-to-Face. All o f the top five teams (and 14 of the top 19) utilized the Internet to their advantage. The most co mmon uses were to esta blish a website (either independently or as a Facebook page) and to reach out to friends and contacts via Facebook (10 teams), WhatsApp (6 teams) and email (2 teams). At the same time, teams also demonstrated a heavy reliance on non -Internet technologies: thirteen teams utilized SM S and eleven utilized voice calls to mob ilize people. There were nine teams that utilized all three technologies: SMS, voice calls, and the Internet. Only three teams relied on Internet technologies alone.

The prevalence of co mmunications outside the Internet is confirmed by our inte rviews with those who received rare phrases. Most often, they heard about the contest via an SMS (n=9) o r face-to-face interaction (n=8) with a friend. Learn ing about the contest from Facebook was less common (n=4). Other ways of learning about the contest included email, phone calls, WhatsApp, etc. Our online survey revealed that even among participants that have access to various Internet-based services, 43% heard about the contest through offline personal conversations with friends and co lleagues.

An examp le of effective use of non-Internet technologies is the runner-up team (IIT Delhi), who relied mainly on a call center approach to reach family members in rural India. As a team leader exp lained to us, "My mom doesn't use Internet", and neither does the majority of India's rural population, wh ich constitutes 72% of the overall population. As another example, a team o f office drivers used only voice calls to manage a team and found two phrases in less than two hours.

One enterprising undergraduate (fro m the Amrita School of Engineering, Co imb atore) claims to have built a team of 200 peers using face-to-face contact alone. He estimates that with the help of his team's combined effo rts, he reached out to at least 1,000 people. While he reports finding three phrases, he did not submit them via SMS because he thought more phrases would be released later. (For this reason, his team does not appear in Table 1.).

These teams' success illustrates that it is also possible to mobilize a sizable cro wd without broadcast technologies such as social med ia or Internet websites. This finding has imp licat ions for social mob ilization in areas lacking Internet connectivity, or for well-connected areas that experience Internet outages during crises .

Reliance on Personal Networks . In prev ious social mob ilization contests, many teams incentivized strangers to join them. However, in the Whodunit challenge, a common thread amongst many teams' strategies was a reliance on personal networks: team leaders reached out to their friends and family, as opposed to incentivizing les ser-known acquaintances or strangers to join their team. Th is is already evident in the strategies for teams 2, 4, and 5, described above, as well as for many other teams. This trend is also corroborated by the online survey where 63 respondents reported relying on friends and colleagues for discovering phrases rather than 16 respondents who incentivized strangers. We also collected anecdotes where participants simp ly borrowed their friends' phones and gave the missed call on their behalf, without even explaining that there was a contest. If a new phrase was received on the friend's phone, it would be forwarded to the participant's phone. Three recipients of rare phrases reported that their phone was borrowed and used in this way. One recip ient of a rare phrase was a vegetable seller who had absolutely no knowledge about the co ntest; we hypothesize that his phone was borrowed without offering him any explanation.

Most Participants not Dri ven by Cash Rewards . Bu ild ing on the prior theme, the primary motivation for most participants was a desire to help a friend or family me mber, rather than any desire for (or even knowledge about) a cash award. Of the top 19 teams, less than half had any p lans to distribute the cash prize beyond the inner circle of team leaders; even the runner-up team did not offer any financial incentive to its members. In teams that did plan to distribute the prize, the ma jority were very vague about how they might reward their full team. In contrast to the challenges conducted in developed countries, team members were mot ivated by non-financial factors, and any reward offered to them would be perceived mo re as a courtesy t han as an owed compensation for their services.

We can quantify this tendency based on our interviews with those who received a rare phrase. Of the 35 respondents, only about one quarter (9) said that they were told about any financial incentive in relation to the contest. The majority [START_REF] Kittur | The Future of Crowd Work[END_REF] were not told about incentives, while the remainder (8) were team leaders or indiv iduals who were working alone. Of the people who were not told about any incentive scheme, the majority (12/ 18) nonetheless shared their phrase with others. This fraction is not significantly different fro m those who shared their phrase with knowledge of an incentive scheme (7/9). Some team leaders offered non-monetary incentives for their members. The leader of a team fro m A mrita School o f Engineering, Co imbatore (#17 in Table 1) pro mised to sponsor a forthcoming industrial visit for his class if they won the challenge. We talked to four team leaders who proposed to throw a party for their friends if they were the winner.

Participants sometimes had intrinsic motivation to participate. Fo r example, one student who received a rare phrase and forwarded it to benefit the winning team (IIIT Delh i) remarked, "I knew about the incentive model. Money was not important. I wanted my institute to win." An Infosys emp loyee chose to participate and forward a rare phrase to a friend's team because he thought the contest itself was creative and worthy of participation. He thought that the purpose of the contest was to understand the role of technology in solving crime. Some teams also experimented with other incentives. One team (not shown in T able 1) approached a charitable foundation, asking them to help publicize their team in exchange for 75% o f the prize money. While the foundation was receptive, it did not promptly post about the challenge on its Facebook page (which has over 20,000 likes) and thus offered little of the anticipated help within the timespan of the contest.

Discussion and Recommendations

While the Whodunit Challenge was quite successful in attracting enthusiastic partic ipants from across India, the lessons learned can also serve as design recommendations to help future crowd mobilizat ion challenges to reach out to a larger nu mber of people, especially in low-income or offline environments.

One of the shortcomings of the Whodunit Challenge was the low level of engag ement by low-inco me low-literate populations, primarily because we did not pro mote the contest widely in o ffline environ ments. The contest and the prize money appeared to be too good to be true for many lo w-inco me people that we interacted with. Many of them were uncertain about the reasons for awarding a high monetary prize just for sending missed calls. Despite our exp lanations, they had reservations about whether they would be charged for sending a call to our system. They were also concerned whether we would misuse their number, e.g., by sending them pesky voice calls or text messages.

To encourage more participation by non-Internet users, one approach would be to restrict pro motions to offline audiences, limiting the visibility to Internet users. Another approach would be to partner with local o rganizat ions that work closely with low-inco me groups, distribute graphic pamphlets in local languages, and conduct outreach efforts led by people who are fro m the target commun ity or have similar socio-economic status. It could also help to make the contest harder, for examp le, by decreasing the frequency of certain phrases or enforcing geographical diversity of team members (in Ind ia, coarse-grained geographic informat ion can be determined fro m the caller ID. [START_REF]telephone numbering in India[END_REF]). As teams are forced to reach out to broader populations, they may derive greater benefit fro m reaching out to the masses of rural and lo werconnectivity residents. Disseminating phrases in audio format rather than text would also enable inclusion of lesser-educated participants, though the cost of phone calls could be a significant deterrent (either for participants or for the challenge organizers, depending on who pays for the calls.) One of our interesting findings is that participants were often motivated by nonmonetary incentives, including social support for friends and recognition for their institution. Future challenges might employ non-monetary incentives to increase participation, for examp le, by offering recognition, goods or services that cater to groups (such as a party or travel vacation).

Our usage of mobile phone numbers as a unique personal identifier was largely successful in prompting the formation of large teams. However, it also led to some subtle imp lications, such as the practice of borrowing others' phones to leverage their participation without their fu ll knowledge or consent. While we d id not observe any serious abuses of this situation, e.g., by stealing phones or feeding misinformat ion to potential participants, these possibilit ies are nonetheless important to consider and guard against in future challenges.

One limitation in the design of the Whodunit Challenge is that it is not possible to know the exact sizes of teams. Addressing this limitation would have required a fu ndamental change in the contest dynamics, for examp le, to require each participant to identify themselves with one or more teams. This would likely require an interaction richer than a missed call, which would have added cost and complexity for partic ipants. Though sending SMS may seem easy, only 185 of 7,739 participants submitted a phrase to our server. So me participants may have been motivated only to share their phrase with their team leader, while other part icipants may have had limited familiar ity with SMS and how to forward them. In any case, finding creat ive techniques to more accurately track the gro wth and co mposition of teams, without adding co mplexity for part icipants, could yield large benefits in the analysis of future challenges. One potential approach could be to host a large number of contest phon e numbers, each advertised to a small nu mber of people. If two participants place calls on diffe rent numbers, it would be unlikely that they are on the same team.

Our final reco mmendation is to take extra care in designing simple ru les and co mmunicat ing them to participants. Though we distilled the challenge rules to very simple language, including several illustrative examp les, many teams misunderstood aspects that prevented them fro m co mpeting well. We found three teams who thought that some phrases would be released at a later date, preventing them fro m being aggressive in the in itial stages. We talked to five teams who assumed that phrases would be distributed across different geographical regions, causing them to seek out more geographies rather than seeking out mo re people. We also spoke with five teams who assumed that all phrases needed to be submitted together, preventing them fro m gaining feedback and recognition for intermed iate progress. It is important to anticipate any possible misconceptions and proactively convey the requisite clarifications . Several individuals misunderstood each phrase to be a puzzle instead of a part of the secret sentence; for examp le, in response to "Who: Rajnikanth", they would respond with "actor". While these details are somewhat specific to the Whodunit Challenge, the broader imp lication is that though it is difficult, it is necessary to design simp le rules that are easily understood and easily communicated fro m one person to another. This is especially important for lesser-educated participants and those who may lack the devices, connectivity or bandwidth to view large exp lanatory materials (such as websites, promotional videos, etc.). We also recommend setting up more accessible informat ion portals, such as an Interactive Voice Response system, to make the rules more accessible for people with low literacy and limited access to the Internet.

Conclusion

This paper presents the first crowd mob ilization challenge conducted in India, a d eveloping-country context where effect ive social mobilizat ion is broader and mo re inclusive than the rich-country settings studied previously. We customized the design of the challenge to incorporate local languages and to enable participation at very low cost by anyone with access to a basic mobile phone. The challenge was successful in attracting broad participation, spanning 7,700 part icipants from all across India in less than a day. While many participants utilized Internet technologies, we also found interesting usage of SMS, voice, and face-to-face co mmun ications that offered benefits in the Indian context. Unlike previous social mob ilization contests, participants relied primarily on their personal networks, and often recru ited team members wit h-out offering any financial incentives. We synthesize our lessons learned as a set of recommendations to help future crowd mobilizat ion challenges extend their reach into low-income, offline environments.

Figure 2 .

 2 Figure 2. Chance of finding N or more phrases as a function of team size

Figure 3 .

 3 Figure 3. Number of unique missed calls vs. time

Figure 4 .

 4 Figure 4. Heat map of received missed calls

  ). Each phone interview lasted around 15 minutes. We took detailed notes during the interview and used open coding to analyze the data. Of the 84 people we interviewed, 65 we re students, 17 were emp loyed in a private job, and 2 were ho memakers. The specific participants interviewed were 31 people (of 32 partic-

ipants) who submitted all five phrases; 1 person (out of 2) who submitted 4 phrases; 6 people (out of 6) who submitted 3 phrases; 38 people (out of 53) who received one of the rare phrases (where, when, or why); and 8 other participants.

Table 1 .

 1 The table lists all 20 teams who submitted or more phrases and, to the best of our knowledge, discovered these phrases without help fro m other teams. While we are certain about the rank o rdering of the first two teams, there is a co mp licat ion in ranking the remain ing teams: the winning team pos ted all of the phrases on the Facebook page of Whodunit Challenge at 4:30p m. Thus, we ran k teams by two criteria: first, by the number of phrases they submitted in advance of 4:30p m, and second, by the total number of phrases they submitted and claimed (during our interview) to have found independently. While 13 teams claimed to have found all the phrases on their own, only 2 teams found all phrases in advance of the leak.

	calls	8000 9000	N=1 Fi rs t s ubmission of N phrase(s) N=4	Wi nner Announced
	Unique missed	2000 3000 4000 5000 6000 7000		N=2 N=3			N=5
		1000
		0
			9:00	9:45	10:45	12:00	1:00	2:07	3:15	4:30	5:45	7:00	8:00	9:15	10:30	11:45

Table 1 .

 1 Top 20 teams in the Whodunit Challenge

	Team Number	Affiliation	Phrases submitted by 4:30pm	Total Phrases Submitted *	Time submitted last phrase	Face-to-face	Voice Call	SMS	WhatsApp	Email	Social media	Prize Benefactors of	Notes
												published	used SM S server;
		IIIT Delhi			2:07							incentive	Facebook group of
	1	(1)	5	5	PM		  			scheme	474
													mostly used voice,
		IIT Delhi			2:14							team leaders	SM S to reach to
	2	(1)	5	5	PM		 				only	friends & family
												published	website with 200
												incentive	registrations;
		IIT Delhi			5:00							scheme	FB event with 392
	3	(2)	4	5	PM							(see text)	replies
		Jansons										shared with	50% reached via
		Inst. of			7:00							team (details	SM S, voice; 50%
	4	Tech.	4	5	PM		 				unclear)	via FB
		Paavai											2 leaders managed
		Eng. Col-			3:30							team leaders	7 sub-teams of 15-
	5	lege	4	4	PM   				only	20 each
													leaders focused on
		IIIT Delhi			7:05							team leaders	different geogra-
	6	(2)	3	5	PM							only	phies
												$180-$270	one-person team;
		IIIT Delhi			2:10							for reporting	calls & WhatsApp
	7	(3)	3	3	PM		    	new phrase	worked best
												given to
												leaders, who
		IIM			2:24							distribute to	focused on calls &
	8	Indore	3	3	PM		  			sub-teams	SM S

In this paper, we use an exchange rate of 1 USD = Rs. 60.

Sending a missed call refers to the practice of calling a number and hanging up before the recipient can answer[START_REF] Pickard | Time-Critical Social M obilization[END_REF]

http://muc.iiitd.edu.in/whodunit/

https://www.facebook.com/groups/528552907178873/

https://www.facebook.com/events/124800261025377


Anders Bruun
email: bruun@cs.aau.dk

Jan Stage
email: jans@cs.aau.dk

An Empirical Study of the Effects of Three Think-Aloud Protocols on Identification of Usability Problems

Keywords: Usability evaluation, thinking aloud, verbalization, think-aloud protocols, empirical study

Think-aloud is a de-facto standard in user-based usability evaluation to verbalize what a user is experiencing. Despite its qualities, it has been argued that thinking aloud affects the task solving process. This paper reports from an empirical study of the effect of three think-aloud protocols on the identified usability problems. The three protocols were traditional, active listening and coaching. The study involved 43 test subjects distributed on the three thinkaloud conditions and a silent control condition in a between-subject design. The results show that the three think-aloud protocols facilitated identification of the double number of usability problems compared to the silent condition, while the problems identified by the three think-aloud protocol were comparable. Our results do not support the common emphasis on the Coaching protocol, while we have seen that the Traditional protocol performs surprisingly well.

Introduction

Software organizations and developers increasingly emphasize usability as a software quality characteristic [START_REF] Rubin | Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests[END_REF], and usability evaluation is a key tool to improve usability [START_REF] Nielsen | Usability Engineering[END_REF][START_REF] Rubin | Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests[END_REF]. User-based usability evaluation is an approach to evaluation that produces two types of results [START_REF] Dumas | A Practical Guide to Usability Testing[END_REF][START_REF] Rubin | Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests[END_REF]: (1) usability measures and (2) usability problems. Usability measures include various quantitative assessments measured directly or indirectly during a usability evaluation. The three factors in the classical ISO definition of usability are examples of this [13]: effectiveness, efficiency and satisfaction.

Usability problems are a list of problems that have been experienced by users du ring a usability evaluation that involves usage of the evaluated software. They are typically expressed in a format that enables developers to enhance the usability of a user interface design. Usability problems are also useful as means to understand a particular usability measure, e.g. why efficiency is low [START_REF] Rubin | Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests[END_REF].

Think-aloud protocols are widely applied in user-based usability evaluations [START_REF] Dumas | A Practical Guide to Usability Testing[END_REF][START_REF] Nielsen | Usability Engineering[END_REF][START_REF] Rubin | Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests[END_REF]. This technique was originally introduced in psychology to p roduce rich data through verbalization of cognitive processes [START_REF] Ericsson | Protocol Analysis: Verbal Reports As Data, revised edition[END_REF]. While there is general agreement about the strength of thinking aloud, there have also been concerns that it affects the task solving process of the test subjects. Yet the amount of systematic empirical studies of the effect of thinking aloud is limited. A recent study of three different thinkaloud protocols characterized the literature on think-aloud protocols as being unclear with respect to the protocols applied. With a focus only on usability measures, i.e. effectiveness, efficiency and satisfaction, they found that different protocols affect the usability measures differently [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF]. However, they did not deal with usability problems, and previous studies of this point in various directions.

This paper reports from an empirical study of the effect of different think-aloud protocols on the usability problems that are identified in a user-based usability evaluation. The study also replicates a previous study of the effect on usability measures [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF], and compares the results on the two types of measures. In the following section, we present related work. This is followed by a description of the method applied in our empirical study. Next we provide the results and discuss these in relation to previous studies as well as their implications for usability practitioners and researchers. Finally, we provide the conclusion and point out avenues of future work.

Related Work

This section presents an overview of related literature on think-aloud protocols.

The Traditional Think-Aloud Protocol

The think-aloud protocol was originally described by Ericsson and Simon in 1984 [START_REF] Ericsson | Protocol Analysis: Verbal Reports As Data, revised edition[END_REF] as a technique to study cognitive processes through verbalization. They divided ve rbalization processes in three levels. Level 1 and 2 are verbalizations that do not distract a subject from the current activity and rely only on information in the subject's short term memory. On these two levels , a participant can be probed or reminded to continue verbalizing, e.g. by saying "Keep talking" or "Um-humm". The assumption is that this probe limits distraction from the current task. Level 3 is verbalizations that distract a subject from the current activity and draw on the subject's long term memory. Ericsson and Simon argued that reliable verbal reports can be produced during task solving provided they do not require participants to carry out additional cognitive processing. They asserted that only verbalizations from level 1 and 2 are valid as rich data to understand cognitive processes , because verbalizations from level 3 distract the subject [START_REF] Ericsson | Protocol Analysis: Verbal Reports As Data, revised edition[END_REF].

The challenge for usability evaluators is that they can see what a user is doing but not why. Thinking aloud has been introduced to overcome this. The idea is that when users are working with an interactive system they express what their intentions are and from that the reasons behind their difficulties can be inferred and compared with what they are actually doing [START_REF] Rubin | Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests[END_REF].

In a usability evaluation with the Traditional think aloud protocol, the test moderator will have minimal interaction with the users and only use the phrase "Keep talking" when the user stops thinking aloud. The test moderator will not embark on any conversation or provide assistance of any kind. Thinking aloud is usually related to laboratory-based testing, but it has also been used in other settings, e.g. [START_REF] Andreasen | What Happened to Remote Usability Testing? An Empirical Study of Three M ethods[END_REF][START_REF] Bruun | Let Your Users Do the Testing: A Comparison of Three Remote Asynchronous Usability Testing M ethods[END_REF][START_REF] Hertzum | Thinking Aloud in the Presence of Interruptions and Time Constraints[END_REF].

Other Think-Aloud Protocols

There is a variety of other think-aloud protocols. Based on field observations of us ability evaluations, Boren and Ramey [START_REF] Boren | Thinking aloud: Reconciling theory and practice[END_REF] conducted observations of usability practitioners in software companies. They found several discrepancies between the trad itional protocol and the way the practitioners conducted the evaluations and in partic ular the probes they used. They argue that the traditional protocol with an entirely passive listener is unnatural, and "a speech communication perspective may well interpret silence interspersed with commands to keep talking as a more abrasive form of contact" [2, p. 269]. Instead, they propose a protocol that reflects the way human beings naturally communicate, with a combination of statements by a speaker and some of feedback or acknowledgment from a listener. The feedback uses a back channel from the listener to the speaker, confirming for the speaker that there is act ually an "active listener". Boren and Ramey [START_REF] Boren | Thinking aloud: Reconciling theory and practice[END_REF] suggest that the feedback that is most effective is the quiet, affirming "Um-humm" response given at relevant times ; and if the test subject goes quiet, the active listener can repeat the last word the speaker said with a questioning intonation.

In usability evaluations with the Active listening think-aloud protocol, the test moderator will be an active listener providing acknowledging expressions. The mo derator will not ask questions directly or start a conversation , but only use verbal expressions like "Um-humm" or "ahh". If users stop thinking aloud, the test moderator will repeat the last word expressed by the user.

There are several think-aloud protocols in usability evaluation practice that go b eyond the traditional and active listener protocols. In practice, a test moderator will often try actively to get the test subjects to talk about their intentions, thinking, understanding and mental model. This is accomplished by using probes that are much more intrusive. Dumas and Redish [START_REF] Dumas | A Practical Guide to Usability Testing[END_REF] present "active intervention" as a technique where the test moderator asks questions in an active manner to get an understanding of the test subjects' mental model of the system and its functioning. Such an intrusive protocol with extensive conversation has been denoted as coaching [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF].

In a usability evaluation with the Coaching think-aloud protocol, the test moderator will be empathetic and encourage the test subject, e.g. by stating that "everybody has problems with this part of the system" if test subjects express frustration or feel ins ecure. The moderator may also express sympathy and give confirming feedback, e.g. "well done, you are doing great" when they have completed. When test subjects come to a complete stop, the moderator may assist by asking questions , e.g. "what options do you have?" or "what do you think happens when you click on that menu item?"

Empirical Studies of Think Aloud Protocols

There is a range of empirical studies of various think-aloud protocols. An early study [START_REF] Wright | M ethod bias and concurrent verbal protocol in software usability testing[END_REF] focused on the coaching protocol, showing that the level 3 verbalizations used in this protocol improved the test subjects' performance, which made them conclude that level 3 verbalizations in usability evaluation imply a bias. Another early study [START_REF] Henderson R D | A comparison of the four prominent user-based methods for evaluating the usability of computer software[END_REF] examined four user-based usability evaluation methods (logged data, questionnaire, interview, and verbal protocol analysis). They found that the verbal protocol analysis was most efficient, and even using two evaluation methods made no statistically significant improvement over the verbal protocol analysis.

A study by Krahmer and Ummelen [START_REF] Krahmer | Thinking about thinking aloud: A comparison of two verbal protocols for usability testing[END_REF] compared the traditional protocol [START_REF] Ericsson | Protocol Analysis: Verbal Reports As Data, revised edition[END_REF] with active listening [START_REF] Boren | Thinking aloud: Reconciling theory and practice[END_REF]. They found that in the active listening condition, more tasks were completed and the participants were less lost on the website. However, the website was untypical, and their active listening protocol included elements of coaching.

A more recent study by Hertzum et al. [START_REF] Hertzum | Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?[END_REF] compared the traditional and the coaching protocol. They found that with coaching, the subjects spent more time solving tasks, used more commands to navigate and experienced higher mental workload.

A study by Rhenius and Deffner [START_REF] Rhenius | Evaluation of concurrent thinking aloud using eyetracking data[END_REF] focused on the relation between verbalization and where test subjects were looking. By using eye tracking they showed that where test subjects were looking was directly related to what they were verbalizing, and concluded that verbalizations and short term memory are synchronized.

Table 1. Overview of the literature that assess think-aloud protocols.

Description of protocol

Assessment of protocol based on

Usability measures

Usability problems

Traditional [6] [10, 14 , 19] [8, 15]
Active listening [2] [14, 19] Coaching [5] [10, 18, 19, 25] [26]

The related work that assesses think-aloud protocols is listed in Table 1. Apparently, these focus mostly on usability measures. We have only found two empirically based assessments of the usability problems identified with different think-aloud protocols. One study compared the traditional with a retrospective think-aloud protocol. The two protocols revealed comparable sets of usability problems [START_REF] Van Den Haak | Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue[END_REF]. Another study compared the traditional and the coaching protocol. They found that coaching identified more usability problems related to dialogue, navigation, layout and functionality, but the unique problems of this condition were less severe [START_REF] Zhao | The impact of two different thinkaloud instructions in a usability test: a case of just following orders? Behaviour & Information Technology[END_REF]. This limited focus on usability problem identification confirms the general critique of usability research for facing major challenges [START_REF] Gray | Damaged M erchandise? A review of experiments that compare usability evaluation methods[END_REF][START_REF] Woolrych | Ingredients rather than recipes: A proposal for research that does not treat usability evaluation methods as indivisible wholes[END_REF] and having little relevance to practice [START_REF] Wixon | Evaluating Usability M ethods: Why the Current Literature Fails the Practitioner[END_REF].

Empirical Studies of Thinking Aloud in Practice

It has been argued that usability evaluation practice needs level 3 verbalizations despite their influence on task solving. Practitioners argue that level 3 gives the most useful data for identifying usability problems in interactive software systems [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF]. There are only few empirical studies of think-aloud protocols in practice. Nørgaard and Hornbaek [START_REF] Nørgaard | What do usability evaluators do in Practice? An explorative study of think-aloud testing[END_REF] observed and interviewed practitioners on the way they conducted usability evaluations. They found that practitioners often asked hypothetical or leading questions, which would elicit level 3 verbalizations [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF]. Another study explored the use of think-aloud protocols by usability practitioners. They found that the thinkaloud technique was generally appreciated, but divergent practice was reported [START_REF] Edwards | Exploring Think-Alouds in Usability Testing: An International Survey[END_REF].

Several researchers have requested additional research in the effects of various think-aloud protocols in usability evaluations, e.g. [START_REF] Boren | Thinking aloud: Reconciling theory and practice[END_REF][START_REF] Krahmer | Thinking about thinking aloud: A comparison of two verbal protocols for usability testing[END_REF], as it is not clear which protocol usability practitioners should employ in different situations.

Method

We have conducted an empirical study of the three think-aloud protocols in Table 1.

The aim was to study the effect of these think-aloud protocols on the identification of usability problems. Part of the study also replicated a study of usability measures [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF].

Experimental Conditions

Our empirical study was designed to compare the following four conditions:  Traditional: the protocol originally suggested by Ericsson and Simon [START_REF] Ericsson | Protocol Analysis: Verbal Reports As Data, revised edition[END_REF], where the test moderator is only allowed to probe with "Keep talking" when the test subject has been quiet for a while.  Active listening: the protocol suggested by Boren and Ramey [START_REF] Boren | Thinking aloud: Reconciling theory and practice[END_REF] where the test administration provide feedback or acknowledgment to the test subject by co nstantly probing with "Um-humm"  Coaching: the protocol that is generally used in usability evaluatio n practice where the test moderator is engaged in a continuous conversation with the test subject.  Silent: the test moderator does not to talk at any point during the evaluation sessions, except when introducing test subjects to the experiment and tasks. The test subjects are specifically asked not to think aloud.

The first two protocols, Traditional and Active listening, support level 1 and 2 verbalization, thereby providing what is assumed to be the most reliable data. The coac hing protocol supports level 3 verbalizations where the probes are expected to initiate use of the long term memory of the test subjects. The Silent condition is a benchmark.

System

The system evaluated is a data-dissemination website (dst.dk) that provides publicly available statistics about educational levels, IT knowledge and skills, economy, employment situation etc. in Denmark. It is the same type of website as in [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF].

Participants

Users. All participating users were invited through emails distributed to the whole university. In total we recruited 43 participating users divided on four different demo-graphical profiles, including 15 participants from technical and administrative personnel from different departments, 13 faculty members from Ph.D. students to professors from different departments, and 15 students in technical and non-technical educations. All participants were given a gift with a value corresponding to 20 USD. We distributed the participants evenly according to their demographic profiles over the four experimental conditions , see Table 2. Test Moderators. We used four test moderators in this experiment, all of whom had previous experience in moderating usability evaluation sessions. We chose to apply a single test moderator for each condition. To avoid test moderators introducing a bias, none of them took part in planning the study, analysing the data or writing this paper.

Evaluators. The two authors of this paper analysed all video material from the 43 evaluation sessions. Both have extensive experience in analys ing video data. To reduce possible bias from learning, the evaluators analysed the collected data in different (random) orders. The moderator/analyst separation is uncommon in practice, but was necessary to reduce bias from experimenter expectancy.

Setting

All usability evaluations were conducted in a usability lab, see Figure 1. The test moderator was in the control room (A) and the user in the test room (B). They com-municated through microphones and speakers. The rooms were separated by one-way mirrors so the user could not see the test moderator. This physical separation of test subjects and moderators is different from typical practice. It was introduced to remove the influence from the tes t moderator's body language and other visible expressions .

Procedure

The usability evaluations were conducted over the course of five days. On the first day of the four test moderators each did two evaluation sessions in order to get acquainted with their protocol. They had received written instructions on how to act in their particular protocol three days in advance, to allow for preparation. After completing their two evaluation sessions, they feedback on how they performed with respect to the protocol. The test moderators did not observe each other's sessions. Each of the following four days was devoted to evaluations with a particular protocol. All were conducted as a between-subjects study and each session lasted one hour. In all evaluation sessions , the users were first asked to complete two tasks using a similar website, to get acquainted with the protocol applied in their session. They were then asked to solve eight tasks using the main web site. At the end of each session, the user filled in a shortened version of the QUIS questionnaire, applied in [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF].

Tasks. The users solved eight tasks that varied in difficulty. For example, the first task was to find the total number of people living in Denmark, while a more difficult task was to find the number hotels and restaurants with 1 single employee in a particular area of Denmark.

Data Collection

All 43 sessions were video recorded. A questionnaire was collected for each user.

Data Analysis

The analysis of video material was divided in two parts: A joint analysis of 16 videos followed by individual analysis of the remaining 27 videos. All videos were tra nscribed in a log file before analysing for usability problems , cf. [START_REF] Skov | A Conceptual Tool for Usability Problem Identification in Website Development[END_REF].

Joint Analysis. In order to achieve consistency in usability problem identification with the different protocols, the authors of this paper first analysed 16 videos together (four videos from each condition). These were analysed in random order. Due to the variety in protocols we found it important to ensure consistency of problem identification in the individual analysis. To support this , we adapted the conceptual tool for usability problem identification and categorization in [START_REF] Skov | A Conceptual Tool for Usability Problem Identification in Website Development[END_REF]. We distinguished used the following ways of detecting a usability problem: A) Slowed down relative to normal work speed B) Inadequate understanding, e.g. does not understand how a specific functionality operates or is activated C) Frustration (expressing aggravation) D) Test moderator intervention E) Error compared to correct approach Examples of usability problems are "cannot find the detailed information for a municipality" or "cannot get overview of the presented information".

Individual Analysis. After the joint analysis, we individually analysed the remaining 27 videos in different and random order using the same tool as in the joint analysis .

Duplicates where more than one user experienced the same problem were removed. Merging Individual Analysis Results . After the individual analysis , we merged our two individual problem lists. We did this by discussing each identified problem to determine similarities between the two individual lists. Across the 27 videos we had an average any-two agreement of 0.44 (SD=0.11), which is relatively high compared to other studies reporting any-two agreements between 0.06 and 0.42 [START_REF] Hertzum | The Evaluator Effect: A Chilling Fact About Usability Evaluation M ethods[END_REF]. For the agreement between evaluators, we found no significant differences between the four conditions using a one-way ANOVA test.

Calculating QUIS Ratings. The satisfaction score was calculated in the same way as in [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF] by combining the sixteen scores from the modified version of the QUIS, each with a score on a Likert scale from 1 to 7. Thus a user could give a total score from 16 to 112 where a high score reflects more user satisfaction with the website.

Results

In this section we present our findings on usability measures , i.e. effectiveness, efficiency and satisfaction, as well as on usability problems.

Effectiveness

We measured the mean number of tasks completed correctly by users in the four co nditions. Users in the Coaching condition had a higher completion rate than users in the other three conditions, while users in the Active listening, Traditional and Silent pe r-formed similarly. A one-way ANOVA test reveals no significant differences between the conditions for effectiveness. We also measured the number of times the users gave up while solving a particular task. Users in the Silent condition had a tendency to give up more often than users in the other conditions, while users in the Coaching condition had the lowest rate in this respect. A one-way ANOVA test reveals no significant differences between any of the conditions with respect to the number of times that users gave up.

Efficiency

Table 4 shows the task completion times in seconds for the four conditions. For each condition, we have calculated the mean value of the total task completion time for each user on all tasks. The users in the Coaching condition on average spent most time while users in the Silent condition had the lowest time. The Active listening and Traditional conditions performed in between these.

A one-way ANOVA test reveals no significant differences between any of the four conditions. However, it should be noted that the difference between the Silent and Coaching conditions is close to significant (p=0.055).

Satisfaction

The QUIS questionnaire included 16 questions where the response for each was a score on a 1-7 Likert scale. Like [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF] we added these numbers together, yielding a score between 16 and 112 where a high score should indicate a high level of satisfa ction. A one-way ANOVA test of our data reveals no significant differences between median ratings in the four conditions. The simple sum combines a broad range of different factors. To complement this, we considered a subset of the questions that deal specifically with the users' overall reactions to the website (each on a scale from 1 to 7):  Terrible -Terrific  Frustrating -Satisfactory  Difficult -Easy Table 5 shows the overall median scores for these three questions. The Coaching and Traditional conditions gave the highest overall medians, while the Active listening and Silent conditions gave the lowest scores. Note that QUIS measure constructs with more questions than those presented in table 5. Picking out individual question items would be debatable practice. This is why we considered all three que stions of the "overall reactions" subcategory in QUIS, cf. [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF].

Table 5. Satisfaction ratings given by users for overall reactions to the website, n = number of users. A one-way ANOVA test reveals significant differences between these one or more conditions (F(3,119)=5.29, 5 % level, p<0.002). We apply a Tukey's pairwise co mparison test on all pairs of conditions in order to detect which are significant. The Tukey test reveals significant differences in overall median ratings between Coaching-Active listening (p<0.03) as well as between Coaching-Silent (p<0.03). The Tukey test also suggests significant differences in overall median ratings between Trad itional-Active listening (p<0.04) and Traditional-Silent (p<0.03). There are no significant differences between Coaching and Traditional, and conversely there are no differences between Active listening and Silent conditions. Thus, users in the Coaching and Traditional conditions express higher satisfaction with the website compared to users in the two other conditions.

Coaching

The questionnaire also included a question on perceived friendliness of the test moderator that was rated on a 7 point scale from unfriendly to friendly. Users in the Coaching condition gave higher ratings than users in all other conditions , but a oneway ANOVA test reveals no significant differences in these ratings.

Usability Problems

Through the detailed video analysis, we identified a total of 165 usab ility problems across all four conditions. Table 6. M ean number of problems identified using the different TA protocols.

Coaching (n=10)

Active listening (n=11)

Traditional (n=11) S ilent (n=11)
M ean (SD) 39.7 (13.8) 37.6 (14.6) 36.7 (8.9) 18.7 (6.1)

Number of identified problems. Table 6 shows the mean number of problems identified per user in the four conditions. We identified most problems in Coaching. This is closely followed by Active listening and Traditional, while the Silent condition revealed only around half of the problems identified in the other conditions. A one-way ANOVA test reveals highly significant differences between one or more of the conditions (F(3,39)=1.4, 5 % level, p<0.001). A Tukey's pairwise test for significance reveals highly significant differences between Silent and Coaching (p<0.001), Silent and Active listening (p<0.002) and Silent and Traditional (p<0.004). We found no significant differences between Coaching, Active listening and Trad itional. In other words, the Silent condition revealed significantly fewer usability pro blems per user compared to Coaching, Active listening and Traditional. This also means that we found no significant difference between any of the think-aloud protocols. In order to test for type II errors in our statistical analysis, we calculated the β value (β = 0.02) which led to a statistical power of 1 -0.02 = 0.98. This is above the threshold of 0.8 [START_REF] Cohen | Statistical power analysis for the behavioral sciences[END_REF], which indicates that sample size, reliability of measurement and strength of treatment are acceptable. Agreement Between Users Within Each Condition. Table 7 shows the mean anytwo agreement between users in the different conditions. This describes the extent to which users in a condition experienced the same or different usability problems; a high number reflects a large overlap in experienced problems. We fou nd most overlap in the Traditional condition, followed by Coaching and Active listening. The least overlap was found in the Silent condition. Table 8 shows the test statistics comparing each of the conditions with respect to the agreement between user. A one-way ANOVA shows significant differences between one or more of the conditions (F(3,206)=7.32, 5 % level, p<0.001). Table 8 shows the results obtained from a Tukeys pairwise comparison test. The Tukey test reveals significant diffe rences between all pairs of conditions and shows significant differences between Silent-Coaching and Silent-Traditional. The differences between Silent-Active listening are close to significant. These findings indicate that users in the Silent condition have a lower overlap in experienced problems compared to the three other protocols that performed similarly. 2 shows the overlap between the three TA protocols where 67 problems out of the 155 (43 %) are common between all. Additionally Coaching revealed 16 unique problems, Active listening 23 and Traditional 28. Thus, the Traditional protocol almost reveals twice as many unique problems compared to the Coaching condition. Note that, for simplicity the Silent condition was intentionally left out in Figure 2. The Silent condition revealed 64 (39 %) of which 54 were identified through TA protocols.

Agreement between Conditions. The Venn diagram in Figure

Types of Problems Identified

In this section we provide a qualitative account of the types of usability problems identified through each of the protocols. To cover individual differences between the conditions this qualitative analysis is based on the problems uniquely identified by each protocol.

In general we found that the problems descriptions of the Silent condition did not extend beyond the notes made in log files. We would observe a problem when users made an error, e.g. choosing a wrong link. A typical example is "Wants to find the number of unemployed persons in 2007. Selects the wrong link, but returns to the previous page". This is very similar to the entries made in the log files for each user, i.e. they are pure observations, which did not provide enough details to extend problem descriptions with interpretations of the cause of the problems.

In contrast, we found that the Coaching condition in general led to identification of several types of problems. This protocol primarily led to the identification of pro blems regarding unclear information, e.g. "Does not understand the meaning of the rows in the table". The coaching condition also led to the identification of problems regarding affordance, e.g. "Wrongfully believes that more details can be obtained by clicking on a table row". Finally, Coaching also revealed problems related to visibility such as "List of tables is immense, difficult to locate the correct one".

In case of the Active Listening condition, we found that most problems concerned unclear information. An example of such a problem is "Difference between menu options is unclear". Otherwise there were relatively few problems in other categories such as affordance and visibility.

The Traditional condition primarily led to the identification of problems related to visibility, e.g. "Does not notice heading of the graph and is therefore looking up numbers in the wrong location". Similar to Coaching, the Traditional condition also revealed problems from the other categories concerning unclear information and a ffordance.

As described above, we found that the three TA conditions led to data from which we could enrich problem descriptions to include notions of why a particular problem was observed. The opposite was the case for the Silent condition where we could observe a problem but not interpret why it occurred. Furthermore we found that the Coaching and Traditional conditions were similar in terms of the breadth of problem types identified. The Active Listening protocol primarily revealed problems related to unclear information.

Problem Severity. We categorized all identified problems according to critical, serious and cosmetic [START_REF]User-Friendly Web Design[END_REF]. We found that the think-aloud protocols reveal about twice as many critical (mean=23.7, SD=2.3), serious (mean=122.7, SD=5.6) and cos metic (mean=248.7, SD=8.6) problem instances as the Silent condition. This follows the tendency of the mean number of problems identified per user (see Table 6). There is a comparable distribution of the severity of problems in each of the conditions, e.g. 9 % of problem instances in the Coaching condition are critical, which is comparable to the 7 % in the Silent condition. Thus, we found no significant differences in this respect.

Categories of Problem Observations

We also categorized the identified usability problems according to the five ways of detecting a usability problem: A) Problem identified due to time delay, B) Shortcomings in a user's understanding of system functionality , C) User frustration, D) Test moderator assistance and E) User makes an error.

As an example, we found that 29% of all problems identified through the Coaching condition were observed based on time delays (category A). For this category of o bservations, we found no significant differences between conditions. This was also the case for category B observations.

As another example, we found that 8% of the problems identified in the Traditional condition were observed on the basis of user frustration (category C). In this case a one-way ANOVA revealed significant differences between conditions (F(3,39)=3.2, 5 % level, p<0.04). A Tukey's pairwise comparison test reveals significant differences between the Silent and Traditional conditions (p<0.03), but no differences between other pairs. The Coaching condition is characteristic in allowing the test moderator to help test subjects. This is also reflected by the fact that 10% of the problems identified in the Coaching condition were observed by test moderator assistance (category D).

Finally, it can be seen that 56% of all problems found in the Silent condition have been observed due to a user making an error (category E). Here we found significant differences between one or more conditions (F(3,39)=15.7, 5 % level, p<0.001). A Tukey's pairwise comparison test shows significant differences between the Coaching condition and all other conditions with p<0.001 in all cases.

These results show that there are no differences between the four conditions on the proportion of usability problems identified because users were (A) slowed down relative to normal work speed or (B) demonstrated an inadequate understanding of the system. With problems identified because users (C) frustration, significan tly more problems were identified in the Traditional condition compared with Silent. For problems identified because of (D) test moderator intervention, there were significantly more problems identified with the Coaching condition compared to all the other. Finally, for problems identified because (E) users made errors compared to correct approach, the Silent condition identified significantly more compared to Coaching.

Limitations

The empirical results presented above have a number of basic limitations. The study involves three specific think-aloud protocols, and there are many alternatives. We chose the three because they are either classical or commonly used by usability practitioners [START_REF] Dumas | A Practical Guide to Usability Testing[END_REF][START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF][START_REF] Rubin | Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests[END_REF]. Our aim of replicating the previous study made us consider the same protocols as they used, cf. [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF].

In each condition, there was one test moderator, and this person was different between the conditions . This was chosen to ensure consistency across all users in each condition. We could have used the same moderator in all conditions, but that could potentially cause confusion and make the conditions overlap. We tried to reduce a possible effect of this by training and assessing the moderators in accordance with the protocol before the tests were conducted.

The empirical study was only based on a single data-dissemination website where users solved 8 pre-defined tasks. To enable replication, we chose the same type of website as the previous study and used exactly the same tasks.

We had 43 users participating in the study. They are not a random or representative sample of the entire population of Denmark. The number of users is limited, but this is comparable to the empirical studies we have in our list of references, where the number of users is between 8 and 40, except for one study based on 80 users [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF].

Discussion

In this section, we compare our results to some of the related work and discuss implications for practitioners.

5.1

Comparison with Related Work

Our empirical study has focused on the consequences of using different think-aloud protocols both on the usability measures and on the usability problems identified.

For usability measures, we have found that users have the highest effectiveness, i.e. they complete more tasks in the coaching condition compared to all other conditions and that active listening, traditional and silent perform similarly. This is in line with the study presented in [START_REF] Wright | M ethod bias and concurrent verbal protocol in software usability testing[END_REF] where it was found that the coaching protocol improved the test subjects' performance. The study in [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF] compared the traditional, active listening and coaching protocols and results from that experiment shows a significantly higher level of effectiveness in the coaching condition than in the other conditions, while there were no significant differences between the traditional and active listening protocols [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF]. Thus, we found similar tendencies in favor of the coaching condition in our study but the differences we found were not significant. This is supported in [START_REF] Zhao | The impact of two different thinkaloud instructions in a usability test: a case of just following orders? Behaviour & Information Technology[END_REF] where the traditional and coaching protocols are compared. Furthermore, the study in [START_REF] Krahmer | Thinking about thinking aloud: A comparison of two verbal protocols for usability testing[END_REF] found a discrepancy compared to our study. In that study the traditional and active listening protocols are compared and it is found that more tasks were completed with active listening. In our study we found almost no difference in effectiv eness between these two protocols. However, the result in [START_REF] Krahmer | Thinking about thinking aloud: A comparison of two verbal protocols for usability testing[END_REF] is not entirely reliable as the active listening protocol included some elements of coaching.

In terms of efficiency we found similar performance between all the think-aloud protocols, while users in the silent condition had considerably lower task completion times. This is similar to [START_REF] Van Den Haak | Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue[END_REF], who showed that the traditional protocol, due to the requirement of thinking aloud while working, had a negative effect on task performance. Similarly, in [START_REF] Hertzum | Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?[END_REF] the traditional and coaching protocols are compared and findings from that study show that the coaching protocol resulted in higher task completion times. Thus, we do witness similar tendencies as reported in the above literature. However, like [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF] we found no significant differences in terms of efficiency.

For the satisfaction ratings, we found that users in the coaching and traditional conditions were significantly more satisfied with the website comp ared to users in active listening and silent. In a similar study [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF] partially agrees they found users in the coaching condition to be most satisfied [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF].

For usability problems, we found that the silent condition revealed significantly fewer usability problems than any of the other conditions ; and even when we were able to observe problems in that condition, we were often unable to explain why they occurred. This is also supported by the finding that most problems were observed simply by users making errors (category E observations). The study in [START_REF] Van Den Haak | Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue[END_REF] compared the traditional with a retrospective think-aloud protocol, which revealed comparable sets of usability problems. In contrast, another study has shown that different protocols can reveal different types of problems. The study presented in [START_REF] Zhao | The impact of two different thinkaloud instructions in a usability test: a case of just following orders? Behaviour & Information Technology[END_REF] compared the traditional and coaching protocols and found that the coaching protocol led to a larger number of usability problems related to dialogue, navigation, layout and functionality . Additionally, they found that the problems which were unique to the coaching condition were mainly at a low level of severity [START_REF] Zhao | The impact of two different thinkaloud instructions in a usability test: a case of just following orders? Behaviour & Information Technology[END_REF]. This is partly supported by our findings that the proportion of critical, serious and cosmetic problems is distributed simi-larly within conditions, also in case of the unique problems identified in each cond ition. However, we did find the types of identified problems to be similar between the coaching and traditional conditions while the active listening condition mainly led to problems concerning unclear information. In terms of problem severity, we found that the all think-aloud protocols revealed twice as many critical, serious and cosmetic problems as the silent condition.

Implications for Usability Practitioners

The most interesting result for usability practitioners is that the think-aloud protocols had only limited influence on user performance and satisfaction, but compared to the silent condition, they facilitated identification of the double number of usability pro blems with the same number of test s ubjects. Recruiting test subjects is a major task in usability evaluation, thus it is important to know how test subjects can be used most effectively. Furthermore, we found it difficult to interpret the causes of problems, which were observed through a lesser richness in problem descriptions.

Our results cannot be used to suggest practitioners to use a specific protocol. For example, the proponents of the Active listening protocol have argued that the Trad itional protocol is unnatural. However, our results do not support this. The two prot ocols facilitate identification of a similar number of usability problems and richness in problem descriptions.

There were some interesting differences in the usability problems that were identified. The Traditional protocol revealed more usability problems in the frustration category, the Coaching protocol revealed more problems identified through test mon itor intervention, and the Silent conditions mainly found usability problems identifie d when users made errors.

The results demonstrate that no single protocol version is superior overall; each has strengths and weaknesses . Yet in the silent condition, we could not capture verbalised problems, because the test subjects were silent, so the comparison is limited by this.

Conclusion

We have conducted an empirical study comparing three think-aloud protocols and a silent control condition. The study assessed the usability problems identified in each condition based on a systematic video analysis. We found that the differences between the three think-aloud protocols were limited. Contrary to the general emphasis on the Coaching protocol, we have no indication that it is superior for identifying usability problems. In fact, there were some aspects where the Traditional protocol performed surprisingly well. Overall, the think-aloud protocols are clearly superior to the silent condition; with the same number of test subjects, they revealed the double number of usability problems compared to the silent condition, and this applied across all levels of severity. Part of the study replicated a previous study on usability measures [START_REF] Olmsted-Hawala | Think-Aloud Protocols: A Comparison of Three Think-Aloud Protocols for use in Testing Data-Dissemination Web Sites for Usability[END_REF]. Here, we found only a few differences between the protocols. There was an indication that efficiency was higher in the Silent condition and that satisfa c-tion with the website was higher for Coaching and Traditional. These limited effects do not contradict the literature as previous studies point in various directions.

Our study has a number of limitations. First, identification of usability problems is not as mechanical as determining usability measuring. Second, the specific experimental design impose limitations on our results, especially with our role as analysts, the moderator/analyst separation, and the physical moderator/user separation, but these were necessary to reduce potential sources of bias.

We deal with usability evaluation and the process of identifying usability problems. This is a controversial topic in HCI where some question the whole approach and its relevance to practice. Contrastingly, others maintain that within certain limits such studies produce relevant results. In order to achieve that, it is not possible to directly copy approaches from practice in the laboratory.

Our study points to interesting directions for future work. Most importantly, the consequences for and approaches used in practice should be stud ied extensively. It is also highly relevant to replicate the study with other types of systems.

Fig. 1 .

 1 Fig. 1. Layout of the usability laboratory. The test moderator was located in the control room (A) and the user in test room 1 (B).

Fig. 2 .

 2 Fig. 2. Venn diagram showing overlap in problems between think-aloud protocols.

Table 2 .

 2 Distribution of participants on conditions and demographic profiles, TAP = Technical and Administrative Personnel n = number of participants.

		Coaching	Active listening	Traditional	Silent
		(n=10)	(n=11)	(n=11)	(n=11)
	M ean age (SD)	34.9 (10.6)	33.2 (10.9)	37.5 (16.5)	40.8 (13)
	Females / M ales	4 / 6	5 / 6	7 / 4	6 / 5
	TAP / Faculty / Students	5 / 2 / 3	3 / 3 / 5	4 / 3 / 4	4 / 4 / 3

Table 3 .

 3 M ean any-two agreement between evaluators, n = number of datasets analysed individually by the evaluators.

		Coaching	Active listening	Traditional	Silent
		(n=6)	(n=7)	(n=7)	(n=7)
	M ean (SD)	0.42 (0.11)	0.44 (0.12)	0.47 (0.08)	0.44 (0.17)

Table 4 .

 4 M ean task completion times in seconds, n = number of users.

		Coaching	Active listen-	Traditional	S ilent
		(n=10)	ing (n=11)	(n=11)	(n=11)
	M ean (SD)	238 (42)	219 (47)	220 (50)	175 (72)

Table 7 .

 7 M ean any-two agreement between users, n = number of all pairs of users.

		Coaching	Active listening	Traditional	S ilent
		(n=45 pairs)	(n=55 pairs)	(n=55 pairs)	(n=55 pairs)
	M ean (SD)	0.18 (0.07)	0.16 (0.09)	0.21 (0.08)	0.14 (0.09)

Table 8 .

 8 Pairwise comparisons with respect to any -two agreements between users, ▫ = almost significant difference, * = significant difference, ** = highly significant difference.

		Coaching	Active lis-	Traditional	S ilent
			tening
	Coaching		p>0.8	p>0.4	P<0.02 *
	Active listening	p>0.8		p>0.09	P>0.1 ▫
	Traditional	p>0.4	p>0.09		P<0.001 **
	S ilent	P<0.02 *	P>0.1 ▫	P<0.001 **

Acknowledgments

We are grateful to Lise Tordrup Heeager, Rasmus Hummersgaard, Rune Thaarup Høegh Mikael B. Skov, Henrik Sørensen and the 43 users who helped us in the study.