Andrés Lucero
email: lucero@acm.org

M Ads Clausen

Using Affinity Diagrams to Evaluate Interactive Prototypes

Keywords: Interaction Design, KJ M ethod, Evaluation, Analysis. 1

Affinity diagramming is a technique used to externalize, make sense of, and organize large amounts of unstructured, far-ranging, and seemingly dissimilar qualitative data. HCI and interaction design practitioners have adopted and used affinity diagrams for different purposes. This paper discusses our particular use of affinity diagramming in prototype evaluations. We reflect on a decade's experience using affinity diagramming across a number of projects, both in industry and academia. Our affinity diagramming process in interaction design has been tailored and consists of four stages: creating notes, clustering notes, walking the wall, and documentation. We draw examples from eight projects to illustrate our particular practices along these four stages, as well as to ground the discussion.

Introduction

Researchers have recently been looking into and studying different (design) methods and reflecting on how they are used in practice by the human -computer interaction (HCI) and interaction design communities. Most of these studies have looked into methods that were originally conceived within and are closely related to design pra ctice, such as probes [START_REF] Lucero | How probes inform and influence the design process[END_REF], [START_REF] Wallace | M aking design probes work[END_REF], workbooks [START_REF] Gaver | M aking spaces: how design workbooks work[END_REF], and mood boards [START_REF] Lucero | Framing, aligning, paradoxing, abstracting, and directing: how design mood boards work[END_REF]. First introduced in the 1960s, affinity diagramming (or the KJ method) [START_REF] Beyer | Contextual Design: Defining Customer-Centered Systems[END_REF], [START_REF] Holtzblatt | M ixedNotes: A Digital Tool to Prepare Physical Notes for Affinity Diagramming[END_REF], [START_REF] Kawakita | The original KJ method[END_REF] has its origins in anthropology and social s cience and has widely been accepted within HCI research. Affinity diagramming is a technique used to externalize, make sense of, and organize large amounts of unstructured, far-ranging, and seemingly dissimilar qualitative data [START_REF] Hartson | The UX Book: Process and Guidelines for Ensuring a Quality User Experience[END_REF]. Common uses of affinity diagramming include analyzing contextual inquiry data [START_REF] Beyer | Contextual Design: Defining Customer-Centered Systems[END_REF], [START_REF] Holtzblatt | M ixedNotes: A Digital Tool to Prepare Physical Notes for Affinity Diagramming[END_REF] clustering user attributes into profiles [START_REF] Kuniavsky | Observing the User Experience: A Practitioner's Guide to User Research[END_REF] or requirements [START_REF] Benyon | Designing Interactive Systems: A Comprehensive Guide to HCI, UX and Interaction Design[END_REF], problem framing and idea generation [START_REF] Geyer | Designing reality-based interfaces for creative group work[END_REF], [START_REF] Sugihara | GKJ: Group KJ method support system utilizing digital pens[END_REF] and prioritizing issues in usability tests [START_REF] Hanington | Universal M ethods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions[END_REF].

In this paper, we reflect on a decade's experience using affinity diagramming to evaluate interactive prototypes. Our affinity teams usually consist of two researchers who collect data from 10 to 24 participants (i.e., observations of use during a task, and semi-structured interviews), independently write affinity notes (i.e., 500 to 2500 notes), and jointly analyze the data (i.e., build an affinity diagram) over a period of two to three weeks. To better suit small to medium interaction design projects in in-dustrial and academic contexts, we have tailored and scaled down Beyer and Holtzblatt's six stages of contextual design [START_REF] Beyer | Contextual Design: Defining Customer-Centered Systems[END_REF], [START_REF] Holtzblatt | M ixedNotes: A Digital Tool to Prepare Physical Notes for Affinity Diagramming[END_REF] to four stages. First, when creating notes, we embrace the affordances of paper [START_REF] Klemmer | The designers' outpost: a tangible interface for collaborative web site[END_REF], [START_REF] Luff | Only touching the surface: creating affinities between digital content and paper[END_REF] and build affinity diagrams with physical paper by producing handwritten sticky notes. Second, when clustering notes, we invite team members to go through each other's notes in sequence, to avoid ownership issues and to create a better understanding of the context when an observ ation of use is made. We also avoid using interview questions to structure the data, letting overarching topics naturally emerge [START_REF] Cox | Supporting collaborative interpretation in distributed Groupware[END_REF] from it. Third, in walking the wall, we take advantage of color-coded sticky notes (i.e., one color per participant) to check at a glance whether enough people have raised an issue. We also discuss practices rela ted to pruning the wall, which include merging, arranging, and removing note clusters. Finally, in documentation, we pick relevant user quotes and count notes to commun icate and quantify our main findings. The main contributions of this paper include: a systematic analysis of affinity diagramming use for prototype evaluations in HCI and interaction design over an extended period; an adaptation of earlier affinity diagra mming techniques such as the ones described by Beyer and Holtzblatt [START_REF] Beyer | Contextual Design: Defining Customer-Centered Systems[END_REF], [START_REF] Holtzblatt | M ixedNotes: A Digital Tool to Prepare Physical Notes for Affinity Diagramming[END_REF] which have been tailored to suit small to medium projects; and a discussion on practices that are relevant for general affinity diagramming.

This paper is structured as follows. We begin by discussing related work on affin ity diagrams, how they are used in HCI and interaction design, and existing support tools. We then take real-life examples from eight industrial research projects to illustrate the four stages of our particular use of affinity diagramming for interactive prototype evaluations. We close by discussing what we have learnt by adapting affinity diagramming to our own practices, followed by conclusions.

2

Related Work

Affinity Diagramming and the KJ Method

Affinity diagramming is a technique used to externalize, make sense of, and organize large amounts of unstructured, far-ranging, and seemingly dissimilar qualitative data [START_REF] Hartson | The UX Book: Process and Guidelines for Ensuring a Quality User Experience[END_REF]. Japanese anthropologist Jiro Kawakita devised the KJ method [START_REF] Kawakita | The original KJ method[END_REF] (on which affinity diagramming is based on) as a tool for use in anthropology to synthesize idiosyncratic observations of raw data obtained through fieldwork to find new hypoth eses. In Japan, the KJ method has become popular as a systematic approach to problem solving in fields such as research, invention, planning, and education. Affinity diagrams (or KJ method charts) are a wall-sized paper-based hierarchical representation of data. The KJ method consists of four basic steps [START_REF] Kawakita | The original KJ method[END_REF]. In label making, the main facts or issues in relation to the data are captured onto separate pieces of paper (or sticky notes). Rather than grouping notes in predefined categories, affinity diagrams are built from the bottom up. Therefore, in label grouping, individual notes are shuffled and spread out on a table, and then read several times. After interpreting and considering their underlying significance [START_REF] Hanington | Universal M ethods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions[END_REF], individual notes are put up on a large empty table or blank wall one at a time, forming teams of labels (or clusters) that are iteratively rearranged [START_REF]The Usability Engineering Lifecycle: A Practitioner's Guide to User Interface Design[END_REF]. 'Lone wolves', or notes that do not seem to fit in the existing clusters, are left aside for later use. Clusters are then given titles (or named) and, if working on a table with pieces of paper, all notes are put together in a pile with the title clipped on top. Cluster names and 'lone wolves' are read and grouped into more abstract groups giving rise to general and overarching themes [START_REF] Koskinen | Design Research Through Practice: From the Lab, Field, and Showroom[END_REF]. In chart making, the resulting clusters are spatially arranged and transferred to a large sheet of paper, where they are annotated using symbols an d signs (i.e., connection, cause and effect, interdependence, contradiction) to show the relationships between groups. Finally, in explanation, the resulting chart is first verbally explained and then described in writing.

Although conceived for individual use, the KJ method is well suited for collaborative data analysis, supporting parallel work and creation of a shared interpretation of the data. The greater-than-human-sized space often used allows people to simultaneously view, discuss, and modify the artifact [START_REF] Klemmer | The designers' outpost: a tangible interface for collaborative web site[END_REF]. The simple skills needed to fill a table with pieces of paper, or a wall with sticky notes, and move notes around to suggest new associations make affinity diagramming (or the KJ method) a tangible, easy, and approachable way to look into complex data.

Different Uses of Affinity Diagrams

Contextual Inquiry. Beyer and Holtzblatt [START_REF] Beyer | Contextual Design: Defining Customer-Centered Systems[END_REF], [START_REF] Holtzblatt | M ixedNotes: A Digital Tool to Prepare Physical Notes for Affinity Diagramming[END_REF] adapted the original KJ method to analyze observational and interview data gathered during contextual inquiries [START_REF] Curtis | Customer-focused design data in a large, multi-site organization[END_REF], [START_REF] Hanington | Universal M ethods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions[END_REF], [START_REF] Koskinen | Design Research Through Practice: From the Lab, Field, and Showroom[END_REF][START_REF] Kuniavsky | Observing the User Experience: A Practitioner's Guide to User Research[END_REF]. Affinity diagramming is often used as a starting point for design [START_REF] Koskinen | Design Research Through Practice: From the Lab, Field, and Showroom[END_REF], helping keep design teams grounded in data as they design [START_REF] Hanington | Universal M ethods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions[END_REF]. The contextual design process begins with contextual interviews, where field data from between four and six different work sites is gathered in an attempt to understand work practice across all cu stomers. Second, interpretation sessions and work modeling allow every team member to experience the interviews and capture key points (affinity notes), after which five models of the user's work are created (i.e., flow, cultural, sequence, physical and artifact models). Third, consolidation and affinity diagram building consist of merging the data from the five models and the affinity notes to represent the work of the targeted population. Fourth, in visioning the team reviews the consolidated data by walking the data and then running a visioning session of how the user's work will be streamlined and transformed. Fifth, storyboarding consists of fleshing out the vision in detail using hand drawn pictures and text. Finally, paper prototypes and mockups interviews consist of designing user interfaces on paper and testing them with users. While Beyer and Holtzblatt's use of affinity diagrams is aimed at the early stages of the design process , our work looks at how affinity diagrams can provide support to analyze interactive prototype evaluations in the later stages of the design process. Moreover, stages such as contextual interviews, work modeling, storyboarding , or paper prototypes and mockups interviews are no longer relevant once interactive prototypes are in place and ready to be evaluated, as those activities should have happened earlier in the process.

User Profiles and Requirements . At the start of developing a new software or product, user profiles (or personas) are constructed to provide a direction to the whole process [START_REF] Kuniavsky | Observing the User Experience: A Practitioner's Guide to User Research[END_REF]. Team members create a list of audience attributes on sticky notes, thus bringing their own views on the user. These attributes are then collectively clustered into three to eight user profiles. Notes that do not fit in exis ting clusters create new ones. Clusters are discussed and notes are moved around until everyone agrees on them. Sometimes it is the customers themselves (i.e., experts in a given field) who create an affinity diagram to specify requirements for a new system [START_REF] Benyon | Designing Interactive Systems: A Comprehensive Guide to HCI, UX and Interaction Design[END_REF], [START_REF] Cox | Supporting collaborative interpretation in distributed Groupware[END_REF]. Starting from the design brief, experts write down fairly s uccinct statements (i.e., requirements, needs, wishes and hopes) on sticky notes capturing the features, properties and expected behaviors of parts of the new system. Requirements are clustered into co mmon themes, while near-duplicate ones are discarded.

Problem Framing and Idea Generation. In design practice, affinity diagrams are used to analyze a design problem or to create first design solutions [START_REF] Cox | Supporting collaborative interpretation in distributed Groupware[END_REF], [START_REF] Geyer | Designing reality-based interfaces for creative group work[END_REF]. With a given design problem in mind, designers write down words, short sentences, or create small sketches on sticky notes to stimulate diversity in ideation phases. Ideas are then clustered to identify common issues and potential solutions, ultimately helping to frame the design problem. Used to achieve consensus and understanding among pa rticipants in a discussion [START_REF] Sugihara | GKJ: Group KJ method support system utilizing digital pens[END_REF], a radial affinity diagram is usually built on a table . A key problem or theme is first placed in the center of the base sheet. Participants then take turns in placing their cards on the table, aligning similar cards in a spoke-like fashion. Visual connections between similar cards can be created using lines of paper clips. After discussion, participants fix the card positions and links. In web design, affinity diagramming is used as a form of collaborative sketch ing [START_REF] Klemmer | The designers' outpost: a tangible interface for collaborative web site[END_REF] to create sitemaps with the structure of a website. Information architects and visual designers first collect ideas about what should be in a website onto sticky notes and then arrange them on the wall into categories, usually on a whiteboard. Sitemaps can grow fast and end up including 200 to 300 notes. Visual designers sketch page designs directly on empty spaces of the whiteboard. Usability Tests. In usability testing, affinity diagrams are also used to help teams prioritize which issues will be fixed and retested [START_REF] Cox | Supporting collaborative interpretation in distributed Groupware[END_REF], [START_REF] Hanington | Universal M ethods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions[END_REF]. At the start of a usability test session, the team assigns a sticky note color to each participant and watches as they perform tasks from an observation room. Observations and quotes are captured on the notes, which are put up on a wall. Common interface issues and problems will emerge. Several colored notes in one issue will indicate that many people experienced a similar problem and should probably be addressed first.

In HCI and interaction design, affinity diagrams have also been used to analyze (post-task) interview data from interactive prototype studies [START_REF] Buur | Video card game: an augmented environment for user centred design discussions[END_REF], [START_REF] Nunes | Sharing digital photographs in the home through physical mementos, souvenirs, and keepsakes[END_REF]. In this paper, the novel and particular use of affinity diagrams in prototype evaluations to analyze both observations of use while participants perform a task and (post -task) interview data is discussed [START_REF] Kildal | Twisting touch: combining deformation and touch as input within the same interaction cycle on handheld devices[END_REF], [START_REF] Lucero | Image space: capturing, sharing and contextualizing personal pictures in a simple and playful way[END_REF], [START_REF] Lucero | M obile collocated interactions: taking an offline break together[END_REF][START_REF] Lucero | NotifEye: using interactive glasses to deal with notifications while walking in public[END_REF].

Affinity Diagramming Support Tools

Software tools have been available to create affinity diagrams. CDTools1 was a software package to support the Contextual Design process offered by InContext De sign. PathMaker2 is a split screen interface that allows recording, dragging, and grouping ideas into affinity sets. StickySorter3 allows working visually with large collections of notes. Koh and Su [START_REF] Koh | Interactions for addressing shortcomings encountered when physically creating and manipulating large affinity diagrams[END_REF] created an affinity diagram authoring environment that provides an infinitely large workspace to post and organize notes, dynamic group layout and repositioning, meaningful zoom levels for navigation (i.e., to fit note, group of notes or selection area), ways to save, restore and export diagrams (JPEG), as well as ways to search for notes and groups (i.e., by text matching and related words). The main disadvantages of these tools include a lack of support for collaboration (i.e., single user) and having to do things in a certain structured way [START_REF] Harboe | Computer support for collaborative data analysis: augmenting paper affinity diagrams[END_REF].

Support for affinity diagramming has also been available in prototype form. The Designers Outpost [START_REF] Klemmer | The designers' outpost: a tangible interface for collaborative web site[END_REF] combines the affordances of paper and large physical workspaces with the advantages of digital media. People write on sticky notes with a normal pen, which are then added to an electronic whiteboard used as a canvas. Using computer vision, the system tracks the position of the notes, which can be physically moved around or removed from the board. Physical notes can be structured, by drawing lines from one note to another, and annotated with digital pens. AffinityTable [START_REF] Geyer | Affinitytable -a hybrid surface for supporting affinity diagramming[END_REF] replicates the benefits of physical affinity diagramming (i.e., copying, clustering, piling and collecting) and enhances current practices (i.e., highlighting, focusing, searching, retrieving images) by combining one vertical and one horizontal display, plus digital pen and paper. Multi-touch gestures and physical tokens provide input to the interactive table. The GKJ system [START_REF] Sugihara | GKJ: Group KJ method support system utilizing digital pens[END_REF] provides a way to digitize affinity diagrams and the process of building them. Using wireles s Anoto-based pens, the system records annotations on cards and on a base sheet. Pen gestures are used to determine the position and orientation of a card, as well as to group/ungroup clusters of cards. Time stamped gestures allow visiting the history of the affinity diagram process, and also serves as an undo function. A PC editor allows further editing the virtual diagram. Harboe et al. [START_REF] Harboe | Two thousand points of interaction: augmenting paper notes for a distributed user experience[END_REF] proposed a distributed system of digital devices (i.e., mobile phones, tablets, stationary camera, and projector) to augment affinity diagramming while aiming to support existing paper-based practices. Using a Magic Lens metaphor, paper notes tagged with unique QR codes can be tracked with the camera of a mobile phone or tablet. Once recognized, the note can be augmented with additional metadata information, which is projected on top of the physical note. The prototypes discussed here aim to augment affinity diagramming, however Klemmer et al. [START_REF] Klemmer | The designers' outpost: a tangible interface for collaborative web site[END_REF] report that some digital features have the potential to interrupt the designers' creative flow and can be considered distracting (i.e., too many things flashing).

Despite the pervasiveness of new technologies, paper remains a critical feature of work and collaboration [START_REF] Luff | M obility in collaboration[END_REF][START_REF] Luff | Only touching the surface: creating affinities between digital content and paper[END_REF]. Luff et al. [START_REF] Luff | Only touching the surface: creating affinities between digital content and paper[END_REF] discuss some of the affordances of paper that seem critical to human conduct. Paper is mobile as it can easily be relocat-ed and juxtaposed with other artifacts, and micro-mobile as it can be positioned in delicate ways to support mutual access and collaboration. Paper can be annotated in ad hoc ways, allowing people to track the development of the annotations and reco gnize who has done what. Paper is persistent [START_REF] Klemmer | The designers' outpost: a tangible interface for collaborative web site[END_REF], retaining its form and the character of the artwork produced on its surface. In addition, paper allows people to simultan eously see its contents from different reading angles, and it can become the focus of gestures and remarks [START_REF] Luff | M obility in collaboration[END_REF]. The affordances of paper have played a key role in our practices with affinity diagrams, including our preference to us e physical paper to digital alternatives, as well as to manually write notes on sticky notes.

Eight Affinity Diagrams

We reflect on a decade's experience using affinity diagramming to evaluate intera ctive prototypes, both in industry and academia. Real-life examples from eight industrial research projects where the technique has been used (see Table 1 for an ove rview) will help illustrate how affinity diagrams are used in practice, as well as to ground the discussion. Prototypes A to E [START_REF] Lucero | M obile collocated interactions: taking an offline break together[END_REF] were related to groupware, exploring the use of mobile phones for collaborative interactions in different physical and social use contexts (i.e., office work, media consumption at home, public expression in a pub, and general group formation). These prototype evaluations were conducted with different nu mbers of participants (i.e., between six and 27 people) in groups of varying sizes (i.e., between three and nine people per group). Prototypes F, G, and H, on the other hand, were evaluated individually (between 10 and 24 participants per evaluation). Prot otype F [START_REF] Lucero | Image space: capturing, sharing and contextualizing personal pictures in a simple and playful way[END_REF] was a social network service built around sharing personal photos. Prototype G [START_REF] Kildal | Twisting touch: combining deformation and touch as input within the same interaction cycle on handheld devices[END_REF] allowed using a flexible handheld interface to provide input for interaction. Finally, Prototype H [START_REF] Lucero | NotifEye: using interactive glasses to deal with notifications while walking in public[END_REF] explored the use of interactive glasses to provide notifications on the go. Most of these evaluations were conducted in controlled lab env ironments, except for prototypes C, F, and H, which took place in public spaces. In all prototype evaluations, two researchers independently made notes as they watched videos of participants performing an interaction task and a semi-structured interview. Handwritten notes were created for all prototypes but one, i.e. C, which used digital notes printed on label templates only. In addition, digital notes printed on sticky notes or on paper (plus removable tape) were produced for four prototypes (i.e., A, B, D, and E), resulting in a mix of handwritten and digitally created notes. The size of the team that built the affinity diagram ranged from two to five people, and always included the same two researchers who made the notes in the first place. The resulting affinity walls differed in size, ranging from 232 to 2243 notes. Around 15% of the notes were discarded, except for two cases, prototypes G (39%) and H (26%).

Affinity Diagramming Process

We outline stages and properties of our particular use of affinity diagramming for interactive prototype evaluations. As was mentioned earlier, Beyer and Holtzblatt's [START_REF] Beyer | Contextual Design: Defining Customer-Centered Systems[END_REF], [START_REF] Holtzblatt | M ixedNotes: A Digital Tool to Prepare Physical Notes for Affinity Diagramming[END_REF] use of affinity diagrams is intended for the early stages of the design process, and some of their stages are not relevant for prototype evaluations. We have tailored their process to better suit small to medium interaction design projects in industrial and academic contexts. More specifically, we have combined their first two stages (i.e., contextual interviews and interpretation sessions and work modeling) into creating notes by placing prototype evaluations at the core, and removing the work modeling activity. Their third and fourth stages (i.e., consolidation and affinity diagram building and visioning) are closely related to clustering notes and walking the wall, respectively. Finally, their last two stages (i.e., storyboarding and paper prototypes and mockup interviews) have been replaced by documentation. Our process consists of four stages: creating notes, clustering notes, walking the wall, and documentation.

Creating Affinity Notes

This first stage of the process starts with the actual evaluation of the prototype. After carefully planning the main research questions, internal ethics co mmittees and privacy reviews, consent forms, the introduction, the task, coffee breaks, the semi-structured interview questions, the debriefing, and rewards, we collect data over a period of one or two weeks (depending on the number of participants). Each evaluation session usually lasts between one and two hours. As a result, we end up with 6 to 24 hours of video to analyze, both from observations of use during a task, and from the semistructured interviews. Interpretation sessions are then conducted wit hin 48 hours after the prototype evaluations. Two researchers with mixed background (i.e., a designer and a psychologist, or a designer and a computer scientist) independently make notes as they watch videos of an interaction task and a semi-structured interview. Affinity notes typically include handwritten text, but can also comprise drawings and annotations (Fig. 5). The number of affinity notes can vary between 500 and 2500, depen ding on the number of interviews, their duration and the level of detail ca ptured. It usually takes us twice as much time to write affinity notes as the length of the videos. Handwritten Sticky Notes. We begin by placing a stack of sheets of A3 paper on our desk (Fig. 1a). These sheets are used as a canvas onto which to stick notes. A3 is a comfortable format to work with while writing affinity notes; it is large enough to put several sticky notes on it, and small enough to fit on a desk in front of a computer. Traditionally, standard 3x3 or 5x3 inch (7.6x12.7 cm) Post-it notes have been used to write affinity notes. However, we have found those sizes to unnecessarily increase the overall size of the affinity wall and thus the amount of walking for the affinity team members. Moreover, larger affinity notes invite more verbose expressions from the note takers. Therefore, we use the smaller 2x2 or 2x1.5 inch (5.1x3.8cm) sticky notes, which are available in different brands, colors, and slightly different sizes (Fig . 1a). These smaller sticky notes fit in a grid of 8 by 6 notes, totaling 48 notes per sheet. We assign a sticky note color for each participant (Fig. 1b). In case of group evaluations (e.g., prototypes A to E on Table 1), we define one color for each group of participants. With smaller groups of people (e.g., three participants), we try to treat them as individuals as much as possible and thus assign three sticky note colors, one for each. Using different note colors for each participant allows us to tell how many people raise a certain issue by glancing at each category.

Blank sticky notes are arranged (usually in columns) and are given a unique ident ifier consisting of a participant or session number, followed by a running sequence number (e.g., P1_01 or S1_01), which is handwritten in the lower-right corner. Note takers do not need to identify themselves on each note as their handwriting provides a quick way to know the author of a given note. We draw a single thick black line to separate task from interview data (Fig. 1b). We place two or three blank sticky notes on the A3 canvas before starting the video to take notes. To optimize our use of time, subsequent notes are then added and identified in parallel during natural transitions or silent moments in the video. However, some parts of th e video do require us to explicitly pause and rewind the video to capture a note in more detail. Those breaks also provide opportunities to add new blank notes to the canvas. Once a sheet of A3 paper is filled with 48 affinity notes, that sheet is moved to the back of the stack. Working with a stack of sheets allows the note taker to quickly consult whether something has been missed or captured earlier. Once all videos have been analyzed, the sheets of A3 paper filled with affinity notes can now be easily transported to the affinity room.

Digital Notes Printed on Paper, Labels or Sticky Notes . We have also experimented with using different digital note types in our evaluations of prototypes A to E (T able 1). First, digital affinity notes are typed on person al computers, then printed on paper, and each note is individually cut with scissors [START_REF] Harboe | Computer support for collaborative data analysis: augmenting paper affinity diagrams[END_REF] (Fig. 1c). In an attempt to provide a similar function and feel as sticky notes, removable tape, blue painter's tape, or yellow masking tape are used so the notes can be easily attached and moved around the affinity wall. The extensive manual work needed to pro duce such notes can easily delay the start of building the affinity wall by a couple of hours. Second, label templates (e.g., Avery®) are another alternative to printing digital affinity notes (Fig. 1d). Since the entire surface of the note then becomes ad hesive, it produces mixed results as we have had difficulties removing the notes from the wall depending on the wall finishing materials (e.g., paint, wallpaper, wood, glass, etc.). Third, affin ity notes can be laser printed on sheets of Post-it® notes. However, priced at almost US$1 per note and available in one-size (3x3 inches), this alternative is both expensive and impractical. We have come up with a cheaper way to print digital affinity notes on standard sticky notes (Fig. 1e). MixedNotes [14] is a software tool that imports text from any text editor, recognizes blank lines as separates notes, adds a unique sequential identifier, optimizes the size of the note to be printed based on the amount of text, and shows an on-screen preview of how notes will be printed on paper. MixedNotes first prints a background sheet, a template onto which sticky notes manually cut in three different sizes are attached, and then the same sheet must be put back into the printer for the digital notes to be printed on the sticky notes. Despite claims that digital notes could improve current practices (i.e., faster note-taking, searching), we have not found this to be the case (e.g., search option only used once).

Clustering Notes

Selecting and Preparing a Room. Holtzblatt et al. [START_REF] Holtzblatt | M ixedNotes: A Digital Tool to Prepare Physical Notes for Affinity Diagramming[END_REF] stress the importance of getting a dedicated team room for the duration of the project to avoid wasting time finding another room, packing up materials, and relocating half way through the process.

However, not all organizations have a room that can be blocked for one or two weeks.

We have used different strategies to secure such a dedicated space for a couple of weeks both in companies and universities. For example, we have reserved an internal meeting room for two weeks (Fig. 2c), used our own shared office (Fig. 2d), used a common innovation space outside our premises, temporarily repurposed a usability lab (Fig. 2a), and used an office space that was emptied before major renovations (Fig. 2e). Room sizes have ranged from a 2.3x3.6m meeting room (Fig . 2c) to a 6x8.4m usability lab (Fig. 2a). The room should provide plenty of wall space for the affinity to spread out [START_REF] Holtzblatt | M ixedNotes: A Digital Tool to Prepare Physical Notes for Affinity Diagramming[END_REF]. We begin by mounting the sheets of A3 paper containing the affinity notes on the wall (between 1.2-2m from the floor) using masking tape (Fig. 2b). We then line up the remaining wall space with white flipchart sheets, butcher paper, or statically charged polypropylene film onto which the note clusters will form. In addition, we sometimes put white cardboard panels on tables (Fig. 2d). When using smaller rooms, we have even used the windows and the door (Fig . 2c). Our ideal room (Fig. 2a) has moveable whiteboards that we line up with white flip chart sheets of paper held up by magnets. Such a space allows us to rearrange parts of the wall, be it entire whiteboard panels or just a couple of sheets. In such a space, we mount affinity notes towards the left and right wall edges, leaving an empty space in the middle for the affinity wall.

Affinity Diagramming for Prototype Evaluations . When building an affinity wall to analyze prototype evaluation data, the two note takers who created the affinity notes are already familiar with the data since they have seen the videos for all se ssions. In cases where the affinity team has included more than two persons (e.g., prototypes B and C), although these people have not created affin ity notes, they have been present earlier as observers during the actual prototype evaluations. Therefore, unlike Holtzblatt et al. [START_REF] Holtzblatt | M ixedNotes: A Digital Tool to Prepare Physical Notes for Affinity Diagramming[END_REF], at the start of building the affinity we do not break the notes up into piles of 20 (i.e., to make it less intimidating). Instead, we invite people to start by going through each other's notes sequentially to avoid potential ownership issues. Our data also more heavily depends on the details of the context when an observation of use was made, therefore we do not mix up the notes (i.e., mix up users). For example, when going through the first part of the evaluation (i.e., the task), it is important for us to be able to identify whether, e.g., a certain function was successfully triggered or a group was having problems on the first, second or third try. An is olated note saying, "they are having difficulties completing the task," will be interpreted differently whether this is happening at the start or by the end of the task.

Despite the seemingly structured way of analyzing the interaction observations, when going through the second part of the data (i.e., the semi-structured interview questions) we avoid as much as possible using the interview questions to structure the data. Instead, we let overarching topics naturally emerge from the data.

Building the Affinity. At the start of building the affinity, team members start by reading each other's notes in silence (Fig. 2c). People will pick notes that raise important issues in relation to the prototype and begin forming rough clusters (Fig. 2d). Once a couple of clusters have been created with a few notes, people will begin ve rbally coordinating where certain notes are being clustered. Questions such as, "where are you putting the notes related to this issue?" will begin to emerge. Thus we alternate between moments of silence and moments of discussion, the former near the start of the process, the latter as the affinity progresses. Clusters with a few notes below them are initially named and labeled with a blue note (Fig . 2e). These clusters are in turn grouped into more abstract groups labeled with pink notes.

Walking the Wall

Discussing and Pruning the Wall. After the team has completed a first round reading all notes, roughly between a third and half of the affinity notes will have been moved from the A3 sheets of paper to the affinity wall, thus forming note clusters (Fig. 3a). Early rounds of discussing the wall will then concentrate on communicating the emerging clusters with the team, checking if these clusters fail to cover some important general topics, and identifying overlapping clusters that could potentially be merged. As a result of these discussions, the teams will agree on an initial set of clusters and (blue and pink) labels. Drawing arcs [START_REF] Hartson | The UX Book: Process and Guidelines for Ensuring a Quality User Experience[END_REF] and sticking stripes of tape [START_REF] Judge | Studying group decision making in affinity diagramming[END_REF] between clusters can be used to visually show related parts of the affinity diagram. A note that belongs in two clusters can be duplicated , or split by ripping the paper and adding tape. The team may also decide at this point to define a tentative cluster hierarchy. On one hand, fixing a hierarchy this early on in the process tends to limit the bottom-up nature of the process. On the other hand, shifting panels around late into the process to define a hierarchy can have a detrimental effect. Social and spatial awareness of the affinity diagram is an important part of building a cognitive model of the data [START_REF] Cox | Supporting collaborative interpretation in distributed Groupware[END_REF], [START_REF] Harboe | Computer support for collaborative data analysis: augmenting paper affinity diagrams[END_REF], [START_REF] Kawakita | The original KJ method[END_REF]. Moving parts of the affinity wall can create confusion and lead to a waste of time when people are trying to find where to place a given note.

In later rounds of discussing the wall, the team will more closely inspect the co ntents of each cluster (Fig. 3b). Specific notes that are unrelated to a cluster may be put on a different cluster, set aside to potentially create a new cluster, or even back in its original location on the A3 sheets of affinity notes. Similarly, clusters may be merged, moved to a different location, or can altogether disappear. Pruning the wall thu s includes merging clusters, arranging the cluster hierarchy, and removing notes (and clusters) from the affinity wall. Adding Notes to Existing Clusters . As was mentioned earlier, the affinity team will switch back to reading affinity notes from the A3 sheets of papers in between rounds of discussing and pruning the wall. As the affinity wall prog resses, team members will be more familiar with the existing clusters and have a better understanding of where a given note could be put up on the wall. As a result, people will evolve from moving one note at a time to the wall, to several simultaneously (Fig. 3c). Each finger can represent a cluster, and several notes can be placed on each finger, and thus ten notes can be moved to the wall at the same time. Another phenomenon that we have observed at these later stages of the process is that team members are more open to verbalize the act of moving a given note up to the wall (Fig . 3d).

-"Listen to this note [reads the note aloud]. It goes here." -"Yes." Such actions are performed to confirm that existing categories remain valid also after new bits of data are added to the wall. Each round of discussing and pruning the wall, and adding notes to existing clu sters often takes between 60 and 160 minutes. Besides alternating between these two different types of activities, it is important to have 30 to 60 minut e breaks in between rounds. Affinity diagramming can be a mentally demanding activity [START_REF] Holtzblatt | M ixedNotes: A Digital Tool to Prepare Physical Notes for Affinity Diagramming[END_REF], especially when it is carried out for three to five days in a row. Natu ral breaks such as lunchtime or taking a coffee, but also an unrelated meeting can provide a much -needed time for the mind to rest and think about something else, other than the ongoing analysis.

Finalizing the Wall. Once there are no more useful notes left on the wall, typically the last 15% of the notes (Table 1), we read each category note by note and make sure each note belongs to that category. We also make sure there are enough notes and more than two people raising an issue in each category. We then check that every blue and pink note still makes sense within the hierarchy. For each blue label, a succinct user statement describing the issue that holds all individual notes together is written on a large sticky note [START_REF] Beyer | Contextual Design: Defining Customer-Centered Systems[END_REF] (Fig. 3e).

Documentation

Creating a Digital Record of the Affinity Wall. Keeping a digital record of the finalized affinity wall (Fig. 4b) allows sharing the results across sites. Miura et al. [START_REF] Sugihara | GKJ: Group KJ method support system utilizing digital pens[END_REF] indicate that in the typical KJ method, there is one participant who digitizes the ou tcome of the diagram by inputting all card content and arranging the card structure using a mouse and keyboard. Similarly, we assign a scribe [START_REF] Klemmer | The designers' outpost: a tangible interface for collaborative web site[END_REF], a member of the affinity team who saves the resulting info. Using PowerPoint or Word, a document with the hierarchical category structure and its descriptions is created (Fig . 4a). Depending on the number of affinity notes and detail of the documentation, roughly one to three PowerPoint slides are made for every pink label. Each slide contains the name of the pink and blue labels, the user statement written in first person that describes all notes under a blue label, plus a selection of the most representative user quotes on that pa rticular issue (typically between 1 and 4 user quotes per blue label). Picking relevant user quotes, ones that capture the essence of what participants tried to tell us, will play an important role in communicating the main (positive and negative) findings of the study to different stakeholders, improving existing designs, and further disseminating the end results (e.g., publication at an academic conference).

In addition to making a digital version of the affinity diagram, we take (highresolution) digital photographs of each panel. Each photo usually contains one pink note, two to five blue notes, plus their corresponding affinity notes that make up those categories. We sometimes also take close-up panel shots (e.g., if we need to leave the room and save the results to a digital document at a later time). Proper lighting cond itions and (color) contrast between the text and note should be considered when ph otographing panels [START_REF] Koh | Interactions for addressing shortcomings encountered when physically creating and manipulating large affinity diagrams[END_REF]. To visually assist the wall documentation process, we sometimes build the final hierarchical structure of categories on a whiteboard (Fig . 4c).

Quantifying Observations of Use and Raised Issues . Another particular use of affinity walls for interaction design that we have developed over the years is to count the total number of notes and the number of people that raised an issue. Counting the total number of notes allows us to check how frequently the participants mentioned an issue or topic. When creating the final affinity wall hierarchy, the overall note numbers for each pink and blue label provide us with an additional way to prioritize one topic over another. Special care should be taken to identify if a category with a large number of notes consists of one or two people mentioning the same issue repeatedly.

Similar as for usability testing [START_REF] Hanington | Universal M ethods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions[END_REF], we also count the number of people that raised an issue. This is where using different note colors comes in handy as we can glance at a category and quickly get a sense of how many different people mention a certain issue (Fig. 4b). By doing this, we are able to quantify our (mostly) qualitative fin dings. An opening statement such as "most participants (16/20) explicitly said the prototype was easy to use" will usually accompany a qualitative finding. Such statements allow us to shed light on and better ground our qualitative findings.

Cleaning Up and Discarding the Affinity Wall. After the affinity diagram has been duly documented, the room must be cleaned up. Besides taking the affinity wall and the (mostly) empty sheets of A3 paper down from the wall (Fig . 4d), the room needs to be arranged back to its original state (i.e., moving tables and chairs, and packing materials). Lining up the wall space with large sheets of paper at the start of the process for note clusters to form can greatly speed up the clean up process. Removing large sheets of paper is easier done and faster than manually removing 500 to 2500 notes one by one. Rolling up the affinity diagram to temporarily store it should be avoided, as the notes will tend to bend and might altogether fall [START_REF] Koh | Interactions for addressing shortcomings encountered when physically creating and manipulating large affinity diagrams[END_REF]. Once the affinity wall has been taken down, and depending on internal practices regarding data ha ndling and privacy, it is time to shred and discard the data (Fig . 4e). Depending on the final number of notes, cleaning up and discarding the wall can take up to two hours.

Discussion

Return on Investment and Impact

A recurring question within organizations when using affinity diagrams in interaction design evaluations is that of resources. Beyer and Holtzblatt [START_REF] Beyer | Contextual Design: Defining Customer-Centered Systems[END_REF] first collect data from 15-20 participants, producing 50 to 100 notes for each two-hour interview, for a total of 1500 notes. They then recommend having one person per 100 note s to build the affinity diagram in one day, for a total of 15 people. Due to the large number of pa rticipants (i.e., 10 to 24) and the resulting notes involved (i.e., 500 to 2500), our affin ity diagramming process takes a team of two researchers between two and three weeks to collect the data (i.e., one week) and complete the analysis (i.e., one to two weeks). It takes us two to five days to individually write affinity notes, two to five days to build the affinity wall with the affinity team, and one or two days for an assigned scribe to document the wall. Holtzblatt et al. [START_REF] Holtzblatt | M ixedNotes: A Digital Tool to Prepare Physical Notes for Affinity Diagramming[END_REF] describe two-person projects where it can take two to three weeks to gather requirements for participant numbers of less than ten. Therefore, we feel that our two-person team being able to analyze data from up to 24 people in the same amount of time is a good success indicator in terms of resources. Thanks to the mix of qualitative and quantitative analysis, plus the level of detail in our findings for each prototype evaluation, we have been able to easily just ify assigning two people full-time to work on an affinity diagram.

Regarding impact, by systematically introducing affinity diagramming to our industrial research projects we were able to not only find existing UX and usability issues with our interaction designs , but also to identify and define new lines of research, requirements that had to be integrated to our designs, and new ideas tha t were filed as invention reports . While most resulting affinity diagram clusters covered issues with the interactive prototype that was currently being evaluated, for every project we had two additional panels, one labeled 'Future Research Areas/Other Topics' and another one 'Ideas/IPR'.

Number of Notes for Observations of Use and Interview

When creating affinity notes for interaction design studies, it is difficult to estimate the ratio of observation of use notes during the task to semi-structured interview notes. For a 45-minute video consisting of 15 minutes of data for the task and 30 minutes for the semi-structured interview, one would perhaps expect a 1:2 ratio in the final number of notes as can be seen from Fig. 1b, where 20 and 40 notes were made respectively. However, we have come across unusually large numbers of notes from the observation part, sometimes even reaching 50% of the total number of notes. The evaluation of Prototype E (Table 1) was related to groupware and explored general group formation using mobile phones for collaborative interactions. The fact that there were six participants trying out different strategies to form a group made it increasingly difficult to keep track of the overall situation. Tasks were taking place in parallel and there were many micro interactions happening. Although the task videos in this case were relatively short (i.e., 15 minutes), the complexity of the interaction data had a big impact on the final number of notes. Similarly, we roughly estimate that for a 45-minute video it would take 90 minutes (or double the time) to generate affinity notes. However, in the case we are currently discussing, the amount of details in the interaction also had an effect on the overall note-taking time.

Our suggestion is for the note takers to start the interpretation sessions , go through the videos for the same two or three participants, and check how many notes each has made for the task and the semi-structured interview parts for each participant. Although there can be differences in the level of detail that each note taker captures, this coordination provides a reasonable estimate of whether someone is focusing on too much detail or being too general before all sessions have been interpreted.

Note Types and Leftover Notes

The most common affinity note types include observations of use (Fig . 5a), good participant quotes for use in publications (Fig. 5b), and design issues or ideas. As stated earlier, roughly 15% of the notes are discarded (i.e., notes that are not included on the final affinity wall). Some of these notes are used to perform general counting (e.g., how many pedestrians and bicycles a participant encountered during the task) (Fig. 5c), to mark certain parts of the evaluation (e.g. "Task given by [the] instructor") (Fig. 5d), to draw an arm posture (Fig. 5e), to personally indicate how well a task was performed (Fig. 5f), to draw different group formation strategies (Fig. 5g), or to record contextual information (e.g., "Police drives slowly past :)") (Fig. 5h). While note types 5c-5h may at first seem superfluous, we argue for their importance from a holistic interaction design perspective as they help researchers thoroughly analyze micro-interactions, social interactions, and other contextual factors that may have an effect on the overall results. Prototypes F, G and H (Table 1) have unusually large numbers of unused notes (23%, 39%, and 26%). Based on the total number of affinity notes, these three projects could be considered average in size at 505, 1037, and 1276 notes. These three evaluations were conducted individually, which may cause note takers to make and write down more disconnected, random, and anecdotal observations. However, we do not believe this to be the sole source for the large number of discarded notes. We also ran out of time to go in more depth and further remove notes from the A3 sheets of paper to transfer them to the final affinity wall, as we spent three or four days to build these affinity walls.

Conclusion

By reflecting on a decade's experience using affinity diagramming across a number of projects, we have discussed how we have tailored the process for use in HCI and interaction design evaluations to four stages: creating notes, clustering notes, walking the wall, and documentation. Digital affinity diagrams can be especially convenient for situations when data are available in digital format (e.g., tweets, Facebook or YouTube comments), allowing single users to perform data coding, clustering, count-ing, and statistics in Excel, and are easy to transport. Despite existing attempts to augment affinity diagramming by making parts of the process digital, we have found that traditional paper affinity diagrams are better suited for collaborative analysis, they support building a cognitive model of the data by social and spatial awareness, and allow people to quickly transition from directly moving data around to having the full overview of the wall by simply taking a few steps away from or towards the wall.

Providing such a flexible access to data in digital format, and in a way that is suitable for collaborative analysis would require the use of very large displays. In addition, we have embraced the affordances of paper by producing handwritten sticky notes, as we have not found alternative digital note types to offer clear advantages (i.e., speed, searchability). In tailoring Beyer and Holtzblatt's affinity diagramming process to provide better support when analyzing interactive prototype evaluations , we have been able to better understand the context of use by checking the data sequentially, perform a micro-interaction analysis by creating and looking into detailed notes that might otherwise be discarded (e.g., counting, body postures, strategies), and to quant ify qualitative findings at a glance using color-coded sticky notes.

Fig. 1 .

 1 Fig. 1. Creating affinity notes. (a) Writing notes on sheets of A3 paper, (b) handwritten sticky notes, (c) digital notes printed on paper, cut with scissors and attached with tape, (d) digital notes printed on labels, and (e) manually printed sticky notes.

Fig. 2 .

 2 Fig. 2. Clustering notes. (a) A typical room at the start of clustering with notes on the left and right, and empty space in the middle, (b) preparing the room by putting up sheets of affinity notes on the wall, (c) going through notes individually, (d) forming first clusters on a white cardboard panel, and (e) clusters have initial names and a few notes below them.

Fig. 3 .

 3 Fig. 3. Walking the wall. (a) Affinity notes have been moved the A3 sheets of paper to form the affinity wall, (b) people slowly start discussing the contents of each category, (c) moving several notes at a time when there are enough categories, (d) verbalizing the act of moving a note to the wall, and (e) user statements in first person are written on larger yellow sticky notes.

Fig. 4 .

 4 Fig. 4. Documenting the wall. (a) Digitally recording the wall using a laptop, (b) a final affinity wall with pink and blue labels, (c) building the final category structure on a whiteboard, (d) cleaning the room to its original state, and (e) shredding the affinity wall.

Fig. 5 .

 5 Fig. 5. Leftover affinity notes. (a) Observations of use, (b) participant quote, (c) general counting, (d) personal separator, (e) drawing of body posture, (f) personal counting, (g) formation strategies, and (h) personal contextual observation.

Table 1 .

 1 Overview of eight affinity diagramming cases.

		Evaluation	Evaluation	Note	Note	Affinity	Affinity	Notes	Notes
	Prototype	T ype	Participants	T akers	T echnique	Notes	T eam	on Wall	Discarded
	A MindMap	Group (2x3)	6	2 Mix	258	2	232	10%
	B Pass-them-around Group (5x4)	20	2 Mix	811	3	740	9%
	C MobiComics	Group (3x9)	27	2 Printed	2433	5 2243	8%
	D EasyGroups	Group (6x4)	24	2 Mix	1096	2	992	9%
	E FlexiGroups	Group (4x6)	24	2 Mix	715	2	673	6%
	F Image Space	Individual	10	2 Handwritten	505	2	391	23%
	G Twisting Touch	Individual	24	2 Handwritten 1037	2	630	39%
	H NotifEye	Individual	13	2 Handwritten 1276	2	946	26%

CDTools. http://incontextdesign.com/process/cdtools/

PathM aker. www.skymark.com/pathmaker/tour/brain.asp

StickySorter. www.youtube.com/watch?v=Wg0WYcYlls0

Acknowledgments. A big shout out to Dima Aliakseyeu, Selene Mota, Marion Boberg, Hannu Korhonen, Jussi Holopainen, Tero Jokela, Akos Vetek, Jari Kangas and Deepak Akkil who co-built the affinity walls shown here. Thanks also to Jacob Buur, Robb Mitchell, and the anonymous reviewers for insightful comments on this paper.


Myunghee Lee

Gerard J Kim
email: gjkim@korea.ac.kr

On Applying Experience Sampling Method to A/B Testing of Mobile Applications: A Case Study

Keywords: Experience Sampling M ethod (ESM ), A/B Testing, Usability

With the advent of mobile devices, the experience sampling method (ESM ) is increasingly used as a convenient and effective way to capture user behaviors of, and evaluate mobile and environment-context dependent applications. Like any field based in-situ testing methods, ESM is prone to biases from unreliable and unbalanced data, especially for A/B testing situations. M itigating such effects can in turn incur significant costs in terms of the number of participants and sessions, and prolonged experimental time. In fact, ESM has rarely been applied to A/B testing nor do existing literatures reveal its operational details and difficulties. In this paper, as a step toward establishing concrete guidelines, we describe a case study of applying ESM to evaluating two competing interfaces for a mobile application. Based on the gathered data and direct interviews with the participants, we highlight the difficulties experienced and lessons learned. In addition, we make a proposal for a new ESM in which the experimental parameters are dynamically reconfigured based on the intermediate experimental results to overcome the aforementioned difficulties.

Introduction

Experience (or environ me nt) sampling method (ESM ) is a system evaluation and behavior capture method by which user evaluative responses are made and recorded at the exact time and place of the system usage. Co mpared to the old paper-andpencil method, with the advent of mobile devices, ESM can be carried out mo re conveniently e.g. as or through functionalities embedded in smart phone sensors and applications. Like any field-based in-situ testing methods, ESM can suffer fro m biases that might otherwise be controllab le in a laboratory setting, but at the same time, they can be mitigated through a high number of repetition s, extended length of experimentation, a large nu mber of part icipants , and thus at higher cost. However, this can also ironincally bring about even more unreliab le and unbalanced data. This is more problematic in the case of a co mparative evaluation in which, fo r a validity and fairness, it is necessary to collect a minimu m and "balanced" amount of reliab le data. In fact, ESM has rarely been applied to A/B testing nor do existing literatures reveal its operational details and difficult ies. In this paper, as a step toward esta blishing concrete guidelines for A/B testing with ESM, we describe a case study of applying ESM to evaluating two competing interfaces for a mobile application.

Based on the gathered data and direct interviews with the participants, we h ighlight the difficult ies experienced and lessons learned. In addition we make a proposal for a new ESM in wh ich the experimental parameters are dynamically reconfigured based on the intermed iate experimental results to overcome the aforementioned difficulties and run the experiment more economically.

Related Work

The Experience Sampling Method (ESM) was first introduced by Larson and Csiks zentmihaly i as a research tool in social science [START_REF] Larson | The experience sampling method[END_REF], but has found great utility especially in mobile HCI research [START_REF] Conner | Experience sampling and ecological momentary assessment with mobile phones[END_REF]. For instance, the Context Aware Experience Sampling Tool developed by Intille et al. [START_REF] Intille | A context-aware experience sampling tool[END_REF], one of the first of its kind, allowed a flexib le data solicitation by scripting in the survey questions and multip le choice answers, and including a functionality for users to easily capture and store mult imed ia data on a PDA. Consolvo et al. used a similarly designed ESM tool, called the iESP, to evaluate ubiquitous computing applications and further analyzed the possible pitfalls and lessons learned of using such a methodology for in -situ HCI evaluation (e.g. the effect iveness of self-reporting using the mob ile devices and the need for tailoring the data collection process for the target subjects ) [START_REF] Consolvo | Using the experience sampling method to evaluate ubicomp applications[END_REF]. In 2007, Froehlich et al. also introduced a more advanced mob ile based ESM tool called My Expereince which offered an XM L based specification method of how to solicit data fro m the user, sensor based context triggered s amp ling and structured storage of logged data to a data base server [START_REF] Froehlich | MyExperience: a system for in situ tracing and capturing of user feedback on mobile phones[END_REF]. Mo mento, a tool developed by Carter et al. is another step in the evolution of the mob ile based ESM tools offereing sampling control and on -line monitoring (e.g. visualizat ion and analysis of inco ming data) fro m a remote desktop server [START_REF] Carter | Momento: support for situated ubicomp experimentation[END_REF]. Maestro further extended the sampling control capability by explo iting long term user behavior and usage patterns for shaping personalized ESM questions to different types of users [START_REF] Meschtscherjakov | MAESTRO: Orchestrating user behavior driven and context triggered experience sampling[END_REF]. As ESM tools become more refined and enter into one of the main stream evaluation methods, its user interface/interaction design itself has emerged as an important issue as well with regards to the requirement and desire to encourage the participants to make faithfu l and reliab le response [START_REF] Consolvo | Conducting in situ evaluations for and with ubiquitous computing technologies[END_REF]. While not usually emp loyed in ESM, the data collection in crowdsourcing can involve "gold standard" questions to ensure the reliability and credibility of the contributors [START_REF] Downs | Are your participants gaming the system?: Screening mechanical turk workers[END_REF]. Answering performanes to the gold standard questions can be used to exclude certain data, e.g. those that are regarded too mechanical or even those of programmed bots. In summary, it can be seen that ESM tools are continuing to evolve and being added with more functionalit ies (for both the participants and experiment administrators) and the methodology extended for a mo re reliable and cred ible results. In most previous related work we have reviwed, ESM was still used for capturing context dependent user behaviors for a "single" application. According to the recent survey of ESM tools by Conner [START_REF] Conner | Experience sampling and ecological momentary assessment with mobile phones[END_REF], the data collection schedule and design are fixed throughout the experiment, e.g. choice of part icipants, number of participants, sampling time, experiment duration, etc. ESM tool capabilit ies and methodological process need to be further extended fo r scalability and efficiency to handle larger subject pools, longitudinal studies and A/B testing with several factors

3

Case Study: ESM for A/B Testing

Test Application and Evaluated Interfaces

In this case study, we evaluate the usability of two co mpeting interfaces for a simp le mobile map-logging application (see Figure 1) in which a user can record and tag short information about the current user location (indicated on the map and sensed by the GPS sensor). The two competing interfaces compared were for entering informat ion through (1) voice and (2) touch typing of text . The ESM is used because the application is mobile and possibly context dependent (e.g. location, time, social se tting, etc.). Either by voluntary initiat ion or by a scheduled prompt, the participant is asked to try out one of the interfaces (chosen in a balanced order) to enter short information (i.e. record in voice, or type in). In addition, several supplemental conte xtual informat ion (that cannot be easily inferred automatically with the sensors) is s olicited using a menu driven interface, asking whether the participant is indoor or outdoor, the location (e.g. restaurant, classroom, streets), on-going activity (e.g. resting, eating, in a meeting), social situation (e.g. alone, with a friend), etc.

ESM based A/B Testing Process and the Support Tool

Typically, a co mparative UI experiment is conducted in a laboratory setting as a cross sectional study with repeated measures taken in batch. On the other hand, ESM is used to attain more relevant data considering the environment and context of usage, but batch collection of repeated usage is often not feasible. Rather the data is collected over some extended period of time. Nevertheless, we still regard our experiment to be "cross-sectional" since we are not (for now) interested in longitudinal change in the user response. Note ESM has primarily been used for capturing user behavior for a single application rather than for making co mparisons of u sability or UX. In our case, the ESM A/B testing proceeds as a within-subject type (i.e. the user tries out both interfaces and make co mparisons) for a g iven period of time with a predetermined number of part icipants to first collect some minimu m amount o f data deemed sufficient for the power of the experiment. While there are several methods to decide on the least amount of required data or number of participants, for now the experiment duration, amount of data solicited (equally nu mber of repetition) and the number of participants were determined arbitrarily but in a conservative fashion (e.g. long enough to gather good amount of data).

The participants were scheduled in a balanced order to fulfill a task, either using the interface A o r B, and make answers to usability and gold standard questions. Due to the difficu lty in inferring particular usage contexts automatically (e.g. whether a pe rson is moving, in a meeting, at a bus station, sleeping, etc.), the data solicitation was done according to a regular time schedule rather than invoked by automat ic context detection. Figure 1 shows the overall flow of the ESM A/B testing process.

Detailed Experiment Procedure

The first phase of the ESM based A/B testing was conducted for six days, and solicited for data entry 8 times a day. For now, the duration of one week for the first phase was decided rather arb itrarily. A total of 30 subjects (20 males / 10 females) mostly in their 20's with various occupational backgrounds participated in the study. The participants were recruited, interviewed and selected through an on-line social messaging system. The part icipants were given instructions as how to download and install the s martphone application, and how and when to try out the tasks and make proper data entries. Prior to the actual experiment, the participants were given instructions for a short training session for getting oneself familiarized with the application, two interfaces and data entry method. The participants were compensated upon the completion of the whole session at rate of $0.23 per answered questions (which totaled to about $11 dollars maximu m). Considering the recommendation by [START_REF] Abdesslem | Mobile experience sampling: Reaching the parts of Facebook other methods cannot reach[END_REF], the task trials and data solicitations were scheduled every two hours only between 7:30 am to 10:30 p m (total of eight times a day). At the scheduled times, the application was invoked on the smartphone device automatically (4 t imes each for the respective interfaces in an alternative order), and an alarm was used to remind and notify the participant. Despite the reminders, it was up to the participants to actually respond. It was also possible that the alarm or smartphone itself was switched off. Thus, a few simp le user behavioral checking measures were imp lemented. For example, 30 seconds of no response was regarded as a refusal of a data entry. It was also checked whether the phone was actually in active use before and after the scheduled time to guess whether the "refusal" was deliberate or not. Such piece of behavioral informat ion was to be used collectively to assess the credibility or reliab ility of the part icipant. Upon a scheduled invocation of the task trial, the user was to enter informat ion as attached to wherever the user was at, enter additional contextual information (as described in Section 3.1) and answer a series of usability questions in a 5 Likert scale (on convenience and ease of use, general satisfaction, annoyance, fatigue). Finally, gold standard questions were given to assess and explicitly confirm the credibility and reliability of the participant. The gold standard questions were designed to be fairly easy with the least cognitive burden, yet not answerable by random guesses, such as solving a simp le arithmet ic (e.g. "what is (5+4)*(1+1)?") o r asking of co mmon sense knowledge in mu ltip le choices (e.g. "who is the president of Korea?"). After co mplet ing the task of logging the map either by voice o r text , there were a total o f 10 questions to answer including the two gold standard quiz. After six days, the collected data were analy zed for sufficiency and (as explained more in detail in the next section) it was determined that another round of data collect ion was deemed necessary. A second phase of data collection for A/B testing was continued for another week. After the whole two week sessions, we ad ministered one final survey, asking the participants about the ESM procedure itself (participants were separately compensated for it). The answers were used, in addition to the operational problems discovered during the case study, to base our proposal for an imp roved and extended ESM. We o mit the presentation of the detailed survey questions and only brief the results in Section 4.4.

4

Results, Observations and Proposals

First Phase: Data Sufficiency and Balance

During the first phase (first six days), with 30 participants and 8 data collection sessions per day, we ideally would have collected 1440 sets of balanced and reliab le task trial and session response data. However, only 463 session data (37.2%) were collected due to reasons such as participant not noticing the inco ming alarm (41.8%) and deliberate refusals (20.3%). At a closer look, the co llected data were even mo re insufficient with respect to different contexts and number of participants. For instance, the data collected for @home context comprised more than half of the total data, while the rest scattered in little proportions to other usage location contexts and thus lacking the power for any meaningfu l analysis. The situation was worse for conjunctive contexts such as for a particular location and time, location and activity, etc. Th is was not only attributed to the fact that the data were collected based on a simp le time based schedule (rather than based on intelligent, but technically difficult, context detection), certain context based usages just do not happen as often as others (e.g. staying home vs. rid ing on a subway). It was also possible that by the nature of the application, the users just was not up to using the application as often as necessary to gather sufficient interaction data in a short amount of time. Thus, in order to reduce the experiment duration, save cost, relieve the burden on the participants and ultimately make the study more focused rather than open -ended and exp loratory, we propose that intermittent data analysis would be necessary (as part of an extended ESM A/B testing study methodology) to check data sufficiency and analysis power, carry out the mid-evaluation if possible, and eliminate certain dependent variable measurements if the analysis results are clear (e.g. very low pvalue, high R 2 , h igh χ 2 etc.). In this study, the experiment continued on for another week (second phase) and about the same amount of data were addit ionally co llected (See Figure 2). The co mparative usability between the text and voice based interfaces for the context of @ho me usage (which had sufficient data for ana lysis after the first week) did not change. Note that comparative qualitative assessments of the interfaces can be gathered as well but would require a subjective analysis. In addition, the data collection process and scheduling must be tailored toward the particular context o f interest. If auto matic context detection is technically d ifficult, then a pilot study should be conducted ahead of time e.g. to recru it a participant who is likely to make a particular context-based usage of interest, or personalize the data entry session schedule at a right timing by an analysis of one's daily activities. Furthermore, the data were unbalanced in many ways, e.g. between the treat ments (e.g. data for voice based interface vs. text based), among the participants, and among the contexts. The missed data entry sessions, deliberate or not, did not originate un iformly among different participants. While the unbalanced amount of data between treatments is a serious problem to making co mparat ive analysis, it can be v iewed as an indication of usability or preference. Thus, through continuous monitoring, the ESM must be ad ministered in such a way to solicit data and closely balance the co mpeting data e.g. by scheduling for t reat ments that lack data, encouraging the nonresponding participants, and analyzing whether the unbalanced response is in fact due to preference or certain operational constraint or contexts. Note that such provisions can contaminate the ba lanced presentation of treatments (for mit igating the learning effect), thus must be applied carefully in an incremental manner.

Participant Reliability

The gold standard quiz is only a part ial and ind irect indicator of participant/data reliability. At any rate, we judged that that the data themselves were reason ably reliab le and credible because only less than 5% of the gold standard quizzes were incorrect overall. In addit ion, the response behavior did not change much over the second week. Thus, it seemed more important to single out the participants who tended to "refuse" (especially deliberately so) the data entry in the first place too o ften. Future ESM tools should have the support capability for monitoring for these participants and replacing them if necessary.

Experiment Extension and the Utility of ESM

Because sufficient data were not collected for a meaningful co mparative analysis for different contexts of usage (except for the @home usage), the experiment was continued on for the second week (see Figure 2). Future ESM tools should administer such an extended experiment in a systematic fashion through data analysis. Note that during the second week, the missed data entry session and data unbalance were still at the similar level. Future ESM tools must take measures to minimize these types of data insufficiency. On the other hand, sufficient data were then collected for the @university usage case, and showed different usability results from the @home usage (e.g. participants preferred text based input more for @university than @home). Thus, this at least confirms the very utility of the ESM in that it can ca pture different usability and user behavior depending on the usage context. After the additional experimentation, nothing much has changed except only the data for @university usage context became sufficient (fo r power of analysis). Data for other contexts were still lacking and unbalanced, and the additional data for @home usage did not change the initial analysis results.

Participant Responses about the ESM Process Itself

Participants mostly acknowledged the experimenter's sentiment of the difficulty in collecting reliable data. They suggested for a system of incentive based compens ation, better alarm mechanisms, and pre-notificat ion of the upcoming data entry sessions. Two main reasons for the missed data entry were not being able to notice the alarms and scheduled events overlapping with uninterruptable on-going activities. They also expressed that the gold standard questioning was effective not only as an indicator for credib ility but also encouraged the participants to be more thoughtful and reliable.

Conclusion and Future Work

In this paper, we described a detailed case study of applying ESM to evaluating two competing interfaces for a mobile application. Based on the gathered data and direct interviews with the participants, we h ighlighted the difficulties experienced and lessons learned. In addit ion, we made several proposals for a new ESM (also our future work) with the capabilit ies to flexib ly rev ise the parameters of the experiment on-line so that the ESM can be run economically, efficiently but with the same reliability.

Fig. 1 .

 1 Fig. 1. The overall process of ESM based A/B testing of two interfaces (using the voice or text input) for a simple mobile map logging application.

Fig. 2 .

 2 Fig. 2. Comparative usability data collected during the first six days (first two bars among the four) and after the second week (cumulative, the second two bars) for different location contexts (@home, @university, etc.). The dark blue bar represents data for text, and the light yellow for voice based interfaces.

Acknowledgements

. Th is research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Planning (No. 2011-0030079) and funded by the Forensic Research Program of the National Forensic Serv ice (NFS), M inistry of Go vernment Administration and Home Affairs, Korea. (NFS-2015-DIGITA L-04).