Patrick Harms
email: harms@cs.uni-goettingen.de

Jens Grabowski
email: grabowski@cs.uni-goettingen.de

Usage-based Automatic Detection of Usability Smells

Keywords: task trees, usage-based, automatic usability evaluation

With an increasing number of supported devices, usability evaluation of websites becomes a laborious task. Therefore, usability evaluation should be automated as far as possible. In this paper, we present a summative method for automated usability evaluation of websites. The approach records user actions and transforms them into task trees. The task trees are then checked for usability smells to identify potential usability issues. The approach was applied in two case studies and shows promising results in the identification of four types of usability smells.

Introduction

Usability engineering is important to develop high quality websites [START_REF] Sarodnick | Methoden der Usability Evaluation: Wissenschaftliche Grundlagen und praktische Anwendung[END_REF]. But a fully-fledged application of traditional usability evaluation methods like useroriented usability tests applied for several devices for displaying websites requires a high effort [START_REF] Krug | Web Usability: Rocket Surgery Made Easy[END_REF]. Therefore, in the recent years usability evaluation strives for automation and, hence, cost reduction.

In this paper, we present a new method for automated detection of indicators for usability issues, so called usability smells, based on recorded user actions on websites. The user actions are first transformed into task trees using a methodology proposed by us in [START_REF] Harms | Trace-based task tree generation[END_REF]. Then, we scan the task trees for patterns of usability smells. As a result, our approach generates a list of detected usability smells. For each smell, it provides details about the user tasks and actions that are related to the smell, as well as the elements of the websites (e.g., links and buttons) that are involved. Our method detects four different types of usability smells. We show in this paper, how the smells are detected and clarify if our approach can be used to automatically detect usability issues.

The paper is structured as follows. First, we introduce the terminology and basic concepts used throughout the paper in Section 2. Then, we describe the detection of usability smells using the generated task trees in Section 3. Afterwards, we depict the validation of our approach in two case studies in Section 4 and discuss the respective results in Section 5. Finally, we compare our approach with related work in Section 6 and conclude on our findings with potential for future work in Section 7.

Usability is the result of executing tasks with effectiveness, efficiency, and satisfaction [START_REF]ISO 9241-11: Ergonomic requirements for office work with visual display terminals (VDTs) -Part 11: Guidance on usability[END_REF]. It is influenced by the usage context which comprises the tasks to be executed, the physical environment, and the users performing the tasks. In this paper we consider the usability of websites and, hence, users of websites and the tasks they execute.

A usability issue is a problem that occurs during the usage of a website and decreases one or more of the factors effectiveness, efficiency, and satisfaction for a specific usage context. The problems can be caused by the design of the website, the functionality, the performance, and other aspects. As an example, a login process of a website can be inefficient (usability issue) because users have a long navigation to the login mask (design).

As usability smell, we refer to exceptional user behavior that may indicate a usability issues. For example, if users typically require many actions to perform a login on a website (usability smell) this may indicate a usability issue with respect to inefficiency of the login process. A usability smell has a description and refers to potential usability issues reflected by the smell. Furthermore, the severity of a usability smell indicates the likelyhood that the smell actually identifies a usability issue.

The following terminology is based on [START_REF] Harms | Trace-based task tree generation[END_REF]. A user utilizes a website by performing individual, atomic actions. Examples for actions are a click on a button or typing text into a text field. The set of all actions executable on a website is the bounded set A. Users combine a subset of actions B ⊆ A to execute a specific task on a website. For example, to perform the task of logging in on a website, a user combines the actions of typing his name and his password into two text fields and clicking on a confirmation button. Users can further combine tasks and actions to perform higher level tasks. For example, for performing the task of submitting a transaction on a bank website, a user performs the subtask of logging in on the website, the subtask for submitting the transaction, and a final action to log out by clicking on a button. We refer to the set of all tasks that can be performed on a website as the set T .

A task t ∈ T has an ordered list of children C t = c 1 . . . c n where each child is either an action or another task (c i ∈ (A ∪ T )). Furthermore, a task defines the execution order of its children using temporal relationships [START_REF] Paternò | ConcurTaskTrees : An engineered approach to model-based design of interactive systems[END_REF]. Our approach is based on the task tree generation mechanism proposed in [START_REF] Harms | Trace-based task tree generation[END_REF], which generates the temporal relationships sequence and iteration. In case of a sequence, a task has one or more children and they are executed in their respective order. In case of an iteration, a task has exactly one child which is executed zero or more times. We refer to tasks with a temporal relationship sequence simply as sequence and to tasks with a temporal relationship iteration simply as iteration.

A task is never directly or indirectly its own child. But a single task or an action can be a child of more than one parent task. With these constraints being applied, tasks form a directed acyclic graph. For simplification, this structure is usually represented as a tree by doubling the representation of tasks occurring several times. Therefore, the directed acyclic graph is called a task tree. The leaf Fig. 1. Example for a task tree of a login process on a website nodes of a task tree are the actions a user performs to complete the root task of the task tree. The intermediate nodes in the task tree divide the root task into a logical execution order of subtasks and actions.

An example for a task tree is shown in Figure 1. It represents a login process on a website. The actions are the leaf nodes. The temporal relationships of the tasks define the order in which the children have to be performed. The task (Sequence 1) starts with an iteration (Iteration 1) of a sequence (Sequence 2) for entering the user name. The user can click on the respective text field (Action 1) several times (Iteration 2). The entering of the password is modeled accordingly (Iteration 3). The overall task (Sequence 1) is completed after the user entered the user name and the password and clicks on the login button (Action 5).

The execution of an action a by a user is called an action instance a . The execution of a task t is called a task instance t . A task instance t is an ordered list of action instances a 1 . . . a n defining the order in which the contained action instances were executed. We define |t | as the number of action instances executed in the task instance t . A task instance must adhere to the task that was executed, i.e., the action instances and their order must match the allowed combination of actions defined by the task, its tree structure, and the temporal relationships. A task t is a representation for the set of all its instances. Therefore, we use the notation t ∈ t to indicate that t is an instance of t. Furthermore, |t| is the number of all instances of t.

Task and action instances contain detailed information about the task execution. For example, an action instance has a time stamp ts indicating when the action was executed. For two action instances a i and a j , we define the function ∆ts(a i , a j ) that returns the time difference in milliseconds between the two action instances. Let a 1 be the first action instance of a task instance t and a n be the last action instance of t . Then ∆ts(a 1 , a n ) is a measure for the execution time of t . The smaller the execution time, the higher is the efficiency of t and, hence, the usability of the task execution.

In [START_REF] Harms | Trace-based task tree generation[END_REF], we describe an approach for generating task trees based on recordings of user actions on websites. The resulting task trees contain only sequences and iterations that were performed by the users and that, therefore, represent actual user behavior. We extended this approach to also dump the instances of the detected tasks in the way they were executed by the users.

Automated Usage-based Usability Smell Detection

Using the generated task trees as input, we identify usability smells. We currently consider the smells missing feedback, important tasks, required inefficient actions, and high website element distance. These are described in the following subsections. For each usability smell, we provide a foundation, a description of the expected user behavior, and a description of our expectations towards the generated task trees. Furthermore, we describe the detection of the smells based on the generated task trees as well as a measure for their severity. For most smells, we also provide an example.

Usability Smell: Missing Feedback

Foundation When utilizing a websites, users require feedback for the actions they perform [START_REF] Polson | Cognitive walkthroughs: A method for theory-based evaluation of user interfaces[END_REF][START_REF] Ferré | Usability basics for software developers[END_REF][START_REF] Norman | The design of everyday things[END_REF][START_REF] Balbo | Leading Web Usability Evaluations to WAUTER[END_REF]. Otherwise, they are not sure if the action has any effect and if it is processed by the system.

Expected User Behavior Users repeat an action, if they don't see any feedback. For example, if users click on the login button during a login process and nothing visible happens, they click the same button again. Users may repeat this action even more than twice until they see some feedback.

Expected Occurrence in Generated Task Trees Repeated actions lead to iterations in the task trees. But not all iterations of actions indicate the smell. For example, a user may repeatedly scroll a page. Therefore, we consider only a subset of iterations of actions which covers iterations of clicks on buttons and links.

Users may accidentally perform a click twice. Furthermore, they may perform a click a second time after a long period of time. For example, they may click on a download link and click the same link again to download the same file a second time. Therefore, we consider repetitions of actions as indicator of missing feedback only if they occurred within a specific time frame. This time frame may not span a full iteration of actions. For example, a user may have clicked a download link for a file three times. The first two times belong to the time frame, the third not. Therefore we consider only subsequences of repeated actions. Let t be an instance of an iteration of an action a and a 1 . . . a |t | are the repeated executions of a. For each pair of action instances (a i , a j ) where 1 ≤ i < j ≤ |t |, we calculate ∆ts(a i , a j ). We consider an iteration of actions only, if for at least one of its instances the time difference ∆ts(a i , a j ) is larger or equal to 1,000 and smaller or equal to 15,000. This means, we consider only repetitions of actions performed in a time frame of 1 to 15 seconds. All other repetitions are considered accidental multiple clicks or multiple clicks for some reason.

Detection We filter the tasks for iterations of actions. We then consider the instances of each iteration. For each instance, we calculate a measure for the missing feedback. For this, we first subdivide the list of repeated actions a 1 . . . a |t | into subsequences subseq i,j = a i . . . a j where 1 ≤ i < j ≤ |t | and for which 1, 000 ≤ ∆ts(a i , a j ) ≤ 15, 000. We choose these subsequences as large as possible and non-overlapping. For this, we scan the list of repeated actions for the first subsequence matching the criterion and for which ∆ts(a i , a j ) is maximized but still smaller or equal to 15, 000. We then search for the next subsequences starting with action instance a j+1 . We repeat this searching until we find no further subsequence matching the criterion. The result of this process is a list of subsequences subseq i1,j1 . . . subseq i k ,j k where k is the number of disjunctive subsequences. For the missing feedback measure of an iteration instance, we define a function mf (t ) as follows:

mf (t ) =    subseq i,j (∆ts(a i , a j ) • (j -i)) |subseq i1,j1 . . . subseq i k ,j k | > 0 0 |subseq i1,j1 . . . subseq i k ,j k | = 0 (1)
Through this calculation, the missing feedback measure increases with the time the user waits for repeated clicks and the number of repetitions. If the user clicks very often, the missing feedback measure increases accordingly. For each iteration of actions t, we then calculate the average missing feedback for all instances as follows:

mf (t) = 1 |t| t ∈t mf (t ) (2)
Severity The average missing feedback measure defines the severity of the usability smell. The usability smell is considered present if mf (t) of an iteration is greater than 50ms.

Example An example for a smelling iteration representing missing feedback is shown in Figure 2a). Iteration 1 has 3 instances. The first instance has a repetition of Action 1 (Click on Button "OK"). Based on the time stamps of the action instances, ∆ts for this instance was calculated to be 3000ms. For the three instances, mf (t) evaluates to 1000ms. As this value is higher than 50ms, the usability smell missing feedback is present.

Usability Smell: Important Task

Foundation Tasks performed regularly should be executed with a minimal number of actions [START_REF] Lecerof | Automatic support for usability evaluation[END_REF][START_REF] Tidwell | Designing Interfaces -Patterns for Effective Interaction Design[END_REF]. If applicable, the tasks can be supported by automation, e.g., using macros.

Expected User Behavior If users perform a task regularly, they execute always the same or similar actions.

Expected Occurrence in Generated Task Trees Regularly performed user tasks are reflected as nodes in the generated task trees.

Detection The task trees were generated based on the ordered set of recorded action instances RA. We search the task tree for tasks being executed most often, i.e., covering most of the recorded action instances ra ∈ RA. We only consider sequences as only sequences represent common combinations of actions and other tasks. For each sequence t, we calculate the number of all action instances covered by all instances t ∈ t and divide it by the number of all recorded action instances. Through this, we get the ratio

r(t) = 1 |RA| t ∈t |t i | (3)
of action instances covered by instances of sequence t with respect to all recorded action instances. The higher this ratio, the more actions are executed in the context of task t. Therefore, the number of actions executed for task t should be minimized or automated for improving the usability.

Severity We use the ratio r(t) as indicator for the severity of the detected usability smell. The smell is considered present, if a task covers at least one percent of the recorded actions, i.e., r(t) ≥ 0.01.

Usability Smell: Required Inefficient Actions

Foundation Some actions are known to have no or minimal effect for fulfilling a task but also do not prevent its completion [START_REF] Patern | Remote usability analysis of multimodal information regarding user behaviour[END_REF]. For example, when filling out a form, vertical scrolling may be required to reach the next element of the form to be filled out but does not semantically advance the process. We call such actions inefficient actions. The number of inefficient actions should be minimized to increase the users efficiency.

Expected User Behavior Users perform inefficient actions during the normal usage of a website. In the case an inefficient action is required to fulfill a task, users will perform it when required.

Expected Occurrence in Generated Task Trees As the task trees are generated based on all recorded actions, they will include inefficient actions. If an inefficient action is required at a specific point in the execution of a task, the generated task trees include this action accordingly.

Detection For each task t, we calculate the average ratio of inefficient actions ria(t). For this, we first consider each instance t ∈ t and determine the number of instances of inefficient actions ia(t ). As inefficient action, we currently consider vertical and horizontal scrolling of single panels and whole pages. Then, we calculate the average inefficient action ratio as

ria(t) = 1 |t| t ∈t ia(t ) |t | (4)
Severity The higher ria(t), the higher is the likelyhood of task t to have a usability issue with respect to inefficiency. Therefore, ria(t) is the severity of the usability smell. We consider this smell present, if ria(t) ≥ 0.1, i.e., more than 10% of the action instances covered by instances of a task are inefficient actions.

Example An example for the smell is shown in Figure 2b). Here, Sequence 1 has two instances. Both instances have two action instances of which one is inefficient (the scroll action). For Sequence 1, ria(t) = 1 2 ( 1 2 + 1 2 ) and evaluates to 0.5 indicating a usability smell with a high severity.

Usability Smell: High Website Element Distance

Foundation The elements of a website required for executing a task, such as buttons and text fields, should be as colocated as possible to ensure an efficient task execution [START_REF] Lecerof | Automatic support for usability evaluation[END_REF].

Expected User Behavior Users utilize website elements in the order that best fits the task they use the website for.

Expected Occurrence in Generated Task Trees The task trees show the order of website element usage.

Detection A website consists of several pages. A page may be subdivided by several panels. Several pages may contain elements that are semantically identical. For example, all pages of a website have a menu with identical structure and menu items. We consider two distinct website elements identical if they serve the same purpose on different pages, such as a menu item. We consider two website elements close to each other, if they are on the same page in the same panel. Two website elements are less close to each other, if they are on the same page but in different panels. Furthermore, two website elements have a high distance if they belong to different pages and do not serve the same purpose. Based on this, we define a measure for the distance d(e 1 , e 2 ) of two website elements e 1 and e 2 as follows:

d(e 1 , e 2 ) =               
0.0 e 1 and e 2 are identical or have the same purpose 0.2 e 1 and e 2 belong to the same panel 0.5 e 1 and e 2 belong to the same page 0.75 e 1 and e 2 belong to the same website 1.0 e 1 and e 2 belong to different websites [START_REF] Paternò | ConcurTaskTrees : An engineered approach to model-based design of interactive systems[END_REF] As a panel in a page, we only consider Hyper-Text Markup Language (HTML) div elements as they are nowadays mostly used for structuring pages. Div elements can contain nested div elements. Furthermore, they may be freely positioned on a page using, e.g., Cascading Style Sheets (CSS). Therefore, the nesting of div elements in the HTML source code of a page is no indicator for the actual positioning on a rendered page. Hence, we consider two website elements in the same panel only if their direct parent div element is the same.

For detecting the smell, we apply the distance measure d(e 1 , e 2 ) as follows. We only consider sequences as only sequences represent common combinations of actions and, hence, website elements. For each sequence instance t ∈ t, we determine an ordered list of website elements in the order they were used in the sequence instance. We remove from this list the website elements used for inefficient actions (see Section 3.3) as these are not semantically required for completing a task. The result is the ordered list E(t ) = e 1 . . . e n of website elements used in t . For each subsequent pair of E(t ), we calculate the distances d(e i-1 , e i ) and sum them up to the cumulative distance between all website elements used in t . Furthermore, we determine the number of distances, which is |E(t )| -1. For the whole sequence t, we sum up the cumulative distances of all sequence instances t ∈ t and divide it by the sum of all distances. This results in the measure gd (t) which represents the average distance between two subsequently used website elements for sequence t.

gd (t) = t ∈t |E(t )| i=2 d(e i-1 , e i ) t ∈t (|E(t )| -1) (6)
Severity gd (t) is the severity of the usability smell. We consider this smell present if gd (t) for a sequence is higher than 0.5, i.e., if in average two subsequently used website elements are not on the same page.

Example An example for the smell is shown in Figure 2c). The tree on the top is an instance of a sequence called "Sequence 2". The tree on the bottom represents the structure of the corresponding website (root node). The website has 2 pages (children of the root node) which consist of several HTML tags and their respective structuring (all other nodes). An HTML tag may have an id to differentiate two HTML tags having the same name.

The instance of Sequence 2 consists of three action instances. The action instances utilize |E(t )| = 3 elements of two different pages of the website: first "link1" on the first page, then "link2" on the second page, and finally button "OK" on the first page again. The cumulative distance of the website elements for this instance is 1.5 and the number of distances is 2. As this is the single instance of Sequence 2, the resulting distance measure for Sequence 2 is gd (t) = 1.5 2 . This value is greater than 0.5 and, therefore, a usability smell is present.

Case Studies

We implemented our approach based on the tool suite for Automatic Quality Engineering of Event-driven Software (AutoQUEST) [START_REF] Herbold | AutoQUEST -Automated Quality Engineering of Event-driven Software[END_REF]. In addition to others, AutoQUEST allows recording user actions for different platforms. We utilized AutoQUEST's ability to record user actions on websites. For this, each page of a website must include a JavaScript provided by AutoQUEST. The recorded action instances are sent to a dedicated AutoQUEST server which stores them intermediately. Afterwards, they are fed into AutoQUEST for further processing.

AutoQUEST provides the mechanism for generating task trees based on recorded user actions which we described in [START_REF] Harms | Trace-based task tree generation[END_REF]. As explained in Section 2, we extended this mechanism with the ability to generate task instances. Furthermore, we implemented the detection of the usability smells described in the previous section. Finally, we used this implementation in two case studies.

Description of the Case Studies

In the first case study, we applied our approach on the website of our research group [START_REF]Software Engineering for Distributed Systems[END_REF] at the University of Göttingen. This website offers information about the members of our research group (e.g., contact information), details and materials of lectures we offer to students, as well as details of our research including publication lists. Members of the research group can login on the website to edit the content. The website offers 18 different types of pages. As type of page, we consider several pages having the same structure. For example, all details pages with information about the group members have the same structure and, therefore, are of the same page type.

The website is based on a content management system in which we integrated AutoQUEST's mechanism for recording user actions. As a result, we recorded 33,690 action instances over a period of 8.5 months. The task tree generation generated 1,847 tasks of which 1,431 were sequences and 416 were iterations. Further details of the case study can be found in Table 1. In the second case study, we applied our approach on a master studies application portal on our university. This portal is subdivided into two major parts. The first part allows student applicants to apply for master studies at our university. In a wizard based fashion, applicants are asked to enter their personal data and information about previous studies and achievements. At the end of the wizard, they get an overview of the entered data in which they are asked to upload their CV, a photo and certificates, e.g., about former studies. In the second part of the portal, a reviewer team can verify the applications and accept or reject them including entering reasons. The portal offers a login mechanism including functions for registration and changing the password.

Case

The application portal is based on a content management system in which we integrated AutoQUEST's mechanism for recording user actions. As a result, we recorded 241,306 action instances over a period of 4.5 months. The task tree generation generated 10,634 tasks of which 9,530 were sequences and 1,104 were iterations. Further details of the case study can be found in Table 1.

The number of reported usability smells in both case studies and their distribution to the different smell types is shown in Table 2. Our approach reported 5,900 smells of which 1,464 were found in the first, and 4,436 were found in the second case study. These were too many to be interpreted manually. Furthermore, many findings referred to tasks that cover only a small set of recorded action instances or that were executed only seldom. Hence, we considered these smells as less valuable. Therefore, we performed a preselection of the detected smells based on a measure for the importance of the tasks for which the smells were found. Similar to the respective smell, we considered a task more important, the more recorded action instances it covers. We selected a variable percentage q of tasks that cover the most recorded action instances and being, therefore, most important. Then we selected only those smells that referred to the selected tasks. We adjusted q so that for both case studies at most 100 smells resulted from the selection. Setting q to 5%, i.e., selecting only smells for those 5% of tasks that cover most of the recorded action instances, resulted in 96 detected smells for the first case study and 75 smells for the second case study. In both case studies, several smells of the same type were reported for a task and also for one or more of its children. We call these reports redundant findings. We considered them as a hint to better analyze the cause of a finding for a parent task. The number of redundant findings for each smell is shown in a dedicated column in Table 2.

Findings for the Smell Missing Feedback

The detection of missing feedback (Section 3.1) revealed 91 tasks with usability smells of which 52 were found in the first case study and 39 in the second. There were no redundant findings. Based on the filtering we considered 12 smells for the first case study and 9 for the second. The smells referred to the following repetitions of clicks:

-First case study

• clicks on 9 links for inside the website (e.g. the menu)

• clicks on 2 links for navigating inside the publication list • clicks on 1 download link for a file -Second case study

• clicks on 6 links and buttons concerned with user registration and login • clicks on 2 buttons used for navigating back and forth in the wizard • clicks on 1 button used to download and view an uploaded file

Findings for the Smell Important Task

The detection of important tasks (Section 3.2) resulted in 32 tasks with 14 redundant findings. 19 smells with 10 redundant findings were found in the first case study and 13 smells with 4 redundant findings in the second. The filtering of the smells did not filter any of the detected smells. This is due to the fact, that the filter for the smells and the filter for important tasks use the same mechanism and that the filter for the smells is less restrictive. Therefore, we considered all found smells for this smell type. The smells referred to the following types of task, that overlap each other:

-First case study • 9 tasks that included at least one inefficient scrolling action • 6 tasks including the navigation via the menu of the website • 6 tasks concerned with the login process • 5 tasks for showing the overview and details of the lectures offered by our group • 4 tasks for showing details of our group members • 2 tasks for filtering the list of publications of our group • 2 tasks for showing details of our research in general -Second case study

• 2 tasks that included at least one inefficient scrolling action • 6 tasks representing the utilization of a date chooser in the wizard • 3 tasks concerned with the login process • 2 tasks for entering details about former studies • 1 task for navigating in the wizard • 1 task for uploading files

Findings for the Smell Required Inefficient Actions

The detection of required inefficient actions (Section 3.3) resulted in 4,731 findings of which 1,077 were redundant. 959 smells with 199 redundant findings were found in the first case study and 3,772 smells with 878 redundant findings in the second. The filtering of the smells reduced the amount of considered smells to 54 with 17 redundant findings in the first case study and 51 with 18 redundant findings in the second case study. The smells referred to the following types of tasks:

-First case study • 26 tasks including the usage of the menu • 19 tasks for showing the overview and details of the lectures offered by our group • 11 tasks for showing details of our group members • 9 tasks concerned with the login process • 7 tasks for filtering the list of publications of our group • 4 tasks for showing details of our research in general • 2 tasks for editing content • 1 tasks for showing news of our research group -Second case study

• 15 tasks for providing application details during the usage of the wizard • 12 tasks for uploading files • 7 tasks for showing details of a specific application on the reviewer part • 6 tasks concerned with the login process • 5 tasks for navigating in the wizard • 5 tasks representing the utilization of a date chooser in the wizard • 4 tasks for providing review details and rejection reasons on the reviewer part • 2 tasks for submitting and revoking an application

Findings for the Smell High Website Element Distance

The detection of high website element distances (Section 3.4) found 1,064 usability smells of which 169 were redundant. 435 smells with 66 redundant findings were found in the first case study and 612 smells with 103 redundant findings in the second. The filtering of the smells reduced the amount of considered smells to 11 with 3 redundant findings in the first case study and 2 with no redundant finding in the second case study. The smells referred to the following types of tasks:

-First case study

• 10 tasks for showing the overview and details of the lectures offered by our group • 9 tasks including the usage of the menu • 3 tasks concerned with the login process -Second case study

• 1 task for navigating in the wizard • 1 task for showing details of a specific application on the reviewer part

Discussion

Our case studies show, that our approach finds usability smells. The number of detected smells that indicate true usability issues are listed in Table 2 in the "true positives" columns. To decide, which of the smells are true positives, we performed a manual check for each of the filtered smells. However, the severity of several found smells is rather low. This should be addressed by adjusting the lower boundary of the severity for each smell which must be exceeded to consider a finding. Furthermore, a ranking of the smells is required based on their severity. The detection of missing feedback shows good results. We consider 17 of the 21 smells detected in both case studies as real usability issues. The findings for internal navigation on our research website are inline with our observations. During the recording period, our research website had a performance problem which may have caused users to click several times on the same link. We addressed this issue by adjusting the configuration of the content management system. We also consider the findings for the login process in the second case study as usability issues, as we also here experienced performance issues. The findings for the next and previous buttons in the wizard of the application portal are false positives as the users utilized these buttons subsequently to navigate between several wizard views. The findings of missing feedback for file downloads in both case studies could not be confirmed. The downloads were usually fast enough and the progress was indicated by the web browser. These findings may show, that the 1 and 15 seconds boundaries used in the smell detection require an adjustment.

The findings for important tasks are helpful in both case studies. We consider 21 of 32 as true positives. The other findings referred to tasks that included only one efficient action and, therefore, did not represent useful action combinations. Based on the findings for our research website, we saw findings for five tasks representing navigations from the start page via the list of lectures to details of a specific lecture. This shows that details of a specific lecture are often required. We now consider to put links to ongoing lectures to the start page to reduce the number of clicks required for accessing this information. The same applies to details of our group members and the login process. For the application portal, the usage of the date chooser is most important. Therefore, we consider to optimize it further to reduce the number of required actions.

The findings for required inefficient actions showed many redundant findings (35 of 105). However, the findings were helpful and we consider 98 of them as usability issues. The others are caused by the fact, that users vertically scrolled pages with a lot of information which are intended to be read or to display an overview. And vertical scrolling of pages while reading is not a usability issue. For our research website, we consider to restructure the pages so that the size of the top banner decreases on all pages but the start page. This would allow to move content of details pages to move upwards on the page. Furthermore, we consider to move the menu from the top of the page to the left and prevent its scrolling out of sight. Through this, we hope to reduce the number of required scrolling for navigating on the website and its details pages. For the application portal, especially the scrolling during the wizard usage was important. A wizard should break down a task into simple subtasks. The required scrolling indicates that still the subtasks are quite complex, require many website elements and, hence, much screen space. Therefore, we consider to check the wizard and the presented forms for their complexity. If required, we will further break down some subtask. Furthermore, scrolling after a file upload was important. This issue is caused by the fact that for a file upload, the current page reloads and starts at the top again. The users have to scroll the page to get to the next file to be uploaded. This issue was already tried to be addressed by adding a link on the top of the page allowing the automatic scrolling to the next file to be uploaded. But still, this is inefficient. Therefore, we propose to change the file uploads to make them possible without requiring scrolling.

The findings for high website element distances are mostly inline with the findings of other smells. We consider 11 of the 13 smells detected in both case studies as real usability issues. In the research website case study the findings showed that users have to navigate via several pages to reach details of a lecture or group member. Therefore, the website element distance is relatively high. Hence, these findings support, e.g., the findings for important tasks. The findings for the second case studies are not considered usability issues. Both findings represent required navigation between pages either for drill down in a large set of information (finding for the reviewer part) or for navigating in the wizard.

Our approach relies on a large number of recorded user actions. Otherwise, the task trees would not represent real user behavior. Therefore, the approach can only be applied on large data sets recorded over a long period of time. Based on the observations in the case studies, the validity of our results seems relatively high. We consider 147 of 171 detected usability smells as indicators for usability issues and, hence, as helpful findings. However, the validity should further be confirmed using other, established usability evaluation methods, e.g., usability tests with thinking aloud, in a comparative study. Furthermore, we should apply our approach in further case studies to determine if we obtain similar results. For example, we need to clarify if the minimum severity used for each smell is also adequate for smaller websites or websites with different structures.

Related Work

The term usability smells is similar to code smells introduced by Fowler [START_REF] Fowler | Refactoring: Improving the Design of Existing Code[END_REF]. Code smells are structures of source code being a hint for a required refactoring. Similarly, usability smells are usage patterns indicating required usability improvements.

Ivory and Hearst [START_REF] Ivory | The state of the art in automating usability evaluation of user interfaces[END_REF] introduced a categorization for usability evaluation methods with respect to automation. The categories are capture, analysis, and critique. Depending on the level of automation, an evaluation method is assigned to one or more of these categories. According to Ivory and Hearst, there are no methods yet covering all these categories at the same time. Our method spans all categories in that it records user actions (capture), transforms the recorded data into task trees representing actual user behavior (analysis), and automatically reasons on aspects of the websites usability (critique).

There are several methods that record users when utilizing a website and subsequently analyze the data to detect usability problems. Examples are Web-Hint [START_REF] Vargas | A method for remote and semiautomatic usability evaluation of web-based applications through users behavior analysis[END_REF], WebRemUsine [START_REF] Paganelli | Tools for remote usability evaluation of web applications through browser logs and task models[END_REF][START_REF] Patern | Remote evaluation of mobile applications[END_REF], and ReModEl [START_REF] Buchholz | Model-based usability evaluation -evaluation of tool support[END_REF]. With respect to Ivory and Hearst [START_REF] Ivory | The state of the art in automating usability evaluation of user interfaces[END_REF], these methods belong to the groups capture and analysis. The methods compare recorded user actions with expected user behavior. The expected user behavior is either a recording of actions of an example session (WebHint) or a manually defined task model (WebRemUsine, ReModEl). The methods only show differences between expected and recorded user behavior. Therefore, they do not belong to Ivory and Hearsts critique category. In comparison to these methods, our approach does not consider expected user behavior. Instead, our approach assesses only real user behavior. This has the additional benefit that no wrong expectations on user behavior influence our analysis. Furthermore, our approach provides details about the location and causes of potential usability issues and does, therefore, belong to Ivory and Hearsts critique category.

John et al. [START_REF] John | Predictive human performance modeling made easy[END_REF] propose an approach to record user actions and generate Goals, Operators, Methods, and Selection Rules (GOMS) models based on them. The GOMS models are used to estimate the average efficiency of users for executing a specific task. In comparison to our approach, this method focuses on efficiency measurement and is only used for example sessions performed by an evaluator. Sessions of real users are not considered.

An approach that belongs to Ivory and Hearsts critique category is the Mental Model workbench (MeMo) [START_REF] Jameson | Looking for unexpected consequences of interface design decisions: The memo workbench[END_REF] and its combination with the Multi-Access Service Platform (MASP) [START_REF] Feuerstack | Automated usability evaluation during model-based interactive system development[END_REF]. In this concept, a Graphical User Interface (GUI) is rendered and an analyzer tool, which simulates a user, evaluates which action is most likely to be chosen by the user in a specific state of the GUI. An extended and iterative version of this approach is proposed by Quade [START_REF] Quade | Towards model-based runtime evaluation and adaptation of user interfaces[END_REF] where the evaluation also performs GUI adaptations for usability improvement. The goal is to iteratively design a GUI guiding users as good as possible in the execution of specific tasks. In contrast to our approach, this technique works at design time to improve a GUI under development. It does not consider real user behavior but simulates it. Furthermore, it requires predefined user tasks and does not expect users to apply a GUI for unexpected tasks.

Conclusion

In this paper, we presented an automated approach for detecting usability smells of websites based on recorded user actions. The detected types of usability smells are missing feedback, important tasks, required inefficient actions, and high website element distance. The approach was applied in two case studies. For all smell types, we gained helpful results but also some room for improvement. For example, the detected smells included redundant findings and false positives. As the majority of the detected smells that we considered in our analysis also indicated usability issues, we conclude that it is possible to detect usability issues using our approach.

In our future work, we plan to improve the described usability smell detection, e.g. to reduce the number of redundant findings. In this context, we will also perform detailed studies with different detection parameters such as the minimal severity that must be reached by a smell to be reported. Furthermore, we will perform studies for specific smell detections with adjusted algorithms to check whether a modified algorithm provides better results. For example, we plan to implement the high website element distances detection without excluding inefficient actions and compare the results with our current findings. Finally, we plan to implement more usability smell detections, e.g. to detect unused but prominently positioned website elements which may also require an improved task tree generation to allow for more sophisticated findings.

Action 2 : 1 Action 4 :

 214 Enter user name into Textfield "username" Sequence Enter password into Textfield "password" Action 1: Click on Textfield "username" Action 5: Click on Button "login" Action 3: Click on Textfield "password"

Instance of Action 1 : 1 Instance of Action 1 :Fig. 2 .

 1112 Fig. 2. Examples of smelling tasks

Table 1 .

 1 Facts of the case studies

		Study 1	Case Study 2
		Website of Research Group	Application Portal
	Page types	18	35
			(21 application part,
			8 review part, 6 shared)
	Recording period	8.5 months	4.5 months
	Start	July 2013	October 2013
	End	March 2014	March 2014
	Recorded actions	33,690	241,306
	Generated tasks	1,847	10,634
	Sequences	1,431	9,530
	Iterations	416	1,104

Table 2 .

 2 Smells detected in the case studies

		Case Study 1		Case Study 2
	Website of Research Group		Application Portal
	all redundant true positive all redundant true positive
	Detected smells 1,464	275		4,436	985
	Missing feedback 52	0		39	0
	Important task 19	10		13	4
	Inefficient actions 959	199		3,772	878
	High distance 434	66		612	103
	Considered smells 96	30	83	75	22	64
	Missing feedback 12	0	11	9	0	6
	Important task 19	10	10	13	4	11
	Inefficient actions 54	17	51	51	18	47
	High distance 11	3	11	2	0	0


Ana De Barros
email: ana.barros@fraunhofer.pt

Sílvia Rêgo
email: silvia.rego@fraunhofer.pt

João Antunes
email: joao.antunes@fraunhofer.pt

Ana Correia De Barros

Aspects of Human-Centred Design in HCI with Older Adults: Experiences from the Field

Keywords: Human-centred design, older adults, interviews, recommendations, qualitative methods, user recruitment, user group

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The World population is getting older. The Eurostat reports an inversion of the demographic pyramid and sharp ageing of the population in Europe until 2060 [START_REF]Directorate-General for Economic and Financial Affairs of the European Commission: The 2012 Ageing Report -Economic and Budgetary Projections for the 27 EU Member States[END_REF]. With ageing, several impairments (e.g. sensorial, motor, and psychological) are likely to come about which, in turn, may have effects on people's activities of daily living, independence, or quality of life. Also rising is the array of Information and Communication Technologies (ICT), which are becoming increasingly sophisticated and hold the promise of bringing health care into older adults' homes [START_REF] Mykityshyn | Learning to Use a Home Medical Device: Me-diating Age-Related Differences with Training[END_REF].

However, solutions designed to be used by this age group must take into account natural ageing decline and the implications it may have on the interaction with ICT. On the other hand, a thorough knowledge of people's needs, expectations and attitudes towards ICT is needed so that solutions being designed indeed meet end-users' needs and are regarded by the latter as appealing.

In order to achieve this purpose, many authors agree, users must be invited to be active participants in the design of these technologies [START_REF] Rogers | Training Older Adults to Use Automatic Teller Machines[END_REF]. Designers can only go so far in trying to put themselves into users' shoes and the more distant the experiences of end-users and those of designers, the more difficult the challenges become [START_REF] Nicolle | USERfit -Design for All Methods and Tools[END_REF]. In the European Ambient Assisted Living Joint Programme, for instance, the involvement of end-users through end-user organisations is indeed mandatory 1 .

While there are reports on how to conduct research activities or on how to set up test sessions with older adults, literature is scarce on the latter and scarcer on what is needed to create and maintain a user group for the purpose of human-centred design (HCD) activities that need to involve older adults, caregivers and other stakeholders. Nevertheless, there are research groups who have engaged in these activities and could share their experience with the community towards promoting the practice of HCD.

This paper i) gathers information from different sources regarding this matter, ii) reports experience from the field in building a user network of older adults, and iii) illustrates some points through statements provided by our centre's researchers about their experience and challenges in conducting HCD with older adults. The experiences from the field come from a user network of older adults and caregivers that we have created and maintained at our research centre since 2011-network Colaborar2 . The network has been growing over time, having at one point reached 400 users, and has supported over 1000 HCD activities required by the manifold research projects at the centre, which is related to ICT solutions towards independent living in old age. Researchers' statements used along the paper were gathered during semi-structured interviews, the major outcomes of which are reported elsewhere [START_REF] Correia De Barros | Young practitioners' challenges, experience and strategies in usability testing with older adults. Assistive technology: From research to practice -AAATE 2013[END_REF].

Previous work

Conducting research with and about older adults is challenging in different fields, namely in clinical research. Mody and co-workers [START_REF] Mody | Recruitment and Retention of Older Adults in Aging Research[END_REF] share methods for the effective recruitment and retention of older adults for research purposes. The methods target four main goals: attaining a representative sample, promoting participation of older adults and other stakeholders in the research, assess feasibility in advance, and retain the participants in the particular study.

Working with older adults in Human-Computer Interaction (HCI) requires specific care not only regarding design, but also regarding the process itself, and how one must prepare sessions, treat participants or deal with idiosyncrasies of the age group [START_REF] Rubin | Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests[END_REF]. Rubin and Chisnell [START_REF] Rubin | Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests[END_REF] have laid out recommendations to address these issues.

Lindsay and co-workers [START_REF] Lindsay | Engaging Older People Using Participatory Design[END_REF] have documented their experience, findings and methods extensively after using participatory design with older adults and people with specific impairments (e.g. dementia). They have developed their own method on top of 4 main challenges they identified from previous work, that were related to focus and meeting structure, representation and taking action, ability to envision future concepts, and design beyond tasks (p. 1201).

Gregor and Newell [START_REF] Gregor | Designing for Dynamic Diversity -Interfaces for Older People[END_REF] have made the case for 'User sensitive inclusive design', an approach to allow and encourage designers to accommodate diversity within the group of 'older adults', which, the authors argue, is far from homogeneous. The au-thors claim that the approach allows overcoming the drawbacks found in traditional user-centred design (UCD) techniques, which are meant for more homogeneous groups.

Researchers from the Universities of Dundee, Abertay Dundee and Glasgow, amongst whom were A. F. Newell and Peter Gregor cited above, have concluded that traditional UCD methods used in HCI require some adaptations in order to be effectively used when working with and for older adults [START_REF] Eisma | Early User Involvement in the Development of Information Technology-Related Products for Older People[END_REF]. Through the UTOPIA project, the researchers have created a database of 160 individuals for research purposes and have outlined the methods used to efficiently elicit requirements about new technologies from older adults. The methods begin with the work towards the creation of a partnership with end-users and the maintenance of it throughout the research and design process.

Bringing end-users into the design team, a principle of participatory design, has long been praised within the Inclusive Design community. Julia Cassim, through her 'Design Challenges' [START_REF] Cassim | Designing Effective User Interactions -Examples from the Challenge Workshops[END_REF], brings people with disabilities into design teams as design partners. Newell, Gregor, and Alm [START_REF] Newell | HCI for Older and Disabled People in the Queen Mother Research Centre at Dundee University, Scotland[END_REF] have brought together not only older adults and designers, but have also fostered get-togethers of older adults where learning of new technologies happens on a peer-to-peer basis, and have reported the benefits of this approach in regards to older adults' engagement in the design activities.

There are different challenges related to participatory design [START_REF] Vines | Configuring Participation: On How We Involve People in Design[END_REF] and different challenges related, in particular, to research and design with and for older adults [START_REF] Vines | Questionable Concepts: Critique as a Resource for Designing with Eighty Somethings[END_REF]. Our previous work has listed an array of difficulties that 6 researchers (R1-R6) experienced when setting up and conducting usability tests with older adults [START_REF] Correia De Barros | Young practitioners' challenges, experience and strategies in usability testing with older adults. Assistive technology: From research to practice -AAATE 2013[END_REF]. These at times support, and at other times add points to existing literature on the creation and upkeep of an older adult user group. The examples outlined above seem to suggest that retention of older adults in a group is an activity that is built not only outside design/research activities, but indeed within the activities themselves. One such example is the theatre promoted by Newell [START_REF] Newell | Design and the Digital Divide: Insights from 40 Years in Computer Support for Older and Disabled People[END_REF].

Creating and maintaining a user network

The existing literature on creating and maintaining groups of older adult users usually reports efforts for a single design or research project. We will go over existing recommendations and add our own experiences from the field.

Recruiting older adults

The challenge begins with finding a large enough number of older adults to take part in the activities. Some authors suggest resorting to nursing homes, day-care centres, and clubs, amongst others. We have found that these are all viable options, although, as Mody and co-workers [START_REF] Mody | Recruitment and Retention of Older Adults in Aging Research[END_REF] point out, reaching out to older adults in nursing homes may pose severe challenges. To build a user network encompassing diversity, we should target as many different institutions as we can. It often happens that if we remain with a single type of institution, we will have a narrow age range.

Furthermore, when we study older adults, it is often the case that we are looking for specific chronic diseases or impairments. Our previous work has suggested that this is an important aspect for researchers [START_REF] Correia De Barros | Young practitioners' challenges, experience and strategies in usability testing with older adults. Assistive technology: From research to practice -AAATE 2013[END_REF]. The user network should be wide and diverse enough to secure participants with these specific characteristics.

In addition, researchers mentioned that in cases when they rely on information from staff at elder institutions where they go to conduct tests, or on the best of their own judgment, they might not get an adequate assessment of participants' cognitive abilities. Four participants [START_REF] Correia De Barros | Young practitioners' challenges, experience and strategies in usability testing with older adults. Assistive technology: From research to practice -AAATE 2013[END_REF] shared stories in which they realized that users were not able to conduct the tests already while the test was underway and at a time when there was no chance for them to turn back. In this context, reliable information regarding participants' particular abilities for the project at hand seems to be of paramount importance to researchers, as it helps avoiding stressful situations for both the practitioner and the older adult participant.

In order to set the terrain for future work, a previous screening should be made. This can be done by talking to the staff in the institutions and getting an idea of the type of group, and through a more formal screening using questionnaires for older adults to fill in with socio-demographic data, general health information or others that cover the specificities required for the tests or activities themselves. From our experience, it is also useful to take time to talk to each potential participant individually prior to conducting any screening, or even planning any activity. This helps us to understand the users, helps to create a relationship of trust and serves as an opportunity to explain to each potential participant what the user network is about.

During the initial contact with the institution it usually helps to mention other institutions in the area that are already part of the network, as it often happens that institutions know one another. In accordance with Eisma and co-workers' experience [START_REF] Eisma | Early User Involvement in the Development of Information Technology-Related Products for Older People[END_REF], we have also found that after the initial contact, and after formalizing the collaboration between the team and the end-user organization (i.e. a signed protocol), the first activity should take place within a short timeframe. Organizations of older adults are very much sought after. Therefore, it is advisable to maintain the contact, so that people do not suspect of researchers' intentions, nor forget about them. These activities may be design or research activities, but may very well be just a leisure activity with older adults.

Maintaining relationships with the group

It must be noted that, for the purpose of a user network that serves many different projects, a large number of older adults is required not only because of the sheer amount of design/research activities of each project, but also because practitioners should be careful not to convert them into a burden for the users in the group. Nevertheless, as going to meet the users is a recurrent activity in HCD, the designers/researchers come to learn through their experience about the groups, their idiosyncrasies, and routines. In day-care centres with which we work, older adults tend to strictly follow their routines and timetables, in particular regarding eating hours, as noted by one of our researchers. Learning about these may be useful for the design and research work [START_REF] Correia De Barros | Young practitioners' challenges, experience and strategies in usability testing with older adults. Assistive technology: From research to practice -AAATE 2013[END_REF], e.g.:

If you try to apply the questionnaires closer to tea time they talk much less, because they know it is tea time and must get on with it. (R2) Rubin and Chisnell [START_REF] Rubin | Handbook of Usability Testing: How to Plan, Design and Conduct Effective Tests[END_REF] alert to the fact that, when welcoming older adults to one's facilities, arrangements should be made for a second person accompanying the first. Our experience confirms this need. As visiting the research centre facilities is a new experience that many users in the network look forward to, we should try to give all an opportunity to come over for research purposes. If this is not possible, we can invite older adults over just to get to know the facilities and the team.

Following the same line of thought, the person in charge of the user network should make sure to keep a record of activities being carried out and the users involved, so as to spread the activities between the available older adults as much as possible. The person should manage the amount and type of activities each user takes part in and make sure that participating in the activities is not a burden for the participants. Users in our network often take notice and complain that they have been less called upon for activities than others in the same group.

The people in charge of managing the user network should also make sure to share the outcomes of the design and research with the users. This helps people understand their contribution and helps to maintain the relationship.

Throughout time we gather much information in video or photographs. These materials are useful to publicize the network and recruit new users. However, it is crucial that we talk individually to each person who can be identified in these materials, so as to gather their authorization to make these materials public in any way. This authorization is signed apart from the particular informed consent used by the research project.

As a result of keeping a strong relationship with older adults and staff at the institutions, introducing the test sessions or other kinds of design/research activities becomes easier to practitioners.

Training researchers

However carefully the relationships with the institutions and the users in the network might be managed, two fundamental points are i) to support, prepare and train researchers to address older adult users in the network and ii) to periodically gather their insights about their contacts with older adults [START_REF] Correia De Barros | Young practitioners' challenges, experience and strategies in usability testing with older adults. Assistive technology: From research to practice -AAATE 2013[END_REF]. The excerpts illustrating some of the points that we will be making in the following two sections were gathered during qualitative research with researchers, reported elsewhere [START_REF] Correia De Barros | Young practitioners' challenges, experience and strategies in usability testing with older adults. Assistive technology: From research to practice -AAATE 2013[END_REF].

Support, preparation and training

Contrary to what one might think, the age gap may lead to difficulties in communication between two generations. Our young researchers have reported that, on their first contacts with older adults in their research, not only did they have to pay special attention to language they used so that complex concepts could be communicated in laymen's terms, but they also actually reported being a bit fearful, on the first encounters, of talking to older adults they did not know because they were not used to talking to any older adults beyond their grandparents [START_REF] Correia De Barros | Young practitioners' challenges, experience and strategies in usability testing with older adults. Assistive technology: From research to practice -AAATE 2013[END_REF]. This is in line with the "cultural and experimental gap between researchers and older people" mentioned by Eisma and coworkers [START_REF] Eisma | Early User Involvement in the Development of Information Technology-Related Products for Older People[END_REF].

If for nothing else, working with and for older adults is a difficult line of work because of the emotional impact it can have on researchers. If contacts are frequent, researchers and users develop a relationship and older adults' personal problems, illnesses or death are likely to affect researchers. New team members need to be alerted and informed about this possibility and prepared to deal with these situations.

The case we are sharing in this paper is of a user network that needs to support multiple research projects. This means that, rather than dealing with the same group of researchers all the time, older adults in the network come to deal with a variety of researchers and a variety of research projects. In order to, on the one hand, support researchers in meeting the users for the first time and, on the other hand, to put users at ease, one personthe one in charge of managing the network, doing the recruitment, and so forthwill accompany researchers most of the times and introduce the activities. This person gives a face to the research centre and always bridges researchers and users. This person is also responsible for helping researchers plan ahead the time and resources they will need to conduct the various activities within their projects. Beyond that, this person is in charge of planning leisure activities (e.g. Christmas parties) and answering requests from the users or institutions in the network (e.g. demonstration of a particular project for an event). Other activities supporting the network are a website of its own, periodical newsletters or periodical dissemination of the network in specific fairs and events.

Researchers' insights

Some of the users' reactions during user research or usability testing are completely unexpected to someone new to the field, but are part of everyday life for those working with the specific user group for a longer time. We have come to observe in time that there are some behaviour patterns within the network.

The experience (and tips) from more experienced researchers is useful to new researchers not only because it helps them plan ahead for some situations, but also deal in the moment with others. This, in turn will help make the best out of the research.

Researchers mentioned several examples of older adults' behaviour during the tests in regard to their own performance, their beliefs about themselves and their capabilities. Researchers commented that older adults in the network often felt they should try to help researchers out and even try to please or try not to look bad in front of researchers-a phenomenon which is documented in the literature concerning not only older adults, but participants in research as a whole [START_REF] Bertrand | Do People Mean What They Say? Implications for Subjective Survey Data[END_REF].

Researchers furthermore comment that older adults they worked with tended to focus on the results of their performance, wanted to do well, and even sometimes regarded usability testing as a competition amongst test participants. In the following excerpt about the process for diary studies regarding games with older adults, the researcher illustrates this aspect of competition as well as the behaviour of older adults when trying to please the researchers: I went to the same [day-care] centre twice a week for a long time. (…) so I handed it [diary] over on a Tuesday, and when I got there on Thursday they would look at me, take out the little books and start filling them in, like "I didn't forget it, it's here, I'm doing it". (…) Sometimes they'd ask me if I could help them here and there (…) (R3) In the same testimony, the researcher continues with an illustration of a behaviour associated with older adults' performance, which another researcher also noticed to be common-that older adults in the network often asked for approval or for help during the tests:

But that was the other thing, they always assumed that I would be evaluating their results: if they'd done well, if they'd done poorly, and it's hard for one to explain that there's no good or bad, and that we just wanted to know which things they liked best. And they insist a lot on that "But did I do well? But did I do badly? How did I do? But is that right, but is that wrong?" (R3) Nevertheless, there might often be unexpected behaviours, such as the one described by researcher R3 during a diary study:

Eventually I stopped asking [for the diaries] because they said they had forgotten, didn't know where the books were anymore. One lady gave it to her grandson to do because she thought it was more fun for him. (R3) In order to face these challenges, all interviewed researchers referred to the importance of having a user network at their service and, more specifically, a person who knows the people in the network and who helps them manage all that is related to the involvement of older adults in HCD.

Discussion and conclusions

The experiences from the field reported in this paper refer to a particular set of over 350 users within a particular socio-cultural setting and might not be applicable to other settings. Some of the lessons learned are, however, aligned with existing reports in the literature.

The paper provides a brief overview of literature reports on the use of HCD with older adults and on the issue of creating and maintaining a user group. It complements existing knowledge by adding the authors' experience in the field and researchers' accounts of their experience with conducting design and research activities with older adults. The contributions of the paper are particularly novel in that they share lessons learned, methods and strategies to recruit and maintain an older adult user network to serve a wide array of projects. In our case the user network is treated as a project in itself, with its own management. One main conclusion is that the project should have, at least, a dedicated person to handle the network and preferably, as is our case, it helps that user network is a project in itself with its own human resources, tasks and budget. This has proved to be helpful for the researchers.

http://www.aal-europe.eu/get-involved/who-can-apply/

www.colaborar.fraunhofer.pt

Acknowledgments.

The authors would like to express their gratitude to all the institutions, staff and users in the network (www.colaborar.fraunhofer.pt).


Kati Kuusinen
email: kati.kuusinen@tut.fi

Improving UX Work in Scrum Development: A Three-Year Follow-Up Study in a Company

Keywords:

This paper presents a three-year follow-up study considering the improvement process of user experience work in a software company utilizing Scrum. Problems encountered in the organization included managing the product vision, timing of UX, lack of cooperation among disciplines, and understanding user needs. We also observed changes in the organization over two years. They included ceasing the centralized UX team and dividing the UX specialists over business lines. UX specialists were given influential roles in regard to product decisions -such as nominated as product owners.

INTRODUCTION

Agile methods [START_REF] Highsmith | Agile software development: The business of innovation[END_REF] are commonly utilized in industrial software development. However, guidance on those methods still lacks user experience (UX) [4] activities. Current recommendations for agile UX development suggest dividing the work into activities that are conducted within agile development and activities that are prior to development Sprints as design upfront work [START_REF] Da Silva | User-centered design and Agile methods: a systematic review[END_REF]. They also typically separate UX related work to its own stream to be conducted by separate UX specialists. However, organizations using such practices struggle with issues such as balancing the amount of design upfront work, and communication between developers and UX specialists [START_REF] Da Silva | User-centered design and Agile methods: a systematic review[END_REF]. Separating the UX designer to another stream seems to keep the UX designer out of the core team, which hinders within-project communication and endangers the realization of UX design as the product vision gets blurred [START_REF] Hodgetts | Experiences integrating sophisticated UX design into agile process[END_REF].

In this paper we report a three-year follow-up study where a company advanced in Scrum 1 and with an established UX team aimed at a better integration of those disciplines. We started the study in 2011 by a current state analysis to explore the situation of that time and to enable and focus improvements. We conducted an international web survey with 31 open-and 19 closed-ended questions, followed by 17 theme interviews [START_REF] Kuusinen | Agile user experience development in a large software organization: good expertise but limited impact[END_REF]. Moreover, during 2012 and 2013, the organization adopted new ways of working and we measured their impact with surveys and interviews.

RESEARCH PROCESS

The studied company was mainly producing specialized software that was supplied via internet service providers (ISP) for consumers. The main product of the company was a software system with massive yearly releases. Large multinational ISPs were dominant when deciding of the feature content for the next release. The company had about 800 employees mainly in Finland, France, and Malaysia. The company was utilizing their own Scrum-based process model in their development. During 2011 the company had a centralized UX team with about 15 members and a few distributed UX specialists. In the beginning of 2013 the UX team as such was ceased and the UX specialists were distributed through the new business lines.

Study conducted in year 2011. We started the study in 2011 with a current state analysis to determine where and how to change working practices in order to improve the UX work in the company. Of the company employees working in development related roles from Finland, France, Malaysia and Russia, 76 responded to a web survey. Participant roles included developers, managers, architects, product owners, scrum masters, UX specialists, and quality engineers. The survey consisted of 50 questions (31 open-and 19 closed-ended) on processes and tools, collaboration and communication, and concepts and knowledge in the company [START_REF] Kuusinen | Agile user experience development in a large software organization: good expertise but limited impact[END_REF]. Moreover, we conducted 17 interviews on UX specialists, product line managers, product owners, scrum masters, architects, developers, and quality engineers on the Helsinki site.

Monitoring the organization performance in year 2012. The cooperation between the researcher and the organization was smaller during 2012. The researcher mainly observed a new practice in the organization and discussed and gave some guidance to a few members of the organization as they implemented changes in the organization.

Study conducted in year 2013. We started the second current state analysis with a preliminary mapping consisting of five interviews in spring 2013. The interview data was enriched with unofficial "corridor discussions" and some email discussions. The aim of the preliminary mapping was both to gain a high-level understanding of the current state in the agile UX work and to create a preliminary list of possible changes to enable further ideation and discussion. We conducted the actual round of measurements inside one business line in October 2013 by surveying and interviewing six persons working for the business line at the Helsinki site of the company. First, all the participants filled in a short web survey individually (Table 1) and later we interviewed them in pairs. The participants represented the same business line from where the majority of the participants of the 2011 study were. We selected participants using theoretical sampling; with an approach that does not aim at representative sample but a sample that offers to compare to previous findings [START_REF] Eisenhardt | Building theories from case study research[END_REF].

Change (question 6): as the business lines in the company had been rearranged recently, we asked respondents to compare the current situation within the business line to the situation that prevailed a year ago in the organization the respondent were then working for. We use the abbreviations of the studied areas presented in Table 2 throughout this paper.

Table 1. Survey questions, their types and used scales 1. The overall satisfaction of UX related work in the business line Measured on a scale from 1 (not at all satisfied) to 7 (completely satisfied) 2. Issues in UX related work that the respondent is currently most dissatisfied with Open-ended question, listing of 0-3 issues 3. Issues in UX related work that the respondent is currently most satisfied with Open-ended question, listing of 0-3 issues 4. Means to improve the current situation Open-ended question, text area 5. Level of performance on certain areas of UX related work and their importance to project success. The studied areas (11 items) are listed in Table 2. Fourfold table : x-axis: performance from poor to excellent, y-axis: importance from insignificant to significant. 6. The change within last year in the level of performance on the studied areas (areas are listed in Table 2) Change was evaluated on the following scale: worsened greatly, worsened somewhat, worsened slightly, no change, improved slightly, improved somewhat, improved greatly

RESULTS

Next, we present the results from our study in two parts. First, we briefly revisit findings from the study conducted during 2011 already reported in [START_REF] Kuusinen | Agile user experience development in a large software organization: good expertise but limited impact[END_REF]. Then, we concentrate on the results of the study conducted in 2013, and address the changes that can be observed.

Year 2011 -Analyzing the problems in agile UX work

The company had had some issues in integrating UX work into other agile development practices. They wanted to clarify their understanding of the problems in order to improve the situation. In a web survey answered by 76 R&D related employees and with 17 interviewees we determined the following: 1) Which issues are considered as the biggest challenges in the agile UX work; 2) Which tasks are considered as the most important duties of the UX team; and 3) How the employees would improve the agile UX work and communication between different R&D roles.

The top three challenges in agile UX work included managing the big picture of the project, changing the UX team's way of working from traditional waterfall practices to agile practices, and proper timing of UX work in agile process. Other major challenges were lack of cooperation between different roles, and being able to understand user needs and fulfilling them by the software (Figure 1). Study participants emphasized the most that they would like to have closer collaboration between UX specialists and other development roles. The most often mentioned manner of improving the cooperation was to include UX specialists in Scrum teams instead of having them work in a separate UX team.

Year 2012 -Implementing new practices in R&D

The company implemented a method called "UX review board" to raise the visibility and understanding of UX related problems of products [START_REF] Kuusinen | Agile user experience development in a large software organization: good expertise but limited impact[END_REF]. The aim was to get UX issues into backlog items to ensure and fasten the process of actually fixing those.

The practice improved the visibility and understanding of UX issues. However, it did not work as plannedin many cases the product owner agreed the found issues are severe but still did not welcome them in the backlog as the PO felt the backlog was already overloaded with other issues. Singh [START_REF] Singh | An Agile Methodology for Promoting Usability[END_REF] reported similar problems; priority of UX tasks remained low and thus they rarely got implemented.

The company adopted a new role of UX engineer to the R&D organization. The role was to work inside Scrum teams as a link between UX designers and developers. UX engineers were people with both coding and UX skills. First, the company had difficulties in recruiting such multi-talents. Later, they noticed they did not manage to maintain the work contents of the role as planned; UX engineers were observed to mainly concentrate on implementing the UI. It became evident that it required changes on the organization level to improve the impact of the agile UX work; the organization structure seemed to hinder from improving the effectiveness of the UX work.

Year 2013 -Changing the organization structure

There were plans to make noticeable changes on the organization level as the previous attempts had not had a desired effect. We started a second round of current state analysis in the beginning of 2013. We used a more lightweight approach than previously to save time and costs. We started by preliminary theme interviews, continued with an expert evaluation and group discussion, and finally conducted a web survey and structured pair interviews. Next, we present the results.

Background mapping. The idea of preliminary interviews and discussions was to gain understanding on how employees would like to change the current state and which issues they consider the major challenges in agile UX work at the moment. A researcher interviewed five persons from the organization and based on the findings formed a suggestion on how to improve the current state of agile UX work. The main items of the suggestion were the following:

 "Away from silos"; UX needs to be considered in every phase from roadmap to delivery and it must be everybody's concern, and team performance metrics should include a UX-related instrument.  Mutual project goals, support goal-setting; developers still need more understanding of UX specialists' work contents and goals.  Spread UX mindset in the organization; continuing the "cultural change"; increasing the understanding of the importance and meaning of UX work.  Prototype and utilize multi-phase testing on actual users.  Emphasize early cooperation and iterative way of working. Expert evaluation: Comparing the situations of years 2011 and 2013. During summer 2013, an experienced UX specialist walked through the report of the study conducted in 2011 and commented on how they thought the situation had changed since. Next, we list the found issues and the comments of the UX specialist. Found issues are numbered and comments of the UX specialist are started with "UXS:" and printed in italics.

Measurement of the situation after the organizational level change

The organization was rearranged and the centralized UX team as such was ceased in fall 2013. UX specialists were divided to different product lines, and they were given more influential positions in the organization.

Issues in UX related work that respondents were most satisfied with. We asked the participants to list issues that they are most satisfied with in the current ways of working. In general, participants reported that the organization has positive and active attitude; work gets done, tasks are novel and interesting, and new technologies and tools are adopted. Both developers' and UX specialists' skills in UX and attitude towards it were appreciated. UX specialists' cooperation with developers, product management, and marketing were all mentioned. Also, user participation both in planning and feedback gathering was considered good.

Nominating an experienced UX specialist as a product owner had given the person more influence on the product vision. However, it was criticized that it was more of a career decision than actual change on the process level: if the person e.g. chooses to leave the company, it is not a company policy to nominate another UX aware person to replace them.

Issues in UX related work that respondents were most dissatisfied with, and means to improve the current situation. Diversity and poor quality of used tools caused dissatisfaction. Both own and several third party UI frameworks were in use which was reported to cause extra work. In addition, delivering several clients instead of building one product with several touchpoints was criticized. Also, it was mentioned that the current development tool did not allow creating high-quality UX.

There was still a small number of UX specialists in the company compared to the amount of UX work. UX specialists were overwhelmed with work and that led to implementing new features without UX design, or with "very basic design" that was mentioned to be often late. Also, both an architect and a UX specialist reported that the visibility of the contents of concept and design work and timing of those tasks should be improved: "…the implementation request often comes as a surprise, or I find out after the implementation has begun"an architect.

Level of performance on studied areas and their importance to project success. We asked the participants to evaluate both the importance of the studied areas to project success (from unimportant to important) and the performance of their organization in those areas (from poor to excellent) on a fourfold table (Figure 2). The most important areas for project success were maintaining the big picture of the project (76.7), understanding user needs (75.0), project team competence (74.5), and proper timing of UX work (71.7). The organization performed best on maintaining the big picture (56.2), project team competence (55.2) and getting user feedback (53.3). There were largest gaps between importance and performance in the following: timing of UX work (29.0), understanding user needs (26.8) and cooperation between UX specialists and product owners (26.0). Participants agreed the most on their level of performance in big picture management (SD=9.6), and welcoming late change (SD=10.3). Largest values of standard deviation were measured for timing of UX work (27.0) and cooperation between developers and UX specialists (25.3). Both the lowest and highest assessments of performance were given by a developer and an architect. It probably indicates that there are project or individual-specific ways of working inside the business line.

SUMMARY AND CONCLUSIONS

This paper presents a three-year follow-up study considering the improvement process of agile user experience work in a software company. In the initial state the organization was following Scrum methodology in software development and it had an established UX team. However, the organization had problems with integrating the work of these two disciplines. The major problems included managing the big picture of the project (product vision), timing of UX work, lack of cooperation between disciplines, and understanding and fulfilling user needs. We conducted major changes in the organization over two years. Changes included ceasing the centralized UX team as such and dividing the UX specialists over different business lines. UX specialists were given more influential roles in regards to product level decisions. These practices were reported to improve the big picture management and understanding user needs. However, there were significant differences in evaluations between individuals in timing of UX work, cooperation between UX specialists and other disciplines, and the agility of UX work. This indicates that the practices still are strongly person-dependent and there is need for more established approaches.

Figure 1 .

 1 Figure 1. Biggest challenges in agile UX work by respondent role (results 2011). Horizontal axis presents the amount of respondents. Legend: QA = quality assurance, PO/SM = product owner/ scrum master, UX = UX specialist, Mgmt = manager, Dev = developer. Total number of respondents was 60; each named 1-3 issues.

Figure 2 .

 2 Figure 2. Averages of performance and importance (middle and rightmost columns) on certain areas (letters T-W) and their standard deviations arranged by the difference between the importance and performance (leftmost bars without standard deviation lines). T = Timing of UX work, U = Understanding user needs, P = PO-UXS cooperation, A = Agility of UX work, D = Developer-UXS cooperation, Q = Quality of UX implementation, B = Big picture maintaining, M = Meeting user needs, C = Competence in project team, F = Getting user feedback, and W = Welcoming late change.

Table 2 . Studied areas of agile UX work. Level of performance and importance for project success and experienced change were measured for these areas (see items 5 and 6 in Table 1).

 2

	Abbreviation	Studied area
	A	Agility of UX work
	T	Timing of UX work
	W	Welcoming late change
	C	Competence in the project team
	B	Maintaining the big picture of the project
	U	Understanding user needs
	M	Meeting user needs
	F	Getting user feedback
	P	Cooperation between product owners (PO) and UX specialists (UXS)
	D	Cooperation between developers and UX specialists
	Q	UX implementation quality

All the UX team's work I see is done in close collaboration with developers

  1. UX work is not really integrated with R&D ways of working. (2011) -UXS: "To the best of my knowledge, this is not true anymore. ." (2013) 2. It seems that the importance of UX work is generally acknowledged but there is a wish that it should not cost or take time. (2011) -UXS: "This, I think has improved … projects try find special

sprints to clean up UI … or there really is no work on UI that is not involving the UX

	team." (2013)
	3. UX issues are not ordered together with other issues in backlog. Lack of
	considering UX when creating the minimum viable product.
	-UXS: "Both issues are (slowly) improving." (2013)
	4. Inadequate communication between architects and the UX team. (2011)
	-UXS: "

This has improved greatly. Architecture is one of the main issues affecting the UX as experienced by consumers

  ." (based on experiences in development of two separate products) (2013) 5. UX work is often conducted in reactive manner. The UX team needs to hurry to get ahead of implementation, but they report often failing in doing so. (2011)

-UXS: "In my experience UX team(s) are quite busy. … In the projects I have been involved in the design has not been reactive. … The work is in any case creative collaboration -so agile." (2013) 6. UX work is not properly integrated with the R&D work. (2011) -UXS: "To some extent I do feel UX will always be a little outside of development; it is a different mindset with different educational background and as a discipline it is UX team's role to challenge." (2013)

Schwaber, K. Agile project management with Scrum, 1 st ed., Microsoft Press.