Jane Billestrup

Anders Bruun
email: bruun@cs.aau.dk

Jan Stage
email: jans@cs.aau.dk

Usability Problems Experienced by Different Groups of Skilled Internet Users: Gender, Age, and Background

Keywords: Usability Evaluation, Test Persons, Demography

Finding the right test persons to represent the target user group, when conducting a usability evaluation is considered essential by the HCI research community. This paper explores data from a usability evaluation with 41 participants with high IT skills, to examine if age, gender, and job function or educational background, has an impact on the amount and types of usability problems experienced by the users. All usability problems were analysed and categorised through closed coding, to group the test persons differently in relation to gender, age, and job function or educational background. The study found that the usability problems experienced across gender, age group and job function or educational background, are approximately the same. This indicates that the usual characteristics of test persons, might not be as important, and opens up for further research in regards to, if users with different skill levels, in regards to internet usage, might be more applicable.

Introduction

Usability evaluation is a strong tool for identifying areas of an interactive system that need improvement. In practice, one of the key challenges for usability evaluators is to find users that can participate as tests subjects. Recruitment of test subjects is challenging, and the time required for test sessions and the subsequent data analysis is usually dependent on the number of the number of test subjects. Therefore, there have been attempts to determine the minimal number of test users required for a usability evaluation [START_REF] Hwang | Number of people required for usability evaluation: The 10±2 rule[END_REF], [START_REF] Lewis | Sample sizes for usability studies: Additional considerations[END_REF], [START_REF] Virzi | Refining the test phase of usability evaluation: How many subjects is enough?[END_REF].

Combining Other researchers have criticised these attempts to define the minimal number. One of the arguments is that different users experience different usability problems [START_REF] Law | Analysis of combinatorial user effect in international usability test[END_REF], [START_REF] Schmettow | Sample Size in Usability Studies[END_REF]. In these discussions, there has been little evidence as to the actual differences between the usability problems experienced by different groups of users.

For specialised systems that are used by a homogeneous group of users, this issue is not particularly relevant. However, for systems that are aimed at diverse and heterogeneous groups of users, it is highly relevant. This paper presents results from an exploratory study of the usability problems experienced by different users. The focus of this study was to what extent different test persons, who are all experienced internet users, experience different types of usability problems, across gender, age, and educational background or job function.

The system we evaluated was a government data dissemination website aimed at a very broad user population. In the following section, the related work is presented, followed by a description of the method used for data collection and analysis. Then the results are presented, and finally, the results are discussed and concluded upon.

Related Work

The question about the number of test subjects needed in a usability evaluation has been discussed for many years. Virzi [START_REF] Virzi | Refining the test phase of usability evaluation: How many subjects is enough?[END_REF] focused on the need exists to reduce the cost of applying good design practices, such as user testing, to the development of user interfaces. He was one of the first to experiment with the number of test subjects needed. Over a series of 3 experiments, he found that 80% of the usability problems were detected with four or five subjects, additional subjects were less and less likely to reveal new information, and the most severe usability problems were likely to be detected with the first few subjects. In the experiments, he used test subjects who were from the surrounding community or undergraduate students. There is no further description of their demography. Lewis [START_REF] Lewis | Sample sizes for usability studies: Additional considerations[END_REF] emphasices that the aim of a usability evaluation is to have representative participants. He reports from an experiment with fifteen employees of a temporary help agency who all had at least three months' experience with a computer system but had no programming training or experience. Five were clerks or secretaries and ten were business professionals. In this study, using five participants uncovered only 55% of the problems. To uncover 80% of the problems would require 10 participants. The results show that additional participants discover fewer and fewer problems. The most important result was that problem discovery rates were the same regardless of the problem severity. Again, there is no concern for the demography of the test subjects.

Caulton [START_REF] Caulton | Relaxing the homogeneity assumption in usability testing[END_REF] argues that the results obtained in these early experiments were based on the assumption that all types of users have the same probability of encountering all usability problems, and he denotes this as the homogeneity assumption. If that is violated, more subjects are needed. He argues that with heterogeneous user groups, problem detection with a given number of subjects is reduced. The more subgroups, the lower the proportion of problems expected. If ten unknown user subgroups exist, 50 randomly sampled subjects should yield 80% of the problems.

Law and Hvannberg [START_REF] Law | Analysis of combinatorial user effect in international usability test[END_REF] have worked more on the influence of subgroups on problem detection through an experiment with usability tests conducted in four different European countries. They conclude that the heterogeneity of subgroups in a test will dilute the problem detection rate. Not only for severe problems but also for moderate and minor ones, the diluting effect implied a reduction. The problem detection rate for the severe problems is significantly higher than for the less severe, but the absolute value for the severe problems is not particularly high. Between nine and ten participants were required to uncover 80% of the severe problems, whereas 15 participants were required to uncover 80% of the minor problems. In addition, they found no significant correlation between problem detection rate and problem severity level. Based on their results, they reject that so-called "magic five" assumption as 11 participants were required to obtain 80% of the usability problems.

More recently, there has been another attempt to define a specific "magic" number [START_REF] Hwang | Number of people required for usability evaluation: The 10±2 rule[END_REF]. This new attempt has been criticised for being flawed [START_REF] Schmettow | Sample Size in Usability Studies[END_REF]. A detailed analysis has been made of the use of the "magic five" assumption. None of these or the previous references in this area have explored in more detail how heterogeneous different subgroups are and how different user groups experience different usability problems.

Method

We have conducted an exploratory study of usability problems experienced by different user groups. This section describes how the data was collected and analysed.

Data Collection

The data was gathered through a usability evaluation of a data dissemination website (dst.dk). This site provides publicly available statistics about the population (e.g. educational level or IT skills), the economy, employment situation, etc. Test Persons. All test persons were invited through emails distributed across the university. For this study data from 41 usability evaluations were included. The test persons consist of 12 faculty members from Ph.D students to professors, from different departments, 15 students in technical or non-technical educations, and 14 participants from technical and administrative staff from different departments. All participants received a gift with a value of approximately 20 USD for their participation. An overview of the participants can be seen in Table 1 on the following page.

All test persons were placed in one of six groups in regards to gender and age. The test persons varied in age between 21 and 66 years and consisted of 19 males and 22 females. All test persons were asked to assess their own skill level in regards to Internet usage on a scale from 1 to 5, where 1 was the lowest and 5 the highest score. The average for each group is shown in the table, none of the 41 test persons assessed themselves lower than 4. Originally 43 usability evaluations were conducted, but the data from two usability evaluations were excluded from this study, due to these test persons assessed themselves at skill level 3 in regards to Internet usage. All test persons were asked if they were familiar with, and used this website. 19 people answered that they had never used the website, 20 answered that they were familiar with the site and used it approximately once a year, and, two people answered that they use the website approximately once a month.

Usability Evaluations. All tests were conducted as think-aloud evaluations in a usability laboratory. The test monitor and test person were placed in different rooms and communicated through microphone and speakers in order to avoid the possibility of the test moderator's body language or other visible expressions, influencing each test person. All test persons were asked to fill out a short questionnaire after the test in regards to their participation.

Tasks. Each user solved eight tasks all varying in difficulty. Examples of this were that the first task was to find the total number of people living in Denmark while a more difficult task was to find the number hotels and restaurants with one single employee in a particular area of Denmark. Data Handling. All usability evaluations were recorded and the collected recordings were analysed by conducting video analysis. All recordings were analysed by two evaluators. Both evaluators had extensive previous experience in analysing video data. The videos were analysed in different random order, to reduce possible bias from learning. The following characteristics were used to determine a usability problem;

(A) Slowed down relative to their normal work speed (B) Inadequate understanding e.g. does not understand how a specific functionality operates or is activated (C) Frustration (expressing aggravation) (D) Test moderator intervention (E) Error compared to correct approach.

The data handling resulted in a list of 147 usability problems after duplicates had been removed. To determine similarities between problems from each list, the usability problems found by each evaluator were discussed. Across the analysis, the evaluators had an any-two agreement of 0.44 (SD = 0.11), which is relatively high compared to other studies [START_REF] Hertzum | The evaluator effect: a chilling fact about usability evaluation methods[END_REF]. Further information about the data collection can be found in [START_REF] Bruun | An Empirical Study of the Effects of Three Think-Aloud Protocols on Identification of Usability Problems[END_REF].

Data Analysis. We also uncovered which types of usability problems that were experienced by the different groups of participants. We did this through closed coding [START_REF] Strauss | Grounded theory methodology[END_REF] where each problem was categorised according to the 12 types listed in Nielsen et al. [START_REF] Nielsen | It's Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field[END_REF]. Two of the authors conducted this coding and did so independently of each other. It was decided in advance that the raters would code all and only use the data from the codings where the authors agreed on the category independently of each other. An interrater reliability analysis using the Fleiss Kappa statistic was performed to validate the result. This determines the level of consistency among the two raters. The result of was a moderate level of agreement (Kappa = 0.44, p < 0.001, 95% CI=0.37, 0.52) [START_REF] Landis | The measurement of observer agreement for categorical data[END_REF]. The 12 categorised used for this study are described next.

Affordance relates to issues on the user's perception versus the actual properties of an object or interface. Cognitive load regards the cognitive efforts necessary to use the system. Consistency concerns the consistency in labels, icons, layout, wording, commands etc. on the different screens. Ergonomics covers issues related to the physical properties of interaction. Feedback regards the manner in which the interface relays information back to the user on an action that has been done and notifications about system events. Information covers the understandability and amount of information presented by the interface at a given moment. Interaction styles concern the design strategy and determine the structure of interactive resources in the interface. Mapping is about the way in which controls and displays correlate to natural mappings and should ideally mimic physical analogies and cultural standards. Navigation regards the way in which the users navigate from screen to screen in the interface. Task flow relates to the order of steps in which tasks ought to be conducted. User's mental model covers problems where the interactive model, developed by the user during system use, does not correlate with the actual model applied to the interface. Visibility regards the ease with which users are able to perceive the available interactive resources at a given time.

The coding and analysis by two raters resolved in a list of 83 coded usability problems, out of originally 147 usability problems. This reduction happened as all usability problems where the raters did not agree on the category was removed from the study.

These categorisations were used to distinguish if test persons experienced the same type of usability problems, or if there were deviations across gender, age, job function or educational background. The results of this analysis are presented in the following section.

Results

In this section, we present the results from conducting this study. The results are presented from four different perspectives. First, the test persons are divided into males and females, then into the three age groups without taking the gender into perspective, then, the test persons are divided into groups both in regards to age and gender, and finally, the test persons are divided into groups in regards to education or work function. This was conducted to show if gender, age or background plays a role in regards to differences in the perceiving of usability problems. The numbers shown in the tables in the result section represent an average number of usability problems found per test person in each category. This was conducted to be able to compare groups containing different numbers of test persons, and still make the numbers comparable.

The results show that problems were found in regards to five of the twelve closed codings. Affordance, Cognitive Load, Feedback, Information, and Visibility, respectively. As problems were not found relating to Consistency, Ergonomics, Interaction Styles, Mapping, Navigation, User's Mental Model, and Task Flow, these categorisations will not be mentioned further.

Note that all results are based on the number of problems to which the two raters agreed on the categorisations, e.g. if the two raters did not agree on the code of a particular problem, this was excluded from the result. Out of the total 147 problems the raters agreed on 83.

Gender

We analysed whether males and females with similar skills in regards to internet usage experienced the same amount and type of usability problems. The results are presented in Table 2. An independent samples t-test revealed no significant differences in the total number of experienced between the genders (t=-0.9, df=39, p>0.2). We did, however find significant differences when considering the problem types related to feedback (t=-1.2, df=10, p<0.01) and information (t=-1.8, df=39, p<0.01).

Age

We also analysed if age had an impact on the experienced amount of usability problems. The results are presented in Table 3 on the following page.

A one-way ANOVA test revealed no significant differences in number of experienced problems between the three age groups (F=1.02, df=40, p>0.3).

Job Function and Educational Background

Finally, we analysed if a large number of test persons with a background in computer science had an impact in regards to the amount of usability problems experienced.

The results are presented in Table 4.

The table shows, that when dividing the test persons into job function or educational background, students which are not in computer science, experience more problems related to cognitive load and information. A one-way ANOVA test revealed no significant differences in the total number of problems experienced across job function or educational background (F=0.6, df=40, p>0.6).

Discussion

This study has focused on comparing the amount of usability problems found when grouping the test persons in regards to gender, age, and job function or educational background. This was conducted as all test persons assessed themselves as experienced internet users, as each rated themselves as either 4 or 5 on a scale from 1 to 5, where five was the highest score. This way, it could be explored if test persons of a high degree of internet skills experienced different types of usability problems, or if they could be considered a homogeneous group, where neither age, gender, and job function or educational background made a difference in regards to the average amount of usability problems.

Comparison with Related Work

Related work has shown that the amount of needed test persons varies [START_REF] Lewis | Sample sizes for usability studies: Additional considerations[END_REF], [START_REF] Virzi | Refining the test phase of usability evaluation: How many subjects is enough?[END_REF]. As demographical data was not included in these studies it is not possible for us to draw any conclusions in relation to the results from this study, though it raises the question of, if the test persons chosen by Virzi [START_REF] Virzi | Refining the test phase of usability evaluation: How many subjects is enough?[END_REF] were more homogeneous than the test persons chosen by Lewis [START_REF] Lewis | Sample sizes for usability studies: Additional considerations[END_REF] in regards to the skills of Internet usage or IT in general. This study has found indications that a user group can be homogeneous though a variety in age and background. Our results indicated that the test persons from this study experience around the same amount of usability problems in regard to each categorization (Affordance, Cognitive Load, Feedback, Information, Visibility), across gender, age, and background. This corresponds with Caultons' conclusions about homogeneous user groups experiencing the same usability problems [START_REF] Caulton | Relaxing the homogeneity assumption in usability testing[END_REF].

This study shows no greater difference in regards to the types of usability problems experienced by the test persons. This does not correspond with the findings of Law and Hvannberg who concluded that the heterogeneity of subgroups in a test will dilute the problem detection rate [START_REF] Law | Analysis of combinatorial user effect in international usability test[END_REF].

Implications for Usability Practitioners

Though further research is needed, this study indicates that recruiting test persons across gender, and age might not be necessary, as these findings show that users with approximately the same level of skills in regards to Internet usage, experience the same amount of usability problems. If, the indication that skill level is key, when recruiting test persons for usability evaluations, this means that the most important is to recruit test persons of all skill levels of the target user group for the website or application, and, that variety in age or gender is not important when recruiting test persons. The implications might especially be of interest, when developing websites or applications for large heterogeneous user groups e.g. public websites or self-service applications, as these types of sites are targeted for all citizens in a country. This will make it challenging to represent all types of users when conducting usability evaluations, as a lot of test persons would need to be recruited, and it would be costly to conduct this amount of usability evaluations. On the contrary, if test persons only need to be recruited in regards to their skill level of Internet usage and IT in general, this would reduce the cost considerably.

Conclusion

This

Limitations

We do recognise that further studies need to be conducted to be able to actually draw conclusions across user groups at different levels of Internet experience and that these results do not provide enough evidence to definitively rejecting the previously mentioned criticism of the "homogeneity assumption" by Law and Hvannberg [START_REF] Law | Analysis of combinatorial user effect in international usability test[END_REF]. This means that further research should be conducted with more homogeneous user groups with different levels of internet skills, and not just one group of experienced users. As it needs to be investigated further if these results also are valid for other user groups with lower skill levels in regards to Internet usage. We also recognise the limitations of our test persons having a higher educational background and a self-reported high expertise in internet usage. Also the fact that a lot of the found usability problems were discarded at the coding phase and therefore not included in the data analysis.

Table 1 .

 1 Demography for the 41 test persons.

	Number of	Age	Age	Gender	Backgrounds	Average
	people in each		average			Internet
	category					experience
	6	< 27	24	M	5 Computer Science students	5
					1 Computer Science faculty member
					5 Computer science students
	8	< 27	22	F	2 humanities students	4.6
					1 office trainee
					4 computer science faculty members
	8	27-44	36	M	1 social science faculty member 1 technical staff	4.8
					1 administrative staff
					1 engineering student
					6 administrative staff
	8	27-44	38	F	1 social science faculty member 1 information science student	4.3
					3 computer science faculty members
	5	44 <	55	M	1 faculty member medicine 1 technical staff	4.8
					4 administrative staff
	6	44 <	50	F	1 faculty member computer science 1 faculty member medicine	4.5

Table 2 .

 2 The average number of usability problems experienced when dividing the test persons by gender.

			Group Statistics
	Gender	N	Mean	Std. Deviation	Std. Error Mean
	Affordance
	Cognitive Load

Table 3 .

 3 Usability problems experienced by different age groups.

Table 4 .

 4 The average amount of usability problems experienced when dividing the test persons in regards to job function or educational background.

  paper presents a study of to what extent different test persons, who are all experienced internet users, experience different types of usability problems. This has been presented across age, gender, and educational background or job function. The results are interesting as it is indicated that the usability problems experienced by users with a high level of internet experience do not vary significantly, across gender, age or background. This means that finding test persons might not have to be balanced in regards to neither gender or age, but that is more important to find test persons on all levels of internet experience in the target user group. Our results also indicate that people with an education in Computer Science do not experience significantly fewer usability problems, than other experienced internet users.

adfa, p. 1, 2011. © Springer-Verlag Berlin Heidelberg 2011

adfa, p. 2, 2011. © Springer-Verlag Berlin Heidelberg 2011

adfa, p. 3, 2011. © Springer-Verlag Berlin Heidelberg 2011

adfa, p. 4, 2011. © Springer-Verlag Berlin Heidelberg 2011

adfa, p. 5, 2011. © Springer-Verlag Berlin Heidelberg 2011

adfa, p. 6, 2011. © Springer-Verlag Berlin Heidelberg 2011

adfa, p. 7, 2011. © Springer-Verlag Berlin Heidelberg 2011

adfa, p. 8, 2011. © Springer-Verlag Berlin Heidelberg 2011

adfa, p. 9, 2011. © Springer-Verlag Berlin Heidelberg 2011

adfa, p. 10, 2011. © Springer-Verlag Berlin Heidelberg 2011

adfa, p. 11, 2011. © Springer-Verlag Berlin Heidelberg 2011

adfa, p. 12, 2011. © Springer-Verlag Berlin Heidelberg 2011


Marius Koller
email: marius.koller@hs-heilbronn.de

Cristian Bogdan

Gerrit Meixner
email: gerrit.meixner@hs-heilbronn.de

Collaborative Task Modeling: A First Prototype Integrated in HAMSTERS

Keywords: Task Modeling, Collaborative Support, HAMSTERS 1

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The creation of task models is a complex undertaking that is usually performed by people from different professions and backgrounds. It is unlikely that just one person is working on task models in a given project. Therefore, task models will often be the result of some kind of collaboration. To support task authors and their daily work, digital support for the collaborative creation of task models is necessary. The needed digital support depends on the nature of task model creation process in each group.

In this demo we propose an integration of collaborative features in the HAMSTERS task modeling tool. We focus on the communication between the collaborators as well as on awareness between them. The proposed and prototyped functions are being implemented and will be evaluated in the future.

Task Models

Task Models are a well-known concept in research and academia. Task modeling has its foundations in the Hierarchical Task Analysis (HTA) that was defined in 1967 by Annett and Duncan [START_REF] Annett | Task Analysis and Training Design[END_REF]. HTA has the goal to analyze the user's tasks and structure it hierarchically. Many different task modeling notations were defined and are used in different, mostly academic, use cases, for different purposes. For example, the Useware Markup Language (useML) [START_REF] Meixner | Model-Driven Useware Engineering[END_REF] aims to generate user interfaces that are based on the created task models. For this purpose, the in useML defined Usemodel is transformed to other models and at the end it is possible to generate the final user interface. The Groupware Task Analysis (GTA) aims to support the task-based development of collaborative applications [START_REF] Van Der | GTA: Groupware task analysis-Modeling complexity[END_REF] (we should however distinguish collaboration during task modeling from collaboration of end users modeled by collaboration-aware task models such as GTA). The Méthode Analytique de Description (MAD) is a semi-formal method to record user tasks based on interviews [START_REF] Rodriguez | Editing MAD* task descriptions for specifying user interfaces, at both semantic and presentation levels[END_REF] that can be used during the requirements analysis phase. The ConcurTaskTrees (CTT) are and well-known approach [START_REF] Paternò | ConcurTaskTrees: an engineered notation for task models[END_REF] which has dominated research approaches lately. CTT aims to analyze and record the user's task and generate models from them. The Human-centered Assessment and Modeling to Support Task Engineering for Resilient Systems (HAMSTERS) [START_REF] Martinie | A Generic Tool-supported Framework for Coupling Task Models and Interactive Applications[END_REF] uses some concepts from CTT and aims to identify possible human errors and to prevent them.

Most of these notations have graphical representations and editors to create graphical task models. For instance, HAMSTERS (see Figure 1) and CTT's editors CTTE [START_REF] Paternò | CTTE: An Environment for Analysis and Development of Task Models of Cooperative Applications[END_REF] and Responsive CTT [START_REF] Anzalone | Responsive Task Modelling[END_REF] use small icons for the representation of tasks. For the different type of tasks there are specific icons defined and used in the models. UseML's editor Udit [START_REF] Meixner | Udit-A Graphical Editor For Task Models[END_REF] represents the items with boxes and differs them using a color-code. CTTE and Udit are stand-alone software whereas Responsive CTT is web-based.

Computer Supported Cooperative Work

The field of Computer Supported Cooperative Work (CSCW) has been aiming to support technology-mediated collaboration for several decades.

Collaborative editors are an important area of CSCW research, and it is collaborative task model editors that we aim to create a system for collaborative editing. It needs to support the following features, according to [START_REF] Greif | A case study of CES: a distributed collaborative editing system implemented in Argus[END_REF]:

1. Distribution: the clients are not co-located but distributed and connected over the internet or an internal network. 2. Sharing: the clients should be able to share documents but be aware with whom they are sharing it. 3. Autonomy: a document has an owner that has the right manage it, e.g. grant access. 4. Reliability: the stored data has to be robust to connection losses and be accessible when a client crashes. 5. Reconfiguration: the clients should easily configure the system and its behavior. 6. Concurrency: the system has to allow editing at one time. 7. Consistency: the stored data has to be consistent even during the concurrent editing. 8. Performance: the responses should have a reasonable delay that allows a fluent working with the system. 9. High-Quality User Interface: the provided user interface should support the clients during their work.

Sutcliffe defines requirements for groupware, some of them are similar to the above [START_REF] Sutcliffe | Applying Small Group Theory to Analysis and Design of CSCW Systems[END_REF]. He adds for example the "group's collective goal awareness"that means the group should be aware of its goal, means why they are collaborate and that may include sub-goals. Also, possible conflicts have to be managed in a transparent and fair way. Sutcliffe's findings do not change the defined requirements but support and extend them.

One challenge in CSCW is the concept of "awareness" defined as "understanding the activity of others in the context of one's own activity" [START_REF] Dourish | Awareness and Coordination in Shared Workspaces[END_REF]. One central question is "what should the user be aware of?" [START_REF] Schmidt | The Problem with `Awareness': Introductory Remarks on `Awareness in CSCW[END_REF].

Collaborative support in Task Modeling environments

To understand the needs for a collaborative support we need to know: who are the collaborators? What are their professions, background? We found in a small pilot-study that many different professions engage in task model authoring. They may not have a technical background and this needs to be taken into account in the collaborative graphical editor design. We identified three groups: Usability Experts, Software Developers/Engineers and User Interface Designers. From literature and other projects, we know that in some cases the management is involved as well. In our own studies we found that the teams may be distributed and the interaction or collaboration -which means the implemented process -differs between companies.

We looked at four task modeling tools and evaluated their possibilities for cooperation support.

At first we identified features that afford some sort of collaboration. The most basic form for developers is the sharing of file with a Version Control System (VCS) like for example Git or Subversion (SVN). Other ways of sharing files, like simple "saving in the cloud" and sharing with othersat the same or different time. Also an important aspect for collaboration is the communication. There are different communicationchannels that could be used: text-based chat, call or video-telephony. For the distributed creation of task models, it is useful if users could comment on tasks, branches of a tree model or on the whole model. Users may want to know how a model evolved during the creation. For that a history feature is useful where the users can see what was changed.

? ?  Chat     Comment     History    
As shown in Table 1, HAMSTERS is the only tool that provides some aspects of collaborative support. HAMSTERS has an integrated version control (VCS) that allows the users to share their project. The history is closely related to the VCS. The view shows the underlying XML-file and shows the differences between the selected version and the current working-copy. There is no possibility to show the differences in the graphical representation. Several incipient collaboration features are supported by other modeling tools as well. In CTTE it is possible to upload the file to a public or private repository at HIIS. Responsive CTTE could be able to share the models. Since it is web-based and the models are saved on servers, it could be possible to share the files. Currently this function is not implemented. Udit has no functionality for collaboration integrated.

Hili et. Al [START_REF] Hili | Innovative Key Features for Mastering Model Complexity: Flexilab, a Multimodel Editor Illustrated on Task Modeling[END_REF] introduce with FlexiLab a tool that offers collaborative functions amongst others. They focus in their work on real-time communication and sharing of models. The sharing enables the users to work on different parts of a model and share it.

Mock-ups of collaborative functions

Since we have some functions that are already integrated in HAMSTERS, we decided to continue working with it adding collaborative support. In order to generate ideas on how a collaborative support could look like, and in order to be able to analyze and judge these design ideas, we developed prototypes for the different collaborative functions. The ideas are being implemented as online prototypes, and prepared for demo. Since HAMSTERS is based on NetBeans™, we intend to stick on the NetBeans tab-based navigation concept.

The first function that we considered is the communication between users. We thus integrated a tab that shows a chat tool We decided to introduce a tab with the same behavior like the other tabs (e.g. tabs could be arranged according to the user's preferences). An important point to keep in mind is that the chat has to be visible for every user, including what was said before he or she joined the talk. The mockup is depicted in figure 2.

Fig. 2. The integrated chat.

As a second major function we identified commenting. We believe that for the collaborative creation it is important to tell the other editors how you are thinking while designing a task model. The comments are anchored on specific parts of a task model. Therefore, we intend to introduce comments that can be attached to a specific task or a branch of the model. The possibility to add a comment will be shown at the object of interest. Depending on the kind of task. e.g. Abstract Task or User Task, the comment will refer to a whole branch or only to this specific task. The first comment in figure 3 includes the possibility to comment on the whole branch while the second comment or conversation is regarding this particular task. It will be possible to start discussions within the comments. The current HAMSTERS implementation of a history is readable only by experts, e.g. Software Engineers, as it is based on version control specific to software. Users from other disciplines that use the application and its models are currently not able to understand the changes without an expert explanation. To support such users, we will introduce a graphical representation of task model history. One design alternative is to display history as grayed-out objects.

The sharing of the model with real-time editing may need changes in the user account structure of HAMSTERS. For example, we need to display the currently active users and which users have access to the model (see figure 4). To achieve that, we have to provide a login for the users. Again, to be consistent with the existing UI design of HAMSTERS we will introduce a tab that contains the currently active users and the users that have access to the model. After implementing such basic collaborative support for task editing, we will look into more advanced support, derived from field studies of task modeling that we are conducting. For example, awareness [START_REF] Greif | A case study of CES: a distributed collaborative editing system implemented in Argus[END_REF][START_REF] Rodriguez | Editing MAD* task descriptions for specifying user interfaces, at both semantic and presentation levels[END_REF] will be initially supported only by our history feature, which is a good beginning is history is shown permanently in a tab, and maybe recent changes are promoted in the tab title. More awareness support can be added to emphasize important task model changes made by collaborators.

Conclusions

We regard task modeling as complex interdisciplinary collaborative work. Different disciplines and profession combine their knowledge and expertise in these artifacts. However, in the current tools for task modeling we found only little collaborative support. Only HAMSTERS includes with SVN and Git which are well-known collaborative tools for software developers. We propose new features that introduce collaborative support in HAMSTERS. We believe that especially the possibility to comment on tasks or branches will support distributed teams that work asynchronously. The other features will ease the creation of task models in groups, too. We are in the process of implementing the collaborative task modeling features and we aim to later evaluate these features.

Fig. 1 .

 1 Fig. 1. The graphical editor HAMSTERS.

Fig. 3 .

 3 Fig. 3. Two comments on tasks.

Fig. 4 .

 4 Fig. 4. Sharing the model.

Table 1 .

 1 Analyzed tools.

	Feature	Udit	CTTE	Responsive	HAMSTERS
				CTTE
	Sharing	


Rui Alves
email: rui.alves@m-iti.org

Nuno Jardim Nunes

Ceiling and Threshold of PaaS Tools: The Role of Learnability in Tool Adoption

Keywords: PaaS, threshold, ceiling, learnability

Cloud services are changing the software development context and are expected to increase dramatically in the forthcoming years. Within the cloud context, platform-as-a-service tools emerge as an important segment with an expected yearly growth between 25 to 50% in the next decade. These tools enable businesses to design and deploy new applications easily, thereby reducing operational expenses and time to market. This is increasingly important due to the lack of professional developers and it also raises a long standing issue in computer-aided software engineering: the need for easy to learn (low-threshold), functional (high-ceiling) tools enabling non-experts to create and adapt new cloud services. Despite their importance and impact, no research to date addressed the measurement of tools' ceiling and threshold. In this paper we describe a first attempt to advance the state of the art in this area through an in-depth usability study of platform-as-a-service tools in terms of their threshold (learnability) and ceiling (functionality). The measured learnability issues evidenced a strong positive correlation with usability defects and a weaker correlation with performance. Remarkably, the fastest and easiest to use and learn tool falls into the low-threshold/low-ceiling pattern.

Introduction

Within the next 30 years the demand for software applications will grow exponentially 1 . Yet, our capability to increase the number of professional software developers and/or their productivity will at best stabilize or grow linearly. Bezivin call it the impossible equation (see Fig. 1) stating that "How, with the same amount of people (code producer and managers, about 1% of the total population), to produce an order of magnitude more of software applications than now?" 2 . This equation seems impossible to solve because, on one hand it is not practical to educate a larger percentage of the world population to develop professional software. On the other hand, lessons from the past show that whatever progress we may achieve in software tools, the productivity of software professionals will not improve by a major factor. As stressed by Bezivin, the key to solving these problems probably lies in the collaboration between professional developers and end-users. This is a long-standing challenge in engineering interactive systems. In order to enable end-users to contribute to software development, we need to provide new languages and development environments that effectively suite their needs and are easy to learn and operate. These tools are known after the seminal paper about "the past, present and future of user interface software tools" [START_REF] Myers | Past, present, and future of user interface software tools[END_REF] as low-threshold and high ceiling tools. In fact, a major trait in tools' acceptance is the role of threshold and ceiling (the threshold is how difficult it is to learn how to use a tool and the ceiling is how much can be done using the tool). Successful tools in the past 25 years are claimed to either be low-threshold/low-ceiling, or high-threshold/high-ceiling [START_REF] Myers | Past, present, and future of user interface software tools[END_REF]. Despite all efforts we struggle to find tools that enable low-threshold and high-ceiling.

Currently, one major trend lies in what is usually known as cloud infrastructure, involving platforms and solution stacks as a service model of computing. These involve software-as-a-service (SaaS), infrastructure-as-a-service (IaaS) and platformas-a-service (PaaS). The focus of this paper is the PaaS model in which the end-user controls software deployment and configuration settings, while the vendor provides everything required to host the application [START_REF] Rimal | A taxonomy and survey of cloud computing systems[END_REF]. PaaS are characterized by providing low initial cost, incremental growing, self-service, resource sharing, automated deployment, management services, lifecycle management and reuse [START_REF] Lawton | Developing software online with platform-as-a-service technology[END_REF].

In this paper we attempt to advance the state of the art by increasing our understanding about what these concepts mean and how they can be effectively measured. We do this through: (1) a methodological approach to assess the usability and functionality of PaaS tools, (2) the results from a comparative study of three PaaS tools with 22 subjects and (3) statistical analysis looking for correlations between dimensions such as learnability issues, usability defects and performance. Lastly, in the discussion section, we contrast our findings with the prevailing literature and then present our conclusions.

State of the Art

Both software engineers and designers often complain that their tools are unsupportive and unusable [START_REF] Myers | Past, present, and future of user interface software tools[END_REF]. While many studies have analyzed and tried to better support general software development practices [START_REF] Iivari | Why are CASE tools not used?[END_REF][START_REF] Seffah | The obstacles and myths of usability and software engineering[END_REF], qualitative studies on user interface related practices, in software development, are relatively rare. Furthermore, the work of Seffah and Kline, showed a gap between how tools represent and manipulate programs and the actual experience of the developer [START_REF] Seffah | The obstacles and myths of usability and software engineering[END_REF]. However, ceiling and threshold related issues were not specifically addressed.

Among the distinct sets of existing development tools, there is a group of literature on computer-aided software engineering (CASE) tools [START_REF] Myers | Past, present, and future of user interface software tools[END_REF][START_REF] Iivari | Why are CASE tools not used?[END_REF][START_REF] Campos | Practitioner tools and workstyles for user-interface design[END_REF][START_REF] Jarzabek | The case for user-centered CASE tools[END_REF], which we think worth assessing if the findings related to these tools prevail in the scope of this work. This prior research suggests that: (1) few organizations use CASE tools; (2) many organizations abandon the use of the tools; and (3) countless developers, working for organizations that own CASE tools, do not actually use them. Moreover, Jarzabek and Huang argued that CASE tools should be more user-oriented and use rigorous modeling, in order to better blend into the software development practice [START_REF] Jarzabek | The case for user-centered CASE tools[END_REF]. Yet, figures on the adoption of these tools seem to contradict the goals driving their development. Though, a new breed of tools has emerged, in particular Platform-as-a-Service (PaaS), which claim to bridge previous gaps and promise easier and faster development, even for non-experts. Intrigued by these apparently conflicting forces, we found room to contribute by researching the role of learnability in tools adoption.

Research Question

Since practitioners often complain about the suitability of tools for their work, we realized that threshold and ceiling are claimed as attributes that play a major role in tools adoption. Moreover, successful tools are said to either be low-threshold/lowceiling or high-threshold/high-ceiling. The issue is that we lack knowledge on how to measure these concepts. Thus, we hypothesized that the threshold is associated with tool's adoption. So, if a tool is hard to learn then it's adoption is more likely to be low, since people will seek out easier solutions. The work presented here is part of a major project, which in the future will help us shed some light on tools adoption and we focus on how to define and measure the core concepts of the framework for tools' threshold and ceiling measurement.

In this paper we measure the initial learnability of tools and assess the relationships between the learnability issues and the usability defects, as well as eventual impacts on subjects' performance. Additionally, recognizing that the terminology is not consistent and before jumping into the details, we would like to clarify the core concepts used in this paper. For instance, learnability is the attribute of a system, which allows users to rapidly begin working with it [START_REF] Holzinger | Usability engineering methods for software developers[END_REF]. In this paper a learnability issue is any problem preventing users from quickly understand how to use the system to do a task. A defect is a breakdown in usability, an obstacle to perform a desired task or a problem in the smooth functioning of the system. A usability defect is generally a mismatch between what the user wants, needs or expects and what the system provides. Finally, performance is the time a participant took to do a task.

Study

The research study described here was conducted over a period of six months (September 2013-February 2014) and expands the sample of the study described in a previous publication [START_REF] Alves | Towards a measurement framework for tools' ceiling and threshold[END_REF]. Participants were asked to play the role of a small shop owner willing to replace his spreadsheet to manage products with a web application. For that purpose, this owner is giving a try to PaaS tools and will try three tools (Knack [START_REF]Knack-the easiest online database builder and web app builder[END_REF], Mendix Business Modeler 4.7.0 [START_REF]App Platform for The Enterprise | Mendix[END_REF] and OutSystems Studio 8.0 [START_REF]OutSystems Platform -High Productivity Platform as a Service -PaaS[END_REF]).

PaaS is a growing and highly competitive sector with more than 70 vendors, making it hard to select which PaaS tools to test. We surveyed the web for comparisons and picked two tools that are popular and comparable to the tool proposed by the vendor that asked us to evaluate their tool. Additionally, prior to deployment, we performed three pilots to improve the test instructions, assess the duration, completeness and suitability of proposed tasks. Furthermore, we faced two major challenges to run this study: (1) how to record subjects' actions so that they did not felt observed and (2) the test duration. To address both, upon subjects' informed consent, we recorded video and audio and captured screen actions using 15" laptop built-in features. Having no extra hardware helped subjects perceive it naturally. Additionally, the amount of tools tested was constrained by the fact that subjects will do the same set of tasks on each tool. As such, it proved unfeasible to add more tools (due to subjects' engagement and motivation) since the test duration roughly spanned from 2,5h to 3h per subject. Because of this plus the fact that we required participants to be completely new (initial learnability) to all tested tools, volunteers were scarce.

Sample

We used a sample of 22 subjects from both genders, divided into two even groups: (G1) comprised people who run small businesses and (G2) IT experts. The rationale for choosing these groups was to contrast their results and assess if PaaS is ready for non-IT experts, as claimed by some vendors. On average, subjects were 30 years old (minimum 22 and maximum 43). Circa 30% had worked for less than one year and 40% had between four and nine years of experience, while 20% had already worked for more than ten years. The vast majority (60%) holds a bachelor degree and 35% hold a master's degree. Additionally, none of the G1 subjects had any experience building web applications or had ever used the tools of the trade, whereas all, but one, G2 subjects were experienced in building web apps. Among the G2 subjects, IDEs were popular tools used to create web apps (50% uses Netbeans, 40% Dreamweaver) but no one had used or even knew of PaaS tools.

Methodology

As previously stated, this paper expands (from ten to 22 subjects) and builds on top of a previous publication [START_REF] Alves | Towards a measurement framework for tools' ceiling and threshold[END_REF]. A detailed explanation of the methodology used was already provided in the methodology section of [START_REF] Alves | Towards a measurement framework for tools' ceiling and threshold[END_REF], please refer to this reference for further detail on the methodology used. Here we focus on the results and provide a statistical analysis of the results to complement the claims and findings stated in [START_REF] Alves | Towards a measurement framework for tools' ceiling and threshold[END_REF].

Results

Three dependent variables were measured in this study, specifically (i) learnability issues, (ii) usability defects, and (iii) performance. Additionally, based on subjects' level of success to complete each task, learnability scores were computed also.

Learnability Issues

Grossman's et al classified learnability issues into five categories: (i) awareness, (ii) locating, (iii) transition, (iv) task-flow and (v) understanding. For detailed explanation on each category please refer to [START_REF] Grossman | A survey of software learnability: metrics, methodologies and guidelines[END_REF]. Table 1 summarizes each category measured weight. The most frequent categories are highlighted in bold and account for 50%+ of issues per group/tool. The transition category is the less frequent, except for G1 in Knack. We used the transition category to compute a ratio between all categories and transition, to learn what categories hinders users the most. Table 2 stresses the proportionality among categories and we found evidence that understanding and task-flow issues hinder users the most, being three times more prevalent than transition issues. In brief, these figures provide empirical evidence that Knack was easier to learn than Mendix and Outsystems. Moreover, all tested tools seem to have a lot of technicalities since all fail to make it easier for non-technical users to quickly learn how to use the tool, risking their putative success in this niche.

Usability Defects

The usability defects were categorized according to their relation to the interface. The icons/graphics category are related to a graphical design issue (similar icons). The bars/windows category is directly related to using bars or high level commands. When the defect was related to buttons or input fields then it was categorized as menus/commands, whereas defects classified as interaction are, for instance, double clicking creates something without questioning the user). Finally, the text/feedback category are issues on the textual terminology and text feedback on the interface. Table 3 summarizes the weight of each usability defect category, per tool and group. In the same table we have highlighted, in bold, the top two categories per group and tool, which account for more than 50% of all defects. Similarly to learnability issues, we have also computed a ratio between all categories and bars/windows, to grasp an idea of what categories have a stronger impact on subjects Table 4. In brief, the trend found on learnability issues is also true here and provides empirical evidence that Knack was easier to use than Mendix and Outsystems. Moreover, all tools seem to suffer from usability defects and fail to make it fully usable for non-technical users, which PaaS vendors claim to be their target.

Performance

Table 5 shows the performance, in hours, per group/tool. Curiously, the tasks that took longer in each tool, took longer for both groups. Additionally, we found that the amount of time spent in each task ranged a lot per tool, although the tasks were the same for all tools tested. For instance, while in Knack the T4 and T5 were the tasks that took more time to do, in Outsystems these two tasks were the quickest ones. These differences could evidence the intrinsic difficulty to accomplish a task in one tool, which in turn could be associated with high, or low frequency of learnability issues, thus impacting tools' threshold. In brief, our results provide evidence that the longer it takes to finish the task, the harder it is to learn how to do it. Performance figures follow the trend found in 4.1 and 4.2, providing empirical evidence that Knack is faster to use than Mendix and Outsystems.

Statistical Analysis

The results highlighted a difference between tools where Knack (classified as lowthreshold/low-ceiling) was the fastest, easiest to learn and use tool being tested, whereas the other two tools produced similar performance results. In this section we investigated if there is statistical evidence, which could ground the previous claims. The tools used in this study were encoded in one independent variable and we measured the differences for each dependent variable (learnability issues, usability defects and performance). Additionally, we also sought for possible correlations.

Differences

We have used a repeated measure analysis of variance (ANOVA) to test if there are differences between conditions (Knack, Mendix and Outsystems), where the same participants are being measured on the same variable (learnability issues, usability defects or performance). The groups are related because they contain the same cases (the subjects) and each group is a repeated measurement on a dependent variable. This led us to the null hypothesis: H0: all related group means are equal. The alternative hypothesis is Ha: at least one related group mean is different, which would provide evidence on statistical differences between tools, if verified. We verified all the assumptions and found no outliers, in learnability issues, by inspecting the boxplots for values greater than 1.5 box-lengths from the edge (dark grey boxplots on Fig. 2). Data is normally distributed in all three conditions, as assessed by Shapiro-Wilk's test (p > 0.05) and Mauchly's Test of Sphericity indicated that the sphericity assumption has not been violated, χ2 (2) = 1.048, p = 0.592. Having satisfied all assumptions, we computed the repeated measures ANOVA, which yielded statistically significant changes in the amount of learnability issues found per tool, F(2, 42) = 8.868, p < 0.001, partial η2 = 0.297. Although this result rejects the null hypothesis and retains the alternative hypothesis, we cannot directly determine where exactly the differences between groups lie. To clarify this uncertainty, a post hoc analysis with a Bonferroni adjustment revealed that the amount of learnability issues significantly decreased from Outsystems to Knack (1.68 (95% CI, 0.25 to 3.12) issues per subject, p < 0.05) and from Mendix to Knack (2.64 (95% CI, 0.97 to 3.76) issues, p < 0.01), but not from Mendix to Outsystems (0.68 (95% CI, -0.98 to 2.34) issues, p = 0.895). As such, these results provide statistical evidence that Knack offers less learnability issues for first time users than the other tested tools.

Fig. 2. Learnability issues and usability defects

The same procedure was applied to usability defects and we found no outliers (light grey boxplots on Fig. 2). Yet, we realized that Knack data was not normally distributed (p = 0.041), although it is normally distributed for Mendix and Outsystems, p > 0.05. Nevertheless, we run the test because the repeated measures ANOVA is fairly robust to deviations from normality. Mauchly's Test of Sphericity indicated that the assumption of sphericity was met (χ2 (2) = 2.46, p = 0.292) and the repeated measures ANOVA test elicited a statistically significant change in the amount of usability defects per tool, F(2, 42) = 12.519, p < 0.0005, partialη2 = 0.373. A post hoc analysis with Bonferroni's adjustment revealed that the amount of usability defects significantly decreased from Mendix to Knack (2.091 (95% CI, 0.01 to 4.2) defects per subject, p < 0.05) and from Outsystems to Knack (3.455 (95% CI, 1.7 to 5.2) defects per subject, p < 0.0005), but not from Mendix to Outsystems (1.364 (95% CI, 0.2 to 2.93) defects per subject, p = 0.102). Given these results, there is statistical evidence that Knack has less usability defects than the other tested tools.

In what concerns performance, the first assumption was violated since there are outliers in Knack and Mendix datasets. Moreover, although the Shapiro-Wilk's test yielded p > 0.05 for Knack and Outsystems, the normality test returned a negative result for Mendix (p = 0.038). Thus, contrarily to what happened with learnability issues and usability defects, we cannot make any statement about statistical evidence on performance differences between the three tools by using this test.

To sum up, we found statistical evidence that (1) Knack was easier to learn (for first time users) than Mendix and Outsystems, (2) Knack was easier to use than Mendix and Outsystems. These findings are in line with the results section. However, we found no statistical evidence that subjects were faster in Knack than in Mendix or Outsystems, although the figures in the results section point towards that direction.

Correlations

There is no statistical test to determine the association between three continuous variables, which is our case and Pearson's correlation was not suitable since there was no linear relationship between variables, as assess by inspecting a scatterplot. As such, we opted for Spearman's correlation. Thus, the null hypothesis is H0: there is no association between the variables and the alternative hypothesis is Ha: there is an association between the variables. Two of the three assumptions of the Spearman's correlation relate to the study design, which we met both (to have two continuous or ordinal variables and the two variables represent paired observations). The third assumption requires to determine whether there is a monotonic relationship between two variables. A monotonic relationship does one of the following: (i) as the value of one variable increases, so does the value of the other variable or (ii) as the value of one variable increases, the other variable value decreases. The Spearman's correlation test unveiled that there was a strong positive correlation between the amount of identified learnability issues and usability defects, rs(20) = 0.724, p < 0.01. Preliminary analysis showed the relationship to be monotonic, as assessed by visual inspection of a scatterplot. In the other two cases, although the monotonic relationship was not fully clear, there was a remarkably positive correlation between the amount of learnability issues and subjects' performance (rs(20) = 0.466, p < 0.01) and a moderately positive correlation between the amount of usability defects found and subjects' performance, rs(20) = 0.352, p < 0.01.

Threshold and Ceiling

As successful tools are said to either be low-threshold/low-ceiling, or highthreshold/high-ceiling, we plotted the tested tools within this framework [START_REF] Myers | Past, present, and future of user interface software tools[END_REF]. We achieved the threshold by measuring the learnability of tools. As aforementioned, Knack is easy to learn whereas Mendix and Outsystems scored as regular. Yet, the other attribute i.e. ceiling, was missing. Since ceiling shows how much can be done using the system, there are two major approaches one can take to assess it: (1) compare tools usage over time or (2) compare the features each tool offers [START_REF] Myers | Past, present, and future of user interface software tools[END_REF]. The first option, which could yield results based on empirical data and could be grounded on actual tool usage, proved impractical due to time constraints and lack of actual users to whom we could refer. Thus, we selected the second option and measured each tool ceiling by comparing the features offered by the tool. To do this we identified another tool (www.force.com), which is supposedly one of the most complete in the PaaS context, and compared it to the tested tools feature set. This approach yielded that from a total of 31 features, Knack matches with seven, Mendix matches 29 features and Outsystems 31 features. Therefore, the computed ceiling was 23% for Knack, 94% for Mendix and 100% for Outsystems.

Additionally, we ranked it according to this ceiling scale: (i) very low [0 to 19%], (ii) low [20 to 39%], (iii) regular [40 to 59%], (iv) high [60 to 79%] and (v) very high [80 to 100%]. This means that according to our measure Knack is low ceiling and Mendix and Outsystems are both very high ceiling. Table 6 summarizes the threshold and ceiling classification based on aggregated data (for initial learnability). Threshold is a measure for how difficult it is to learn a tool and is obtained by inverting the learnability scores. Upon surveying the literature, we found several learnability metrics from which we selected two: (M1) the percentage of users who complete a task without any help and (M2) the percentage of users who complete a task optimally [START_REF] Grossman | A survey of software learnability: metrics, methodologies and guidelines[END_REF]. In M1 users could not ask task-flow related questions. In M2 users must complete the task without help [START_REF] Linja-Aho | Evaluating and Improving the Learnability of a Building Modeling System[END_REF]. The aggregated results based on all 330 tasks (22 subjects did five tasks each in three distinct tools) performed, provide empirical evidence that Knack is the easiest tool tested, which reached the easy to learn score with 63%, followed by Outsystems with a score of regular (53%) and Mendix ranked last with 46%. This means that if a tool is easy to learn then the threshold is low or if a tool is very difficult to learn then the threshold is very high. Inversely, the ceiling is determined directly (100% matching of features is very high ceiling). As such, only one tool (Knack) matches the low-threshold/low-ceiling or high-threshold/highceiling pattern identified in [START_REF] Myers | Past, present, and future of user interface software tools[END_REF]. The other tools do not match this pattern. The difference in Outsystems was only 7%, which made us wonder if a bigger sample would make a difference or if it will make the gap between tools even bigger.

Discussion

The work presented here is part of a major project addressing other tools and development environments, here we focused only on the building blocks of the framework for tools' threshold and ceiling measurement, because they were reported as key attributes in tools adoption [START_REF] Myers | Past, present, and future of user interface software tools[END_REF]. As such, we measured the learnability issues and usability defects found per subject and tool, as well as subject performance.

With regard to the learnability issues, the Grossman et al classification [START_REF] Grossman | A survey of software learnability: metrics, methodologies and guidelines[END_REF] was helpful for clustering issues. The raw data and the statistical analysis provided evidence that Knack was easier to learn than Mendix and Outsystems. This is arguably of little interest for generalization purposes. Yet, the fact that, in general, task-flow and understanding related issues are three times more prevalent than transition issues, presents a clear hint for aspects that vendors need to observe in order to increase their tools learnability. This previous claim is moderately in line with the findings from [START_REF] Alves | Towards a measurement framework for tools' ceiling and threshold[END_REF] where understanding and task-flow related issues hinders users the most, although with distinct proportionality. One fact that could justify this difference is the sample size which comprised ten subjects in [START_REF] Alves | Towards a measurement framework for tools' ceiling and threshold[END_REF] and 22 here.

Regarding the usability defects, which were reported as a major issue preventing adoption [START_REF] Malavolta | What Industry Needs from Architectural Languages: A Survey[END_REF], we found statistical evidence that Knack was easier to use than Mendix and Outsystems. Additionally, the differences between categories of defects was not as evident as the learnability issues, yet, defects related to interaction, menus/commands and text/feedback happen twice as often as defects related to bars and windows. Again, tools creators should take this evidence into account so that they can diminish the existing usability barriers and make these tools fully usable for nontechnical users, which some PaaS vendors target for. Moreover, the performance data shows that Knack was quicker to use than Mendix and Outsystems, although no statistical evidence was found (not all needed assumptions to run the repeated measures ANOVA test were met).

Remarkably, the correlation tests revealed a strong positive correlation between the number of learnability issues and the usability defects, which stresses the visceral relationship between these two attributes. Noteworthy is also the positive correlation between the number of learnability issues and performance issues (this result hints that the more learnability issues, the longer it takes to do the task) yet, correlation is not an implication. Lastly, a moderately positive correlation between usability defects and performance should also be highlighted.

Additionally, regarding the ceiling/threshold, our findings are in line with Myers et al, who claim that successful tools are either low-threshold/low-ceiling or highthreshold/high-ceiling [START_REF] Myers | Past, present, and future of user interface software tools[END_REF]. The fact is that the fastest, easiest to learn and use tool was the only low-threshold/low-ceiling tool tested (Knack). Yet, the success and adoption of tools is far more complex than simply measuring a tool's threshold and ceiling (social and motivational factors, for instance, are claimed to play a major role). As such, since our sample comprises only three tools we cannot make any claim based on this finding. Having said that, we came to realize that both academia and industry are still failing, to build tools that allow us to create sophisticated systems easily (lowthreshold and high ceiling) [START_REF] Myers | Survey on user interface programming[END_REF], and that this is a demanding challenge which requires further research to address it.

Jeng claims that learnability is inferred from the amount of time required to achieve user performance standards [START_REF] Jeng | Usability assessment of academic digital libraries: effectiveness, efficiency, satisfaction, and learnability[END_REF]. Yet, in our study, we had no access to real users' performance figures. To do that we would need to perform an extended test with more tools and users. Nevertheless, our goal in this phase was to measure initial learnability, and was not intended to be a longitudinal study.

A note of caution when interpreting these results should be observed due to the intrinsic subjectivity of these evaluations. The fact that all measures were obtained from analyzing video and audio recordings is prone to subjective interpretation. For instance, when identifying learnability issues, one researcher can consider a fact as an issue whereas another person could skip it. However, if there is no perfect way to this, it does not mean that we should give up. Instead, to know and understand the underlying facts that impact tool adoption is an objective worth pursuing, and one which could benefit the entire community. As such, we suggest that a pool of analysts should review the recordings and doubtful cases should be pair reviewed to reduce bias. Another hypothetical weakness of this work is the sample size, which is arguably small. However, we cannot ignore how complex this kind of study is. Knowing that (1) subjects should have no experience at all in all three tools, and that (2) each subject spent roughly 2,5 to 3 hours using these tools and got no incentives to do so, we can easily understand why did it took us half a year to reach 22 subjects, which heavily limited our efforts to have a bigger sample, with a larger set of tools.

Conclusion

The cloud infrastructure is pushing the requirements of development tools in the face of the increasing needs of software and the growing importance of cloud infrastructure. PaaS is becoming an important because it facilitates the deployment of applications or services reducing cost and complexity. PaaS are required to handle tasks from editing code to debugging, deployment, runtime, and management. The prevailing literature on development tools hints at several factors contributing to PaaS adoption: (1) software engineers and designers often complain that their tools are unsupportive and unusable [START_REF] Myers | Past, present, and future of user interface software tools[END_REF], (2) there is a demand for tools more user-oriented tools, which better blend into the software development practice [START_REF] Jarzabek | The case for user-centered CASE tools[END_REF] and (3) a gap between how tools manipulate programs and the developers' actual experiences [START_REF] Myers | Past, present, and future of user interface software tools[END_REF].

Since tools ceiling and threshold are among the key factors impacting tools adoption [START_REF] Rimal | A taxonomy and survey of cloud computing systems[END_REF] and, due to the lack of existing knowledge on how to measure these attributes, we researched how to measure the threshold by assessing tools initial learnability. For that purpose, we have used a hybrid protocol by making use of thinkaloud and question-asking. In brief, all tested tools seem to fail to make it easy to learn, use and perform for non-technical users, which some PaaS vendors claim to be their major target market. Additionally, although threshold and ceiling measurement proved to be challenging and the findings must be thoughtfully analyzed (for instance, high learnability may be necessary for a low threshold, but high learnability may not guarantee a low threshold), we think this is an effort worth pursuing, in light of contributing to our understanding on how to design the tools of the future.

The authors would like to thank Claudio Teixeira, Amanda Marinho and Monica Nascimento for their help to conduct the study described in this paper.

Fig. 1 .

 1 Fig. 1. The impossible equation (from http://modelseverywhere.wordpress.com)

Table 1 .

 1 Learnability issues weight per tool.

			Knack			Mendix		Outsystems
	Category	G1		G2		G1		G2		G1		G2
		∑	%	∑	%	∑	%	∑	%	∑	%	∑	%
	Awareness	5	8	11	26	16	17	9	15	16	16	8	17
	Locating	11	18	9	21	26	27	20	33	14	14	7	15
	Transition	6	10	5	12	9	9	6	10	3	3	6	13
	Task-flow	10	17	5	12	33	35	12	20	25	26	13	27
	Understanding	28	47	13	30	11	12	13	22	40	41	14	29
	Total	60		43		95		60		98		48

Table 2 .

 2 Relative impact of each learnability issue category.

	Category	G1	G2	Knack	Mendix	Outsystems	Total
	Awareness	2	2	1	2	3	2
	Locating	3	2	2	3	2	2
	Task-flow	4	2	1	3	4	3
	Understanding	4	2	4	2	6	3
	Transition	1	1	1	1	1	1

Table 3 .

 3 Usability defects weight per tool.

				Knack			Mendix		Outsystems
	Category		G1			G2		G1		G2		G1		G2
		∑		%		∑	%	∑	%	∑	%	∑	%	∑	%
	Bars/Windows		6		8	3	5	13	13	8	12	22	16	9	12
	Icons		3		4	5	8	21	21	9	13	27	20	10	14
	Interaction		2		3	22	37	12	12	5	7	27	20	18	24
	Menu/Commands	5	2	2	2	14	24	32	32	27	40	20	14	10	14
	Text/Feedback	0	2	6	3	15	25	21	21	18	27	25	18	20	27
	Total	3	7	0		59		99		67		13		74
		7										8

Table 4 .

 4 Relative impact of each usability defect category.

		Category	G1	G2	Knack	Mendix	Outsystems	Total
		Icons	1	1	1	1	1	1
		Interaction	2	2	5	1	1	2
		Menus/Command	2	3	4	3	1	2
	s	Text/Feedback	2	3	4	2	1	2
		Bars/Windows	1	1	1	1	1	1

Table 5 .

 5 Performance (aggregated time spent) in hours.

	Task		Knack			Mendix			Outsystems
		G1		G2		G1		G2		G1		G2
		∑	%	∑	%	∑	%	∑	%	∑	%	∑	%
	T1 T2	0.81 1.08	17 23	0.55 0.85	14 22	1.08 2.56	16 39	0.55 2.05	12 43	0.67 2.49	10 39	0.66 2.01	13 38
	T3	0.44	9	0.39	10	0.78	12	0.43	9	2.18	34	1.67	32
	T4	1.21	26	1.00	26	0.53	8	0.44	9	0.69	11	0.47	9
	T5	1.16	25	1.08	28	1.68	25	1.26	27	0.40	6	0.46	9
	Total	4.70		3.86		6.64		4.72		6.42		5.27

Table 6 .

 6 Overall threshold and ceiling classification.

	Threshold	Very	Low	Regular	Ceiling High	Very High
	Very High	Low
	High
	Regular					Mendix/Outsystems
	Low		Knack
	Very Low

http://semat.org

https://modelseverywhere.wordpress.com/2013/02/15/one-percent-software-professionals-inadvanced-countries/