Johannes Feiner
email: johannes.feiner@fh-joanneum.at

Keith Andrews
email: kandrews@iicm.edu

Usability Reporting with UsabML

Keywords: formative evaluation, usability findings, exchange, XML, reporting format

de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Formative usability studies form an integral part of iterative software design and development. Heuristic evaluations or thinking aloud tests, for example, can help identify problems in a user interface. Usability practitioners typically compile the findings and deliver them as a written report, say in PDF format. This paper discusses the UsabML format and its applications to usability reporting, particularly with regard to reuse of usability data during software development.

The are many examples of written report styles for formative tests, for example Molich et al. [START_REF] Molich | Rent a Car in Just 0, 60, 240 or 1,217 Seconds? -Comparative Usability Measurement, CUE-8[END_REF] and the NIST IUSR Formative Project [START_REF]NIST: Common Industry Format (CIF) IUSR Formative Project[END_REF]. There is even an ISO standard [START_REF]Software Engineering -Software Product Quality Requirements and Evaluation (SQuaRE) -Common Industry Format (CIF) for Usability Test Reports[END_REF] for written reports of summative tests (formal experiments). However, results on paper or in written reports are hard to reuse (see Figure 1). It is difficult to import the findings into other systems, such as bug-tracking systems used by software engineers. In the age of agile development, fast feedback cycles are essential. The ability to automatically import usability findings into the issue-tracking systems associated with software code repositories would be extremely valuable.

Usability Reports printed on paper

The definition of the UsabML usability reporting exchange format is an important step towards formalising the reporting of usability results and will greatly simplify the handover of such findings to software development teams.

The UsabML Markup Language

For exchange of data, the extensible markup language (XML) is convenient, because XML can be readily parsed with standard tools and is extensible to further functionality without breaking existing code. Formulating UsabML in XML guarantees not only well-formed XML documents, but allows documents to be validated against the XSD schema provided. A tree hierarchy of tags defines the sections, subsections, and details for the various parts of typical usability evaluation reports. Tags hold their main information as XML text nodes. Tag attributes are used only for computer generated information or metainformation, such as unique identifiers, references to other tags, and sort-order.

? xml version=" 1 . 0 " encoding="UTF-8"?> <x s : s c h e m a x m l n s : x s=" h t t p : //www. w3 . o r g /2001/XMLSchema" a t t r i b u t e F o r m D e f a u l t=" q u a l i f i e d "> <x s : e l e m e n t name=" p r o j e c t "> <xs:complexType> <x s : s e q u e n c e> <x s : e l e m e n t name=" t i t l e " /> <x s : e l e m e n t name=" d e s c r i p t i o n " /> <x s : s e q u e n c e i d=" r e p o r t s " maxOccurs=" unbounded "> <x s : e l e m e n t r e f=" r e p o r t " /> </ x s : s e q u e n c e> </ x s : s e q u e n c e> . . . Separate UsabML schemas are defined for the two different kinds of reports currently supported: thinking aloud (TA) test and heuristic evaluation (HE). The overall outline of UsabML is shown in Figure 2, where the internal structures of heuristic evaluation and thinking aloud test reports are listed side-by-side, so differences within the schemas become evident. Within a schema, only very few sections are marked as required to give users of the UsabML standard more freedom to omit parts they do not require. The detailed schemas can be found online at the project web site [START_REF] Feiner | UsabML: The Usability Markup Language[END_REF]. Roughly speaking, the structure shown in Listing 1.1 is enforced by each schema. In particular:

-The root xml tag project (see Listing 1.1) holds general information (title, description) about a software project and one or more usability reports. -The report xml tag specifies the kind of evaluation (for example "TA" or "HE"), further meta-information ("generated" timestamp), and the main report contents. This includes first the title, date/time of the report, author(s), description and summary, a general introduction and a description of the methodology. Some of these tags may hold pre-formatted text, which is used later as introductory text in the generated PDF or HTML reports.

For a HE report, the heuristics, evaluators their environment come next, followed by the heuristicissues:

-A heuristic section provides a fixed list of (about ten) heuristics. Many practitioners use those suggested by Jakob Nielsen [START_REF] Nielsen | Ten Usability Heuristics[END_REF]. More specialised sets of heuristics are sometimes used when evaluting specific kinds of interface, such as mobile devices or information dashboards. -The heuristicissues section stores the positive and negative findings of the HE evaluators. Detailed information about findings including their severity and steps to reproduce them can be entered.

For a TA report, the users, test environment, tasks and questionnaire sections follow the introductory part. The findings are available in detail in the sections named taskresults and questionnaireresults:

-The users section holds several TA test persons with their profile (gender, education, itexperience and so forth). Information about test users is connected to the video clip(s) of their thinking aloud test. -An environment section allows the specification of the environment used for the TA test, including the location and any hardware (computers, monitors, cameras) and software (screen capture) used. -The section tasks denotes the tasks prepared in advance by the test team for the test users. A task requires a title, description, and prerequisites as well as possilbesolutionpath, endingcriteria, and scheduledduration. -Details of the element taskresult are start, end, actualduration, and percent of completion. -A questionnaire represents a list of questions and information about any rating scale (Very easy -Very hard) used. The answers to the questions are normalised values (say, points from 0 to 6 for a 7-point rating scale) stored for each answer per user.

-For each user, an interview and a test transcript may optionally be provided.

The final part of a report are the discussion section, which contains an interpretation of the evaluation results, and an appendix:

-The discussion section holds information about findings, including a list of recommendations. For a TA as well as for a HE report, there is the possibility to store details like video clips and codereferences. -In the appendix the material used is referenced. URLs to online resources are given.

Several tags, for example heuristicissues, findings, or transcript-logs, are designed to hold one or more (codereference) items, indicating a source code location (say, a potential origin of a bug).

With all this detailed data at hand, it is easy to automatically generate both comprehensive and aesthetically pleasing reports, in say HTML or PDF formats. Note, that not all elements hold numeric information such as numbers and dates, many hold descriptive free text passages. Reports are generated by applying a stylesheet (XSL) transformation to the UsabML data. Different XSL stylesheets might be provided to generate highly customised reports for different target groups. For example, managers might generate just a short report with executive summary and a main overview. Programmers might export prioritised bug lists. Other stylesheets allow the generation of comma-separated values (CSV) files of selected data for later statistical analysis. Stylesheets to generate HTML are available at the project web site [START_REF] Feiner | UsabML: The Usability Markup Language[END_REF].

Usability Reporting Manager

To create a report in UsabML, evaluation managers could hand-edit XML directly. A more comfortable alternative is to use web-based software called the Usability Reporting Manager (URM). URM provides a web interface to enter, manage and export data in UsabML and is shown in Figure 3. Optionally, the exported reports can be styled with stylesheets for rendering in browsers as HTML.

The main web page is shown in Figure 4a and the details for entering information about recommendations can be seen in Figure 4b. The equivalent information rendered in HTML can be seen in Figure 4c. Further formats can be created simply by adding further style sheets (XSL files).

The URM software is open source and can be downloaded from the GIT repository [START_REF] Feiner | github -Usability Reporting Manager[END_REF]. More detailed specifications and documentation is available from the project web site [START_REF] Feiner | URM -Usability Reporting Manager[END_REF].

Reuse and Exchange of Usability Findings

UsabML is great for reuse and exchange, because it is standardised XML and can be processed accordingly. It is easy to parse the files and data points can be extracted automatically at any given time. It is possible, for example, to imagine tools to compare evaluation results from different studies, like a diff on two data sets in respect of number of findings or severity of findings. This would allow a more automated approach to running studies like Rolf Molich's Comparative Usability Evaluation series [START_REF] Molich | CUE -Comparative Usability Evaluation[END_REF].

UsabML supports the exchange of information between different domains. Usability testers can hand over their findings to software engineers, who then integrate the findings automatically into their bug tracking system. That is a huge improvement over reports on paper, where bugs must be extracted and entered into bug tracking systems manually. We are currently working on import tools (see Figure 5) for well-known open source bug tracking systems such as Trac, Bugzilla, and Mantis. Automated import can be accomplished by con-verter scripts which parse UsabML and submit new issues via XML-RPC (Trac, Bugzilla), ReST (Bugzilla), or a SOAP web service (Mantis) as appropriate. The finding (problem report) shown in Figure 4c is converted into a corresponding Trac ticket in Figure 6.

Related Work

The importance of usability evaluation in general was promoted by Nielsen [START_REF] Nielsen | Usability Engineering[END_REF]. The use of usability evaluation to increase the quality of software is discussed in many papers such as Komiyama or Law [START_REF] Komiyama | Usability Evaluation Based on International Standards for Software Quality Evaluation[END_REF][START_REF] Law | Maturing Usability: Quality in Software, Interaction and Value[END_REF]. Many books [START_REF] Barnum | Usability Testing Essentials: Ready[END_REF][START_REF] Krug | Don't Make Me Think!: A Common Sense Approach to Web Usability[END_REF][START_REF] Rubin | Handbook of Usability Testing: Howto Plan, Design, and Conduct Effective Tests[END_REF][START_REF] Stone | User Interface Design And Evaluation[END_REF] advocate the use of usability methods. Howarth et al [START_REF] Howarth | Supporting Novice Usability Practitioners With Usability Engineering Tools[END_REF] discuss the need for tool support, especially for novice usability practitioners.

A collection of the most common usability evaluation methods is described in detail in Andrews [START_REF] Andrews | Evaluation Comes in Many Guises[END_REF]. For formative usability evaluation, the two classical methods are heuristic evaluation (HE) and thinking aloud (TA) testing. Both are supported by UsabML to manage and generate reports. A standard written report structure for reporting the results of summative evaluations (formal experiments) was developed by NIST as the Common Industry Format (CIF) [START_REF]NIST: Common Industry Format for Usability Test Reports[END_REF] and is now an ISO standard [START_REF]Software Engineering -Software Product Quality Requirements and Evaluation (SQuaRE) -Common Industry Format (CIF) for Usability Test Reports[END_REF]. A similar effort to standardise written reports of formative evaluations was started under the name IUSR Formative Project [START_REF]NIST: Common Industry Format (CIF) IUSR Formative Project[END_REF], but has yet to produce a draft standard (Theofanos [19]).

The Extended Structured Problem Report Format (ESPRF) for capturing problem reports during a usability inspection (heuristic evaluation) was introduced by Cockton et al. [START_REF] Cockton | Reconditioned Merchandise: Extended Structured Report Formats in Usability Inspection[END_REF]. However, each ESPRF covers the reporting of a single problem rather than an entire structured report, and furthermore is limited to the case of heuristic evaluation.

The general idea of a markup language for usability findings to promote their electronic reuse was proposed in previous work by Feiner, Andrews and Krajnc [START_REF] Feiner | UsabML -The Usability Report Markup Language[END_REF].

As shown with the Usability Reporting Manager URM, evaluation managers can automate parts of their workflows using software tools. Loitzl [START_REF] Loitzl | The Heuristic Evaluation Manager (HEM)[END_REF] presents a web based software tool for data collection and reporting when performing heuristic evaluations.

UsabML generated out of URM focuses on formative evaluation and differs therefore from Wilson et al. [START_REF] Wilson | The Whiteboard: Tracking Usability Issues: To Bug Or Not To Bug?[END_REF] and Spacco et al. [START_REF] Spacco | Tracking Defect Warnings Across Versions[END_REF] who view issues more or less as standard software bugs.

Wilson et al. compare the approach of storing usability issues directly in the standard bug tracking system with the technique of creating a dedicated usability issues database. With URM we provide a standalone usability issues database with the possibility to export into bug databases, hence both approaches are supported.

Spacco implemented the idea of comparing issues from one software version to another by creating extensions for code analysis tools. In contrast, the UsabML format prepares for comparison between versions of usability evaluation data in a tool agnostic way, requiring XML comparison feature only.

Concluding Remarks

UsabML defines a new standard file format for formalised usability reporting. The XML format allows easy reuse of formative evaluation reports. It provides advantages for structuring a report and for transforming usability reports into different formats. Furthermore, it has the built-in possibility to store crossreferences from usability findings to corresponding source code locations (these cross-references must be established manually by project managers). UsabML has the potential to bridge part of the gap between usability specialists and software developers and improve overall software quality.

With the URM web application, UsabML can be created on a server by entering data through the web. The information is stored in a central database and can be modified or appended any time. On demand, one or several reports in different formats can be generated. So, in future, a cycle of creation and reuse is feasible. The electronic handover (see Figure 5) of usability findings to software engineers is now becoming possible.

Although UsabML as described currently only supports the generation of HE and TA reports, it can of course in future be extended analogously to cover other forms of formative evaluation, such as cognitive walkthrough.

UsabML and URM will be evaluated in the near future during an upcoming project with industry partners. A case study will demonstrate the potential advantages of reusing usability data for development of real world software applications.

The UsabML specification, schema files, and example reports are available at the project web site [START_REF] Feiner | UsabML: The Usability Markup Language[END_REF].

Fig. 1 .

 1 Fig.1. Life-cycle of evaluation reports: The information exported via UsabML can be processed by other systems. For example, software engineers can import findings into a bug-tracking system.

Listing 1 . 1 .

 11 Top elements of the schema definition file hold information about the project and one or more reports.

Fig. 2 .

 2 Fig. 2. The internal structure of heuristic evaluation and thinking aloud test reports.

Fig. 3 .

 3 Fig. 3. URM enables evaluation managers to enter reporting data via a web interface.

Fig. 4 .Fig. 5 .

 45 Fig. 4. (a) URM allows management of reports in a single, interactive AJAX and HTML5 enabled page. (b) Multimedia information, for example recommendations with screenshots and videos, can be entered. (c) The browser renders XML with selected XSL for pretty printing.

Fig. 6 .

 6 Fig.6. Automated import into bug tracking systems becomes possible with scripts using web services to create new tickets out of UsabML.


Derek Flood
email: [derek.flood@brookes.ac.uk

Rachel Harrison
email: rachel.harrison@brookes.ac.uk

Claudia Iacob
email: iacob]@brookes.ac.uk

Lessons Learned from Evaluating the Usability of Mobile Spreadsheet Applications

Keywords:

It is estimated that 90% of all the analysts in business perform calculations on spreadsheets. Due to advances in technology, spreadsheet applications can now be used on mobile devices and several such applications are available for platforms such as Android and iOS. Research on spreadsheets revolves around several themes, but little work has been done in evaluating the usability of such applications (desktop or mobile). This paper presents lessons learned and usability guidelines derived from laboratory usability testing of mobile spreadsheet applications. Twelve participants were given a task to be solved using a mobile spreadsheet application and based on the video recordings of their interactions with the application patterns of recurring actions and sequences of actions were derived. Navigation, selection, feedback, and transparency of features were some of the main themes in the results of the testing, pointing to a set of guidelines which are also generalizable across other types of mobile applications.

Introduction

Spreadsheets are widely used for tasks such as inventory administration, educational applications, scientific modeling, financial systems, etc. [START_REF] Hermans | Supporting professional spreadsheet users by generating leveled dataflow diagrams[END_REF]. It is estimated that 90% of all analysts in business perform calculations on spreadsheets [START_REF] Winston | Executive education opportunities[END_REF], the financial business being the domain where spreadsheets are most used. Due to advances in technology spreadsheet applications may be used on mobile devices, several such applications being available for platforms such as Androids or iOS. A recent study has shown that 79% of the participants (all recruited through the European Spreadsheet Risk Interest Group) required access to a spreadsheet while away from a desktop computer, mostly in the following contexts: a) daily commute, b) demonstrating data to clients, c) inbound email onto a mobile device, d) discussing urgent changes with coworkers [START_REF] Flood | Spreadsheets on the move: An evaluation of mobile spreadsheets[END_REF]. However, the majority of the participants used mobile spreadsheet applications less than once a month, mostly because of the usability challenges such applications bring.

Research on spreadsheets has focused on various aspects, including expanding spreadsheet language through direct manipulation and gestures [START_REF] Burnett | Graphical definitions: expanding spreadsheet languages through direct manipulation and gestures[END_REF], communicating unit error messages in spreadsheets [START_REF] Abraham | How to communicate unit error messages in spreadsheets[END_REF], type inferences for spreadsheets [START_REF] Abraham | Inferring templates from spreadsheets[END_REF], graphbased visualizations [START_REF] Kankuzi | An end-user oriented graph-based visualization for spreadsheets[END_REF], testing [START_REF] Rothermel | A methodology for testing spreadsheets[END_REF], and web based spreadsheet-mediated collaboration [START_REF] Ginige | User requirements for a web based spreadsheet-mediated collaboration[END_REF]. However, little work has been done in identifying the usability issues revolving around spreadsheet applications (mobile or desktop-based) and ways to go about resolving them. To address this gap, this paper describes the results of a series of human-centered laboratory usability tests performed on mobile spreadsheet applications, pointing to some of the lessons learned and guidelines applicable to the mobile context design.

Related Work

Zhang and Adipat highlighted a number of issues that affect the usability of mobile applications, such as the mobile context, small screen size and different display resolutions, limited processing capability and power and tedious data entry methods [START_REF] Zhang | Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications[END_REF]. Efforts have been made to look into some of these issues, focusing mostly on mobile web browsing [START_REF] Geven | Depth and breadth away from the desktop: the optimal information hierarchy for mobile use[END_REF][START_REF] Schmiedl | Mobile phone web browsing: a study on usage and usability of the mobile web[END_REF][START_REF] Shrestha | Mobile web browsing: usability study[END_REF] and mobile guides [START_REF] Kjeldskov | Evaluating the usability of a mobile guide: the influence of location, participants and resources[END_REF]. Schmield et al. describe the results of a multidimensional study which investigates usage scenarios as well as the usability of mobile tailored websites, answering questions such as: what kinds of websites are most often accessed using a mobile phone and who are the users etc. [START_REF] Schmiedl | Mobile phone web browsing: a study on usage and usability of the mobile web[END_REF]. The findings show (for example) that the typical mobile phone user is male, between 20 and 29, and of the mobile tailored sites investigated, 55% fell in the category 'Information services', while 20% were social networking sites.

A discussion on the optimal information hierarchy for mobile use and the usability effects for reduced screen size is presented in [START_REF] Geven | Depth and breadth away from the desktop: the optimal information hierarchy for mobile use[END_REF]. Slow reading speed, poor comprehension and poor information retrieval performance are just a few of the usability effects reduced screen size brings [START_REF] Jones | Improving Web Interaction on Small Displays[END_REF]. Ways of dealing with such issues include adapting the way information is structured in hierarchies [START_REF] Geven | Depth and breadth away from the desktop: the optimal information hierarchy for mobile use[END_REF]. A comparative study of four hierarchies on three different devices found that users preferred narrower hierarchies on all the devices. In [START_REF] Burigat | diagram, and web page navigation on mobile devices: the effectiveness of zoomable user interfaces with overviews[END_REF], a comparative study of zoomable UIs (with and without overviews) used in the context of mobile maps, diagrams, and web pages shows that for zoomable UI overviews the "effectiveness in search tasks is highly dependent on the type of information overviews can provide and on the structure of the considered space". For example, applications involving maps with dynamic content could benefit from zoomable UIs with overviews. The results of a usability study in which users' mobile web browsing experience was evaluated in comparison to desktop web browsing have also been described [START_REF] Shrestha | Mobile web browsing: usability study[END_REF]. All the participants managed to complete all the tasks given to them on the desktop, but only one participant achieved this on the mobile device. Moreover, 80% of the time was dedicated to completing the tasks on the mobile device. Also subjects found it more difficult to correct their mistakes when using the mobile device. Overall, even if the subjects found the changes of the usual page structure on the mobile device easy to understand and learn, they experienced difficulties while interacting with it and this affected their satisfaction.

For evaluating the usability of mobile spreadsheet applications, laboratory usability tests were run with 12 participants, whose actions were video recorded, coded, and further analyzed. The participants were given the task of editing nine cells and locating seven pieces of information located on a given spreadsheet using an iPhone device (which had the Spreadsheet app installed, the most currently used and representative spreadsheet mobile app). Out of the 12 participants, 75% of the participants rated their level of experience with spreadsheets as being intermediate. Only 1 participant (8.33%) rated their level of experience as being novice. As far as experience with the iPhone was concerned, 60% of participants rated themselves as either intermediate or expert. With the exception of one participant, the remaining participants all had experience on other smart phone devices. In terms of age, the majority of participants (42%) were between the age of 21 and 30. During the laboratory tests, the actions of the participants were video recorded while using the mobile spreadsheet application. These recordings were coded for analysis by associating each action with one of the codes depicted in Figure 1. The most often performed action while using the mobile spreadsheet application was scrolling from left to right. This may be explained by two points: 1) Zooming out was performed infrequently (less than 2% of the video fragments being assigned this code) which led to the need to navigate within a spreadsheet using scrolling and 2) There was very little feedback from the application on the participants' actions, so the participants were at times uncertain as to which part of the spreadsheet they were viewing and how this part was related to the rest of the spreadsheet. Navigation was common also between screens, the participants switching from one spreadsheet to another quite frequently.

Figure 1 highlights the high percentage of actions coded with "sameCellSelect" (i.e. the user clicks on the same cell they clicked on during the previous action) and "wrongCellSelect" (i.e. the user clicks on a different cell than the one they intended to click on). The discussion would also include "multipleCellSelect" (i.e. the user clicks on multiple cells at once by mistake) even if the percentage for this is much lower. What often happened was that the participants would aim to select one cell, but they were either selecting another cell close to the one desired or they were not getting any feedback on the action so they tried to select the same cell again to ensure the cell was selected. The percentage of actions coded with "newCellSelect" (i.e. the user clicks on the right cell at a first attempt) was just a little bit higher than "sameCellSelect". Particularly important, "actionFeedback" would mark the times when the participants perceived and were supported by feedback from the application on their interactions with it (e.g. trying to close one spreadsheet, they would be asked for confirmation). However, such occurrences were rare, the distribution of such actions being the lowest. Even so, the participants (with few exceptions) did not give up the task or parts of it.

Figure 1 -Action codes distribution for the lab tests "Whatsthis" was used to code actions during which the participants were puzzled by the answer of the application to their actions. Apart for this, the system provided very little feedback, and lacked support for issues such as: a) was the action successful (e.g. Is the background color set?), b) what are the consequences of an action, especially if it was made by mistake (e.g. Selecting the wrong cell), c) where is the cursor located, etc. There was very little use of features such as fisheye, zoom in and zoom out.

In addition to analyzing the frequency of each individual action, we also analyzed the common sequences of actions (Figure 2). Some recurring sequences of actions participants performed bring to light potential issues with the application itself. The act of Saving changes indicates that the participant successfully made the desired modifications. In more than a half of the cases, this led to navigating the spreadsheet further on. Navigation was mostly performed through scrolling, zooming actions being performed rarely. In 37.9% of the cases, scrolling from left to right led to more scrolling. This is partially explained by the fact that most of the time a scroll to the left would immediately be followed by a scroll to the right and vice versa. Similarly, scrolling up and down led to more scrolling. Also, not surprisingly, switching to another screen mostly led to navigating within the screen through scrolling actions.

Figure 2 -Sequence code distribution for the lab tests "SameCellSelect" led to the same action in 66.9% of the cases, showing that selecting the same cell over and over again was common. Due to lack of feedback from the application, the participants would select the same cell several times in a row. The maximum repetition of this action was 9 times. Right after selecting a wrong cell (different from the one desired), the participants would select the one desired in more than half of the cases. However, 17.9% percent of the cases point to the situation where the same cell (the wrong one) was selected. Selecting an option (e.g. selecting the background color for a cell) led to saving the modifications in 73.3% of the cases. However, at times, selecting an option would lead to reselecting that particular option. This was partially because the application would provide no feedback leading to confusion on the completion of the action. As an answer to that, the participants would at times try to repeat the action in order to check or validate that the action had been performed. 75% of the cases when the participants intended to close the application were followed by a cancelation of the action. This was particularly interesting since in most cases it was not clear to the participants that the close button would close the whole application and not just the open spreadsheet. Only when asked whether they want to save the work (as result of closing), did it become clear that the close button closes the application entirely. In the majority of cases, the participants would hide the keyboard right after typing some characters because otherwise the keyboard would impede the navigation within the spreadsheet. Preceding "whatsthis" coded actions were: a) zooming actions (the participants were uncertain of what caused the zooming and how to go back to the previous state) and b) canceling an action (without any feedback the participants sometimes became confused).

To determine the level of satisfaction participants had, they were asked to complete a SUS questionnaire, the average overall SUS score for the spreadsheet app being 50.42%. In addition to the low score, when asked if they would use the mobile spreadsheet application again, only half of the participants said that they would.

Lessons Learned, Guidelines, and Discussion

Based on direct observation and on the results drawn from the study, this section describes a set of 4 guidelines and lessons learned.

Make navigation features obvious.

The laboratory tests showed that features such as fisheye, or zooming in and/or out were rarely used, leading to poor navigation and overall awareness. Most of the participants were not even aware of the existence of these features. Such features should be clearly signaled such that the user is aware of the full potential of the application.

2. Provide feedback. Feedback supports users in understanding their actions and the impact of their actions in relation to the application. During the study, five types of needed feedback became evident. a) Validation feedback. Once the user performs an action on the application, it is helpful for them to understand whether that was a correct action with respect to the interaction context they are in. For example, selecting a spreadsheet cell should be accompanied by feedback on whether the cell was the right cell or not. b) Location feedback. Due to the small size of mobile devices, it is easy for users to get disoriented with respect to the cursor's location on the screen or to the position of the part they are visualizing in the overall document For such disorientation to be avoided, it is important for the user to be provided with feedback on the location of what they are currently viewing in the overall picture. c) Neighborhood feedback. Participants needed to navigate through the document for extensive periods of time also to become aware of the data contained by the cells surrounding those visible on the screen. Feedback on such information would reduce the users' navigation overload. d) Selection feedback. One of the main issues observed during the study was that the participants were not always sure that the cell they were trying to select was indeed selected. In answer to that, they kept selecting the same cell repeatedly. In part, this problem could be solved by providing feedback once a cell is selected by highlighting its margins or bringing the cell to the front. Similar techniques could be used for applications which use other types of selections. e) Keyboard feedback. The location of the keyboard was not always clear to the participants. At times, they struggled to understand how to hide the keyboard or, on the contrary, how to make it accessible. The application should support the user by providing feedback on the current state of the keyboard at all times.

3. Hints as to the consequences of each action. Users might be unaware of the consequences of some actions and realize only after the action has been performed what it led to. Even worse, in some cases users might not perceive the answer of the application to their action or fail to understand it. Hints on what each action led to would help users. As a concrete example, the color changing of a cell in the spreadsheet application could be modeled as a wizard where all the steps are specified a priori and the user is aware of what the next step is.

Provide evident milestones.

At times during the study, participants faced difficulties in trying to cancel an action by clicking mistakenly on the button closing the application. It was not always clear how to save, cancel or quit the application and this affected their interaction with the application. Such features should be made evident and distinct, so that at any time during his/her interaction with the application the user is aware of how such features work and how they are different.

Conclusions

In this paper, we looked at the results obtained through a pilot study addressing the usability evaluation of mobile spreadsheet applications. We learned that several types of feedback are particularly important when interacting with a mobile application, some of them being feedback on the state and location of the keyboard, feedback on the location of the cursor and on the currently viewed part of the overall document. Also, navigation features should be made evident since only experienced users made any use of zooming or fisheye features. Functions such as saving the work done, closing the application, or canceling an action should be clear and the difference should be made clear to the users. Our results will indicate to mobile spreadsheet application designers some of the difficulties and challenges users face while using such applications.


Emilia Mendes
email: emilia.mendes@bth.se

Improving Software Effort Estimation Using an Expert-Centred Approach

Keywords: Software Effort Estimation, Expert-centred Approach, Process Improvement, Cost Estimation, Project Management

A cornerstone of software project management is effort estimation, the process by which effort is forecasted and used as basis to predict costs and allocate resources effectively, so enabling projects to be delivered on time and within budget. Effort estimation is a very complex domain where the relationship between factors is non-deterministic and has an inherently uncertain nature, and where corresponding decisions and predictions require reasoning with uncertainty. Most studies in this field, however, have to date investigated ways to improve software effort estimation by proposing and comparing techniques to build effort prediction models where such models are built solely from data on past software projects -data-driven models. The drawback with such approach is threefold: first, it ignores the explicit inclusion of uncertainty, which is inherent to the effort estimation domain, into such models; second, it ignores the explicit representation of causal relationships between factors; third, it relies solely on the variables being part of the dataset used for model building, under the assumption that those variables represent the fundamental factors within the context of software effort prediction. Recently, as part of a New Zealand and later on Brazilian government-funded projects, we investigated the use of an expert-centred approach in combination with a technique that enables the explicit inclusion of uncertainty and causal relationships as means to improve software effort estimation. This paper will first provide an overview of the effort estimation process, followed by the discussion of how an expert-centred approach to improving such process can be advantageous to software companies. In addition, we also detail our experience building and validating six different expert-based effort estimation models for ICT companies in New Zealand and Brazil. Post-mortem interviews with the participating companies showed that they found the entire process extremely beneficial and worthwhile, and that all the models created remained in use by those companies. Finally, the methodology focus of this paper, which focuses on expert knowledge elicitation and participation, can be employed not only to improve a software effort estimation process, but also to improve other project management-related activities.

Introduction

The purpose of estimating effort is to predict the amount of effort (person/time) required to develop an application (and possibly also a service within the Web context), often based on knowledge of 'similar' applications/services previously developed. The accuracy of an effort estimate can affect significantly whether projects will be delivered on time and within budget; therefore effort estimation is taken as one of the main foundations of a sound project management. However, because effort estimation is a complex domain where corresponding decisions and predictions require reasoning with uncertainty, there are countless examples of companies that underestimate effort. Jørgensen and Grimstad [START_REF] Jørgensen | Software Development Effort Estimation: Demystifying and Improving Expert Estimation[END_REF] reported that such estimation error can be of 30%-40% on average, thus leading to serious project management problems.

Fig. 1 provides a general overview of an effort estimation process [START_REF] Mendes | Cost Estimation Techniques for Web Projects[END_REF]. Estimated characteristics of the new application/service to be developed, and its context (project), are the input, and effort is the output we wish to predict. For example, a given software company may find that to predict the effort necessary to implement a new e-commerce Web application, it will need to estimate early on in the development project the following characteristics:

• Estimated number of new Web pages.

•

The number of functions/features (e.g. shopping cart, on-line forum) to be offered by the new Web application.

•

Total number of developers who will help develop the new Web application • Developers' average number of years of experience with the development tools employed.

•

The choice of main programming language used.

Of these variables, estimated number of new Web pages and the number of functions/features to be offered by the new Web application characterise the size of the new Web application; the other three, total number of developers who will help develop the new Web application, developers' average number of years of experience with the development tools employed, and main programming language used, characterise the project -the context for the development of the new application, and are also believed to influence the amount of effort necessary to develop this new application. The project-related characteristics are co-jointly named 'cost drivers'.

No matter what type of development it is (of an application or of a service), in general the one consistent input found to have the strongest effect on the amount of effort needed to develop an application or service is size (i.e. the total number of server side scripts, the total number of Web pages), with cost drivers also playing an influential role.

Evidence also shows that for most part of existing software & service projects, effort estimation is based on past experience, where knowledge or data from past finished applications & projects are used to estimate effort for new applications & projects not yet initiated [START_REF] Jørgensen | Software Development Effort Estimation: Demystifying and Improving Expert Estimation[END_REF]. The assumption here is that previous projects are similar to the new projects to be developed, and therefore knowledge and/or data from past projects can be useful in estimating effort for future projects. This process is also illustrated in Fig. 1. Those steps (some or all) can be executed more than once throughout a given software development cycle, depending on the process model adopted by the company. For example, if the process model adopted by a company complies with a waterfall model this means that most probably there will be an initial effort estimate for the project, which will remain unchanged throughout the project. If a company's process model complies with the spiral model, this means that for each cycle within the spiral process a new/updated effort estimate is obtained, and used to update the current project's plan and effort estimate. If a company uses an agile process model, an effort estimate is likely to be obtained for each of a project's iterations (e.g. sprints). In summary, a project's process model drives at what stage(s) an effort estimate(s) is/are obtained, and whether or not these estimates are revisited at some point throughout a project's development life cycle. [START_REF] Mendes | Cost Estimation Techniques for Web Projects[END_REF] Note that cost and effort are often used interchangeably within the context of most effort estimation literature since effort is taken as the main component of project costs. However, given that project costs also take into account other factors such as contingency and profit [START_REF] Kitchenham | Modelling Software Bidding Risks[END_REF] we will use the word "effort" and not "cost" throughout this paper.

Fig. 1. Effort Estimation process

The remaining of this paper is organised as follows: the next Section provides a motivation for using an expert-centred approach to improving software effort estimation, followed by another two Sections where first the approach we propose is briefly introduced and second explained in more detail using the experience from eliciting six different expert-centred real models. Finally, our last two Sections Estimated size Deriving an effort estimate

Estimated effort

Data and/or knowledge on past finished projects

Step 3

Step 4

Step 1

Estimated cost drivers

+

Step 2

Effort

Model Building discuss our experience building six expert-centred models to improve effort estimation, and lessons learnt, respectively.

Motivation to Employing an Expert-Centred Approach

Most research in software & Web effort estimation has to date focused on solving companies' inaccurate effort predictions via investigating techniques that are used to build formal effort estimation models, in the hope that such formalisation will improve the accuracy of estimates. They do so by assessing, and often also comparing, the prediction accuracy obtained from applying numerous statistical and artificial intelligence techniques to datasets of completed software/Web projects developed by industry, and sometimes also developed by students. Recent literature reviews of software and Web effort estimation studies are given respectively in [START_REF] Jørgensen | A Systematic Review of Software Development Cost Estimation Studies[END_REF] and [START_REF] Azhar | A Systematic Review of Web Resource Estimation[END_REF]. The variables characterising such datasets are determined in different ways, such as via surveys [START_REF] Mendes | Investigating Web Size Metrics for Early Web Cost Estimation[END_REF], interviews with experts [START_REF] Ruhe | Cost estimation for Web applications[END_REF], expertise from companies [START_REF] Ferrucci | A Case Study Using Web Objects and COSMIC for Effort Estimation of Web Applications[END_REF], a combination of research findings [START_REF] Mendes | Web metrics -Metrics for estimating effort to design and author Web applications[END_REF], or even a researcher's own consulting experience [START_REF] Reifer | Web Development: Estimating Quick-to-Market Software[END_REF]. In all of these instances, once variables are defined, a data gathering exercise takes place, obtaining data (ideally) from industrial projects volunteered by companies. Except when using research findings to inform variables' identification, invariably the mechanism employed to determine variables relies on experts' recalling, where the subjective measure of an expert's certainty is often their amount of experience estimating effort.

However, in addition to eliciting the important effort predictors (and optionally also their relationships), such mechanism does not provide the means to also quantify the uncertainty associated with these relationships and to validate the knowledge obtained. Why should these be important?

Our experience developing and validating single-company expert-centred Software and Web effort prediction models that incorporate the uncertainty inherent in this domain [START_REF] Mendes | Using Knowledge Elicitation to Improve Web Effort Estimation: Lessons from Six Industrial Case Studies[END_REF] showed that the use of an expert-centred structured iterative process in which factors and relationships are identified, quantified and validated [START_REF] Mendes | Knowledge Representation using Bayesian Networks A Case Study in Web Effort Estimation[END_REF][13] [START_REF] Mendes | Building an Expert-based Web Effort Estimation Model using Bayesian Networks[END_REF] leads the participating companies to a much more thorough and deep understanding of their mental processes and their decisions when estimating effort, when compared to just the recalling of factors and their relationships. The iterative process we use employs Bayesian inference, which is one of the techniques employed in root cause analysis [START_REF] Ammerman | The Root Cause Analysis Handbook: A Simplified Approach to Identifying, Correcting, and Reporting Workplace Errors[END_REF]; therefore it aims at a detailed analysis and understanding of a particular phenomenon of interest.

In all the case studies we conducted, the original set of factors and relationships initially elicited was always modified as the model evolved; this occurred as a result of applying a root cause analysis approach comprising a Bayesian inference mechanism and feedback into the analysis process via a model validation. In addition, post-mortem interviews with the participating companies showed that the understanding they gained by being actively engaged in building those models led to both improved estimates and estimation processes [11][12][13] [START_REF] Mendes | Building an Expert-based Web Effort Estimation Model using Bayesian Networks[END_REF].

We therefore contend that the use of an expert-centred structured iterative process provides the means to elicit a more robust set of predictors and relationships, when compared to other means of elicitation. We argue that the recalling mechanism used in surveys and interviews to elicit the important factors (and also occasionally their relationships) when estimating effort does not provide any means for experts to understand thoroughly their own decision making processes via the quantification of the uncertainty part of that decision process, and the validation of the factors they suggested during the elicitation. This means that the list of factors elicited is most likely based on a superficial process.

An Overview of the Expert-Centred Approach

Technique Used

The expert-centred approach proposed herein is based on a technique called Bayesian Networks (BN). This technique and corresponding process are briefly introduced in this Section, and further details are given in [START_REF] Mendes | Building an Expert-based Web Effort Estimation Model using Bayesian Networks[END_REF].

A BN is a model that enables the characterisation of a knowledge domain in terms of its factors, their relationships, and the uncertainty inherent to that domain. It has two parts [START_REF] Ammerman | The Root Cause Analysis Handbook: A Simplified Approach to Identifying, Correcting, and Reporting Workplace Errors[END_REF]. The first part, known as the BN's qualitative part, results in a graphical structure comprising the factors and causal relationships identified as fundamental in the domain being modelled. This structure is depicted by a Directed Acyclic Graph (DAG) (see Fig. 2(a)). In addition to identifying factors and relationships, this part also includes the identification of the states (values) that each factor should take (e.g. Small (1 to 5), Medium (6 to 15), or Large (16+) in Fig. 2(a)).

The second part, known as the quantitative part, represents the relationships identified in the qualitative part, and their quantification, done probabilistically. This quantification represents the uncertainty in the domain being modelled, and in order for it to be accomplished, a Conditional Probability Table (CPT) is associated to each node in the graph. A parent node's CPT describes the relative probability of each state (value) (Fig. 2(b) CPTs for nodes 'Total Number of Web pages' and 'Total Number of Images'); a child node's CPT describes the relative probability of each state conditional on every combination of states of its parents (Fig. 2(b) CPT for node 'Total Effort'). Each row in a CPT represents a conditional probability distribution and therefore its values sum up to one [START_REF] Ferrucci | A Case Study Using Web Objects and COSMIC for Effort Estimation of Web Applications[END_REF]. Such probabilities can be attained via expert elicitation, automatically from data, from existing literature, or using a combination of these. However, within the context of this research all probabilities were obtained via expert elicitation. Once both qualitative and quantitative parts are specified, the BN is validated using data on past finished projects, where one project at a time is entered as evidence (see Fig. 2(d)) and used to check whether the BN provides the highest probability to a value (range of values) that includes the real actual effort for that project, which is known. If not, then the BN is re-calibrated.

The building of a BN model is an iterative process where one can move between the three different steps of this process -building the BN's structure, or qualitative part, building the CPTs, or the quantitative part, and validating the model. Once a BN is validated (see Fig. 1(c), evidence (e.g. values) can be entered into any node, and probabilities for the remaining nodes automatically calculated using Bayes' rule [START_REF] Ferrucci | A Case Study Using Web Objects and COSMIC for Effort Estimation of Web Applications[END_REF] (see Figs. 2(d) and 1(e). This was the validation method employed herein. In summary, BNs can be used for different types of reasoning, such as predictive (see Fig. 2(d)), diagnostic (see Fig. 2(e)), and "what-if" analyses to investigate the impact that changes on some nodes have on others [START_REF] Ammerman | The Root Cause Analysis Handbook: A Simplified Approach to Identifying, Correcting, and Reporting Workplace Errors[END_REF].

Process We Employed to Building the Expert-Centred Models (BNs)

The process that was used to build and validate the expert-centred models focus of this research is an adaptation of the Knowledge Engineering of Bayesian Networks (KEBN) process proposed in [START_REF] Woodberry | Parameterising Bayesian Networks[END_REF] (see Fig. 3). As shown in Fig. 3, this process iterates over three steps -Structural Development, Parameter Estimation, and Model Validation, until a complete BN is built and validated. Each of the steps is detailed next:

Structural Development: This step represents the qualitative component of a BN, which results in a graphical structure comprised of, in our case, the factors (nodes, variables) and causal relationships identified as fundamental for effort estimation of software & Web projects. In addition to identifying variables, their types (e.g. query variable, evidence variable) and causal relationships, this step also comprises the identification of the states (values) that each variable should take, and if they are discrete or continuous. In practice, currently available BN tools require that continuous variables be discretised by converting them into multinomial variables, also the case with the BN software used in this study. The BN's structure is refined through an iterative process where existing literature in the field can also be used as input to the process. This structure construction process has been validated in previous studies (e.g. [START_REF] Woodberry | Parameterising Bayesian Networks[END_REF] [START_REF] Druzdzel | Building Probabilistic Networks: Where Do the Numbers Come From?[END_REF]) and uses the principles of problem solving employed in data modelling and software development [START_REF] Tang | Developing Complete Conditional Probability Tables from Fractional Data for Bayesian Belief Networks[END_REF]. Throughout this step the Kowledge Engineer (responsible for eliciting the knowledge from the Domain Expert(s) (Des)) also evaluates the structure of the BN, checking whether variables and their values have a clear meaning; all relevant variables have been included; variables are named conveniently; all states are appropriate (exhaustive and exclusive); a check for any states that can be combined. Once the BN structure is assumed to be close to final the KE may still need to optimise this structure to reduce the number of probabilities that need to be elicited or learnt for the network. If optimisation is needed, techniques that change the causal structure (e.g. divorcing [START_REF] Jensen | An introduction to Bayesian networks[END_REF]) are employed.

Parameter Estimation: This step represents the quantitative component of a BN, where conditional probabilities corresponding to the quantification of the relationships between variables [START_REF] Jensen | An introduction to Bayesian networks[END_REF] are obtained. Such probabilities can be attained via Expert Elicitation, automatically from data, from existing literature, or using a combination of these. When probabilities are elicited from scratch, or even if they only need to be revisited, this step can be very time consuming. In order to minimise the number of probabilities to be elicited some techniques have been proposed in the literature (e.g. [16][17]). Model Validation: This step validates the BN resulting from the two previous steps, and determines whether it is necessary to re-visit any of those steps. Two different validation methods are generally used -Model Walkthrough and Predictive Accuracy. Model walkthrough represents the use of real case scenarios that are prepared and used by DEs to assess if the predictions provided by the model correspond to the predictions experts would have chosen based on their own expertise. Success is measured as the frequency with which the model's predicted value for a target variable (e.g. quality, effort) that has the highest probability corresponds to the experts' own assessment.

Predictive Accuracy uses past data (e.g. past project data), rather than scenarios, to obtain predictions. Data (evidence) is entered on the model (see example in Fig. 2(d)), and success is measured as the frequency with which the model's predicted value for a target variable (e.g. quality, effort) showing the highest probability corresponds to the actual value from past data.

Detailing the Expert-Centred Approach

This Section revisits the adapted KEBN process (see Fig. 3), detailing the tasks carried out for each of the three main steps part of that process within the context of six expert-centred effort estimation models that were separately elicited by the author with the participation of Domain Experts (Des) from six different Companies (five in New Zealand and one in Brazil).

In all cases, prior to eliciting the expert-centred effort models, the DEs from all participating companies were presented with an overview of the technique that was going to be used, and examples of "what-if" scenarios, using a made-up BN model. This, we believe, facilitated the entire process as the use of an example, and the brief explanation of each of the steps in the KEBN process, provided a concrete understanding of what to expect. We also made it clear that the KE was a facilitator of the process, and that the companies' commitment was paramount for the success of the collaboration. The effort required by each company to have their Expert-centred models created and the characteristics of each model are detailed in Table 1. The DEs who took part in the case studies were all project managers of wellestablished companies in either Auckland (New Zealand), or Rio de Janeiro (Brazil), each with at least 10 years of experience in project management. These companies varied in their size, measured as the total number of employees. In addition, all six companies were consulting companies and as such, developed a wide range of applications, from conventional software (only company E), and static & multimedia-like to very large e-commerce solutions. All six companies employed a wide range of technologies, mostly focusing on the development of Web 1.0, 2.0 and Web 3.0 applications. Finally, when approached, they were all looking at improving their current effort estimates, and agreed to participate for two main reasons: i) because the models being created were expert-centred single-company models geared towards their specific needs; ii) and also because their expertise and participation were acknowledged as essential to eliciting those models.

Detailed Structural Development and Parameter Estimation: In order to identify the fundamental factors that the DEs took into account when preparing a project quote we used the set of variables from the Tukutuku dataset [START_REF] Mendes | Investigating Web Size Metrics for Early Web Cost Estimation[END_REF] as a starting point (see Table 2). We first sketched them out on a white board, each one inside an oval shape, and then explained what each one meant within the context of the Tukutuku project. Our previous experience eliciting expert-centred models in other domains (e.g. ecology, resource estimation) suggested that it was best to start with a few factors (even if they were not to be reused by the DE), rather than to use a "blank canvas" as a starting point [START_REF] Woodberry | Parameterising Bayesian Networks[END_REF].

Table 2. Tukutuku Variables

Variable Name

Description

Project Data

TypeProj

Type of project (new or enhancement). nLang

Number of different development languages used DocProc

If project followed defined and documented process.

ProImpr

If project team involved in a process improvement programme.

Metrics

If project team part of a software metrics programme.

DevTeam

Size of a project's development team.

TeamExp

Average team experience with the development language(s) employed. Once the Tukutuku variables had been sketched out and explained, the next step was to remove all variables that were not relevant for the DEs, followed by adding to the white board any additional variables (factors) suggested by them. This entire process was documented using digital voice recorders and also text editors. We also documented descriptions and rationale for each factor proposed by the DEs. The factors proposed were indeed influenced by DEs' hunches and insights; however DEs decisions and choices were also very much influenced by their solid previous experience managing Web projects, and estimating development effort.

Application

Next, we identified the possible states that each factor would take. All states were discrete. Whenever a factor represented a measure of effort (e.g. Total effort), we also documented the effort range corresponding to each state, to avoid any future ambiguity. For example, to one of the participating Web companies, 'very low' Total effort corresponded to 4+ to 10 person hours, etc. Once all states were identified and thoroughly documented, it was time to elicit the cause and effect relationships. As a starting point to this task we used a simple medical example from [START_REF] Jensen | An introduction to Bayesian networks[END_REF] (see Fig. 4).

Fig. 4. An example of a cause and effect relationship

This example clearly introduces one of the most important points to consider when identifying cause and effect relationships -timeline of events. If smoking is to be a cause of lung cancer, it is important that the cause precedes the effect. This may sound obvious with regard to the example used; however, it is our view that the use of this simple example significantly helped the DEs understand the notion of cause and effect, and how this related to Web effort estimation and the BNs being elicited. Once the cause and effect relationships were identified, we worked on the elicitation of probabilities to quantify each of the cause and effect relationships previously identified. In all four cases, there was an iterative process between the structural development and parameter elicitation steps.

Detailed Model Validation: Both Model walkthrough and Predictive accuracy were used to validate all six expert-centred models, where the former was the first type of validation to be employed in all cases. DEs used different scenarios to check whether the node Total_effort would provide the highest probability to the effort state that corresponded to the DE's own suggestion. However, it was also necessary to use data from past projects, for which total effort was known, in order to check the model's calibration. Table 1 details the number of projects used by each company as validation set. In all cases, DEs were asked to use as validation set a range of projects presenting different sizes and levels of complexity, and being representative of the types of projects developed by their companies.

For each project in a validation set, evidence was entered in the expert-centred model, and the effort range corresponding to the highest probability provided for 'Total Effort' was compared to that project's actual effort. Whenever actual effort did not fall within the effort range associated with the category with the highest probability, there was a mismatch; this meant that some probabilities needed to be adjusted. In order to know which nodes to target first we used a Sensitivity Analysis report, which provided the effect of each parent node upon a given query node. Within our context, the query node was 'Total Effort'. Whenever probabilities were adjusted, we re-entered the evidence for each of the projects in the validation set that had already been used in the validation step to ensure that the calibration already carried out had not been affected. This was done to ensure that each calibration would always be an improvement upon the previous one. Once all projects were used to calibrate a model, the DE(s) assumed that the Validation step was complete.

Each of the five New Zealand expert-centred models has been in production for at least 18 months, and the Brazilian model has been in production since May 2011. Due to shortage of space we cannot show the models; however, more details about these six expert-centred models are given in [START_REF] Mendes | Using Knowledge Elicitation to Improve Web Effort Estimation: Lessons from Six Industrial Case Studies[END_REF].

Further Gains from Eliciting Expert-Centred Models

Except for Company E, where the expert-centred model is also used to estimate effort for non-Web-based projects, all the five remaining models represent solely the knowledge elicited from domain experts relating to their previous experience estimating effort for Web development projects; therefore we believe that by aggregating their knowledge we can obtain a wider understanding on the fundamental factors affecting effort estimation (mainly Web effort estimation) and their causal relationships.

The type of aggregation mechanism we are suggesting herein applies solely to the qualitative parts of the expert-centred models elicited (factors and relationships only), as aggregating their quantitative parts and also the different categories used to measure each factor would prove to be a herculean (if not impossible) task; in addition, our goal is not to obtain a cross-company expert-centred model. Some of the advantages of such type of aggregation are as follows [START_REF] Baker | Aggregating Expert-driven Causal Maps for Web Effort Estimation[END_REF]:

• The aggregation from different experts of knowledge relating to the same phenomenon has the obvious advantage of providing an opportunity to amplify and broaden our overall understanding of that phenomenon. • The graph (map) resulting from the aggregation mechanism uses as input factors and relationships from models that were built and validated using a process based on a root cause analysis technique [START_REF] Ammerman | The Root Cause Analysis Handbook: A Simplified Approach to Identifying, Correcting, and Reporting Workplace Errors[END_REF]; such technique, by requiring a thorough and deep understanding of experts' mental processes and their decisions when estimating effort, provides the means to truly portray the phenomenon focus of this research, i.e. effort estimation. • The introduction of more structure into the effort estimation process, as such map can be used as a checklist to help improve judgment-based effort estimates [START_REF] Jørgensen | Software Development Effort Estimation: Demystifying and Improving Expert Estimation[END_REF]. • Anecdotal evidence we obtained throughout the elicitation process and postmortem meetings with several companies revealed that the use of a map aggregating companies' expert knowledge with regard to factors and relationships relevant for Web effort estimation would be extremely useful to help them elicit factors and relationships when building their own Web effort prediction models; therefore they would like to use such aggregated map at the start of their elicitation process. • Our aggregated map shows graphically not only the set of factors and relationships from the input models, but also a way to identify visually the most common factors and relationships resulting from the aggregation. Such knowledge may also be useful to project managers to revisit the factors they consider when estimating effort for new projects.

• The aggregated map can be used to provide companies with a starting point to building a single-company expert-based Web effort estimation model. This is also an approach suggested in [1][21].

Herein we will just present the main patterns observed from aggregating those six expert-centred models [START_REF] Mendes | Using Knowledge Elicitation to Improve Web Effort Estimation: Lessons from Six Industrial Case Studies[END_REF]; however, further details on the aggregation mechanism employed and an example of the sort of aggregated map we are focusing herein is given in [START_REF] Baker | Aggregating Expert-driven Causal Maps for Web Effort Estimation[END_REF].

The main patterns observed from aggregating the six expert-centred effort estimation models are presented below.

Apart from total effort, which was identified by all participating companies, there were three factors that were chosen by five of the six Companies:

• Average Project Team Experience with Technology Except for Project Risk Factor and Average Project Team Experience with Technology, all the remaining factors were related to the effort to accomplish certain tasks, such as adapting or developing a new feature, testing and interface design. Note that these factors are very much related to more dynamic Web applications, which offer a large set of features (this requiring more detailed testing). It is also interesting that the effort to develop the user interface was also chosen by four of the six companies. Nowadays, with the plethora of Web technologies and possibilities available, good interface design and usability can also add very much so to a company's competitive advantage on the global market.

All six expert-centred models presented several effort-related factors as predictors of Total effort. Note that these factors represent tasks that are part of an effort estimation process and therefore, their relationship with Total effort has an associative nature, however not of type cause&effect.

Lessons Learnt

The elicitation of expert-centred models, and their aggregation has provided numerous lessons, as follows:

First: engaging with industry. At the start of this research, in order to reach out to industry, we invited the local NZ IT industry to attend a seminar about software & Web effort estimation and how to improve their estimates. The seminar provided an introduction to using expert-centred models, their value as estimation tools, and their capability for running "what-if" scenarios. Many of the participating companies saw the immediate value in such an approach, in particular because it enabled the very close and fundamental participation of in-house domain experts while building and validating the company-specific model. Several companies sign up to collaborate.

Second: Time constraints. Depending on the complexity of the expert-centred model, the elicitation of probabilities can be very time consuming. Motivated by this issue, we have also made a preliminary attempt at investigating mechanisms to enable the automatic generation of probability tables. An attempt was devised and used with two NZ companies that participated more recently in this research. This solution comprised the comparison between different probability generation algorithms and expert-driven probabilities. A tool was implemented as a result of this work [START_REF] Baker | Evaluating the Weighted Sum Algorithm for Estimating Conditional Probabilities in Bayesian Networks[END_REF]; however, further work is needed to validate the proposed solution.

Third: Value for a company. Except for the company in Brazil, the other participating companies were contacted for post-mortem interviews. The main points highlighted were the following:

• The elicitation process enabled experts to think deeply about their effort estimation process and the factors taken into account during that process, which in itself was considered advantageous to the companies. This has been pointed out to us by all the DEs interviewed. • Once a company's expert-centred model was validated, DEs started to use their model not only for obtaining better estimates than the ones previously prepared by subjective means, but also sometimes as means to guide their requirements elicitation meetings with prospective clients. They focused their questions targeting at obtaining evidence to be entered in the model as the requirements meetings took place; by doing so they basically had effort estimates that were practically ready to use for costing the projects, even when meetings with clients had short durations. Such change in approach proved to be extremely beneficial to the companies given that all estimates provided using the models turned out to be more accurate on average than the ones previously obtained by subjective means. • Clients were not presented the models due to their complexity; however by entering evidence while a requirements elicitation meeting took place enabled the DEs to optimise their elicitation process by being focused and factor-driven. • One of the participating companies, the largest company in total number of employees, and also the one that built the largest BN model, provided the following feedback: The DEs who participated in the causal structure and probabilities' elicitation changed completely their approach to estimating effort as follows: they presented the BN model to all of their development teams, and asked that from that point onwards every estimate for any task should be based on the factors that had been elicited. This means that an entire team started to use the factors that have been elicited, as well as the BN model, as basis for their effort & risk estimation sessions. In addition, the DEs presented the model at a meeting with other company branches, so to detail how the Auckland branch was estimating effort and risk for their healthcare projects. The other branches were so impressed, in particular the one from the US, that they increased the number of Healthcare software projects outsourced to the NZ Branch, as they recognised the benefits of using a model that represented factors and uncertainties. Overall, such change in approach provided extremely beneficial to the company.

All the companies remained positive and very satisfied with the results. We believe that the successful development of these six expert-centred models was greatly influenced by the commitment of the participating companies, and also by the DEs' experience estimating effort.

Fig. 2 .

 2 Fig. 2. Parts of a Bayesian Network an Types of Reasoning

Fig. 3 .

 3 Fig.3. KEBN, adapted from[START_REF] Woodberry | Parameterising Bayesian Networks[END_REF]

Table 1 .

 1 Expert-Centred Models' Characteristics and DEs

		Companies in New Zealand		Co. Brazil
	Characteristics	A	B	C	D	E	F
	Number of DEs	1	1	2	2	7/2	1
	Number of Employees	~5	~5	~20	~30	~100	~30
	Number of 3-hours elicitation sessions	12	6	8	12	12/12	20
	Total hours to elicit & validate model	36	18	24	36	98	60
	Effort to elicit & validate model (person/hours)	72	36	72	108	324	120
	Number of factors	14	13	34	33	38	19
	Number of relationships	18	12	41	60	50	37
	Number of past projects used as validation set	22	8	11	22	22	9

  Within the context of the Tukutuku project, based on collected data, a new higheffort feature/function and a high-effort adapted feature/function require respectively at least 15 and 4 hours to be developed by one experienced developer.

	TotWP	Total number of Web pages (new and reused).
	NewWP	Total number of new Web pages.
	TotImg	Total number of images (new and reused).
	NewImg	Total number of new images created.
	Num_Fots	Number of features reused without any adaptation.
	HFotsA	Number of reused high-effort features/functions adapted.
	Hnew	Number of new high-effort features/functions.
	TotHigh	Total number of high-effort features/functions
	Num_FotsA Number of reused low-effort features adapted.
	New	Number of new low-effort features/functions.
	TotNHigh	Total number of low-effort features/functions

Acknowledgments. We thank all the Web companies who participated in this research. This work was sponsored by the Royal Society of New Zealand (Marsden research grant 06-UOA-201), and by CAPES/PVE (Brazil).