Enes Yigitbas

Ivan Jovanovikj

Stefan Sauer

On the Development of Context-aware Augmented Reality Applications

Keywords: augmented reality, context-awareness, model-based development

Augmented Reality (AR) is a technique that enables users to interact with their physical environment through the overlay of digital information. With the spread of AR applications in various domains (e.g. product design, manufacturing or maintenance) and the introduction of concepts such as Pervasive Augmented Reality (PAR), the aspect context-awareness started to play an important role. By sensing the user's current context and adapting the AR application accordingly, an adequate user experience can be achieved. Due to the complex structure and composition of AR applications, their development is a challenging task. Although, context-awareness for AR systems was addressed to some extent, a systematic method for development of context-aware AR applications is not fully covered yet. Therefore, in this paper, we identify the main challenges for development of context-aware AR applications and sketch our solution idea for a model-based development framework. The benefit and applicability of our solution idea is shown on the basis of two example mobile AR applications supporting context-awareness.

Introduction

Augmented Reality (AR) is a user interface metaphor, which allows for interweaving digital data with physical spaces. AR relies on the concept of overlaying digital data onto the physical world, typically in form of graphical augmentations in real-time [START_REF] Azuma | A survey of augmented reality[END_REF]. Augmented reality has been researched for a considerable amount of time, with first implementations as early as Sutherland's headmounted three dimensional display "The sword of Damocles" [START_REF] Sutherland | A head-mounted three dimensional display[END_REF] from 1966. The expression Augmented Reality was first coined by Tom Caudell in 1992 in his work on the "Application of Heads-Up Display Technology to Manual Manufacturing Processes" [START_REF] Caudell | Augmented reality: an application of heads-up display technology to manual manufacturing processes[END_REF]. In more recent years, AR technology is strongly on the rise, with many different devices available. One main technology are Head-Mounted-Displays (HMDs) like Microsoft's HoloLens 1 or the Magic Leap One 2 : Headsets with integrated display and optics. Some of them also have built in hardware to process the programs that run on the HMD, while other headsets need to be connected to a computer and only serve as a special kind of display which also includes control functions. An alternative way in AR-technology is to use a smartphone as the main hardware. The smartphone can be worn in a headgear ("Head-mounted Smartphone"), which is not very common for AR applications yet, as many of the headgears only support VR, for example because the phone-camera's lens is simply covered by the gear. More often smartphones are used in their original purpose, as handheld AR devices.

With the spread and increasing usage of Augmented Reality (AR) techniques in different domains, the need for context-awareness in AR was underlined in previous work [START_REF] Grubert | Towards pervasive augmented reality: Context-awareness in augmented reality[END_REF]. Supporting context-awareness, can greatly enhance user experience in AR applications, for example by adjusting to the individual needs of each user. It also makes the usage more intuitive and effective: The more the application can adjust to the user and his situation, the more natural the AR is experienced and the more ergonomic it is to work with. However, due to the complex structure (tasks, scenes) and composition (interrelations between real and virtual information objects) of AR applications [START_REF] Vitzthum | SSIML/AR: A visual language for the abstract specification of augmented reality user interfaces[END_REF], the development of context-aware AR applications is a challenging task. While context-aware AR applications were introduced for specific application domains, e.g. maintenance [START_REF] Zhu | A context-aware augmented reality assisted maintenance system[END_REF], a systematic method for supporting the efficient development of context-aware AR applications is not fully covered yet. Therefore, in this paper, we discuss the main challenges in developing context-aware AR applications and sketch a first solution idea for a model-based development framework for context-aware augmented reality applications.

The rest of the paper is structured as follows: In Section 2, we discuss main challenges in developing context-aware AR applications. In Section 3, we present architectural patterns as basic solution concepts for addressing these challenges. Section 4 provides an overview of our integrated model-based framework supporting the development of context-aware AR applications. In Section 5, we present case studies to show the applicability of our framework. Section 6 gives an overview of related work. Finally, Section 7 concludes our work with an outlook on future work.

Challenges

The challenges in developing context-aware AR applications can be divided up in to three main categories: multi-platform capability, adaptation capability, and round-tripping capability. In the following, we describe each category in more detail.

Multi-Platform

An augmented reality application can be used across heterogeneous computing platforms spanning over head-mounted display (HMD) devices to mobile hand held devices. There are a number of AR capable devices on the market which are developed by different companies and organizations. Target devices could be smartphones, tablets or HMDs as the Microsoft Hololens, RealWear Glasses or Google Glass Enterprise Edition, just to mention a few. The cost of a device, its comfort in using it and the ability of a device to help the user accomplish her task are some of the reasons that influences the type of equipment that different users and organizations use to acquire them. In a real world scenario, most users and organizations would like to have the freedom of choosing devices for task execution on an AR application which are most suitable for them if they require to use AR solutions. Mostly, their concerns are to have a mixture of several devices to suit the varying nature of tasks available and also to meet their budget constraints. Each computing platform can have different properties regarding hardware and sensor, operating system, used AR SDKs etc. To support multiplatform AR experience across heterogeneous computing platforms, an efficient way of developing various AR applications is needed.

Adaptation

For supporting context-aware and adaptive AR applications various aspects have to be taken into account.

First of all, context monitoring is an important prerequisite for enabling context-aware applications in general. An important challenge in this regard is to continuously observe the context-of-use of an AR application through various sensors. The context-of-use can be described through different characteristics regarding user (physical, emotional, preferences etc.), platform (Hololens, Handheld, etc.), and environment (real vs. virtual environmental information). Due to the rich context dimension which is spanning over the real world and virtual objects, it is a complex task to track and relate the relevant context information to each other. The mixture of real (position, posture, emotion, etc.) and virtual (coordinates, view angle, walk-through, etc.) context information additionally increases the aspect of context management compared to classical context-aware applications like in the web or mobile context.

Based on the collected context information, a decision making process is required to analyze and decide whether conditions and constraints are fulfilled to trigger specific adaptation operations on the AR application. In general, an important challenge is to cope with conflicting adaptation rules which aim at different adaptation goals. This problem is even more emphasized in the case of AR applications as we need to ensure a consistent display between the real world entities and virtual overlay information. For the decision making step it is also important to decide about a reasoning technique like rule-based or learningbased to provide a performant and scalable solution.

As AR applications consist of a complex structure and composition, an extremely high number of various adaptations is possible. The adaptations should cover text, symbols, 2D images and videos, as well as 3D models and animations. In this regard, many adaptation combinations and modality changes increase the complexity of the adaptation process.

Round-trip

Beside the before mentioned challenges, it is important for a context-aware AR application to support the flexible usage of various information objects. On the one hand, information objects can be text, symbols, 2D and 3D objects which are predefined and available in an existing object repository. On the other hand, it should be also possible to digitize existing real world physical objects, e.g. through a 3D scan, so that further objects can be stored in the object repository and reused at run-time. We call this flexible way of transferring real world physical objects in to a repository and making them reusable again as round-trip. In order to support such a round-trip for the development of context-aware AR applications, a solution should enable the following steps: scan physical objects, refine object model, store object model in repository. Beside that, the import and usage of the created object models the development environment should be supported.

Solution Idea

In order to support the development of context-aware augmented reality applications, we have identified basic architectural patterns to address the identified challenges: Multi-platform, Adaptation and Round-trip capabilities.

Multi-platform capability

For increasing the efficiency of multi-platform user interface development in the context of AR, we envision to establish a model-based development process. Based on the CAMELEON Reference Framework [START_REF] Calvary | A unifying reference framework for multi-target user interfaces[END_REF], as described in Figure 1, we propose a step wise model-based development process.

The top layer Task & Concept includes a task model that is used for the hierarchical description of the activities and actions of individual users of the AR user interface. The abstract user interface (AUI) is described in the form of a dialogue model that specifies the user's interaction with the user interface independent of specific technology. The platform specific representation of the user interface is described by the concrete user interface (CUI), which is specified by a presentation model. The lowest layer of the framework is the final user interface (FUI) for the target platform. The vertical dimension describes the path from abstract to concrete models. Here, a top-down approach is followed, in which the abstract description of relevant information about the user interface (AUI) is enriched to more sophisticated models (CUI) through modelto-model transformations (M2M). Subsequently, the refined models are transformed (model-to-code transformation, M2C) to produce the final augmented reality user interface (AR FUI). Based on this architectural pattern, it is possible to enable multi-platform capability for the different UIs that are generated during the development process.

Adaptation capability

Based on our previous work in the area of UI adaptation for web and mobile apps [START_REF] Yigitbas | Self-adaptive UIs: Integrated model-driven development of UIs and their adaptations[END_REF], we propose an extended version of IBM's MAPE-K architecture (shown in Figure 2) to support context-aware AR applications.

AS depicted in Figure 2, the MAPE-K architecture consists of two main parts Adaptation Manager and Managed Element. In our case, the Managed Element is an AR application consisting of Tasks, Scenes and Interrelations between them. The Adaptation Manager is responsible for monitoring and adapting the AR application through sensors and effectors in order to provide a highly usable AR experience. In the following, the functionality of each sub-component of the Adaptation Manager is briefly described.

The monitor component is responsible for observing the context information. Context information changes are then evaluated by the analyze component to decide whether adaptation is needed. If so, the planning of an adaptation schedule is done by the plan component. Finally, the adaptation operations are performed by the execute component, so that an adapted UI can be presented. The knowledge management base is responsible for storing data that is logged over time and can be used for inferring future adaptation operations. At design time, the described model-based development process supports to generate the final AR user interfaces for various target platforms. The generated final UI is deployed to a specific AR client which enables the described round-trip functionality at run-time. Also, the generated final UI of the AR application is monitored and adapted through the Adaptation Manager at run-time as described in the previous section. In addition to the provided framework, we elaborate on the adaptation process as it is a crucial prerequisite for enabling context-aware AR applications.

Round-trip capability

To address the adaptation process at different development stages, we combine our previous work on model-driven development of adaptive UIs for web and mobile apps [START_REF] Yigitbas | Self-adaptive UIs: Integrated model-driven development of UIs and their adaptations[END_REF] with an existing method for structured design of AR UIs [START_REF] Vitzthum | SSIML/AR: A visual language for the abstract specification of augmented reality user interfaces[END_REF]. As shown in Figure 5, our solution concept addresses three different aspects: AR UI, Context, and Adaptation. Regarding the AR UI aspect, shown in the leftmost column in Figure 5, we rely on the approach and the SSIML/AR language of Vitzhum [START_REF] Vitzthum | SSIML/AR: A visual language for the abstract specification of augmented reality user interfaces[END_REF]. SSIML/AR (Scene Structure and Integration Modeling / Augmented Reality) is a visual modeling language which provides model elements for modeling virtual objects and groups in a virtual scene. Additionally, the relations between application classes and the 3D scene can also be specified. Using SSIML/AR, an abstract specification of the user interface of the AR application is created. This Abstract AR UI Model is the input for the AR UI Generator, which generates the Final AR UI. In order to support the creation of contextaware AR apps, we complement the development method with two additional aspects, namely the Context and Adaptation, originally presented in [START_REF] Yigitbas | Self-adaptive UIs: Integrated model-driven development of UIs and their adaptations[END_REF].

Case Studies

In this section, two case studies are presented which show the benefit of using our model-based development framework for developing context-aware AR applications. The first case study deals with the development of a context-aware mobile AR app which supports UI adaptation capabilities. The second case study shows an example implementation of our object repository supporting the mentioned round-trip capability for scanning and reusing various objects in AR applications.

Context-aware Mobile AR app supporting adaptation capability

For addressing the adaptation capability, described in Section 2, we shortly present the implementation of a context-aware mobile AR application for printer maintenance tasks. Usually, printers have to be maintained and various tasks like refilling paper or changing cartridges should be performed on a regular basis. For such tasks, the printer device usually comes up with a manual. However, most users find it cumbersome to read through the whole manual to accomplish straightforward tasks. Due to this reason and to illustrate the benefit of context-aware AR applications, we implemented an AR application for this example scenario by using our development framework. The context-aware printer maintenance AR (CAPMAR) app makes use of the used smartphone's hardware sensors to observe various context characteristics about the user, platform, and environment. Beside that, we have a preliminary starting questionnaire where newly starting users of the CAPMAR app are asked about their task experience, age etc. Based on those static and dynamically observed context information at run-time, the CAPMAR app is able to trigger AR interfaces adaptations to better suit the context-of-use. Figure 6 is showing a screenshot from the CAPMAR app, where on the left app screen a shorter help message is shown, while on the right screen for a beginner user who is not familiar with the task, a more detailed description of the task is depicted. In summary, the implementation of the CAPMAR app, shows the interface adaptation capability of our model-based development framework for context-aware AR applications.

Object Repository supporting round-trip capability

In order to address the round-trip capability, described in Section 3.3, we have implemented an object repository which supports the digitization of physical 7, we can use third party photogrammetry applications to scan a real object and make a 3D model out of it. The derived 3D object model can be imported into our object repository which enables to mange various of those 3D objects. The object repository is implemented as a web server and supports the persistence of various objects in varying levels of quality detail. Upon the need of the objects, they can be searched and reused based on our mobile client which can position the derived objects in the real scene. In summary, the combination of such an object repository with the concept of context-awareness brings many advantages to ease the developers work in devising context-aware AR apps.

Related Work

In previous work, different approaches were introduced to address the development of context-aware augmented reality applications. In the following, according the challenges introduced in Section 2, we analyze the related work regarding context-awareness in augmented reality applications, multi-platform capability, and object round-tripping.

While context-awareness has been exploited in various types of applications including web [START_REF] Yigitbas | Self-adaptive UIs: Integrated model-driven development of UIs and their adaptations[END_REF], mobile [START_REF] Yigitbas | Component-based development of adaptive user interfaces[END_REF], cross-channel [START_REF] Yigitbas | Engineering context-adaptive uis for task-continuous crosschannel applications[END_REF] and virtual reality [START_REF] Yigitbas | A context-aware virtual reality first aid training application[END_REF] to improve the usability of an interactive system by adapting its user interface, there are only few existing works focusing on the topic of context-awareness in augmented reality.

In [START_REF] Grubert | Towards pervasive augmented reality: Context-awareness in augmented reality[END_REF], the concept of Pervasive Augmented Reality (PAR) is introduced. A taxonomy for PAR and context-aware AR that classifies context sources and Context-aware Mobile Augmented Reality (CAMAR) [START_REF] Oh | Camar: Context-aware mobile augmented reality in smart space[END_REF] is an approach on context-awareness in mobile AR focusing on user context, which is measured using the user's mobile device. It enables the user to customize the presentation of virtual content and to share this information with other users selectively, depending on the context. Furthermore, a framework called UCAM (Unified Context-aware Application Model) [START_REF] Hong | A new paradigm for user interaction in ubiquitous computing environment[END_REF] can be used to create CAMAR-enabled applications. UCAM is a framework which beside the acquisition, process, and awareness of contextual information provides also a unified way of representation with respect to user, content, and environment.

The framework presented in [START_REF] Lindlbauer | Context-aware online adaptation of mixed reality interfaces[END_REF] focuses on context-aware adaptation of interfaces in mixed reality, with the main adaptation points being what content is displayed, where it is shown and how much information of it is displayed. It is designed to adjust the content display depending on the user's tasks and their cognitive load and archives this using a combination of rule-based decisions and combinatorial optimization. The framework uses parameters about the applications that are to be displayed as input additionally to the context-specific parameters to achieve a fitting layout optimisation. The framework is mentioning mixed reality as its base, but regarding that it shows contents in the real world and does not create a whole new virtual world, it can safely be said that AR is supported.

While above mentioned approaches address the development of context-aware applications, the main focus is not lying on a model-based development framework for context-aware AR applications. As such, the required concerns and features regarding context-awareness and interfaces adaptation are mostly manually implemented while our solution idea is aiming a model-based approach that increases reusability, maintainability, and portability of the AR app for different target platforms. The AR Foundation project3 , aims to the creation of AR apps that run both on iOS and Android. It provides a common API which covers the core functionality of both Android ARCore4 and iOS ARKit5 , thus making it possible to develop AR applications for both platforms from a single code base.

Beside the above described approaches focusing on context-awareness in augmented reality applications, we shortly describe prior work that addresses the topics of round-trip capability and object repositories for prototyping and development of context-aware AR applications. One important resource to assist the development of AR applications is given by object repositories such as Google Poly 6 or Unity Asset Store7 . Main idea of such object repositories is to provide a public repository for all sorts of assets (images, 3D models, sound and visual effects, etc.) which can be reused in the development of augmented reality applications for different domains and in various projects. Beside that, existing approaches like DART [START_REF] Macintyre | DART: the designer's augmented reality toolkit[END_REF] or ProtoAR [START_REF] Nebeling | Protoar: Rapid physical-digital prototyping of mobile augmented reality applications[END_REF] aim to support the rapid prototyping of AR applications. While these approaches ease the prototyping and development of AR applications even for non-technical developers, their focus is not lying on support for addressing context-awareness and interface adaptation. Therefore, our model-based development framework for context-aware AR applications makes use of synergy effects between existing prototyping approaches and object repositories that are combined with a model-based development approach for context-aware AR apps.

Conclusion and Outlook

This paper discusses the main challenges in developing context-aware augmented reality applications and presents architectural solution patterns to address them. Based on the identified architectural solution patterns, we propose an integrated model-based development framework for context-aware AR applications. Furthermore, we elaborate on the adaptation process and propose a model-based solution architecture for the development of adaptive AR applications. At the end, we show that our framework can be used for the development of contextaware AR applications.

In future work, we plan to implement tool-support for model-based development of context-aware AR applications. Our goal is to support the efficient development of context-aware AR applications for different application scenarios covering different target platforms.

Figure 1 .

 1 Figure 1. Multi-platform support

ForFigure 2 .Figure 3 .Figure 4 .

 234 Figure 2. Adaptation support

Figure 5 .

 5 Figure 5. Model-based Solution Architecture for Adaptive AR Apps

Figure 6 .

 6 Figure 6. Case Study 1: Context-aware AR Printer Maintenance App

Figure 7 .

 7 Figure 7. Case Study 2: Object Repository for AR Apps

https://www.microsoft.com/en-CY/hololens

https://www.magicleap.com/magic-leap-one

https://unity.com/unity/features/arfoundation

https://developers.google.com/ar

https://developer.apple.com/augmented-reality/

https://poly.google.com/

https://assetstore.unity.com/


Torkil Clemmensen

Jacob Nørbjerg

Digital Peer-Tutoring: Early results from a field evaluation of a UX at work learning format in SMEs

Keywords: collaborative robots, assistive technologies, UX at work 1

Digital Peer-Tutoring is a new learning format that enables production workers in Small to Medium Sized Enterprises (SMEs) to co-design their interaction with assistive technologies such as collaborative robots.The videobased learning format is based on design thinking and helps shop floor workers create and document solutions to robot interaction problems, and share their how-to knowledge with their colleagues. Early field evaluation results indicated that workers benefit from the Digital Peer-tutoring learning format and produced how-to videos for their colleagues. Furthermore, the Digital Peer-tutoring learning format was also found useful by the company management and ownership as means of documentation and customer communication. Thus, the learning format can also support SMEs on their path to digitalization

Introduction

In this paper we propose a new learning format 'Digital Peer-tutoring' as a means to design and share solutions to worker-technology interaction problems in small to medium sized enterprises1 (SMEs).

Peer tutoring has long been suggested as a way to help students deal with design problems [START_REF] Schleyer | Peer tutoring in conceptual design[END_REF]. Design, understood here as design thinking [START_REF] Kolko | Abductive Thinking and Sensemaking: The Drivers of Design Synthesis[END_REF], is an iterative process consisting of generative and evaluative stages, which eventually converge on a solution to the design problem. Design thinking is typically applied to solve non-routine, wicked problems in an organization, when there is a need for novel how-to knowledge. To engage increation and sharing of new how-to knowledge requires hands-on experience, which is where peer tutoring becomes very helpful. The new learning format Digital Peer-Tutoring aims to help workers interacting with collaborative robots on the shop floor touse digital media to engage in teaching and learning with colleagues about their user experiences. We ask the questions: Can a Digital Peer-Tutoring learning format enable shop floor workers to design positive UXs for them-selves and their colleagues? What kind of ethical stance is implied by the use of Digital Peer-Tutoring?

The paper reports from the initial part of a research project aiming to develop a Digital Peer-Tutoring learning format for shop floor workers in SMEs. The project aims to develop capabilities among shop floor workers to design and document, with short videos, solutions to operational and collaboration issues related to collaborative robots.

The research is situated within the KomDigital regional development project that brings together 18 of the Copenhagen Capital Region's companies, unions, employer associations, and educational institutions. The project aims to improve digital competencies in a broad sense among the employers and employees in SMEs thereby enabling the companies to adopt and implement digital technologies. The target companies come from all sectors, including construction and building, small scale production, product development, and finance, and the technologies include data mining and analysis, collaborative robots and other forms of production automation, AI based financial advice, and more.

KomDigital achieves its goals through new digital learning formats, which can be made available to target companies and organizations. The formats are tailored to the working conditions and needs of companies and employees, so that both employees, managers, companies and organizations can use new digital technologies to expand and grow.

2

Related work

Digitalization in SMEs

SMEs depend on their workers' knowledge and innovative capabilities to create new ways of working with technology, and they generally lack the capability and capacity for comprehensive digital transformation [START_REF] Ludwig | Revive Old Discussions! Socio-technical Challenges for Small and Medium Enterprises within Industry 4.0[END_REF][START_REF] Mucha | Application Areas for Human-Centered Assistive Systems[END_REF]. Collaborative robots that work alongside a human worker can be integrated into the production without radical reconfiguration or automation of established workflows. A human worker can program a collaborative robot to perform tasks such as lift, pick and place, move, or otherwise process physical objects [START_REF] Kragic | Interactive, Collaborative Robots: Challenges and Opportunities[END_REF][START_REF] Sanchez | Collaborative Robots in Surgical Research: a Low-Cost Adaptation[END_REF][START_REF] Schulz | Collaborative robots learning spatial language for picking and placing objects on a table[END_REF].Thus, worker designed interaction with collaborative robots and other assistive technologies is a useful first step towards digitalization in an SME.

Peer tutoring

Peer tutoring [START_REF] Magin | Peer tutoring in engineering design: A case study[END_REF][START_REF] Schleyer | Peer tutoring in conceptual design[END_REF]overlaps somewhat with other notions of providing informal technical help between colleagues, such as over-the-shoulder-learning [START_REF] Michael B Twidale | Over the shoulder learning: supporting brief informal learning[END_REF], over-theshoulder-guidance in tertiary education [START_REF] Angela | Towards over-the-shoulder guidance following a traditional learning metaphor[END_REF], peer-assisted learning [START_REF] Martinez | Using peer-assisted learning to teach and evaluate residents' musculoskeletal skills[END_REF] and peer teaching [START_REF] Secomb | A systematic review of peer teaching and learning in clinical education[END_REF] in the medical domain, andover-the-shoulder appropriation [START_REF] Draxler | Supporting the collaborative appropriation of an open software ecosystem[END_REF] and peer interaction [START_REF] Murphy-Hill | How Do Users Discover New Tools in Software Development and Beyond?[END_REF] in software development.

In this paper we build primarily on the approach put forward by Twidale [START_REF] Michael B Twidale | Over the shoulder learning: supporting brief informal learning[END_REF] in that we aim to support the provision ofinformal technical help between colleagues. Similar to Schleyer et al. [START_REF] Schleyer | Peer tutoring in conceptual design[END_REF] we acknowledge the role of peer tutors at various levels towardsdeveloping problem solving skills among colleagues. Specifically, we introduce a new role of digital competence facilitator, a 'Digital Coach', as explained below.

Digital Peer-Tutoring

What distinguishes 'digital peer-tutoring' from traditional peer-tutoring is that the concept builds entirely on the use of video. The idea is that workers learn from creating and redesigning videos while sketching [START_REF] Ørngreen | The Learning Potential of Video Sketching[END_REF]as part of applying design thinking to design their own and their colleagues' work flow and interactions with collaborative robots. Ørngreen et al. [START_REF] Ørngreen | The Learning Potential of Video Sketching[END_REF] suggested to link sketching techniques and creativereflection processes to video productions, and we extent this proposal to cover linking all parts of design thinking (problem definition and user needs finding, sketching, prototyping hypotheses, evaluation) to workers' video production. Secondly, we propose that video-based reasoning, instead of paper or verbal exchange, empower workers to explore and take ownership of their work. Vistisen et al. [START_REF] Vistisen | Animating the ethical demand: Exploring user dispositions in industry innovation cases through animationbased sketching[END_REF] proposed to support ethical userstances during the design process of products and services, andproposes using animation-based sketching as a design method. We follow that line of thought, though we are less interested in professional designers, and more interested in workers' own production (and consumption) of videos-as-digital-peer-tutoring.

Case setting: A collaborative robot in specialized glass manufacturing

After learning from initial talks with three different SMEs in Denmark, we agreed with the ABC company to adapt and evaluate the digital peer-tutoring learning format in one of their production facilities. The ABC company is a European SME specializing in glass processing. The company produces individual pieces and small batches with special specifications as well as entire series of several thousand units. About a year prior to our visit, the ABC company purchased and installed a 100,000€ collaborative robot in order to explore if and how it could be used in their production. At the time of our visit, the robot was used only during the final polishing steps of one large scale order, and it was idle much of the time. Workers and management agreed, however, that the robot could be used for other purposes as well, and thus enable the company to accept more large batch orders, but no initiatives had been implemented for several months due to lack of time to experiment with the robot. Furthermore, the initial design decision had been a stationary installation, that is, the robot could not be moved to other positions on the floor where it could interact with other machines or workers.

The initial design decisions seemed to be related to a limited initial understanding of the robot's capability and a lack of strategic intent. In any case, it was clear that there was an unexplored potential (and risks) for enhancing the factory's capacity while empowering workers and help them design their own user experiences with the robot.

4

Method: Action design research with SMEs

Our approach to building new digital competences in SMEs is inspired by action design research (ADR). ADR argues that IT artifacts are 'ensembles' formed by the organizational context during development and use. Research in this tradition interweaves constructing the IT artifact, intervention in the organization, and evaluating outcomes [START_REF] Sein | Action Design Research[END_REF].

We visited the company 6 times over a six-week period during the spring 2019. The purpose of our first visit was to develop insights into the company, the motivation for purchasing the robot, and challenges with its current as well as potential future useof the robot. We observed the robot's current (very limited) use, interviewed and discussed with robot vendors, managers and shop-floor workers, and observed work and demonstrations of the robot.

The digital peer-tutoring learning format (see section 5) was implemented in four sessions over the next four visits, followed by a final evaluation on the sixth visit.We documented all observations, interviews, and learning sessions with video and audio recordings and photos, and we collected the videos produced by the workers.

The learning format was evaluated after each session and at a final one-day meeting with participation from all key stakeholders.

5

The digital peer-tutoring learning format The digital peer-tutoring learning format consisted of an ensemble of instructionvideos, quizzes, example-solution-video, and worker-created-how-to-videos. Together with the case company production site, we designed and implemented four training sessions with selected shop-floor workers (Table 1).

We developed short (3-5 minutes) instruction videos for each session that explained the theme, introduced techniques that the participants could use to investigate problems and describe solutions, and concluded with an exercise where the participants should develop a short video (3-5 minutes). We also produced short example videos with our 'answers' to the video assignment for each session.

All video materialincluding instruction material was recorded with standard smartphone hardware and software, and published without editing, in order to promote a 'simple-yet'sufficient' attitude towards to video production.

For each session, a 'digital competence facilitator' (student assistant) travelled to the factory and discussed the material with the participants, and helped them produce their own 'employee-videos', which were subsequently uploaded to a shared (secure) site for later download and knowledge sharing within the company.

Field evaluation results

The evaluation of the 'Digital Peer-Tutoring' learning format consisted of weekly evaluations after each of the four sessions, and a final evaluation with participation from all key stakeholders. Here, we report about the initial results from the finalevaluation; a one-day meeting in the location of the factory of the case company.The participants in the evaluation were all those present at the upstart meeting 6 weeks before. They were: company managers (Company manager J and Company manager K), learning format users (Worker Br, Worker H, Worker Bi), corporate learning consultant (corporate learning consultant F), educational institution teacher(s) (Teacher J, Teacher T), pilot project manager(s) (Teacher T, Teacher J), pilot project documentarist (Documentarist F), and a digital competence facilitator (Digital competence facilitator S).

The initial results from the final evaluation reveal both short-and long-term benefits and challenges of Digital Peer-Tutoring.

Short term benefits

The workers liked the learning format and found it useful: "...worker-video on iPad [could beuseful]...", [Worker Br]. This confirms previous findings on the usefulness of video [START_REF] Ørngreen | The Learning Potential of Video Sketching[END_REF], and extends it to the shop floor workers. However, the workers found that the instruction videoswere too long and complicated. "[They should be cut down to a list of four points" [Worker Br]. Too long videos can be an expression of an 'apathetic ethical stance', a stance that reduces the worker-user to be a mean of input for the intended final design [START_REF] Vistisen | Animating the ethical demand: Exploring user dispositions in industry innovation cases through animationbased sketching[END_REF].

On the other hand, the workers expressed that they could use video to both think about a problem, sketch different solutions, and evaluate their use:"Sketches .... I had read up on it, go and think about it...." [Worker Br], and "the workershould be able to pause the video ..." [Worker Bi]. Thus, there were indications that the format helped workers explore new technologies from an emphatic ethical userthat is, from their ownperspective [START_REF] Vistisen | Animating the ethical demand: Exploring user dispositions in industry innovation cases through animationbased sketching[END_REF]. Company manager K supported this: "We, as a business must spend more time on [workers' use of video to innovate]." The management perspective adds a new layer to understand short term benefits of video-sketching and ethical design, and thus center our focus on the multi-layered essence of user experiences at work.

Long-term benefits

The stakeholders also commented on the long-term benefits of the learning format:

1. The format could be used to tackle issues in the manufacturing, as "help videos" [Worker Bi], and a "'Company database of videos that could be accessed even years after production" [Company manager J], 2. New employees could be introduced to the job: "A new one that is totally novice [could use worker-created-how-to-videos]" [Company manager J], 3. Help dyslexic employees who could watch how to do things, rather than read, 4. Supplier courses could be made memorable by "record [ing] and "The video can be used to "squeeze" a good idea out of an experienced employee who will have to think a little about the idea" [Teacher T]. 6. Finally, the stakeholder group discussed that the learning format could also be used to produce videos for customers for marketing purposes and quality documentation.

These benefits allude to a diversity of user experiences in work situations, and perhaps tell us that the ethical stances taken by workers-as-designers-of-their-own-work may be confounded by management's strategic interest in how-to knowledge.

Discussion and conclusion

We conclude that our proposed Digital Peer-Tutoring learning format enabled shop floor workers to design positive UXs for themselves and their colleagues, and beyond ways that we expected. The participating shop floor workers stated in various ways that they liked the Digital Peer-tutoring how-to videos and foundthem useful.This corresponds to the claim made by Twidale (2005) [START_REF] Michael B Twidale | Over the shoulder learning: supporting brief informal learning[END_REF]that it is possible to use peer tutoring to give informal technical help between colleagues, and with Ørngreen et al.

(2017) [START_REF] Ørngreen | The Learning Potential of Video Sketching[END_REF]who suggest to link various sketching techniques and creative reflection processes to video productions.The videos helped workers create ideas about robot use, identify problems not formulated before, sketch alternatives, test solutions, and demonstrate them to colleagues.

Company owners, management, and workers had unexpected ideas about how to use the peer-tutoring videos within and outside the company, in for example internal quality control and customer communication. Thus, similar to the point made about peer tutoring [START_REF] Schleyer | Peer tutoring in conceptual design[END_REF], we should acknowledge the role of Digital Peer-Tutoring in developing problem solving skills at various organizational levels.

Based on the categories proposed in [START_REF] Vistisen | Animating the ethical demand: Exploring user dispositions in industry innovation cases through animationbased sketching[END_REF],we furthermore observe that the ethical stance built into the'Digital Peer-Tutoring' learning format can be characterized as 'apathetic', when too long and complex instructional videos, intended to teach workers' design thinking and enabling their own video-production, tendto make workers give up. However, the learning format also showed to be 'empathetic', as workers produced their own videos and evaluated solutions together, effectively co-designing work procedures.

We developed the Digital Peer-Tutoring learning format to improve workers' capability to create and share solutions to human-robot collaboration challenges in SMEs.

Thereby we also answer the call for research into how SMEs can adopt and implement new technologies that build upon and enhance worker capabilities, skills, and knowledge [START_REF] Hannola | Empowering production workers with digitally facilitated knowledge processes-a conceptual framework[END_REF][START_REF] Ludwig | Revive Old Discussions! Socio-technical Challenges for Small and Medium Enterprises within Industry 4.0[END_REF].

Table 1 .

 1 Overview of training sessions.

	Sessions Themes	Topics	Worker-created-how-to-
				videos
	1	The problem	Personas	1. A persona
			Interaction	2. An interaction problem
			Collaboration with tech.	3. A collaboration problem
	2	Solution	How to sketch a solution	Three design ideas for
		sketch	Interaction	4. Interaction
			Collaboration	5. Collaboration
	3	Design proto-	Interaction and collabora-	6. Elaborate one design idea
		type	tion prototypes	into a prototype
	4	Evaluate	How to evaluate/test	7. Test the prototype with a
		prototype	prototypes	colleague

  what the supplier shows on the shop floor" [Corporate learning consultant F], and "Cut out what is not useful [from the supplier teaching]"[Company manager K] 5. Starting new ways to produce, for example "recording the results from the company's informal and formal experiments on the shop floor" [Company manager J, Company manager K], and "recording order-specific ideas for how-to, so next time this order comes in, the video shows what to do" [Worker Bi],

The category of micro, small and medium-sized enterprises (SMEs) consists of enterprises which employ fewer than

persons with an annual turnover not exceeding 50 million euro, and/or an annual balance sheet total not exceeding 43 million euro.


Marta Kristin Larusdottir

Marcel Kyas
email: marcel@ru.is

Assessing the Performance of Agile Teams

Keywords: User experience, Accessibility, Security, Agile development, Team collaboration, Agile teams

scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

When public authorities want to make new software systems to be used by citizens and employees for solving various tasks they often negotiate with software companies for developing the software. The selection of the software company for making the software needs to be free and open for competition according to European Union legislation, so the public authorities must issue a public request for proposal (RTF).

Typically the RTF contains two sections: (1) the requirements and needs for the system to be developed, and (2) the selection criteria [START_REF] Jokela | Usability in RFP's: The current practice and outline for the future[END_REF]. Often the selection criteria is based on the cost solely, so the software companies estimate the hours needed to be able to develop the software fulfilling the requirements and needs stated. The company with the lowest prize gets the job [START_REF] Jokela | Usability in RFP's: The current practice and outline for the future[END_REF]. In a case study of four software companies in Denmark developing for public authorities, the software companies focused on what the public authorities are willing to pay for and what they wanted to citizens to be able to do [START_REF] Billestrup | A Case Study of Four IT Companies Developing Usable Public Digital Self-Service Solutions[END_REF]. So the software companies did not include quality factors like user experience (UX) or security issues, in their proposal, if it was not requested in the RFT.

In some cases the selection criteria is based on both the prize and quality factors, so the price could weight 60% and the quality criteria 40% for example [START_REF] Jokela | Usability in RFP's: The current practice and outline for the future[END_REF]. Requirements for quality factors, like user experience (UX) and security, can be included in the requirement section of the RTF defining the level of the UX and security in the developed system. The requirements can also be included in the selection criteria, defining how much weight in the selection process the UX and security factors have [START_REF] Tarkkanen | Evaluation for Evaluation: Usability Work during Tendering Process[END_REF]. Typically, the usage of particular methods like user testing and the frequency of using those methods would be stated in the selection criteria. Another option would be that the public authority may state performance criteria for the users, for example that the users will be able to accomplish a particular task within a particular time limit [START_REF] Tarkkanen | Evaluation for Evaluation: Usability Work during Tendering Process[END_REF]. One possibility is to base the selection criteria on the competences of the software team getting the job, but that is not frequently done. The selection criteria should state the wanted knowledge, skills and competences of the team, in that case. Possibly, the criteria could also include the focus on quality aspects that the team should have. In any case, the objective of the process is to find the best team for the job according the predefined criteria and thereby get the best outcome for the money spent.

There are many aspects that affect a project outcome. A study of four similar software teams developing software to fit the same needs, described 1 to 6 variation in the prizes of the outcome [START_REF] Schwaber | Scrum development process[END_REF]. The teams were similar in technical competences. The quality of the outcome was also assessed and the team with the next lowest price scored best on the three quality aspects in the study, usability, maintainability and reliability. That team had one project manager, one developer and one interaction designer in the team, but the other teams had two developers and one project manager. The best team used intermediate process models for the development, with analysis and design in the first four weeks, then implementation from week 4 to 10 and testing in the last six weeks of the project [START_REF] Schwaber | Scrum development process[END_REF].

In this paper we describe an approach, where the performance of five agile software development teams was evaluated as a part of the selection criteria for selecting the best agile team for making a web service. The performance factors included: 1) team collaboration and user experience focus, 2) user stories delivered and 3) quality of code including accessibility and security. The performance factors were assessed during and after a one day workshop with the team, where the teams were observed and their deliverables reviewed. The performance factors weighted 70% and the cost 30% in the selection criteria for the best agile team.

Related Work on the Performance Factors

In this section we briefly describe the related literature on the performance factors evaluated in this study. First we give a brief overview of agile development and team collaboration, we explain the format and usage of user stories and then we briefly describe the concept of user experience and code quality.

Agile Development and Team Collaboration

The agile process Scrum [START_REF] Sjøberg | The relationship between software process, context and outcome[END_REF] has gained popularity in the software industry in recent years. According to an international survey, Scrum was the most popular process of the agile processes with more than 50% of the IT professionals surveyed were using it [START_REF]13th Annual State of Agile survey[END_REF].

A similar trend is seen in the software industry in Iceland, but the lean process Kanban [START_REF] Ohno | The Toyota Production System: Beyond Large-Scale Production[END_REF] has also been gaining popularity lately [START_REF] Law | Whose experience do we care about? Analysis of the fitness of Scrum and Kanban to User Experience[END_REF]. A characteristic of Scrum is the observation that small, cross-functional teams historically produce the best results. Scrum is based on a rugby metaphor in which the team's contribution is more important than each individual contribution. Scrum teams typically consists of people with three major roles: 1) a Scrum Master that acts as project manager/buffer to the outside world; 2) a Product Owner that represents stakeholders, and 3) a team of developers (less than 10). One of the twelve principles behind the agile manifesto is: "The most efficient and effective method of conveying information to and within a development team is face-to-face conversation." [START_REF]Manifesto for Agile Software Development homepage[END_REF]. In agile development the teams should collaborate openly and all the team is responsible for delivering a potentially shippable product after each sprint. Some of the more important artifacts and ceremonies within Scrum is the Sprint, which defines 15-30 days iteration, the Product backlog of requirements described by user stories and managed by the Product Owner and the Daily Scrum meeting, which is the daily meeting for the team and the Scrum Master to plan the work of the day and report what was done the day before [START_REF] Sjøberg | The relationship between software process, context and outcome[END_REF].

User Stories

In Scrum, the user requirements are usually described by user stories. The most common format for describing a user story is: "As a [user role], I want to [do some task] to [achieve a goal]" [START_REF] Cohn | User Stories Applied[END_REF]. The user stories are used to describe the requirements for the whole system being developed kept in the Product Backlog. During the Sprint planning meeting, the team, the Scrum Master and the Product Owner select the user stories that the team will work on during the next sprint in accordance to how many user stories it is possible to implement during the time of a sprint. The Product Owner describes the priorities of the user stories, so the most important user stories will be selected for the particular sprint according to the Product Owners criterias. During the daily Scrum meeting, the team members report what user stories and tasks they will be working on during the day and what the finished they day before.

User Experience

UX has gained momentum in computer science and is defined in the ISO 9241-210 in the following way [START_REF]Ergonomics of humansystem interaction -Part 210: Human-centred design process for interactive systems[END_REF]: "Person's perceptions and responses resulting from the use and/or anticipated use of a product, system or service". Researchers agree that UX is a complex concept, including aspects like fun, pleasure, beauty and personal growth. An experience is subjective, holistic, situated, dynamic, and worthwhile [START_REF] Hassenzahl | User experience and experience design[END_REF]. A recent survey on what practitioner's think is included in the term UX shows that respondents agreed that user-related factors, contextual factors and temporal dynamics of UX are all important factors for defining the term UX [START_REF] Lallemand | User experience: A concept without consensus? Exploring practitioners' perspectives through an international survey[END_REF]. The temporal dynamic of UX also reached consensus amongst the respondents. Many methods have been suggested for active participation of users in the software development process with the aim of developing software with good user experience. Some of the methods for focusing on either the expected UX or the UX after users have used a particular system, including interviews with users, surveys, observations and user testing [START_REF] Preece | Interaction design: beyond human-computer interaction[END_REF]. IT professionals rated formal user testing as the most useful method for active participation of users in their software development for understanding the UX of the developed system [START_REF] Jia | The usage of usability techniques in Scrum projects[END_REF].

Quality of Code Including Security and Accessibility

Code quality is generally hard to define objectively. Desirable characteristics include reliability, performance efficiency, security, and maintainability [START_REF] Curtis | CISQ Recommendation Guide[END_REF]. Metrics to assess code quality usually include volume of code, redundancy, unit size, complexity, unit interface size, and coupling [START_REF] Heitlager | A practical model for measuring maintainability[END_REF][START_REF] Baggem | Standardized code quality benchmarking for improving software maintainability[END_REF]. The process of measuring properties like complexity and the decision on what unit size is acceptable depend on the context and is often subjective.

Accessibility of web application is typically realized by conforming to the WCAG 2.0 recommendation [START_REF] Caldwell | Web Content Accessibility Guidelines (WCAG) 2[END_REF]. Following these recommendation allows a web page to be interpreted and processed by accessibility software. For example, by a.o. preferring relative font sizes over absolute ones allows the web page to be rendered in any font size and making it accessible to users with visual impairments. The WCAG is seen as an important part of making web pages accessible [START_REF] Kelly | Forcing standardization or accommodating diversity?: A framework for applying the WCAG in the real world[END_REF]. Indeed, for any web application and any mobile application used by the public sector in the European Economic Area must conform to the WCAG [START_REF]2102 of the European Parliament: Directive (EU) 2016/2102 of the European Parliament and of the Council of 26 October 2016 on the accessibility of the websites and mobile[END_REF].

3

The Case -The Financial Support RTF Reykjavik city has decided to make the digital services easy to use for all the citizens of Reykjavik. The motivation came from two new employees that wanted to change the web services to being more user centered. One of the first projects for this attempt had the objective to make the application for financial support more usable to citizens, but to focus also on security and reliability of the code. An official request for proposals was made to select "the best" team for taking part in developing a web service in collaboration with IT professionals at Reykjavik city. One of the constraints was that the team had to follow an agile development process similar to Scrum, by using user stories, conducting daily Scrum meetings and focus on the values of agile team work and collaboration.

The teams that submitted a proposal were evaluated according minimal technical requirements and their performance and delivery after a one day workshop. There were five steps in the selection process: a) First the team submitted a proposal, b) The applying teams were evaluated according to the minimum technical requirements, c) the teams fulfilling the technical requirements were assessed according to performance criteria, d) the hourly prices of each team member were evaluated and e) the final selection of a team was decided. In this section we describe the minimal technical requirements for the teams and the three performance factors assessed during and after the one day workshops.

The Minimal Technical Requirements

The minimal technical requirements were described in the request for proposals document. The teams had to provide at least 5 team members, whereof at least: 1. Two members had to be skilled backend programmers, which had experience in writing code that was tested for security. For confirming these skills, the team members were asked to deliver a list of projects were they had worked on security issues for the system. They also had to list at least 5 software projects that they had been involved in. They had to be experienced in automated testing and have knowledge of .NET programming. 2. 1 member had to be a user interface programmer. This persons had to have the experience of making apps or web services that fulfilled the accessibility standard, European Norm EN 301 549 V1.1.2 [START_REF]Accessibility requirements suitable for public procurement of ICT products and services in Europe[END_REF] that includes the WCAG 2.0 Level A and Level AA and are scalable for all major smart equipment and computers. This person had to describe his/her involvement in five software development projects. 3. 1 member had to be interaction designer or a UX specialist. This member had to have taken part in developing at least 5 software systems, (apps or web services), with at least 100 users each. They should describe their experience of user centered design with direct contact with users and what methods they had used to integrate user in the development. 4. 1 member should had to be an agile coach or a Scrum Master. To fulfill this, the person had to have led at least one team with at least three members with at least 10 two week sprints. This member should describe his experience regarding coaching team members.

The Workshop Organization

Five teams fulfilled the above minimum technical requirements. Each of them were invited for a one day performance workshop. The workshops took place at an office at the IT department of Reykjavik city in October and November 2018. The teams got four user stories to as possible tasks to work on during the workshop.

The user stories were the following: 1. As a citizen of Reykjavik that has impaired intellectual ability I want to be able to apply for financial assistance via web/mobile so that I can apply in an simple and easy-to-understand manner. 2. As a employee of Reykjavik city with little tech know-how I want to be able to see all applications in a "employee interface" so that I have a good overview of all applications that have been sent. 3. As a Reykjavík city employee which is colorblind I want to be able to send the result of the application process to the applicant so that the applicant can know as soon as possible if the applicant is eligible for financial assistance. 4. As a audit authority for financial assistance I want to be able to see who has viewed applications so that I can perform my audit responsibility.

The workshops were organized by a project manager at Reykjavik city. The schedule was the following: 1. The team got a one hour introduction to the schedule of the day and to the work environment at Reykjavik city, the services and systems, the organization and work practices. Also the user stories were introduced briefly. 2. The teams were asked to do a daily Scrum meeting for 15 minutes for selecting the tasks for the day and to organize the day for 15 minutes. The experts focusing on team collaboration and UX focus observed this part of the workshops. 3. The teams worked on developing their deliverables during the day. 4. The last 45 minutes of the day, the teams were asked to present to all the involved experts and the organizing team, their work practices and their deliverables. The teams could plan these 45 minutes as they preferred. They had been introduced to the performance factors that were being assessed, so some of the teams deliberately organized the presentation according to these factors.

The Performance Factors Assessed During and After the Workshops

The workshops had the goal of assessing the following three performance factors:

1. The teams collaboration and user experience (UX) focus 2. Their delivery of user stories 3. The quality of the code delivered An assessment scheme was conducted for each of the three factors. Four external experts were asked to conduct the assessment. The team collaboration and UX focus contained four sub factors and in total these gave the maximum of 25 points. These were assessed by two external experts by observing the teams twice during the one day workshop. The delivery of user stories and the quality of the code delivered were assessed after the workshop. Two external experts in security issues and performance were asked to review the code delivered. The user stories delivered gave maximum 10 points and the quality of the code 35 points. In total these three performance factors added up to 70 points. The hourly price for the team members could give a maximum of 30 points. Experts at Reykjavik city reviewed the hourly prizes. The agile team could get 100 points in total, if they got the maximum points for all the three performance factors and the hourly prizing. We will describe the process of the data gathering for assessing the three performance factors resulting from the workshops in the next section.

Data Gathering for Assessing the Performance Factors

In the following we will describe the process of gathering data to be able to evaluate the team collaboration, the user stories delivered and the quality of the code.

Data Gathering for Assessing the Team Collaboration and UX Focus

Two experts in team collaboration and UX focus were asked to assess this performance factor.

Four sub factors were defined: 1. How well did the team perform at the daily meeting (max 4 points)? 2. How problem solving oriented was the team (max 8 points)? 3. How much did the team emphasize UX (user experience) (max 8 points)? 4. How well did the team present their work at the end of the workshop (max 5 points)?

The two experts observed the teams during an half an hour session in the morning, when the teams had a daily Scrum meeting and when selecting tasks for the day. The experts took notes and assessed the first sub factor. They tried to keep silent and not ask questions so the five workshops would be as similar as possible. Forty five minutes were used as the last part of the workshop for presenting the work practices that the team used during the day and the deliverables. The two experts observed the presentation and took notes. The experts only asked, if there were issues, which the experts were about to assess, that were not mentioned during the presentation, to have better information on all the performance factors.

There was a short meetings with all the experts involved and the organizing team at Reykjavik city right after each workshop. The goal was to discuss the first impression of the workshop of that day. For each of the sub factors there were predefined questions that the experts answered for assessing each sub factor.

For the UX emphasis sub factor, the questions were five: 1) Does the team emphasize UX when presenting their solution? 2) Does the team justify their decisions according to the needs of the users? 3) Does the team justify the flow and the process in the UI according to the user needs? 4) Has the team planned some user evaluations? 5) The team consulted with users during the workshop? Each of the experts rated all the questions for the four sub factors for the teams within 48 hours and noted an argument for each of the ratings. The two experts met shortly after that assessment and discussed their individual answers to the questions and the rating of each sub factor and made a consolidated rating for the team that was sent to the project manager of the workshops. When all the teams had been assessed, the two experts met to make the final comparison of all the ratings andmake the final ratings that were sent to the project manager of the workshops..

Data Gathering for Assessing the User Stories Delivered

A second team of two experts was assigned the task of assessing whether the user stories had been successfully implemented. The second team had to rely on the documentation of the submission to identify the code that was supposed to implement the feature described by the user story and the test cases for that story.

Each agile team submitted their project as a dump of a git repository. Some teams also submitted sketches, mock-ups and photographs of all documentation written down during the workshop day. In addition, some teams kept a test instance of their system running for the two experts to test.

The assessment criteria were:

1. Did the submitting team make a claim that a user story was implemented? Lacking such a claim the experts would assume that the story was not implemented. 2. Did the submitting team document what functions were used to implement the user story? The experts would look at the code only for names that related to concepts in the user story. 3. Did the submitting team provide test cases to test the user story?

The verdict for each user story was pass or fail. The score was with respect to the maximum achieved by all teams. One team managed to implement 3 stories, which gave the maximum number of 10 points. All other teams scored a fraction of three, according to the number of stories they achieved. A finer distinction than pass and fail was rejected, because the experts could not agree on how that should be done objectively, and they felt that it was not worth the effort.

Data Gathering for Assessing the Quality of the Code

As mentioned above, each team submitted their code as a clone of a git repository. This enabled the experts to assess the way the teams were documenting their software development process.

The properties that the two experts assessed were: Both the experts and the City of Reykjavik believe that the quality of documentation of the code, the architecture, and the log messages in the version control system are predictors of product quality and maintainability of the resulting project. Points 3 and 4 were most relevant to the interaction with the user. The experts used the WAVE web accessibility assessment tool to assess the quality of web accessibility and to check compliance with WCAG 2.0 at levels A and AA [START_REF]Accessibility requirements suitable for public procurement of ICT products and services in Europe[END_REF]. The experts investigated the choice of colors by hand and by using filters to simulate how color vision deficient users would see the web site. Overall, all submissions had some issues with web accessibility, like laying out information in the wrong order, missing alt tags for images, and so forth.

The two experts referred to the way erroneous behavior is conveyed by the user for assessing the error handling in the interface. The experts checked whether the error messages were displayed in a meaningful manner, how an encountered error would be addressed, and whether a pointer to assistance was provided.

The difference between error handing in the user interface and the error handling in the code was defined. If a possible user would makean error in using the system, or if the system would fail to perform the task the user wanted to achieve without being able to recover, the experts checked whether the error message delivered to the user provided a meaningful explanation of what happened and if it provided an assistance in how the user could recover from the error. If a possible error related to an internal API and could be handled programmatically, they considered it to be a code error. The experts expected such errors, typically thrown exceptions, to be caught, logged and handled.

Functionality of the data base scripts referred to how the data of the users was represented in the data model, how that model was represented as data base schemata, whether they were set up correctly and later used correctly.

The final parameter error free function was assessed through a demonstration of the teams, a cursory code review, and the presence of tests and their traceability to requirements.

No formal audit was defined concerning security. The assessment of secure coding standards was guided by the documents of the Open Web Application Security Project [START_REF]OWASP Homepage[END_REF]. The two experts audited the submitted projects for possible injection attacks and sufficient logging and monitoring, as well as security configuration. However, ensuring security of the system and verifying that security goals have been met was outside of the scope of the assessment.

Results

Five agile software development teams (team A, B, C, D and E) that fulfilled predefined technical requirements were invited to take part in one day workshops. Reykjavik City wanted to assess both how the teams performed during the workshops and the quality of the deliverables they handed in. The three performance factors were: 1) team collaboration and user experience (UX) focus, 2) user stories delivery and 3) the quality of the code. The performance factors were assessed by four experts, two experts assessed the team collaboration and UX focus during the workshops, and two experts assessed the user stories delivery and the quality of the code after the workshops by reviewing the deliverables. The results from the assessment of the performance factors are shown in table 1.

Team A got the highest number of points in total for the three performance factors. This team had an interesting approach. They only focused on one user story, which was user story 1, during the workshop, but all the other teams selected more than one user story to focus on. This is why Team A got the lowest number of points for the user stories delivered. The user story that Team A selected was the only story that included the citizens of Reykjavik, the other three user stories included employees of Reykjavik city.

Table 1. The total points that each team received for the three performance factors evaluated Performance factor Team A Team B Team C Team D Team E Team A got the highest number possible for team collaboration and UX focus. This was the only team that contacted a domain expert to understand the needs of this particular group of citizens. They called a person at the service center and interviewed her/him to enhance their understanding of the needs of the user group. One of the team members also went to the service center, which was in the same building, and tried out how the application process was during the day of the workshop. The other teams did not contact any people outside the team for gathering information on the users and only imaged how the users would behave. These teams got much lower score on the user experience focus sub factor than team A.

The results on the team collaboration factors were more similar for the teams, but still there were some differences. For some teams we did not see much communication during the daily Scrum meeting and the organizing meeting, so the team members did sit by their computers and work individually. This is against the fundamental rules of agile, where team communication and collaboration is vital [START_REF]Manifesto for Agile Software Development homepage[END_REF].

The aggregate score for the quality of the code had much less variation in the rating. Teams A, D, and E received almost the same score on code quality. Each of these teams were very competent. The experts observed some differences in each of the 8 categories among these teams but the differences averaged out.

Team B did not document their code and did not trace decisions to requirements and stories. Exceptional behavior was not handled, and no tests were provided. Team C did not document parts of their code well, had many non-descriptive messages like "log in stuff" as commit messages to their version control systems, and did not take care of exceptional code paths. One error message displayed to the user was: "An unexpected error happened" and some errors were silently ignored. They aimed to implement three of the four stories, but only managed to finish two of them. Team D worked on a technical level, planning to implement all the user stories with a high standard of quality. At the same time, they chose the simplest stories. Team D and E received the same scores on code quality but aspects of code quality differed, e.g., team E had worse documentation of their process and the code, but handled web accessibility, error handling, and software architecture better than Team D. To summarize the findings, it was surprising for all the experts how much variation there was in how the teams worked and what they delivered. All the teams included IT professionals with the technical requirements fulfilled. Team A got the job since they got the highest score of the summary of all the performance factors and their prize estimations were in line with the other teams, so they got the highest total score and the job. They were the only team that reached out to understand the users of the service, while focusing on the code quality in parallel.

Lessons Learned

In this section we will summarize the lessons learned for taking part in the assessment of these performance factors.

For assessing the performance of the teams during the workshops it was essential to have the questions behind each sub factor to decide what to focus on during the assessment. Some of the sub factors were harder to assess than others. The sub factor on how the planning meeting for the day was conducted was rather straight forward to assess, since there are procedures on how a good meeting should be conducted from the literature. The experts observed, for example: if all team members were active, if it was clear who was doing what and if they prioritized the user stories. The sub factor on how problem solving oriented the team was, was maybe the hardest to assess, since there is not one activity or time frame, where this should happen. This is more like an assessment of the underlying values in the team and is therefore harder to assess.

For planning the assessment there were a lot of work done on deciding and describing the assessment criteria. This is vital in such work. A recommendation to readers that would want to conduct a similar process is to plan the assessment well and focus on issues that are valuable, but rather straight forward to assess. Some of the technical measurements had little impact on the overall assessment. For example, the experts concluded that the commit messages had no impact on the total result. This is certainly a result of the workshop format. The experts looked at the commit messages that the teams generated during the day. Four out of five teams focused on implementing as many story points as possible. Therefore, the experts have observed the same poor quality of comments and gave the same grade. The second recommendation to readers is not to assess this factor using the work of a single day.

The quality of web accessibility was assessed. The experts concluded that this score was mostly determined by the choice of the web framework used by each team. The experts observed that little was done to provide accessibility beyond what the framework provided. In this particular project many of the users of the web solution have special needs. For this reason, we recommend to include and emphasize user stories that involve users with special needs when planning such workshops, e.g. a visually impaired user or a colorblind user. The experts commented that while this may make the teams display skills in web accessibility, this comes with the caveat that the assessment may include the same factor twice: as a technical score and as part of the story points' delivery.

Error handling in the interface was not a priority of the teams during the day either. This was surprising to the experts. Input validation was not consistently performed, and invalid inputs were too often reported in obscure ways or not at all. Cryptic error messages in English (not the language of the interface) and stack traces were displayed too often. While the experts gave low scores for this factor, they cannot recommend to ignore this.

The technical factors that had biggest impact on the selection of the agile team were the documentation in the code and error handling in code. The experts could observe that the teams that attempted to document their architecture, the activities their system performed, and that tried to trace code to requirements produced better code. Those teams also tended to handle errors well in code, typically by logging the error and displaying an error message. Again, care has to be taken to not account a deficiency twice. The experts graded an error handled well in code even if the result was a stack trace displayed in the user interface.

A summary of the recommendations includes:

 The planning of the assessment should be done thoroughly and carefully, keeping in mind the values that the agile team should focus on during the actual project. In this particular project many of the users of the web solution have special needs. Therefore one of the sub factors assessed was the emphasis of the teams on user experience during the workshops.

 Meaningful stories should be provided to the teams that involve users with special needs. For these users the focus on accessibility and user experience is very important. Assessing the teams focus on these quality factors is vital in these cases. If the teams do not focus on the quality aspects for these users they are not likely to focus on these aspects for other users.

 If the work of a single day workshop is assessed,, the assessment should include the focus on the documentation and how it manages to trace code to requirements,  The assessment should include to check how the application handles exceptional cases and how it provides feedback to the user in exceptional cases.