Olga Druzhinina

Ebba Thora Hvannberg

Feedback in a Training Simulator for Crisis Management Compared to Feedback in a Real-Life Exercise

Keywords: Feedback, work analysis, human computer interaction, interaction design, virtual environment, training simulator, crisis management training

Designing feedback to trainees in a training simulator for complex cognitive domains is demanding but has not been given adequate attention. This paper aims to understand the importance of early work analysis in a real context during the design of such a simulator. More specifically, the aim of this research was to learn whether there were differences between the types of feedback given to the learner in an already planned design and a real-life training exercise. Therefore, a comparison between feedback given to trainees in a reallife training exercise and feedback planned for a virtual environment supporting training of crisis management was made. The results showed that there were several significant differences.

Introduction

While training for crisis management in the advent or after an airplane crash is vital, it is complex and resource demanding, involving many organisations such as medical, police, rescue, airlines and airport operators. The tasks of operational units involve field work, such as medical aid, transportation of casualties and triage of wounded.

Coordination and decision making are also important tasks [START_REF] Stolk | Gaming and multimedia applications for environmental crisis management training[END_REF]. Training is carried out regularly in comprehensive real-life exercises, with tens or even several hundred participants, trainees, instructors and organizers. Trainees representing emergency personnel engage in activities where fast situation assessment and reaction are the main tasks [START_REF] Hansen | System Master Plan -Deliverable 3.3. of the CRISIS project[END_REF]. To keep the cost down, while maintaining the level of training requirements, simulators have been developed [START_REF] Raybourn | Applying simulation experience design methods to creating serious game-based adaptive training systems[END_REF][START_REF] Rudinsky | Consolidating models of requirements analysis for crisis management training simulator[END_REF]. Designing such simulators is a difficult task, since crisis management requires not only knowledge domain of managing crises, but also skills that enable workers to respond to unexpected and variable events and situations which neither may have apparent causes or foreseeable consequences to reactions [START_REF] Stolk | Gaming and multimedia applications for environmental crisis management training[END_REF]. Another aspect of the design of such a simulator is to decide the instructional design. Instructional design is a systematic practice of analysing learning requirements and based on them, designing a set of instructional specifications [START_REF] Van Merriënboer | Blueprints for complex learning: The 4C/ID-model[END_REF]. Instructional design models span over the lifecycle of an educational artefact through the stages of planning, designing, developing, implementing and updating where applicable. The instructional design of a crisis management simulator needs to provide trainees with experiential learning, variable uncertainty of events and situational complexity [START_REF] Wong | The Variable Uncertainty Framework[END_REF].

Crisis management involves teamwork in a safety critical domain requiring actors to collaborate and communicate to manage information, make decisions, plan and delegate tasks. Teams are formed at different levels. There are organizational teams, e.g. fire fighters, medical personnel and police, and there are actors from these organizations which form teams at various posts, e.g. at command centres or at the accident site. In crisis management simulation, the activities of trainees are divided into two main categories: field and command [START_REF] Hansen | System Master Plan -Deliverable 3.3. of the CRISIS project[END_REF] [START_REF] Salas | THE SCIENCE OF TRAINING: A Decade of Progress[END_REF] [START_REF] Hansen | System Master Plan -Deliverable 3.3. of the CRISIS project[END_REF] [START_REF] Hansen | System Master Plan -Deliverable 3.3. of the CRISIS project[END_REF] [START_REF] Hansen | System Master Plan -Deliverable 3.3. of the CRISIS project[END_REF] [START_REF] Hansen | System Master Plan -Deliverable 3.3. of the CRISIS project[END_REF] [START_REF] Hansen | System Master Plan -Deliverable 3.3. of the CRISIS project[END_REF]. Field activities are carried out by trainees representing emergency personnel, who aim to learn to assess and respond to a situation as effectively and quickly as possible. These activities mainly involve rescue, triaging casualties, providing medical aid, transporting resources and casualties. Command activities involve the command and coordination staff, who is responsible for planning, strategizing, decision making and coordinating on a larger scale. Command activities include organizing resources, e.g. people, transportation, medical aid and hospital resources, mitigating risks, assessing situations, responding to them and planning [START_REF] Hansen | System Master Plan -Deliverable 3.3. of the CRISIS project[END_REF]. A wide range of expertise appears during training and operations of crisis management, where responders and commanders include novices to experts [START_REF] Dreyfus | Peripheral Vision: Expertise in Real World Contexts[END_REF]. Furthermore, to add to the heterogeneity some are professionals but others are volunteers from search and rescue organizations, Red Cross etc.

In addition to simulating work and situations, simulators need to include a stimulating and rewarding training environment for trainees. Experience has shown that during simulator tenders technical issues are prioritised over functionalities for supporting training [START_REF] Naikar | Work Domain Analysis for Training-System Definition and Acquisition[END_REF]. To encourage better practices, Naikar and Sanderson [START_REF] Naikar | Work Domain Analysis for Training-System Definition and Acquisition[END_REF] have illustrated how work domain analysis [START_REF] Vicente | Cognitive Work Analysis, Toward Safe, Productive, and Healthy Computer-Based Work[END_REF] can be extended to define training needs. They have argued that common techniques such as Instructional System Development (ISD) [START_REF] Kirby | Instructional systems design and the learning sciences: A citation analysis[END_REF], which has been applied extensively by the air force in military training, is inadequate since they do not support work that is characterised by cognitive work, adaptive behaviour, random events and unpredictable situations.

Besides identifying training needs, designers of simulators have to decide how to deliver training. Training needs to be planned, carried out and assessed. Much has been written on professionals' reflection in action [START_REF] Schön | The Reflective Practitioner: How Poressionals Think in Action[END_REF] and practitioners' ability to self-assess or self-monitor [START_REF] Eva | I'll never play professional football" and other fallacies of self-assessment[END_REF]. Feedback is one of the most powerful techniques to help students learn. It provides learners with information on their performance when gaining or applying new knowledge and can help them finding the right answers [START_REF] Alvarez | An Integrated Model of Training Evaluation and Effectiveness[END_REF][START_REF] Salas | THE SCIENCE OF TRAINING: A Decade of Progress[END_REF]. Feedback is an essential aspect for any learning environment, as it provides assessment of the learner´s progress and results, motivates the learner, and enhances learning [START_REF] Cyboran | Designing feedback for computer-based training[END_REF]. Either feedback can be intrinsic, that is available in a normal job situation, or extrinsic, which is available from an instructor either during or after a training exercise [START_REF] Hawkins | Human Factors in Flight[END_REF]. The latter one is also termed explicit feedback [START_REF] Hays | Feedback Specificity and the Learning of Intercultural Communication Skills[END_REF] which can come from a human tutor, a pedagogical agent, disembodied coach or other intelligent tutor [START_REF] Lane | Promoting Metacognition in Immersive Cultural Learning Environments Human-Computer Interaction. Interacting in Various Application Domains[END_REF]. Feedback can be provided by peers and is one of the five components of teamwork [START_REF] Entin | Measures for Evaluation of Team Processes and Performance in Experiments and Exercises[END_REF][START_REF] Cannon-Bowers | Team performance and training in complex environments : Recent findings from applied research[END_REF]. Since teamwork is prevalent in crisis management, looking at such feedback becomes even more important than feedback provided in a single learner situation. As mentioned above, real-life training has many drawbacks, e.g. the costs of equipment and organization, and menace to environment [START_REF] Stolk | Gaming and multimedia applications for environmental crisis management training[END_REF]. However, it remains unclear whether the level of feedback delivery in a simulation, an essential part of training, may be the same or comparable as in a real-life exercise.

To meet this requirement of learning through feedback, most educational systems need to provide information on the assessment of learning to the trainee in some form [START_REF] Moreno-Ger | Educational game design for online education[END_REF], whether it is given by the system, or by a person, e.g. either a teacher/instructor or a fellow student. Although a simulator for crisis management training is not a game, we expect that knowledge from that field can be adopted. Researching designs of learning in games, Liu et al. [START_REF] Liu | Development of a Multiplayer Online Role-Playing Game-based Learning System for Multiple Curriculums[END_REF] state that assessing the progress of a player´s learning activity is one of the most rewarding interactive behaviour. To assist the player in the learning experience [START_REF] Kiili | Digital game-based learning: Towards an experiential gaming model[END_REF], such systems should provide appropriate feedback and they should ensure continuous flow to motivate the player to proceed [START_REF] Paras | Game, Motivation, and Effective Learning: An Integrated Model for Eucational Game Design[END_REF].

Despite the substantial emphasis on including feedback in games or simulators, it seems that its design, or methods for its design, has not been given as much room as required [START_REF] Renaud | Feedback in Human Computer Interaction: Characteristics and Recommendations[END_REF]. However recently, there have been examples of methods to provide ingame assessments of objects and actions taken in real time. These assessments correspond to logged, time-stamped events that undergo statistical analyses that are displayed in real time or during a review after the game [START_REF] Raybourn | Applying simulation experience design methods to creating serious game-based adaptive training systems[END_REF].

Motivated by the above need, the objective of this research is to compare, in terms of feedback, a plan for designing a training simulator to data gathered in a real-life training exercise for responding to an airplane crash. Thus, through an extensive analysis of the crisis management work domain and training exercises, we hope to learn about the kinds of feedback learners receive during training. The rest of the paper is organised as follows. After describing forms of feedback as they appear in the literature in the next section, the methodology of the research study is detailed in section 3. Section 4 describes the results of the research study which is a case study of a comparison between two different training environments. In the discussion of the results in section 5 we compare them with a feedback framework and suggest a theory of feedback. The paper concludes with section 6 presenting its contributions.

Forms of Feedback

Feedback in any learning environment may be categorized [START_REF] Cyboran | Designing feedback for computer-based training[END_REF]. For example, feedback may be qualitative or quantitative, e.g. "Good", or "65 points" [START_REF] Connellan | Sustaining Knock Your Socks Off Service AMACOM[END_REF]. It may be negative, positive or neutral [START_REF] Renaud | Feedback in Human Computer Interaction: Characteristics and Recommendations[END_REF][START_REF] Spink | Human-computer interaction in information retrieval: nature and manifestations of feedback[END_REF]. Also, feedback can be corrective and/or motivational, where corrective feedback has the purpose to correct the learner when he/she has made a mistake, while motivational feedback motivates the learner, based on his/her individual preferences [START_REF] Pyke | A Closer Look at Instructor-Student Feedback Online: A Case Study Analysis of the Types and Frequency[END_REF]. According to Pivec and Dziabenko [START_REF] Pivec | Game -based learning in universities and lifelong learning : UniGame : Social Skills and Knowledge Training game concept[END_REF], learning by mistake is a primary way for trainees to learn and motivating them to keep on trying.

Feedback may be in a form of elaboration or verification. Verification feedback refers to a notification of the result of the learner´s answer, i.e. whether it is correct or not. Elaborative feedback provides an explanation of the result of the learner´s answer and can give a hint to a solution [START_REF] Narciss | How to Design Informative Tutoring Feedback for Multimedia Learning[END_REF][START_REF] Pridemore | Control of feedback in computer-assisted instruction[END_REF].

In a team, the process of providing feedback is not the same as in single person-toperson communication. According to Sonntag [START_REF] Sonntag | Teleteaching: From Unidirectionalism to Multidirectionalism[END_REF], in unidirectional teaching environment, where feedback is teacher-to-student and vice versa, feedback is a simple process, while in multidirectional environment (in a team) feedback becomes a more complex process. While Sonntag has looked at the integration of aspects within the teaching environment, the issues raised are relevant to a training simulation. To pass the right message onto students, Sonntag has emphasised a teacher's coordination of students and organization of materials and feedback in a multidirectional environment. The main concern is that students may misinterpret, underestimate or ignore bits of information [START_REF] Sonntag | Teleteaching: From Unidirectionalism to Multidirectionalism[END_REF]. In a similar manner, crisis trainees may misinterpret or not respond to information from an operational unit. Or, for example, if a trainee receives contradictory information from two other trainees, s/he may be left confused unable to proceed in such a case. Thus, unlike unidirectional feedback, multidirectional ones hold a rather arduous character due to the distributed interaction nature of the environment. Therefore, when designing scenarios, an important aspect of crisis management training is learning how to deal with wicked problems due to multi-directional feedback. Since unidirectional feedback occur in crisis handling as well, the two need to be distinguished. Another type of a teaching strategy is called peer-review, an approach when student-to-student interaction is used to provide feedback and advice [START_REF] Richer | A video game-based framework for analyzing human-robot interaction: characterizing interface design in real-time interactive multimedia applications[END_REF]. Peer-review feedback exhibit similar complex behaviour of a multi-directional environment and poses a risk of credibility of information provided by the student.

Above, we have described some types of feedback in the learning context. More generally, feedback is provided in conversations, and more specifically in the context of systems, such as in the context of human computer interaction, where feedback is a report issued by the system as either a reply to the action of a user or a notification of the progress of a user's task [START_REF] Perez-Quinones | A collaborative model of feedback in humancomputer interaction[END_REF]. Hence, when discussing feedback in the context of learning systems, all these aspects come into play.

Looking more closely at the system aspect, feedback in a virtual environment can be categorized according to senses, i.e. visual, haptic, or sound [START_REF] Liu | Development of a Multiplayer Online Role-Playing Game-based Learning System for Multiple Curriculums[END_REF][START_REF] Kiili | Digital game-based learning: Towards an experiential gaming model[END_REF][START_REF] Richer | A video game-based framework for analyzing human-robot interaction: characterizing interface design in real-time interactive multimedia applications[END_REF]. Further, there are two types of feedback in virtual environments, either immediate (real-time) feedback that can be issued to the player during the game or delayed or archival feed-

Yes

Delayed or archival

After the training session

Yes back that is issued as a summary upon completion of a task or a game [START_REF] Cyboran | Designing feedback for computer-based training[END_REF][START_REF] Ahn | Designing games with a purpose[END_REF]. Immediate feedback which is automatically generated by an application can be of any form mentioned above, while delayed feedback is usually given in text or graphical, form [START_REF] Cyboran | Designing feedback for computer-based training[END_REF][START_REF] Liu | Development of a Multiplayer Online Role-Playing Game-based Learning System for Multiple Curriculums[END_REF][START_REF] Renaud | Feedback in Human Computer Interaction: Characteristics and Recommendations[END_REF]. For example, action feedback is an immediate interactive response in real-time, while text feedback can be both immediate and delayed [START_REF] Liu | Development of a Multiplayer Online Role-Playing Game-based Learning System for Multiple Curriculums[END_REF]. Immediate feedback may be provided during the execution of a task, usually when the task is complex, or after task completion if the task is relatively straightforward [START_REF] Renaud | Feedback in Human Computer Interaction: Characteristics and Recommendations[END_REF]. For example, in a car simulator, the player may get immediate haptic feedback in the form of vibration during the execution of a task.

Liu et al. [START_REF] Liu | Development of a Multiplayer Online Role-Playing Game-based Learning System for Multiple Curriculums[END_REF] and Kiili [START_REF] Kiili | Digital game-based learning: Towards an experiential gaming model[END_REF] state that feedback is accepted more seriously by the player if it is provided immediately, instead of being issued after the game completion in the form of instructional text material. Also, according to Garzotto [START_REF] Garzotto | Investigating the educational effectiveness of multiplayer online games for children[END_REF], when an action taken by the player is incorrect, it may be necessary and more effective to provide immediate than delayed feedback, since the player might later forget the incident, and, hence, not learn from delayed feedback. However, Renaud and Cooper [START_REF] Renaud | Feedback in Human Computer Interaction: Characteristics and Recommendations[END_REF] argue that archival feedback is also important, and may be presented in the form of text and/or graphically. Where appropriate, graphical feedback is more helpful than text, as it depicts the information as a whole [START_REF] Renaud | Feedback in Human Computer Interaction: Characteristics and Recommendations[END_REF]. Furthermore, feedback may be provided as an action replay in the form of a video [START_REF] Renaud | Feedback in Human Computer Interaction: Characteristics and Recommendations[END_REF]. Table 1 summarises the different forms of feedback described in this section divided into two categories depending on the source of introduction. By no means are the categories mutually exclusive, e.g. a qualitative feedback can be positive and delayed.

Research Methodology

The research method is a work analysis of trainees' performance during the real-life exercise of emergency management. The feedback in the real-life exercise is compared to a system master plan of a virtual environment of a crisis management training simulator using categories from the literature. The real-life exercise training was conducted at an international airport in Europe and included almost 250 participants, with 90 participants creating the training scenario and serving as instructors, and the rest training for crisis management. The main exercise lasted 4-5 hours and included all organizations involved in a regular response to an aircraft incident. Seven participants of the CRISIS project served as observers, taking written notes and photos. The CITE system from VSL was used to enter notes during and after the exercise. In total, 548 observations were made for four hours at five different locations of the exercise: Emergency Operations Center (EOC), Casualty Assembly Point (CAP), Rendezvous Point (RVP), On Scene Command (OSC), and Accident Site. In addition, a few observations were made by instructors of the exercise, who were using the CITE system as a pilot. Partners of the CRISIS project have collaboratively developed a virtual environment training simulator (CRISIS VE) for responders and commanders responsible for crisis management. The target sector is transport security, such as at for airport and railway operations. CRISIS VE was designed to support the 4C/ID instructional design model, which is meant to be used for long-period training with multiple complex tasks and learning objectives [START_REF] Narciss | How to Design Informative Tutoring Feedback for Multimedia Learning[END_REF]. The 4C/ID instructional design model focuses on complex learning tasks and comprises four components, a) learning tasks, b) supportive information, c) just-in-time information and d) part-task practice [START_REF] Van Merriënboer | Blueprints for complex learning: The 4C/ID-model[END_REF]. Early on in the project, a system master plan (SMP) was written of the CRISIS VE including a detailed description of the system design and architecture, description of key concepts, such as events, variable uncertainties, and the CRISIS training cycle. Furthermore, the system master plan includes reference scenarios and example scenarios. Finally, it contains a list of roles for each of the scenarios and extensive lists of func- tional requirements for different modules and technologies of the simulator. Comprising well over 300 pages, the system master plan, is written after a thorough requirements analysis including site visits, interviews, questionnaires and a visit to one reallife exercise.

The classification of the feedback data from the real-life exercise and its analysis was carried out independently by two researchers. Each researcher classified data according to different feedback types, which are based on an analysis of the literature presented in section 2. Afterwards, they discussed and merged their results. Out of 548 observations noted down by observers, 49 observations on feedback were made.

4

A Case Study

An Analysis of a System Master Plan with Respect to Feedback

In this section we describe an analysis of the SMP [START_REF] Hansen | System Master Plan -Deliverable 3.3. of the CRISIS project[END_REF] to see what requirements have been specified for the feedback available to the trainee. CRISIS VE supports functions to diagnose and to make a full After Action Review (AAR) of a trainee's action, decision and behavior based on data collected throughout the simulation game. Thus, the AAR module provides a quantitative summary report of the trainee´s overall performance and details of an exercise. Based on this data, the instructor submits his/her own qualitative comments on the trainee´s performance. CRISIS VE supports immediate visual and action feedbacks. CRISIS VE supports immediate visual feedback, e.g. when a trainee places a Casualty Assembly Point (CAP) at a location, the system immediately draws a CAP object on the screen. An example of how CRISIS VE supports immediate action feedback is when the system prevents the trainee from placing the CAP in an area too close to the site of an accident, since it would endanger the casualties in case of fire spreading. Thus, the system prevents the trainee from making a mistake, but signals him or her implicitly that an incorrect action has been carried out. Furthermore, CRISIS VE facilitates, by supporting communication with fellow trainees, learning by mistake. Trainees will be able to receive real-life verbal feedback through communication devices implemented in the simulator, such as radio or mobile phone, from other trainees or instructors. CRISIS VE also supports both unidirectional and multidirectional feedback between the trainee and the instructor, as well as peer-reviewing between all trainees.

CRISIS VE does not support sound as a form of feedback. Although it includes sounds to resemble the real-life environment [START_REF] Rudinsky | Designing soundscapes of virtual environments for crisis management training[END_REF], it does not notify the trainee of the success or failure of a task completion. While some research is underway for using haptic equipment, i.e. for sensing heart beats of casualties [START_REF] Einarsdóttir | Simulation of Hearbeat, Respiration and Heat using Haptic Feedback and Thermal Unit Faculty of Industrial Engineering[END_REF], the SMP does not include such support for physical sense feedback. Error! Reference source not found. presents an overview of the feedback forms found in SMP.

An Analysis of a Real-Life Exercise with Respect to Feedback

Using the data from a real-life exercise, this section describes feedback a trainee gave another trainee or an instructor gave a trainee or trainees. We have put feedback types, either from the literature or the real-life exercise data, in bold. Quotes from the data are presented in quotes and italics. Error! Reference source not found. gives an overview of the feedback types. All 49 instances of feedback during the real-life exercise were verbal and qualitative. An AAR was presented to trainees verbally at a meeting after the event, but a written report was planned as is customary in such exercises.

Trainees were not rewarded, i.e. they were not provided with positive feedback, upon the successful completion of a task. Although there were many examples of verification feedback [START_REF] Entin | Measures for Evaluation of Team Processes and Performance in Experiments and Exercises[END_REF], such as when one commander tells the other that they "need better information on weapons", most instances were of elaborative feedback type [START_REF] Pivec | Game -based learning in universities and lifelong learning : UniGame : Social Skills and Knowledge Training game concept[END_REF] with an explanation of the error or the problem, such as when "airline operator discusses with Red Cross commander how to send the list" after feedback had been provided on not receiving the list although, reportedly, it had been sent. Another example of verification feedback was when the commander made decisions and organized the other trainees after having concluded: "transport capacity is not enough to transport all red casualties". Elaborative feedback occurred during every status meeting.

Many of the feedback during the exercise were of a corrective character [START_REF] Cannon-Bowers | Team performance and training in complex environments : Recent findings from applied research[END_REF]: "Wounded still on the scene", implying that they should not be on the scene, and only a few feedback instances were motivational in nature [START_REF] Kirby | Instructional systems design and the learning sciences: A citation analysis[END_REF]: "EOC commander concludes that the transport capacity is not enough to transport all red casualties. She asks Resource manager for more resources". Another example of corrective feedback was when an instructor stated that "ambulances should already be on the scene, but they are not". Some of the feedback is only unidirectional [START_REF] Alvarez | An Integrated Model of Training Evaluation and Effectiveness[END_REF], e.g. in the case of providing an assessment of the situation, at RVP: "It took too long time for activating resources from the Rendezvous point to the accident site", or at the OSC post: "[We] need basic information on the status of the triage immediately in the beginning". A further example implies negative feedback when commander states "there should be better flow". Another example of negative feedback happened when at the EOC it was observed "NECC (National Emergency Coordination Center) has complained about not receiving a passenger list". Feedback incidents that took place during the status meetings were peer-reviews with trainees explaining their current state of work and problems. Most (39) feedback instances were peer-reviews, as they involved different trainees trying to solve one task. For example: "EOC commander concludes that the transport capacity is not enough to transport all red casualties. She asks the Resource manager for more resources. The radio operator informs that OSC has not requested resources, only to move existing resources closer". There were no multidirectional feedback instances.

In two cases we noted feedback in the form of explicit teaching. OSC realised that the rescue coordinator had misunderstood the role of the OSC. In response, they attempted to inform the coordinator of the correct procedures: Commander to Planning: "How is the flow?" -Looking at an overview at the display. Planning to Resources: "Do you want to have the ambulances closer?" Commander says: "Transport coordinator should resolve this" -explains the procedure and role of transport coordinator.

Another example of teaching trainee procedures was when a commander at OSC said: "I want to iterate that EOC should organize transport of casualties out of the scene -this is not the responsibility of OSC but an EOC."

Planning personnel at OSC communicated to the EOC: "You should talk directly to the transport coordinator" -Commander looks at the hierarchy with plans and says that according to the hierarchy, the medical coordinator and the transport coordinator should organize the transport among themselves.

Probing for information was noted as a trigger for feedback. At the OSC at a start of regular status meetings: "We don '

t have confirmation that the scene has been secured. There they come (i.e. the rescue team) (Looking through the window). How much of rescue teams did he send? Do we have numbers from security coordinator?"

During the real-life exercise, trainees were constantly burdened with various hiccups that persisted along the way, such as the equipment failure or data loss. For example, "the police officer reports a radio problem and asks for assistance for fixing it or replacing". This is important to mention since it could play a big role in determining the triggers for feedback in a simulation. There were other important triggers for feedback that took place during a real-life exercise.

Many feedback instances were triggered through discussions (22 out of 49) and information sharing [START_REF] Cristancho | A framework-based approach to designing simulation-augmented surgical education and training programs[END_REF], e.g. during regular status meetings when numbers of casualties were reported. Many instances triggered reasoning [START_REF] Cannon-Bowers | Team performance and training in complex environments : Recent findings from applied research[END_REF] and organization [START_REF] Liu | Development of a Multiplayer Online Role-Playing Game-based Learning System for Multiple Curriculums[END_REF] among participants of an exercise, e.g. when the commander made decisions and organized the other trainees after having concluded that transport capacity was not enough.

More examples of when feedback initiated a discussion were triggered after an issue had been raised [START_REF] Schön | The Reflective Practitioner: How Poressionals Think in Action[END_REF]. At the EOC there was an incident when "The radio/computer operator informs that NECC has been trying to reach EOC. The EOC commander replies she is about to contact them". At the EOC, another example which showed feedback which raised an issue: "We need transport capabilities from [City excluded for the sake of anonymity] -there is nothing on this in the database. This is a problem".

It is also important to mention that on some occasions the feedback resulted in a learning outcome. For example, in two feedback cases the trainee was learning by mistake as a result of feedback, e.g. a trainee learned not to inform patients of the death incidents as they became "upset when hearing about dead patients over a nearby tetra unit".

A Comparison between the Planned Design and the Real-Life Exercise

In the previous section, we presented how trainees received feedback of different types. In this section, we discuss how the design of a real-life exercise may contribute to the design of a simulation in terms of credibility of learning, by comparing it to the planned design.

Although the purpose of feedback in real-life exercise and CRISIS VE is to train participants to respond in a crisis situation, some of the feedback practices in CRISIS VE contradict with the real-life exercise. There are differences in the delayed feedback during the real-life exercise and in the training simulator. The archival feedback provided in the crisis simulator is only available in text form, and it is quantitative with the possibility that the instructor may provide qualitative comments on the performance. In the real-life exercise, the feedback which is given to the trainees as an aftermath of their performance is qualitative and is presented verbally or as text.

The immediate feedback in the real-life exercise is provided by trainees, usually in the form of elaboration, both during the task execution and/or after task completion. In the training simulator, immediate feedback is provided in the form of action by the system, and can be received through conversing with other players via communication devices. The difference here is that the trainee does not get any immediate automatic feedback in the real world, so if s/he makes a mistake, it will have consequences at a later stage of the training. The system, however, tries to prevent the trainee from making a mistake without explanations, and with no further consequences. For example in real-life, if a trainee breaks something then other trainees will have to continue without it, and will have to use another object instead as a replacement.

While CRISIS VE supports some learning by mistake function, it does not provide feedback to the trainee on the nature of a mistake. For example, in the case of placing a CAP area too close to the accident site, the simulator does not allow for the CAP to be placed. There is no explanation on why the CAP cannot be placed. Thus, the trainee may be unsure whether it is a fault in the software or if the reason is the proximity to the accident site. Although in real-life a trainee does not get any messages of the sort, in the training environment s/he may become confused if there is no message or alert generated when something is happening unexpectedly. Cyboran [START_REF] Cyboran | Designing feedback for computer-based training[END_REF] has explained that learners do not benefit when the system does not specify its response towards learner´s action, particularly when the nature of response may not be clear or obvious. Analysing the system master plan it was unclear if incorrect actions or mistakes were logged for the AAR report.

Furthermore, the work analysis revealed that in real-life many feedback incidents are connected to unexpectedness of situations, particularly equipment failures. However, in the master plan for the CRISIS simulator, there are only some vague suggestions of triggers for unexpected incidents. While the master plan has a general design of feedback and unexpected situations, the training exercise showed that in real-life there are several typical situations that would need specifically designed feedback.

Discussion

In this paper we showed how feedback design may be overlooked if it is not supported by a work analysis that aids in explaining the relationship between feedback and learning during crisis management training. The current planned design reveals that the design of feedback, one of the most important learning tactics in education, contradicts with the real-life practices of crisis management training.

The work analysis has shown that during the real-life exercise feedback incidents, similar to typical of crisis management training, occurred. Such similar feedback incidents have been classified and grouped, and can be used in the design of a simulator for crisis management training. In preparation for the real-life exercise, exercise organisers said they were not expecting a lot of feedback given to trainees by instructors. The analysis reported in this paper showed that although not extensive, feedback was more apparent than expected. In future studies, it would be interesting to contrast this result to trainees' reflection on their own work [START_REF] Schön | The Reflective Practitioner: How Poressionals Think in Action[END_REF] as professionals or in an after action review and see how trainees expertise might determine this reflection vs. feedback.

We analysed what the feedback implied in terms of learning in a real-life exercise, i.e. whether trainee would gain knowledge as a result of this feedback or get the message that training needs to be improved in the future. For example, for assessment of a situation, a fellow trainee pointed out to the trainee that the situation was deficient, and that in future situations this would need be avoided or improved. Or, for raising an issue or probing for information, the feedback indicated to the trainee that a prob-lem had arisen that could be solved now but little or no instructions were provided. Or, in the case of explicit teaching, the trainee received instructions and guidance on how to accomplish a goal.

For this research, we have gathered empirical data to show how work analysis can incorporate and shape the design of the feedback incidents through comparison of the work analysis and an earlier design plan. In comparison, Narciss and Huth [START_REF] Narciss | How to Design Informative Tutoring Feedback for Multimedia Learning[END_REF] carried out research, based on literature, to derive guidelines on how to design feedback incidents. In their research, they emphasise how the cognitive task analysis plays a crucial part in the design of feedback.

Narciss and Huth [START_REF] Narciss | How to Design Informative Tutoring Feedback for Multimedia Learning[END_REF] proposed three facets of feedback, functions of feedback (cognitive, meta-cognitive or motivational), contents of feedback (evaluative or informative towards the knowledge of the result) and presentation of feedback contents (timing, schedule, adaptivity). Based on this framework, procedures are detailed on how to design feedback. Instruction contexts, such as objectives, tasks and errors contributed to the contents of feedback; and individual factors such as learning objectives, prior knowledge and skills and academic motivation contributed to functions of feedback. Narciss and Huth [START_REF] Narciss | How to Design Informative Tutoring Feedback for Multimedia Learning[END_REF] viewed the contents of feedback in terms of evaluative and informative components. Comparing their classification to ours, the assessment of a situation and raising an issue or probing for information may be considered as evaluative components, while explicit teaching falls in the category of an informative component.

According to Gredler [START_REF] Gredler | Educational Games and Simulations: A Technology in Search of a (Research) Paradigm[END_REF], simulations may be of two types: symbolic or experiential. Symbolic simulation presents tasks where a participant must often use or improve theoretical knowledge to solve a problem, e.g. business manager, physician, or mathematician. Experiential simulation presents tasks where a participant is trained to gain knowledge of how to operate a device or perform a set of routine tasks, which require little to no need for scientific concepts or theorems [START_REF] Gredler | Educational Games and Simulations: A Technology in Search of a (Research) Paradigm[END_REF]. Since Narciss and Huth studied learning environments for a symbolic type of simulator where trainees solve problems by presenting knowledge of the result, such as subtraction, we do not know if their findings can be directly translated to crisis management training, which is an experiential simulator. Nonetheless, it can be valuable to compare the two types to see how well the design guidelines can be translated from one type of learning environment to another.

The research has some limitations. One limitation was that the observers may not have reported all of the incidents that occurred. Although the data set that we analysed was large, it was only from one real-life exercise. Thus, the feedback incidents retrieved are specific to that particular training scenario.

Additionally, for more detailed modelling of a learning environment in a simulation-based training, a study of gathering feedback from actual trainees, and not just observers, may need to be carried out. Castor et al. [START_REF] Castor | Decision processes in simulation-based training for ISAF vehicle patrols[END_REF] performed a study for the purposes of gaining insight into the understanding of the cognitive processes of the trainees. The study was experimental in nature, and no well-defined model was derived as an outcome, although suggestions of cognitive modelling approaches were made. CRISIS VE could benefit from such an analysis in determining the functional design of artificial intelligence components within the simulator, such as for unexpected events or computer-played characters.

Conclusion

Previous research on feedback seems to be scattered in individual areas ranging from communication to learning sciences, onwards to system related areas such as HCI, educational technologies and learning games. And thus, the theory of feedback in the context of HCI is rather weak [START_REF] Spink | Human-computer interaction in information retrieval: nature and manifestations of feedback[END_REF]. Furthermore, what makes it difficult is that feedback is of varying abstractions along different dimensions, ranging from low lexical level to a high semantic level, from a drill type of training to training more cognitive complex tasks in disperse application domains, ranging from Sports training [START_REF] Iskandar | Pedagogy in Computer-Based Sport Training[END_REF] to Medical training [START_REF] Cristancho | A framework-based approach to designing simulation-augmented surgical education and training programs[END_REF] onwards to Pilot training [START_REF] Hawkins | Human Factors in Flight[END_REF]. This research has shown that it is possible to categorize data from a real-life exercise for use in designing a simulator. Specifically, we showed how this data may be useful in recognizing what practices in real-life are frequent and need to be incorporated, and thus should be given more attention when designing a simulation. For example, many vague suggestions of unexpected situations were provided in a system master plan, while in real-life there were several typical unexpected situations. By comparing data from a real-life training exercise to data from a system master plan, we have shown that it is necessary to pay more attention to learning aspects of a simulator, such as feedback. In particular, since the comparison revealed significant differences between the system master plan and the real-life exercise, the research has shown that while it is vital to carry out analysis of the domain tasks, roles and scenarios, the development of a learning system needs to take into account an analysis of learning tactics provided under a real-life setting.

Table 1 .

 1 Feedback forms as discussed in the literature

		Brief descrip-	Learning	System / Vir-
		tion		tual environ-
				ment
	Qualitative	Textual	Yes
	Quantitative	Numerical	Yes
	Positive	Positive out-	Yes
		come of as-
		sessment
	Negative	Negative out-	Yes
		come of as-
		sessment
	Corrective (ex-	Give a correct	Yes
	plicit)	solution
	Motivational	Encouraging	Yes
		to proceed
	Unidirectional	Teacher to	Yes
		student or vice
		versa
	Multidirectional Teacher to a	Yes
		team
	Peer-review	Among peers	Yes
	Visual/graphical Visual repre-		Yes
		sentation
	Textual	Textual repre-		Yes
		sentation
	Haptic	Touch and		Yes
		kinaesthetic
	Immediate,	Right away by
	automatic	the system

Table 2 .

 2 Feedback forms in the SMP

	Feedback form	Description or purpose
	Qualitative AAR	Comments by instructor
	Quantitative AAR	A summary report
	Immediate Visual	Verification
	Immediate Action	Preventing mistakes
	Verbal feedback	Communication with
		fellow trainees
	Unidirectional	One-way feedback
		comments by instructor
	Multidirectional	Discussion with instruc-
		tor
	Peer-reviewing	Feedback by fellow
		trainees

Table 3 .

 3 Feedback Forms in a Real-Life Exercise

	Feedback forms	Literature / New
	Verbal qualitative	L
	Verification	L
	Elaborative	L
	Corrective	L
	Motivational	L
	Negative	L
	Unidirectional	L
	Peer-reviews	L
	Explicit teaching	N

Acknowledgement

The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no. [FP7-242474].


Luis Velhinho
email: luis.velhinho@gmail.com

Arminda Lopes
email: aglopes@ipcb.pt

A framework in Support of Multimodal User Interface

Keywords: Company Frameworks, Content Management, Graphical User Interface, Information Technologies, Work Analysis

Technologies (IT) Companies' Frameworks have been one of the solutions for the increasing demand for new products and services that are more and more complex and need to be delivered in the least amount of time possible. This paper presents work in progress. The goal was to evaluate Frameworks used by business enterprises and to state the advantages and disadvantages in their use. The chosen methodology for data collection and analysis was the Grounded Theory Method from which it was possible to obtain some conclusions based on given answers. With this research, the intention was is to present some recommendations for companies that work with this type of products to improve their productivity and profit.

Introduction

The business market has sought, in last years, to take out advantage of new technologies as a differentiation factor for better productivity and to reach new customers. This increased demand has brought pressure to the suppliers of IT to provide products and services of greater complexity on a reduced time.

The IT companies meet their expectations supported on their customers' claims, and they found solutions thinking in profit incomes. For many companies one of the ways to be different and to respond to this challenge has been the Frameworks development which allows accumulating knowledge and experience that exist within the company and to improve their products' quality.

According IBM [START_REF] Ibm | Building Object-Oriented[END_REF] "A Framework is a set of blocks of pre-fabricated software that programmers can use to apply, or change specific computing solutions. (…) With the Frameworks the software developers do not need to start from beginning each time they start building an application. Frameworks are built from a collection of objects, so far as the design codes to be reused."

In this research we try to analyze the impact of these types of tools inside companies, and we will try to understand the following concerns:

• To understand the segment markets where the Companies Frameworks are used furthermost;

• To identify the advantages of using Frameworks for enterprise businesses as well as for the development teams; • To understand the decision factors considered for the choice of a framework and the customers' satisfaction; • To detect key requirements in the creation of Frameworks;

• To identify areas of improvement on the Frameworks.

Frameworks

Frameworks can be classified according two dimensions [START_REF] Sauvé | Tipos de frameworks[END_REF]: where the framework is used and how it is used. And they can be regarded in three distinct ways:

• White-Box -where the developer modifies or extends existing functionality by defining or overlapping methods.

• Black-Box -where the developer uses functions already presented in the Framework (the internal modules of the Framework cannot be viewed or altered and interfaces already defined must be used).

• Hybrids -has the same behavior as the White-Box and Black-Box

Frameworks are used by companies with different goals. It depends on the type of use that the company intends to do. In some cases, they only intend to have a tool that helps in a simple project's development, but in other cases, they intend to develop a tool that has a huge amount of data and logic definitions.

Framework Wizy

Wizy is an agile development Framework that we use within the enterprise. It enables rapid creation of web applications based on web front ends (Error! Reference source not found..). It was created as a platform to allow fast development of multiple application projects. Conversely, it offers an entire multi-runtime application with embedded functionality for content management and workflow. The dispersed data is aggregated in a standard manner allowing content management in a flexible way, providing libraries and components that let prompt development for custom applications. Typically, a framework is composed for several organizational layers that allow a higher abstraction view for anyone working with the Framework [START_REF] Johnson's | Frameworks Home Page[END_REF].

Error! Reference source not found.. presents an example of the Framework Wizy structured on 4 layers. The first layer, External Systems, is the whole systems that interact with Framework. The second layer, Content Provider, provides interfaces to interact with internal and external systems. The third layer, the Content Aggregation, is where all contents are aggregated into one default or custom ASPX page. The fourth layer Content Presentation is shared between Server and Client: the server sends the page on XML and applies XSLTs and CSS styles. The client browser interprets the whole data and shows the page to the user.

Research Methodology and Methods

Grounded Theory Methodology was developed by Glaser & Strauss [START_REF] Glaser | The discovery of grounded theory[END_REF]. It is a general methodology for developing theory that is based on data systematically collected and analyzed [START_REF] Strauss | Grounded Theory methodology: An overview[END_REF]. Grounded Theory (GT) is a general method to use on any kind or combination of data, and it is particularly useful with qualitative data.

The main goal of this approach consists of a set of steps which will "guarantee" a good theory as the outcome. It involves two phases in the analysis of qualitative data. Data fragments are compared in order to derive concepts and categories which catch their analytically relevant properties. The second phase is used to elaborate, refine and reduce results of the first phase.

Research Methods

This research was prepared for business companies which mission is to invest in software development for clients who wants customized applications to manage and improve their business. This study was conducted through the use of different research methods. Data was collected from staff interviews, observations, and surveys. Interviews were conducted with staff from 11 Portuguese companies whose core business was IT consultancy operating in several target markets with over 5 million euros.

Interviews were face to face, on a private room, and with the interviewees' authorization. They were audio recorded. Later, a transcript of the interview was made. Based on a script several questions were asked and the researcher took also some notes.

Finally, an anonymous online survey for IT developers was made through programming forums, social networks, e.g. LinkedIn. The forum's participants were software developers who worked with multiple Frameworks.

Work Analysis Procedures

The main motivation of this research was the researcher's concern about several rejected projects from customers at the end of an order. The intention was to try to understand and find out the reasons that led projects to this end. This research was carried out through several phases. Initially, we started reading literature, namely, Fayad [START_REF] Fayad | Object-Oriented Application Frameworks[END_REF] and technical documentation about object-oriented application frameworks. When applying GTM preconceived conceptions it should not be feared but instead serve as a starting-point for looking at data. After this initial data collection and analysis phases, we decided to make interviews. Interviews permitted to have rich data through a constant comparative method which constitutes the core of grounded theory, ideas started to be organized but needed to be validated. To validate the analyzed data and to keep the GTM process of constant comparison analysis, we decided to do an anonymous online survey. Several IT companies were invit-ed to collaborate in completing the survey. Fig. 3. presents the several phases for data collection, analysis and comparison.

Fig. 3. Research Methodology Phases

On Phase 1 we start by searching and reading bibliography to begin the data collection GTM process. On Phase 2 the interviews were conducted and respective data analysis and collection was made. Interviews demanded a preparation process before: background information about Companies' Frameworks were collected through books and internet articles which permitted to understand, as well as, to describe Frameworks' components and the way they work. The researchers only considered to interview people with around six years of experience in the companies. After this data collection phase the data was compared with the previous dataset.

Online surveys for data validation, on phase 3, were designed. Phase 3 also included the data gathered analysis and respective comparison with other phases. On Phase 4 we start defining the final theories of the process.

The work analysis process, following GTM, was, initially, based on the literature, different set of profiles were identified to establish a set of questions according to interviewees' experience (see table 1.).

Profile Description

Developer

Basic element of the hierarchy that develops projects over the Framework.

Team Leader

Coordinator element of the team development that manages the resources in accordance with the project plan.

Project Manager

Project manager who controls and analyzes the various stages of the project, is the bridge between the development team and the client.

Support

Element responsible for providing application support to the Framework or to the products built upon the Framework.

Tester

Element that performs the tests of the project according to the requirements.

Architect

Element responsible for defining the architecture of the solution. He is the person who defines the structure of the Framework and defines the features to be implemented.

Soft Director

Responsible for I&D is who controls the investment and define the product roadmap.

After face-to-face interviews, we did memo-writing which has high priority in the process of constructing a grounded theory. Memo-writing is the pivotal step between data collection and the draft. When an idea was got, ''It prompts you to analyze your data and codes early in the research process'' [START_REF] Charmaz | Constructing Grounded Theory. A Practical Guide Through Qualitative Analysis[END_REF].

The reading and note taking processes about frameworks led to a spreadsheet elaboration with some questions that should be asked to interviewees. It was realized that their different profiles required different kinds of questions. A developer, for example, sees the Framework on a different way that a supporter or an architect do.

Defined the profiles and questions, meetings were scheduling with all the persons. It was decided to start with some of researcher's colleagues to get more experience and to help in the questions' redefinition.

When the interviews were concluded a new phase arose, the data transcription and analysis. From that moment, different codes were defined to give rise to several categories. This analysis process was made with other spreadsheet and from there charts and diagrams were constructed. An example of data analysis for one of the answered questions by interviewees is presented on fig. 4.

Question: What are the advantages of using Frameworks by business enterprises? About 89% of respondents have worked with a Framework and from these 43% have between 6 and 10 years of experience.

Considering the main categories found within the GTM analysis we could understand that the benefits for developers who have worked with the frameworks were among others the speed of implementation, structuring, reuse and efficiency. The major identified drawbacks were the lack of documentation and the costs of introducing new features as well as versatility of the Framework in accordance with certain customer requirements.

For companies, the most obvious advantages for the use of frameworks are the increase on productivity associated with a shorter development time, the profitability improvement and the creation of standards inside the company (Fig. 6.). Fig. 6. Frameworks advantages Some keys factors considered good practices to the creation of Frameworks were also identified: formation, implementation speed, performance and security. It was interesting to see that the average investment by companies on these products is between 10% and 15% of their budget.

From the interviewees' responses it was considered that the support for multiples systems is a challenge for the future, side by side with the frameworks' versatility and adaptability, and also the continuous evolution and the introduction of new features in the framework.

The companies that are betting on this type of product, a framework, although they are achieving great profitability, they need to do some more work concerning new solutions to their customers.

Conclusions

Two of the major evident concerns are the lack of documentation and learning courses about the frameworks' use. On these referred areas, companies need to do some effort to improve the quality of utilization of the Framework.

In resume the Frameworks are useful for companies, they allow to reduce costs and increase productivity, besides allowing offering new and innovative products to its customers.

Fig. 1 .

 1 Fig. 1.Wizy Structure Organization [9]

Fig. 2 .

 2 Fig. 2. Framework Wizy Layers[START_REF] Wedotechnologies | Framework Wizi Portugal[END_REF]

Fig. 4 .

 4 Fig. 4. Grounded Theory Method

Fig. 5 .

 5 Fig. 5. Activity Sectors

Table 1 -

 1 Profiles Description

This research was important and it permitted to have some answers to the initial questions from the interviewees and from the online survey.The data analysis showed that there are some activity sectors that invest more in this type of products than others, for example, the Telecommunications' sector.Many users highlighted some key attributes on the advantages of using Frameworks like: time development reduction and the increase of profitability.


Catherine Burns
email: catherine.burns@uwaterloo.ca

Cognitive Work Analysis: New Dimensions

Keywords: cognitive work analysis, interface design, task analysis, humancomputer interaction

Cognitive work analysis (CWA) originated in the late 1970's and 1980's through the work of Jens Rasmussen, being collected and built into an effective methodology through the 1990's work of Kim Vicente, and culminating in his book Cognitive Work Analysis [1]

. Since that time, CWA, and in particular its derivative design approach Ecological Interface Design (EID), has been widely applied in a vast range of complex, control-oriented systems. Since the 1990's however, there has been an explosion of a new type of system -networked and distributed systems. These systems are characterized by their highly social dimension, resulting in new challenges in team problem solving, community building, and trust allocation across distributed teams. Our recent work in CWA has focused on adapting CWA to face these new challenges and provide a solution that fits a truly social technical system.

Introduction

In 1999, Vicente published his seminal book, Cognitive Work Analysis (CWA) [START_REF] Vicente | Cognitive Work Analysis, Toward Safe, Productive, and Healthy Computerbased Work[END_REF]. This book brought together several analytical views of cognitive work that had been developed earlier by Jens Rasmussen and his associates at the Riso National Laboratory in Denmark [START_REF] Rasmussen | Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models[END_REF][START_REF] Rasmussen | The role of hierarchical knowledge representation in decision making and system management[END_REF][START_REF] Rasmussen | Mental procedures in real-life tasks: A case study of electronic trouble shooting[END_REF]. The application of CWA as a method was demonstrated through the use of the method on a sample experimental process control system, DURESS [START_REF] Vicente | Ecological interface design: Theoretical foundations[END_REF]. DURESS was later used through multiple studies to confirm and to test the principles of CWA and Ecological Interface Design. While DURESS served as a solid test bed for validating the new approach, DURESS had limitations due to its simplicity compared to real world systems. In particular, DURESS could be operated with a single operator, and with relatively low competency levels, which meant that while this system could demonstrate the first three phases of CWA, Work Domain Analysis (WDA), Control Task Analysis (ConTA) and Strategies Analysis (StA), it remained a weak example for the last two phases, Social Organizational Analysis (SOA) and Worker Competency Analysis (WCA) [START_REF] Vicente | Cognitive Work Analysis, Toward Safe, Productive, and Healthy Computerbased Work[END_REF]. For many years, as CWA was becoming more established, this pattern continued. Multiple researchers demonstrated WDA, ConTA and in some cases StA, gradually refining these methods, making them simpler and building a set of examples [START_REF] Burns | Ecological Interface Design[END_REF][START_REF] Bisantz | Applications of Cognitive Work Analysis[END_REF].

These first three phases were also demonstrated in design, showing that they could effectively define useful design requirements and develop designs that were often more effective than existing designs. Essentially, through time, the value proposition for using CWA as a method became more convincing as more examples and designs emerged. In the background, however, the two final phases of CWA languished with little attention and poor development. In many cases, researchers made cursory statements that "of course" systems operated in a social-organizational context and "of course" the business would ensure workers had adequate competencies to operate the design. It was assumed, in fact, that in most cases, social-organizational factors, and worker competencies were out of the scope of the designer and were essentially elements of context that could not be changed.

Behind the scene, however, the world was changing. In 2002 LinkedIn was created and in 2004 Facebook was started. While it would be several years before these sites had become established, the reality was that the technology of social connectivity had arrived. Social connectivity meant that quick and fast access to people, as well as the development of new flexible social structures in the form of social networks. Social networks began among groups of friends but before long the business world had also adopted chats and social groups. Social networking allowed for faster and more flexible teamwork, mentoring and the rich access to expertise. Even a plant operator worked in a team, had email and chat to others in his or her organization, and talked to other operators and colleagues in his or her profession. Social organizational factors were becoming more important.

Early attempts to apply CWA to team situations produced a variety of results. Hadjukiewicz et al. [START_REF] Hajdukiewicz | Modeling a medical environment: An ontology for integrated medical informatics design[END_REF] mapped work domain zones in the operating room. We adopted this for a multiple objective domain in a military context [START_REF] Burns | Work domain analysis for establishing collaborative work[END_REF]. Naikar developed the contextual activity template as an approach for teams [START_REF] Naikar | Designing teams for first-of-akind, complex systems using the initial phases of Cognitive Work Analysis: Case study, human factors[END_REF][START_REF] Naikar | Analyzing activity in complex systems with cognitive work analysis: Concepts, guidelines, and case study for control task analysis[END_REF] and Jenkins and Stanton showed various team roles across different phases of CWA [START_REF] Jenkins | Using Cognitive Work Analysis to explore activity allocation within military domains[END_REF]. Through this work it became apparent that as CWA tackled social questions, a variety of approaches would be needed. We decided to focus on the development of a range of CWA inspired social methods to help tackle these new questions. We identified different needs for teams and for communities. In this paper we are defining teams as relatively small, goal focused work teams. In contrast communities are larger, more open, and less tightly focused. Team CWA is discussed first and in the following section, community based CWA is discussed. While Team CWA is fairly well developed, we are just beginning to explore CWA for community-based design.

Team CWA

Several Team CWA as an approach recognizes that in many cases, people do not work in isolation but function with others to achieve goals. In designing systems for teams, many questions arise such as "what view should team leaders have of team member work?" "What do people need to know of each other's work" and "how do we facilitate effective communication between team members?". Traditional CWA as a base provides a useful structure for gathering requirements, but does not particularly lead analysts towards developing the models or exploring the requirements that would answer these questions.

In developing Team CWA, we quickly realized that team concerns influenced all levels of the analysis and could not be easily reduced to a single analysis phase [START_REF] Ashoori | Team Cognitive Work Analysis: Structure and tasks[END_REF]. In fact, isolating team requirements from the primary CWA could be ineffective and lead to a loss of information. We argue that, when design for a team is required, that team requirements can be handled through the traditional CWA phases with some modifications. We suggest a full four phases looking at team WDA, team ConTA, team StrA, and team Competency Analysis (Figure 1). In the next sections these phases are described, in the context of a healthcare example taken from [START_REF] Ashoori | Team Cognitive Work Analysis: Structure and tasks[END_REF]. The basics of the team CWA analyses are discussed below.

Team Work Domain Analysis

The objective of Team WDA is to understand how work domain constraints influence a team. There are several distinct possibilities:

1. Team members may have shared objects or processes. 2. Team members may have objects or processes that are not shared.

3. Team members may have shared purposes. 4. Team members may have different or conflicting purposes. When team members have their own individual scope of control that is distinct from others, team work is relatively simple. However, when team members share objects or processes, team relevant design requirements become important. Must team members share object x? or pass it between them? Do they use it for different purposes? Do they have different information needs from object x? Similarly, which team members have shared purposes? And which team members may have conflicting purposes? Team members with shared purposes may find it useful to see how they are each contributing to the shared goal, while team members with conflicting pur-

Team Control Task Analysis

Team ConTA should identify how control tasks are shared across team members. It is also an opportunity to identify when various team collaborate on tasks synchronously, or across time, asynchronously. The contextual activity template can be used for this, multiple decision ladders, and we have recently explored the development of a decision wheel (Figure 3). The decision wheel has some advantages in showing individual team members as slices and separate teams as wheels. As well, the wheel tends to concentrate the evaluation processes into the bulls-eye, helping to identify key team members involved in more complex decision-making. We have also used this structure to show synchronous and asynchronous activity. Sample wheels are shown in Figure 3.

Team Strategies

Strategies, in CWA are alternative ways of achieving control tasks. While we see individual strategies shift with factors like expertise and workload, we also see teams show strategies. To respond to different work constraints, teams may adopt different configurations, roles, and expertise levels. The example below in Figure 4 shows different team structures under normal and emergency situations in the mother-baby unit example. Note that in emergency situations, the team shifts to highly experienced personnel, different types of expertise, and has access to different resources. In general, there are different kinds of strategies that can be modeled through CWA strategy analysis, from structural strategies provided by the work domain itself, team development strategies, operational strategies, and coordination strategies. The team strategies in this example are a case of coordination strategies.

Social competencies

Where worker competencies would define the skills, rules and knowledge needed for an operator to perform their job, social competencies define the skills and capabilities needed in various team roles. Some team members need to provide leadership, some mentorship, some negotiation, and others need to take direction and understand objectives. In the next two tables, we include the worker competencies (Table 1) and the social competencies (Table 2) from the mother-baby unit observations. It is clear from this example, that different social capabilities are required for different roles and an incorrect assignment of people to roles could significantly affect the coordination and performance of the team. The obstetrical resident should be aware of the fact sheets and the best practices and be able to discuss them with the peers.

The obstetrical resident should be able to effectively describe the observation, discuss the options with the peer, analyze the situation, and make a decision about the reason for the pain.

CWA in Communities

While improving CWA for teams is effective for smaller, more goal directed situations, there are cases where one can be asked to develop new designs for larger communities. Designing for communities can be a challenge as there are very few human factors methods that suit these kinds of problems. To work with CWA in communities, we explored community of practice concepts (CoP) [START_REF] Euerby | Designing for social engagement in online social networks using communities of practice theory and cognitive work analysis: A case study[END_REF]. The CoP ideas originate from the work of Wenger [START_REF] Wenger | Communities of practice: Learning, meaning, and identity[END_REF] and seek to understand why communities exist and how they grow. A difficulty with using the CoP concepts is that they were never developed for design. In the example below we show the use of CoP concepts in WDA. In this case, we were designing a social networking environment for a development organization. The key to CWA in community contexts is realizing that while a community has goals (in this case meeting certain development goals, the left hand column of the WDA in Table 3), a community also has a need to build and sustain itself as an entity (the right hand column of the WDA in Table 3). The CoP concepts were very helpful in understanding community-building requirements. In this sense, and in the terminology of WDA, a community has purposes, values and principles, processes, and people working together. Note that it can be helpful in this context to adjust the labels of the WDA to fit with more community relevant terms. In this particular case, this analysis was used to redesign a site for the community. Over a six-month longitudinal study, significant improvements in community struc-ture and presence were obtained [START_REF] Euerby | Improving social connection through a communities of practice inspired Cognitive Work Analysis approach[END_REF][START_REF] Euerby | Increasing social activity through a Community of Practice inspired design[END_REF]. This provides early encouragement that this approach to design for communities may be effective.

4

Discussion: when to use what?

The intention of this work is not to turn CWA into a massive analysis of individual, team and community, multiplied by 4 phases of analysis. But given systems with different needs, CWA needs to provide a responsive methodology to help designers to face these new systems. CWA is a toolbox and while doing every analysis at every level of detail is ideal, it is also unrealistic in many cases. Designers need to understand the scope of their design problem, its scale, and the key objectives of the redesign. There are many examples of successful designs developed solely from WDA. We have also had strong success using ConTA and StA to inform design in environments like healthcare where work domain information can be difficult to obtain. Just as the traditional phases of WDA can be applied in part to solve a design problem, so too can a more team or community oriented view be adopted if that suits the problem at hand. Figure 5 below provides some guidance on how to choose approaches. Individual Operator, Working in a Team Team WDA, Team ConTA, StA Team of Operators Known roles Team WDA, Team ConTA, Team StA Team of Operators Flexible roles Team WDA, Team ConTA, Team StA, Social Competencies, WCA

Community of Operators

CoP-WDA, other approaches to be developed

Conclusion: next steps

Evolving CWA, and indeed Human Factors Engineering to tackle modern sociotechnical systems is just beginning. There are, at this point, still relatively few methods adapted for social or team systems and few examples. The work presented here is also preliminary, but expected to be a core research direction for the Advanced Interface Design Lab over the next few years. Several key research directions are apparent. First, further method development is possible. We have not yet integrated the ideas of macro cognition and emergent team processes into CWA and there is evidence that these processes play an important role. Further, the community building approach could be deepened through an understanding of community processes, strategies, and competencies. Completing these directions would provide a rich analytical approach to understanding cognitive work in a variety of environments.

Understanding the design support for these environments is critical. Drawing from a base the ecological psychology foundation of ecological interface design, we anticipate a new class of affordances, social affordances that need to be understood. These affordances already exist, and we use them every day (friend counts, like buttons, mail and chat are just a few), but a systematic understanding of the influence of these affordances and how to best use them in design does not exist. Furthermore, there are social processes such as trust and engagement that are very influential in encouraging, or discouraging human behaviour that are amenable to influence by design but remain poorly understood. We intend over the next few years to explore the design of social affordances as well as how trust and engagement are influenced through design.

6

Fig. 1 .

 1 Fig. 1. Team CWA

Team

  Work Domain Analysis Team Control Task Analysis Team Strategies Analysis Worker Competency Analysis (Technical and Social) Team Cognitive Work Analysis poses may need to accommodate each other to achieve overall success. Team members with individual and unshared purposes may need to focus on their own part of the work domain, with less distraction from the team. Using a basic WDA, team WDA can use maps of team member zones on the work domain, contextual activity templates, or collaboration tables to keep track of various team interactions in the work domain. Below (Figure 2) we show an example of mapped zones in a healthcare domain.

Fig. 2 .

 2 Fig. 2. Team member mappings on a work domain model.

Fig. 3 .

 3 Fig. 3. The decision wheel (from Ashoori and Burns, Journal of Cognitive Engineering and Decision Making in press), copyright ©2012 Human Factors and Ergonomics Society. Reprinted by Permission of SAGE Publications.

Fig. 4 .

 4 Fig. 4. Team strategies.

Fig. 5 .

 5 Fig. 5. Choosing CWA approaches.

Table 1 .

 1 Worker competencies.

	Actions	Skill-based behavior	Rule-based behavior	Knowledge-based behav-ior
	Collecting	An experienced obste-	The obstetrical resi-	The obstetrical resident
	information	trical resident should	dent should be able to	should be able to analyze
		be aware of the symp-	look up the fact sheets	the symptoms and plan
		toms and know what	or best practices to	for further assessments, if
		signs and symbols to	find a list of signs and	required.
		observe.	symbols to observe.
	Identifying	An experienced obste-	The obstetrical resi-	Severe headache on the
	the reason for	trical resident should	dent should be able to	day of surgery can have
	the pain	know that a lumbar	look up the fact she-	many reasons. The obste-
		puncture can cause a	ets, best practices, or	trical resident should be
		headache.	instructions to find the	able	to	analyze
			cause for a post-dural	supplementary
			puncture headache.	information and make a
				decision about the cause
				of the headache.

Table 2 .

 2 Social competencies.

	Function Role	Team role	Social skills required (Belbin,1981)
	Care facilitator at
	MBU Care facilitator at	Coordinators	Mature, confident, a good chairperson, should be able to clarify goals, promote decision making, and delegate well
	BU
	Primary nurse Staff nurse	Team-workers	Cooperative, perceptive and diplomatic. Should be able to listen, build, and avert friction
	Anesthesiologist Obstetrical resi-dent	Specialists	Single-minded, self-starting, dedicated to providing knowledge and skills in rare supply

Table 3 .

 3 Work domain analysis including community domain elements (Euerby, Burns Journal of Cognitive Engineering and Decision Making 6 (2), 194-213. © 2012 by Human Factors and Ergonomics Society. Reprinted by Permission of SAGE publications.

		Domain	Community
	Functional Purpose	Achieve the UN Mil-	Agreed upon role in the	Desired State of the
		lennium Development	larger context; Strength	Learning Community
		Goals (MDGs)	in relationships to
			actively discuss differ-
			ences with respect to
			the domain and prac-
			tice;
			Established what
			knowledge should be
			shared and how to share
			it.
	Value and Priority	Span of the external	Participation vs. reifica-	Tradeoffs and Priori-
	Measures	network vs. strength of	tion;	ties of the Negotiation
		the connections;	Designed vs. emergent;	of Meaning
		Social action vs. re-	Identification vs. nego-
		search results;	tiability;
		Power of the researcher	Local vs. global.
		vs. power of the com-	Increase alignment,
		munities.	engagement and imagi-
		Mobilize knowledge	nation.
		and mobilize communi-
		ties;
		Bring awareness about
		MDGs and impel col-
		lective social action
		toward their achieve-
		ment;
		Educate, engage and
		empower communities.
	Processes	Building formal part-	Building connections	Practices
		nerships, hosting net-	between the core partic-
		working and public	ipants, identifying
		outreach events, collab-	opportunities to provide
		orating with partners.	value in the larger
			context, finding the
			ideas and insights that
			are worth sharing with
			other members, identi-
			fying gaps in the
			knowledge, refining the
			CoP's role in the larger
			context
	People, Relationships,	Network Facilitator, Executive Team Members,	People, Relationships,
	Projects, Events	Regional Coordinators, peripheral members,	Projects, Events
		Knowledge Partners, Organizational Partners,
		Memorandums of Agreement with Organizational
		Partners, Executive Meetings, Stand-Up Against
		Poverty Events, Bridging the World virtual events.

Acknowledgments

This work was made possible through an NSERC Discovery Accelerator Supplement Award to Catherine Burns. The Accelerator Supplement Award identifies Canadian researchers who are most likely to become world class researchers and make a lasting impact in their field.