Marie-Stéphanie Bracq
email: marie.stephanie.bracq@univ-rennes2.fr

MPsy Estelle Michinov

Professor Pierre Jannin

Marie- Stéphanie Bracq

Virtual reality simulation in non-technical skills training for healthcare professionals: a systematic review

Keywords: virtual reality simulation, non-technical skills, healthcare training, medical education, systematic review. Virtual reality simulation

This systematic review, conducted in accordance with PRISMA guidelines, is aimed to review current research in virtual reality for healthcare training, specifically pertaining

Introduction

With the publication of To Err is Human in 1999 [START_REF] Kohn | To Err Is Human: Building a Safer Health System[END_REF] and Safety at the Sharp End [START_REF] Flin | Safety at the sharp end, a guide to non-technical skills[END_REF] , attention was drawn to the importance of human factors in operational teams, identifying skills that were first described for crew resource management in aviation and other high-risk industries. Flin described these non-technical skills (NTS) as "the cognitive, social and personal resource skills that complement technical skills, and contribute to safe and efficient task performance" [START_REF] Flin | Safety at the sharp end, a guide to non-technical skills[END_REF] . In Flin's taxonomy, NTS included individual cognitive skills (e.g., situation awareness, decision-making, coping with stress and management of fatigue) and interprofessional social skills (e.g., cooperation and teamwork, conflict resolution, leadership). Evidence that human factors have an impact on surgical performance [START_REF] Hull | The impact of nontechnical skills on technical performance in surgery: a systematic review[END_REF] led to the development of training programs focusing on NTS to improve patient safety [START_REF] Gordon | Non-technical skills training to enhance patient safety: a systematic review: Non-technical skills education: a systematic review[END_REF] . Using Flin's taxonomy, several tools (i.e. ANTS, NOTSS, SPLINTS) have been designed to provide surgical teams with a common language to discuss and develop the human factors that are critical for patient safety [START_REF]Non-technical skills for anaesthetists, surgeons and scrub practitioners (ANTS, NOTSS and SPLINTS)[END_REF] .

Several studies have demonstrated the efficiency of simulation for knowledge acquisition and for technical [START_REF] Agha | The role and validity of surgical simulation[END_REF][START_REF] Harder | Use of Simulation in Teaching and Learning in Health Sciences: A Systematic Review[END_REF][START_REF] Mcgaghie | Does Simulation-Based Medical Education With Deliberate Practice Yield Better Results Than Traditional Clinical Education? A Meta-Analytic Comparative Review of the Evidence[END_REF] and non-technical [START_REF] Cumin | A Systematic Review of Simulation for Multidisciplinary Team Training in Operating Rooms[END_REF][START_REF] Von Wendt | Simulation in Interprofessional Clinical Education: Exploring Validated Nontechnical Skills Measurement Tools[END_REF] skills training in healthcare. Training healthcare professionals to manage rare or critical events in a standardized manner and without risk for the patient has become a major challenge [START_REF] Gaba | The future vision of simulation in health care[END_REF] . Although the benefits of simulation have been well documented, the human resources required for mannequin or standardized patient-based simulation and the availability of human resources in simulation centers remain scarce. However, various simulation methods have recently been developed using real actors, mannequins, standardized patients, computer simulators or serious games.

With the development of technology and the "laparoscopic surgery revolution" [START_REF] Windsor | Role of simulation in surgical education and training[END_REF] , Virtual Reality (VR) simulators are being used more widely in both professional practice and education programs [START_REF] Mantovani | Virtual reality training for healthcare professionals[END_REF][START_REF] Palter | Virtual reality in surgical skills training[END_REF] . VR is a broad concept that encompasses three categories of simulators: screen-based VR simulators, virtual worlds, and immersive virtual reality environments. First, screen-based VR simulators have been used since the nineties to develop psychomotor skills for endoscopic surgery [START_REF] Van Dongen | European consensus on a competency-based virtual reality training program for basic endoscopic surgical psychomotor skills[END_REF] . They consist of an interface comprising a computer and monitors coupled to mechanical devices or haptic units [START_REF] Van Herzeele | Cognitive training improves clinically relevant outcomes during simulated endovascular procedures[END_REF] . This kind of simulator requires very little set-up time and can be used repeatedly by learners for practice in different pathologies and with a number of anatomical variations [START_REF] Blackburn | Role of simulation in training the next generation of endoscopists[END_REF] . Second, virtual worlds are three-dimensional virtual environments based on multiplayer online gaming, allowing users to free themselves from geographical proximity or time constraints (individual connection and full time access) [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF] . For health professionals, medical furniture, instruments, devices, tools and characters are added to create dedicated medical virtual worlds [START_REF] Dev | Virtual worlds and team training[END_REF] . Lastly, immersive virtual reality environments combine three-dimensional imaging, interactions with the environment, possible haptic feedback and head-mounted displays (HMD) or cave automatic virtual environments (CAVE, room-sized cube VR environments) to immerse the user and occlude the real world in order to provide a feeling of presence [START_REF] Mantovani | Virtual reality training for healthcare professionals[END_REF][START_REF] Kilmon | Immersive Virtual Reality Simulations in Nursing Education[END_REF] .

The effects of VR-based training on healthcare professionals have mostly been studied in relation to the development of technical skills, either surgical (e.g., procedure, planning, knowledge of instruments) or psychomotor (e.g., dexterity, accuracy, speed) [START_REF] Choudhury | Fundamentals of neurosurgery: virtual reality tasks for training and evaluation of technical skills[END_REF][START_REF] Gallagher | Virtual Reality Simulation for the Operating Room: Proficiency-Based Training as a Paradigm Shift in Surgical Skills Training[END_REF] . The use of VR simulation-based training for the development of NTS seems to be less common. In view of the recent developments described above, a systematic review of how VR has been used for NTS skills training and assessment could provide powerful new insights into the value and efficiency of this technique. This paper provides such a review, focusing on the use of virtual reality in the evaluation and development of a predefined set of cognitive and interprofessional social skills required by healthcare professionals.

METHODS

Data source and search strategy

This systematic review followed the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [START_REF] Moher | Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement[END_REF] . The PICOS worksheet and search strategy [START_REF] Rielding | Learning to learn[END_REF] were used to organize our research topics and terms, combining five concepts:

Selection criteria

Inclusion of articles was based on the following criteria: 1) Focus on healthcare professionals working in teams; 2) Focus on Virtual Reality used in an educational or training context that includes outcomes data; 3) Focus on NTS in line with Flin's pre-defined set of NTS referring to cognitive skills (situation awareness, decision-making, stress and management fatigue) and interprofessional social skills (cooperation and teamwork, conflict resolution, leadership). To get the most reliable source of scientific information, only publications in peer-reviewed journals were considered. Since VR is a relatively new technology, the period chosen for study selection was between 2007 and 2017. Finally, to have an international overview, only articles written in English have been retained.

Study selection and eligibility criteria

We extracted 1373 references, 460 on Psycinfo and 913 on Medline. Four others found on Google Scholar while reading on the subject were added, thus making a total of 1377 references. We removed 188 abstracts in duplicate, resulting in 1189 titles and abstracts to read for selection. At this stage of the process, 1109 references were excluded.

In accordance with the first selection criterion, study populations that did not involve health professionals (e.g. pedestrians, cyclists, students, athletes or patients) were removed, as well as those working in smaller teams or solo (e.g. dentists, pharmacists and vets).

Following the second selection criterion, articles dealing with learning methods that were not the focus of this study were deleted (e.g., e-learning, e-classroom, web-based learning, telemedicine, mobile applications, social networks). According to the same criterion, some references were also excluded because the term "virtual" was used without referring to virtual reality (e.g., virtual teams, virtual communities, virtual classroom, virtual consultations and virtual patients). There is a vast literature on the "Virtual Patient", focusing on the interpersonal social skills of healthcare professionals required to facilitate accurate diagnosis, give appropriate advice, and instruct patients about treatment. As these skills do not come within the scope of our predefined set, these references were not included in the present review.

Finally, according to the third criterion, articles that did not address NTS were excluded: knowledge representation (anatomy, physiology, histology, 3D planning for radiotherapy), technical skills, interpersonal patient-physician relationship (diagnosis, notification of critical results or pathology), psychiatric disorders (dementia, post-traumatic stress disorder), or remediation (cognitive behavioral therapy, rehabilitation) that also use VR technologies but are out of the scope of this review. This inital selection included 80 articles with abstracts that were insufficiently clear to decide whether or not they were eligible, and these were uploaded in order to read the full text. However, five remained unavailable even after their main authors were contacted by email, which reduced the number of articles to read to 75.

After the full-text analysis, 49 articles were excluded for four reasons: 1) they did not involve NTS (n= 31), 2) they did not use virtual reality (n= 15), 3) they did not involve health professionals (n= 2), or 4) it was a review article with no experiment (n=1).

Selection bias (interrater reliability)

To check the validity of the study, a sample of the articles (30%) was independently coded by the second author. Based on the eligibility criteria, the second coder decided whether or not to examine an article in depth. After reviewing the abstract and full text, the two coders, meaning the first and second authors, agreed on 26 articles, 11 that should be selected and 15 rejected. Their opinions only diverged on the implementation of NTS in one article (3.7%), which was eventually excluded (Cohen Kappa = .92). The selection process is summarized in the flowchart (see Figure 1).

Figure 1. Flowchart of systematic review

To obtain a descriptive analysis of the results of this review, we drew up a comparative table, focusing on medical specialties, VR simulator typologies, study populations, and assessd NTS. For the latter, when assessment tools were mentioned, presence or absence of their validity evidence in the virtual environment was also examined. Another criterion for the analysis was outcomes, according to Kirkpatrick's levels (see Table 1, Supplemental Digital Content 1, which shows the characteristics of the studies included in the review). This framework was chosen because the Kirkpatrick Model [START_REF] Kirkpatrick | Evaluating training programs the four levels[END_REF] is frequently used to evaluate training programs; for example, it was recently used to evaluate serious games for training healthcare professionals [START_REF] Wang | A Systematic Review of Serious Games in Training Health Care Professionals[END_REF] . This model has four levels: 1) affective reactions after the training program, 2) improvement of knowledge and skills, 3) change of behavior and transfer of skills in professional contexts, 4) increased patient care quality and reduced costs.

RESULTS

Study characteristics

VR is a recent technology and, as expected, the number of articles published per year over the last decade on this subject is rather low although it has been increasing, with a peak of seven articles in 2015. The median number of articles published per year is two

(interquartile range [IQR] = 3-1, min = 0, max = 7).
Regarding sample size, the average number of participants was 39.69 (min = 10, max = 148, SD= 37.33). Looking more closely at their study design, 17 studies used an experimental design (pre-test/post-test, group comparisons or control/test group comparisons) [START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Creutzfeldt | Using Virtual World Training to Increase Situation Awareness during Cardiopulmonary Resuscitation[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Khanal | Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles[END_REF][START_REF] Maschuw | The impact of self-belief on laparoscopic performance of novices and experienced surgeons[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF] , nine were observational studies [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF][START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF][START_REF] White | Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit[END_REF][START_REF] Umoren | TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners[END_REF] , and one was based on qualitative interviews [START_REF] Rogers | Developing simulations in multi -user virtual healthcare education[END_REF] .

Medical specialties

Emergency medicine [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF][START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF] and health education [START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Umoren | TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners[END_REF][START_REF] Rogers | Developing simulations in multi -user virtual healthcare education[END_REF] stand out from the rest with five articles each, but the latter includes issues that potentially concern all specialties or health professionals in different sectors. Next are urology [START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF] and gastroenterology [START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF] with four articles each. A further eight articles dealt with interprofessionality [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF][START_REF] White | Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit[END_REF][START_REF] Umoren | TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners[END_REF] .

Study populations

Most of the studies include a range of professions: nurses feature in eight articles [START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF][START_REF] White | Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit[END_REF] , surgeons [START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khanal | Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF] in seven, and five articles mention other professionals [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF] . With regard to students, 12 articles involve residents or postgraduate trainees [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khanal | Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles[END_REF][START_REF] Maschuw | The impact of self-belief on laparoscopic performance of novices and experienced surgeons[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF] , ten concern health students regardless of their level or specialty [START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Creutzfeldt | Using Virtual World Training to Increase Situation Awareness during Cardiopulmonary Resuscitation[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF][START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF][START_REF] Umoren | TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners[END_REF][START_REF] Rogers | Developing simulations in multi -user virtual healthcare education[END_REF] , and three articles mention trainees [START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF] .

VR systems

In the studies selected for review, two technologies are equally represented: Screenbased VR simulators in 12 studies [START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Maschuw | The impact of self-belief on laparoscopic performance of novices and experienced surgeons[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF][START_REF] White | Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit[END_REF] and virtual worlds in 12 articles (18- 20,31,32,35,37,41,44,47,50,51) . Immersive virtual worlds are less common, as they are found in only three studies [START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF] .

NTS and assessment metrics

Among the 26 studies, the most frequently investigated NTS is teamwork, which is mentioned in 19 articles [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Creutzfeldt | Using Virtual World Training to Increase Situation Awareness during Cardiopulmonary Resuscitation[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Khanal | Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF][START_REF] White | Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit[END_REF][START_REF] Umoren | TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners[END_REF][START_REF] Rogers | Developing simulations in multi -user virtual healthcare education[END_REF] . After that is communication (15/26) [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF][START_REF] White | Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit[END_REF][START_REF] Umoren | TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners[END_REF] , followed by situation awareness (10/26) [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Creutzfeldt | Using Virtual World Training to Increase Situation Awareness during Cardiopulmonary Resuscitation[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF][START_REF] Rogers | Developing simulations in multi -user virtual healthcare education[END_REF], decision-making (8/26) [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF] , leadership (8/26) [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF] , and stress management (6/26) [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Maschuw | The impact of self-belief on laparoscopic performance of novices and experienced surgeons[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF] . Most studies measured more than one NTS construct, with the exception of three articles focusing specifically on stress management [START_REF] Maschuw | The impact of self-belief on laparoscopic performance of novices and experienced surgeons[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF] , three others on teamwork [START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Khanal | Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF] , and one on communication [START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF] . Specific metrics for NTS assessment are mentioned or described in detail in 16 articles [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Creutzfeldt | Using Virtual World Training to Increase Situation Awareness during Cardiopulmonary Resuscitation[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Khanal | Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles[END_REF][START_REF] Maschuw | The impact of self-belief on laparoscopic performance of novices and experienced surgeons[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF][START_REF] White | Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit[END_REF] , ten of which [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF] mention and use specific validated assessment tools for NTS in the operating room or in a clinical environment (i.e. NOTECHS,

NOTSS, T-NOTECHS, CGRS, M-OSANTS, TOSCE, T-TAQ, SURG-TLX, EMCRM).

These assessment tools are presented as validated for simulation [START_REF] Sevdalis | Reliability of a revised NOTECHS scale for use in surgical teams[END_REF][START_REF] Yule | Development of a rating system for surgeons' non-technical skills[END_REF] , but none of them mention their validation in the very specific context of VR simulation. Four articles [START_REF] Creutzfeldt | Using Virtual World Training to Increase Situation Awareness during Cardiopulmonary Resuscitation[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Maschuw | The impact of self-belief on laparoscopic performance of novices and experienced surgeons[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF] use scales designed for a specific NTS, but are not specific to the operating room or to health professionals (i.e. situation awareness, self-efficacy, anxiety). Six articles [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Khanal | Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] White | Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit[END_REF] use their own tools or checklists, or open-ended questions rather than a questionnaire. Again, none of these tools or checklists take into consideration the specificity of VR simulation, but are simply transferred from real life or traditional simulation contexts.

Outcome measures

The levels of assessment of the simulators or scenarios used in the studies reach Kirkpatrick's level 1 (affective reactions) in 19 articles [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF][START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF][START_REF] Rogers | Developing simulations in multi -user virtual healthcare education[END_REF] and level 2 (learning: attitude, knowledge) in 18 [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Creutzfeldt | Using Virtual World Training to Increase Situation Awareness during Cardiopulmonary Resuscitation[END_REF][START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF][START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Khanal | Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles[END_REF][START_REF] Maschuw | The impact of self-belief on laparoscopic performance of novices and experienced surgeons[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF][START_REF] White | Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit[END_REF][START_REF] Umoren | TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners[END_REF] . Level 3 (behavior) is only reached in two articles [START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF] that describe a test of skills transfer to the clinical environment in order to evaluate behavioral changes among trainees. No studies were found that reached level 4 (results) evaluating the effect of simulation on patient care quality and cost.

Overall, this review demonstrates that few medical specialties use virtual reality for NTS training. The most frequently used systems are screen-based VR simulators or virtual worlds, while immersive virtual worlds are rarely used. The most frequently studied NTS are the interpersonal and interprofessional social skills needed for effective medical teams, including teamwork and communication, together with situation awareness, which is a crucial cognitive ability.

Objectives and findings of the studies

Two main categories of study objectives emerged from our review (see Table 2, Supplemental Digital Content 2, which shows the objectives, measures and main findings of the studies).

The majority of studies are simulator or scenario centered, with a clear goal of establishing the acceptability of the technology for NTS training. This is the goal of 20 articles [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Creutzfeldt | Using Virtual World Training to Increase Situation Awareness during Cardiopulmonary Resuscitation[END_REF][START_REF] Khanal | Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles[END_REF][START_REF] Maschuw | The impact of self-belief on laparoscopic performance of novices and experienced surgeons[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF][START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF][START_REF] White | Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit[END_REF][START_REF] Rogers | Developing simulations in multi -user virtual healthcare education[END_REF] . As VR has only been introduced in healthcare training recently, it seems logical to assess this first. The general conclusion of these studies is that VR simulators offer promising opportunities for NTS training of health professionals. Their acceptability as an NTS training tool is validated either directly [START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF] or through one or more of its predictors as defined by Nielsen's model of system acceptability [START_REF] Nielsen | Usability engineering[END_REF] : validity or fidelity [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF] , usefulness or utility [START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF] , efficiency, effectiveness or efficacy [START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Khanal | Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles[END_REF][START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF] , and usability [START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF] . The acceptability of VR simulators is not assessed alone but together with teamwork [START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Khanal | Collaborative virtual reality based advanced cardiac life support training simulator using virtual reality principles[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF][START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF][START_REF] Rogers | Developing simulations in multi -user virtual healthcare education[END_REF] , situation awareness [START_REF] Hudson | Second life simulation as a strategy to enhance decision -making in[END_REF][START_REF] Creutzfeldt | Using Virtual World Training to Increase Situation Awareness during Cardiopulmonary Resuscitation[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF] , stress management [START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF] , self efficacy [START_REF] Maschuw | The impact of self-belief on laparoscopic performance of novices and experienced surgeons[END_REF] , and leadership [START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF] , or with several NTS using a dedicated assessment tool such as NOTECHS or NOTSS [START_REF] Cohen | Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF] .

The purpose of the six remaining articles, among the most recent ones of this review, is to propose different uses of VR simulation. One includes a virtual reality program that focuses on communication tools as an initial introduction to team communication strategies [START_REF] Umoren | TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners[END_REF] . The other five are curriculum centered, their objectives being to incorporate NTS training modules using VR simulators in the medical curriculum: four of them describe distributed simulation [START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF][START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF] , in other words, the use of VR simulators initially designed for teaching technical skills in wider settings or scenarios [START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF] , and the fifth [START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF] addresses emergency preparedness using a virtual world.

DISCUSSION

There are few medical specialties that use virtual reality simulation for NTS and their goals vary. First, virtual worlds are used in healthcare education and in emergency medicine to provide practice opportunities for rare events or extreme situations that are difficult to set up in real life. Second, screen-based VR simulators are used for technical skills training in domains involving laparoscopic surgery or robotic-assisted surgery, such as gastroenterology [START_REF] Grover | Impact of a simulation training curriculum on technical and nontechnical skills in colonoscopy: a randomized trial[END_REF][START_REF] Khan | Simulation-Based Training of Non-Technical Skills in Colonoscopy: Protocol for a Randomized Controlled Trial[END_REF] and urology [START_REF] Brunckhorst | Simulationbased ureteroscopy skills training curriculum with integration of technical and non-technical skills: a randomised controlled trial[END_REF][START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF] curricula. In the latter, NTS are introduced progressively in VR simulation scenarios as trainees develop their technical skills [START_REF] Arora | Framework for incorporating simulation into urology training[END_REF] . This is congruent with the conclusions of Shamim-Khan et al. (2013) [START_REF] Khan | Development and implementation of centralized simulation training: evaluation of feasibility, acceptability and construct validity: Centralized simulation training[END_REF] and Rudarakanchana et al.

(2014) [START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF] who observed that junior trainees focus more on technical skills and feel that the introduction of NTS adds stress and anxiety due to the need to multitask. It also follows the hierarchy of core, procedural and team skills proposed by Windsor (2009), who illustrates it with a musical analogy of note, melody and harmony: just like musicians begin to learn playing a note before trying to play a melody and then join an orchestra, core skills such as knot tying, dissecting or suturing, for example, should be trained before procedural skills (i.e. how to dissect out a pathology). Then, team skills can be trained [START_REF] Windsor | Role of simulation in surgical education and training[END_REF] . Nevertheless, this progression does not enjoy consensus and must be considered for each set of NTS. For example, the addition of cognitive skills likely to help error-detection to technical skills training can result in more effective learning [START_REF] Kohls-Gatzoulis | Teaching cognitive skills improves learning in surgical skills courses: a blinded, prospective, randomized study[END_REF] .

Screen-based VR simulators or virtual worlds are thus the most frequently used systems. Immersive virtual worlds are less commonly used and are mentioned in only three articles [START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF] , two of them using head-mounted displays [START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF] . They were published between 2015 and 2016, which corresponds to the time when HMD such as Oculus Rift or HTC Vive became mainstream, even though immersive VR simulation is not a new technology in healthcare as it has been used since the early 2000s [START_REF] Krijn | Treatment of acrophobia in virtual reality: The role of immersion and presence[END_REF] . It will be interesting to see how these devices will be used in the future as their comfort and fields of view improve, and how they can be integrated into new surgical training modules.

One of the major interests of VR is its realism, which is stressed in several studies [START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Rudarakanchana | Endovascular repair of ruptured abdominal aortic aneurysm: technical and team training in an immersive virtual reality environment[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF][START_REF] Abelson | Virtual operating room for team training in surgery[END_REF][START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF]. . Even though avatars may sometimes behave awkwardly, and haptic feedback may be approximate or missing, participants still recognise the main features of their environments or work organizations. This contributes to the feeling of immersion mentioned in two articles [START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Sankaranarayanan | Face and construct validation of a next generation virtual reality (Gen2-VR) surgical simulator[END_REF] . These two notions are of particular interest in that they help trainees gain confidence [START_REF] Heinrichs | Training healthcare personnel for mass-casualty incidents in a virtual emergency department: VED II[END_REF][START_REF] Brewin | Full immersion simulation: validation of a distributed simulation environment for technical and non-technical skills training in Urology[END_REF][START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF][START_REF] Willaert | Efficient implementation of patient-specific simulated rehearsal for the carotid artery stenting procedure: part-task rehearsal[END_REF] by providing them with new learning opportunities, such as discovering an emergency room without stress [START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF] , or being able to develop the skills needed to react to disasters when training in an unfamiliar environment [START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF] . However, the degree of realism expected in a VR simulator has to be questioned for each scenario: if more haptic feedback, such as somatosensations or kinematic interactions, is expected with a high degree of fidelity, too much realism for an avatar in virtual worlds can lead to anxiety and rejection, due to the uncanny valley phenomenon. According to this theory, participants react favorably to environments that are very similar or very dissimilar to reality, but are uncomfortable with intermediate realism [START_REF] Mori | The uncanny valley [from the field[END_REF][START_REF] Howard | Investigating the simulation elements of environment and control: Extending the Uncanny Valley Theory to simulations[END_REF] .

The affective component of learning has been described as one of the four key criteria for simulation-based learning, which is centered on the learners' needs and for which motivation and self-efficacy are key concepts [START_REF] Kneebone | Evaluating Clinical Simulations for Learning Procedural Skills: A Theory-Based Approach[END_REF] . With this in mind, many VR simulators allow participants to replay their session, helping them recognize and analyze both their interactions and their emotions [START_REF] Fertleman | A Discussion of Virtual Reality As a New Tool for Training Healthcare Professionals[END_REF] . The emotional impact of VR simulation on self-efficacy is emphasized and appreciated by trainees [START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF] , as well as the opportunity provided by some scenarios to communicate with other disciplines before any clinical practice [START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF] or to experience human interactions in problematic environments [START_REF] Rogers | Developing simulations in multi -user virtual healthcare education[END_REF] . In terms of motivation, VR simulation seems to be of particular interest as it is described as a highly rated learning experience [START_REF] Riesen | Improving interprofessional competence in undergraduate students using a novel blended learning approach[END_REF] , preferred to standard didactic lectures [START_REF] Dorozhkin | OR fire virtual training simulator: design and face validity[END_REF] , and seen as excellent preparation for clinical situations [START_REF] King | Developing interprofessional health competencies in a virtual world[END_REF] .

Another interest of VR simulation is data generation because VR simulators can track and record every action. The data are used to give learners feedback on their performance and progress over time through their profile, allowing them to verify their skills acquisition and become proactive in their learning. But they also help educators better understand their students' learning processes, allowing them to adjust their inputs and complement their traditional teaching methods with appropriate simulations [START_REF] Umoren | TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners[END_REF] . In addition, as most VR simulators and scenarios can be used in different cultural and geographic environments, the collected data could also be used for further intercultural studies of NTS [START_REF] Umoren | TeamSTEPPS Virtual Teams: Interactive Virtual Team Training and Practice for Health Professional Learners[END_REF] .

The most frequently studied NTS are interpersonal and social skills, such as teamwork and communication. As a cognitive skill, situation awareness also features frequently, as it is a crucial personal skill, especially in dynamic environments such as medical settings where it can impact decision-making and communication [START_REF] Endsley | Toward a theory of situation awareness in dynamic systems[END_REF] . While the capacity of VR environments to recreate realistic situations makes it suitable for stress management training, none of the articles selected for this review discuss fatigue management. Even though VR is used to that end for patient-focused psychological therapy, healthcare professionals may not perceive fatigue and stress management as an issue warranting the use of simulation scenarios.

Most studies are set up to assess a simulator or scenario, reaching Kirkpatrick's level 1 (affective reaction) or 2 (learning). According to these articles, VR simulators offer NTS training opportunities for healthcare students, as their feasibility, usability, validity, acceptability or effectiveness have been validated in different situations. However, their conclusions recommend further studies. The impact of VR simulation on NTS training still requires more systematic assessment in routine clinical practice in order to validate a possible transfer of skills for health professionals and to determine whether it achieves its ultimate goal of improving patient safety (Kirkpatrick's level 4). This is challenging because patient safety is multifactorial, and because, as this review shows, validated tools to assess NTS are not systematically used, making large-scale comparisons difficult. Furthermore, few studies mention the inclusion of NTS training using VR simulators in the curriculum as a way of helping students progress.

To continue the development of VR systems for NTS training, four lines of research are required. First, studies are needed regarding the validation of specific NTS assessment tools for VR simulation. So far, these tools have not been validated, which may impact their evaluation in these specific environments. Second, studies are needed to estimate the different technical possibilities offered by VR technologies for NTS training, such as mixed and augmented reality. Third, debriefing is considered a key element for skills transfer in simulation-based training [START_REF] Fanning | The Role of Debriefing in Simulation-Based Learning[END_REF] . While this has been studied in different simulation contexts [START_REF] Cheng | Debriefing for technology-enhanced simulation: a systematic review and meta-analysis[END_REF] and with different frameworks [START_REF] Sawyer | More Than One Way to Debrief: A Critical Review of Healthcare Simulation Debriefing Methods[END_REF] , specific debriefing methods for VR simulation are scarce.

Avatar-based debriefing [START_REF] Nagendran | A Unified Framework for Individualized Avatar-Based Interactions[END_REF] , for example, could be developed for health education. Fourth, future studies should evaluate the effects of these training methods at different levels of learning: attitudes, skills, transfer of skills, and cost-benefit ratio. In particular, more studies are needed in the near future to investigate the transfer of skills to the operating room (Kirkpatrick's level 3) and to validate VR simulation as an efficient training tool for NTS. But what is really missing is the impact of these tools on the quality of care for patients and on the overall cost of care (Kirkpatrick's level 4), which has not yet been established.

Limitations

As our search was limited to 2 databases, there is a risk that some relevant articles were missed. The decision to select only articles written in English also excludes studies published in other languages. Our search may also have been limited by the definitions of VR and NTS, as the keywords may not have used a very specific terminology, such as avatar, 3dimensional, HMD, mixed or augmented reality. Augmented reality, the integration of digital information into the physical world in real time, or mixed reality, the use of real objects to enhance simulations, are related to virtual environments or VR settings [START_REF] Stone | A "mixed reality" simulator concept for future Medical Emergency Response Team training[END_REF] . Thus, some studies may use simulators based on VR technology but not specify it in their keywords. However, in order to be as comprehensive as possible for this review, the search request was replicated, replacing virtual reality with augmented and mixed reality: 58 articles were found but none of them dealt with NTS training.

Another limitation of this review is that NTS are not always defined and explicitly operationalized in studies and are sometimes concealed behind technical skills on which they are highly dependent. Finally, a limitation concerns Virtual Patient simulation, focusing on the interpersonal social skills of healthcare professionals with their patient. It was decided that this did not come within the set of skills defined for this review. However, there is a vast literature on Virtual Patients, which are frequently used in virtual worlds and virtual immersive environments. A systematic literature review concerning the development of interpersonal social skills and communication with patients via VR could thus be examined in another paper. Nevertheless, the development of VR technologies and the portability of VR systems offer a very promising outlook for the future training of healthcare professionals. The wide range of possible scenarios that can be simulated, especially for NTS training, will undoubtedly contribute to the "successful integration of simulation throughout the fabric of healthcare" [START_REF] Gaba | The future vision of simulation in health care[END_REF] .

Conclusion

  training OR medical education) AND (non-technical skill* OR cognitive skill* OR team* OR leadership OR communicat* OR decision making OR task management)". We searched two main databases, Psycinfo and Medline, because they both cover our fields of interest (healthcare, VR and NTS) and because their query forms are very similar, allowing us to use the same research equation. The search was conducted in May 2017 with a final extraction for screening on 11th December 2017.

  In conclusion, VR simulation systems are a recent development in health education. The use of VR simulators has increased for technical skills training, but to a lesser extent for NTS (i.e. cognitive and interprofessional social skills). This systematic review of articles published from 2007 to 2017 shows that screen-based VR simulators or virtual worlds are the most frequently used systems, and teamwork, communication or situation awareness are the most frequently addressed NTS. The evaluation of VR systems as training tools is essential, but there has, so far, been little systematic research. The majority of studies evaluate the usability and acceptability of VR simulation, and few studies have measured the effects of VR simulation on non-technical skills development.

Figure 1 .

 1 Figure 1. Flowchart of systematic review

  Population, Intervention, Comparison, Outcome and Settings. Based on this framework, we produced a comprehensive research equation: "(nurs* OR scrub nurse OR surgeon* OR medical student OR nursing student OR health* OR clinical) AND (virtual reality OR virtual* OR virtual gaming simulation OR immers* OR serious game) AND (skills training OR

Table 1 .

 1 Characteristics of studies included in the review Supplemental Digital Content 2. PDF,

Table 2 .

 2 Objectives, measures and main findings of studies included in the review Figure 1. Flowchart of systematic review

	Records identified through database
	searching
	(n = 1373)
	Screening

Table 2

 2 Experienced team leaders were significantly faster than trainees  Realism of the environment was scored very high and realism of the VR simulator was rated high  Trainees rated the simulation more useful for technical skills training, and experts believed it more useful in enhancing communication skills  Feasability, face and construct validity of a realistic crisis scenario integrating a VR simulator has been shown

		module in VEST	 Open-ended	questions :
		simulator			improvements	and
						preferences
							 All students improved postcourse disaster preparedness knowledge scores
	Greci et al., 2013 (27)	 Develop evaluate a virtual and learning curriculum	 Open-ended technical challenges, course questions : content, immersion  Interviews and focus groups	 Emerging themes : team communication, team planning, team decision making  Functioning in an unfamiliar environment was evaluated as requiring similar skills as during a disaster where rapid decision making with incomplete information
		 Validate	a	 Global	performance :
		simulation-based		Integrated Scenario Global	 Participants significantly outperformed control group with respect to
	Grover et al.,	curriculum	for		Rating Form (ISGRF)	colonoscopy-specific performance, communication skills and global
	2015 (28)	cognitive	and	 Communication	skills :	performance during the integrated scenario format assessment 4 to 6 weeks
		integrative			Communication	Global	after training
		competencies			Rating Scale (CGRS)
		 Determine efficiency
		of	a	Virtual	 Immersion, level of comfort,	 68% of the participants felt immersed
	Heinrichs et al., 2010 (43)	Emergency Department to train mass-casaulty		confidence, usefulness for clinical skills and team training : questionnaire	 Everyone felt they learned how to interact in the simulation  "Useful","Very Useful", or "Extremely Useful" for clinical skills training for 82% participants
		incidents	(team		Focus group	 Participants gained confidence in ability to handle incidents
		skills)
		 Examine perceived
	Hudson et al. , 2015 (44)	usability of Second Life (SL) as an immersive virtual environment  Study clinical	 Perceived usability : System Usability Scale (SUS)  Situation awareness : questionnaire with 27 items	 SL considered usable in providing practice with complex scenarios of insulin administration. Perceived usability decreased among experienced nurses  No significant association between years of nursing experience and SA scores was found.
		decisions
	Khan et al., 2017 (29)	 Evaluate effectiveness of a simulation-based training curriculum of NTS on novice endoscopists' performance clinical colonoscopy. of	 Modified Form (ISCRF) Structured Assessment NTS Objective (M-OSANTS)  ISGRF  Integrated Scenario Communication Rating	 To inform the potential implementation of NTS into postgraduate gastrointestinal curricula, non-technical performance will be determined by comparing the scores from the M-OSANTS, ISGRF, ISCRF and GSE for both conditions and at 3 different times



  Performance decreased with added distractions and interruptions  Subjects rated interruptions very high in their ablity to affect performance and music distraction received the lowest mean rating  Simulators rated as realistic to present distractions and interruptions in a simulated OR, immersion evaluated as intermediate. Learner recognition of the SBAR communication tool was high accross groups  Knowledge of which component of SBAR was missing was lower accross groups  Students demonstrated increased correct recognition of strategies as they progressed through the scenarios  When they had the choice, students were more likely to chose the Two-Challenge Rule than the CUS A substantial percentage of participants did not share 3 critical items and 87% of the participants missed a dosage error  Items on Communication Checklist were missed by a substantial number of participants (introduction of self and task, closed-loop communication)  No statistically significant relationship between scores and years of nursing 

	Shamim Khan et al., 2013 (35) Umoren et al., 2017 (48)  White et al.,  Interviews : perception of  Construct-validity established : Seniors performed significantly better than  Propose an introduction to TeamSTEPPS communication tools for nursing and medical students  MCQ questions during the progression of the scenarios : designation of a TeamSTEPPS strategy, identification of a missing component of this strategy and possible selection of another strategy  Study quality of 2015 (47) information transfer and teamwork during a simulated critical event  Assess gathering and sharing of critical information  Communication skills: Critical Patient Information checklist and Interprofessional Communication Skills checklist  Non-technical skills :  Willaert et al., 2011 (37) NOTSS  Evaluate whether a  Face validity and part-task rehearsal of usefulness : questionnaire a surgical procedure  Emotional, cognitive and on a VR simulator is physical stress : short as effective as a full-version of State Trait task one Anxiety Inventory (STAI)  Establish feasability questionnaires for NTS and construct validity : the unknown and interplay between technical and non-technical skills simulation training  Feasability, acceptability  Increased cognitive load for trainees on VR simulator : pressure/anxiety about and acceptability of simulated environment junior trainees in all simulation sessions questionnaire
	Sweigart et al., 2016 (36)	 Test utility and acceptability of a virtual learning environment (VLE)  Examine change in communication in interprofessional teamwork attitudes	 Effectiveness : TeamSTEPPS (Teamwork Questionnaire)  Utility : Time to complete -TAQ Attitude type questions  Acceptability : Likert-scale questions within scenarios scenarios and answers to	 Positive student feedback on ease of use and perceived effectiveness for teaching communication and professionalism t  Scores on the T-TAQ revealed significant positive changes in leadership, situation monitoring, mutual support, and communication



  Both groups scored acceptable scores in all categories of NOTSS  Simulated procedure was found highly realistic. Simulation helped participants in decision-making, confidence, reduction of anxiety, and communication. Both strategies were as effective on stress levels  For a moderately difficult case, a part-task patient specific VR rehearsal is as effective as a full-task one Training resulted in a decrease of time, but significantly slower performances when crises  The more workload was experienced, the poorer was the surgical performance.  Telephone call seemed more disturbing compared to patient discomfort

	Wucherer et al., 2015 (38)  Youngblood et  Measure usability of simulator  Explore relationship between mental workload and surgical performance during crisis  Cognitive workload : 3-item questionnaire and Surgery Task Load Index (SURG-TLX)  Questionnaire : face validity and training value al., 2008 (39)  Evaluate VLE for leadership  Leaderdhip skills : EMCRM and (Emergency Medicine Crisis trauma management Resource Management) by comparing users' experience with a scale



  All participants evaluated simulation as "useful" or "very useful" to assess and manage trauma patients in Emergency Department (ED)  All participants showed significant improvement in team leadership  Students emphasized emotional impact of simulation in VLE  Both mannequin-based and VLE simulation of ED cases are valid training methods to improve EMCRM team leadership skills

	high-fidelity patient	 Assessment of learning
	simulator (PS)	experience : debriefing and
		questionnaire
		6

Funding This work is supported by the French Ministry of Research and Education as part of the "Laboratoires d'Excellence", Labex CominLabs, SunSet project: "Scrub Nurse Non-Technical Skills Training System".

Conflict of Interest

The authors declare that they have no conflict of interest. [START_REF] Sweigart | Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students[END_REF] Ambulance practitioner, surgical registrars, emergency department consultants X Second Life® + OpenSimulator X X X X X X X X emergency medicine

Greci et al., 2013 [START_REF] Kirkpatrick | Evaluating training programs the four levels[END_REF] Emergency department nurses and hospital administrators X MUVE (Multi User Virtual Environment) using Second Life® X X X X X emergency medicine

Heinrichs et al., 2010 [START_REF] Wucherer | Vertebroplasty Performance on Simulator for 19 Surgeons Using Hierarchical Task Analysis[END_REF] Physicians and nurses X VED II (Virtual Emergency Department) an online virtual world X X emergency medicine

King et al., 2012 [START_REF] Abelson | Virtual operating room for team training in surgery[END_REF] Medicine, nursing, respiratory therapy, pharmacy, speech language pathology, diagnostic medicine sonography students [START_REF] Youngblood | Design, development, and evaluation of an online virtual emergency department for training trauma teams[END_REF] Nurses X Second Life® X X X X health education Riesen et al., 2012 [START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF] Recent graduates from nursing, paramedic, police and child and youth service programs

Rogers, 2011 [START_REF] White | Using a Critical Incident Scenario With Virtual Humans to Assess Educational Needs of Nurses in a Postanesthesia Care Unit[END_REF] Nursing students X Second Life® X X X health education Sweigart et al., 2016 [START_REF] Maschuw | The impact of self-belief on laparoscopic performance of novices and experienced surgeons[END_REF] 9 Health professional students X VLE with avatar X X X X X health education Umoren et al., 2017 [START_REF] Paige | From the flight deck to the operating room: an initial pilot study of the feasibility and potential impact of true interdisciplinary team training using high-fidelity simulation[END_REF] 4 Nursing and medical students X VLE with avatar X X X [START_REF] Greci | vTrain: a novel curriculum for patient surge training in a multi-user virtual environment (MUVE)[END_REF]  Improve interprofessional competencies  Determine acceptability of a blended learning environment  Self-perceived changes in interprofessional attitudes and competence : IEPS, ICCAS  Team performance assessment : TOSCE  Students perceptions : program assessment tool, and 16-item questionnaire  Significant differences pre and post workshop were found in ICCAS and IEPS scores  Significant improvement accross the 3 simulations in all competencies  Program and learning experience were highly rated  Learner confidence and performance can be improved through education delivered in a virtual environment