PhD Stanley Mugisha
email: stanley.mugisha@edu.unige.it.

PhD Mirko Job

PhD Matteo Zoppi
email: matteo.zoppi@unige.it

PhD Marco Testa

PhD Rezia Molfino
email: rezia.molfino@unige.it

Zhenhong Li

Wei Meng

Tianliang Li

Vamsi Krisha Guda
email: vamsikrishna.guda@ls2n.fr

Christine Chevallereau
email: christine.chevallereau@ls2n.fr

Damien Chablat
email: damien.chablat@cnrs.fr

Immersive Virtual Reality

Keywords: BiTRRT Bi-directional transition-based RRT BRRT Bi-directional RRT CAD Computer Aided Design CAE Computer-aided engineering CAVE Cave Automatic Virtual Environment CT Conventional therapy Stroke-Rehabilitation-Virtual Reality-Immersive Virtual Reality-Computer Mediated Therapy Trajectory planning, Human safety, Haptic interface, Intermittent contact interface cobot, human motion prediction, human safety haptic devices, response time, human-robot interaction, virtual reality, eye-gaze tracking

Virtual reality has been recognized as a potent tool for creating more natural and intuitive human-computer interfaces and has been found to be beneficial in so many applications. However, the inability to interact in a virtual environment through touch compromises its realism and usefulness. Haptic interfaces with intermittent contacts allow users to reach out and touch the virtual objects physically to simulate contact between the user and the environment with use of tactile sensation to increase the realism of interaction. They allow a wide range of physical interactions throughout the user's workspace, with a physical input that resembles reality. These devices are faced with challenges such as cost, a small workspace, limited speed and user safety. In this thesis, we developed a haptic interface using a cooperative robot to address these challenges. Several motion strategies, a trajectory generation scheme and user interaction techniques to ensure safety were developed and evaluated. Two case studies were used as application areas. The first is an industrial application for analysis of the interior material of the car during the early stages of development while the second one is a haptic interface for upper limb rehabilitation training. User studies conducted to evaluate the efficacy of the motion strategies showed significant improvement in device speed, response, and user safety.

Virtual reality

For decades, scientific research has been ongoing in the field of Virtual Reality (VR), recognizing it as a potent tool for creating more natural and intuitive human-computer interfaces (HCI). The phrase "Virtual Reality" was invented by Jaron Larnier in 1994 Ken, Pimentel and Kevin (1994) to describe a real-time, computer-based, interactive, multi-sensory simulation environment that enables users to engage in activities inside environments that resemble real-world artifacts and events to varying degrees Saposnik et al. (2016). Other closely related terms include "Artificial Reality," and, more recently, "Virtual World" have been used to describe a Virtual environment (VE) Krueger et al. (1985). Users in a typical VR environment have several feedback senses such as visual, auditory, and tactile rather than the only vision information available in most computer graphics applications and can interact with virtual objects naturally and intuitively.

Fundamentals of virtual reality

An essential concept in a VR system is immersion, which refers to the user's perception of being fully immersed in an artificial, 3D world entirely generated by a computer. Immersion is a significant advancement over traditional 3D computer graphics animation and CAD modeling packages, mainly natural and intuitive user interaction. Because the line between immersion and non-immersion is becoming increasingly blurred, the term "Virtual Reality" is now also applied to applications that are not fully immersive. Due to the existence of the various terms used in different applications, VR systems currently take many forms, including cyberspace, synthetic environment, artificial reality, virtual world, and virtual environment. The standard features of all such VR-related systems include a natural or intuitive interface for user interaction, real-time 3D graphics 1 1.1 Virtual reality for synthetic presentation, and a sense of immersion. Based on the interactive means used and the number of sensory channels engaged in simulation, VRbased systems are divided into two major types. The first type is immersive VR(IVR) which completely replaces the user's real-world environment with a simulated one Kim et al. (2017). Thus users get the sense of being transported into three-dimensional, interactive worlds from where they can participate in various activities in imaginary environments through 360 o immersion in an alternate reality. IVR is based on immersive display technologies like head-mounted displays (HMDs) or stereo projections and video captures systems such as the Immersive Rehabilitation Exercise (IREX) system and the Cave Automatic Virtual Environment (CAVE) Chen et al. (2019). However, HMDs and head position trackers are challenging to use for extended periods of time in an IVR system, while the CAVE and IREX are quite expensive. The other type of VR system is non-immersive (NIVR) which has emerged from 3D CAD animation technologies. NIVR systems are typically less expensive than IVR systems, users view and interact with virtual objects in a 3D environment displayed on a desktop or workstation using technologies such as stereoscopic displays. The individual can see their avatars reflected on the screen Mirelman et al. (2016); Ögün et al. (2019). However, NIVR systems lack the experience of immersive features such as scene changes with head movements, which could play a role in the experience of presence. People can use devices such as 3D trackers, electrical hand gloves, and haptic force feedback devices to interact with the 3D world. Additional features such as voice input recognition and sound feedback output can be used to improve the usability of a VR system without the need for expensive additional hardware.

VR applications

Recent advances in VR have led to a proliferation of low-cost consumer VR devices available on the market affordable for everyone, and its applications have become practically limitless. Today, VR technologies are widely used in healthcare Laver et al. (2017); Mugisha et al. (2022b), product and process design Banerjee & Zetu (2001), in flight simulators and human factors Oberhauser & Dreyer (2017) among others. It is expected that VR will reshape the interaction interfaces between the user and computer technology by providing new approaches for information communication, process visualization, and creative expression of ideas. However, I shall limit the scope of this thesis to only industrial and healthcare applications.

Virtual reality in industrial applications

As advanced input and output hardware devices are used in VR, it is often regarded as a natural extension of 3D computer graphics. This technology has recently matured to the point where people can use it in industrial applications. Integrating this new technology with software systems for industry, engineering, design, and manufacturing will boost computer-aided engineering (CAE). At the moment, there is evidence of increasing global market competition. As a result, the industry is under increased pressure to reduce product life cycle costs, maintain product quality, improve product performance, and reduce product design cost and fabrication time Banerjee & Zetu (2001). By speeding up the product development process, improving product quality, and reducing product design errors, virtual product design and development can be considered one of the enabling technologies for the rapid development of information technology infrastructure in this area.

Virtual reality in healthcare applications

Virtual reality technologies provide numerous benefits to the medical community. These include more realistic certification procedures (for example, objective measures of surgical skill) and more exciting treatments such as the case of virtual rehabilitation of patients with neurological disorders such as stroke. Stroke represents a severe problem for public health with a significant prevalence Feigin (2017). Recovery is always incomplete, and most survivors are left with motor, sensory and cognitive impairments with a consequent increase in the burden of health care expenses during adulthood Levine et al. (2013). The economic cost of related neurological disorders is increasing (Olesen et al., 2012) and this burden has raised the need to pursue new cost-effective rehabilitation strategies both independent of and complementary to traditional methods such as physiotherapy, occupational therapy, and speech therapy (Sihvonen et al., 2017). Owing to the rising number of neurologically impaired survivors, patients with neurological conditions often require rehabilitation in the early stages, and some require rehabilitation regularly throughout their lives. Virtual reality is a promising tool for rehabilitation to help patients regain their ability to live independently. VR therapies aim to create more fun, practical and task-specific rehabilitation programs while giving patients feedback on their performance, meaningful goals, and a personalized experience to support motor learning Perrochon et al. (2019). In addition, by gamifying the training through VR applications, the rehabilitation process is made fun and enjoyable. The experience of immersive features (e.g scene changes with head movements, sensations) plays a significant role in the experience of presence and, therefore, in the therapeutic benefits of that experi-1.2 Haptic systems ence (Colloca et al., 2020). However, IVR has not been used extensively as NIVR in the rehabilitation of neurological disorders so there are a few studies have been conducted to show the benefits of immersive VR. This thesis seeks to build the knowledge gap by evaluating the efficacy of IVR compared to NIVR and CT in the rehabilitation of stroke Laver et al. (2017) using a systematic review and meta-analysis of randomized controlled trials Mugisha et al. (2022b).

Haptic systems

A milestone in VR history was the head-mounted display (HMD) by Ivan Sutherland in 1968 Sutherland (1965). His stereoscopic display system made it possible to perceive three-dimensional environments visually. Until recently, VR was defined as "technology that allows the user to perceive and experience sensory contact with a non-physical world" Easton et al. (1997). The user interacts with the contents of a VE but cannot touch any element of the environment itself. Considering how much of life revolves around physical interaction with the environment, which is only possible through touch, the significance of touch becomes clear. It is frequently used to supplement what we see and hear and it provides information about object properties that the other senses do not have access to (e.g., temperature). As a result, it is fair to say that the inability to interact with a VE through touch compromises its realism and usefulness Durlach & Mavor (1995). Biggs et al. point out that incorporating the sense of touch into VR is critical for simulating tasks where touch is essential, such as medical applications Biggs & Srinivasan (2002). There is evidence to suggest that VR with touch is perceived as more realistic than VR without touch Hodges et al. (2001). Therefore, incorporating touch into VR translates to more natural and realistic interactions with VEs, improves the accuracy and time required to perform simple manual tasks, such as manipulating virtual objects Biggs & Srinivasan (2002); Durlach & Mavor (1995). For example, consider the task of reaching, picking up, and placing an object such as a ball from one location to another in the absence of information from the sense of touch. For a user to complete the task with only visual and auditory information, he would be unable to feel the contact between their hand and the object, the weight of the object, or any contact the object makes with the surface to which it is placed. The lack of this information would have a negative impact on how easily the individual could complete the task. To enable task completion in the thesis, I added the sensation of touch in VEs by designing an exergame to enable users to reach and touch balls in a Virtual environment for rehabilitation training. The term haptics is derived from the Greek word haptesthai, which means "to touch," linked to the science of applying touch (tactile) sensation and control to computer application interaction. Photorealistic 3D en-

Haptic systems

vironments can be modeled and rendered almost in real-time through a process called haptic rendering to reproduce virtual, dynamic interactions mechanically, thus reducing the gap between the virtual world and the real world. Haptic rendering is done by computing and generating forces in response to user interactions with virtual objects so that an individual using VR cannot just see objects but also touch and feel them Salisbury et al. (1995). Thus, haptic rendering can be considered a gateway between the virtual and physical worlds. However, realistic haptic rendering is still an open issue in the domain of gaming, robot control LIU et al. (2019), training/rehabilitation Bortone et al. (2018), and human-machine interaction Sanfilippo & Pacchierotti (2020); Sanfilippo et al. (2015). Since the potential of the sense of touch and the convincing aspects of haptic sensation through haptic feedback can improve interaction in a VE, researching novel interfaces is paramount to increase presence. By combining a 3-dimensional visual representation of the task and a hopefully realistic physical interpretation of the virtual world via a haptic force feedback device, haptic devices can allow a user to be immersed in a virtual scene Hannaford et al. (1991).

Haptic systems should allow for a wide range of physical interactions and manipulations throughout the user's workspace, with a physical input that resembles reality. One promising approach to achieve this is the paradigm of encounteredtype haptics (Yokokohji et al., 1996) with intermittent contacts. Encountered type haptic devices (EHDs) are devices that autonomously position physical props for virtual objects in the real world at a target appropriately, thus allowing users to reach out to and touch the virtual objects physically, just like in the real world. They use a mobile physical prop, usually actuated by a fixed or mobile robot, that follows the user's hand and only comes into contact with it when necessary to simulate contact between the user and the virtual environment.

Applications of haptic devices

Haptic devices have been widely used in many sectors such as education and training, entertainment, and product design. Since the application areas are broad, the scope of this thesis will be restricted to applications in healthcare and product design.

Haptics in Health care

With the latest technological improvements in virtual reality and robotics, a growing body of research is investigating robotics to support rehabilitation and functional assessment following neurological conditions such as stroke and motorcognitive impairments. Studies highlighting the efficacy of haptic-robotic approaches to upper-limb rehabilitation after Stroke and other neurological disor- (2011). Many new, relatively low-cost haptic technologies, such as SensAble's PHANTOM OmniR and the Novint Falcon, are now being produced with a small form factor that allows these devices to be used in subjects' homes.

Robotic devices can provide high-intensity, task-specific, and repetitive therapy comparable to CT. These technologies aim to replace conventional, labourintensive therapy techniques and typically require one-on-one sessions with therapists. In addition to the potential for long-term cost savings from a therapist's relaxed involvement, patients can perform robotic exercises under the supervision of a non-specialist clinician or in a patient's own home. Haptic rendering has shown indications of enhancing learning progression and user experience in people with reduced motor skills due to stroke (Bortone et al., 2018).

The physiotherapeutic benefits of immersing a person in a virtual world with the aid of haptic rendering include: physical engagement of patients, keeping the training on a stationary rig for optimal monitoring and assessment, the possibility to create controlled, user-specific exercises and difficulties, and to include objectives and rewards in the training routines (Mirelman et al., 2016).

However, including haptic rendering in physiotherapy also poses challenges. Firstly, the small workspace of the devices and limited speed. Secondly, there is always a safety concern in situations where a machine directly collaborates with a human. Therefore, many considerations must be made when designing and developing a haptic render-based rehabilitation system for physiotherapists and home use. These considerations, along with ethical problems and cost-overfunction, all contribute to limiting its current use as tools used by health experts. This usage limitation affects research on the topic and impairs the advancement within the field.

Haptic interfaces for product design in the automotive industry

In the automotive industry, research professionals in perceived quality and sensory perception are responsible for discovering and selecting relevant materials and shape associations to ensure perceptual coherence and homogeneity of an entire project. The evaluation of their choices is based primarily on the modalities of vision and touch Crochemore et al. (2004); Posselt et al. (2017). Haptic interfaces have also been beneficial in product design by engineers to evaluate the textures and appreciate the quality of the materials of the interior of a vehicle in the early stages manufacturing process. Digital mock-up modifications are much easier and faster to perform, allowing for exploring many more hypotheses than traditional physical mock-ups.

Motivation

While current immersive visualization systems can provide an acceptable level of realism, touch simulation remains a significant challenge. ETH devices with intermittent contacts combine the flexibility of programmable active haptic devices with the inherent ability of passive mechanical devices to provide high-fidelity tactile and kinaesthetic feedback.

VR allows the rendering of interior car components, and the interaction between a user and a robot is made possible by the analysis of data from different integrated sensors. These interactions allow for creating a touch-sensitive interface or ICI Araujo et al. (2016); Kim & Kim (2016). The scenario allows the user inside a virtual car to interact with its environment by getting sensory feedback from different surfaces, thanks to a six-face prop providing the different textures. This use case aims to use an EHD to provide high-fidelity tactile and kinaesthetic feedback in an IVR system that is safe for the user while providing a level of interaction between VR environments and the real world.

Motivation

Among the main shortcomings of haptic interfaces are a limited workspace, low stiffness, speed, high cost, and user safety.

While current immersive visualization systems can provide an acceptable level of realism, touch simulation remains a significant challenge. Furthermore, despite recent technological and scientific advances in touch simulation, achieving high fidelity tactile simulation remains difficult. This thesis seeks to address the above challenges.

For the applications envisaged in this thesis, a Cobot carries several texture specimens on its end-effector and a ball to allow contact between a user's finger or hand and the robot CLARTE et al. (2020); Posselt et al. (2017).

Due to the user's immersion via a VR headset, the system must ensure the user's safety, as he cannot see the robot's location. Therefore, it is necessary to implement trajectory planning techniques to avoid unwanted interactions between the robot and the user. The system must consider the obstacles present (environment or user). A virtual sphere is modeled to estimate the user's position and give the system a model to plan the movements. The most important thing to consider in a human-robot interaction system is the user's safety. Safety is guaranteed through the generation of a motion planning scheme that takes into consideration the obstacles present in a VR environment and human intention detection. Due to workspace and speed limitations, it is challenging for real-time interaction to organize physical props that accurately replicate the virtual world due to practical constraints, (Gonzalez et al., 2020). Speed limitations delay the de-

Research objectives

The main objective of this thesis is to address the challenges currently prohibiting the implementation of an ICI in two contexts: industrial and health care. I developed an ICI prototype based on a cobot to meet security requirements in the event of involuntary contact with the user through the generation of a motion planning scheme that takes into consideration the obstacles present in a VR environment, then human intention detection and motion prediction. Specifically, this thesis has the following objectives. 1. To evaluate the efficacy of different forms of VR treatments in improving the physical and psychological status of stroke patients.

2. Develop a haptic Interface with props for renderable virtual shapes and materials user interaction.

3. Develop safe motion planning schemes taking into account the cobot's low velocity and collision between every robot segment and the user.

4. Develop human intention detection and prediction models to both specify a target location for the prop and a user-safe trajectory to reach that location.

5. Conduct user studies to evaluate the interaction techniques and prediction models.

Research Contributions

1. Evaluated the efficacy of VR treatments as either IVR or NIVR compared to CT in improving physical and psychological status in patients with stroke. Chapter 3 details the implementation of the motion planning scheme to ensure the user's safety and implement a reliable system. This section describes the motion planning scheme that considers the obstacles in a VR environment. This one has to be guaranteed in order to be able to implement a reliable system. This study is a continuation of the work presented by CLARTE et al. (2020); Guda et al. (2020) where the main objective is to generate a safe motion scheme that takes into consideration the obstacles present in a VR environment with the application for analysis of the material of an interior of the car Gutierrez et al. (2022). The work has been developed using MoveIt software in ROS to control a UR5 industrial robot. With this, I will be able to set up the planning group, which is confirmed by the UR5 robot along with a 6-faced prop and the base of the manipulator to plan feasible trajectories for it to execute within the environment.

Chapter 4 describes movement strategies for the robot to be as fast as possible in the contact zone while guaranteeing safety. This work uses the concept of predicting the user's position through his gaze direction and the position of his dominant hand (the one touching the object). The case study presented here analyses the material of the interior of the car for the automotive industry Guda et al. (2022); Mugisha et al. (2021).

Chapter 5 details the development of a VR exergame for reach and grab tasks designed for upper limb rehabilitation training. I further present more prediction strategies based on eye gaze and conduct a user study to evaluate the efficacy of each strategy's effect of eye gaze window and hand threshold on the speed and response of a haptic device Mugisha et al. (2022a).

Discussions and future work are presented in chapter 6.

Introduction

Stroke has been described as one of the significant causes of death and disability globally, representing a severe problem for public health with a significant prevalence in men and women of all ages. 1,2 Recovery is always incomplete, and most survivors are left with motor, sensory and cognitive impairments with a consequent increase in the burden of health care expenses during adulthood. 3 Owing to the rising number of neurologically impaired survivors, several computer-mediated programs for stroke rehabilitation have recently been developed to help patients regain their ability to live independently. In particular, the advancement of digital technology has favored the assertation of virtual reality (VR) as an accessible solution to give patients feedback on their performance, meaningful goals, and a personalized experience to support motor learning. 4,5 Training can be gamified through various applications, making the rehabilitation process fun and enjoyable. VR is a real-time, computer-based, interactive, multisensory simulation environment that enables users to engage in activities inside environments that resemble real-world artifacts and events to varying degrees. 6À8 Depending on the quantity of visual sensory channels engaged in simulation, VR can be categorized as either immersive or non-immersive. Immersive VR (IVR) replaces the user's real-world environment with a simulated. 9,10 Users get the sense of being transported into three-dimensional interactive worlds through 360°i mmersion in an alternate reality such as a head-mounted display (HMD) or video capture systems such as IREX, which enables them to participate in various activities in imaginary environments. 11À13 In non-immersive VR (NIVR), the user mainly interacts with virtual objects displayed either in a 2D or 3D environment that can be directly manipulated on a conventional graphics workstation using a keyboard and a mouse. As in IVR, animation and simulation are interactively controlled to the user's direct manipulation with some NIVR systems and allow individuals to see their avatars reflected on the screen. 14À17 Non-immersive systems are characterized by a lower level of immersive features (e.g., scene changes with head movements), which could play a role in supporting the feeling of presence and its therapeutic benefits. 13 For instance, it has already been pointed out how immersion in the virtual simulation plays a pivotal role in pain management by inducing relaxation.

Previous reviews have studied the use of VR in the rehabilitation of stroke. 18À21 However, none of these works differentiated their results according to the VR modalities. Since the type of VR seems to influence the rehabilitation outcomes differently depending on the level of immersion, 18,19 it is vital to unravel which form yields the best treatment effects for motor rehabilitation and other outcomes important to people with stroke such as quality of life and participation.

Hence, by differentiating between IVR and NIVR systems, this systematic review and meta-analysis of randomized control trials aimed to evaluate the efficacy of VR treatments in improving physical and psychological status in patients with stroke compared to conventional therapy (CT).

Methods

Registration number

The review protocol and inclusion criteria were prespecified and registered on the National Health Service Prospero Database under the registration number: CRD42019134806. This systematic review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

Electronic searches

The literature search was conducted on the following databases: MEDLINE (via PubMed), Scopus, Web of Science, ACM Digital Library, and IEEE Xplore. Additional articles were retrieved by scanning the reference lists of those studies that passed the "full-text" screening stage.

The search strategy was organized by extracting a set of keywords from three primary groups representing:(i) technology, (ii) rehabilitation, and (iii) pathology-related semantic fields. Terms were connected using the OR boolean operator for within-group connections and the AND boolean operator between-group relations. Both free and MeSH terms were used to build the final search string, reported in detail in Appendix I. Results obtained from the database research were filtered to include only those published after 2015, chosen as the lower temporal limit because Immersive Virtual reality is relatively a new technology, and its use in neurological rehabilitation is still in the early stages. Other works were retrieved by scanning the reference lists of all included studies.

Eligibility Criteria

To be included, articles had to meet the following eligibility criteria. (i) a randomized controlled trial that (ii) considered subjects more than 18 years old, (ii) affected by neurological disorders, (iii) and compared computermediated treatments against conventional therapy (iv) for the upper limbs or lower limb motor functions (v) or postural control. Moreover, only full-text articles written in English were included in the screening process. We excluded those articles that: (i).compared one or more different types of computer-mediated reality-based treatments without an alternative control group; (ii) involved high-cost devices such as treadmills, CAVE, and any form of a robotic manipulator. (iv) Other neurological conditions apart from a stroke.

Study selection

In May 2020, we started a comprehensive systematic search. Duplicates across databases were removed, and the remaining studies were screened for titles, abstracts, and descriptors by the two reviewers independently (MS, MJ) to assess whether they met the predefined inclusion criteria. Controversies between the reviewers concerning the eligibility were resolved in a consensus meeting. After reaching an agreement, full texts of potentially eligible studies were retrieved and further assessed against the inclusion criteria, and reasons for excluding the studies were documented.

Risk of Bias

We used the Critical Appraisal Skills Programme for randomized controlled trials 22 to assess the risk of bias in the included trials. It is 11 questions checklist with three sections that assess the following items: validity of study results, what the results are, and how they are helpful locally. Each question in each subsection required a positive('yes'), neutral (Can't tell), or negative answer('No').

The articles were classified as low risk, moderate risk, or high risk of bias according to the number of items that received a negative appraisal. Those articles in which all checklist items were appraised positively were considered low risk of bias works. Articles in which one or two of the checklist items were appraised negatively were considered a moderate risk of bias. Articles in which three or more items were appraised negatively were considered high risk of bias works.

The two authors independently assessed the quality of the work, and any disagreements that arose during the process were resolved through discussion.

Data extraction

Two review authors (MS and MJ) independently extracted data into a a custom data table and the following data were chosen to be extracted from each study:

(i) citation details, (i) Population characteristics; (ii) inclusion and exclusion criteria (ii) Type of intervention (iii) Technology Used (iv) outcome measures; (v) main results. One author (M. S) extracted data, and another (M. J)checked for accuracy. For studies where values were provided in an unconventional format (i.e., medians [interquartile range], or means [minimum-maximum range] 23À25 ) the sample mean and standard deviation were estimated as described in Wan et al. 2014. 26 From this, the effect sizes of the main outcomes were calculated.

Data synthesis

Two review authors (MS and MJ) independently classified outcome measures in terms of the domain assessed ((i)Upper limb activity and function (ii) lower limb activity and function, (iii) balance, (iv) activity of daily life, and (v) adverse events). When more than one outcome measure for the same domain was presented in a study, the most frequently used across studies was considered in the analysis. The standardized mean differences (SMD) were calculated for continuous outcomes and Cochrane's Review Manager 5 (Review Manager 2014) software was used for all analyses. The effect of the intervention was measured using cohen's d on the primary outcome of each study. In trials with three-armed interventions, 27 the VR therapy was compared to conventional interventions.

Meta-analysis

Pooled results were used to present an overall estimate of the treatment effect using a random-effects model in the analysis across studies. Heterogeneity was assessed through the I 2 statistic. 28 The level of heterogeneity was considered substantial if the I 2 statistic was greater than 50%. A meta-analysis was considered not appropriate where the level of heterogeneity was substantial or when only one study was identified for the desired outcome. In this case, a narrative summary of the results was given. Where data pooling was decided, forest plots were provided along with a description of the results. When applicable, a sensitivity analysis was performed, including only studies at low risk of bias, and the results were compared to the primary analysis, including all the trials.

Results

Search results

The study inclusion workflow is displayed in detail in Fig. 1. We identified a total of 1573 possible record from the database research (ACM: n=18,Pubmed: n=1382, Cochrane: n=17, IEEE: n=29, Science direct: n=60, Scopus: n=38, Web of Science: n=29). Additional 151 Records were identified through secondary sources. After removing records duplicates, 1680 studies were evaluated for title and abstract, resulting in 95 full-text articles assessed against the previously defined eligibility criteria. At the end of this process, 22 fullÀtext articles were included in the present review (Adie et 29,30 were excluded from the meta-analysis. 40 studies did not meet the inclusion criteria because they were published before 2015. 8 studies were excluded from this review because the VR intervention methods used expensive devices such as treadmills and robotic manipulators or actuated devices 31À36, 37 3 studies were excluded because they compared virtual reality with no intervention. 38À40

Risk of bias

The critical quality appraisal of the 22 included trials is presented in Table 1. Relying on the CASP guidelines 22 for a randomized trial, we identified 2 of the included studies to be at high risk of bias, 42,47 3 at low risk of bias 17,46,[START_REF] Shin | Effects of virtual reality-based rehabilitation on distal upper extremity function and health-related quality of life: a single-blinded, randomized controlled trial[END_REF] and 17 at moderate risk of bias. 8,23,25,27,29,30,41,43À45,48À53,55 In 2 trials, 42,47 the authors did not report the information about the randomization method of patients. In. 48 , the treatment between groups was not the same. The experimental group received 30 extra minutes of VR training for 20 sessions making up 10 hours. 7 studies reported a large treatment effect of the Intervention measured using Cohen's d, 17,42,45,47À49,52,54 while the rest had a medium to small effect size. It is also important to consider that small sample sizes characterized most studies. A total of 13 ) had a small sample population of about 10 to 16 participants in the experimental group, resulting in broader confidence intervals and therefore were regarded to be of high risk of bias. Due to the nature of the study, blinding of the patients, physiotherapists, and assessors who supervised the treatments was not possible. Therefore it was not considered as a possible risk of bias.

Population characteristics

Of all 22 trials, 7 trials had a sample size of more than 50 participants. 8,17,27,41,45,50,[START_REF] Schuster-Amft | Effect of a four-week virtual reality-based training versus conventional therapy on upper limb motor function after stroke: a multicenter parallel group randomized trial[END_REF] And 6 trials had less than 25 participants. 47,49,51,[START_REF] Mekbib | A novel fully immersive virtual reality environment for upper extremity rehabilitation in patients with stroke[END_REF][START_REF] Zondervan | Home-based hand rehabilitation after chronic stroke: Randomized, controlled single-blind trial comparing the music glove with a conventional exercise program[END_REF][START_REF] Shih | Effects of a balance-based exergaming intervention using the Kinect sensor on posture stability in individuals with Parkinson's disease: a single-blinded randomized controlled trial[END_REF] . A detailed overview of the population characteristics is provided in Tables 2 and3 for IVR and NIVR interventions respectively.

Inclusion and exclusion criteria

Most studies specified the inclusion and exclusion criteria apart from two where the exclusion criteria were not given. 42,49 Most Participants in the included studies appeared to be relatively young, with mean ages ranging from 29 to 75 years in all studies. Studies omitted medically ill participants, such as specified by the presence of a disease in which exercise was contraindicated. Participants were included if they were cognitively intact, as defined by cut-off scores on the MMSE. Medically unstable participants were excluded, as defined by having a disease in which exercise was contraindicated, severe visual disorders 46,51,[START_REF] Zondervan | Home-based hand rehabilitation after chronic stroke: Randomized, controlled single-blind trial comparing the music glove with a conventional exercise program[END_REF] and neurodegenerative disorders. 23

Interventions

In all retrieved trials, the active control group performed similar exercises as those proposed in the VR intervention. For IVR, 3 studies implemented an equally matched dose of CT, 17,[START_REF] Schuster-Amft | Effect of a four-week virtual reality-based training versus conventional therapy on upper limb motor function after stroke: a multicenter parallel group randomized trial[END_REF][START_REF] Zondervan | Home-based hand rehabilitation after chronic stroke: Randomized, controlled single-blind trial comparing the music glove with a conventional exercise program[END_REF] while 1 trial consisted of a combination of VR with CT. [START_REF] Shin | Effects of virtual reality-based rehabilitation on distal upper extremity function and health-related quality of life: a single-blinded, randomized controlled trial[END_REF] For NIVR interventions, 7 trials had the intervention group that performed only VR exercises, 8,27,29,41,42,44,46 7 studies had a VR intervention group performing a combination of VR exercises augmenting CT. 23,25,30,45,48À51 Fig. 1. Preferred reporting items for systematic reviews and meta-analyses flowchart of study retrieval, screening, and eligibility.

One trial consisted of three-armed interventions including a VR group, an active control group performing similar exercises within a conventional physiotherapy setting, and a passive control group. 43 In all trials, therapy sessions lasted between 45 minutes and 2 hours per day, with a minimum duration of therapy lasting for 16 hours and a maximum of 40 hours.

Technology

Participants in the IVR group received training using immersive devices such as head-mounted displays, 17,47,[START_REF] Mekbib | A novel fully immersive virtual reality environment for upper extremity rehabilitation in patients with stroke[END_REF] a leap motion controller, 17,[START_REF] Mekbib | A novel fully immersive virtual reality environment for upper extremity rehabilitation in patients with stroke[END_REF] and Htc Vive controllers in. 47 While For NIVR interventions, participants mainly used the Microsoft Xbox Kinect connected to a display monitor, 23,42,46,48 Nintendo Wii gaming system, 8,27,29,30,41,44,45 motion tracking sensors, 25,50,51 and data gloves. 53À55 A detailed overview of the IVR and NIVR interventions is provided in Table 4 andTable 5, respectively.

Outcomes

Due to various intervention approaches, a wide range of outcome measures was retrieved. The outcome measures were collected at the baseline and soon after the intervention. In all the trials, a post-intervention follow-up assessment was done in only 7. Of these, 5 had a postintervention follow-up of less than 3 months, 8,27,45,[START_REF] Shin | Effects of virtual reality-based rehabilitation on distal upper extremity function and health-related quality of life: a single-blinded, randomized controlled trial[END_REF][START_REF] Zondervan | Home-based hand rehabilitation after chronic stroke: Randomized, controlled single-blind trial comparing the music glove with a conventional exercise program[END_REF] and 2 had more than 6 months. 30,41 An overview of all outcome measures for each predefined outcome category and results for the primary outcome in the included studies can be found in Tables 6 and7 for IVR and NIVR respectively.

Immersive virtual reality vs CT (Short term Effects)

Upper limb activity and function Upper limb function was accessed using FM as the outcome measure. In 3 trials with 106 participants, 17,47,[START_REF] Mekbib | A novel fully immersive virtual reality environment for upper extremity rehabilitation in patients with stroke[END_REF] Table 1. Critical appraisal table. Indicates positive, negative responses from both authors, respectively shown in green, and red.

COMPUTER-MEDIATED THERAPIES FOR STROKE REHABILITATION

improvements were reported in both training groups post-intervention. A meta-analysis revealed significant inter group differences with greater improvements in the IVR group compared to CT (smd:1.37, 95% CI [0.80, 1.93], I 2 =35%, P<0.00001). A forest plot is shown in Fig. 2.

A sensitivity analysis of results without 47 considered to be a high risk of bias revealed similar results(smd:1.57, 95% CI [1.03, 2.11], I 2 =15%, P=0.2).

Upper limb activity was assessed by BBT as the outcome measure. Results from one study 47 with 18 participants reported improvements in both groups. IVR registered a higher improvement than CT. However, the difference between the groups was not significant IVR: (0.08 §0.14), CT(0.04 §0.06), (smd= 0.35, 95% CI [-0.58, 1.29], p>0.05). This was a low-quality study with large confidence intervals.

Activities of daily life (ADL)

Three trials 17,47,[START_REF] Mekbib | A novel fully immersive virtual reality environment for upper extremity rehabilitation in patients with stroke[END_REF] investigated the effects of IVR on ADL. A meta-analysis indicated greater improvement in the IVR group with a moderate effect size (smd = 0.54,95% [CI 0.15 to 0.93], I 2 = 0%, p=0.007). A forest plot is shown in Fig. 3.

A sensitivity analysis without 47 considered high risk of bias revealed similar results (smd = 0.61,95% [CI 0.18 to 1.04], I 2 = 0%, p=0.005).

Adverse events

In all studies, the occurrence of adverse events was not reported.

Non-Immersive VR Vs CT (Short Term Effects)

Upper limb activity and function

Nine trials reported results on upper limb function using FM as the outcome measure. 23,25,44,46,48,50,[START_REF] Shin | Effects of virtual reality-based rehabilitation on distal upper extremity function and health-related quality of life: a single-blinded, randomized controlled trial[END_REF][START_REF] Kong | Efficacy of a virtual reality commercial gaming device in upper limb recovery after stroke: a randomized, controlled study[END_REF] Significant improvements were reported in both groups post-intervention. A meta-analysis was not performed because of high heterogeneity (I 2 = 82%) in the data. According to results from individual studies, NIVR reported greater improvements than CT.

Four trials with a total population of 172 participants reported greater improvement in NIVR compared to CT with a large effect size, Aşkın et In one study. 30 , with 41 participants, we could not get data in a suitable format for analysis. However, they reported no significant difference between groups.

Three trials with 109 participants measured upper limb activity using BBT as the outcome measure. 23,[START_REF] Schuster-Amft | Effect of a four-week virtual reality-based training versus conventional therapy on upper limb motor function after stroke: a multicenter parallel group randomized trial[END_REF][START_REF] Zondervan | Home-based hand rehabilitation after chronic stroke: Randomized, controlled single-blind trial comparing the music glove with a conventional exercise program[END_REF] A metaanalysis revealed no significant difference between the groups. The overall effect size was small (smd = 0.19, 95% CI [-0.19, 0.57], I 2 = 0%, P=0.33) as shown by a forest plot in Fig. 4.

Lower limb activity and function

Three trials with a total of 92 participants used TUG to assess mobility, 42,45,49 Improvements were reported in both groups post-intervention. A meta-analysis done revealed no significant difference between the groups (smd = 0.33, 95% CI [-0.08, 0.75], I 2 = 0%, P=0.11) as shown by a forest plot in Fig. 5. A sensitivity analysis that excluded results from one study 42 deemed to be a high risk of bias revealed similar results (smd = 0.34, 95% CI [-0.12, 0.81], I 2 = 0%, P=0.15).

In 3 trials with 65 participants, the 10mw Test was used as the outcome measure, 42,45,51 A meta-analysis revealed Nintendo Wii, projector no significant difference between the groups (smd = -0.06, 95% CI [-0.60, 0.48],I 2 = 0% P=0.82). A forest plot is shown in Fig. 6. A sensitivity analysis which excluded results from. 42 (smd = -0.22, 95% CI [-1.00, 0.56],I 2 = 42% P=0.58) revealed similar results.

Balance

Four trials with 123 participants used BBS as the outcome measure for balance. 45,46,49,51 The VR group had greater improvements with a moderate overall effect size. (smd:0.46, 95% CI [-0.01, 0.93], I 2 =37%, P=0.06). However,

Quality of life and participation

Eight trials with 686 participants measured Quality Of Life using various outcome measures. 8,25,27,41,43À45,50 A meta-analysis showed no significant difference between groups with (smd = 0.04 95%, CI[-0.11, 0.19], I 2 = 0%, P=0.79) . A forest plot is shown in Fig. 8.

Activities of daily life

In nine trials 8,27,30,45,48,50,53À55 with 550 participants and several outcome measures. A meta-analysis revealed a significant difference between the groups but the overall effect size was low. However, the statistical heterogeneity was moderate. (smd = 0.25, 95% CI, [0.02 to 0.49], I 2 =43.0%, p=0.04). A forest plot is shown in Fig. 9.

Non-immersive virtual reality Vs CT(Long term Effects)

Upper limb function and activity

Two trials, studied the long-term effects of VR on upper limb function using FM as the outcome measure. A metaanalysis was not done due to high statistical heterogeneity in the data. However, in 27 with a population of 64 participants, after 15 weeks follow up, greater improvements were reported in the VR group but there was no significant difference between the treatment groups. The overall effect size was small. (VR:40.4 § 20.7, CT:34.5 § 19.5, smd=0.29, 95% CI[-0.20,0.78]). While in, [START_REF] Shin | Effects of virtual reality-based rehabilitation on distal upper extremity function and health-related quality of life: a single-blinded, randomized controlled trial[END_REF] with a population of 23 participants, after 1-month follow-up, VR In two studies [START_REF] Schuster-Amft | Effect of a four-week virtual reality-based training versus conventional therapy on upper limb motor function after stroke: a multicenter parallel group randomized trial[END_REF]8 with 153 participants, upper limb activity was measured by BBT. After 2 months 53 and 4 weeks 8 follow up assessment, there were no significant differences between the groups. (smd = -0.06 95% CI [-0.38, 0.26], I 2 = 0% P = 0.78).A forest plot is shown in Fig. 10.

Lower limb activity

Only one trial 45 with a total of 47 participants reported long-term effects on lower limb activity. Three months after intervention using TUG, greater improvements were reported in the VR group. However, there was no significant inter-group difference (NIVR: -23.52 §10.96, CT:-28.67 §18.73, smd = 0.34, 95% CI [-0.24, 0.92]).

Balance

In One trial 45 which used BBS as the outcome measure, a follow-up assessment conducted 3 months post-intervention with 137 participants reported no significant difference between the two intervention groups. (VR: 46.31 § 5.8, CT 45 §5.06 smd= 0.23, 95% CI [-0.34 to 0.81]).

Quality of life and participation

Four trials 8,27,41,45 with 406 participants, conducted a post intervention follow-up on the quality of life. Improvements were noted in both groups without significant intergroup difference (smd=0.04, 95% CI[-0.15 to 0.24], I 2 =0.0%, p= 0.67). A forest plot is shown in Fig. 11.

Activities of daily living

6 trials 8,27,30,41,45,[START_REF] Schuster-Amft | Effect of a four-week virtual reality-based training versus conventional therapy on upper limb motor function after stroke: a multicenter parallel group randomized trial[END_REF] with a total of 493 participants, reported a follow-up post-intervention. A meta-analysis of results revealed no significant difference between the groups. (smd, = 0.00 95% CI [-0.22, 0.23],I 2 =28.0%, p= 0.97). A forest plot is shown in Fig. 12.

Adverse events

Eight trials monitored adverse events. 8,23,27,29,30,45,[START_REF] Gandolfi | Virtual reality telerehabilitation for postural instability in Parkinson's disease: a multicenter, single-blind, randomized, controlled trial[END_REF][START_REF] Allen | An interactive videogame for arm and hand exercise in people with Parkinson's disease: a randomized controlled trial[END_REF] However, no serious adverse event related to the treatments was reported.

Discussion

This work investigated the effectiveness of IVR and NIVR compared to conventional therapy for stroke rehabilitation based on 22 included trials.

In general, the statistical analysis revealed that both VR interventions positively affected patients' functionality in a comparable way to CT.

As a comparison of NIVR to CT, this meta-analysis discovered positive improvement in treatments in favor of VR with small to medium effect sizes but with no significant difference between the different techniques. Higher values of effect sizes in favor of the NIVR indicated that patients had improvements in upper limb function and activity, mobility, balance, and ADL. However, their level of independence, which is the aim of rehabilitation strategies did not improve as their counterparts who received CT. The magnitude of this effect was comparable to that observed in previous systematic reviews. They concluded that using VR-based therapy systems enhanced upper limb function, quality of life, 18,20 mobility, and balance 21 in people with stroke but not significantly greater than CT. There is an indication that VR may alleviate upper limb motor impairments and encourage motor activities and societal participation among stroke survivors.

More improvements in VR could be due to several crucial factors. The first may be the ability to provide therapists with various training programs. The therapist may select different treatment modes and construct an individualized training program that adapts the intensity and difficulty level of the training to the patient's current motor status. [START_REF] Ballester | The visual amplification of goal-oriented movements counteracts acquired non-use in hemiparetic stroke patients[END_REF] The potential of VR to scale difficulty levels and give adequate rewards to users in the context of gaming and level progression is vital to the implementation of effective VR training systems. Secondly, VR systems incorporate task-specific workouts in addition to an appropriate level of exercise intensity and repetition, from which patients can benefit. The majority of the included trials in our review comprised graded training regimens to induce optimal neural plasticity and continuous active engagement, both of which are essential for successful motor recovery after a stroke. [START_REF] Brunner | Virtual reality training for upper extremity in subacute stroke (VIRTUES): a multicenter RCT[END_REF] A recent review found that custom-built VR systems had a more significant effect on the recovery of upper limb extremity function and activity than using CT. [START_REF] Maier | Effect of specific over nonspecific VR-based rehabilitation on Poststroke motor recovery: a systematic meta-analysis[END_REF] Customized VR systems are usually constructed based on a valid hypothesis to provide an effective rehabilitation regimen beneficial to patients. 19,[START_REF] Brunner | Virtual reality training for upper extremity in subacute stroke (VIRTUES): a multicenter RCT[END_REF][START_REF] Cameirao | The combined impact of virtual reality neurorehabilitation and its interfaces on upper extremity functional recovery in patients with chronic stroke[END_REF] Therefore, the ability to configure a VR therapy system with multiple training alternatives may be critical for upper-limb rehabilitation of motor deficits following stroke. Another reason could be the numerous types of sensory feedback like visual and aural present in VR therapeutic systems, which make the training more enjoyable. The use of virtual reality (VR) can assist in the creation of environments in which more repeated actions are done in a playful context, hence enhancing motivation and adherence to therapy. The majority of the selected studies have taken into account this essential factor, either through participation scales or questionnaires assessing the level of motivation.

By differentiating the effects obtained by immersive and non-immersive solutions, our evidence suggests that IVR may be more beneficial than NIVR for upper limb function and daily life activities. However, it is important to note that the results of IVR intervention are based on short-term effects and few studies with a limited number of participants. We observed that the effect size values of the results from studies that used IVR were higher than NIVR.

A possible reason behind this phenomenon may be the missing depth cues in the 2D environments that characterize NIVR applications. [START_REF] Durgin | Controlled interaction: strategies for using virtual reality to study perception[END_REF] Piggott et al. ( 2016) observed in their study that subjects using 2D VR systems such as cyber gloves tend to decrease their wrist extension. [START_REF] Piggott | Haptic neurorehabilitation and virtual reality for upper limb paralysis: a review[END_REF] This could be resolved by using a head-mounted display (HMD). On the contrary, IVR systems take into account depth cues, 17 which are responsible for accelerated cortical reorganization, [START_REF] You | Virtual reality-induced cortical reorganization and associated locomotor recovery in chronic stroke: an experimenter-blind randomized study[END_REF] thus allowing the central nervous system to control the position and orientation of body segments and adapt to the simulated environment.

The effect of different neurological characteristics on VR rehabilitation outcomes also needs examination. Some studies suggest that hemorrhagic stroke may result in more severe cognitive, motor, and functional impairment than ischemic stroke. [START_REF] Andersen | Hemorrhagic and ischemic strokes compared: stroke severity, mortality, and risk factors[END_REF][START_REF] Bhalla | Differences in outcome and predictors between ischemic and intracerebral hemorrhage: the South London Stroke Register[END_REF] Future investigations would benefit from a comparison of these stroke types to test the impact of both forms of VR.

The absence of adverse events related to the treatment suggests that NIVR can be considered a safe treatment. This data is consistent with results from a review by Laver et al., 18 who found little or no adverse events from VR treatment after stroke. This was because most studies took place in settings that applied extra safety measures such as supervision or walking harnesses in a laboratory or rehabilitation setting. It would also be important to note that IVR studies did not monitor occurrence of adverse events among participants, therefore it would important to examine the safety and psychological outcomes associated with the use of head mounted displays.

Limitations

The presented results should be interpreted considering some limitations. High heterogeneity in the data made it difficult to perform a meta-analysis of results for some of the outcomes of interest. For example, a meta-analysis on the beneficial effects of NIVR on upper limb function using FM was not feasible.

There was a high diversity between the VR training scenarios, which made comparing the results across the studies difficult. Therefore, we could not make a firm conclusion on the benefits of this technology compared to CT, though all studies reported post-intervention improvements.

The IVR studies were single-center designs characterized by high dropout rates mainly due to medical reasons and compliance issues 17 reported a high dropout rate in the CT group (22.5%) and (23%) in the IVR group. Our results should be validated on additional works based on larger populations.

Another limitation was the wide variety of outcome measures used in the included articles, which precluded the possibility to compare different evidence across studies, as trials used different versions of the same scale or different measuring units.

NIVR is already established and has many studies. On the contrary, the utilization of IVR systems for motor rehabilitation programs is still in the early stages. IVR is a relatively new technology and remains partially known, with a lot of the work limited to pain and phobias treatment. 13,[START_REF] Benham | Immersive virtual reality for the management of pain in community-dwelling older adults[END_REF] There are few RCTs on the effectiveness of immersive virtual reality systems in stroke rehabilitation. Furthermore, we did not find studies that examine the long-term benefits of IVR, and therefore more trials are needed to validate the intensity of efficacy.

Additionally, most of the included trials had small sample sizes, which resulted in low certainty in the effect measures and low statistical power. We recommend larger studies in the future, with power calculations pointing to more than 25 participants per group.

Conclusion

This study on the efficacy of virtual reality therapies applied to the rehabilitation of patients with stroke highlights the benefits of VR. However, evidence of the clinical effectiveness of the different forms of virtual reality (as either immersive or non-immersive) is scarce. Results from this review suggest that IVR therapies may be more effective than NIVR but not CT to improve upper limb activity, function, and daily life activities. The results of IVR intervention are based on short-term effects with small effect sizes. Therefore, there is no evidence that IVR treatment is long-lasting. NIVR provides the same benefits as CT for mobility, balance, quality of life, and daily life activities among patients with stroke. While the current literature evaluates VR as a viable alternative to conventional therapy in stroke rehabilitation, much attention is accorded to non-immersive solutions due to their wide use in clinical and research fields. By exploiting the increasing use and availability of IVR systems, additional controlled trials with larger sample sizes should be carried out in the future to reliably assess the long-term effects and promising benefits of this technology. Abstract. In a human-robot interaction system, the most important thing to consider is the safety of the user. This must be guaranteed in order to implement a reliable system. The main objective of this paper is to generate a safe motion scheme that takes into account the obstacles present in a virtual reality (VR) environment. The work is developed using the MoveIt software in ROS to control an industrial robot UR5. Thanks to this, we will be able to set up the planning group, which is realised by the UR5 robot with a 6-sided prop and the base of the manipulator, in order to plan feasible trajectories that it will be able to execute in the environment. The latter is based on the interior of a vehicle, containing a user (which would be the user in this case) for which the configuration will also be made to be taken into account in the system. To do this, we first investigated the software's capabilities and options for path planning, as well as the different ways to execute the movements. We also compared the different trajectory planning algorithms that the software is capable of using in order to determine which one is best suited for the task. Finally, we proposed different mobility schemes to be executed by the robot depending on the situation it is facing. The first one is used when the robot has to plan trajectories in a safe space, where the only obstacle to avoid is the user's workspace. The second one is used when the robot has to interact with the user, where a mannequin model represents the user's position as a function of time, which is the one to be avoided.

Trajectory planning in

Introduction

In human-robot interaction systems, knowing how to compute a path for the robot to follow, while taking into account the human position, is a crucial task to ensure the safety of the individuals around the robot. This is where path and trajectory planning plays its role in the field of robotics, where achieving realtime behaviour is one of the most challenging problems to solve. The result is arXiv:2202.11336v1 [cs.RO] 23 Feb 2022 a constant demand for research into more complex and efficient algorithms that allow robots to perform tasks at higher speeds, reducing the time they need to complete them, resulting in increased efficiency. But this also comes at a cost: to achieve higher speeds and shorter times, robot actuators must work under more demanding conditions that can shorten their overall life or even damage their structure. High operating speeds can also affect the accuracy and repeatability of manipulators. Therefore, it is important to generate well-defined trajectories that can be executed at high speeds without generating high accelerations (to avoid robot wear or end effector vibrations during stopping). Path planning is the generation of a geometrical path from an initial point to an end point and the calculation of the crossing points between them. Each point of the generated trajectory is supposed to be reached by the robot end effector through a specific movement. When the robot is supposed to interact with a human, its velocity and acceleration must be zero at the end of the trajectory. Another important element to take into account is the environment in which the task or the movement is going to be performed. This is what allows the system to identify the robot's environment and the colliding objects that might be present, thus determining the areas in which the robot must be constrained or limited to ensure the safety of the user.

The Lobbybot project is a project that allows interaction between a user and a cobot. These interactions allow for the creation of a touch-sensitive interface or intermitant contact interface (ICI). The scenario used allows the user to be inside a car with the possibility to interact with its environment by getting a sensory feedback of the different surfaces thanks to a 6 faces prop providing the different textures. Due to the immersion of the user via a VR headset, the system must ensure the safety of the user, as he cannot see the location of the robot. Therefore, it is necessary to implement trajectory planning techniques to be able to avoid unwanted interactions between the robot and the user. To do this, the system must take into account the obstacles present (environment or user). A virtual mannequin is modelled using data from the HTC Vive trackers which provide an estimate of the user's position, and will give the system a model to plan the movements. Thus, the goal of the LobbyBot project is to provide an immersive VR system that is safe for the user and gives them the ability to interact with the environment at different locations, providing a new level of interaction between VR environments and the real world.

State of the art

Intermittent contact interface

In the area of human-robot interaction and haptic perception, the ability to reproduce the sense of touch to appreciate different textures and motion sensations through the use of cobots has been addressed in [1], where a rotatable metaphorical accessory approach (ENTROPiA) has been proposed to provide an infinite surface haptic display, capable of providing different textures to render multiple infinite surfaces in VR (virtual reality). Studies in [2] [3] have focused on the perception of stiffness, friction, and shape of tangible objects in VR using a wearable 2-DoF (degrees of freedom) tactical device on a finger to alter the user's sense of touch. In [4,5], a 6-DoF cobot is used in a VR environment to simulate the interior of a car, where interaction between the robot and the user is expected just at specific, instantaneous points. This proposal is to use ICIs (Intermittent Contact Interfaces) [6] to minimise the amount of human-robot interactions to increase safety. In order to use the proposed implementations in this study in a real-time environment that involves human movement, it is important to ensure the safety of both the user and the robot to avoid potential collisions or accidents. This is where it is necessary to implement proper path and trajectory planning, in order to determine a feasible path to the desired goal, while avoiding interaction with the human until said goal is reached, generating a human-robot interaction just at the desired time.

Path Planning

Path planning refers to the calculation or generation of a geometric path, which connects an initial point to an end point, passing through intermediate viapoints. These trajectories are intended to be followed by the end effector of a robot in order to execute a desired task or motion. This geometric calculation is based on the kinematic properties of the robot as well as its geometry (included in its workspace). In the simplest case, path planning is performed within static and known environments. However, this problem can also be generated for robotic systems subjected to kinematic constraints in a dynamic and unknown environment.

Path planning can be done using a previously known map. This is called global planning. This method is commonly used to determine the possible paths to follow to reach the final position. It is used in the case of a known and static environment, where the position of the obstacles does not change. This operation can be performed offline, as it is based on previously known information. In the case of dynamic environments, it is necessary to perform local path planning, which relies on sensors or any other type of interface providing data to obtain updated information about the robot's environment. This planning can only be done in real time, as it depends on the dynamic evolution of the environment. Figure 1 presents the main differences between local and global path planning [7,8].

There have been multiple proposals on path planning algorithms over the years. In [9], one can find a review of the basics and workings of the most common algorithms most commonly found in the robotics literature. The main methods are the following:

-The Artificial Potential Fields (APF) approach [8] introduced by O. Khatib in 1985 and further developed by [10] [11]. -The Probabilistic Road-maps approach [7] consists in generating random nodes in the configuration space (C space ) in order to generate a grid (so called, the road-map). -The Cell Decomposition algorithms [12].

-The Rapidly Exploring Random Trees (RRTs) [13], introduced by S. LaValle in 2001 as an optimisation from the classical Random Trees algorithm.

Algorithm Comparison

In robotics, path planning is one of the most difficult tasks in real-time dynamic environments. Among the presented algorithms, APF and its variations offer a good adaptation to path planning in dynamically changing environments, where any obstacle entering the C space generates a new repulsive field that can be taken into account to generate a new path. But the local minima problem requires the use of alternative algorithms to overcome it. The case of PRM, it is well known for its ability to find a path without needing to explore the whole C space , but it is also a graph based algorithm, which requires the use of shortest path method like A * . It works well in static environments and can handle initial and final configuration changes, but if the objects in C space change position, the connections between the nodes must be redone. Some alternatives propose to keep the previously generated nodes and recheck whether they belong to C f ree or C obs , then rebuild the graph based on this information and find a new path. This is also the case for cell decomposition methods, where the graph search has to be reconstructed again. Nevertheless, these methods have proven to be viable options in real time, capable of adapting to a dynamic environment.

Finally, regarding the RRT and RRT* methods and their alternatives, they are known to be good path planning methods, with the limitations that the generated trees are related to the initial configuration and have high computational demands. The proposal of the different alternatives allows to obtain very optimal real-time path planners. The limitations of this type of algorithms are that they require a large memory capacity, as the entire tree must be stored at all times, and that they only work in bounded environments, with unbounded and long distance environments remaining a challenge.

Setup of the experimentation

In this section, we will present the tools used in the development of the project, such as the laboratory system, the software used, a description of the system environment as well as the laboratory setup.

System Architecture The architecture of MoveIt is based on two main nodes, the node move group and the node planning scene, which is part of the first one. The move group node is responsible for obtaining the parameters, configuration and individual components of the robot model being used, in order to provide the user with services and actions to use on the robot.

Within the planners available in the OMPL library there are:

-PRM methods (PRM [7], PRM* [14], LazyPRM [15], LazyPRM* [15] [14]), -RRT methods (RRT [13], RRT* [16], TRRT [17]), BiTRRT [18], LBTRRT [19], RRTConnect [20], -Expansive Spacial Trees (EST) methods (EST [21], BiEST [21]).

Collision detection Collision checking in MoveIt is configured within a planning scene using the CollisionWorld object. Collision checking in MoveIt is performed using the Flexible Collision Library (FCL) package -MoveIt's main collision checking library.

Kinematics MoveIt uses a plugin infrastructure, specifically designed to allow users to write their own inverse kinematics algorithms. Direct kinematics and Jacobian search are built into the RobotState class itself. The default inverse kinematics plugin for MoveIt is configured using the KDL numerical solver [22] based on Jacobians. This plugin is automatically configured by the MoveIt configuration wizard.

ROS-Industrial ROS-

Industrial is an open-source project that extends the advanced capabilities of ROS software to industrial hardware and applications. For this project, we used the ROS-Industrial-Universal-Robots metapackage [23], which provides and facilitates the main configuration files for the use of Universal Robots cobots in the ROS environment, providing the different descriptions of the robot, configuration files such as joint boundaries, UR kinematics, etc.. This package also facilitates the use of the robot in MoveIt, providing the setup for its use in simulation or in real implementations.

HTC Vive

The HTC Vive is a motion tracking system that allows users to be immersed in a VR system [24]. It consists of trackers, which can attach to any rigid object, and work with the VR headset. The tracker creates a wireless connection between the object and the headset and then allows the user to represent the objects movements in a virtual world.

Laboratory Setup

The laboratory setup consists of a UR5 robotic system and a car chair in a face-to-face configuration (Figure 2). The location and height of the robot was determined by [4] to be 75 cm above the floor. This position is optimal enough for the robot to reach all the interaction points that the system is interested in reaching. For the user, the VR headset and trackers are attached to the body (the humerus and palms), in order to obtain data and locate the user's location in the VR environment (Figure 3).

Selection of the optimal trajectory planning and its application

We present the setup associated with the choice of the optimal trajectory generator available within the MoveIt software and its application for the LobbyBot project.

MoveIt Setup

The installation of MoveIt consisted of configuring and defining the planning group, as well as making it compatible to work in Gazebo. The start-up phase was very important to analyse the behaviour of the different movement alternatives found in the MoveIt API. For this, it was important to configure the simulation environment in Gazebo so that we could test without compromising the real robot.

Planning group

The planning group is defined as the group of elements that make up the entire robotic system. These are the UR5 robot, the 6-faced prop and the robot support. These three elements are the ones that the trajectory planning algorithms must consider in order for them to avoid any collision state existing with one of these elements. The robot support was modelled to match the size of the real system that was optimally defined [4]. For the configuration of the plannig group, MoveIt has an integrated graphical interface to create all the configuration files related to the kinematics, controllers, Semantic Robot Description Format (SRDF) and other files for the usage of the robot in ROS. This interface is called MoveIt Setup Assistant. The MoveIt Setup Assistant creates all the mentioned files based on the robot description given to it, in this case the UR5 robot description files provided by [23] where taken and modified to include the robot support (included in the URDF definition of the robot) and also the mesh file for the prop.

User's Model

To model the user, a mannequin was defined in a URDF robot model. The main torso of the model is fixed, while the arms are structured as a serial robot with seven revolute joints, where the first three constitute the shoulder, the fourth joint represents the elbow, and the last three revolute joints represent the wrist of the arm. In the model, two small dots have been created in the humerus and palm links, which represent the location of the sensors in the user, as shown in Figure 4. Regarding the movement of the mannequin model, a kinematic model has been developed in parallel to this project in [25], where the connection between the sensor data and the model is defined. This will allow the system to recognise the user's movements and represent it in the simulation 4.

Motions

To generate collision-free trajectories, the different algorithms implemented in the MoveIt API have been analysed. All the tasks related to planning group movement are handled by the move group class. By specifying the planning group we want to consider, we are able to use all the different functions that the class offers for it, such as getting information about the current values of the joints, the target, configuring the planning algorithm we intend to use, and performing the planning and execution of the movements in the environment.

Types of movement

The move group class has the ability to perform path planning through different types of movements. These options can be chosen according to the nature of the task. For example, we can define a given pose in workspace or a desired joint value as the goal. Given the nature of the system, we will work with joint value goals, as we hope to achieve the different points in a specific configuration that provides a higher level of safety to the user (elbow up configuration for the UR5). Another important feature is the ability to specify whether one wants to achieve each of the requested objectives or not. As the implementation will receive constantly changing goals, the best implementation is to plan and move towards said goal by allowing the system to replan if the goal changes, meaning that we do not need to reach the initial goal. To do this, the move group class relies on the move group.execute(my plan) function to strictly reach the goal and on the move group.asyncExecute(my plan) function to execute the planned path with the possibility of re-planning during this execution.

In Figure 5, two trajectories are calculated from an initial configuration, to an intermediate goal, and then to a final goal. In this case, by using the function move group.execute(my plan), we ensure that the robot will completely execute each of the trajectories and achieve both goals. This is illustrated in Figure 5, where the speeds drop to zero as the robot comes to a stop. In the case of figure 6, we have calculated the same two trajectories as before, but using the function move group.asyncExecute(my plane), which allows replanning during the execution of the first plane. In this case, in figure 6(a), we can see the two plans one after the other, while in figure 6(b), we show the representation of the segment that was not executed from the first plan, because a replanning scenario was set up. In this case, the current positions of the first plan were taken as the initial positions for the second plan, resulting in Figure 6(c), showing the two plans that were executed. Algorithm Selection Another parameter to select was the planning algorithm that best suited the task. As mentioned earlier, MoveIt has several built-in path planning algorithms that can be used. In order to determine the best option, we went through all the available options and performed a planning task to a desired target configuration, measuring the time required for each algorithm and recording the data. We ran each of the 12 available planning algorithms five times through nine different paths. We then took the average time it took them to find a solution, to simplify the trajectory (only for the algorithms that had this feature) and calculated the total average time. Using this data, we were able to select the algorithms that performed best with the shortest planning times (Figure 7). After performing these calculations, given the large difference in planning times for some of the algorithms, we select the six best algorithms to compare them on 12 trajectories (Figure 8). Another analysis that allowed us to select the algorithm which behaved the best for the implementation, was to perform an analysis on the generated trajectories with each one of the algorithms for a fixed task. Based on the six best algorithms from the previous analysis as a starting constraint, we computed the average execution time and via-points number for a set of trajectories. The BiTRRT algorithm wins for both comparisons. This analysis was performed for the same trajectories as in the previous graphs for a total of ten iterations for each algorithm, but instead of considering only the computation time (Figure 9), we also took into account the amount of via-points generated (Figure 10). Between each via-point, a linear interpolation is performed in the joint space. For theses trajectories, the mannequin was placed in its seat so that it was avoided in the calculation, in order to test each algorithm's ability to plan around it. This also allowed us to see how consistent the behaviour of each algorithm was.

Unity's Virtual Environment In parallel to the development of the project, and to better explain the developed implementation, it is important to specify how it will fit into the project. The system will receive a desired goal configuration which will be the q goal for the planning algorithm, from the current q init configuration. This goal selection is done in Unity by a Point selection algorithm which determines the interaction point the user intends to reach [26] (Figure 11).

Planning Scene

For the definition of the planning environment and scene, MoveIt has instances that allow the manipulation and monitoring of the scene to keep it up to date. These instances are :

-PlanningSceneInterface: Is responsible for adding and removing objects in the scene. -PlanningSceneMonitor: Takes care of keeping track of the planning scene in order to keep it updated.

The last of these instances is absolutely necessary to perform the collision check, as we need to ensure that the scene being processed is the last one available.

Mobility Schemes

Based on the Unity information, two different motion or mobility schemes and scenarios has been proposed depending on the nature of the task we want to achieve at the moment. One for which no interaction with the user is required, and another one for when it is. These two scenarios have their own environment to consider, presenting in general two different behaviours.

Movement outside user's workspace

The first scenario is based on [26] where a distinction for velocity zones is made and where a plane divides the environment (space with the user and space where the user cannot go).Based on the same idea, we represented the effective working space of the mannequin as a sphere surrounding the model (Figure 12). The mobility scheme consists of alternating between different "Safe positions". These positions are so called because they are points out of reach of the user, which means that there is no need to constrain the robot's speeds. Therefore, the movement from one point to another just has to take into account the defined sphere, as we do not want to "collide" with it. Following this idea, we have performed a calculation of all existing trajectories between the different "Safe positions" and stored them in a data file. This allows us to perform offline path planning, and then at runtime, depending on the initial and desired goal, we can access the pre-calculated paths to execute them directly, eliminating the computational time that would otherwise be required by performing online planning. The algorithm 1 allows the storage of the trajectory.

Then, the second part of the device consists of loading the pre-registered data and being able to use them on demand (Algorithm 2). We wait until we know the position we want to reach. Unlike [26], we have used a spherical surface here to divide the two areas of the space instead of a plane, as this allows greater flexibility for the planning group to consider more configurations when calculating the path between points. It also allows for more feasible trajectories for the robot. for j < 0 ; j < nop ; j + + do 5:

if i = j then 6:

Ø ← Plan and Exec to(points[i]) {Move to initial point of the plan} 7:

plan end for 20: end for workspace, which means that the movements have to take into account the user's model in order to avoid any collision with him. We also have to take into account that the speed of these movements must be limited, in order to ensure safety.

Unlike the first scheme, in this case the environment consists of moving obstacles, which requires constant updating of the scene and constant tracking of the objects in it (Figure 13). For this reason, we used the images of the mannequin model to obtain its current positions and orientations in order to track their movement and link it to the objects created in the scene.

We also need to be able to determine whether a computed plan will collide or not, which requires taking several aspects into account. First, based on the calculated path to the desired goal, we check whether the path remains valid during the execution of the plan. To do so, we check for all calculated via-points of the path, whether the respective configurations are currently colliding with any other object present in the scene. If there are no collisions, we continue the execution. In the case of a collision present in any of the remaining states of the path, we instruct the robot to stop the execution of the computed path and replan it based on the updated scene information.

To test our framework, we performed an initial trajectory planning. Then, during the execution, we created an obstacle. Then, by checking the validity of the trajectory, we are able to detect that an object is in collision with the

Algorithm 2 Trajectory upload and execution

Require: A desired frame to go to des f rame. A home pose home. Number of elements noe. Initial positions init pos id. Goal positions goal pos id. Planned trajectories plan. A counter i. 1: for i < 0 ; i < noe ; i + + do 2:

{Extract the data from the file} 3: end for 23:

start name[i] ← init pos id[i] 4: f inal name[i] ← init
end if 24: end while planned trajectory. We then instruct the robot to stop the current execution and replan towards the same goal, taking into account the updated planning scene. This work is intended to be extrapolated to work according to the size of the mannequin. Thus, we can take into account the user moving in the environment as an obstacle to be avoided (Figure 14).

Conclusions

In this paper, we have presented motion generation algorithms that can be used by a cobot to create an intermittent contacts interface. A framework was presented including a UR5 cobot, ROS nodes, HTC Vive sensors and a car chair. Taking into account the objects present in the environment, a comparison of trajectory planning algorithms is presented. The selected algorithm is then used in two examples. An experimental validation is in progress and will be presented in the final version of the paper. The aim of virtual reality is to immerse a human being in a virtual environment using all his senses. In most collaborative systems, the main senses are sight, then hearing, and finally, touch [1]. The sense of vision can be rendered using large screens that occupy the user's entire field of vision or using a head mounted display (HMD). In the latter case, the user's vision becomes completely disconnected from the real world and all his movements can become dangerous. In some cases, user may lose his spatial landmarks and have the feeling of falling on the ground. Sound immersion further increases this immersion and separation from the real world. By using immersion HMD and headphone, the user can free himself from his environment.

Haptic interfaces, such as Virtuose 6DOF [2], are used in product design by engineers [3]. In Refs. [4,5], a five-fingered haptic interface robot with a 6 degree of freedom (DOF) arm and a 15 DOF hand was used to provide multipoint contact between the user and a virtual environment through force and tactile feeling to the fingertips of the human hand. These interfaces are safe and well mastered, but if the user can apply force/torque, he cannot really feel the textures and appreciate the quality of the materials. Among the main shortcomings of these interfaces are limited workspace, low stiffness, and high cost.

New haptic interfaces using an industrial robot or a cobot (robots specially designed to work in human-robot environments) can be used as haptic interfaces with intermittent contacts [6]. For the application envisaged in this document, the cobot carries several texture specimens on its end-effector to allow contact between a user's finger and the robot. They are called intermittent contact interfaces (ICIs) [7].

When the user uses HMD vision interfaces and has to perform haptic evaluations, he no longer sees the real scene, but only a virtual world. His physical reference points quickly disappear except for objects he touches such as his seat and the floor.

When users reach to grasp objects, they look at the target first, then bring the hand to the center of gaze to grasp the object.

Eye-hand coordination is a fundamental behavior that humans use to interact with the world [8-10]. The head movement facilitates subsequent gaze shifts toward the future position of the hand to guide object manipulations, thus leading to a strong correlation between head and hand movement parameters [11-13].

Through the user's gaze and hand movements, as well as the position of areas to be studied, it is possible to predict the tasks that the user will perform. The purpose of this study is to ensure that the robot end-effector will be available for intermittent contact in complete safety when the human hand is close to the surface to touch.

The outline of this article is as follows. First, we present the context of the study and the material used. Then, a human-robot interaction framework is introduced with its hardware, the virtual environment, and its data flow. Next, we present two velocity profiles to ensure user safety and improve the performance of our system by introducing safe-points. Four strategies are then introduced to predict the intention of the user by taking into account the movement of his hand and his gaze, and five criteria are introduced to characterize the performance of these strategies. This work concludes with an experiment and analysis based on seven trajectories recorded from three users. From this analysis, several Pareto fronts are calculated.

Description of the Context

The context of the study is the evaluation of the perceived quality of a virtual car interior during the first design phases. In a given scenario, the user sits in the real world for a visual virtual reality experience inside the car. The user wears a HMD and cannot see the robot, which explains the safety problem (Fig. 1). While the user is trying to interact with the virtual object of the environment, the robot must come and position a sample of the material associated with the local surface to provide a tactical sense of touching the object [14,15]. A motion capture system based on HTC vive trackers is used to know the position of the body and especially the hand used for interaction as well as the position of the chair and the robot [16] (Fig. 2). Currently, the prop can carry six different materials. The robot is fixed on a 0.8 m high table and the user sits on a seat 0.6 m above the floor. The placement of the robot in the scene has been chosen to be able to reach all the places where the user's hand will want to have haptic interaction with the robot's probe [17].

A virtual model under the UNITY® software represents the fixed objects in the environment, as well as the moving objects, which are the robot thanks to the encoders of the motors and the user thanks to HTC trackers located on the hands and on its seat. The industry partner provided the virtual model of the car design (Fig. 3).

An industrial robot can perform powerful and fast movements that can be dangerous for the humans around it. Involuntary contact between the robot and humans is a threat. This is particularly important in a virtual reality context where humans equipped with an HMD will not be able to anticipate the robot's movements. Today, more than ever, humans work closely with robots. In the case of ICI, contact is inevitable between humans and robots. Cobots are best suited to such a scenario, but in terms of human safety, accident prevention can always be improved [18]. These robots are designed to work at limited speeds during potential contacts. Moreover, it must be ensured that the desired contact with the robot during interaction will not result in a necessary restart of the robot after a safety stop [19].

Human Intention Prediction in Virtual Reality Environment

3.1 Detection of the Target of Human Motion. Robots need to anticipate human's future actions and act accordingly while performing collaborative tasks. In most human-robot collaboration systems, the motion of robots is based on some predefined programs, which are task-based. However, most tasks are highly complex and it is difficult to redefine a complete set of instructions for such situations. In such tasks, the roles of the robot should be changed from purely automated machines to autonomous companions. Previous works relied on supervised learning methods to build models of human motion, which relied on understanding the environment, offline training, or manual labeling, adaptation to new people, and motion styles.

An expectation-maximization (E-M) algorithm and a neural network to infer human intentions in a three-dimensional (3D) space were used in Ref. [20]. They modeled a function with intentions as parameters and developed a neural network to learn human arm dynamics. In Ref. [21], time series analysis for the motion of the human arm based on demonstrations of human arm reaching motion, which synthesized anticipatory knowledge of human motions and subsequent action steps to predict, was used. A combination of a two-layer framework of Gaussian mixture models and unsupervised learning to predict a remainder of the trajectory from a prior observed human arm motion in reaching tasks was used in Ref. [22].

In Ref. [23], a Markov decision process to anticipate a belief about possible future human actions was used by modeling the human's and robot's behavior and then constructed a graph to represent the human motion and interaction with objects.

Human intention is mainly expressed through the behavior of humans and the objects they interact with. Most of the current research on human intention prediction just focuses on action classification, in which the human action is classified into several categories, such as running, walking, jumping [24] which is inadequate for accurate inference of human intention in human-robot collaboration.

We propose an human-robot interaction framework that combines hand motion with gaze direction to build models on the fly which predict human intention in virtual reality and move the robot to the required position in a virtual space without offline training.

Proposed

Model. The aim of the work is for the human to make contact with different parts of the car, in a design phase where only a virtual model exists, to be able to assess the quality of the materials. The areas to be explored are limited, driver's door, passenger seat, dashboard, and touch pad. Depending on where the human wants to touch, the robot must position itself so that the human can touch the appropriate material placed on the probe. The probe has a certain surface area, so a limited number of regions of interest (ROI) in the car have been defined that the robot will have to reach to allow contact with the human. The set of 18 ROI considered is described in Sec. 3.3. The objective is therefore to determine as soon as possible the ROI that the user wants to reach and even more so that the robot's probe is positioned as soon as possible on this ROI. If the probe arrives before the human, the human will be able to make contact without being aware that he is in a virtual world, otherwise the waiting time before making contact should be as short as possible.

The major elements involved in our approach are summarized in Fig. 4. Measurements of the pose of the hand and the gaze direction via the orientation of the HMD are used to select ROI where the human hand will touch the prop of the robot.

It should be noted that as the objective is that the robot arrives at the target as soon as possible, several strategies are possible and can be combined:

• Detect the target at the earliest.

• Move the robot as soon as possible in the right direction even if the final target is not yet known. • Move the robot as quickly as possible.

As we are in a cobotic context with a human locked in a virtual world that does not see the robot (the robot can also be visualized in the virtual mode but the immersion will be less), safety is a priority. A description of the methods implemented to have a fast speed of movement of the robot and ensure safety will be discussed in Sec. 4.2.

Scene Information.

From the model of the car in UNITY virtual reality software, we defined the ROI the user is to interact with. Each ROI is represented as a capsule placed at the center of the surface. For each surface, the desired orientation of the probe is defined. We have defined 18 ROI to be studied inside the car (Fig. 5). They are located as follows:

• Four capsules on the door.

• Four capsules on the chair.

• Four capsules on the dash board.

• One capsule on steering wheel.

• One capsule on touch pad.

• Three capsules on glove compartment.

• One capsule on speedometer. ROS [25] is the middle ware that communicates with the robot and UNITY. Based on this information, pre-computed trajectories are selected so that the robot reaches the desired positions while knowing the current states of all objects in the scene. Once the trajectory is selected, we communicate with the UR5 robot using the ur modern driver. Thus, we can move the UR5 robot with the robot operating system (ROS) control, and send as output the current states of the robot's joints for visualization in UNITY.

4 Safe and Fast Motion of the Robot 4.1 Cobot Motion and User Avoidance. The robot will navigate between a finite number of points which are our ROI. However, the movements must ensure that collisions with the human are avoided. To do this, we will generate offline robot movements that ensure that no part of the robot enters an area encompassing the human at rest in the driver's seat. The area to be avoided is composed of a sphere and is illustrated in Fig. 7. The dimension of the sphere covers the human head and torso and part of the arm but the hands can be outside since they must be able to reach the ROI. For this, we need to construct 18 × 17 offline trajectories that we will assemble in line according to the ROI detected to accomplish the task. These trajectories being close to the human, they are realized with a maximum speed of 0.25 m/s to ensure the use of the UR5 cobot according to the ISO standard for human-robot collaboration [26]. This condition guarantees that a possible collision with the human will not hurt him.

Definition of Velocity

Zones. The robot must be moved closer to the target point to prepare for the interaction. The movement must be fast so that the robot has arrived before the human and thus avoid unpleasant waiting but the maximum speed of the robot must be limited for safety reasons. Figure 8 shows the scene of the virtual reality (VR) environment, and it consists of car interior and user model.

Based on this, we distinguish three velocity zones:

• The human workspace (HW), defined as two spheres whose radius is the size of the arm centered on the shoulders of the mannequin. This workspace will evolve according to the movements of the human. We could also consider a constant Transactions of the ASME space if we limit the realistic movements of the torso. This space is represented by blue circles in Fig. 9. • The inside of the car (IC): this space delimits the area where we know the human must move. Even if the Unity model is complex, this zone can be approximated by a larger simple region that includes the real interior of the car. The gray rectangle, in general, represents the entire Unity model and we define a plane, depicted by the dark line in Fig. 9, that separates the region that can be reached by the user. • The free space (FS) cannot contain points that are in the HW.

We can have a certain safety margin to define this zone. In our example, this zone is simply limited by a plane represented by a dark line in Fig. 9.

The limit on the robot velocity is chosen according to the space:

• When the robot moves in FS, it can do so at maximum speed V m (all parts of the robot are in FS). • When the robot moves outside of FS, it must move at reduced speed V r .

The speeds are chosen such as V m ≥ V r ≥ 0. The different spaces are shown in Fig. 9. The blue hollow circle is the robot workspace, and two blue-filled circles are the workspace of the user's hands. The gray rectangle is the complete interior model of the car and the dark line is the plane that we use to differentiate the reachable and unreachable parts of the car by the user.

Velocity Profiles Based on Zones.

To ensure safety and also have better response time, we defined two velocity profiles of V r = 0.25 m/s and V m = 4 m/s based on the zones defined above. This idea imposed that we move the robot in the FS. To illustrate the idea, we devise the scene as shown in Fig. 10. We define four points:

• Two points A ′ and B ′ are on the plane boundary, these are in the FS, and fast motion between these points can be produced.

For the application studied in the paper, the capsule denoted that 20-25 are in the FS. • Two points B and A are inside the plane boundary, one point on the dashboard and another on the passenger's seat. These points play the same role as the ROI 1-18.

We analyze two different scenarios based on different velocity combinations.

• The shortest way: Knowing the target point, the robot moves towards it and adapts its speed according to the spaces it crosses. In the studied example, it goes directly from A to B with a velocity of less than 0.25 m/s. • Safe-points: We use safe-points to keep the robot's speed high.

In the studied example, the points A ′ and B ′ belongs on the plane that limits FS. The robot goes from A to A ′ with a maximal velocity less than 0.25 m/s, then from A ′ to B ′ with a maximal velocity less than 4 m/s, B ′ to B with a maximal velocity less than 0.25 m/s.

The movement of the robot inside the car should be performed with reduced speeds for safety reasons. In some cases, when the desired point is far away from the user space, it takes longer to reach it due to the low speed. For such situation, we use the via. points on plane boundary in FS, called safe-points, between which the robot can move at high speed. The path is longer but its execution can be faster.

New trajectories are calculated offline to connect the ROIs and the safe-points with the two motion speeds.

Fig. 9 The two spheres described the human workspace when seated. The square part shows the car model. The transparent sphere is the robot's working area. The dark line delimits an area where the speed of the robot can be higher because there is no risk of collision with the human. The interest of moving the robot in the FS is also to increase the safety of the operator by moving the robot away from the human. To quantify this notion of safety, we will define an average distance between the robot's end-effector and the sphere encompassing the human's torso shown in Fig. 7. The higher this distance, the safer the human/robot interaction will be. This distance is calculated in an approximate way from the points of passage of the robot (PP 1 , …, PP 18 and SP 20 , …, SP 24 ) by making the hypothesis of a straight line displacement at constant speed between the points (defined as the distance between the points divided by the duration of the displacement).

d s = N i=1 ( p i -C -R) N (1)
where the robot motion is sampled in N instant, p i is the coordinate vector of the end-effector of the robot for sample i, C is the coordinate vector of the center of the sphere shown in Fig. 7, and R is its radius.

Considering the user safety and robot velocity it is of interest to pass through points on the plane boundary in FS, when we have long robot trajectories to make. In the next part of the study, we will show that this can also be useful when a movement is initiated by the human by hand and gaze but without knowing yet where the human will stop. Placing the robot on one of the safe-points, SP 20 , SP 21 , SP 22 , SP 23 , and SP 24 , will allow the robot to get closer to the goal more quickly.

Comparison of Motion With or Without Safe-Points.

An example is illustrated in Fig. 11 where A = P 17 , B = P 5 , A ′ = SP 24 , and B ′ = SP 21 . The points A ′ and B ′ are positioned on the plane that divides the two different velocity zones.

We compared the time taken by the robot to move between two points, taking into account the presence of safe-points and without them.

The results in Fig. 11 show the position of points in X-coordinate with respect to time. The Fig. 11 is a simple representation to show the point and at what time the robot reaches that point. From the recorded trajectories of the robot to reach the points, the X-coordinates of the robot are plotted at the beginning and end of the trajectory (and connected by a straight line) against time in Fig. 11. It proves that by passing through safe-points A ′ and B ′ , the robot takes less time than going directly.

By traveling through safe-points, the robot starts from point A and moves through A ′ , B ′ at a higher velocity and then to the final point B. The total time taken to reach the final point was 0.41 s. For the motion inside the car, the robot arrives at the final point after 0.45 s.

Proposed Strategies to Predict Human Intention

We will study and compare four strategies that integrate the position of the hand, the direction of the gaze, and the use of safe-point to efficiently move the robot to one of the ROI that the human wants to reach.

Strategy A.

As the user's objective is to touch with his hand the ROI, the first and simplest strategy proposed is to consider that the point to be reached is the closest to the hand position. The strategy is presented in Fig. 12, in the example the selected point is P 2 . The main advantage is the simplicity in the approach. For the search of the nearest point, a nearest neighbor search algorithm is used to find the nearest point to the hand. An implementation of this algorithm for VR environment is done in Ref. [27]. The strategy is summarized in Algorithm 1. Using hand pose as a query point q, return nearest point from the k-d tree.

4:

return P 5: end function detected, which will allow the robot to start its movement before the desired end point is detected.

Strategy B.

Head gaze direction is introduced to limit the detection of points to only what the user can see. The detection of the interest point is only possible if the point is in the field of view. The strategy is presented in Fig. 13. The pose of the hand is used to select two points, the closest to the hand, in the example, P 1 and P 2 . From these two points, the closest to the gaze direction is selected, by comparison of the angle between the line connecting the point and the line of view. In the example, the closest point is P 2 . The strategy is summarized in Algorithm 2. Using hand pose as a query point q, return nearest two points from the k-d tree.

4:

Find the gaze direction as a unit vector from the central point of the eyes and draw a ray in the gaze direction.

5:

Calculate the distance l 1 of each point in n p from the ray. 6:

Find the angle λ i for each point such that λ i = l i /L i , where L i is the distance from the HMD to the projection of the point on the line (shown in Fig. 13).

7:

The point with min(λ i ) is the closest point P.

8:

return P 9: end function

The aim of this approach is to try to find the point of interest with a little anticipation compared to the previous method, by being able to choose a point that may be a little further from the hand but directed according to the direction of the gaze.

Strategy C.

If the hand is far from the point of interest, it is probably on the way but still far from the goal, so it may be appropriate to move the robot to a safe-point to prepare for a higher speed movement. This strategy is an extension to strategy B, but with added extra safe-points. This strategy is designed such that the robot will always go to the safe-point if the distance between the hand and closest point is above a threshold, here 0.2 m. The strategy is presented in Fig. 14. In the example, the point P 2 , the closest to the view line among the two closest to the hand, is at more than 0.3 m from the hand, thus the robot will go to the safe-point which is the closest to P 2 among SP 20 , SP 21 , SP 22 , SP 23 , and SP 24 . The strategy is summarized in Algorithm 3. The main characteristics of this strategy are as follows:

• The difference between this approach and strategy B can only be seen for long displacements (of more than 0.6 m between the points) for which the hand displacement can be quite far from the points P i , i = 1, …,18. • There is a risk that the robot will move to a safe-point in an inefficient way with a longer path that will not allow the robot to arrive faster • The results obtained can vary with the choice of the threshold T h .

Strategy D.

All the strategies presented so far are based on a selection or pre-selection of the point of interest based on the hand position. Here the approach is different and the selection is based on the direction of the gaze which can greatly anticipate the movement of the hand. As in strategy C, safe-points will be used if the point of interest is more than T h = 0.3 m away from the hand. The strategy is presented in Fig. 15. In the example, the point P 2 , the closest to the view line among the points in the frustum of the HMD. As the point P 2 is at more than 0.3 m from the hand, the robot will go to the safepoint which is the closest to P 2 among SP 20 , SP 21 , SP 22 , SP 23 , and SP 24 . The strategy is summarized in Algorithm 4.

Algorithm 4

Strategy D: Predictions with head gaze and safe-points n i = points in the view frustum of HMD.

Input: Hand position P h ∈ R 3 , HMD position P s ∈ R 3 , head orientation O s ∈ R 4 ,

3:

Find the gaze direction as a unit vector from the central point of the eyes and draw a ray in the gaze direction.

4:

Calculate the distance of each point in n i from the ray. 5:

Find the angle λ i for each point such that λ i = l i /L i where l i represents the distance between a point and its projection on the line as calculated in previous step and L i the distance from the HMD to the projection of the point on the line (shown in Fig. 15).

6:

The point with min(λ i ) is the closest point P. This strategy is based on the assumption that the task will be carried out with coordination of gaze and movement. In general, it is reasonable to think that the gaze anticipates the movement. In the context of the study, where the human is enclosed in a virtual world, it is likely that he will not be disturbed by external elements and that he will remain focused with his gaze directed towards the point of interest. The context should therefore be favorable to this approach.

Strategy to Move the Robot.

When the target is defined, the robot must be moved. For this, a series of trajectories that avoid obstacles have been defined (see Sec. 4.1) between point of interest and/or safe-point. While the robot is moving, a new point of interest can be defined. One could stop the robot's movement and recalculate an obstacle-free trajectory online. However, in order to avoid wasting time in this calculation, while predefined trajectories have generally short execution times, so the robot is let to perform its movement. And it will take into account the new target at the end of its movement.

Definition of the Criterion to Compare the Strategies.

The main research question was to find a selection strategy which maximizes user safety while minimizing robot response time. Four strategies explained in the previous section were tested against each criterion. The strategy selected needs to minimize robot time to reach a desired target pose while ensuring maximum user safety.

To evaluate the strategies, the following criteria were considered for strategy.

(1) Efficacy:

• Q 1 : If the strategy detects the final point or not. A value of 1 or 0 was assigned if final end point was detected or not. (2) Time for detection:

• Q 2 : Time taken by the strategy to detect the desired/final point the user want to reach. • Q 2norm : In order to be able to compare the results for several strategies, we defined a normalized criterion. It a ratio of each value of Q 2 for a trajectory divide by the minimum value of Q 2 for this trajectory and the four strategies analyzed. [Q 2 /min (Q 2 )]. For the best strategy with respect to this criterion, Q 2norm = 1. (3) Time for robot:

• Q 3 : Time taken by the robot to reach the final point (to move from start to desired point including all via points). It is the sum of the duration of all the precomputed trajectories according to the strategy of motion of the robot according to Sec. 5.5 and the time that the robot waited to have new point where to go. • Q 3norm : As for the criterion Q 2 , we defined a normalized criterion to be able to compare the strategy for several trajectories. It is a ratio of each value of Q 3 for a trajectory divide by the minimum value of Q 3 for this trajectory and the four strategies analyzed. [Q 3 /min (Q 3 )]. For the best strategy with respect to this criterion, Q 3norm = 1. (4) Robot distance:

• Q 4 : The distance traveled by the robot from start to end point for all via points the robot travels through. • Q 4norm : The ratio of distance traveled by the robot (from start to end point for all via points the robot travels through) with distance between start and end point. (5) User distance:

• Q 5 : The mean distance between the sphere centered on the driver's seat with a radius of 0.5m and all points on the trajectory. This distance is evaluated via Eq. ( 1) and characterizes the safety of the user. The further the robot is from this sphere, the safer it is.

6 Experiments and Analysis of the Results 6.1 Experimental Setup. We used the hand motion sensor as a proximity sensor. The objective is to find the closest ROI with respect to the hand. This is an optimization problem of finding a point in a closed set that is closest to a given point. Using the head mounted display, we find the points in the user view and if the gaze is directed towards a point P i . We classify the distance of the points to the direction of the gaze as a function of l 1 /L 1 . The user will direct his hand towards a capsule (discrete set of N points). The goal is to detect as soon as possible which point to be reached by the human and to move the robot to that point:

(1) If the direction is known, the robot can be moved to intermediate points to facilitate the task. (2) Safe-points SP i (i = 20 : 24) are defined on the boundary plane, which is located outside the car's interior space. Between these points therobot can move quickly. (3) At each moment, from the sensor data, we define the target point among the set of N points P i (i = 1 : 18).

Seven trajectories were considered in the experimental design: two consisted of long distance trajectories (from points 2 to 11 and 5 to 18), three medium distances (from points 5 to 11, 5 to 15, and 12 to 15), and two short distances (from points 3 to 4 and 17 to 16). The seven trajectories have been done by three different participants.

041012-8 / Vol. 14, AUGUST 2022

Transactions of the ASME

The participant was seated in the car seat 0.6 m above the ground and at a position of 0.9 m in y and -0.1 m in x from the robot base frame. The sphere used for the obstacle avoidance of the user is centered at this reference point. An HTC vive HMD was worn by the user and vive sensors were attached to the user's dominant hand as shown in Fig. 2. Then for each trajectory, the user was instructed to move his hand from a start point to a defined end point.

For each trajectory performed by the user, data were recorded. It is composed of position of the hand tracker, the head position, and orientation. The user can do the task at the speed he wants. During this experiment, the robot was not moved to ensure the security of the user. As the objective is to compare the strategy with exactly the same data as input, thus there is no reason to move the robot with one chosen strategy. The data recorded were used to perform the analysis in parallel with the four strategies in order to compare them on the same data set.

6.2 Analysis of One Experiment. The four strategies are described and illustrated on one example, a trajectory done by one subject for moving his hand from P 2 to P 11 was recorded. A visual trail of the user hand is shown in Fig. 16. This recorded motion was used to analyze the four proposed strategies.

Detection of Points of Interest.

Figure 17 shows the sequence of points that are detected for the four studied strategies. It can be visualized that different strategies have different intermediate points selected except for strategies A and B that produced the same sequence of points detected.

Based on the results from the different strategies, it can be observed that strategies A and B select intermediate points which are inside the car while strategies C and D select the safe-points as some of the intermediate points. For this example, all the strategies allow to find the desired final point P 11 . However, strategy D succeeds to detect this point earlier than strategies A, B, C that detect the final desired point at the same time (as it can be seen in Table 1).

The obtained sequence of points of interest is now detailed:

• Strategy A: P 2 , P 12 , P 6 , P 8 , P 9 , P 10 , P 11 .

The points are selected based on the least distance to the hand, and the points selected can be easily explained by the hand trail described in Fig. 16.

• Strategy B: P 2 , P 12 , P 6 , P 8 , P 9 , P 10 , P 11 .

The selection of this strategy is similar to strategy A for this example because all points were all the time the point closest to the hand is also closest to the direction of gaze. The set of points detected is the same as for the strategy A. The robot motion will be similar and analyzed simultaneously.

• Strategy C: P 2 , SP 20 , SP 21 , SP 23 , P 9 , P 10 , P 11 .

For strategy C, a threshold is introduced around the detected points to choose whether the robot should go to the point or to a safe-point. It is observed in Fig. 16 that the hand passes at a distance > 0.3 m from the points P 12 , P 6 , P 8 . Consequently, the selected points will be the associated safe-points SP 20 , SP 21 , and SP 23 . Then points P 9 , P 10 , P 11 were detected and found to be within the required threshold of the distance from the hand.

• Strategy D: P 2 , P 3 , SP 20 , SP 21 , SP 22 , SP 23 , SP 24 , P 10 , SP 24 , P 11 . Strategy D uses the direction of gaze as the primary criterion and it is therefore more difficult to predict the sequence of points detected based on Fig. 16. In this strategy, P 3 is selected, it is done based on the user gaze and since the hand moved not far from this point P 3 is selected. For the following points selected by the gaze, the hand is farther from the points so the associated safe-points were selected. Then, the gaze is directed toward the point P 10 ; since the distance from the hand was below a threshold T h , the point P 10 was selected. Then the gaze is probably oriented to point P 11 ; since its distance from the hand is above the threshold, the robot has to return to the safe-point P 24 and finally when the hand is close to the final point P 11 , the point is detected. This happens at a moment when the point P 11 is not yet the closest point to the hand and is therefore not yet detected by strategy A. The points detected in this example show that Journal of Mechanisms and Robotics AUGUST 2022, Vol. 14 / 041012-9

Q 2 Q 2norm Q 3 Q 3norm Q 4 Q 4norm Q 5 St. A 1
the gaze does not go directly to the goal but sweeps along the path accompanying the hand. The results also show a certain sensitivity of the point sequence to the value chosen for the threshold.

6.2.2

The Robot Motion for the Four Strategies. For the three different selections of points (strategies A, B, strategy C, strategy D), the robot motion and the safe distance are now commented. Figures 18-20 illustrate, starting from the sequence of points detected represented as function of time, the corresponding robot motion which is represented as function of time. On the same figure, the distance between the end-effector of the robot and the sphere encompassing the human is shown as function of time and expressed in meter. This representation shows only the instants corresponding to the start and end points. Between these points straight dotted lines are drawn. This curve is directly calculated based on the robot motion and will be used for the evaluation criterion in Table 1. As for the robot motion, the exact value are calculated only at initial and final points and interpolated by straight line.

The sequence of progression of robot motion with time from start to end point is described below: 6.2.3 Strategies A and B:. The progression of the robot motion is indicated by the dark line as shown in Fig. 18. Starting from point P 2 , the robot waits for a new point. When P 12 is detected, the robot starts moving and before it arrives, P 6 is detected. However, the robot has to continue and complete the motion so then arriving at P 12 the robot now detects P 11 as new desired point, so the robot moves directly to P 11 . The robot avoids all the points that are detected during the motion towards P 12 . This pattern continues until the robot reaches the last point detected. 20. The robot starts from P 2 and when P 3 is detected, it moves to point P 3 . When P 20 is detected, the robot is still in motion to P 3 , so it ignores the point. Furthermore, SP 21 , SP 22 , SP 23 , and SP 24 are detected, but when the robot reaches P 3 , the prediction system still predicts SP 24 as it moves to SP 24 without waiting. Again during the motion, P 10 and SP 24 are detected, but before arriving, P 11 is detected, so on arrival at SP 24 , it does not wait but continues to P 11 , where it finally stops.

Criterion Analysis for Single Trajectory.

Table 1 shows the complete data for all criteria proposed, for the single trajectories 2-11. Q 1 is the success of the strategy detecting the final goal state. It can been seen that all strategies are able to detect the goal point. From the table it can be seen that Q 2norm and Q 3norm have its best value for strategy D. These strategies have a very fast detection and robot travel time. However, the distance traveled is not the best as this strategy allows the selection of safe-points which increase the robot travel distance. When considering safety, strategy D is second best. The best safety Q 5 is provided by strategy C.

After this detailed analysis for one trajectory, a discussion of the criterion evaluation for seven trajectories recorded is presented in the following sections.

Analysis of All Recorded

Experiments. The results obtained are summarized in Table 2. For all the tests, the final desired position is detected, thus criterion efficacy Q 1 is one and the criterion is not summarized in Table 2. The analysis of the results will be separated into two parts. First, an observation of the results obtained for the different trials will enable us to arrive at certain conclusions about less variations of the results as function of the trajectories. Then, in a second part, we will use the average values of the different criteria, taking into account the seven trajectories, to highlight the particularities of each strategy by comparing the criteria two by two.

Variations of the Results for the Different Trajectories.

For almost all trajectories, strategies B and A produce the same result, but for one test, the direction of the gaze is not well directed at the end of motion and a delay is observed with strategy B, contrary to what is expected. This delay affects the time of detection and also the time for the motion of the robot (trajectories 12-15). From the point of view of the efficiency of detection of the contact point, strategies A and D were the most efficient: two times for strategy A and five times for strategy D. There is no clear correlation between the efficiency of the strategy and the length of the trajectory. However, for short trajectories, strategy D is the most efficient.

User Distance

Versus Time for Detection. For the purpose of analyzing, -Q 5 has been used so as the goal would be to minimize all the selected criteria. A comparison of -Q 5 and Q 2 norm shows that strategies C and D belong to the Pareto front for these two criteria. Strategy D takes the least time to detect a desired point the user would like to reach and a slightly higher mean distance from the sphere as shown in Fig. 21. Strategy C has the largest user distance. However, it takes the longest time to detect a point than all the strategies.

User Distance Versus Robot Distance.

A comparison of -Q 5 and Q 4norm as presented in Fig. 22 shows that strategies A-B and C belong to the Pareto front for these two criteria. Since strategies A and B don't use any safe points they have smallest user distance (max -Q 5 ) and always take the minimal robot distance to arrive to a desired point (min Q 4 norm ). Strategies C and D use safe points so, have higher user distance (min -Q 5 ) and robot distance Q 4 norm . While strategy D uses head gaze as primary selection, it gives a value of Q 5 better than A and B but the worst robot distance. Strategies A-B and C belong to the Pareto front for these two criteria.

User Distance Versus Time for Robot.

A comparison of -Q 5 and Q 3norm as presented in Fig. 23 shows that strategies C and D belong to the Pareto front for these two criteria. As known Strategy C gives a higher value for safety (min -Q 5 ) followed by strategy D. However strategy D far outperforms strategy C in terms of time for the robot. points helps it in reaching the desired point faster, as the robot travels at higher velocity than in strategies A and B. Strategy A has lots of intermediate point, but as the hand moves closer to the desired point, the robot gradually moves closer. This is one of the reason why this strategy is the second best. Even though strategies B and C have a head gaze, the primary selection is still based on the hand. Unless the hand is closer to the point, the robot does not move to the desired point. For strategy D, it is the user gaze that helps in primary selection of points.

6.3.6 Robot Distance Versus Time for Robot. Visualization of Q 4 norm vs Q 3 norm is shown in Fig. 25 strategies A and D belong to the Pareto front for these two criteria. In contrary to the assumption that longer robot distance implies longer time for robot, the results show that strategy D has minimum time for robot but has longer robot distance. This result achieved is due to combination of fast time for detection and use of safe-point as intermediate points. Strategy C uses also safe-points, but it alone does not guarantee a fast response time. 6.4 Discussion. A comparative analysis of data from all the trajectories shows that if the objective is to maximize safety, strategies C and D would be better. Both strategies C and D ensure safety by selecting safe-points when the hand is far away from the desired point. The safe-points are located outside the user reach, such that the robot can travel fast and does not collide with the user. The selection of the safe-points mean that the robot will have to travel a longer distance to reach the desired point. While for strategies A and B, they select intermediate points which are inside the car and no safe-points. So the points are all inside the car, and the robot does not travel longer distance but has reduced velocity.

Strategy D gives second best safety and at the same time minimize the time to detect/reach a desired point. Therefore, it can be seen as the best strategy. The detection time for strategy D is the smallest because we used the gaze of the user to pre-select the points. This plays a big role in giving priority to vision information over information from the hand position. Fastest detection time allows the robot to start moving to the desired point at the earliest time and reach the desired point the fastest.

Conclusions

In this article, a collaborative robot is used as a haptic interface with intermittent contact. Four motion prediction strategies are used to select the areas with which the user intends to interact and to move the robot as fast as possible while ensuring user safety.

We introduce two speed profiles for the user's safety. The robot moves at a higher speed when it is outside the user's workspace. In situations where there is a large distance between two points within the workspace, we introduce via points to reduce travel time. The time needed to go through via points can be less than the time needed to go directly inside the car while being much safer.

Seven trajectories done by three users were analyzed, thanks to five criterion. A compromise must be made between user safety and speed for the robot to reach its target.

A simple realtime demonstration of the above system can be found. 1 In future works, we will perform other experiments with different users to test the robustness of our analysis and to know the influence of the threshold value. We will also realize the robot movements at the same time as the user's movements to validate his feeling when he/she hears the robot moving. Additional work is also in progress Transactions of the ASME to reassure the user by showing for example the position of the robot during its movement.

Introduction

Haptic systems enable user interaction in virtual reality by automatically recreating virtual scenes for dynamic interactions through haptic rendering, thus creating a link between a virtual world and the real world. Haptic systems should allow for a wide range of physical interactions and manipulations throughout the user's workspace, with a physical input that resembles reality. One promising approach to achieve this is the paradigm of encountered-type haptics (EHDs) [1]. EHDs are devices that autonomously position physical props for virtual objects in the real world at a target appropriately, thus allowing users to reach out to the virtual objects physically, just like in the real world. However, it is challenging for real-time interaction to organize physical props that accurately replicate the virtual world due to practical constraints, such as speed and workspace limits. In addition, the virtual environments are always much more extensive and richer in variety than the tracked physical space [2]. Speed limitations delay the device's arrival to some targets, creating discrepancies between what the user can see and what they feel. The resulting position and orientation mismatch between the virtual object and haptic proxy and latency negatively impacts the user experience [3,4]. While these issues may be partly solved by improving device hardware, factors such as cost, safety, and complexity often lead to design decisions that make device workspace and speed constraints unavoidable. Control approaches from the state-of-the-art, such as hapticretargeting [2] and user motion prediction, have been employed to address speed and latency issues [5]. Our study addresses this problem through motion prediction using the human eye-gaze tracking and hand motion. Previous studies have shown that the head movement facilitates subsequent gaze shifts toward the future position of the hand to guide object manipulations [6,7]. Thus, tracking eye movements is a natural way to learn about an intended reach target [8]. With eye-gaze information, hand movements, and the information in virtual environment, we can predict the tasks that the user will perform. Eye-tracking systems have been found to play an increasingly important role in assistive robotics as hand-free interaction interfaces for motor-impaired people [9], social gaze control for humanoids [10], robotic guidance [11], creating artistic drawings [12], and safe robot interactions in patients with speech and motor impairments [13]. Eyetracking combined with action observation tasks in a virtual reality display has been used to monitor motor deficits derived from stroke and, consequently, for the rehabilitation of stroke patients [9,14]. This study aims to develop and evaluate motion prediction strategies by analyzing the hand motion and eye gaze of adults when selecting targets. The strategies are used for upper limb training exercises to simulate activities of daily living tasks for people with motor impairments.

The main contributions of this work are:

1. We introduce and compare three strategies to detect human intention using the eye gaze and the hand motion to improve the human immersion.We use the eye-gaze detection rather than the eye-gaze attention used in [2,15,16]; 2.

We introduce a framework to implement the strategies; 3.

We implement a proof of concept that illustrates our proposed approach; 4.

We study the effect of the eye-gaze field of view and the threshold by comparing our approach to state-of-the-art eye-gaze-based robot control.

The remaining part of the paper is structured as follows. Section 2 discusses work related to haptic displays and prediction strategies. Section 3 describes the context of the study, the intention prediction strategies, the design and setup of the human-robot interaction model to contextualize the contribution of this research, the evaluation criteria, and the experimental design. Section 4 presents the results of the analysis of the performance of the strategies, and Section 5 discusses the results.

Related Works

This section focuses on haptic display devices, and the options researchers seek to improve surface rendering. Then, the state-of-the-art on human intention detection through motion predicting algorithms is presented.

Haptic Displays

There is a significant amount of studies on haptic devices in the literature. Our review will focus on encounter-type haptics, which employ a prop attached to a robot. The earliest work, Mcneely [17], presented the concept of encountered-type haptic device. The system places a haptic device at the desired location in time and waits for the user's encounter. It has the extra benefit of allowing the user's hand to move freely in open space and the use of physical props attached to a robot to represent virtual objects with varying sizes and shapes [18,19]. Other devices followed, such as the shape approximation device [20], haptic simulation of the refrigerator door [21], and a robotic turret with switches [22]. Surface rendering with texture and temperature characteristics [23] and new forms of EHDs, including shape-changing displays [24], surrounding platforms [25], mobile robots [26,27], and drones [28,29] also followed. To enable smooth interactions, EHDs need to achieve a high level of spatial-temporal consistency between the visual and haptic sensory inputs [1]. However, EHDs have limitations that lead to discrepancies between what the user can see and what can be felt, including limited workspace volumes, positional inaccuracy, and low speeds that may not support real-time interactions.

User Motion Prediction

Motion prediction strategies to determine the next target the human would like to reach and action to take can overcome timing constraints that affect most EHDs. This section explores the prediction and intention detection in the literature. Mostly, machine learning techniques, such as neural networks [30-32], Bayesian methods [32,33], principal component analysis [34], dynamic movement primitive [35], and hidden Markov models [36], have been used. A probabilistic principal component analysis was used for the recognition and prediction of human motion through motion onset detection by relying on a motion detection database of various motion models and an estimation of the execution speed of a motion [34]. Li et al. used a Bayesian predictor for the motion trajectory of the human arm in a reaching task by combining early partial trajectory classification and human motion regression in addition to neural networks used to model the non-linearity and uncertainty of human hand motion [32]. A combination of hidden Markov models and probability density functions was used in [36] to model the human arm motion and predict regions of the workspace occupied by the human using a 3D camera. In a related work, a Bayesian inference model [33] was used to infer the hand target and to promptly allow for the robot to reach a position within the scene. Based on observations from a 3D camera sensor, Ravichandar et al. [30] trained a neural network using a data set containing demonstrations of a human reaching for predefined target locations in a given workspace to infer a goal location for the human hand reach. However, all of the above models require vast amounts of training data. Furthermore, the performance of the models is dependent on the data acquired. Therefore the performance is affected when new measurements are received due to arm motion dynamics or different conditions of the human subjects. Other techniques that do not use training data are based on a distance metric [37]. This method selects an object closest to the user's hand by calculating the distance of all objects of interest in the scene from the hand and selecting the best. However, this method only detects the next desired object only when the hand has crossed the midpoint or has gone beyond the current minimum distance; therefore, if two objects are far apart, detecting the next one will take a longer time.

Since the hand position is one of the most informative features in human manipulation movement, the above works on intention inference based on hand motion. However, based on assumptions from studies on human behaviour, for most tasks involving object manipulation, humans reach to grasp an object and look at the target first. The gaze direction is always in the direction of the hands and the object manipulated [6,7], and therefore can be used to determine targets for interaction.

The eyes are considered as a window into the human mind because they can reveal information about human thoughts and intentions, as well as our emotional and mental states and where we are paying attention to [38]. Thus, the eye gaze can be used as a direct input to control robots and predict users' targets. Gonzalez et al. [2] used gaze fixation to predict an element of a virtual scene the user wants to reach. If the robot could not arrive at that target in time, they remapped the virtual element to a physical point within the EHD's reachable space. Stolzenwald et al. [15] introduced a model that predicts users' interaction location targets based on their eye gaze and task states using a hand-held robot. This model derives intention from the combined information about the user's gaze pattern and task knowledge. Castellanos et al. used eye-gaze information to predict the user target and provided haptic assistance for people with physical disabilities [16]. These works use gaze fixation to select the desired target. To classify an object as the target, they wait for a time ranging from 200 ms to 4 s when the eyes are fixated on an object. However, this approach results in unnecessary delays and may not be practical for smaller objects.

Using additional data from the head-mounted display, we use the gaze direction and only consider the points in the user-facing direction. The desired point is selected from a few candidate points within a defined threshold distance from the hand and a user view cone. In this approach, points that were not in the gaze direction or above the threshold were not considered, even if they were close to the user's hand. Our approach aims to pre-select all objects the user views and then select the desired object in the eye-gaze direction. Our method is designed to work with hand motions during real-world interaction and to give participants the freedom to make their own decisions along the way.

Methods

Context of Study

We based this study on an exergame designed for upper limb rehabilitation training for both the right and the left hand. The task aims to simulate reaching and grabbing balls in a virtual space. The study is inspired by the work presented in [39] for upper-limb and postural rehabilitation. Different balls are displayed to the player at different locations at a given time instance. He/she has to reach and grasp a ball of choice and release it above the virtual basket on the fl oor to gain points. The exergame is designed to record the active range of the motion of the user's hand using HTC Vive trackers. Then, the data are used by a control algorithm to generate virtual objects within the patient's comfort zone initially, and then to gradually push them further out of the comfort zone. The virtual world application allows the user to perform daily life activities while providing abundant repetitive movements and giving the patient visual feedback. The game was developed in collaboration with researchers and physiotherapists at the University of Genoa and LS2N. In this scenario, the user sits in the real world on a chair for a visual virtual reality experience and must reach out to pick balls with one of his or her hands. While the user is attempting to interact with a virtual object in the environment, the robot must position a ball to provide a tactical sense of touching the object [18,19]. A motion capture system based on HTC Vive trackers is used to determine the position of the hand used for interaction and the position of the chair and the robot. A tennis ball was attached to the robot's fl ange, as shown in Figure 1. The robot was mounted on a 0.8 m high table. The user was seated on a seat, positioned 0.6 m above the fl oor and 0.7 m from the robot. The robot's placement in the scene was chosen in order to allow it to reach all of the locations where the user's hand will want to have a haptic interaction with the robot's prop, as shown in Figure 2a,b. The arrangement of the balls fixed in the environment is represented by a virtual model created by the Unity© software. The main components for this study were:

1.

An encountered-type haptic device comprising a grounded Universal Robots UR5 robotic arm. A spherical prop was attached to give the sensation of touching a ball using a dominant hand; 2.

A motion capture system. The HTC Vive pro eye VR headset/head-mounted display for eye-tracking, a Vive tracker and base stations for tracking users' hand position, and another Vive tracker at the robot's base for robot positioning; 3.

Virtual Environment: Virtual objects were rendered using the Unity software along with the intention detection strategies. The Tobii XR SDK (Tobii Technology Inc., Stockholm, Sweden) captured and processed gaze data; 4.

Motion planning and obstacle avoidance. The algorithms for collision-free path and execution of the desired trajectories were implemented in ROS by using MoveIt function [40]. In the implementation, we ensured that the new objective is defined only when the robot has stopped. To avoid the computation of collision-free path, all trajectories used were pre-computed, and no new trajectory was generated during the experiments. The details of the implementation of the trajectory planning, collision, and obstacle avoidance algorithms are explained in [41].

Detection Strategies

Since the user has many balls presented in a virtual 3D environment at a given time instance, they have to choose one at a time. The robot's task is to arrive at the desired position in time. Different strategies were proposed to read the intention of the human in order to predict which ball he/she may want to reach.

Strategy 1: The Nearest Neighbour Approach

The most commonly used approach depends on finding the object closest to the hand. Implementation was carried out by computing the distances from the hand to all points of interest in search space, as used in [37], or, alternatively, by searching through a k-d tree, as used in [5]. In this study, we used a k-d tree to store the positions of all objects in the scene. Using hand location based on data from the tracker, we searched for the nearest to the hand from the k-d tree using Algorithm 1, as shown in Figure 3. The desired point is the closest to the hand, corresponding to min(d i , d N ): in this case, P 2 .

However, the main drawback is that the target is detected only when the human hand has almost reached the point. Moreover, if two points are close, switching between two points can occur. Using hand pose as a query point q, return nearest point from the k-d tree.

4:

return P * 5: end function

Strategy 2: Hand Position with Threshold

To detect the next desired point, a threshold distance between the hand and the current point of interaction was used to detect if the user intends to move their hand or if their hand is close to a point. Once the distance between the hand and the current point is above the threshold, we maintained the previous, and if the next point the hand approaches is within the threshold, it was taken as the indented target. The threshold ensures that only points in close contact are selected as explained in Algorithm 2. In this way, we aimed to reduce the detection of intermediate points and, hence, reduce the number of erroneous points detected. In Figure 4, the best point would be P 1 . 1: Build a k-d tree for all points P i in the scene. In addition to the hand position, strategy 3 uses eye-gaze direction to determine the next point. The next target is the point closest to the ray from the midpoint of the eyes in the gaze direction. As in previous works, the difference with this approach is that we do not wait for the gaze fixation on a specific object. In this approach, the detection is guaranteed to be fast. A threshold distance λ d was added onto the hand to detect when the user intends to move. If the hand to the point distance is within a threshold, we assumed that the user is still interacting with the current point. The value of λ d was chosen so that only one point P i can be inside. However, if the distance is above the threshold, the human wants to move to the next point; therefore, a new target was selected based on the eye-gaze direction. In this case, the threshold serves two roles. The first is to detect the intention of the user to move and then to cut off the selection of the next point by the head gaze. The threshold stops the robot from moving when the hand is near a point.

The search by gaze direction starts with only the points in the view frustum of the HMD. We carried this out to limit the search space and improve the detection speed.

In addition, we added a limit α on the angle from the gaze line to restrict the points selected by the eye gaze. The angle can be varied from 1%, as was used in [2] for a visual attention task. Another study [42] on visual attention perspective for social robotics modeled the threshold as a cone model of 30 • , whereas [43] used a slightly wider aperture of 40 • .

If there is no point within the limit α, the previous point was maintained. The ray in the gaze direction was then used to determine the next target. If the point-to-hand distance is above the threshold λ d , the point selected by the head gaze was taken as the desired target. Otherwise, it was ignored, and robot motion was restricted to the point near the hand. As shown in Figure 5a, the best point selected is P 1 .

We started by building a list of all points in the user view and then calculated the angle α i using Equation (1) for each point P i ∈ P. The next target is the point with a minimal α i < α value.

α i = tan -1 l i L i ( 1
)
l i is the projection of a point P i on the ray in the gaze direction and L i is the distance of the projection point to the center of the eyes. Algorithm 3 describes the procedure in two steps:

1.

Case 1: The hand is very close to a point as in Figure 5a. Search for the best P i using the strategy 1 If ||(P i , P h )|| < λ d , then P * ← P i ; 2.

Case 2: All points are very far as in Figure 5b. Next point is determined by eye gaze. P * ← P next from Equation ( 1). Vive tracking system was set up in a room without external disturbance, and the user was positioned at a distance of 0.7 m from the robot. A tracker was attached to the user's hand for motion capture in 3D space for interaction within game activities. The user held no other devices.

To ensure safety of the user, the workspace was divided by a safety plane into the human workspace and the robot workspace. The safety plane was used to restrict the motion of the robot to the robot workspace by using a motion planning algorithm described in [41]. In this study, the plane was considered as a static obstacle to be avoided. In addition, there was an emergency switch making it possible to cut off power to the whole system by fl icking a switch.

Experimental Task

The task comprised 16 tennis balls displayed in a virtual environment, located at points P 1 to P 16 , and spawned within the robot workspace, as shown in Figure 7. Three volunteers participated in this experiment. They included 1 female and 2 male participants with a mean age of 32 years. None of them had experience with eye-tracking displays; however, 1 of them had used a VR display. All participants were right-handed and provided written informed consent prior to the start of the experiment. Each participant was told to move the dominant hand from a ball specified by a number to a target ball also specified by a number.

The participants performed the task of reaching toward and grasping a ball with a radius of 7 cm and matching 3D virtual renderings as shown in Figure 8. The physical object was 3D-printed thermoplastic. Participants wore a head-mounted display to provide a 90 Hz virtual picture update frequency and scene sound effects while a tracker was attached to the hand. They viewed green-colored virtual renderings of these objects and a virtual rendering of the hand in a custom 3D immersive virtual environment designed in UNITY (ver. 19.4.1f1, Unity Technologies, San Francisco, CA, USA). The objects in the virtual environment were placed at different locations corresponding to the length of the arm 1/3 length of the arm at 25 cm from the centre (near), 2/3 arm length (middle) at 50 cm from the centre, and full arm length at 75 cm from the centre (far), each corresponding to a level of difficulty. A computer with an Intel Core i7-7700 processor and an NVIDIA gtx 2070 graphics processor was used to create the virtual environment.

Design of the Experiment

The main objective was to study the effect of the eye gaze on the detection time of the desired point, intermediate points, the time taken by the robot to reach the desired point, and intermediate stops of the robot. For this, three strategies were compared. The nearestneighbour method [5,37]

Data Collection

We recorded the participant's hand position, head position, and eye-gaze direction for each point-to-point trajectory. Data for the following trajectories were recorded:

Results

Out of the 39 recorded trajectories, one was discarded due to recording errors, and the remaining 38 were used for analysis. We first present a detailed analysis of an individual trajectory, then a summary of the results from 38 trajectories on Q1, Q2, Q3, and Q4, then an analysis on the effect of the hand threshold, and finally the effect of the eye-gaze window. It is important to note that, for the analysis of the results, the values of λ d = 0.15 m and the value on the eye-gaze threshold in strategy 3 was 60 • .

Analysis of the Trajectory from P 1 to P 16

We took, as an example, one of the user's motion trajectory from point P 1 to P 16 to analyze the results of the three strategies proposed based on the four criteria Q1, Q2, Q3, and Q4 (as shown in Table 1). A user view is shown in Figure 9 using strategy 3. The robot motion corresponding to each strategy is shown in Figures 10 and11. In Figure 10, we represent the hand trajectory and the resultant robot motion for the different strategies. For P 1 , we only show when the motion started. For the rest of the points, we indicate the time at which the hand was closest to each point, and the time the robot stopped at any point. A video of a user performing a motion from point P 1 to P 10 and P 1 to P 16 using strategy 3 along with the robot is provided in the Supplementary Materials. With strategy 1 as illustrated in Figure 12a, the points detected were P 1 , P 8 , P 13 , and P 16 . The desired point was detected at t = 4.86 s. Two intermediate points were detected: P 8 and P 13 . Points 8 and 13 are along the path of the straight line. Therefore, each of them was detected as the hand moved. The robot stopped at all the points detected, as indicated by the green line. The hand left from P 1 at t = 0.6 s. P 8 was the first point to be detected by the strategy and the robot received the point and moved towards it. However, before reaching P 8 , the strategy detected P 13 . Since the trajectory from P 1 to P 8 was not yet completed, the robot reached P 8 , stopped, and then started a new trajectory from P 8 to P 13 . It then waited for new information to go to P 16 .

Strategy 2

The motion of the hand, the robot, and the selection by strategy 2 is shown in Figure 12b. From P 1 , the strategy selected P 16 at t = 5.34 s. There were no intermediate points detected. This was possible because the hand threshold limits the selection of a point until the condition is met. This can be an advantage if the objective is to minimize the detection of unwanted points. However, it comes with a cost of late detection of the desired point when compared to other strategies, as shown in Figure 11. The robot moves directly to the desired point. However, it arrives after the hand has already reached the point.

Strategy 3

Figure 12c illustrates the progression of strategy 3. The strategy started with point P 1 , then selected P 13 , the best point in the user eye-gaze direction shown in Figure 9 by the red line on the camera icon, and, then, finally, P 16 . The robot started from point P 1 , then to the intermediate point P 13 , where it waited for a new point, and then to P 16 . As can be observed in the graph, the robot arrived at the final point earlier than the hand and the other strategies.

Analysis for All Trajectories

For all of the objectives Q1, Q2, Q3, and Q4, the data distribution was checked for normality using the Shapiro-Wilk test [45]. We used the strategies as a three-level factor and strategy 1 as the baseline for comparison. A one-way analysis of variance (ANOVA) model was used to fit the data. Results showed that there were significant differences among the strategies (p < 0.05) for all the objectives, with Q1 (F(2, 111) = 10.66 and p = 0.000), Q2 (F(2, 111) = 19.21 and p = 0.000), Q3 (F(2, 111) = 10.77 and p = 0.0), and Q4 (F(2, 111) = 30.62 and p = 0.000). Therefore, we reject the null hypothesis and conclude that the mean detection time, the number of intermediate points detected, robot arrival time, and the intermediate points detected by the robot are different for all the strategies. The results indicated that the effect of the eye-gaze tracking was significant for all of the objectives. A post hoc analysis was performed to find out the strategy-wise differences using the Bonferroni [46] and the Tukey test [47].

The Tukey test showed that the time for detection in strategy 3 was significantly lower than strategy 1 (p = 0.004) and strategy 2 (p = 0.000). Overall, strategy 3 was the best with the lowest time, as shown in Table 2 and Figure 13a. Compared to the baseline, the time difference was 0.92 s, representing a 37% reduction. However, there were no significant differences between the other strategies. These results indicate that the participants always looked in the direction of the desired point before moving their hand. Mutasim et al. [48] discovered similar results in a study of gaze movements in a VR hand-eye coordination training system. They found that the target was detected, on average, 250 ms before touch with eye gaze. Therefore, the use of eye-gaze direction tracking significantly reduced the detection time. A post hoc analysis using the Tukey test showed that strategy 2 had a significantly reduced number of intermediate points detected compared to strategy 1 (p = 0.000). The results can be seen in Table 2 and Figure 13b. The difference between strategy 3 and strategy 1 was insignificant, although strategy 3 had a lower number of intermediate points by 20%. Due to the rapid eye movements (the saccades), eye-gaze direction tracking can result in the detection of intermediate points. However, the hand threshold prevented the selection of a new target when the hand was close to a point, hence reducing the number of intermediate points in strategy 3.

Concerning the robot time, a post hoc analysis showed that the overall time taken for strategy 3 was significantly lower than strategy 1 (p = 0.025) by 23% and strategy 2 (p = 0.000), as shown in Table 2 and Figure 13a. The result indicates that eye-gaze tracking greatly improved the robot time. Even though the number of intermediate points detected by strategy 1 and 3 was similar, the motion planning algorithm ignored many points due to saccades, so they did not affect the results. In addition, strategy 3 improved the arrival time for the robot because of a lower detection time.

The number of robot stops was significantly higher in strategy 1 than strategy 3 (p = 0.000) by 69% and strategy 2 (p = 0.05) by 77% , as shown in Table 2 and Figure 13b. Although the difference in the number of intermediate points detected was insignificant between strategy 1 and strategy 3, the robot did not stop for all intermediate points. This implies that selections by strategy 3 due to saccades did not significantly affect the robot motion, thanks to the motion planning algorithm, which discarded new information received before a trajectory finished its execution.

Analysis of the Effect of Parameters on the Performance of the Strategies

The performance of strategy 3 depends on the values of the hand threshold parameter λ d and the eye-gaze window parameter α. Therefore, we conducted experiments to determine the effect of λ d and α on Q1, Q2, Q3, and Q4.

The Effect of the Hand Threshold

We experimented with different values of λ d , with λ d = 5 cm as the baseline, compared to λ d = 10 cm, 15 cm, 20 cm, 25 cm, and 30 cm. The results based on a data set with 34 different trajectories are presented below.

A one-way ANOVA model revealed a significant effect of the λ d on Q1 F(5, 198) = 8.099, p = 0.000. Post hoc comparisons using the Tukey HSD test [49] indicated that the mean time for λ d = 5 cm was statistically lower than that for λ d = 25 cm (p = 0.000) by 1.11 s and λ d = 30 cm (p = 0.000) by 1.19 s. Specifically, our results suggest that increasing the value of the threshold generally increased the time to detect the final point. A small threshold allows for the detection of the hand's intention to move away from the current point. This leads to an early detection of the desired point by the eye gaze. However, λ d had to be greater than 20 cm to notice a significant effect. Details are shown in Table 3 and in Figure 14a. A one-way ANOVA revealed no significant effect of λ d on Q2, with (F(5, 198) = 0.294, p = 0.916). The results are shown in Table 3 and in Figure 14b. The difference was not significant because of the following reasons. First, a lower value of λ d triggered selection by the eye gaze, which is affected by saccades, as observed in [16], resulting in a high number of intermediate points. Increasing the threshold would reduce the saccades because the target selection is by hand. However, this would mean that the strategy would tend to behave like strategy 1, increasing the number of intermediate points. Thus, selecting the correct value for this criteria is a trade-off between selection by eye gaze and selection by user's hand. The best balance was λ d = 10 cm or 15 cm.

A one-way ANOVA model revealed a significant effect of λ d on Q3 F(5, 198) = 7.486, p = 0.000). Post hoc comparisons showed significant differences between λ d = 5 cm, 10 cm, 15 cm, and λ d = 25 cm and 30 cm. Overall, λ d = 5 cm took the least time, as shown in Figure 14a and Table 3. The results show that the robot took a shorter time to reach the desired point for a small threshold.

A one-way ANOVA test showed that the hand threshold had a significant effect on Q4: F(5, 198 = 7.486), p = 0.0. The results are shown in Table 3 and in Figure 14b. Post hoc comparisons revealed that λ d = 5 cm was significantly different from λ d = 25 cm and 30 cm; however, it was not significantly different from λ d = 10 cm and 15 cm. The results show that a larger threshold increased the number of intermediate points detected by the robot. The mean values were similar for the lower values of λ d = 5 cm, 10 cm, and 15 cm. Then, the slope of the graph changed with increasing values of λ d . This pattern is different from the results obtained from the number of intermediate points selected by the algorithm. The algorithm selected a significant number of intermediate points for lower thresholds due to the saccades in the eye-gaze tracking. Therefore, the robot discards most of them thanks to a robust motion planning algorithm. On the contrary, as the threshold increases, the selection of points is mainly by hand. In this way, the algorithm behaves like strategy 1, which accounts for the increased number of intermediate points detected.

Overall, there was no significant difference between λ d = 5 cm, 10 cm, and 15 cm for all of the objectives Q1, Q2, Q3, and Q4. For this study, the best value selected was λ d = 15 cm, in accordance with the dimension of the environment. People hold a ball with a diameter of 7.5 cm. The tracker is placed on the top of the hand at a distance of approximately 5 cm from the palm. Therefore, the total distance from the center of the ball to the tracker was approximately 8 cm.

Eye-Gaze Window

Previous studies [2,10,42,43] have used different values of α, ranging from 1 • , 30 • , and 40 • , which have been used for selecting objects in the gaze window. However, there was no standard value for the appropriate gaze window size. Based on a data set with 37 different trajectories, we present results of the effect of α by comparing α = 5 • , 10 , and 60 • to the baseline α = 1 • . Normality checks were carried out and the assumptions were met.

A one-way ANOVA test showed that α had a significant change on Q1, with F(7, 288) = 2.230 and p = 0.032, as shown in Table 4 and Figure 15a. The results from a post hoc analysis showed that α = 1 • has a significantly longer detection time (p = 0.054) than α = 20 • , α = 25 • , α = 30 • , and α = 60 • , with a difference of 0.68 s. These results show that decreasing α delayed the detection of a point because of the smaller selection. A point cannot be selected until it is within the gaze window. A threshold greater than 10 • would give a view cone greater than 20 • , which would be large enough to accommodate several points in the user's gaze direction.

A one-way ANOVA test showed that Q2 was significantly affected by α, with F(7, 288) = 4.237 p = 0.000. More specifically, a post hoc analysis showed that α = 1 • had the lowest number of intermediate points, with a value significantly lower than α = 10 • (p = 0.003), α = 15 • , α = 20 • , α = 25 • , α = 30 • , and α = 60 • (p = 0.000). The results are shown in Table 4 and Figure 15b. This suggests that, when α was set to a value less than 10 • , the detection of intermediate points decreased significantly. A small selection window will block out many points, whereas a large window gives room for saccades. This relationship is depicted in Figure 15b.

There were significant differences in the time taken by the robot to reach the desired point: F(7, 288) = 5.451, p = 0.001. The time taken using α = 1 • was significantly greater than the rest (p = 0.000). These results showed that reducing α to a value < 10 • significantly delayed the robot. However, the difference was not noticeable between large values, as can be observed in Figure 15a. There was no significant effect of α on Q4. Adjusting the threshold had no effect on the intermediate stops of the robot, as observed in Figure 15b. These results follow a similar pattern to the results from Q2. However, in this case, the number was lower thanks to the robust motion planning algorithm.

Discussion

This study on the development and evaluation of strategies for user motion prediction was motivated by the need to improve detection speeds and increase the response time in EHDs.

Most importantly, our solution relied on the eye-gaze direction and hand position to determine human motion intention and desired targets. We analyzed data from three participants to determine the time taken by each strategy to detect the desired point, the number of intermediate points detected, the time taken by the robot to reach the final point, and the total number of intermediate stopping points of the robot. Strategy 3 gave the best detection time, robot time, and fewer robot stops. These results showed that the eye gaze significantly improved the response time while minimizing the number of robot stops. Our results were coherent with the literature on hand-eye coordination and target selection, which has identified that humans typically fix their gaze in the direction of the target, slightly before or after the hand begins to move, as shown in Figure 9.

The results suggest that visual behaviour for target selection with a haptic system is similar to behaviour when carrying out the task with hands in everyday life. Thus, the proposed system should work for people with motor impairments.

The prediction strategy based on the eye-gaze direction demonstrated a pattern to detect more intermediate points because of the saccadic movements. To minimize this behaviour, recent studies [15] in which the gaze direction is used to predict human intention utilized gaze attention models. In such models, they wait for a window period ranging from 200 ms to 4 s when the gaze is fixated on an object to validate it as a target. Such models affect robot arrival times and are applicable for large objects. In our case, the balls are not big. Thus, we used a threshold on the hand to limit the selection of the next point. The detection by the eye gaze was cut off when the point-to-hand distance was less than a threshold. In addition, the path-planning algorithm of the robot was designed to complete a trajectory before starting a new one. Thus, rapid trajectory changes due to saccades were always discarded. This implies that our model can be used for both small and large objects, as long as a suitable threshold on the hand is selected.

In this study, the hand threshold plays a vital role in detecting human motion intention. In studies where the nearest point to the hand method is used [2,18], the target was detected whenever the hand had crossed half the distance between any two points. However, when coupled with the eye gaze, a hand threshold was used to detect the user's intention to move to another point. In this way, the hand-to-point was below the threshold, and the user intention was interpreted as a desire to remain at a point. Therefore, the robot remained stationed at the point. The threshold also served to restrict the detection of a new point. Thus a threshold plays a significant role in determining the detection time of the target and the intermediate points. Thus, it affects the robot arrival time and the intermediate points detected along the robot trajectory. We experimented with different threshold values on the hand to determine a suitable threshold value. The analysis revealed that a lower threshold was associated with a faster detection time. We attribute this to the fact that a lower threshold value indicated an earlier detection of the intention by the user to move to another point.

Due to the lack of clear agreement on the standard size of the gaze view window in studies investigating eye gaze and hand coordination patterns [2,10,42,43], we examined the effect of the threshold on the eye gaze. Our results showed that a view angle greater than 20 • , as used in strategy 3, had similar results for all of the research questions Q1, Q2, Q3, and Q4. However, a reduced threshold ≤ 10 • was associated with a significantly increased time for detection, but reduced the number of intermediate points. A small threshold implied that a few points would be selected at a time. Thus, it would take longer to have a valid selection, which greatly increased the time to detect the desired point and consequently delayed the robot. From the analysis, we discovered that the gaze fixation model, as used in [2], increased the detection time, hence delaying the robot.

Previous studies [16,50] pointed out that the visual gaze is full of rapid eye movements between fixed points (saccades), and it was the primary reason why gaze fixation was the widely used approach. However, we do not use gaze fixation because it takes longer to detect a point. In addition, it is unsuitable for smaller objects. Instead, we used a threshold on the hand to restrict robot motion when a point lies within the stated threshold. We select the best point depending on the angle α and not the visual ray directly. By combining the hand threshold and a good motion-planning algorithm, saccadic movements do not significantly affect the robot motion when a large threshold on the eye gaze is used. Thus, our approach is robust to saccades and highly responsive.

Our findings show that prediction based on the eye gaze improved the response time for the robot. However, optimizing the detection time from human predictions comes at the cost of increasing the intermediate points detected. We observed this through an analysis of the threshold values on the hand. A lower value resulted in a good detection time and a higher value of intermediate points detected. Thus, a compromise has to be made to improve the detection time and reduce the detection of the intermediate points. Therefore, we recommend finding a suitable threshold on the hand and the eye-gaze window to suit the task.

In addition, our system only uses positions in a 3D space; it would be good to extend the interaction to 6-DOF to study the implications of prediction and robot time in haptic rendering systems where positions and orientations of virtual objects are essential. Although VR hand-eye coordination significantly improved the detection time, we observed that participants spent some time searching for a target. Therefore, further research is needed to minimize the time spent searching for the next target to increase the user's performance and the eye-hand coordination training system. Our work was preliminary on the proof of concept with a few healthy participants. It would be essential to evaluate the haptic system with many people with motor disabilities.

Eye-gaze detection to predict targets for haptic devices is a promising solution to improve intention detection and robot response. However, due to saccades during decision making and target search, additional studies are needed on methods to process gaze data.

Conclusions

Haptic systems enable physical user interaction with a virtual world by automatically recreating virtual scenes for dynamic interactions through haptic rendering. However, speed constraints present a challenge for the real-time interaction of such systems. We have addressed this problem through motion prediction using eye-gaze direction and the user's hand. This study developed motion prediction strategies in a virtual environment for reaching tasks. Based on data from three participants, our study confirms the principle that the eye gaze precedes hand movement for reaching tasks. Furthermore, our results confirm that the strategy using eye-gaze-based prediction significantly reduced the detection of the desired point and reduced intermediate points. This significantly improved the robot response, with fewer intermediate robot stops. More specifically, our approach showed better results than the state-of-the-art, which relies on gaze fixation. Therefore, this approach may be helpful to communities using haptic systems for upper extremity rehabilitation training and tasks for rapid prototyping in industrial design [51] to improve the response time and device speed.

6.1 Summary of the thesis ACM Digital Library, Medline (via PubMed), Cochrane, IEEE Xplore, Web of Science, Scopus, and science direct. The effect sizes of the primary outcomes were calculated using Cohen's d. Pooled results were used to present an overall estimate of the treatment effect using a random-effects model. The evaluated trials suggest that IVR therapies may be more effective than NIVR but not CT in improving upper limb activity, function, and daily life activities. However, the evidence showed promising benefits of immersive virtual reality technology.

The next aspect was implementing a motion generation scheme for the haptic interface envisaged, which considers the obstacles present in a Virtual Reality (VR) environment. The most essential thing to ensure was the user's safety for the proposed human-robot interaction system. This one has to be guaranteed in order to be able to implement a reliable system. This study was a continuation of the work presented by Guda et al. (2020); Posselt et al. (2017) where the main objective is to generate a safe motion scheme that takes into consideration the obstacles present in a Virtual Reality (VR) environment. The work was developed using MoveIt software in ROS to control a UR5 industrial robot. With this, we were able to set up the planning group, which is confirmed by the UR5 robot along with a 6-faced prop and a ball attached to the robot flange to plan feasible trajectories for the robot to execute within the environment.

We first studied the software capabilities and options for path planning, along with the different ways of executing motions. We then compared the different path-planning algorithms to determine which one best suits the task. Finally, we proposed different mobility schemes for the robot to execute depending on the situation faced. The first one is when the robot needs to plan trajectories in a safe space, where the only obstacle to avoid will be the user's workspace. We ran two different experiments to choose a planning algorithm using MoveIt. The first one was intended to determine which one of the 12 available planning algorithms was better suited for path planning within the system. The second set of experiments considered the execution times of the trajectories generated by the algorithms. For each path-planning algorithm, both RRT and RRTConnect, the solutions found were less optimal, sometimes generating longer trajectories with high deviations from the optimal path. However, even if both algorithms take more or less the same time to find a solution, BiTRRT was found to have faster execution times. This is reflected in the fact that for the BiTRRT algorithm, the trajectories computed were more "consistent", obtaining similar trajectories every time. Thus we decided to pick the BiTRRT algorithm as our path-planning algorithm for the implementation of this project. One of the advantages of this implementation is that by directly identifying which trajectory we want to execute, we can access the pre-computed plan and execute said trajectory directly. The motion generation scheme was then used to execute point-to-point trajectories of the robot based on data received from the virtual environment in the Unity 6.1 Summary of the thesis software. Two different virtual environments based on different applications were used as case studies to develop further user interaction techniques and motion prediction algorithms based on sensor data from the HTC Vive tracking System.

The first case study was based on the interior of a vehicle containing a person (which would be the user in this case) for which the set-up was made in order to be taken into account within the system. The user interacts with a 6-faced prop which rotates to present different textured material for user haptic perception. We developed intermittent contacts using an industrial robot for human-robot interaction, such that the robot moves to the user when contact needs to be made. Since the user is immersed in the virtual world via a VR HMD, he cannot perceive the danger of a collision when he changes his area of interest in the virtual environment. The objective of this study was to describe four movement strategies for the robot to be as fast as possible in the contact zone while guaranteeing safety. This work uses the concept of predicting the user's intention through his gaze direction and the position of his dominant hand (the one touching the object), and safe points outside the human workspace. We introduce two-speed profiles for the user's safety. The robot moves at a higher speed when it is outside the user's workspace. In situations where there is a large distance between two points within the workspace, we introduce via points to reduce travel time. The time needed to go through via points can be less than the time needed to go directly inside the car while being much safer. Seven trajectories done by three users were analyzed thanks to five criteria. A compromise must be made between user safety and speed for the robot to reach its target. Experiments are done and analyzed with a Pareto front with a UR5 robot and an HTC Vive tracker system for an industrial application involving the analysis of materials in the interior of a car. The study showed that to maximize user safety, there has to be a compromise on speed. So by dividing the workspace into different zones, velocity modulation techniques were able to improve device speed.

The second case study presents an experimental study on an exergame for upper limb rehabilitation training. A person sits on a chair and has to reach out balls in a virtual scene. He interacts with a ball attached to the robot and must be at the exact location of the virtual ball the user intends to touch. An interactive VE developed in unity3D software was used for total user immersion, and a sensor is attached to his dominant hand. The objective was to improve the prediction time and reduce the robot's time to reach the desired position. This study presented several motion prediction strategies to infer user intention and predict intended targets in a VE for reaching tasks based on hand motion and eye gaze tracking. Point-to-point trajectories were executed using the motion planning scheme for user safety. To assess the performance of the strategies, we conducted three subject-based experiments using 39 trajectories. The experimental results in this study revealed that eye-gaze-based prediction significantly improved the 6.2 Limitations and future perspectives detection time by 37% and the robot time taken to reach the target by 27%. Further analysis was done to provide more insight into the effect of the eye-gaze window and the hand threshold on the device response for the experimental task. This study confirmed that using eye-gaze-based tracking significantly reduced the detection of the desired point and greatly improved device speed and response, with fewer intermediate robot delays. Our approach showed better results than the state-of-the-art, which relies on gaze fixation. Therefore, this approach may be helpful to communities using haptic systems for upper extremity rehabilitation training and tasks for rapid prototyping in industrial design (Posselt et al., 2017).

Limitations and future perspectives

The utilization of IVR systems for motor rehabilitation programs is still in the early stages of adoption. IVR is a relatively new technology and remains partially known, with a lot of the work limited to pain and phobias treatment. There were few studies on the effectiveness of IVR systems in stroke rehabilitation that examine its long-term benefits. Therefore more trials are needed to evaluate the benefits of IVR technologies in improving physical and psychological status among people with neurological conditions.

We developed a motion generation scheme. However, one of the limitations is that, given the nature of performing pre-recorded trajectories, the system needs to fully execute each of the trajectories that a user performs once the robot has received a target. Therefore, the path planning algorithm needs to accommodate re-planning when a new target is detected.

Although VR hand-eye coordination significantly improved the detection and response time of the robot, we observed that participants spent some time searching for a target. Therefore, further research is needed to minimize the time spent searching for the next target to increase the user's performance for the eye-hand coordination training system.

Our work was preliminary on the proof of concept. The motion strategies were evaluated based on a few participants. Future work should conduct more studies based on a large number of users, including people with motor disabilities, and more trajectories to test the robustness of our analysis.

Using Eye-gaze tracking to predict targets for haptic devices is a promising solution to human-robot interaction and robot response. However, due to saccades during decision-making and target search, additional studies are needed on methods to process gaze data.

Further work is also envisioned to evaluate the system by realizing the robot's movements simultaneously with the user's movements to validate his feeling when 90

Conclusion

he/she hears the robot moving.

Conclusion

VR presents immense benefits for industrial and medical care applications, including so many benefits such as reduced costs, among others. This thesis studies the development of a novel haptic interface and motion planning strategies for safe human-robot interaction. We developed motion strategies and user interaction techniques to overcome inherent challenges such as speed, workspace, and safety associated with using industrial robots as encountered type haptic devices with intermittent contacts. Two case studies are presented for the analysis of the developed haptic interface and the motion planning strategies. The first is an industrial application for analysis of the material of the interior of the car during the early stages of development, while the second is the application of the haptic interface to upper limb rehabilitation training. We developed an immersive virtual environment for user interaction using Unity software, human motion intention detection, and motion prediction algorithms based on gaze direction and hand motion to predict the points of interest for the user. A motion planning algorithm based on velocity modulation was developed to move the robot as fast as possible in areas outside the user's workspace and slowly in areas close to the human. Different path-planning libraries to execute safe point-to-point trajectories were evaluated, and the best one suitable for the task was selected. An obstacle avoidance algorithm was proposed to limit collision between the user and the robot and prevent the robot from colliding with itself. For both applications considered in this thesis, an evaluation of the motion strategies was conducted based on user studies. Results were analyzed to choose the best motion strategy to adopt for each application to overcome the inherent problems of haptic devices. Title: Motion strategies for a haptic interface with intermittent contacts to ensure safe human-robot interaction.

Keywords: intermittent contacts interface, Haptic interface, Human robot interaction, Virtual reality, human safety Abstract: Virtual reality has been recognized as a potent tool for creating more natural and intuitive human-computer interfaces and has been found to be beneficial in so many applications. However, the inability to interact in a virtual environment through touch compromises its realism and usefulness. Haptic interfaces with intermittent contacts allow users to reach out and touch the virtual objects physically to simulate contact between the user and the environment with use of tactile sensation to increase the realism of interaction. They allow a wide range of physical interactions throughout the user's workspace, with a physical input that resembles reality. These devices are faced with challenges such as cost, a small workspace, limited speed and user safety. In this thesis, we developed a haptic interface using a cooperative robot to address these challenges. Several motion strategies, a trajectory generation scheme and user interaction techniques to ensure safety were developed and evaluated. Two case studies were used as application areas. The first is an industrial application for analysis of the interior material of the car during the early stages of development while the second one is a haptic interface for upper limb rehabilitation training. User studies were conducted to evaluate the efficacy of the motion strategies in improving device speed, response, and user safety.

1. 1

 1 Virtual reality 1.1.2.

1. 2

 2 Haptic systems ders have shown that Robot-assisted therapy is as effective as conventional therapy Kwakkel et al. (2008); Laver et al. (2017); Norouzi-Gheidari et al. (2012); Sivan et al.

1. 4

 4 Research objectivesvice's arrival at some targets, creating discrepancies between what the user can see and feel. The resulting position and orientation mismatch between the virtual object and haptic proxy and latency negatively impacts the user experience (Di Luca et al., 2011; Lee et al., 2016). In addition, virtual environments are always much more extensive and richer in variety than tracked physical space. Workspace limits mean that the device can recreate only a limited part of the Virtual environment. So finding a suitable place for the robot to reach the areas of interest in VR is another challenge. The limited workspace of haptic devices used in rehabilitation, such as the Novint Falcon limits the range of possible motions that can be perfomed by a user in a VE. Therefore, cannot support many upper limb rehabilitation exercises involving arm and shoulder movement D'Auria et al. (2016).

  ; C-Control; CT-Conventional Therapy; CAHAI -Chedoke-McMaster Arm and Hand Activity Inventory; COPMP-Canadian Occupational Performance Measure Performance; COPMS-Canadian Occupational Performance Measure Satisfaction; DGI-Dynamic Gait Index; ES-Effect Size: ESAS -Edmonton Symptom Assessment Scale; FM-Fugl-Meyer; FIM-Functional Independence Measure; FRT-Functional Reach Test; IVR-Immersive Virtual Reality; JTT -JebsenÀTaylor hand function test; LOS-Limits of stability; MAL-Motor Activity Log; MAS-Modified Ashworth Scale; MBI-Modified Barthel Index; MFT-Manual Function test; M-PAES-Modified Physical Activity Enjoyment Scale; NHPT-Nine Hole Peg Test; NIHSS-National Institutes of Health Stroke Scale; NIVR-Non-Immersive Virtual reality; PASS-BADL-Performance Assessment of Self-Care Skills À basic activities of daily living; PASS-IADL-Performance Assessment of Self Care Skills À instrumental activities of daily living; PGT-Purdue pegboard test; QoM-Quality of Movement; SF-Short Form Health Survey; WMFT-Wolf Motor Function Test the results were not statistically significant. A forest plot is shown in Fig. 7.

Fig. 3 .

 3 Fig. 3. Forest plot of comparison: IVR Versus CT (Short term Effects), outcome: Activities of Daily Life.

Fig. 4 .

 4 Fig. 4. Forest plot of comparison: NIVR Vs CT (Short term Effects), outcome: 3.3 BBT.

Fig. 5 .

 5 Fig. 5. Forest plot of comparison: Non-Immersive VR Vs CT (Short term Effects), outcome: TUG.

Fig. 2 .

 2 Fig. 2. Forest plot of comparison: 1 IVR Versus CT (Short term Effects), outcome: FM

Fig. 6 .

 6 Fig. 6. Forest plot of comparison: NIVR Vs CT(Short term Effects), outcome: 10mW.

Fig. 7 .

 7 Fig. 7. Forest plot of comparison: NIVR Vs CT(Short term Effects), outcome: BBS.

Fig. 8 .

 8 Fig. 8. Forest plot of comparison: NIVR Vs CT(Short term Effects), outcome: QOL.

Fig. 9 .

 9 Fig. 9. Forest plot of comparison: NIVR Vs CT(Short term Effects), outcome: ADL.

Fig. 11 .

 11 Fig. 11. Forest plot of comparison: NIVR Vs CT (Long Term Effects), outcome: Quality Of Life.

Fig. 10 .

 10 Fig. 10. Forest plot of comparison: NIVR Vs CT (Long Term Effects), outcome: Upper Limb Activity. BBT.

Fig. 12 .

 12 Fig. 12. Forest plot of comparison: NIVR Vs CT (Long Term Effects), outcome: ADL.

Fig. 1 .

 1 Fig.1. Global[7] and local[8] path planning.

Fig. 2 .Fig. 3 .

 23 Fig. 2. Laboratory Setup

Fig. 4 .

 4 Fig. 4. Planning group and mannequin model of the user

Fig. 5 .

 5 Fig. 5. Planned paths using move group.execute(my plan): Back-to-back plans (left) and Velocities for both plans (right).

Fig. 6 .

 6 Fig. 6. Re-planned paths using move group.asyncExecute(my plan) (a) Back-to-back plans. (b) Segment not executed due to re-planning. (c) Final executed plan.

Fig. 7 .Fig. 8 .

 78 Fig. 7. Comparison of all planning algorithm's times for nine trajectories

Fig. 9 .

 9 Fig. 9. Comparison of average execution times of the algorithms.

Fig. 10 .

 10 Fig. 10. Comparison of average amount of generated via-points.

Fig. 11 .Fig. 12 .

 1112 Fig. 11. Unity VR system and representation of interaction points

Fig. 13 .Algorithm 1 1 :

 1311 Fig. 13. Moving obstacles

Fig. 14 .

 14 Fig. 14. Original planned path (left) and Re-planned path from detected collision (middle and right)

Fig. 1 Fig. 2

 12 Fig.1Conceptual scheme of the experimental platform with a user touching a prop carried by a cobot and wearing a HMD[17]

Fig. 4 3 3. 4

 434 Fig. 4 Diagram of the inputs used to choose a robot movement strategy

Fig. 6

 6 Fig. 6 Flow of data between the systems

Fig. 10

 10 Fig. 10 Illustration for the comparison of motion using safe-points

Algorithm 1

 1 Strategy A: Predictions with hand Input: Hand position P h ∈ R 3 . Output: Nearest point P in the set of P i , i = 1 • • • 18. 1: Build a k-d tree for all points P i in the scene. 2: function STA (P h ) 3:

Fig. 11

 11 Fig. 11 Comparison of motion through safe-points and without safe-points

Algorithm 2

 2 Strategy B: Predictions with head gaze Input: Hand position P h ∈ R 3 , HMD position P s ∈ R 3 , head orientation O s ∈ R 4 . Output: Nearest point P. 1: Build a k-d tree for all points in the scene. 2: function STB (P h , P s , O s ) 3:

Algorithm 3

 3 Strategy C: Addition of safe-pointInput: Hand position P h ∈ R 3 , HMD position P s ∈ R 3 , head orientation O s ∈ R 4 , hand threshold T h (shown in Fig.14). Output: Nearest point P. 1: Build a k-d tree for all points in the scene.

Fig. 14

 14 Fig. 14 Pictorial representation of strategy C

Fig. 16

 16 Fig. 16 User hand trail for motion from points 2 to 11

6. 2 . 4

 24 Strategy C. A graph of robot motion is indicated by the dark line as shown in Fig. 19. The robot starts at P 2 and waits for a new point. When point SP 20 is detected, it starts moving but along the way, a new point SP 21 is detected, so it has to complete the motion to SP 20 first. By the time it has reached the point SP 20 , points SP 21 , SP 23 , and SP 24 have been detected and passed by the prediction algorithm. All the points that were detected and passed during the motion of the robot have been ignored by the robot. Once a new point has been detected, the goal state of the robots updates and neglects the previous points that have not been reached. After reaching SP 20 , the prediction now shows point P 11 as the desired destination. So the robot moves to P 11 . 6.2.5 Strategy D. The motion of the robot is indicated by a dark line as shown in Fig.

Fig. 18 11 Fig. 19

 181119 Fig. 18 Robot motion, user distance, and time for detection for strategies A and B for trajectories 2-11

6. 3 . 5

 35 Time for Robot Versus Time for Detection. Visualization of Q 3norm versus Q 2norm is shown in Fig. 24. It shows that the faster the strategy detects the point, faster the robot reaches the point. The best being strategy D. It uses the head gaze and an added use of safe-

Fig. 21

 21 Fig. 21 Comparison of time for detection (Q 2norm ) versus user distance (-Q 5 ) for the four strategies, all trajectories. Fig. 22 Comparison of robot distance (Q 4norm ) versus user distance (-Q 5 ) for the four strategies, all trajectories

6. 3 . 7

 37 Time for Detection Versus Robot Distance. Visualization of Q 2norm versus Q 4norm shown in Fig. 26 strategies A and D belong to the Pareto front for these two criteria. From Fig. 26, it can be seen that Strategy A and B have least robot distance, but it detects the goal later than Strategy D. Strategy C is not ideal in both criteria.

Fig. 23

 23 Fig. 23 Comparison of time for robot (Q 3norm ) versus user distance (-Q 5 ) for the four strategies, all trajectories

Figure 1 .

 1 Figure 1. The designed prop, a physical representation of the virtual objects presented to the user during interaction.

Figure 2 .

 2 Figure 2. Experimental setup with the robot, the balls, and the user. (a) The side view. (b) The front view.

Figure 3 .Algorithm 1

 31 Figure 3. Pictorial representation of strategy 1. Algorithm 1 Strategy 1: Predictions with hand. Input: Hand position P h ∈ R 3 . Output: Best point P * in the set of P i , i = 1 . . . 16. 1: Build a k-d tree for all points P i in the scene. 2: function ST1(P h ) 3:

Figure 4 .Algorithm 2

 42 Figure 4. Pictorial representation of strategy 2. Algorithm 2 Strategy 2: Hand position with threshold. Input: Hand position P h ∈ R 3 , threshold distance λ d . Output: Best point P * in the set of P i , i = 1 . . . 16.1: Build a k-d tree for all points P i in the scene. 2: function ST2(P h , P prev )

Figure 5 .

 5 Figure 5. Strategy 3 prediction with eye gaze tracking. (a) Case 1: Point within λ d , (b) Case 2: All points outside λ d .

Figure 7 .

 7 Figure 7. Virtual environment rendering of the scene.

Figure 8 .

 8 Figure 8. The user performing the experimental task in unity without motion of the robot.

  (strategy 1) was used as the baseline. The null hypothesis was that eye-gaze-based prediction had similar results as the selection with the hand-alone strategies for the detection time, intermediate points detected, robot time, and intermediate points for the robot. For the research objective, the following evaluation criteria were defined: • Q1: The time taken by the strategy to detect the desired point; • Q2: The number of intermediate points detected by the strategy before the desired point was detected; • Q3: The time taken by the robot to reach the final point. This was the sum of the duration of all of the pre-computed trajectories plus the waiting time of the robot; • Q4: The total number of intermediate stopping points of the robot.

Table 1 .Figure 9 .

 19 Figure 9. User's hand trail and selection by eye gaze for motion from P 1 to P 16 . The hand trail is shown as a pink line. The ball selected by the eye-gaze direction is P 13 , indicated by a slim red line from the camera, represented as an icon.

Figure 10 .

 10 Figure10. A representation of the actual hand motion and the resultant robot motion projected on the y-z plane. The time the robot stops at a point is indicated for each strategy, as well as the time the hand is closest to each point. For p 1 , the time at which the motion starts is indicated. For p 8 , p 13 , and p 16 , the time the robot stops is indicated. For p 16 , the time the hand stops is indicated. For p 8 and p 13 , the hand that is the closest is indicated.

Figure 11 .

 11 Figure 11. Comparison of the strategies for the trajectory from P 1 to P 16 . The dotted line indicates the time the hand is at the start and the target point. 4.1.1. Strategy 1

Figure 12 .

 12 Figure 12. Results of individual strategies selection with the hand and robot motion for the trajectory from P 1 to P 16 showing the points selected by the strategies, the robot, and the hand stops. The dotted lines represent the motion of the hand and the robot. The graphs only show the points and the time the hand and robot stop. (a) Selected points using strategy 1 along with the robot and hand stops. (b) Strategy 2 points selection along with the robot and hand stops. (c) Strategy 3 selection along with the robot and hand stops.

Figure 13 .

 13 Figure 13. Results for each strategy. (a) Q1: Time taken for each strategy to detect the desired point. Q3: Time taken by the robot to reach the desired point. (b) Q2: The number of intermediate points detected by each strategy. Q4: The number of intermediate stopping points of the robot.

Figure 14 .

 14 Figure 14. Results for each value of the hand threshold. (a) Q1: The time taken by the strategy to detect the desired point. Q3: The time taken by the robot to reach the desired point. (b) Q2: The number of intermediate points detected by the strategy. Q4: The total number of intermediate stopping points of the robot.

Figure 15 .

 15 Figure 15. Investigating the effect of the eye-gaze threshold using different values of α. (a) Q1: The time taken by the strategy to detect the desired point. Q3: The time taken by the robot to reach the desired point. (b) Q2: The number of intermediate points detected by the strategy. Q4: The total number of intermediate stopping points of the robot.

Titre:

  Stratégies de mouvement pour une interface haptique avec des contacts intermittents pour assurer une interaction homme-robot sûre.Mots clés :Interface contacts intermittents, Interface haptique, Interaction homme-robot, Réalité virtuelle, Sécurité humaine Résumé : La réalité virtuelle a été reconnue comme un outil puissant pour créer des interfaces homme-machine plus naturelles et intuitives et s'est avérée bénéfique dans de nombreuses applications. Cependant, l'incapacité d'interagir dans un environnement virtuel par le toucher compromet son réalisme et son utilité. Les interfaces haptiques avec des contacts intermittents permettent aux utilisateurs d'atteindre et de toucher physiquement les objets virtuels pour simuler le contact entre l'utilisateur et l'environnement en utilisant la sensation tactile pour augmenter le réalisme de l'interaction. Ils permettent un large éventail d'interactions physiques dans l'espace de travail de l'utilisateur, avec une entrée physique qui ressemble à la réalité. Ces appareils sont confrontés à des défis tels que le coût, un petit espace de travail, une vitesse limitée et la sécurité des utilisateurs. Dans cette thèse, nous avons développé une interface haptique utilisant un robot coopératif pour relever ces défis. Plusieurs stratégies de mouvement, un schéma de génération de trajectoire et des techniques d'interaction avec l'utilisateur pour assurer la sécurité ont été développés et évalués. Deux études de cas ont été utilisées comme domaines d'application. Le premier est une application industrielle pour l'analyse du matériau intérieur de la voiture pendant les premières phases de développement tandis que le second est une interface haptique pour l'entraînement en rééducation des membres supérieurs. Des études d'utilisateurs ont été menées pour évaluer l'efficacité des stratégies de mouvement dans l'amélioration de la vitesse, de la réponse et de la sécurité de l'utilisateur.

  The remaining part of the thesis is structured as follows: Chapter 2 describes the role of VR in stroke rehabilitation. I evaluate the efficacy of different forms of VR treatments as either IVR or NIVR in comparison to CT in improving the physical and psychological status among stroke patients Mugisha et al. (2022b).

	1.6 List of publications 1.7 The structure of the thesis
	2. Developed specific path planning techniques for collision and obstacle avoid-1.7 The structure of the thesis
	ance.
	3. Presented velocity modulation schemes to ensure user safety and improve
	the performance of encountered haptic devices.
	4. Introduced human intention detection and target prediction strategies by
	taking into account hand motion and eye gaze.
	5. Contributed towards user motion prediction in VR by learning from the
	hand trajectory data.
	; Chab-
	lat, D.(2022) "Improving Haptic Response for Contextual Human Robot
	Interaction." Sensors. https://doi.org/10.3390/s22052040
	5. Mugisha, S., Job, M., Zoppi, M., Testa, M., & Molfino, R. (2022). Computer-
	Mediated Therapies for Stroke Rehabilitation: A Systematic Review and
	Meta-Analysis. Journal of Stroke and Cerebrovascular Diseases, 31(6),
	106454.

1.6 List of publications 1. Mugisha, S, Zoppi, M, Molfino, R, Guda, V, Chevallereau, C, & Chablat, D. "Safe Collaboration Between Human and Robot in a Context of Intermittent Haptique Interface." Proceedings of the ASME 2021 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. Volume 8B: 45th Mechanisms and Robotics Conference (MR). Virtual, Online. August 1719, 2021. V08BT08A007. ASME. https://doi.org/10.1115/DETC2021-71518 2. Gutierrez, A., Guda, V., Mugisha, S., Chevallereau, C., & Chablat, D. (2022). Trajectory planning in Dynamics Environment: Application for Haptic Perception in Safe HumanRobot Interaction. 24TH International Conference on Human-Computer Interaction, Jun 2022, Gothenburg, Sweden 3. Guda, V., Mugisha, S., Chevallereau, C., Zoppi, M., Molfino, R., and Chablat, D. (2022). "Motion Strategies for a Cobot in a Context of Intermittent Haptic Interface." ASME. J. Mechanisms Robotics. doi: https://doi.org/10.1115/1.4054509 4. Mugisha, S.; Guda, V.K.; Chevallereau, C.; Zoppi, M.; Molfino, R.

  al. 2017; Allen et al. 2017; Aşkın et al. 2018; Choi, Shin, and Bang 2019; Henrique et al. 2019; Huang and Chen 2020; Ikbali Afsar et al. 2018; In, Lee, and Song 2016; Kiper et al. 2018; K. H. Kong et al. 2016; Llorens et al. 2015; McNulty et al. 2015; Mekbib et al. 2021; € Og€ un et al. 2019; Pedreira da Fonseca et al. 2017; Saposnik et al. 2016; Schuster-Amft et al. 2018; Shin et al. 2016; Shin, Bog Park, and Ho Jang 2015; Da Silva Ribeiro et al. 2015; Bin Song et al. 2015; Zondervan et al. 2016. And these 2 studies

, 2021; Pedreira da Fonseca et al., 2017; Saposnik et al., 2016; Zondervan et al., 2016

	COMPUTER-MEDIATED THERAPIES FOR STROKE REHABILITATION et al., 2017; Llorens et al., 2015; McNulty et al., 2015; Mek-
	bib et al.

studies Aşkın et al., 2018; Bin Song et al., 2015; Choi et al., 2019; Da Silva Ribeiro et al., 2015; Huang & Chen, 2020; In et al., 2016; H.-C. Lee

Table 2 .

 2 IVR vs CT Participant Information.

	Study	Study		Participants
		Type
			population	pathology duration	age (years) (Mean § SD)	Sex
				(mean § SD),
	1. Huang and	SC	n=17,VR(9),CT(9)	CT(7.87 §7.07),IVR:	IVR(59.48 §15.02),CT	CT(F:2,M:7),IVR
	Chen 2020 47			(9.69 §3.5)	(55.36 §10.48)	(F:1,M:8)
	2. Mekbib et al.	SC	n=23,VR(12),CT(11) IVR(1.23 §0.73),CT	IVR(52.17 §13.26),CT	IVR:(M:9,F:3),
	2021 52 Og€ un et al. 3. €	SC	(39.36 §18.08) n=84,VR(42),CT(42) IVR:(1.32 §0.25),CT:	(61.00 § 7.69) IVR:(61.5 §10.9) CT:	CT(M:8,F:3) IVR:(M:28,F:5)
	2019 17			(0.51 §0.32)	(59.8 § 8.1)	CT:(M:23,F:9)
	Abbreviations: IVR: Immersive Virtual Reality; C: control, CT: Conventional Therapy, mo: months

Table 3 .

 3 NIVR vs C Participant Information.

	1. 2. 3. 4. 5. 6. 7. 8. 9.	Study Adie et al. 2017 41 Aşkın et al. 2018 23 Bin Song et al. 2015 42 Choi, Shin, and Bang 2019 43 Da Silva Ribeiro et al. 44 SC Study type SC SC SC SC H.-C. Lee et al. 2017 SC Henrique et al. 2019 46 SC Ikbali Afsar et al. SC In, Lee, and Song 2016 49 SC	Participants VR(1.91 §1.61),CT(1.88 §1.67) pathology duration. (mo) (mean § SD) VR(20.27 §5.47),CT(19.40 §4.48) VR(14.8 §6.1), CT(14.3 §3.4) n=36,VR(12),CT(12),C(12) CT(28.91 §15.80), VR(26.33 §15.51),C Population n=235,VR(117),CT(118) n=40,VR(20),CT(20) n=40,VR(20),CT(20). (29.00 §19.21) n=30,VR(15), CT(15) VR(42.1 §26.9),CT(60.4 §44.1), N=50,VR(26), CT(24) VR(28.00 §23.97), CT(21.77 § 19.66) n=31,VR(16) CT(15) CT(17.07 § 10.00),VR (15.63 § 6.60) n = 35,VR(19), CT(16) VR(2.94 §1.88),CT(2.29 §1.31) n=25,VR(13), CT (12) VR(12.5 §4.1), CT(13.6 §5.3)	age (yrs) (mean § SD) VR(66.8 §14.6),CT (68.0 §11.9) VR(53.3 §11.2), CT(56.6 § 9.9) VR(51.37 §40.6),CT (50.10 §7.83) CT(58.00 §5.15),VR (59.58 §11.87), (59.33 § 13.63) VR(53.7 §6.1), CT(52.8 § 8.6) VR(59.4 §8.95), CT(55.8 § 9.6) CT(76.20 §10.41),VR (76.19 §10.09) VR(69.4 §8.6),CT (63.4 § 15.7) VR(57.3 §10.5),CT	Sex VR(F:51,M:66),CT (F:53, M:65) VR(F:5,M:13),CT(F:6, M:14) VR(M:10,F:10),CT(M:12, F:8) CT:(M:7,F:5),VR(M:7, F:5),C(M:9, F:3) VR(M:06,F:09),CT(M:05, F:10) VR(M:16,F:10), CT (M:18, F:3) CT(M:7,F:8),VR (M:7, F:9) VR(F:7,M:12,CT(F:8, M:8) VR(M:8,F:5),CT (M:7,	COMPUTER-MEDIATED THERAPIES FOR STROKE REHABILITATION
						(54.4 §11.4)	F:5)
	10. Kiper et al. 2018 50	SC	(N=136),VR(68), CT(68)	VR(52.8.4 §33.6),CT(49.2 §38.4)	VR(62.5 §15.2), CT	VR(M:37,F:31), CT
						(66.0 §12.9)	(M:43, F:25)
	11. Kong et al. 2016 27	SC	N=105,VR(35), CT (35),	VR(0.47 §0.30),CT(0.47 § 0.31), C	VR(58.1 §9.1),CT(59.0 §	VR(M:27,F:9), CT(M:25,
				C: (35)	(0.44 § 0.29)	13.6),C(55.8 § 11.5)	F:10) C(M:25 F:10)
	12. Llorens et al. 2015 51	SC	n=20,CT(10), VR (10)	CT(19.59 §7.4),VR (13.58 §7.75)	CT(55.0 §11.6),VR (58.3	CT(M:5,F:5), VR(M:4,
						§11.6)	F:6)
	13	McNulty et al. 2015 30	SC	n=41,VR(21) CT (20)	VR(11¢0 §3¢1),CT(6¢5 §1)	VR(59¢9 §13¢8) CT(56¢1	VR(F:8,M:13), CT(F:2,
						§ 17¢0)	M:18)
	14. Pedreira da Fonseca	SC	n=27,VR(14), CT(13)	VR(44.1 §25.0),CT(64.5 §41.9)	VR(53.8 §6.3), CT(50.9 §	VR(F:10,M:4),CT(F:9,
		et al.				10.9)	M:4)
		2017 29
	15. Saposnik et al.2016 8	MC	n=141,VR(71),CT(70)	VR(102 §16¢8)CT(102 §19¢2)	VR(62 §13)CT(62 §12)	VR(M:46,F:25), CT
							(M:48, F:31)
	16. Schuster-Amft et al. 2014 24	MC	n=42,VR(22),CT(32)	VR(28.8 §28.8),CT(43.2 §44.4)	VR(61.3 §13.4), (CT	VR(F:6,M:16),CT (F:9,
						61.2 §11.2)	M: 23)
							(Continued)

Table 4 .

 4 IVR Interventions, Comparator, and Technology

		Study	Intervention	Comparator	VR Technology
	1.	Huang and Chen 2020 47	30 mins IVR games + 60 mins CT including a	90 mins CT of Upper limb training using a	HTC VIVE HMD, Controllers
			Climbing bar, Ball bearing, and Pulley for	Climbing bar, Ball bearing, and Pulley for
			20 sessions. Tot. 30 hrs	20 sessions. Tot. 30 hrs.
	2.	Mekbib et al. 2021 52	60min IVR activities like reaching, grasping,	120 min CT of daily living activities, balance	HTC VIVE HMD + Leap Motion
			and releasing tasks + 60min CT for 8 ses-	control, gait training, weight shift, and distal
			sions. Tot 16 hrs.	and proximal UE
				functional movements for 8 sessions, Tot 16
				hrs.
	3.	€ Og€ un et al. 2019 17	60 min IVR activities to facilitate hand	60 min CT of upper extremity exercises com-	Leap Motion, HMD
			motions, stimulating forearm supination and	prising the same tasks as used in the IVR
			pronation, flexion, and abduction 18 ses-	group.+ 15min Sham IVR for 18 sessions.
			sions. Tot: 18 hrs.	Tot:18 hrs.
	Abbreviations: IVR-Immersive Virtual Reality, CT-Conventional Therapy, HMD-Head Mounted display

Table 5 .

 5 NIVR Interventions, Comparator, Technology.

	COMPUTER-MEDIATED THERAPIES FOR STROKE REHABILITATION
	Study	Intervention	Comparator	VR Technology
	1. Adie et al. 2017 41	15 min warm-up + 45 min Wii	15 min warm-up + 45 min CT	Nintendo Wii
		VR per day; Tot: 17hrs	per day. Tot: 17 hrs
	2. Aşkın et al. 2018 23	20 sessions of CT + VR games	20 sessions of CT activities to	Xbox Kinect, TV screen,
		that required the upper extrem-	improve the active range of	laptop
		ity use, Tot 20 hrs.	motion, strength, flexibility,
			transfers, posture, balance,
			coordination, and activities of
			daily living. Tot 20 hrs.
	3. Bin Song et al. 2015 42 VR 30 min/session, Tot 20 hrs	Ergometer bicycle training	Xbox Kinect
		(Kinect games for body bal-	30 min/session, Tot 20 hrs
		ance and limb motion)
	4. Choi, Shin, and Bang	30 min CT + VR mirror therapy	30 min CT + 30min Conven-	Leap motion, a monitor, a
	2019 43	30 min of lifting the arms,	tional Mirror Therapy: 15 ses-	mirror
		moving the arms to the left and	sions. Tot 15 hrs.
		right, bending
		and stretching the elbows, rais-
		ing and lowering the hands,
		lifting the wrists, lowering the
		wrists, flexing the wrists
		inward,
		flexing the wrist, and finger
		gripping,15 sessions. Tot 15
		hrs.
	5. Da Silva Ribeiro et al.	10 min stretching of UL, LL, and	10-min stretching + 50min CT of	Nintendo Wii, projector
	2015 44	trunk muscles. + 50-min of VR	trunk activities, active or
		games. 8 sessions Tot 16 hrs.	active-assisted diagonal move-
			ment of the Lower Limbs, bal-
			ance training, stationary and
			side gait, anteroposterior and
			laterolateral movements, gait
			training. 8 sessions. Tot 16 hrs.
	6. H.-C. Lee et al. 2017 45 45 min CT + 45 min VR balance	CT for 90 min focusing on	television, Microsoft
		games based on common bal-	strengthening, endurance	Kinect + commercial
		ance problems experienced	training, ambulation, and ADL	game
		after stroke. for 12 sessions Tot	training. for 12 sessions. Tot 18
		18 hrs	hrs.
	7. Henrique et al. 2019 46 VR exergame 30 mins for 24	CT 30 mins exercise similar to	Motion Rehab 3D, projec-
		sessions weeks, Tot 12 hrs.	VR group, for 24 sessions, Tot	tor, Kinect, PC
		exergame for upper limb motor	12 hrs.
		function and balance rehabili-
		tation of stroke survivors,
		including flexion exercises,
		shoulder abduction and
		adduction, horizontal shoulder
		abduction and adduction,
		elbow extension, wrist exten-
		sion, knee flexion, hip flexion,
		and abduction
	8. Ikbali Afsar et al.	CT 60 min + VR 30 min pro-	CT 60 min consisting of static	Microsoft Xbox Kinect
	2018 48	grams for active movements of	and dynamic control of posi-	system, TV screen
		the upper extremity, bilateral	tion, balance skills, weight
		shoulder abduction, and adduc-	shift, and activities of daily liv-
		tion, and active elbow flexion	ing for 20 sessions. Tot 20 hrs.
		and extension movements, per-
		formed flexion and extension
				(Continued)

Table 5 (

 5 Continued)

	Study	Intervention	Comparator	VR Technology
	15. Saposnik et al. 2016 8	VR 60 min with the goals of	recreational therapy, 10 sessions,	Nintendo Wii
		enhancing flexibility, range of	60 min Tot 10 hrs.
		motion, strength, and
		coordination of the affected
		arm 10 sessions, Tot 10 hrs.
	16. Schuster-Amft et al.	VR 45-min for use of the arm	45-min CT which included neu-	PC, gloves.
	2014 24	and/or hand movements, mir-	romuscular interventions, body
		roring of the real movements of	structural interventions
		one	perceptual and sensory inter-
		arm and/or hand and following	ventions, 16 sessions.
		the movements of one arm	Tot:12hrs.
		and/or hand.,for 16 sessions
		Tot: 12hrs.
	17. Shin et al. 2016 54	VR 30min + 30min OT for	60 min OT (20 sessions) Tot:	smart glove
		movements of the distal upper	20hrs.
		extremity such as the forearm	Same categories of movements
		pronation/supination, wrist	of the distal upper extremity as
		flexion/extension, wrist radial/	those in the VR group
		ulnar deviation, finger flexion/
		extension, and complex move-
		ments .20 sessions tot: 20hrs.
	18. Shin, Bog Park, and			Depth sensor, 3D aware-
	Ho Jang 2015 25			ness sensors, infrared
				projectors, and image
				sensors.
	19. Zondervan et al.	1hr VR games of self-guided	1hr CT of self-guided therapy of	Music Glove, laptop
	2016 55	therapy for hand and finger	tabletop hand and finger exer-
		exercises for 9 sessions, Tot:	cises 1hr for 9 sessions, Tot:
		9hrs	9hrs

30 min CT + 30 min VR games for active arm and trunk movements and promote successful rehabilitation. 20 sessions. Tot 20 hrs. 1 hr of CT which includes a range of motion and strengthening exercises for the affected limb, table-top activities, and training for activities of daily living for 20 sessions. Tot 20 hrs. Abbreviations: VR Virtual Reality; C control, CT Conventional Therapy, hr hour

Table 6 .

 6 Study Outcomes of IVR Vs. CT

		Study	Upper limb function	Activity limitation/ ADL	Results
	1.	Huang and Chen 2020 47	Pri:FM(IVR:0.13 §0.12,	FIM	Greater improvement in
			CT:0.05 §0.05,		VR in FM and FIM
			ES:,0.83), BBT
	2.	Mekbib et al. 2021 52	Pri:FM(IVR:12.25 §4.58,	BI	Greater improvement in
					VR on FM
	3.

CT:7.7 §2.54, ES:,1.17), € Og€ un et al. 2019 17 Pri:FM(IVR:46.54 §7.91, CT:40.06 §8.33, ES:0.79,) ARAT PASS-BADL, PASS-IADL, FIM Greater improvement in VR on FM, ARAT, FIM and PASS COMPUTER-MEDIATED THERAPIES FOR STROKE REHABILITATION

Table 7 .

 7 Study Outcomes for NIVR Vs CT

	Study

3. Bin Song et al. 2015 42 TUG(NIVR:21.9 §7.9, CT:19.5 §7.5, ES:0.31),10mWT (NIVR:21.4.2 §8.9, CT:19.1 §8.8.ES:0.26) balance ability, (NIVR:24.7 §19.01, CT:20.37 §21.34, ES:0.61)forward LOS, (NIVR:3311.7 §19.01, CT:4322.6 § 565.5 ES:0.28)Backward LOS (NIVR:1895.9 §2097.5, CT:2889.7 §2769.7, ES:0.4) Pri:FM(NIVR: 14.69 § 0.67, CT:9.07 §1.34, ES:5.22), Pri:BBS(NIVR:7.87 §8.6, CT:5.6 §0.69: ES:0.25), Greater improvement in VR on FM. 8. Ikbali Afsar et al. Pri:FM(NIVR:18.74 § 7.67,CT:13.94 § 6.58, ES:0.65), BBT,BRSarm,BRS-Hand, FIM Greater improvement in VR for BRS and BBT. Pri:FM,(NIVR:32.8 § 18.2,CT:29.2 §17.25, ES:0.2), ARAT SIS FIM Number no significant difference in all outcome measures between groups. 12.Llorens et al. 2015 51 10-mWTest. TP-OMA Pri:BBS(NIVR:3.8 §2.6, CT:1.8 §5.7,ES:,0.43), BBA greater improvement VR for BBS and 10mWT. 10mWT-10m Walking Test; ADL-Activities of Daily Life; ABC-Activities-specific Balance Confidence; AOU-Amount of use; ARAT-Action Research Arm test; AROM-active range of motion; BBA-Brunel Balance Assessment; BBS-Berg Balance Scale; BBT-Box and Blocks Test; BDI-Beck Depression Inventory; BI-Barthel Index; BRS-Brunnstrom Recovery Stages

  names[i] {Same id's as we are iterating through all points}

	5: 6: end for plan array[i] ← plan[i]
	7: Ø ← Plan and Exec to(home) {Move to home pose} 8: {Reference to home position as current} 9: init f rame ← "home 10: aux des f rame ← "home 11: while running do
	12:	if des f rame == init f rame then
	13: 14:	{The robot is in position.} else
	15:
		aux des f rame) then
	19:	execute(plan array[i])
	20: 21:	init f rame ← aux des f rame end if
	22:

aux des f rame = des f rame {Update the desired position} 16:

for i < 0 ; i < noe ; i + + do 17:

{Search in the list of plans the one that matches the init and final frames} 18:

if (start name[i] == init f rame) and (f inal name[i] ==

  hand threshold T h . Output: Nearest point P. 1: function STD (P h , P s , O s , T h ) 2:

Table 1

 1 Strategy analysis for the trajectories 2-11

			Trajectories 2-11
		Time for detection	Time for robot	Robot dist.
		Efficacy		User dist.
	Strategy	Q 1

Table 2

 2 Complete analysis for seven trajectories

				Long trajectory	Medium trajectory	Short trajectory	Analysis
		Criteria	Strategy	2-11	5-18	5-11	5-15	12-15	3-4	1 7 -16	Mean	St. dev.
	Time for detection	Q 2	A	2.0868	2.3029	2.5878	2.1767	1.9842	0.6418	1.0381	1.8312	0.6597
			B	2.0868	2.3029	2.5878	2.1767	2.3366	0.6418	1.0381	1.8815	0.6825
			C	2.0868	2.5370	2.5878	2.1767	2.3366	0.6418	1.0381	1.9150	0.7076
			D	1.9687	2.5370	2.2383	2.0428	2.3366	0.3258	0.8709	1.7600	0.7687
		Q 2norm	A	1.0600	1	1.1561	1.0655	1	1.9701	1.1919	1.2062	0.3190
			B	1.0600	1	1.1561	1.0655	1.1776	1.9701	1.1919	1.2316	0.3085
			C	1.0600	1.1016	1.1561	1.0655	1.1776	1.9701	1.1919	1.2461	0.2995
			D	1	1.1016	1	1	1.1776	1	1	1.0399	0.0663
	Time for robot	Q 3	A	2.1818	2.4212	2.6828	2.2642	2.0717	0.7506	1.1440	1.9309	0.6560
			B	2.1818	2.4212	2.6828	2.2642	2.4241	0.7506	1.1440	1.9812	0.6780
			C	2.1818	2.6846	2.6828	2.2642	2.4241	0.7506	1.1440	2.0189	0.7080
			D	2.1431	2.6846	2.3434	2.1303	2.4241	0.4345	0.9777	1.8768	0.7740
		Q 3norm	A	1.0181	1	1.1448	1.0629	1	1.7273	1.1701	1.1605	0.2399
			B	1.0181	1	1.1448	1.0629	1.1701	1.7273	1.1701	1.1848	0.2309
			C	1.0181	1.1088	1.1448	1.0629	1.1701	1.7273	1.1701	1.2003	0.2214
			D	1	1.1088	1	1	1.1701	1	1	1.0398	0.0651
	Robot dist.	Q 4norm	A	1.1959	1.4943	1.0948	1.1582	1.0225	1.0000	1.3042	1.1814	0.1601
			B	1.1959	1.4943	1.0948	1.1582	1.0225	1.0000	1.3042	1.1814	0.1601
			C	1.2810	1.6766	3.9707	1.5271	3.0161	1.0000	2.9585	2.2043	1.0273
			D	1.7904	3.9039	7.4400	2.8315	4.2133	1.0000	2.6207	3.4000	1.9464
	User dist.	Q 5	A	0.4370	0.3301	0.4846	0.4143	0.3285	0.3535	0.4429	0.3987	0.0570
			B	0.4370	0.3301	0.4846	0.4143	0.3285	0.3535	0.4429	0.3987	0.0570
			C	0.4804	0.4134	0.4932	0.4149	0.3165	0.3535	0.4684	0.4200	0.0616
			D	0.4428	0.3836	0.4199	0.4327	0.3510	0.3535	0.5044	0.4126	0.0506

Note: Optimal values are given in bold.

•

  Long trajectories: P 1 -P 6 , P 1 -P 16 , P 7 -P 16 , P 1 -P 7 and P 4 -P 16 ; • Medium trajectories: P 1 -P 13 , P 7 -P 15 , P 12 -P 13 and P 8 -P 12 ; • Short trajectories: P 6 -P 14 , P 6 -P 9 , P 3 -P 11 , P 13 -P 16 and P 14 -P 16 .

Table 2 .

 2 Mean and standard deviation of Q1, Q2, Q3, and Q4 for different strategies.

		Q1		Q2		Q3		Q4
	Strategy	Mean	SD	Mean	SD	Mean	SD	Mean	SD
	1	2.47	1.34	2.63	1.62	4.18	1.36	1.84	1.13
	2	2.82	1.36	0.50	0.86	4.89	2.06	0.42	0.68
	3	1.54	0.99	2.32	2.12	3.23	1.14	0.58	0.72
	(a)					(b)

Table 3 .

 3 Analysis of Q1, Q2, Q3, and Q4 for different threshold values.

	λ d	Q1		Q2		Q3		Q4
		M	SD	M	SD	M	SD	M	SD
	5 cm	1.12	0.88	2.47	2.69	2.74	1.04	0.53	0.71
	10 cm	1.22	0.79	2.06	2.33	2.98	1.06	0.53	0.66
	15 cm	1.48	0.77	2.38	2.09	3.09	0.96	0.53	0.61
	20 cm	1.86	1.00	2.65	2.17	3.58	0.92	1.24	0.78
	25 cm	2.23	1.36	2.59	2.11	3.85	1.20	1.38	0.85
	30 cm	2.31	1.32	2.47	1.93	3.97	1.21	1.59	0.99
	(a)					(b)

  • , 15• , 20 • , 25 • , 30

Table 4 .

 4 Mean and standard deviation of Q1, Q2, Q3, and Q4 for different α.

	α	Q1		Q2		Q3		Q4
		M	SD	M	SD	M	SD	M	SD
	1 •	2.19	1.07	0.65	1.01	4.23	1.69	0.41	0.69
	5 •	1.55	0.76	1.14	1.29	3.55	1.26	0.54	0.69
	10 •	1.53	1.01	1.92	1.67	3.25	1.09	0.54	0.65
	15 •	1.51	0.98	2.22	1.96	3.19	1.06	0.57	0.69
	20 •	1.51	0.98	2.22	2.06	3.18	1.10	0.54	0.69
	25 •	1.51	0.98	2.22	2.06	3.17	1.10	0.54	0.69
	30 •	1.51	0.98	2.22	2.06	3.17	1.10	0.54	0.69
	60 •	1.51	0.98	2.22	2.06	3.17	1.10	0.54	0.69
	(a)					(b)

S. MUGISHA ET AL.

Acknowledgements I appreciate Prof. Matteo Zoppi and Rezia Molfino for providing continuous support and design ideas for the project and the various the opportunities to participate in different activities related to my Acknowledgements This research is part of LobbyBot: Novel encountered type haptic devices, a French research project funded by ANR.

Acknowledgment

This work was funded under the LobbyBot project, ANR-17-CE33 [28]. The authors of the article thank the members of the project for their help in carrying out this work, Lionel Dominjon, Sandrine Wullens, Allexandre Bouchet, Javier Posselt, Anatol Lecuyer, and Victor Rodrigo Mercado Garcia.

Data Availability Statement

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request.

Chapter 2 Systematic literature review of VR use in rehabilitation Summary

This Chapter is intended to evaluate the efficacy of different forms of virtual reality (VR) treatments as either immersive virtual reality (IVR) or non-immersive virtual reality (NIVR) in comparison to conventional therapy (CT) in improving the physical and psychological status among stroke patients.

Author contributions

MS and MJ contributed to the study design, data collection, data interpretation and drafted the manuscript. MT and MZ contributed to the study design, data interpretation, and manuscript revision. RM contributed to the study design and revision of the manuscript. All authors have revised and approved the manuscript.

Declaration of Funding sources/sponsors

None

Declaration of interests

None

Chapter 3 Motion Planning Summary

This chapter describes the generation of a safe motion scheme that considers the obstacles present in a Virtual Reality (VR) environment. It explains the stages of development of the work using MoveIt software in ROS to control a UR5 industrial robot. This was done by setting up the planning group, which is conformed by the UR5 robot and a 6-faced prop and the base of the manipulator to plan feasible trajectories for it to execute within the environment. The study was based on the analysis of the material in the vehicle's interior. A person inside the cockpit is to be taken into account as an obstacle. Different software capabilities and options for path planning, along with the different ways of executing motions, were studied, and different path planning algorithms were compared to find the software that best suits the task. Furthermore, I proposed different mobility schemes for the robot to execute depending on the situation faced.

Chapter 4

Motion strategies for a haptic interface for industrial application Summary This chapter describes movement strategies for the robot to be as fast as possible in the contact zone while guaranteeing safety. This work uses the concept of predicting the user's intention through his gaze direction and the position of his dominant hand (the one touching the object). A motion generation algorithm is proposed and then applied to a UR5 robot with an HTC vive tracker system for an industrial application involving the analysis of materials in the interior of a car. From the list of interfaces used in virtual reality systems, haptic interfaces allow users to touch a virtual world with their hands. Traditionally, the user's hand moves the end-effector of a robotic arm. When there is no contact in the virtual world, the robotic arm is passive; when there is contact, the arm suppresses mobility to the user's hand in certain directions. Unfortunately, the passive mode is never completely seamless to the user. Haptic interfaces with intermittent contacts are interfaces using industrial robots that move towards the user when contact needs to be made. As the user is immersed in the virtual world via a virtual reality head mounted display (HMD), he cannot perceive the danger of a collision when he changes his area of interest in the virtual environment. The objective of this article is to describe four movement strategies for the robot to be as fast as possible on the contact zone while guaranteeing safety. This work uses the concept of predicting the user's intention through his gaze direction and the position of his dominant hand (the one touching the object) and safe-points outside the human workspace. Experiments are done and analyzed with a Pareto front with a UR5 robot, an HTC vive tracker system for an industrial application involving the analysis of materials in the interior of a car.

Vamsikrishna Guda

[DOI: 10.1115/1.4054509]

Conflict of Interest

There are no conflicts of interest.

Chapter 5 Motion strategies for a haptic interface for rehabilitation training Summary

This chapter describes the development of motion strategies for a haptic device for upper limb rehabilitation training. First, I describe the set-up and then the development of in VR exergame for reach and grab tasks designed for arm and shoulder rehabilitation. I further present more prediction strategies based on eye gaze and conduct a user study to evaluate the efficacy of each strategy, the effect of eye gaze window and hand threshold on the speed and response of a haptic device using the method of analysis of variance (ANOVA).

Algorithm 3 Strategy 3: Predictions with head gaze and threshold on eye-gaze angle. end if end if P prev ← P * 15: end function

Input: Hand position

Data Flow and System Integration

The data exchange for the above system components is shown in Figure 6. The proposed architecture describes the different interactions each system element has and provides an insight into how the instances share the information and communicate to each other.

The ROS component receives just the desired goal as an input. Later, based on this information, the move_group can generate a plan for the robot to reach the desired positions using pre-computed trajectories. Once the plan is generated, we communicated to the UR5 robot by using the " ur_modern_driver" [44]. With it, we can move the UR5 robot with ROS control and send, as an output, the current joint states of the robot for the Unity system to work with.

Experimental Setup

The UR5 Universal Robot was used to implement the system. This robot was programmed to receive a desired position and orientation from Unity software and move the prop. Participants used their right hand to touch the prop. For the training, the HTC Supplementary Materials: The following supporting information can be downloaded at: https:// www.mdpi.com/article/10.3390/s22052040/s1, A participant performing the experimental exercise.

Institutional Review Board Statement: Not applicable.

Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.

Conflicts of Interest:

The authors declare no conflict of interest.

Chapter 6 General Discussion

Summary

This chapter discusses the results obtained in the thesis. It also addresses the challenges and limitations and future proposals to advance the research project.

Summary of the thesis

This thesis focused on the development of motion strategies for a haptic interface with intermittent contacts to ensure safe human-robot interaction. In the studies conducted, two applications were considered. The first context was the industrial context for the analysis of the texture of the material of a car in the early stages of development and the second was a medical application for upper limb rehabilitation training. We proposed solutions to the most common problems of haptic devices which include cost, safety, and speed constraints. The software architecture used within the development of this thesis includes extensive use of Unity for the virtual environment with the prediction strategies implemented in C#. The Robot operating system was used as the middleware for motion planning to generate collision-free trajectories with nodes to perform tasks written in C++. Matlab software was for simulations and R software was for statistical analysis of results. Additionally, CATIA was used to design CAD models and create 3D parts for the props.

We first investigated the efficacy of VR in comparison to CT in improving physical and psychological status among stroke patients since the evidence of the clinical effectiveness of the different forms of VR (either immersive or nonimmersive) is scarce. A literature search was conducted on seven databases: