Bruno Patrão
email: bpatrao@isr.uc.pt

Paulo Menezes

Paula Castilho

Development of Mixed Reality Systems to Support Therapies

Keywords: Affective Computing, Sensing Technologies, Embodiment, Mixed Reality, and Computer Graphics

This work is focused on the development of Mixed Reality-based Systems suitable for their use in psychological therapies. Recently, on-going research about Immersive Virtual Environments has shown its applicability in psychological domains. Firstly, we intend to explore the bio signals data for emotion recognition. Secondly, we aim to develop an innovative system that allows the creation and manipulation of experiments oriented to psychological therapy. Finally, these experiments will be tested and evaluated in real scenarios by therapists with clinical patients. The results from this work will enable the assessment of the efficacy of these experiments and its improvement in order to apply in a generalized way to individuals with emotional, behavioural and psychological problems.

Introduction

Virtual reality has been establishing itself as a powerful tool for the treatment of panic disorder and anxiety. Among these disorders, virtual reality has been used, for example, for the treatment of: fear of flying, driving, heights, public speaking, storms; claustrophobia; agoraphobia; arachnophobia; social phobia; panic disorder and posttraumatic stress due to traffic accidents.

Research carried out so far, where it uses virtual reality to exposure therapy, demonstrated the potential of this technology for the treatment of various types of phobias. The most common and effective treatment of certain phobias is through gradual exposure, i.e., the patient is exposed to conditions that stimulate gradually anxiety. Initially the anxiety increases although as the treatment progresses it tends to decreases. Another form of treatment is through imagery and recall of the dangerous situations and describing the experience to the therapist [START_REF] Emmelkamp | Anxiety disorders: a practitioner's guide[END_REF].

Blascovich et al. enumerated the methodological advantages of virtual-realitybased studies for social psychological research including increased safety and control, the treatment is more efficient and it becomes easier to schedule; it is more efficient and effective for therapists realize the concerns of patients, streamlining the diagnostic and treatment processes; unlimited number of repetitions of feared situations, and the therapist has better control in the environment in which the patient is exposed [START_REF] Blascovich | Immersive virtual environment technology as a methodological tool for social psychology[END_REF]. Virtual reality can be used to induce affect in the treatment of anxiety, particularly for exposure therapy. A meta-analysis showed effects comparable to clinical in vivo exposures [START_REF] Powers | Virtual reality exposure therapy for anxiety disorders: A meta-analysis[END_REF].

For virtual reality to be effective in the treatment, it is necessary that the virtual environments are able to cause anxiety in patients, i.e. there is a need for patients to have a sense of being in fact experiencing a situation. In virtual reality, this is associated with the concepts of presence and immersion. The feeling of presence refers to the sense of involvement and excitement with the environment and the objects in it. When experiencing virtual environments, the sense of presence is a key concept, which is translated by the feeling of being present in another location or space [START_REF] Sanchez-Vives | From Presence to Consciousness through Virtual Reality[END_REF]. Regarding the immersion, it is a psychological state that is characterized by a sense of involvement in the environment. This perception is created through images, sounds, and other stimuli that allow the interaction with an environment that provides a series of stimuli and experiences to the participant. The sense of presence is vital in any experience of this kind, as it will define if the participant will perceive or not the virtual world as real. One way to achieve the sense of presence is through multisensory stimulation. By increasing the number of senses stimulated in a virtual reality, it is possible to dramatically improve the feeling of presence of a user and consequently, his/her satisfaction and memories of the experiment.

Motivation

The general aim of this work is to develop a system capable of induce, identify and classify human emotions and physiological activation. We intend to explore two complementary parts: the induction of basic emotions related to threat cues (e.g. selfdisgust, anxiety) which are consistently associated with psychopathology, and on the other side, the evaluation of physiological reaction, reflexive behaviours and identification of stimuli linked to emotion recognition. Thus, it will be possible to have a closed loop system capable of controlling the emotional state of the patient during a therapeutic process.

The first part will be based on the analysis of body motion (e.g. body language, eye tracking, facial expression) and measured bio-signals (e.g., heart rate, body temperature, skin conductance, respiratory rate). The second part will drive the development of a framework specially designed for the intended goals of therapeutic procedures, in which visual, auditory or haptic stimuli will be combined.

Finally, the developed solution will be tested and evaluated in real therapeutic scenarios, under the supervision of specialists and following the ethical and legal procedures. This work will enable the assessment of the efficacy of these technologies and how they can be improved and applied in a generalised way to individuals with psychological and emotional problems.

Cyber-Physical Systems

The recent availability of low cost immersive devices, the growing computing power of portable devices and cloud services create excellent opportunities for the application of virtual realty systems in therapeutic contexts, which is one of the areas that can have a real benefit with these systems.

The proposed tool intends to be used in therapeutic context but not only limited to it. Using the advantage of nowadays fast Internet connections it is possible to use this tool anywhere and the therapist can always remotely design or control the experience on the fly. Actually, there are two main advantages of using this kind of immersive technologies, first is that the therapist can manipulate all the environment and guide the patient through a specific situation without physically interfere, it can be applied to most of exposure based therapies. The second advantage is that the therapist can, in real time, observe the physiological responses of the patient and act in accordance to it. Furthermore, the therapist can have a virtual representation in the same virtual environment as the patient.

State of the Art

In virtual reality, participants are presented with digital (and sometimes photorealistic) images of what looks like real-world people, objects, scenes, and events, which are combined with tracking of the user's movements to allow him/her to become immersed in and interact with this artificial world as if it were real. Virtual reality allows a person to immerse himself/herself in a social situation or a scene in a first-person point of view (as opposed to viewing the scene in a third-person way), a distinction that appears to have specific neural correlations [START_REF] Ochsner | Reflecting upon feelings: An fMRI study of neural systems supporting the attribution of emotion to self and other[END_REF] [START_REF] Ruby | How would you feel versus how do you think she would feel? A neuroimaging study of perspective-taking with social emotions[END_REF]. Although virtual reality has great potential as an affect and emotion induction method, this method is, up until now, rarely used.

Rizzo et al. proposed virtual warfare scenarios to use therapeutically with soldiers with post-war trauma [START_REF] Rizzo | A virtual reality exposure therapy: Application for Iraq war military personnel with post-traumatic stress disorder: From training to toy to treatment[END_REF]. Also relevant, are studies conducted by Mel Slater et al., where they developed virtual environments for the study of neurological rehabilitation and in other psychological domains [START_REF] Pan | Socially Anxious and Confident Men Interact with a Forward Virtual Woman: An Experimental Study[END_REF].

Other notable work is Project EMMA (Engaging Media for Mental Health Applications) that examines how emotion contributes to "presence" (feeling part of, or immersed in) of a virtual environment. Here, a virtual urban park with multisensory features (e.g., sounds, sights, and different kinds of affective stimuli) is used to induce changes such as anxious, relaxed, or neutral affective states [START_REF] Riva | Affective interactions using virtual reality: The link between presence and emotions[END_REF].

The success of these immersive systems depends on the "sense of presence" level that they can induce on a participant taking part in such experiences. The sense of presence refers to the sense of "being there", in the virtual environments created by the technology. Botvinick & Cohen found that a fake body part can be incorporated into human body representation through synchronous multi-sensory stimulation on the fake and corresponding real body part, e.g. the "rubber hand illusion" [START_REF] Botvinick | Rubber hands 'feel' touch that eyes see[END_REF].

For instance, people experiencing virtual reality may have the illusion of being in a virtual place and, consequently, carry out actions as if the situation and events depicted were really happening. Accompanying these actions, we may observe physiological changes (e.g., heart rate, body temperature, skin conductance, and respiratory rate), reflexive behaviours (e.g., eye blinking or smiling at a virtual human character) and emotions arousal. As far as we know, few authors had combined virtual reality with physiological data.

Research Contributions and Innovation

The expected contributions of the current work are the analysis of body motion and bio signals data for emotion recognition and the development of an innovative framework that allows the creation and manipulation of experiments oriented to psychological therapy, to this end we have created different setups.

Figure 1 represents the diagram for the proposed system; the dashed boxes represent the on-going work. Furthermore, the therapist can manipulate some parameters of Stimuli System, such as, scene type and intensity of stimuli (input data) and receive the patient's biofeedback (output data). The first prototype was designed to elicit fear and avoidance reactions. It consists in a virtual mirror room and, at a certain point, instructing the participant to touch the mirror triggers a set of grim visual and auditory events that culminate in the severance of the avatar's hand. To construct this prototype several steps had to be followed in order to produce the necessary components that upon integration result in the intended setup. These steps are described in the next subsections.

Model Acquisition System

First we start to reconstruct the user's 3D model so he/she can have a virtual representation (Figure 2). The 3D reconstruction solves two problems by itself; in one hand is the user's height being used to place the point of view on the virtual world helping the spatial awareness, because environments are modulated in a 1:1 scale, in other hand, the user will be able to have a fastest acceptance of their virtual representation and consequently a better immersion feeling.

Bio-Signals System

It is well known that physiological signals change in response to physical activities or emotional changes. There are morphological and physiological connections between these emotional states and autonomic nervous system, including its sympathetic and parasympathetic divisions. Sympathetic (SNA) and parasympathetic nervous activity (PNA) contributes to those responses, such as, the heart rate variability, skin conductance variability or body temperature. While the PNA controls the body's response in rest, SNA is responsible for the internal response in body's fight-or-flight, meaning the reaction that occurs in response to a perceived harmful event, attack, or threat to survival. However, it is very difficult to identify a direct correlation between physiological data and a particular emotion, but possible to detect the arousal of an emotion when presenting specific contextual stimulus, i.e., it is possible to detect fear when presenting a horror situation.

The following statements describe how we will use the physiological data in our system.

Electrocardiography. Essential tool to access the Heart Rate Variability (HRV) associated with changes in the Heart Rate (HR) strongly related to stimuli activation.

Electro-Dermal Activity. Measures electrical skin conductance varying with the wet level of sweat glands. These are controlled by the sympathetic system that affects EDA once it is active, especially when an individual is anxious or stressed.

Body Temperature. Varies depending on the place where it is measured. In our system the body temperature is measured upon the skin and upper torso.

Body Acceleration. When attached to a moving body, accelerometers are capable of accurately sensing it movements. We use it in several body parts (arms, torso, head) to detect psychomotor agitation.

Breathing Activity. The process involves the movement of the diaphragm, which is expressed in movements of expanding and compressing of the rib cage and abdominal area. This produces patterns that are used by the system to identify events.

Aiming to detect emotional arousal, the signals are acquired from a set of sensors, pre-processed to remove any noise and produce the features needed by the classification process.

Stimuli System

The majority of the development of immersive systems focuses on vision since it is the most dominant sense. Thus, the head mounted displays (HMD) are considered the core technologies for this end. By consequence the essential ingredient of Virtual Reality is a tracked HMD that lets user see new views of the visual world as he/she moves his head. Wearing an HMD, the user can look around and see the simulated virtual world just like in the real world.

Furthermore, objects producing sounds or the sound of objects interacting with other objects are part of our daily experience. Humans as many animals have the ability to detect the sound source location (i.e. where the sound is coming from). The synchronicity of the sounds and object-related events and coherency between object motion seen and the corresponding sound source displacement is something that we humans are very sensible to. In other words if synchronicity and coherency fail they are immediately detected, affecting the perception and immersion in the environment. For this end we are using the HMD Oculus Rift with audio phones combined with our graphical engine, OpenAIR.

Tracking System

With a level of importance similar to the sensory stimuli systems used, the tracking system is the cornerstone that transforms a simple wearable visualisation system into an immersive system, where the user can be active. An unobtrusive tracking mechanism is required to register any head and body motion and providing the data to the computer to make the required changes in viewpoint and position.

In order to have a full body tracker we are using a RGB-D sensor (Kinect) to collect all joints orientation and map them to the 3D model skeleton. This allows the user to interact with the virtual world and have the virtual representation mimicking, synchronously his/her movements, thus improving the immersion.

Additionally, there are reflexive behaviours when experiencing virtual environments very important to collect, such as, gaze orientation, eye blinking or smiling at a virtual character. In order to get this information, we are at the moment designing a prototype with cameras tracking the lower face and eye movements to be installed on the HMD covering that entire region.

Discussion of Results and Critical View

We started to develop a small virtual room with a mirror where the participant can freely move and look around. At the mirror, his/her reflection mimics all the movements creating a good way to adapt and feel immersed in the system. After a minute of adaptation the user is asked to touch his/her own reflection and when he/her does it a guillotine falls cutting the hand and left it bleeding. At this point different participant's reactions occur in their motor and physiological systems.

With this experiment in the virtual mirror prototype, our tests showed high peaks in Electro-dermal Activity (EDA) on most participants, confirming that physiological data has a direct relation with emotional responses and stimuli reactions (Figure 3). After these preliminary results, we decided to add more scenarios, and extend our data acquisition to include other bio-signals such as Heart Rate (HR), Respiratory Rate (RR), Body Temperature (BT), and Body Movements (BM). The addition of these bio-signals will enable a more complete analysis of the human physiological responses to emotional states or activation in an attempt to establish a correlation between them. Figure 4 shows these new acquired signals in two different situations, the left image corresponds to a set of stressful situations in a haunted house and the right one to a peaceful environment.

Conclusions

Being able to inhabit a virtual world through the same perspective as in real life unlocks the possibility for creating plausible enough experiences to be used as a tool in clinical therapy. Our tests have been confirming this claim by showing that different experiences can indeed elicit different physiological responses in the participant. These responses can be used by the therapist as an additional source of information about the evolution of the patient during the therapy sessions. This leads us to believe that our system can indeed help the aid of several psychopathologies.

Fig. 1 .

 1 Fig. 1. Proposed system diagram.

Fig. 2 .

 2 Fig. 2. User 3D reconstruction.

Fig. 3 .

 3 Fig. 3. Immersive Virtual Mirror Experiment (dashed line represents the moment when guillotine falls).

Fig. 4 .

 4 Fig. 4. Stressful environment: Haunted house (left). Relaxing environment: Zen garden (right).


Pedro Morais
email: jpe.morais@campus.fct.unl.pt

Carla Quintão

Pedro Vieira

Brain-Computer Interfaces by Electrical Cortex Activity: Challenges in Creating a Cognitive System for Mobile Devices using Steady-State Visually Evoked Potentials

Keywords: BCI, EEG, SSVEP, Mobile Device, Tablet, Smartphone

The research field of Brain-Computer Interfaces (BCI) emerged in an attempt to enable communication between paralyzed patients and technology. Identifying an individual's mental state, through his brain's electric activity, a typical BCI system assigns to it a particular action in the computer. It is known that when the visual cortex is stimulated with a certain frequency, it shows activity with the same frequency. This Steady-State Visually Evoked Potential (SSVEP) activity can be used to achieve the aforementioned communication goal. In this work, we first analyze the spontaneous electrical activity of the brain, to distinguish two mental sates (concentration/meditation). Then, following an SSVEP type of approach, we divide the stimulating screen in four areas, each of which flickering at a distinct frequency. By observing the responding frequency from the occipital lobe of the subject, we can then estimate the 2 bit decision he made. We observe that such a setup is efficient for real time BCI, and can be easily integrated in mobile devices. Besides, the user is able to change voluntarily her/his decisions, interacting with the system in a natural manner.

Introduction

A biomedical cognitive control system must be able to interpret the electrical signals produced in our brain and distinguish different levels of activity. There have been various approaches taken in this area, such as event-related desynchronization and synchronization (ERD/ERS) [START_REF] Lopes Da Silva | Event-Related Desynchronization[END_REF], evoked potentials with latency of 300 ms (P300) [START_REF] Fazel-Rezai | speller for brain-computer interface[END_REF][START_REF] Sellers | BCIs that uses P300 event-related potentials[END_REF] and visual evoked potentials in stationary mode (SSVEP) [START_REF] Allison | BCIs that use steady-state visual evoked potentials or slow cortical potentials[END_REF][START_REF] Silberstein | Steady state visually evoked potential (SSVEP) topography in a graded working memory task[END_REF]. The ERD/ERS engaged in the study of alpha and beta waves, characterized by frequency between 8 Hz and 12 Hz and 12 Hz to 30 Hz, respectively, which can be observed, for example, during an imagination task motion. P300 in an evoked potential with a latency of about 300 ms, which appears after a visual or auditory stimulus that requires attention and cause some surprise. SSVEP are elicited by retinal stimulation with a signal whose frequency can vary between 3.5 Hz and 75 Hz and consist of a continuous and periodic signal detected in the visual cortex with the same frequencies [START_REF] Pastor | Human cerebral activation during steady-state visual-evoked responses[END_REF]. The publications in this area are many and varied, ranging from the presentation of tools to control different devices, to new technological solutions. Among the applications that resulted from the above described techniques, we highlight some technological advances such as: 1. Activation of a mobile robot using a BCI [START_REF] Barbosa | Activation of a mobile robot through a brain computer interface[END_REF], wherein a robot is controlled through four imagined movement (foot, tongue, left arm and right arm) using ERD/ERS; 2. Application of dried EEG sensors to mobile BCIs [START_REF] Chi | Dry and noncontact EEG sensors for mobile brain-computer interfaces[END_REF], which are placed on the hair, and exhibit very similar results to traditional sensors that use saline solution or conductive gel for electrical contact; 3. Construction of a simple communication system using SSVEP based on BCI [START_REF] Sanchez | Simple communication using a SSVEPbased BCI[END_REF] where a user gives answers like "yes", "no", "good", "bad"; 4. Creation of an online BCI using static visual evoked potential [START_REF] Liu | An online brain-computer interface using nonflashing visual evoked potentials[END_REF] where through one EEG channel the user can write a word and perform a search on Google; 5. Characterization of stimuli based on P300 amplitude [START_REF] Kleih | Motivation modulates the P300 amplitude during brain-computer interface use[END_REF], wherein the study shows that there are several factors which influence the potential analyzed, for example, the effect of motivation as possible physiological influence on the amplitude P300 and, finally; 6. Development of a mobile phone based on BCI for communication on a day-to-day [START_REF] Wang | A cell-phone-based braincomputer interface for communication in daily life[END_REF] through SSVEP stimulation.

In this context it is rather evident the importance of a transversal knowledge between neuroscience and computer science to reach new insights in BCI area, solving problems related to the acquisition, storage and retrieval of brain information, as well as creating new approaches to identify different actions on the same interface.

This work aims to answer the question: "How to develop a solution where an user can interact with mobile devices naturally changing voluntarily their mental task?". We propose a system consisting of two specific phases: First, the detection of an individual's state of concentration and second, the selection of the action to be taken through SSVEP after confirming the previous condition. To fulfill the first phase, the spontaneous electroencephalographic (EEG) signals were classified based on the traditional band frequency analysis (alpha band, between 8 to 13 Hz and beta band, between 13 and 30 Hz) [START_REF] Fisch | Fisch and Spehlmann's EEG Primer[END_REF][START_REF] Niedermeyer | Electroencephalography: Basic Principles, Clinical Applications, and Related Fields[END_REF]. In the second phase, the SSVEP permit to distinguish between at least 4 different actions that the subject would like to performed.

Relationship to Cyber-Physical Systems

A cyber-physical system (CPS) comprises the junction of computing elements with nature physical processes. This approach provides the development of more specific applications such as process control, instrumentation, medical devices and smart structures. In the coming age of internet of everything, the development of this type of system contributes to a new era of products where everyone will be connected everywhere. To reach such solution is necessary a flexible architecture with new interfaces. The research for user-friendly interfaces is increasing and one of the aims will be to replace keyboard and mouse computers for more effective means of communication. The use of touch screens, commonly available in tablets and smartphones, is a clear example of this demand. The project described in this paper presents a system that establishes the connection between machines and humans using brain-computer interfaces (BCI) / Electroencephalography (EEG). The EEG is used as a non-invasive electrophysiological monitoring approach to identifying behavior patterns of brain electrical activity, while BCIs promote a direct communication channel between the brain and an electronic device. Using brain electrical activity recorded at scalp level enables intelligent monitoring systems in real-time. The use of sensory channels becomes a new form of input in addition to providing information about the status and user intent. Using this type of information, systems can adapt dynamically contributing to the task that one wants to run. To develop new solutions in this field, it is essential to understand how the brain works and manages the information. Referring to EEG and other biological signals such as electrocardiography (ECG) and electromyography (EMG), opens up a new form of communication between humans and electronic devices. This approach presents several challenges such as the efficiency of embedded systems, the implementation of algorithms using brain electrical impulses and distribution architectures that add autonomy to the devices and increase the efficiency to the communication mechanisms. At the end, this solution must have a high degree of robustness, and should enable connection to the cloud, not confined to the local control devices.

Materials and Methods

To answer the challenges in creating this BCI solution it is used the EPOC equipment [START_REF]Emotiv -EPOC neuroheadset[END_REF] to record the EEG signals. The system comprises 14 channels (AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, AF4) and two reference electrodes located in P3 and P4 channels (Common Mode Sense active electrode and Driven Right Leg passive electrode -CMS/DRL). The device records the signals sequentially with a rate of 128 Hz, with a resolution of 14 bits per channel and have a frequency response between 0.16 and 43 Hz. In addition, it provides wireless data transmission using a frequency of 2.4 GHz, with a battery allowing for 8 hours of continuous work. One of the greatest facilities of this device is the use of saline solution electrodes instead of the common conductive gel to establish the contact between the electrodes and the scalp. The acquisition control and the complete signal classification process were performed by two software platforms: OpenVibe and EEGLAB. The OpenVibe is an open source application, multi-platform, containing various pre-programmed modules for signal processing. In parallel, the EEGlab, a Matlab toolbox for EEG and event related potentials processing, is also used, being necessary to install the plug-in acquisition BIOSIG data for reading data in GDF format (General Data Format) [START_REF] Schlogl | GDF -A general dataformat for biosignals[END_REF].

Acquisition Protocol for Classification of the Mental State

The signals from the 14 channels are recorded while the subject is seated with a straight posture with the palms on the knees, avoiding any muscular movement. A complete test lasts a total of five minutes. The subject is asked to switch from one mental state (meditation) to the other (concentration), every 30 seconds, being notified with a beep. In meditation he should get away from any thought focusing his attention on the breath which should be long and deep. For the state of concentration he should countdown of 3 in 3 from 100, while he visualizes the results.

Signal Processing and Classification

A band pass analog filter from 0.16 Hz to 85 Hz, together with a 50 Hz notch filter, is applied to the signals. Since the meditation/concentration detection states is the main purpose of this phase of the project, we focalized our analysis on the prefrontal cortex electrodes (AF3, AF4), relating with attention activities [START_REF] Rebollo | Atención y funciones ejecutivas[END_REF] and in occipital electrodes (O1, O2), where the alpha rhythm, which measure the level of arousal of the subject, is particularly intense.

We used the OpenVibe application to implement the following steps: 1. The electrodes are selected; 2. A 4 th order Butterworth digital band pass filter between the alpha band frequencies (8-12Hz) is applied to the signals; 3. The signal is sub-divided into various time windows lasting 5 seconds with an overlap of 4.9 seconds; 4. The power spectral density is computed; 5. A moving average is applied.

The identification of meditation and concentration states was performed by realtime signal acquisition using the following criteria: if the alpha band spectrum in AF3 electrode is below a certain threshold (in our arbitrary units this threshold is 20) then the state will be classified as "concentration", else, the state will be classified as "meditation". If the alpha band spectrum exceeds the value 80 the corresponding EEG epoch is considered an artifact.

Choosing Actions using SSVEP

In a second stage, after checking the "concentrate" state, it is necessary to identify an action by means of SSVEP. To fulfill this objective it is presented to the subject a screen divided into four areas, each one with an image formed by a black/white checkerboard that oscillates at a predefined frequency. The subject option is identified depending on the area/image where the subject focuses its attention.

The stimulation is made using the Psychophysics version 3, a Matlab toolbox, which allows imposing frequencies at 8.6 Hz, 10 Hz, 12 Hz and 15 Hz to the screen. The detection of the frequency emitted by the different screen areas is performed through the analysis of the power spectrum obtained from the electrodes O1 and O2, located in primary visual areas in the occipital lobe. This process allows the recognition of the action to be taken in a natural way.

In this stage the signals were filtered using a 4 th order Butterworth digital band pass filter between 8-30 Hz, which comprises the alpha and beta bands. Once again, the analysis was performed on windows during 5 seconds with an overlap of 4.9 seconds. Finally, the spectrogram of those signals is generated and the results were compared with the frequencies of the visual stimuli in order to identify the action that should be taken.

Results

Meditation/Concentration

The states identification of meditation and concentration which corresponds to the 1st phase of the project was performed by real-time signal acquisition. One example is shown next, for electrodes AF3 and O1 [Figure 1]. As expected, during the concentration periods the alpha activity decreases substantially to values that rarely exceed the value 20 (arbitrary units). It should be noted in this example that the two peaks seen in between 30.0-60.0 and 150.0-180.0 observed in O1 electrode, it is a state deconcentration unscheduled during the visual imagination of the subtraction results. Analyzing the values obtained, there is a clear distinction between the two states meditation/concentration, both on the front and occipital zone, although in the latter the difference is more pronounced.

These results demonstrate that it is possible to search and identify patterns of brain activity consistent with the classification of this information in real time.

SSVEP Results

After completing the 1st stage of the process, with the concentration state identification, it follows the 2nd phase results, corresponding to the choice of the action using SSVEP. The results obtained in the spectrogram show a clear identification of the 8.6 Hz, 10 Hz, 12 Hz and 15 Hz frequencies [Figure 2] (the ones at which the four areas of the screen flick) . It was also realized that the checkerboard images in which the oscillating frequencies were applied, should be dispersed in the screen away from each other as much as possible. Besides, it is expected that the use of preset options will provide faster responses, compared to description of the desire. Taking an example of someone who wants to quench his thirst, using a grid with the image of a glass of water as one of the default options, will save significant time compared to writing this same intention.

Conclusions

The junction of the identification of a concentrated state with SSVEP will allow the implementation of a hybrid system with high accuracy. BCI approach allows the user to change voluntarily their mental task interacting with the system naturally. This work aims to contribute to the development of an autonomous system which allows monitoring in real-time mobile devices from the electrical brain activity. Associated with this application are the interfaces that will interact with the user. The flickering stimuli using checkerboard images can be replaced with icons that identify the user's intent with a clear differentiation of the actions to be taken. At the same time it should be evaluate the disruption that may be caused by peripheral vision of a concentrated individual, in order to remove this undesirable activity from the signal. We also believe that the use of higher frequencies will provoke a decreasing in visual fatigue caused by the oscillation of the image, making the system more comfortable. The presented approach should be tested in a statistical significant number of individuals to identify more precisely the degree of accuracy of the results.

Fig. 1 .

 1 Fig. 1. -Results of a 5 minutes essay in which the subject is switching from the meditation to concentration stage. Each state lasts 30 seconds starting with meditation. The graph represents the amplitude of the alpha waves in O1 and AF3 electrodes.

Fig. 2 .

 2 Fig. 2. -Spectrogram with frequencies 8.6 Hz, 10 Hz, 12 Hz and 15 Hz identification using the electrodes O1 and O2 during the SSVEP stimulation.


Ali Abdollahi Gharbali
email: a.gharbali@campus.fct.unl.pt

José Manuel Fonseca

Shirin Najdi
email: s.najdi@campus.fct.unl.pt

Tohid Yousefi Rezaii

Automatic EOG and EMG Artifact Removal Method for Sleep Stage Classification

Keywords: Sleep stage classification, Wavelet packet, Adaptive filtering, Artifact removing, Artifact detection

In this paper, a new algorithm is proposed for artifact removing of sleep electroencephalogram (EEG) with application in sleep stage classification. Rather than other works which used artificial noise, in this study real EEG data contaminated with electro-oculogram (EOG) and electromyogram (EMG) are used for evaluating the proposed artifact removal algorithm's efficiency using classification accuracy. The artifact detection is performed by thresholding the EEG-EOG and EEG-EMG cross correlation coefficients. Then, the segments considered contaminated are denoised by normalized least-mean squares (NLMS) adaptive filtering technique. Using a single EEG channel, four sleep stages consisting of Awake, Stage1 + REM, Stage 2 and Slow Wave Stage (SWS) are classified. A wavelet packet (WP) based feature set together with artificial neural network (ANN) are deployed for sleep stage classification purpose. Simulation results show that artifact removed EEG allows a classification accuracy improvement of around 14%.

Introduction

Sleep is an essential part of a human's daily life that significantly affects his/her health, productivity, mental and moral states. Therefore, the diagnosis of sleep related disorders is of great importance in sleep research. Sleep scoring with the use of polysomnographic (PSG) recordings is one of the principal requirements of diagnosis procedure of sleep disorders. PSG recordings include electroencephalogram (EEG), electromyogram (EMG), electro-oculogram (EOG), electrocardiogram (ECG), snoring and other physiological signals to detect body movements. Currently sleep scoring is mainly done by a domain expert who visually analyses PSG data, diagnoses and prescribes treatment. However, this process is time consuming, tedious and highly subjective. Therefore, several algorithms for automatic sleep scoring have been proposed to assist the expert and increase the reliability of the results [START_REF] Niedermeyer | Electroencephalography: Basic Principles, Clinical Applications, and Related Fields[END_REF].

EEG is widely adopted for the automatic detection of sleep stages and neuronal activity evaluation during sleep. However, EEG is usually contaminated with several artifacts such as power line noise, EMG, EOG and electrode movements. Removal or attenuation of the noise and unwanted signals is a prerequisite for most of the EEG signal processing applications. The presence of artifacts makes the EEG analysis difficult, since it may introduce spikes that can be confused with original EEG trend decreasing the reliability of the subsequent processing stages.

The basics for the artifact removing are diverse and are closely related to the specific application in which the algorithm is going to be used. A commonly used method for avoiding artifacts is rejecting the contaminated segments of the recorded EEG [START_REF] Devuyst | Automatic Processing of EEG-EOG-EMG Artifacts in Sleep Stage Classification[END_REF]. This method although simple, results in huge data loss. Instead, denoising the contaminated EEG segments would not only preserve the amount of data, but also would probably contribute to the increase of accuracy in the automatic sleep stage classification [START_REF] Croft | EOG correction: A comparison of four methods[END_REF]. Following this idea, an extensive number of studies have tried to extract the clean EEG out of the contaminated recording in different research areas but still no optimal method is agreed upon [START_REF] Sweeney | Artifact removal in physiological signals-practices and possibilities[END_REF].

Considering the fact that nowadays portable devices for patient monitoring and automatic sleep stage classification could be a helpful assistance for experts on the analysis of sleep signals, the main motivation for the current work is the lack of a systematic method for automatic artifact detection and cancellation which leads to an improvement in the automatic stage classification accuracy compared to the original acquired data. In view of this gap, the following research question emerges:

What would be a suitable methodology to detect, identify and remove the EOG and EMG artifacts from EEG in EEG-based sleep analysis, so that the overall artifact level is decreased and the automatic sleep stage classification accuracy is positively influenced?

In the rest of the paper we will try to answer this question. The paper is organized as follows: Section 2 explains how the proposed automatic artifact removal system can benefit cyber-physical systems. Section 3 gives a brief overview of how EEG artifact removal problem is treated in the literature. Section 4 provides detailed description of the proposed algorithm, and simulation results. Section 5 presents the results discussion and Section 6 finalizes the paper with the conclusions and future work directions.

Relationship to Cyber-Physical Systems

Cyber-physical systems (CPS) are an emerging technology that attracted a lot of attention in recent years. CPS can be regarded as the next generation computing systems that monitor, coordinate, control and integrate communication and computation abilities to interact with physical systems [START_REF] Wang | A Secured Health Care Application Architecture for Cyber-Physical Systems[END_REF]. CPS are assumed to have a wide range of applications in several areas including defense, transportation, agriculture, energy, healthcare, etc.

In the healthcare area there are currently a limited number of CPS available (such as bedside monitors) and many of their components still work isolated. With the extensive use of CPS, all these components will be integrated to a networked closed loop system together with the human [START_REF] Philip | Cyber-Physical Systems -a Concept Map[END_REF]. Wearable health monitoring systems, that allow the subject to continue normal life while his/her vital signs are recorded and analyzed, are a good example of such solutions. In most of the cases a whole night PSG signal is necessary for diagnosing a sleep related problem. This data is usually recorded in a hospital or sleep clinic while the change of sleeping environment and noise may distract the subject and induce significant alterations in the recorded data. Medical CPS like wearable sleep monitoring systems seem to be the perfect solution for recording whole night data while the subject is saved from any change in the sleeping environment. The medical data recorded this way is the main indicator of the subject's health status and its reliability is very important for further signal processing. Therefore, denoising of biological data is one of the most important requirements for improved medical CPS [START_REF] Milenković | Wireless sensor networks for personal health monitoring: Issues and an implementation[END_REF], [START_REF] Haque | Review of Cyber-Physical System in Healthcare[END_REF]. This work is developed in such a context to support the relation between CPS and sleep monitoring.

State of the Art

EEG is mainly intended for recording cerebral activity, yet other extra electrical activities are also recorded. These extra activities are usually considered harmful artifacts that can be either physiological like EOG, EMG and ECG or extra physiological like power line interference. EOG measures are captured mainly by frontal electrodes, but they are strong enough to also affect other electrodes. About the EMG, the degree and type of contamination depends on the contracted muscle, the recording purpose and the environment [START_REF] Urigüen | EEG artifact removal-state-of-the-art and guidelines[END_REF]. Some of these artifacts are easily removed by a finite impulse response (FIR) or infinite impulse response (IIR) filter if their power spectrum doesn't have overlap with the EEG power spectrum. However, EMG and EOG, which have significant power spectrum overlap with EEG are not easy to remove requiring careful consideration [START_REF] Urigüen | EEG artifact removal-state-of-the-art and guidelines[END_REF].

The state of the art in EEG denoising is quite broad. The most classic methods used in the last years are regression (especially for ocular interferences), blind source separation or component base techniques [START_REF] Romero | Ocular Reduction in EEG Signals Based on Adaptive Filtering, Regression and Blind Source Separation[END_REF] and Wiener and Bayes filtering methods [START_REF] Sweeney | Artifact removal in physiological signals-practices and possibilities[END_REF]. Also adaptive filtering [START_REF] Sweeney | Artifact removal in physiological signals-practices and possibilities[END_REF], wavelet denoising [START_REF] Unser | A review of wavelets in biomedical applications[END_REF] and empirical mode decomposition (EMD) [START_REF] Safieddine | Removal of muscle artifact from EEG data: comparison between stochastic (ICA and CCA) and deterministic (EMD and wavelet-based) approaches[END_REF] are among the most widely used denoising techniques. A great range of studies pay particular attention to the improvement of existing methods or using more objective performance criteria [START_REF] Fatourechi | EMG and EOG artifacts in brain computer interface systems: A survey[END_REF].

In the context of sleep stage classification, Estrada et al. [START_REF] Estrada | Wavelet-based EEG denoising for automatic sleep stage classification[END_REF] proposed a denoising method based on applying a predefined threshold to the wavelet coefficients of noisy EEG with the objective of finding the best threshold defining rule and value. The noisy signal is constructed by adding white noise and 50 Hz power line sinusoidal noise. The coefficients that contribute to the noise components are zeroed out using a threshold discrimination filter. They use Mean Squared Error (MSE) and signal-to- noise ratio (SNR) as criteria for evaluating the performance of the proposed denoising scheme.

Materials and Methods

Figure 1 shows an overview of the sleep stage classification with proposed sleep EEG artifact removal scheme.

Data

In this study, PSG records of 22 Caucasian males and females with the duration of nine hours each were used from the Sleep EDF Database [START_REF]The Sleep-EDF Database[END_REF]. Except for a slight difficulty in falling asleep, subjects were healthy and without any medication. All EEG (from Fpz-Cz and Pz-Oz electrode locations), EOG (horizontal) and submental chin EMG recordings were performed with a sampling rate of 100 Hz. All PSGs are divided into 30-seconds epochs and annotated by well-trained technicians according to the Rechtschaffen and Kales (R&K) manual. In this study, Pz-Oz EEG channel, horizontal EOG and submental chin EMG recordings of all the subjects are used.

Pre-Processing

Visual inspection of the data reveals that occasionally there exist one or more consecutive epochs in which the energy of the signal is zero probably due to hardware failure. These epochs are removed in order to eliminate the confusion that they could introduce on the sleep stage classification algorithm. Following the AASM manual [START_REF]The AASM Manual for the Scoring of Sleep and Associated Events -Rules, Terminology and Technical Specifications[END_REF] recommendations for the scoring of sleep, the EEG and EOG signals are filtered with a band-pass filter with lower cut-off of 0.3 Hz and higher cut-off of 35 Hz. In this work we use WP based decomposition and reconstruction as a filter for removing unwanted frequency band of physiological data [START_REF] Wiltschko | Wavelet filtering before spike detection preserves waveform shape and enhances single-unit discrimination[END_REF].

Artifact Detection and Removal

Conventionally, it is assumed that the measured EEG is a linear combination of cerebral activity with one or more kind of artifacts. Therefore, in this paper, for detecting the EOG and EMG contamination, the filtered EEG, EOG and EMG recordings are divide into 1000-sample segments and then the cross correlation of each EEG segment is calculated with the corresponding EOG and EMG segment. If the absolute value of the EEG-EOG cross correlation coefficients or EEG-EMG cross correlation coefficients is more than threshold 1 or threshold 2 respectively, the corresponding segment will be fed to an artifact removal block which is based on NLMS adaptive filtering. Adaptive filtering [START_REF] Haykin | Adaptive Filter Theory[END_REF] has been extensively used in EEG artifact removal algorithms. It uses a recorded reference of the artifact (in our case horizontal EOG and submental chin EMG) to adjust a vector of weights that models the contamination according to an optimization algorithm.

If the thresholding conditions for cross correlation coefficients are not satisfied, the relevant EEG segment will be copied to the output without any change.

Feature Extraction

In order to perform sleep stage classification, the output of the pre-processing block is used for feature extraction. There are five types of main brain waves that can be distinguished by their frequency range. These frequency bands are called Delta (0-3.99 Hz), Theta (4-7.99 Hz), Alpha (8-13 Hz) and Beta (>13 Hz) [START_REF]The AASM Manual for the Scoring of Sleep and Associated Events -Rules, Terminology and Technical Specifications[END_REF]. Therefore, EEG is conventionally analyzed in the frequency domain. Moreover, EEG is a nonstationary signal and simultaneous time-frequency analysis can be quite useful. In this study, a WP tree with 7 decomposition levels and Daubechies order 2 (db2) mother wavelet is used for feature extraction. Different frequency bands of EEG including Delta, Theta, Alpha, spindle, Beta1 and Beta 2 are extracted according to the scheme proposed in [START_REF] Ebrahimi | Automatic sleep stage classification based on EEG signals by using neural networks and wavelet packet coefficients[END_REF]. The following statistical features are calculated for each epoch using the WP coefficients:

1) Energy of the WP coefficients for each frequency band (F1-F6) 2) Total Energy (F7) 3) Mean of the absolute values of WP coefficients for all frequency bands (F8) 4) Standard deviation of WP coefficients for all frequency bands (F9) 5) Energy ratio of various frequency bands (F10 to F14) F10 is the ratio between the energy in the Alpha band and the sum of the energy in the Delta and Theta bands. F11 is the ratio between the energy in the Delta band and the sum of the energy in the Alpha and Theta bands. F12 is the ratio between the energy in the Theta band and the sum of the energy in the Alpha and Delta bands. F13 is the ratio between the energy in the Alpha band and the energy in the Theta band and F14 is the ratio between the energy in the Delta band and the energy in the Theta band.

Normalization

Features are normalized to standardize their range. Since the range of values of raw EEG varies broadly, to avoid that features with larger numeric values dominate those with smaller numeric values affecting the accuracy of the classification technique, each feature ( ij x ) is independently normalized by applying the following equation:

ij i ij x x σ - ′ = i x x (1)
where i

x and σ i x are the mean and the standard deviation of each independent feature vector i x .

Classification

In this study ANN were used for the classification of sleep stages. The two-layer feed forward network consisting of 14 input neurons, 12 hidden neurons and 4 output neurons for discrimination between the four sleep stages Wake, REM+S1, S2 and SWS is used. A sigmoid transfer function in the hidden layer and a linear transfer function in the output layer were selected. Levenberg-Marquardt training algorithm is chosen.

Experimental Results and Discussion

The performance of the proposed method was assessed using the six subjects selected from the dataset mentioned in section 4.1. In our experiment for wavelet packet based filtering Daubechies order 20 (db20) was used as mother wavelet. In the artifact detection stage a threshold of 0.5 (Threshold 1) for EEG-EOG cross correlation coefficients and 0.25 (Threshold 2) for EEG-EMG cross correlation coefficients were selected. These thresholds are selected empirically considering highest classification accuracy. Three different result validation approaches including subjective and objective methods were applied.

The cross correlation coefficients for EEG-EOG and EEG-EMG which were detected by thresholding before and after applying the artifact removal algorithm are shown in Figure 2. A significant reduction in the correlation coefficients is noticeable after artifact removal.

Figure 3 and 4 illustrate the cancellation of EOG and EMG artifacts from contaminated EEG segments. It can be seen that the artifacts can be correctly eliminated without distorting the original EEG.

After the completion of the artifact removal stage, the data is fed to feature extraction algorithm. For training the ANN, unlike the more conventional approaches in the literature, which import all the existing stages to the neural network, we used a quantity of training data to be selected out of each patient. This method is suitable for large databases helping on the reduction of the computational complexity of the classifier training stage. In order to assess the effectiveness of our artifact removal algorithm, we studied the sleep stage classification accuracy for raw (after removing zero energy epochs), filtered and artifact removed data. Table 1 shows the results of statistical analysis for comparison of each stage and overall accuracy for all the above-mentioned data. The results are validated using repeated random sub-sampling method which is also known as Monte Carlo cross-validation technique. It is observed that there is an improvement in the performance of the classifier after filtering the data but the best performance is achieved by applying the proposed artifact removal algorithm.

Conclusions and Future Work

To the best of our knowledge there is a lack of EEG artifact removing studies in the sleep stage classification context that simultaneously removes the EOG and EMG artifact from EEG without rejecting epochs and also evaluates the performance of the classifier for de-noised data. This paper is a contribution in this regard. In conclusion our findings show that the proposed method for artifact cancelation is reliable for sleep stage classification giving a promising accuracy improvement.

Due to the database available at the moment for this research, EOG and EMG channels were utilized for cancellation of their contamination in EEG. The performance of proposed method can be improved in the future using other PSG channels like ECG.

Fig. 1 .

 1 Fig. 1. Flowchart of the proposed algorithm.

Fig. 2 .

 2 Fig. 2. Absolute Value of cross correlation coefficients, (a) EEG-EOG before artifact removal, (b) EEG-EOG after artifact removal, (c) EEG-EMG before artifact removal, (d) EEG-EMG after artifact removal algorithm.

Fig. 3 .

 3 Fig. 3. EOG artifact cancelation from contaminated EEG.

Fig. 4 .

 4 Fig. 4. EMG artifact cancelation from contaminated EEG.

Table 1 .

 1 Results of the statistical analysis for comparison of each stage and overall accuracy.

		Wake	REM + S1	S2	SWS	Overall
		(%)	(%)	(%)	(%)	(%)
	Raw	77.56	87.08	74.67	78.11	63.70
	Filtered	79.44	78.75	83.26	90.74	70.60
	Proposed method	87.08	87.25	87.38	90.93	77.80

Acknowledgment. This work was partially funded by FCT Strategic Program UID/EEA/00066/203 of UNINOVA, CTS.


João Lourenço
email: jp.lourenco@campus.fct.unl.pt

Leonardo Martins
email: l.martins@uninova.pt

Rui Almeida
email: rui.almeida@ngns-is.com

Claudia Quaresma

Pedro Vieira

Low Cost Inertial Measurement Unit for Motion Capture in Biomedical Applications

Keywords: motion capture, sensor fusion, cyber-physical systems, home rehabilitation, work risk analysis

A low-cost inertial measurement unit has been developed for accurate motion capture, allowing real-time spatial position registration (linear and angular) of the user's whole-body. For this, we implemented a dedicated circuit for 9 degrees of freedom motion sensors, composed of an accelerometer, gyroscope and a magnetometer. We also applied signal processing and data fusion algorithms to prevent the inherent drift of the position signal. This drift is known to exist during the sensor integration process and the implemented algorithms showed promising results. This system is meant to be used in two specific biomedical applications. The first one is linked to the development of a low-cost system for gait analysis of the whole-body, which can be used in home-based rehabilitation systems. The second application is related to the realtime analysis of working postures and the identification of ergonomic risk factors for musculoskeletal disorders.

Introduction

Pervasive and ubiquitous environments have been spreading in our society, especially focusing on tracking user's location and activities by monitoring physiological signals, motion and orientation.

There are two main cyber-physical approaches to this field, using two different sensorial components, the first is based on computer vision devices for human motion capture, such as using single or a network of cameras, infrared cameras or other optical devices [START_REF] Moeslund | A survey of advances in vision-based human motion capture and analysis[END_REF], [START_REF] Corke | An introduction to inertial and visual sensing[END_REF], while the task of human movement analysis has also been influenced by using inertial sensors, such as accelerometers and gyroscopes that are present in common smartphones or by using dedicated sensors connected to microcontrollers (which can also communicate to smartphones or other computation devices) [START_REF] Corke | An introduction to inertial and visual sensing[END_REF], [START_REF] Yang | A Review of Accelerometry-Based Wearable Motion Detectors for Physical Activity Monitoring[END_REF].

The remote analysis of the user's position and body posture can provide major benefits in the healthcare and manufacturing industries. Our studies will focus on the real-time analysis of two main problems. The first problem will analyse the position of each sensor to enable the gait analysis of the whole-body, which can in turn be used in low-cost and home-based rehabilitation systems. The second will use pattern recognition techniques to analyse static working postures, taking into account potential ergonomic risk factors of musculoskeletal disorders caused by poor work situations. From these open problems, there is a main research question that arises:

"How to design a low-cost system that is capable of accurate detection in real-time of the user's whole-body?"

In this paper we propose a system that answers this question (a relation of this work to Cyber-Physical Systems is presented in section 2). Section 3 gives a more detailed overview of the state of the art and the related literature to inertial measurement units and the associated applications of such systems. Section 4 provides a summarized description of our proposed physical system and the associated algorithms of signal processing, the 9 degrees of freedom data fusion and the implementation of a real-time data transmission protocol. The initial laboratory results are presented in Section 5, while Section 6 gives a summarized conclusion of our working progress and a preview of our future work plan.

Research Connection with Cyber-Physical Systems

Cyber-Physical Systems (CPS) have been emerging as the next computing revolution because they integrate the available communication and computational capacities to create new interactions between cyber and physical components [START_REF] Geisberger | Cyber-Physical Systems -Driving force for innovation in mobility, health, energy and production[END_REF]. These systems are merging the physical and virtual world and several applications have been identified in communication, transportation, infrastructure, energy, robotics, manufacturing and healthcare [START_REF] Geisberger | Cyber-Physical Systems -Driving force for innovation in mobility, health, energy and production[END_REF], [START_REF] Khaitan | Design Techniques and Applications of Cyberphysical Systems : A Survey[END_REF].

Cyber-Physical frameworks have already been proposed for use in rehabilitation systems [START_REF] Ma | A Cyber-physical System based Framework for Motor[END_REF]. Such systems can be used for home-based physiotherapy services processes, such as the monitoring of users in senior care, patients with reduced mobility (for example with hip or knee replacement surgery), stroke and heart attack patients and other subjects that require occupational therapy to regain day to day skills.

CPS applications are already inside the healthcare system, which has been extensively reviewed [START_REF] Haque | Review of Cyber-Physical System in Healthcare[END_REF], and require a special architecture that can handle the privacy of the data, that can also handle communication between the hospital/rehabilitation centre, the storage unit and the sensors. Finally they also need to handle the computational resources that arise from a feedback system that receives real-time data from various patients and a large network of sensors and requires a real-time classification or response back to the users [START_REF] Wang | A secured health care application architecture for cyber-physical systems[END_REF].

In the manufacturing area, there have been several advances in hazard risk management and their approach to the preservation of worker's safety and health has been highly improved, by developing new work equipment features and definition of more secure tasks [START_REF] Lazaro | Integrated and Personalised Risk Management in the Sensing Enterprise[END_REF].

A big trend that has been emerging in this area is the definition of a sensing and smart factory, which uses information from a network of physical sensors and virtual databases to make real-time risk assessments [START_REF] Lazaro | Integrated and Personalised Risk Management in the Sensing Enterprise[END_REF], which has been extensively overviewed for the construction industry [START_REF] Pinto | Occupational risk assessment in construction industry -Overview and reflection[END_REF]. These sensors can also be used to monitor and track the assembly line, by using real-time information to guide the manual assembly process [START_REF] Bader | Tracking Assembly Processes and Providing Assistance in Smart Factories[END_REF]. Right now, most studies in this area use visual based motion capture frameworks for to identify workers ergonomic risk factors [START_REF] Han | A vision-based motion capture and recognition framework for behaviorbased safety management[END_REF], even adapting systems such as the Kinect TM sensor for body kinematic measurement in the workplace [START_REF] Dutta | Evaluation of the Kinect TM sensor for 3-D kinematic measurement in the workplace[END_REF]. Most of these studies lack the possibility of real-time management and will be hard to implement in a real setting, due to inherent problems in these systems: there is necessity of adaptation of the background (other objects can influence the field of view, the lightning can affect the measurements) and the user's to be completely inside the field of view of the cameras. To solve such a problem, some groups have been fusing data coming from physiological sensors (e.g. ECG, EEG, EMG) location sensors (e.g. GPS or Ultra-Wideband technology) and normal cameras to make a real-time monitor tasks in the workplace and analysis of ergonomic risks [START_REF] Cheng | Data fusion of Real-time Location Sensing and Physiological Status Monitoring for Ergonomics Analysis of Construction Workers[END_REF].

State of the Art

Motion capture sensorial systems are usually divided into two categories: optical sensors and non-optical sensors [START_REF] Corke | An introduction to inertial and visual sensing[END_REF]. Optical systems use cameras to acquire motion information of the subject being studied. Due to their scalability and high sensitivity to the motion, they are regarded as the standard on several fields such as gait analysis, film production and video game animation. However, they have some disadvantages: they are usually very expensive, do not work on every lighting environment and require special software for different tasks. On the other hand, there are the nonoptical systems, which are comprised of every other system e.g. magnetic arrays, microelectromechanical systems (MEMS)-based [START_REF] Corke | An introduction to inertial and visual sensing[END_REF].

MEMS-based inertial motion capture systems use inertial sensors (e.g. gyroscopes and accelerometers) to acquire motion information, have been subject of great interest due to constant miniaturization of the sensors, ability to work on any environment and relative low cost when compared with other systems. They do, however, also have several drawbacks, namely the lack of information about the full 3D motion and the inherent noise in some sensors.

The MEMS-based sensors have been more recently implemented for real-time human motion tracking in various physical activities, such as boxing, golf swinging, and table tennis [START_REF] Liu | Realtime human motion control with a small number of inertial sensors[END_REF], but also have been able to actually identify some simple activities using fuzzy logic, such as walking, sitting or stair climbing [START_REF] Liu | Using Accelerometers for Physical Actions[END_REF]. These human activity measurements can be useful in the identification of bad postures in the workplace and the identification of ergonomic hazard risks in the workplace.

Both issues have been tackled simultaneously in two fronts, the first is using combinations of sensors to get the most information (e.g. combining 3D accelerometers with 3D gyroscopes to get information about the acceleration and angular velocity in 3 Dimensions). The second is by using Data Fusion algorithms to mitigate the effect of the measurement noise, for example using Kalman filtering in a limited frequency band [START_REF] Ligorio | Extended Kalman Filter-Based Methods for Pose Estimation Using Visual, Inertial and Magnetic Sensors: Comparative Analysis and Performance Evaluation[END_REF] or other methodologies to filter the inherent noise over the whole frequency band of interest [START_REF] Skaloud | Detection and Filtering of Short-Term ( 1 / f γ ) Noise in Inertial Sensors[END_REF].

A final approach is by fusing both visual (both with and without using optical markers) and inertial sensors, in order to solve the intrinsic problems in each system [START_REF] Hol | Robust real-time tracking by fusing measurements from inertial and vision sensors[END_REF]. Concerning the rehabilitation procedure of gait disorders, all the approaches, such as using just inertial sensors, or optical sensors or fusing both of them have been tested and studied [START_REF] Ali | Gait disorder rehabilitation using vision and non-vision based sensors: A systematic review[END_REF], [START_REF] Zhou | Human motion tracking for rehabilitation-A survey[END_REF], [START_REF] Zheng | Position-sensing technologies for movement analysis in stroke rehabilitation[END_REF], [START_REF] Tao | Integration of Vision and Inertial Sensors for Home-based Rehabilitation[END_REF].

Materials and Methods

In this section we provide a representation of the physical sensor architecture, with the implemented sensor elements and a description of the fusion and signal processing algorithms.

Our current prototype is using the LSM9DS1, a 9 degrees of freedom (gyroscope, accelerometer, magnetometer) digital sensor unit from STMicroelectronics, communicating the sensor output to a Raspberry Pi acting as signal processor, data logger and as gateway to a local network, which allows not only the sensor to be configured remotely but also the sensor output data to be accessed by any computer on the network using a software that allows the user to watch not only the data in text format, but also a 3D Model that mimics the sensor orientation in real-time. In Figure 1 we show a simple scheme of the prototype and the experimental assembly of the prototype.

To address the noise issue described in Section 3, we implemented the Mahony's filter in Direction Cosine Matrix form in both Python and MATLAB following the algorithm in [START_REF] Premerlani | Direction cosine matrix IMU: Theory[END_REF], a detailed discussion about the problem and filtering algorithms have been extensively discussed in [START_REF] Premerlani | Direction cosine matrix IMU: Theory[END_REF][START_REF] Madgwick | Estimation of IMU and MARG orientation using a gradient descent algorithm[END_REF] respectively.

Initial Laboratory Results and Discussion

In Figure 2 we present the results of the static test in which the sensor was idle for 15 minutes with a sampling rate of 119 Hz on both the gyroscope and accelerometer and 20 Hz on the magnetometer, it shows the difference between the uncorrected signal (A) and the corrected signal (B), the initial behaviour is the filter response. Alpha, beta and gamma correspond to the relative rotation (to the initial orientation) around the x,y,z axis respectively. We can notice that although there is an offset after the drift correction, the values tend to be stable after 15 minutes. This offset is a consequence of the filters signal response and can be easily removed after a few a few minutes of its stabilization. It is important to note, that this drift issue is still a problem, even in a recent study on rehabilitation systems for gait disorders, which solved this problem by fusing the inertial sensors with information coming from an infrared camera [START_REF] Ma | Wearable Gait Analysis using Vision-aided Inertial Sensor Fusion[END_REF] Fig. 2. Difference between the uncorrected signal (A) and the corrected signal (B)

For the next experiment, we tested the sensor for rapid and slow movements to check the filter response, using a goniometer to rotate the sensor around the z axis (gamma), while trying to maintain the other axis stabilized. The rotation sequence was the following: at 2.5 s, we moved from 0º (all the angles are relative to this starting point) to 24º, and then we moved to 165º with 2 additional steps of 5º. Afterwards we move to -100º (8 s), which followed to a fast rotation to 90º (11.5 s) then to -45º, then to 150º. At 15 s, we rotated back to 0º, then a fast rotation to 160º, with a slow rotation back to 90º, then again to 160º, with a gradual slow rotation to -25º. At 36 s, we rotated to 45º, back to -110º, then to 120º and a slow rotation to -125º, finalized by a rotation to 110º.

Figure shows the respective test around the z axis (gamma) with a sampling rate of 60 Hz of both the gyroscope and accelerometer and 20 Hz on the magnetometer, (A) represents the uncorrected signal and (B) the corrected one. Although the differences are barely noticeable in gamma plot, on the alpha and beta plots there is a significant effect of the drift, showing that we can also correct this problem when the sensor is moving.

Conclusions and Future Work Plan

In this work, we implemented a low-cost cost measurement unit that been developed for accurate motion capture, by implementing a dedicated circuit with 9 degrees of freedom sensors (accelerometer, gyroscope and magnetometer), connected to a Raspberry Pi, which transmits the data to a local network, allowing access by authorized computers. We implemented signal processing and data fusion algorithms to correct the inherent drift process that occurs even when the sensor is stopped, as shown in figure 2. As future work we can use an absolute position coordinate system (e.g. north east down system) instead of the existing coordinate system.

We are planning to use this system in two specific biomedical applications, as the first one is linked to the real-time analysis of working postures and the identification of ergonomic risk factors for musculoskeletal disorders. For this we are planning to start with a network of at least 3 sensors to test the analysis in the upper limbs. We will make a feedback system that uses the ISO 11226 recommendations for the holding time for upper arm elevation in a specific posture and the acceptable shoulder, forearm and hand positions in a static posture [START_REF]International Organization for Standardization: ISO 11226 -Ergonomics-Evaluation of static working postures[END_REF]. To analyse a dynamic working posture, we will use a similar framework to the project FAST ERGO_X [START_REF] Nunes | FAST ERGO_X -a tool for ergonomic auditing and work-related musculoskeletal disorders prevention[END_REF], which is based on fuzzy rules. Instead of using camera sensors, we will be using inertial sensors, which will give the ability to make the analysis in real-time.

The second application is related to the development of a low-cost system for gait analysis of the whole-body, which can be used in home-based rehabilitation systems, we will use the same network of sensors for the upper limbs to validate them using the gold standard sensors used in rehabilitation centres.

Fig. 1 .

 1 Fig. 1. (A) Prototype schematic (B) Experimental assembly of the prototype.

Fig. 3 .

 3 Fig. 3. Difference between the uncorrected signal (A) and the corrected signal (B)

Acknowledgments

Leonardo Martins is supported by a PhD Scholarship with the reference SFRH/BD/88987/2012 and is also partially funded by FCT Strategic Program UID/EEA/00066/203 of UNINOVA, CTS, funded by the Portuguese funding institution FCT -Fundação para a Ciência e a Tecnologia. We also acknowledge the technical support of the engineers of NGNS -Ingenious Solutions.


João Quintas
email: jquintas@ipn.pt

Paulo Menezes

Jorge Dias

Auto-Adaptive Interactive Systems for Active and Assisted Living Applications

Keywords: Human-Machine Interaction, Context, Active and Assisted Living, Social Agents, Adaptive Systems

The objective of this work is of improving the efficacy, acceptance, adaptability and overall performance of Human-Machine Interaction (HMI) applications using a context-based approach. In HMI, we aim to define a general human model that may lead to principles and algorithms allowing more natural and effective interaction between humans and artificial agents. This is paramount for applications in the field of Active and Assisted Living (AAL). The challenge of user acceptance is of vital importance for future solutions, and still one of the major reasons for reluctance to adopt cyber-physical systems in this domain. Our hypothesis is that, we can overcome limitations of current interaction functionalities by integrating contextual information to improve algorithms accuracy when performing under very different conditions and to adapt interfaces and interaction patterns according user intentions and emotional states.

Introduction

There is an emerging trend focused in auto-adaptable and self-reconfiguring ambient intelligence systems in order to support smarter habitats. The associated technological challenges include active perception features, mobility in unstructured environments, understanding human actions, detect human behaviors and predict human intentions, access to large repositories of personal and social related data, adapt to changing context. In the case of social artificial agents, systems must incorporate features that allow an agent to be capable of delivering a sociable experience with the user. These can be classified as cyber-physical systems, as they are designed as a network of interacting elements with physical input and output instead of as standalone devices. These features are paramount for applications in the field of Active and Assisted Living (AAL). In AAL, the primary goal is to provide solutions that help people through ageing, by promoting active and healthy living. Part of being active and healthy includes socializing. In a vast number of cases, this activity is done in care centers or nursing homes. In such scenarios, the demand for social stimulation as part of care service aggravates the need for scarce qualified human resources. Thus, technological solutions are seen as a benefice but the challenge of adoption remains associated to user acceptance. However, most of interactive approaches rely on explicit information (e.g. direct commands) to achieve expected behaviors from the system. How can we take into account implicit information to achieve human-like interaction skills in cyber-physical systems? Our hypothesis is that, we can overcome limitations of current interaction functionalities by integrating contextual information (i.e. implicit) to improve algorithms accuracy when performing under very different conditions, to select most adequate algorithms to provide a given functionality, and to adapt interfaces and interaction patterns according user intentions and emotional states. In section II we present our contribution to the field of cyber-physical systems focusing in applications for the AAL domain. Section III describes a survey of relevant literature. In sections IV and V we present our approach and initial results. Sections VI and VII conclude the paper summarizing the major findings and stating further work.

Contribution to Cyber-Physical Systems

Our contribution aims to develop automatically adaptive cyber-physical agents that will evolve according to the end-user needs and intentions. The overall goal is improving acceptance of interactive technologies, with special focus to robotics and cyber-physical systems, by users of assistive technologies, developed to address challenges from the Active and Assisted Living (AAL) domain.

From experience, it is common that the misunderstanding of functionalities and handling of new technology often leads to rejection or even to fear if the system is not behaving as expected. Thus, the challenge of user acceptance is of vital importance for future systems and still one of the major reasons for reluctance to deploy or introduce cyber-physical agents in AAL applications. The analysis of the results of relevant research projects allows us to identify common needs, which are repeatedly requested by end-users. In summary, users expect intuitive interaction with social agents. Therefore, Natural Interaction features (e.g. gestures, speech, etc.) and adaptation to the user's general profile, specific needs and intentions, were identified as high priority requirements. In the AAL domain, these features must address the needs of users with cognitive or physical skills degradation (e.g. elderly or impaired people) and aim to compensate these limitations, hence enriching user experience and accessibility.

Taking into account this list, we conclude that interaction aspects are a central point for end-users. Therefore, we can summarize the major features according Fig. 1.

Fig. 1. Scientific and technological challenges related with HMI related features, as summarized from end-user needs

To refine reasoning and to integrate context information about a person within his/her home environment in the broadest sense possible, namely the identification of activities of daily life (ADLs) in order to provide the correct services to attain enduser goals, models must be developed that are capable 1) of using a priori knowledge, either hardcoded or from experience, and 2) of evolving in time. However, there is not available yet a satisfactory implementation that is capable of reason and learn based in of context information in a distributed system, which involves sharing contextual information between different components of the system [START_REF] Quintas | Context-based perception and understanding of human intentions[END_REF].

Related Works

The current state of the art in the domain of user interaction with cyber-physical agents is still mainly based on graphical user interfaces implemented on stationary (often wall-mounted) touch panels (or terminals), portable devices (like touch tablets or mobile phones) or by using TV-sets (often in combination with set-top-boxes) and their remote controls as front end for the elderly. The graphical user interfaces are often text or icon based. Speech input is becoming more and more stable but is still in the research and development phase since surrounding noise and sounds, speaker localization and optimum input with ambient microphones (by avoiding wearable microphones) are still a challenge and are addressed by several research projects (e.g. Companionable etc.). Recent approaches as well apply avatar technology to enhance (increase acceptance and entertainment value) and personalize user interaction [START_REF] Morandell | Avatars@home interfaceing the smart home for elderly people[END_REF]. Human-machine interaction (HMI) technologies and corresponding cognitive capabilities of agent systems have seen many developments in the last few decadessee [START_REF] Goodrich | Human-robot interaction: A survey[END_REF] for an extensive survey. Solutions for multisensory active perception and attention allocation have greatly evolved [START_REF] Ferreira | A bayesian framework for active artificial perception[END_REF], as has multimodal human emotion and dialogue analysis and human-like emotion and dialogue synthesis [START_REF] Prado | Visuo-auditory multimodal emotional structure to improve human-robot-interaction[END_REF] and also human behavior analysis [START_REF] Aliakbarpour | HMM-based Abnormal Behaviour Detection Using Heterogeneous Sensor Network[END_REF][START_REF] Quintas | Context-based understanding of interaction intentions[END_REF]. In summary, we identified that existing cyber-physical systems that have been developed recently in order to provide assistance in domestic, professional and public environments are based on closed architectures, thus being operational only in specific contexts of usage, equipment and data. Furthermore, one drawback of existing implementations is commonly that perceptual models neglect the context and the spatial relation during the perception process. Our conclusions from the literature survey point us in the direction of proposing a human-machine interaction framework that considers integration as keystone to overcome existing challenges. This framework goes beyond the state-of-the-art approaches by taking into account contextual information in the interaction process.

Research Contribution and Innovation

Our main contribution aims to create a reference framework that targets improving the current state of the art of auto-adaptive interactive systems for Active Assisted Living applications. The innovative aspect of this framework is the use of contextual information to orchestrate the behaviors of the social agent (i.e. perception and actuation features involved during interaction). This paper extends the concepts and design presented from our previous work in [START_REF] Menezes | The Role of Context Information in Human-Robot Interaction[END_REF].

Context-based Human-Machine Interaction (CB-HMI) Framework

We consider the integration of contextual information a vital aspect in HMI approaches. In our approach we consider context recognition to be the process that leads to auto-adaption of the system. We call this process the Contextual adaptation loop. As depicted in Fig. 2, this is a periodic process that operates in the background of the system, while the user is interacting with it. It plays the role of detecting changes in the context and provides this information to the agent's main execution loop (e.g. perception-action).

The execution of this process in mainly related with the Interaction inputs, Environment analysis and Interaction control components depicted in Fig. 3, which invokes the adaption of the agent's interaction behavior (i.e. changes the orchestration of Interaction outputs components in Fig. 3). This process must take into consideration restrictions or rules that may apply for different situations. This information corresponds to Context models that are learnt and stored, in the Context recognition component, for future reference. The CB-HMI framework is designed to be modular, loosely coupled and to take advantage of context models to orchestrate different NI modules and user interfaces. The goal is to provide features allowing the agent to be able to adapt its behavior according to the user's expectations and intentions, as illustrated in Fig. 3.

The components are grouped into four main categories. Interaction inputs include the components related with user interaction inputs (i.e. Speech Recognition, Gaze recognition, Gesture recognition, Touch/ Haptics recognition and Psyco-physiological recognition). These modules contribute to the overall system by perceiving the user's explicit state (i.e. intentions and emotions). Environment analysis is also related with perception mechanisms, but it contributes with implicit states. It refers mainly to what can be recognized and identified from the environment and from user habits profile (i.e. Object recognition, User identification, ADLs recognition, User behavior analysis, Time, Location). Interaction control components receive inputs from the Interaction inputs components and handle the data fusion and fission in order to update the agent's internal state and determine appropriate response (i.e. Multimodal data fusion, Multimodal data fusion, Context recognition, Behavior synthesis and Dialogue management). Interaction outputs are directly linked to the synthesis components generating the appropriate verbal and non-verbal behavior of the companion and communicate relevant information to the user via the graphical user interface (i.e. Expression synthesis, Movement synthesis) and vocal outputs (i.e. Sound synthesis).

Implementation and Results

To validate our approach we started the implementation of a cyber-physical social agent that took the form of a virtual assistant. Our objective at this stage was to create a working prototype in line with the state of the art to establish a baseline for future work. In this first version of the prototype we addressed the Main execution loop depicted in Fig. 2. The prototype included an initial implementation of some of the components listed in Fig. 3, namely: Speech recognition, Dialogue Management, Expression synthesis, Sound synthesis and Movement synthesis. The system operates on an all-in-one computer (Lenovo ThinkCentre Edge 93z All-in-One) and supports various interaction modalities. According to the preferences of the user and depending on the distance between the user and the hardware, interaction can be done either by speech (2-3 m) or via the Graphical User Interface (GUI) on the computer's touchscreen (arm length). The virtual assistant interface (Fig. 4a) closely simulates human conversational behavior through the use of synthesized voice and synchronized non-verbal behavior such as head nods, posture shifts, facial expressions and hand gestures. The interface is built on top of the Behavioral Markup Language (BML) to describe the physical realization of behaviors, such as speech and gesture and the synchronization constraints between these behaviors.

Experimental Validation

The current implementation of the system was validated in three different sites. Test trials were conducted in Switzerland, in the Netherlands and in Portugal. The criteria for selecting the participants stated that the target group should be 65+ years of age, living alone, and independent and or with light impairments related with ageing. In Table 1 we present detailed information regarding participant characteristics.

Evaluation process: In all test sites, the common approach to introduce the system to the elderly was by gathering small groups of 2 to 3 and then the elderly interact with the system individually (Fig. 4b). These sessions worked as focus group. The Think aloud method and the System Usability Score (SUS) were used as tools to assess the overall impressions and usability of the system.

Results

System Usability Scores have been calculated based on the 10 questions of the validated questionnaire for SUS. In this tool, a score between 50-70 is positive, where above 68 is considered as 'above average'. Besides, a total SUS score below 70 is problematic since this value seems to be the threshold for a "good" usability. The overall results from the trials are depicted in Fig. 5.

Discussion

In general all participants found the system very interesting and showed a lot of motivation in participation. The SUS scores, for the Swiss and Dutch users, were all above 68 and especially below 70 which means that the system is as "above average" and below the threshold for a "good usability". It seems to summarize what the participants reported and the results shown: the idea behind the system was very good but the system needs to be improved, to be more intuitive in its use. In Portugal, the overall of SUS scores were 67, which is still considered as positive but falls bellow "above average". This may be an indication that, for those users, the system requires a deeper adaptation. Evidence from the study revealed that the (cognitively impaired) older adults found it difficult to gain insights into the system's possibilities and cope with the limitations. Age related cognitive degradation, specifically age related memory changes and their effects on learning, make older adults feel mentally stressed when interacting with new technologies.

Conclusion and Further Work

The paper a framework that is being developed to improve the humanmachine interaction process within cyber-physical systems. We identified HMI as a vital aspect in the adoption of Active and Assisted Living solutions. Moreover, we identified key HMI features that are commonly referred by end-users as a must, which are still imposing relevant scientific and technical challenges. To address these challenges, we implemented a virtual agent that aims to serve as artificial and interactive companion. The current version of the system was tested in three end-users organizations, in three different European countries. The results showed that the current version of the system can reach positive scores, but it is still considered to be in the borderline of usability. Future improvement will require further development of the proposed framework.

Fig. 2 .Fig. 3 .

 23 Fig. 2. Context verification process

Fig. 4 :

 4 Fig. 4: Implemented prototype (a) and user testing (b)

Fig. 5 :

 5 Fig. 5: SUS scores results

Table 1 .

 1 Characteristics of trials participants

		Topic	Switzerland Netherlands Portugal
		Participants	N=7	N=13	N=12
	Demographics	Average Age	76,4	79,4	79,9
	Technologic experience Computers (1-5)	4	3,1	1
		Tablets (1-5)	2,6	3,8	1
	Quality of life	Satisfaction daily life 87,5%	100%	NA
		Memory (1-10)	Qualitative 6,9	NA
	Device used during trials All-in-one	4	1 (used by N) 1 (used by N)
		Tablet	3	0	0