Huatian Wang
email: hwang@lincoln.ac.uk

Qinbing Fu
email: qifu@lincoln.ac.uk

Hongxin Wang
email: howang@lincoln.ac.uk

Jigen Peng
email: jgpeng@gzhu.edu.cn

Shigang Yue
email: syue@lincoln.ac.uk

Constant Angular Velocity Regulation for Visually Guided Terrain Following

Keywords: Insect vision, Flight control, Angular velocity, Terrain following

Insects use visual cues to control their flight behaviours. By estimating the angular velocity of the visual stimuli and regulating it to a constant value, honeybees can perform a terrain following task which keeps the certain height above the undulated ground. For mimicking this behaviour in a bio-plausible computation structure, this paper presents a new angular velocity decoding model based on the honeybee's behavioural experiments. The model consists of three parts, the texture estimation layer for spatial information extraction, the motion detection layer for temporal information extraction and the decoding layer combining information from pervious layers to estimate the angular velocity. Compared to previous methods on this field, the proposed model produces responses largely independent of the spatial frequency and contrast in grating experiments. The angular velocity based control scheme is proposed to implement the model into a bee simulated by the game engine Unity. The perfect terrain following above patterned ground and successfully flying over irregular textured terrain show its potential for micro unmanned aerial vehicles' terrain following.

Introduction

The detection of visual motion has been researched for decades to understand how insects, like honeybees and locusts, use visual information to guide their flight behaviours [START_REF] Seidl | Die sehfelder und ommatidien divergenzwinkel von Arbeiterin, Königin und drohn der honigbiene (Apis mellifica)[END_REF][START_REF] Srinivasan | Honeybee navigation en route to the goal: visual flight control and odometry[END_REF][START_REF] Wang | A directionally selective small target motion detecting visual neural network in cluttered backgrounds[END_REF]. Executing a visually guided terrain following is one of the most challenging parts in visual flight control for flying insects. To accomplish this, the insect might either measure the flight speed and adjust the height accordingly, or estimate the flight height directly. However, it is hard for their tiny brains, with insufficient computation resources, to measure the flight speed and the distance to surface. So a much simpler strategy, holding the angular velocity (the ratio of forward speed and distance) constant, is taken by honeybees for this visual control task [START_REF] Franceschini | A bio-inspired flying robot sheds light on insect piloting abilities[END_REF][START_REF] Serres | Optic flow-based collision-free strategies: From insects to robots[END_REF][START_REF] Srinivasan | Visual control of honeybee flight[END_REF].

The angular velocity can be measured by the angular displacement ∆φ in a small time interval ∆t, that is ω = ∆φ ∆t . In terrain following scenario, denoting v x as the forward flight speed and d as the distance to the surface, the angular velocity of the image motion perceived by ventral part of the compound eyes can also be expressed as ω = vx d . If the forward speed is maintained by a constant forward thrust, then the flight altitude will change automatically as the distance to ground varies by regulating the angular velocity to a constant value. Problems therefore arise as how insect estimates the angular velocity and keeps it constant.

Biological experiments show that honeybees can fly in the central path of the patterned tunnel by balancing the angular velocity of the image motion on both eyes. What's more, the estimation of the angular velocity is largely independent of the spatial frequency and the contrast [START_REF] Baird | Visual control of flight speed in honeybees[END_REF][START_REF] Srinivasan | Visual control of honeybee flight[END_REF]. Further, the spike recordings also show that the responses of some descending neurons in the honeybee's central nerve cord grow as the angular velocity of the image motion increases [START_REF] Ibbotson | Evidence for velocity-tuned motion-sensitive descending neurons in the honeybee[END_REF][START_REF] Ibbotson | Neural basis of forward flight control and landing in honeybees[END_REF]. Both indicate that honeybees are capable of estimating the angular velocity. However, the neural mechanism underlying this ability is still elusive.

Due to the limitation of the computation ability of insect's tiny brain, the differential techniques, matching or feature-based approaches [START_REF] Fleet | Measurement of image velocity[END_REF] which are widely used in computer vision are not working here. Hassenstein and Richardt propose a classic elementary motion detecting model describing the mechanism of the motion sensing for animals [START_REF] Hassenstein | Systemtheoretische analyse der zeit-, reihenfolgenund vorzeichenauswertung bei der bewegungsperzeption des rüsselkäfers chlorophanus[END_REF]. The Hassenstein-Reichardt (HR) motion detector produces a much higher response when a progressive motion movement presents (see Fig. 1(a)). HR-balanced detector which has a mirror symmetrical structure with a balance parameter can detect motion from both directions [START_REF] Zanker | Speed tuning in elementary motion detectors of the correlation type[END_REF] (see Fig. 1(b)). to enhance the response of a preferred direction motion [START_REF] Hassenstein | Systemtheoretische analyse der zeit-, reihenfolgenund vorzeichenauswertung bei der bewegungsperzeption des rüsselkäfers chlorophanus[END_REF]. (b) HR-balanced detector can detect both preferred and opposite direction motions [START_REF] Zanker | Speed tuning in elementary motion detectors of the correlation type[END_REF].

An angular velocity sensor based on the HR model is designed [START_REF] Ruffier | Optic flow regulation: the key to aircraft automatic guidance[END_REF] to accomplish visual guided aircraft terrain following by Franceschini and Ruffier [START_REF] Franceschini | A bio-inspired flying robot sheds light on insect piloting abilities[END_REF][START_REF] Ruffier | Optic flow regulation in unsteady environments: A tethered mav achieves terrain following and targeted landing over a moving platform[END_REF]. However, both the HR model and the HR-balanced model are tuned for particular temporal frequency (number of gratings passed over the photoreceptor per second) rather than angular velocity [START_REF] Zanker | Speed tuning in elementary motion detectors of the correlation type[END_REF]. So the output of their sensor shows a large variance when tested by patterned ground in flight [START_REF] Ruffier | Optic flow regulation: the key to aircraft automatic guidance[END_REF]. Therefore, an angular velocity specific model is needed to improve the terrain following performance.

According to numerical calculations, Zanker et al. [START_REF] Zanker | Speed tuning in elementary motion detectors of the correlation type[END_REF] point that the ratio of two HR-balanced detectors is angular velocity tuned. Based on this idea, Cope et al. [START_REF] Cope | A model for an angular velocity-tuned motion detector accounting for deviations in the corridorcentering response of the bee[END_REF] propose C-HR model, using the ratio of two HR-balanced detectors with different temporal delays. Their model perform well especially around 100 • /s. However, this is different from the fact that the honeybees usually keep a constant angular velocity around 300 • /s [START_REF] Baird | Visual control of flight speed in honeybees[END_REF]. Riabinina and Philippides [START_REF] Riabinina | A model of visual detection of angular speed for bees[END_REF] build up the R-HR model, using a fully temporal dependent channel as the denominator to produce angular velocity tuned responses. But the response of their model slightly depend on the spatial frequency of the moving grating and the dependence increases as the angular velocity grows. Wang et al. [START_REF] Wang | A model for detection of angular velocity of image motion based on the temporal tuning of the drosophila[END_REF] propose a model based on the neural structure of Drosophila's visual system which compares the results from detectors with different sampling rates, to get a spatial independent response. Nevertheless, the independence still needs to be more significant before guiding the terrain following.

Previous mentioned models use ratio of two channels to get a angular velocity tuned response, but the spatial independence of the response is not strong enough to reproduce honeybees' flight behaviours. At the same time, the performance is affected when the angular velocity is small or large due to the division. To address this issue, this paper presents a new model using texture estimation layer to estimate the spatial frequency of the image. The proposed model preforms better by combining both texture and temporal information to decode the angular velocity of the image motion.

Methods

Angular Velocity Decoding Model

The model mainly contains 3 parts, the texture estimation part for spatial information extraction, the motion detection part for temporal information extraction and the decoding part for angular velocity estimation. The structure of the model is shown in Fig. 2 explaining how these parts are connected with each other. In the proposed model, every motion detector receives the light intensity change from the neighbouring photoreceptors. And the light intensity change is separated into ON and OFF pathways and then processed by two HR-balanced detectors. The texture information and the motion information from the average of the detectors across the whole vision field are combined. Then angular velocity is decoded from this composite information. (1) Texture estimation layer. The simulated input signals received by ventral part of the compound eye are processed by a texture estimation layer where the image contrast and the spatial frequency of the gratings are estimated by the light intensities of different locations. This is based on a hypothesis that insects can have a basic sense of complexity of the texture. And the estimation method requires only low computation ability to give the global texture information of the spatial frequency and image contrast. Here we use Michelson contrast which is defined as the following:

C = I max -I min I max + I min (1)
where the I max and I min (I max , I min ≥ 0) indicate the highest and the lowest light intensities of the input signal in vision filed.

Then in order to decrease the cost of computing, binarization of the input image is performed using the relative intensity threshold I thre = (I max -I min )/2. The spatial frequency is estimated by counting the number of boundary lines of the binary image in whole visual field. This simple method works well for sine-wave and check-board gratings in our simulations. For more complex background, boundary number also indicates the complexity to some extent.

(2) Motion detection layer. Motion detection layer mainly captures the motion information. The input image frames are first processed by the lamina layer where the light intensity change, which insects interest more than the intensity itself, are computed to get the primary information of visual motion. The output of a cell in this layer is defined by the following:

P (x, y, t) = I(x, y, t) -I(x, y, t -1) + m i=1 p i P (x, y, t -i) (2)
where P (x, y, t) corresponds to the luminance change of pixel (x,y) at time t; m denotes the maximum number of the time steps that the persistence of the luminance change can last and the persistence coefficients p i ∈ (0, 1) is defined as following equation respectively [START_REF] Yue | Collision detection in complex dynamic scenes using an lgmdbased visual neural network with feature enhancement[END_REF]:

p i = (1 + e µi ) -1 (3)
Then the luminance changes are separated into two pathways, ON pathway and OFF pathway [START_REF] Fu | Shaping the collision selectivity in a looming sensitive neuron model with parallel on and off pathways and spike frequency adaptation[END_REF][START_REF] Fu | Modeling direction selective visual neural network with on and off pathways for extracting motion cues from cluttered background[END_REF]. Specifically, the ON pathway deals with light intensity increments; whilst the OFF pathway processes brightness decrements. Denoting f + = max(0, f ) and f -= min(0, f ), then we can express the outputs of the cells in this two pathways as following:

P ON (x, y, t) = P + (x, y, t), P OF F (x, y, t) = P -(x, y, t). (4)
Considering the delay of the visual signals received by neighbouring cells, we denote D ON (x, y, t) and D OF F (x, y, t) as the output of the ON and OFF detectors for horizontal motion which are computed by following the structure of the HR-balanced detector (Fig. 1(b)). Using a pure time delay of magnitude τ , then we have the following expression:

D ON (x, y, t) =P + (x, y, t -τ ) • P + (x, y + 1, t) -αP + (x, y, t) • P + (x, y + 1, t -τ ) ( 5
)
where α is chosen from Zanker's paper [START_REF] Zanker | Speed tuning in elementary motion detectors of the correlation type[END_REF] setting as 0.25 forming a partial balanced model. And D OF F (x, y, t) can be expressed similarly. Then the output of all motion detectors are averaged to give a response containing the motion information.

(3) Angular velocity decoding layer. In order to decode angular velocity of the image motion from texture information and response of the motion detection layer, here we take only one detector for example to analyse how the response is affected by the input signals. Let S 1 , S 2 denote the input signal of photoreceptor A (left) and B (right), and S D 1 , S D 2 denote the temporal delayed signal of A and B, then according to the structure of HR-balanced detector (Fig. 1(b)), the response of the detector R 0 can be expressed as

S D 1 • S 2 -αS D 2 • S 1
, where the bar means the response is averaged over a time period to remove fluctuation caused by oscillatory input.

If the input signals are simulated using the following sinusoidal grating frames (6) with spatial period λ moving at an angular velocity ω:

I(x, y, t) = (sin( 2πω λ (t - ϕ(y -1) ω )) + 1/C)/(1/C + 1) (6)
where (x,y) denotes the location of the ommatidium, t indicates the time and C ∈ (0, 1] denotes the image contrast. Then we can get the output of each detector by [START_REF] Fu | Shaping the collision selectivity in a looming sensitive neuron model with parallel on and off pathways and spike frequency adaptation[END_REF]. It can be roughly expressed in theoretical [START_REF] Zanker | Speed tuning in elementary motion detectors of the correlation type[END_REF] as the following equation:

R 0 ≈ 1 -α (1 + C) 2 + C 2 2(1 + C) 2 [sin( 2π(ϕ -τ ω) λ ) -α sin( 2π(ϕ + τ ω) λ )]. ( 7
)
In fact, the angular velocity of the background moving is caused by the flying of the insects. Considering when a simulated bee performing terrain following, the consistency of the image motion speed in vision field helps us simplify the problem so that we can average the output signals from all ON and OFF detectors in visual field to get the final response R(ω, λ) which encodes the angular velocity.

However, it is hard to derive angular velocity directly from [START_REF] Hassenstein | Systemtheoretische analyse der zeit-, reihenfolgenund vorzeichenauswertung bei der bewegungsperzeption des rüsselkäfers chlorophanus[END_REF]. But we can decode the angular velocity information from the response R(ω, λ) using an approximation method. Though there is an inevitable fitting error, we can decrease it into an acceptable level if the fitting function is chosen well. One decoding function can be chosen as following to approximate the actual angular velocity:

ω = a * λ b * 1 + C 2 C √ R (8)
where ω denotes the decoded angular velocity, λ is the estimated spatial period and C is the estimated contrast from texture estimation layer. Parameters a * and b * can be learned by minimizing the difference from the ground truth using alternate iteration method:

(a * , b * ) = arg min a,b (ω -aλ b 1 + C 2C R(ω, λ)). (9)

Control Scheme for Automatic Terrain Following

The proposed model can be used to simulate the automatic terrain following of the honeybees by maintaining a constant angular velocity. Using AVDM, we can estimate the angular velocity in flight. By regulating it accordingly to a constant value, the altitude will change automatically even without the exact altitude or forward speed in flight. The close loop control scheme for terrain following is given in Fig 3 . For simplicity, we assume the forward flight speed is maintained the same by a proper constant forward thrust. So we only need to adjust the vertical lift according to the difference between the preset angular velocity and the estimated value by AVDM to control the altitude correspondingly. Here, the preset angular velocity is also estimated by the AVDM in the beginning phase when the vertical lift is set to the same value as the gravity and where the ground is flat. After that, when the ventral angular velocity varies caused by terrain undulates, the vertical lift controller will change the lift according to the difference ε between ventral angular velocity estimated and the preset value. If the difference ε is positive, the lift will increase and vice versa. During the terrain following approach, the vertical speed v z is relatively small, and the air resistance can be approximated as f = kv z . Then the vertical dynamics can be described using the following differential equations:

m dv z dt = T -kv z -mg (10)
T = ρ(ω est -ω set ) (11)
v z = dz dt ( 12
)
where m is the mass of the simulated bee, g is the gravity acceleration and T is the vertical lift. Given the initial conditions, then the flight trajectory can be computed step by step. In our simulation, this process can be achieved by the physics engine of Unity.

Parameter Setting

Parameters of the proposed model and the control scheme are shown in TABLE 1. Parameters are mainly tuned manually based on our empirical knowledge and stay the same in the following simulations unless particularly stated.

Table 1. Parameters of the model and the control scheme

Eq. Parameters (2) m = 10 (3) µ = 1 (5) τ = 0.08s, α = 0.25 (6) ϕ = 2 • (8) a * = 48.84, b * = 1 (10) k = 0.1, g = 9.
81 [START_REF] Riabinina | A model of visual detection of angular speed for bees[END_REF] ρ = 0.04

Experiments and Results

Within this section, we present the experiments and results. The proposed model is first tested by synthetic grating stimuli to show its spatial independence in Matlab ( c The MathWorks, Inc.). Then the model is implemented into a virtual bee using Unity ( c Unity Technologies) to simulate the automatic terrain following behaviours of honeybees.

Moving grating experiments

In the first kind of experiments, we aimed to inspect the spatial frequency independence of the proposed angular velocity decoding model. Therefore, the spatial period of the grating is chosen widely from 12 • to 72 • and the model is tested by these gratings moving at different angular velocities to get a general result. As you can see from Fig. 4, the proposed model show indeed the expected independence of spatial frequency when tested by gratings of different spatial periods. The angular velocities are well decoded with little variance except when the grating is too narrow. This is caused by the much higher temporal frequency when the angular velocity is larger than 700 • /s for grating of 12 • . Actually it does not affect the honeybee's flight in most of the cases since honeybees tend to maintain a constant angular velocity of 300 • /s [START_REF] Baird | Visual control of flight speed in honeybees[END_REF], around which our model shows pretty enough spatial independence.

Terrain following simulations

In the second kind of experiments, the proposed model is implemented in a simulated bee in terrain following simulations where the ground is covered with different textures. The flight trajectories and the ventral responses are recorded to see if the simulated bee can perform automatic terrain following by estimating the angular velocity of the image motion and regulating it to a constant value.

The simulated bee with AVDM implemented is first tested on a regular terrain covered with sinusoidal gratings. The simulated bee is released around a given height at a certain forward speed. In beginning phase, the agent is set to fly forward without changing its altitude (by setting the vertical lift equals gravity), and the preset angular velocity value is estimated using AVDM after the first few frames. Then the control scheme described in previous section starts to take over the control of the vertical lift according to the difference between the angular velocity estimated and the preset value. The result is shown in Fig. 5. From the flight trajectory, we can see the flight altitude changes automatically to keep a distance from ground using only visual information. This result is similar to the experiments on regular terrain with gratings [START_REF] Ruffier | Optic flow regulation: the key to aircraft automatic guidance[END_REF][START_REF] Ruffier | Optic flow regulation in unsteady environments: A tethered mav achieves terrain following and targeted landing over a moving platform[END_REF]. Using Unity engine, the images received by ventral camera can be processed in real time to estimate the image motion angular velocity. Neither the flight speed nor the flight altitude is necessary to perform this visual guided task.

In order to see whether the model is stable under more complex terrain, we also tested the simulated bee above an irregular mountain terrain covered with sinusoidal gratings. The result is shown in Fig. 6. The flight trajectory of the agent undulates above the mountain shape ground which indicates that the flight altitude changes automatically according to the distance to ground using only visual information. Whenever the distance is close to ground, the increasing angular velocity will trigger the controller to provide a high vertical lift to help the agent get away from ground. Further, the model is also tested using mountain terrain covered with white snow and black rock in our simulations. In this scenario, the texture information captured by ventral camera is irregular. Especially when there is too much snow in the vision field, the response will drop due to the low contrast. In order to inspect the terrain following ability in this situation, several simulations are performed (see Fig. 7). Though not all flight trajectories follow the terrain well, they all complete the flight tests without crashing. The result is obviously not robust as the terrain following using grating patterns. This is mainly caused by the lack of contrast and in accordance with the phenomenon that bees can plunge straight into calm water where low contrast provided [START_REF] Heran | Windkompensation und seitenwindkorrektur der bienen beim flug über wasser[END_REF].

Conclusion and Discussion

We proposed a bio-plausible model, the angular velocity decoding model (AVD-M), for estimating the image motion velocity to perform automatic terrain following with only visual information. The model combines both spatial and temporal information from moving frames received by ventral camera to give a relatively accurate image motion angular velocity. The response curves show large spatial independence in grating simulations which is in accordance with the biological experiments. It also provides a possible explanation of how insects' visual circuits detect image motion speed.

The close loop control scheme based on angular velocity estimation is also proposed to simulate the terrain following of honeybees in game engine Unity. And the result indicates that the model is well capable of guiding the flight course using only visual information. Similar strategy can be used in simulations of the honeybee's wall following behaviour [START_REF] Serres | A bee in the corridor: centering and wall-following[END_REF]. The control scheme proposed mainly accounts for vertical lift regulating using only ventral part of the panoramic compound eye. In fact, similar strategy can be used to control forward thrust by lateral visual information. Combining both may give a more complete control scheme which can deal with more complex visual flight tasks. The proposed model will be implemented on UAV and be tested in a real environment in the near future.

Fig. 1 .

 1 Fig.1. HR detector and HR-balanced detector. (a) HR detector uses the multiplication (M) of the delayed signal from left photoreceptor and the non-delay signal from right to enhance the response of a preferred direction motion[START_REF] Hassenstein | Systemtheoretische analyse der zeit-, reihenfolgenund vorzeichenauswertung bei der bewegungsperzeption des rüsselkäfers chlorophanus[END_REF]. (b) HR-balanced detector can detect both preferred and opposite direction motions[START_REF] Zanker | Speed tuning in elementary motion detectors of the correlation type[END_REF].

Fig. 2 .

 2 Fig.2. The proposed Angular Velocity Decoding Model. The visual information of the grating movement is received by photoreceptors. The global spatial frequency and image contrast information is estimated by texture estimation part, and the temporal information is processed by motion detectors. Angular velocity decoding layer combines both texture and temporal information to estimate the angular velocity.

Fig. 3 .

 3 Fig. 3. The AVDM-based close loop control terrain following scheme. The vertical lift controller is triggered by the difference ε between preset angular velocity ωset and the estimated angular velocity ωest.

ig. 4 .

 4 The estimated angular velocity curves from decoding under different angular velocities when tested by moving gratings of different spatial periods[START_REF] Ruffier | Optic flow regulation: the key to aircraft automatic guidance[END_REF] • , 19• , 38 • , 54 • and 72 • ).

Fig. 5 .

 5 Fig.5. The bee trajectory (blue line), terrain height (black line) and the angular velocity (red line) estimated by the ventral eye are shown in the same graph. A demo video can be found at https://youtu.be/jaYSuCJGAfc.

Fig. 6 .

 6 Fig. 6. The bee trajectory (blue line) and the angular velocity (red line) estimated by the ventral eye are shown above the mountain terrain covered with gratings. The simulated bee can adjust its flight height by regulating the angular velocity to a constant value.

Fig. 7 .

 7 Fig. 7. The simulated bee released around the given height in four times. The flight trajectories are shown above the snow mountain terrain in the same graph. Whenever the distance to ground decreases, the angular velocity increases causing the vertical lift goes up, and so does the flight altitude to help it fly over the terrain.

Acknowledgments

This research is funded by the EU HORIZON 2020 project, STEP2DYNA (grant agreement No. 691154) and ULTRACEPT (grant agreement No. 778062); the National Natural Science Foundation of China (grant agreement No. 11771347)


A Kallipolitis
email: nasskall@unipi.grmichaelgalliakis@unipi.gramenychtas@bioassist.grimaglo@unipi.gr

Emotion Analysis in Hospital Bedside Infotainment Platforms using Speeded up Robust Features

Keywords: Healthcare, Platforms, Affective, Computing, Hospital Bedside Infotainment Systems, WebRTC, Speeded Up Robust Features (SURF), Emotion Analysis

Far from the heartless aspect of bytes and bites, the field of affective computing investigates the emotional condition of human beings interacting with computers by means of sophisticated algorithms. Systems that integrate this technology in healthcare platforms allow doctors and medical staff to monitor the sentiments of their patients, while they are being treated in their private spaces. It is common knowledge that the emotional condition of patients is strongly connected to the healing process and their health. Therefore, being aware of the psychological peaks and troughs of a patient, provides the advantage of timely intervention by specialists or closely related kinsfolk. In this context, the developed approach describes an emotion analysis scheme which exploits the fast and consistent properties of the Speeded-Up Robust Features (SURF) algorithm in order to identify the existence of seven different sentiments in human faces. The whole functionality is provided as a web service for the healthcare platform during regular Web RTC video teleconference sessions between authorized medical personnel and patients. The paper discusses the technical details of the implementation and the incorporation of the proposed scheme and provides initial results of its accuracy and operation in practice.

1.

Introduction

While the relation between the psychological status of human beings and their health was acknowledged in numerous studies [START_REF] Dubois | Relationships between positive psychological constructs and health outcomes in patients with cardiovascular disease: A systematic review[END_REF][START_REF] Burger | The effects of a novel psychological attribution and emotional awareness and expression therapy for chronic musculoskeletal pain: A preliminary, uncontrolled trial[END_REF][7] in the past years, conventional medicine failed to exploit this notion. In practice, it is only recently that medical experts, in parallel with the routine treatment, are investing in the improvement of the emotional status of their patients to reinforce the effects of provided therapy. Towards the same direction, bioinformatics researchers are investigating methods to better interpret, distinguish, process and quantify sentiments from various human expressions (body posture [START_REF] Dael | Emotion Expression in Body Action and Posture[END_REF], speech [START_REF] Gunawan | A review on emotion recognition algorithms using speech analysis[END_REF], facial expression [START_REF] Ko | A Brief Review of Facial Emotion Recognition Based on Visual Information[END_REF]), all summarized in what is called Affective Computing (AC) or Artificial Emotional Intelligence. Depending on the source of the human expression, affective computing is divided in three main categories: a. Facial Emotion Recognition (FER), b. Speech Emotion Recognition (SER), c. Posture Emotion Recognition (PER). The importance of affective computing systems is highlighted by the engagement of many IT colossi (Google [START_REF]Google Cloud Vision API Homepage[END_REF], IBM [START_REF]IBM Watson Visual Recognition[END_REF], Microsoft [9]) to implement systems of realtime affective analysis of multimedia data depicting human faces and silhouettes. As far as healthcare platforms are concerned, integrating equivalent schemes in systems which bear the responsibility of monitoring and managing patients' biosignals is of great significance to the healing procedure, especially in the case of chronic diseases. In brief, the generation of positive emotions assists in keeping the patient in a stable psychological condition, which is the basis for fast and efficient treatment [START_REF] Chakhssi | The effect of positive psychology interventions on well-being and distress in clinical samples with psychiatric or somatic disorders: a systematic review and meta-analysis[END_REF], whereas negative ones have the opposite effect. Apart from the integration of affective systems in healthcare platforms, rapid development of emotional AI techniques has been reported in a wide range of areas, namely Virtual Reality, Augmented Reality, Advanced Driver Assistance and Smart Infotainment as part of a general trend leading towards the alignment with human-centered computing. In this paper, we describe the design and deployment of a FER system, incorporated in a healthcare management system as a web service to provide functionalities through the entire lifecycle of the Medical Staff -Kinsfolk -Patient interaction. Motivated by the improved results that a treatment can have when combined with the psychological management of the patient, this system will provide the ability of real time measurement and quantification of the patient's emotions for the medical staff to assess and act upon. Moreover, correlating the emotion measurements with health-related markers collected by the system may lead to important newly discovered knowledge. The remainder of this paper is structured in 6 sections, as follows: Section 2 presents the related research works, while Section 3 describes the proposed emotion analysis system architecture. Section 4 describes the system in practice and Section 5 reports the experiments conducted and the corresponding results. Finally, Section 6 concludes the paper.

Related Work

As stated earlier, the analysis, recognition and evaluation of human sentiment via pattern recognition techniques does not rely solely in the processing of facial expressions, but in the quantification of body posture and speech as well. Focusing on SER, several approaches have been proposed in the literature for the extraction of vocal features and their exploitation in forming appropriate classifying models. Methods based on the extraction of low-level features like raw pitch and energy contour [START_REF] Schuller | Hidden markov model-based speech emotion recognition[END_REF][START_REF] Nwe | Bhattacharyya distance based emotional dissimilarity measure for emotion classification[END_REF] are outperformed by high level features utilizing Deep Neural Network [START_REF] Han | Speech Emotion Recognition Using Deep Neural Network and Extreme Learning Machine[END_REF] to an extend of 20% better accuracy. PER is the least examined territory related to the field of AC. The interpretation of human emotions from body posture in an attempt to assist individuals that suffer from autism spectrum disorder is described in [START_REF] Libero | Attribution of emotions to body postures: An independent component analysis study of functional connectivity in autism[END_REF], while an approach based on theoretical frameworks investigates the correlation between patterns of body movement and emotions [START_REF] Dael | Emotion expression in body action and posture[END_REF]. On the other hand, FER methodologies vary from the exploitation of Deep Belief Networks combined with Machine Learning Data Pipeline features [START_REF] Uddin | A facial expression recognition system using robust face features from depth videos and deep learning[END_REF], the utilization of a Hierarchical Bayesian Theme Model based on the extraction of Scale Invariant Feature Transform features [START_REF] Mao | Hierarchical Bayesian Theme Models for Multipose Facial Expression Recognition[END_REF], the capitalization of Online Sequential Extreme Learning Machine method [START_REF] Cossetin | Facial expression recognition using a pairwise feature selection and classification approach[END_REF] to Stepwise Linear Discriminative Analysis with Hidden Conditional Random Fields [START_REF] Siddiqi | Human Facial Expression Recognition Using Stepwise Linear Discriminant Analysis and Hidden Conditional Random Fields[END_REF]. In addition, hybrid implementations of all the above-mentioned approaches that combine FER and SER are available in the literature to complete the puzzle of Affective computing methodologies [START_REF] Ekman | Facial expression and emotion[END_REF].

In general, affective computing has been widely deployed in the blooming field of electronic healthcare. As examples of such applications, patients' breathe is managed via emotion recognition carried out by Microsoft Kinect sensor in [START_REF] Dantcheva | Emotion facial recognition by the means of automatic video analysis[END_REF], while in [START_REF] Tivatansakul | Healthcare system focusing on emotional aspect using augmented reality: Emotion detection by facial expression[END_REF] sentiments are analyzed via a facial landmark detecting algorithm from patients suffering from Alzheimer. Another application in electronic healthcare systems is the detection of potential Parkinson patients by recognizing facial impairment when certain expressions are formed with the generation of specific emotions [START_REF] Almutiry | Automated integration of wireless biosignal collection devices for patient-centred decision-making in point-of-care systems[END_REF]. An innovative notion concerning healthcare solutions is the hospital bedside infotainment systems (HBIS). These systems are designed to enhance medical staffpatient communication and promote patients' clinical experience. Comprising internet, video, movies, radio, music, video or telephone chatting with authorized personnel or kinsfolk and biosignals monitoring in one device and connected to the Electronic Health Record (EHR), it can be proven a productive tool for healthcare ecosystems [START_REF] Dale | One Size Does Not Fit all: Design and implementation considerations when introducing touch-based infotainment systems to nursing home residents, computers helping people with special needs: 15[END_REF]. Furthermore, constant monitoring of patients can assist in the improvement of their health status and lead to early detection of potential setbacks like detection of outliers [START_REF] Fouad | Continuous Health-monitoring for early Detection of Patient by Web Telemedicine System[END_REF], poor medication adherence, changes in sleep habits. Despite the fact that HBIS and emotion analysis services exist as stand-alone cloudbased applications, the combination of the aforementioned advances in a platform is a newly breed idea with positive effects concerning the timely intervention of specialists and kinsfolk when negative emotions or depression is detected.

3 System Architecture

Overview

The FER Restful web service is built to provide functionality as an additional feature of an existing hospital bedside infotainment system and assisted living solution [25]. The target group of this system are patients who suffer from chronic diseases or are obliged to stay in rehabilitation centers for long periods due to the reduced mobility. Another group of people affected are the elders who live independently or in far regions and conduct routine medical teleconsultations with doctors and caregivers [START_REF] Panagopoulos | Utilizing a homecare platform for remote monitoring of patients with idiopathic pulmonary fibrosis[END_REF]. Although the existing system provides numerous features like the monitoring of patients' biosignals through a mobile application while conducting measurements via wearables and Bluetooth enabled devices as illustrated in Figure 1, the contribution of this paper is focused on the real-time video communication functionality through which the ability of communication with their medical experts and kinsfolk in a 24/7 basis is rendered. The FER service operates in parallel with the video communication functionality and is called upon request of the doctor. As mentioned earlier, the importance of automated analysis of facial emotion expression is high, especially to patients and elderly people whose health status is strongly connected to their psychological condition and emotion management. In reference to the FER service, it is divided in two modules: a) the face extraction module, b) the emotion recognition module. The face extraction module takes place in the web browser of the client side, while the emotion recognition module occurs on the cloud platform (server side). 4), d) resizing the face picture to 256x256 pixels (if needed), e) analysis of the face picture for emotions, f) presentation of the emotions to the medical expert during the video conference, g) storage of generated results to the patient's personal health record. The analysis of facial images and their classification in seven different sentiments (anger, disgust, happiness, neutral, sadness, surprise, fear) is accomplished by the extraction of Speed Up Robust Features (SURF) which form a k dimensional vector as a result of the Bag of Words technique to the extracted features. Given a collection of r images, an algorithm that extracts local features is utilized to create the visual vocabulary (Visual Vocabulary). In our case the Speeded Up Robust Features (SURF) algorithm [START_REF] Bay | Speeded Up Robust Features[END_REF] extracts n 64-dimensional vectors where n is the interest points which are automatically detected by the algorithm and, in turn, described by using a Fast Hessian Matrix (SURF Descriptor) in each one of the r images (Fig. 4). Upon completion of the feature extraction process from the r images, a collection of r x n 64-dimensional vectors is formed, which represent corresponding points in a 64-dimensional space. This collection is grouped using a clustering algorithm (Kmeans++ is utilized) in k groups. The centroid of each group represents the visual word, resulting in the formation of a visual vocabulary of k visual words. The process of extracting SURF features is implemented utilizing ImageJ [27], face detection is based on the OpenCV library [START_REF] Bradski | Learning OpenCV: Computer vision with the OpenCV library[END_REF], while the processes of clustering and classification are using the WEKA tool [START_REF]Weka 3, Data Mining Software in Java Homepage[END_REF]. The emotion recognition service is called during a video call (Fig. 5). A sequence of image frames (1 frame per second) is captured during the WebRTC video conference. In order to avoid additional overload on the network, the Face detection module is executed locally on the web browser. Cropping the image to a face bounding box reduces the amount of data being sent from client to server which in turn results to overall improved performance of the system. This is accomplished by the utilization of the recent implementation of OpenCV library in JavaScript, which provides the functionality of OpenCV models in the JavaScript runtime environment of web browsers. The System in Practice

The functionality of the proposed solution takes place transparently as far as the users are concerned and upon selection of the medical experts. This provides the discreet capability of monitoring and registering emotional status of the patients while performing a regular video conference 'visit' (Fig. 6). The results of FER are returned from the cloud service in JSON format (Figure 7) and consequently, visualized in the user interface.

{prediction": { angry": 0,12 disgust": 0.11, fear": 0.14, happy": 0.08, neutral": 0.20 sad": 0.12, surprise": 0.23}, status": 0} Fig. 7 Sample response of the FER Service

Testing the system in practice was performed by conducting 50 video sessions of 1minute duration. In these sessions, the client-side burden was handled by a PC with Quad core Intel Core i5-7400, while the server-side (cloud services) was deployed to an IaaS Cloud environment with two cores of Intel Xeon CPU E5-2650. Necessary internet connection for the communication along the two sides was provided by a typical 24Mbps ADSL connection. The images that are captured by the camera had a resolution was set to 640x480 pixels. Average time in milliseconds for basic operations conducted in client side and server side are measured and depicted in Table 1 and 2 respectively.

Average time allotment for the uploading of the cropped image file from client to server is 70ms. Observation of the measurements in both sides demonstrates that the most timeconsuming operation is the feature extraction from the cropped image (server side), followed by the uploading of the image file in the client side. In addition, operations in the server side are far more expensive in time than those in client side, which was expected and strategically planned for the discharge of all the computationally demanding tasks from the web browser. Further experimentation on the requirement of running face detection on the front end is presented in Table 3. The Table illustrates the produced overhead for network, browser's memory and CPU for the two scenarios, one for the image size set to 320x240 indicated as s (s for small) and the other set to 640x480 accordingly indicated with (l). When idle, image is processed in 640x480; therefore, idle for (s) scenario does not exist.

The experiment was conducted using the Mozilla Firefox browser (version 0.66), but the module also operates in Opera 58.0.3135.117 (64bit) and Google Chrome 73.0.3683.86 (64-bit) without any issues. The experiments demonstrated that memory consumption is insignificantly influenced in all scenarios, whereas large variations are evidenced in data length as expected.

Experimental Results

While the main objective of this paper is the presentation of integration of a FER web service into a homecare platform, initial results for two scenarios of the classification of the JAFFE [START_REF] Lyons | Coding Facial Expressions with Gabor Wavelets[END_REF] dataset are provided. The first scenario splits the dataset into two emotional categories (positive and negative emotions, an assumption is made that anger, fear, disgust, sadness are negative emotions) and the second scenario seven emotional categories (anger, fear, disgust, neutral, happiness, surprise, sadness-Fig. 3) are provided with the utilization of various classifiers. The procedure is conducted following the 10fold cross-validation of the whole JAFFE dataset (214 images, 256x256 pixels, grayscale). In order to discover the more efficient space representation of the training dataset, extensive testing of the Bag of Visual Word scheme was conducted. Kmeans++ method (350 clusters, 70 seeds) was selected among Kmeans, Canopy and Farthest First WEKA's implementations for its ability to better distinguish inter-class and intra-class relationships. Kmeans++ improves the initialization phase of the Kmeans clustering algorithm by selecting strategically the initial seeds [START_REF] Arthur | Proceedings of the eighteenth annual ACM-SIAM symposium on Discrete algorithms[END_REF].

Conclusion

Whereas other affective computing systems operate as stand-alone applications, this paper presents an facial emotion recognition web service, integrated in a healthcare information system for monitoring and timely management of emotional fluctuations of elders and patients with chronic diseases as part of a human-centric treatment. The value of the provided functionality to classify faces into corresponding sentiments real-time during video communication sessions is of great significance especially in cases of patients with diseases related to their psychosomatic condition. Future work will be focused in the realization of a service that can execute emotion recognition in the web browser. This feature will liberate the application from its cloud based imposed restrictions. Concerning the classification performance towards the improvement of accuracy of the current prediction model, other schemes of Bag of Words techniques will be tested in an effort to provide weighted and localized information of the Visual words. Although results are promising, further testing with the utilization of larger and Caucasian oriented labeled dataset should be performed towards more thorough evaluation of the system. Correlating emotion recognition results along with information related to the biosignals and everyday routine activities of individual patients can lead to the discovery of specific patterns and valuable knowledge to the medical community.

Fig 1 .

 1 Fig 1. Overview of the homecare platform after the integration of the FER service 3.2 The Emotion Analysis Process In general, the basic skeleton of methodologies related to FER consists of five steps: a) Preprocessing of images, b) Face's acquisition, c) Landmarks acquisition (if necessary) d) Facial Feature extraction, e) Facial Expression classification. The proposed method, specifically, comprises six steps as described in the pseudocode in Fig. 2 and as follows: a) frame extraction from the real-time streaming video, b) face detection, c) cropping of

Fig 2 .

 2 Fig 2. Emotion Analysis Process as pseudocode. Green color is indicative of the functions performed client-side, red color shows the operations performed server-side

Fig 3 .Fig. 4 (Fig. 5 .

 345 Fig 3. Jaffe Image Dataset depicting seven different emotion expressions

Fig. 6 .

 6 Fig. 6. Medical expert interface showing the FER results of the patient during a video conference

FER

Table 1

 1 Average time allotment for performing operations in the client side

	Web service		FER Operations (Server Side)
	invocation
			Visual	Image to Vector	Prediction	Image	Time
		Feature	Vocabulary	Transformation	Model	Classification	(sum)
		Extraction	Loading		Loading
			(only once)		(only once)
	Beginning of session	200ms	178ms	142ms	157ms	1ms	678ms
	All other calls	200ms	-	142ms		1ms	243ms

Table 2

 2 Time allotment for performing operations in the server side

Table 3 .

 3 Resources allotment for face detection on the web browser.

Classifier Experiments Scenario 1 (Two emotional categories) Scenario 2 (Seven emotional categories)

	Random Forest	92,01%	77,46%
	Naïve Bayes	87,79%	54,92%
	MLP	93,48%	73,23%
	Sequential Minimum Optimization	93,42%	74,64%
	Logistic	92,01%	71%
	K Star	93,42%	84,03%
	LMT	89,67%	67,43%

Table 4 .

 4 Classification accuracy results for JAFFE datasetThe accuracy of the emotion detection module is provided in Table4. A Multilayer Perceptron (learning rate:0.3, momentum rate: 0.2, epoch number: 500, threshold for number of consecutive errors: 20) reaches a 93,48% classification accuracy for the first scenario while the selection of K Star classifier (manual blend:20%, value missing replaced with average) from the Weka library, achieves the best accuracy (84,03%) for the second scenario.

Acknowledgment

This research has been co-financed by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH -CREATE -INNOVATE (SISEI: Smart Infotainment System with Emotional Intelligence, project code: T1EDK-01046).


Linfeng Sui

Xuyang Zhao

Qibin Zhao
email: qibin.zhao@riken.jp

Toshihisa Tanaka
email: tanakat@cc.tuat.ac.jp

Jianting Cao
email: cao@sit.ac.jp

Localization of Epileptic Foci by Using Convolutional Neural Network Based on iEEG

Keywords: Epileptic, Focus localization, iEEG, CNN, STFT

Epileptic focus localization is a critical factor for successful surgical therapy of resection of epileptogenic tissues. The key challenging problem of focus localization lies in the accurate classification of focal and non-focal intracranial electroencephalogram (iEEG). In this paper, we introduce a new method based on short time Fourier transform (STFT) and convolutional neural networks (CNN) to improve the classification accuracy. More specifically, STFT is employed to obtain the time-frequency spectrograms of iEEG signals, from which CNN is applied to extract features and perform classification. The time-frequency spectrograms are normalized with Z-score normalization before putting into this network. Experimental results show that our method is able to differentiate the focal from non-focal iEEG signals with an average classification accuracy of 91.8%.

Introduction

According to the World Health Organization, Epilepsy is one of the most common neurological diseases globally, approximately 50 million people worldwide suffer from it [1]. Epilepsy is a chronic disorder of the brain which is a result of excessive electrical discharges in a group of brain cells. Different parts of the brain can be the site of such discharges [1]. Recent studies have shown that up to 70% of patients can be successfully treated with anti-epileptic drugs (AEDs) [1]. For patients with drug-resistant focal epilepsy, resection of epileptogenic tissues is one of the most promising treatments in controlling epileptic seizures. Hence, it's very important to determine the seizure focus in surgical therapy. The nature that focal iEEG signal is more stationary and less random than non-focal iEEG signal enable iEEG to be used for identification of location [START_REF] Andrzejak | Nonrandomness, nonlinear dependence, and nonstationarity of electroencephalographic recordings from epilepsy patients[END_REF].

As it is not certain that symptoms will present in the EEG signal at all times, interictal iEEG from epilepsy patients are monitored or recorded in long-term. In this process, large amounts of data are generated that detection of seizure from the iEEG recordings with visual inspection by neurological experts is timeconsuming. That can cause a delay of hours or even weeks in the patient's course of treatment. Therefore, many methods of automatic detection of epileptic foci have been proposed to assist neurologists by accelerating the reading process and thereby reducing workload, such as classification of normal, interictal and seizure [START_REF] Schirrmeister | Deep learning with convolutional neural networks for eeg decoding and visualization[END_REF]. For drug-resistant focal epilepsy, automatic detection of seizure focus localization from the interictal iEEG signal is required. In order to determine the epileptic seizure focus according to the iEEG signals, it essential to extract the most discriminative features, followed by classification of features into the focal part or non-focal part.

For feature extraction, methods in common use are entropy, empirical mode decomposition (EMD) [START_REF] Itakura | Epileptic focus localization based on bivariate empirical mode decomposition and entropy[END_REF] and time-frequency analysis such as Fourier transform, wavelet transforms (WT) [START_REF] Dash | A discrete-wavelet-transform-and hidden-markovmodel-based approach for epileptic focus localization[END_REF], etc. Particularly, it is demonstrated that the timefrequency domain extracted with the aid of STFT is suitable for classifying EEG signal for epilepsy [START_REF] Kıymık | Comparison of stft and wavelet transform methods in determining epileptic seizure activity in eeg signals for realtime application[END_REF]. For classification, in recent years, automatic classification of EEG by machine learning techniques has been popular, including support vector machines (SVM) [START_REF] Itakura | Epileptic focus localization based on bivariate empirical mode decomposition and entropy[END_REF], K-Nearest-Neighbor method (KNN) [START_REF] Bhattacharyya | Tunable-q wavelet transform based multiscale entropy measure for automated classification of epileptic eeg signals[END_REF] and deep learning, such as the CNN and recurrent neural network (RNN) [START_REF] Roy | Chrononet: A deep recurrent neural network for abnormal eeg identification[END_REF]. Several computer-aided solutions based on deep learning that used the raw iEEG timeseries signal as input had been proposed to localize seizure focus in epilepsy [START_REF] Acharya | Deep convolutional neural network for the automated detection and diagnosis of seizure using eeg signals[END_REF]. However, the performance of the combination of the time-frequency domain and CNN has not been widely tested for this task.

In this paper, inspired by successes in CNN with the raw EEG time-series signal [START_REF] Acharya | Deep convolutional neural network for the automated detection and diagnosis of seizure using eeg signals[END_REF], we propose a deep learning approach for the classification of focal and non-focal iEEG signal combining time-frequency analysis and CNN, in which simple features are extracted from the time-frequency domain with the use of STFT, discriminative features are learned with convolutional layers and classification is performed with a fully connected layer.

Dataset

The iEEG signals used in this study are obtained from the publicly available Bern-Barcelona iEEG dataset provided by Andrzejak For STFT, the process is to divide a longer time signal into shorter segments of equal length and then use the Fourier transform to compute the Fourier spectrum of each shorter segment.

Given a determined signal x(t), the time-frequency domain at each time point can be obtained by the following formula (1). The Z-score normalization defined as:

ST F T {x(t)}(τ, ω) = ∞ -∞ x(t)w(t -τ )e -jωt dt (
z = x - x S ( 2
)
where x is the mean of the dataset and S is the standard deviation of the dataset.

Convolutional Neural Network (CNN)

The CNN architecture contains three different types of the layer: convolutional layer, pooling layer, and fully connected layer.

Convolutional Layer: The ultimate preprocessed STFT data is used as input to convolutional layers. In the convolutional layers, the input time-frequency spectrogram is convoluted by the learnable filter (kernel) which is a matrix, and the stride is set to control how much the filter convolves across the input time-frequency spectrogram. The output of the convolution, also known as the feature map, are obtained after additive bias and non-linear map by an activation function.

Pooling Layer: In the pooling layer, also known as the down-sampling layer, feature maps from the upper layer are down-sampled to lower the calculation complexity and prevent overfitting. There are many kinds of pooling operation, max-pooling is used in this study to obtain the maximum value of each region of the feature map and consequently reducing the number of output neurons.

Fully Connected Layer: In the fully-connected layer, all the 2D feature maps from the upper layer are represented by a one-dimensional feature vector as the input of this layer. In this study, the output is obtained by doing dot products between the feature vector and learnable weights vector, adding learnable bias and then responding to the activation function. The accuracy of the validation set across classification all ten-folds is shown in Fig. 5.

Compared with the published works record in Table 1 [START_REF] Acharya | Characterization of focal eeg signals: a review[END_REF], although our proposed method does not achieve the best in terms of classification accuracy, it is still managed to obtain 91.8% accuracy. And the advantage of this method is that it is less preprocessing for feature extraction and selection than other methods such as EMD and entropy.

  et al. at the Department of Neurology of the University of Bern [2], collected from five epilepsy patients who underwent long-term intracranial iEEG recordings. All patients suffered from long-standing drug-resistant temporal lobe epilepsy and were candidates for surgery. Signals recorded at epileptogenic zones were labeled as focal signals, otherwise, it was labeled as non-focal signals. The dataset consists of 3750 pairs of focal iEEG signals and 3750 pairs of non-focal iEEG signals, and a pair of iEEG signals from adjacent channels were recorded into each signal pair, sampled of 20 seconds at a frequency of 512 Hz. In order to guarantee to get rid of the seizure iEEG signals, the iEEG signals recorded during the seizure and three hours after the last seizure were excluded. An example of a pair of the focal and non-focal iEEG signals are shown in Fig. 1, respectively.

Fig. 1 .

 1 Fig. 1. An example of the focal and non-focal iEEG signals

Fig. 2 .

 2 Fig. 2. Proposed method of classification

  1) where w(t) is the Hann window function centered around zero. Examples of spectrogram of iEEG signals are shown in Fig. 3.

Fig. 3 .

 3 Fig. 3. STFT of focal and non-focal iEEG signals

CNN Architecture:

  The ultimate preprocessed spectrogram transformed by STFT, which is a time-frequency spectrogram of size 257×101, exploited by the CNN architecture to conduct convolution operation. The local features are extracted individually based on the local correlation among the time-frequency domain. Overall features are built by connecting the local features. The CNN architecture is proposed in Fig. 4. The network is trained by setting with five pooling layers (P1 to P5) after each convolutional layers (C1 to C5), following six fully connected layers, and output includes two neurons corresponding to the focal iEEG signal and non-focal iEEG signal. And the training process builds relationships between iEEG signals and labels. The specific training process is as follows [11]: 1. Time-frequency spectrogram of 257×101 size is convoluted by a 3×3 filter sliding in C1 with stride 1 and set 10 feature maps, which are the same size as the input to represent each input spectrogram. 2. The P1 is done pooling operation by a 3×3 filter sliding with stride 2 and set 10 feature maps to represent the output of C1, and its output size is 129×51. 3. The C2 to C5 and P2 to P5 are similar to C1 and P1, except that their sizes of input and output are decided by former, and size of the feature map increase exponentially. The final output obtained is 160 feature maps of size 9×4. 4. The fully connected layer has 160×9×4 neurons connected to the feature maps obtained from the P5, and the output layer has two neurons connected to the fully connected layer for classification. Finally, each signal is trained to correspond to one kind of label.

Fig. 4 .

 4 Fig. 4. CNN architecture in the method

Fig. 5 .

 5 Fig. 5. Accuracy of the validation set classification

Table 1 .

 1 Detection results of focal and non-focal EEG signals of published journal articles using the Bern-Barcelona EEG databaseSince manual visual inspection of iEEG is a time-consuming process, an effective classifier that automates detection of epileptic focus will have the potential to reduce delays in treatment. We propose a new recognition method for iEEGbased localization of epileptic focal based on STFT and CNN with additional preprocessing and we implement a 15-layer CNN model for automated iEEG signal classification in this paper. The results with 91.8% accuracy demonstrates that this method is effective with much efficient and fast preprocessing step for localization of focal epileptic seizure area.

	Author (Year)	Techniques proposed	Performance (accuracy in %)
	Sharma et al. (2015a) [13]	EMD, entropy, LS-SVM	87
	Deivasigamani et al. (2016) [14]	DT-CWT, mean, SD,ANFIS	99
	Das et al. (2016) [15]	EMD-DWT, entropy, KNN	89.4
	Sharma et al. (2017) [16]	TQWT, entropy, LS-SVM	95
	Gupta et al. (2017) [17]	FAWT, entropy, Kruskal-Wallis test, LS-SVM	94.41
	Bhattacharyya et al. (2017) [7]	TQWT, entropy, LS-SVM	84.67
	Sriraam et al. (2017) [18]	Statistical, frequency-based, entropy, FD, Wilcoxon test, SVM	92.15
	Arunkumar et al. (2017) [19]	Entropies, non-nested generalized exemplars	98
	5 Conclusion

Acknowledgement

This work was supported by JST CREST Grant Number JPMJCR1784, JSPS KAKENHI (Grant No. 17K00326 and 18K04178).


Dimitrios Arampatzis
email: arabatzis@ics.forth.gr

Maria Doulgeraki

Michail Giannoulis
email: giannoulis@csd.uoc.gr

Evropi Stefanidi
email: evropi@ics.forth.gr

Theodore Patkos
email: patkos@ics.forth.gr

EduBAI: An Educational Platform for Logic-based Reasoning

Keywords: Answer Set Programming, Event Calculus, Logic-based Reasoning

The field of logic-based Knowledge Representation and Reasoning has produced powerful formalisms for modeling commonsense knowledge in Artificial Intelligence. In this paper, we present EduBAI, an educational platform that helps users familiarize themselves with the main tenets of commonsense reasoning in dynamic, causal domains by means of an interactive entertaining environment. We present the design and implementation of the platform, along with the rationale of sample game tactics of diverse modeling complexity.

Introduction

Formal theories for reasoning about action and change are well-established logical theories for reasoning about the dynamics of changing worlds, contributing solutions to domains as diverse as high-level robot cognition, Ambient Intelligence, complex event detection, and others. The heterogeneous application areas of these formalisms is attracting the interest of researchers from a variety of disciplines and with diverse backgrounds. Given the quite advanced technical training needed to master logic-based reasoning, it becomes evident that the educational process of understanding both the theoretical and the practical aspects of commonsense reasoning is a challenging task.

This paper aims to facilitate the learning process of logic-based reasoning, by introducing EduBAI (Educational Basketball playing using AI), a game platform and accompanying material that can assist students and researchers having basic knowledge of the underlying formalisms in writing, executing and evaluating logic-based axiomatizations. The main contributions of this paper can be summarized as follows:

This project has received funding from the Hellenic Foundation for Research and Innovation (HFRI) and the General Secretariat for Research and Technology (GSRT), under grant agreement No 188 -We present a fully operational, interactive educational platform, that enables users to focus on improving their logic programming skills in an entertaining way, hiding technical details regarding in-game communication, visualization of outputs etc. -We offer a testbet for experimenting with a rich range of reasoning features. The game scenario involves automating the decision making of 3-player teams competing against each other in a basketball match. The gameplay gives the ability to explore aspects related to executing causality-based reasoning, geospatial reasoning, multi-player coordination, temporal reasoning, probabilistic reasoning, function optimization, among others. -We offer a set of example tactics of diverse level of modeling complexity that can be used as educational material for users interacting with the platform.

The gamification nature of our approach aims to offer a more entertaining way of introducing logic-based programming, motivated by the observation that toy problems in the form of grid settings and logic puzzles have proven to be an effective way of understanding insights of AI programming.

The rest of the paper is structured as follows. Section 2 describes in detail the game dynamics, the methodology used, as well as the possibilities for learning reasoning aspects through diverse team tactics. Section 3 presents implementation details, while Section 4 discusses related work and future directions. EduBAI is an interactive educational platform implementing a basketball game between two 3-player teams. The overall goal of the platform is to enable users model the intelligence of their team using formal languages, such as the Event Calculus and ASP, in the context of a dynamic, nondeterministic domain.

Platform Design and Methodology

The Event Calculus [START_REF] Kowalski | A logic-based calculus of events[END_REF][START_REF] Miller | Some alternative formulations of the event calculus[END_REF] is a narrative-based many-sorted firstorder language for reasoning about action and change, which explicitly represents temporal knowledge, enabling reasoning about the effects of a narrative of events along a time line. It also relies on a non-monotonic treatment of events, in the sense that by default there are no unexpected effects or event occurrences. The game dynamics of EduBAI, such as the players' actions, their effects etc, are described as Event Calculus axioms.

ASP [START_REF] Gelfond | The stable model semantics for logic programming[END_REF][START_REF] Marek | Stable models and an alternative logic programming paradigm[END_REF] is a declarative problem solving paradigm oriented towards complex combinatorial search problems. A domain is represented as a set of logical rules, whose models, called answer sets, correspond to solutions to a reasoning task, such as progression or planning. Sets of such rules, or answer set programs, come with an intuitive, well-defined semantics, having its roots in research in knowledge representation, in particular nonmonotonic reasoning. EduBAI implements a translation of Event Calculus theories into ASP rules, which are then executed by the state-of-the-art Clingo ASP reasoner 3 .

Figure 1 shows the main user interface of EduBAI, displaying a snapshot of a game, including the position and state of each player in the court, as well as informative messages describing the progress of the game. The platform is free-to-use and available online for testing and play. 4

Main Concepts, Rules and Game Mechanics

Each of the six players can move within the premises of the EduBAI basketball court, represented as a 5x7 grid. The players can perform only a limited set of actions, namely shoot, move (up, down right, left) and pass. At most one action can be performed at each timepoint by each player, except from the player holding the ball, who cannot stay inactive. Player actions are contingent on appropriate preconditions; for instance, a player can perform a pass only if she is in possession of the ball. The players cannot move beyond the boundaries of the court, but any number of players can be located in the same cell at any moment.

At each timepoint, first the attacking team decides its actions for all its players and, then, the defending team takes turn, having observed the opponent's moves. All player actions are sent to the Sports Caster Server, a component that decides non-deterministically the outcome of all actions, based on the following probabilities:

a shoot action has 95% probability of being successful if it occurs in the same cell as the basket. Distance from the basket reduces the prior probability by 17% per cell. Each opponent in the same cell reduces the probability by 25%. a pass action has 99% probability of being successful if the teammate receiving the ball is in the same cell. Distance reduces this probability by 2% per cell. Each opponent in the same cell as the person making a pass reduces the probability by 25%. a move action always succeeds.

A successful shot within 1-cell distance from the basket or less counts for 2 points, all other shots count for 3 points. Indicatively, a player shooting from medium range distance having 1 opponent has 53% success probability; 3-pointers without defense succeed 61% of the times, at best; even the longest pass without defense is rather easy (79%).

The game proceeds in rounds. At the beginning of each round, the human users decide the starting position of their players, as well as the defending and attacking tactic that their team will follow during the round; these tactics cannot change in the course of the given round, therefore once this initialization step is completed, the game proceeds without any human intervention. A round concludes if a successful shot is made or if the ball goes out of play due to an unsuccessful pass. Finally, 10 consecutive moves by the attacking team without a shot cause the attack to end and the possession of the ball to change hands.

Axiomatizing the Game Dynamics

One of the objectives of the EduBAI platform is to help users understand the main principles of causality-based reasoning. This is achieved by separating the axiomatization of the game dynamics, such as the rules and constraints, from the team tactics described in the next subsection. The platform includes a sample encoding of the game dynamics to be used as reference, which the users can choose to study, use or ignore, if wishing to define their own game environment.

The game dynamics are axiomatized using the Event Calculus formalism and cover a range of features that are commonly met in a typical dynamic domain. The formal definition of the main domain objects is needed first, such as the players, the ball, the cells etc. The Event Calculus relies on the notion of fluents to characterize the state of any time-varying property these objects may possess. Such properties, for instance, may concern the position of each player, the player holding the ball etc. Since the state of fluents changes as a result of event occurrences, users need to also define appropriate actions, as specified in the game description (Section 2.1), along with any other event considered relevant.

The domain axiomatization then follows, covering various causality features, such as the effects of events, preconditions on effects, as well as preconditions on event occurrences, triggered events caused by the occurrence of other events, concurrent event execution, causal constraints etc. The commonsense law of inertia, a key concept in causal domains, becomes also part of the axiomatization, enabling users to specify the state of fluents that remain unaffected by events. Finally, the initial state of all fluents needs to be specified, in order to complete the axiomatization of a game instance. Example of event occurrence constraint: a player cannot shot unless holding the ball : -happens(shoot(T M ), T ), not holdsAt(ball holder(T M ), T ), time(T ), player(T M ).

Table 1 shows example rules comprising part of the game environment axiomatization, expressed as Event Calculus axioms and written in the ASP syntax that is used by the Clingo reasoner. 5 For example, the "move left" action is axiomatized as two rules specifying which the new set of position coordinates should be (initiates axiom), invalidating at the same time the old coordinates (terminates axiom). Similarly, an example of a constraint is shown at the bottom of the table. A constraint axiom is written as a rule with a body, but without a head, meaning that the body should not be satisfied. In this case, the shoot event cannot happen by a player not possessing the ball. After some familiarization with the syntax, the declarative nature of such rules often makes it easier for the human user to understand the intended meaning and the rule correlation.

Axiomatizing the Team Tactic Intelligence

The modeling of the dynamics of the EduBAI domain, described previously, covers certain elementary, yet crucial aspects in the process of representing causal domains. More advanced features of logic-based reasoning can be introduced as an attempt to implement intelligent team tactics to outsmart the opponent. Next, we discuss aspects that can be taken into consideration when implementing the reasoning part of a basketball team in EduBAI, specifying the features that can be learned by the user. 6 Table 2 give examples of encodings.

Simple, pre-defined tactics: Users can implement a rich set of simple tactics by only moving their own players along the lines of a predefined plan, ignoring any information about the opponents, the probability of action success etc. Examples include moving directly towards the basket before shooting without risking any pass, moving the teammates away from the player holding the ball to avoid crowding defenders in that cell, etc.

Such tactics can easily be implemented by only relying on simple Event Calculus constructs that model the effects of actions, in addition to a powerful construct provided by ASP called choice rule. Choice rules are typically used in ASP programs to specify which combinations of predicates should be included in an answer set. In our case, they are used to produce the set of all possible plans of actions to be taken by the players. For instance, the first rule shown in Table 2 generates all combinations of valid shoot or move toward the basket actions to be considered as a potential next step. Appropriate state constraints can also be added, to eliminate for instance answer sets suggesting a shoot action when the distance is above a given threshold.

Considering the opponent players' positions: Apparently, taking into account the position of opponent players is the next reasonable step, especially if defensive tactics are to be implemented. This can be as simple as mirroring the opponents' moves, but it can also involve the axiomatization of topological relations, such as maintaining sufficient distance to accomplish double-player blocking when a shoot is attempted. Defining proper predicates to capture such topological relations offers a very preliminary introduction to geospatial reasoning.

Taking into account probabilities: Taking advantage of the percentage of success of actions can significantly improve the intelligence of a team tactic. The modeling of uncertainty can directly be introduced in domain axioms or it can be implimented with more advanced probabilistic extensions of the Event Calculus.

Optimizing a cost function: The coupling of modeling probabilities with ASP optimization statements can produce powerful tactics. In particular, teams can easily implement a rich variety of cost functions to minimize relevant aspects, in order to help their teams decide whether to move, shoot or pass, given the state of their opponents and the success percentages. The ruleset shown in Table 2 is part of a tactic that decides which attacker each defender should cover, attempting to minimize the Manhattan distance of all defending players from the attackers.

Considering past/future actions: The incorporation of temporal reasoning can also improve significantly the effectiveness of team plans. The easiest step is to consider future timepoints, in order to estimate whether following a given course of actions can reach a possible state that achieves high probability of success for a shoot or pass, given the current state of players. Even more challenging can be to also take into consideration the history of opponent's moves, in order to figure out their tactics and react accordingly. The explicit modeling of time, as supported by the Event Calculus, facilitates temporal reasoning in practical domains, where the time clock needs to also be taken into account.

Implementation

EduBAI consists of multiple web services that expose their functionality either via a Rest API or a web-socket connection. The core components of the platform, shown in Figure 2, are as follows: User Service: this is a nodeJS service that stores, manages, and serves data about users. It provides a Restful API which allows users to register, login and query the system about other users. A noSQL database is used to store data.

User Status Service: This service is responsible for monitoring if a user is offline, online or in a game session (i.e., in a basketball match). This service also provides an invitation mechanism, where a user can start a game session by inviting another online user. Users can access the service via a Restful API alongside with a websocket for real time communication.

Tactics Service: It manages custom tactics created by users. It provides a Restful API developed using nodeJS that allows the uploading of ASP code.

Players Positioning Service: This service provides a real time connection between users, using nodeJS and webSockets, when they are in the same game session. This connection is used in order to send data about the positioning of the players on the court and for setting up the initial position of the players and ball in each new round.

Game System: This is a web service responsible for running a game session. The Game System is developed using Java and provides a websocket server where multiple users can connect. ASP code is executed using Clingo, a command line tool that is executed as a child process. After the execution of the logic program defining the tactics, the results are forwarded to the Sports Caster Server. This component implements a simple multi-thread java server that receives XML messages arriving from the Game Master and decides the outcome of the player actions based on the aforementioned probabilities (Section 2.1).

Graphical User Interface (GUI): The GUI is Web based and implemented using HTML, CSS and JavaScript. It is designed with respect to Human-Computer Interaction (HCI) principles. Learnability, i.e., the ease of effective interaction between a new user and the system, is achieved by following specific HCI guidelines, such as adopting a common Look 'n' Feel throughout the whole system, in order to achieve consistency among all pages. In addition, Web design guidelines are followed for predictability when positioning Web elements; e.g., the logout button is on top right of the page, which is where a user expects to find it.

Related Work and Conclusions

Probably the closest to ours platform in terms of scope and objectives is the React! tool [START_REF] Dogmus | React!: An interactive educational tool for ai planning for robotics[END_REF]. React! is an educational environment designed to teach AI students the main formalisms and principles of cognitive robotics. It is a domain-agnostic tool that provides guidelines and explanations for modeling dynamic causal domains to solve planning problems. The user can choose among different SAT and ASP solvers to execute the reasoning tasks.

In a broader sense, a similar purpose is being served by platforms that support users in the complex task of producing syntactically correct and valid logic programs. For instance, the SeaLion system [START_REF] Busoniu | Sealion: An eclipse-based ide for answer-set programming with advanced debugging support[END_REF] is an IDE for facilitating the task of ASP encoding.

Such tools though do not employ some scoring mechanism or adopt a competitive style of interaction to motivate the participation of users in the learning process. Such a practice is typically found in systems following the Games With A Purpose (GWAP) paradigm, which has become a rather popular approach. There are many relevant systems, as for instance the one presented in [START_REF] Rodosthenous | A hybrid approach to commonsense knowledge acquisition[END_REF], which relies on the Event Calculus, similarly to EduBAI. It implements a game where players take the role of a teacher to train a robot to answer simple questions.

Overall, EduBAI is a platform that intends to help users improve their logic programming skills through an entertaining interactive environment. Relying on the set of predefined tactics as guidelines, the users can apply their own creative thinking, in order to devise complex and effective tactics to win human or computer opponents. Part of our future plans involve extending the platform with friendly interactive features that will enhance the educational character, as for example an embedded ASP editor to enable users to encode their tactics or a tutorial-style assistant to walk new user through the main functionalities.

Fig. 1 .

 1 Fig. 1. Snapshot of the EduBAI UI

  Example of fluent and event definitionsf luent(player pos(T M, (X, Y ))) : -player(T M ), pos(X, Y ). event(pass(T M 1, T M 2)) : -player(T M 1), player(T M 2), T M 1! = T M 2.Example of a context-dependent effect axiom: the move left action initiates(move(T M, lef t), player pos(T M, (X, Y -1)), T ) : -holdsAt(player pos(T M, (X, Y )), T ), time(T ). terminates(move(T M, lef t), player pos(T M, (X, Y )), T ) : -holdsAt(player pos(T M, (X, Y )), T ), time(T ).

Table 2 .

 2 Sample of team tactics axiomatization. Example of a choice rule generating all combinations of specific actions (exactly one) 1{happens(doN othing(D), T ); happens(move(D, lef t), T ); happens(shoot(D), T )}1 :player(D), time(T ). Example of optimization: assign player markings minimizing overall distance holdsAt(marking(Def, Att), T ) : -def ender(Def ), attacker(Att), time(T ). : -holdsAt(marking(Def 1, Att), T ), holdsAt(marking(Def 2, Att), T ), Def 1! = Def 2. holdsAt(manhattanDist(Def, Att, |X1 -X2| + |Y 1 -Y 2|), T ) : -holdsAt(marking(Def, Att), T ), holdsAt(player pos(Def, (X1, Y 1)), T ), holdsAt(player pos(Att, (X1, Y 1)), T ). #minimize{Dist : holdsAt(manhattanDist(Def, Att, Dist), T )}.

Fig. 2 .

 2 Fig. 2. EduBAI system architecture

Table 1 .

 1 Sample of the game environment axiomatization.

https://potassco.org/

The EduBAI platform: http://139.91.183.97:8181/edubai/

Arguments starting with a capital letter denote universally quantified variables.

Fully implemented tactics can be found at https://github.com/DimitrisArr/EduBAI