Jiawei Zhu
email: jiawei.zhu@utbm.fr

Fabrice Lauri
email: fabrice.lauri@utbm.fr

Abderrafiaa Koukam
email: abder.koukam@utbm.fr

Vincent Hilaire
email: vincent.hilaire@utbm.fr

Scheduling Optimization of Smart Homes based on Demand Response

Keywords: smart grid, smart home, demand response, particle swarm optimization, energy

Demand response can potentially lead to economic and environmental advantages, but non-coordinated scheduling and operation of controllable devices in a set of smart homes will make peak rebounds at periods with lower electricity prices happen, which may damage the power grid, cause unforeseen disasters, and reduce the global profit. In this work, we advocate the use of a metaheuristic algorithm based on Cooperative Particle Swarm Optimization to optimize scheduling and operation of time-shiftable and power-shiftable devices in a set of smart homes of a district. By employing this method, user comfort is guaranteed, electricity cost is reduced and total load on the main grid is flattened so that the global energy efficiency is improved.

Introduction

Corresponding to the increasing demands for environment, comfort, and energy, smart home has become an active research topic. Smart home, also known as automated home or intelligent home, is the residential extension of smart building. It is an automation system that incorporates a diverse of home appliances with ubiquitous computing, networking, and controlling abilities, forming an "Internet of Things", to provide better home life service and experience to residents [START_REF] Ma | The converged service oriented architecture in smart home service[END_REF][START_REF] Ricquebourg | The smart home concept : our immediate future[END_REF]. A smart home system does not simply turn devices on and off. It can also monitor their working states and detect the environment to make better decisions. The two most important aspects in smart homes that should be considered carefully are comfort level and energy efficiency.

Generally, different occupants have different definitions of comfort level and at the same time certain extents of uncomfortableness are tolerated for them. Therefore it allows the dynamic balance of comfort obtained and energy consumption in smart homes. When we talk about the energy efficiency, it represents on one hand the efficiency of generating units, and on the other hand the energy cost paid by occupants. In this field, smart homes can be treated as integral additions to smart grid, which is a modernized electrical grid using information and communication technologies to improve the efficiency, reliability, economics, and sustainability of the production and distribution of electricity [3]. Communication between smart homes and the smart grid allows flexible home appliances shifting working time or adjusting working power during on-peak periods to balance the usage of electricity. In this sense, demand response is one of the key energy management strategy adopted both by the smart grid and smart homes. It refers to changes in electric usage by end-use customers from their normal consumption patterns in response to changes in the price of electricity over time, and helps to reduce peak demand, therefore tempering the need to operate high-cost and high-emission generating units as well as cutting down residents' electricity bills [START_REF] Siano | Demand response and smart gridsa survey[END_REF].

In recent years, great efforts have been made to activate demand response potentials of residential customers and incorporate it in smart home systems that optimize the scheduling and operating status of relative intelligent appliances. In [START_REF] Chen | Real-time price-based demand response management for residential appliances via stochastic optimization and robust optimization[END_REF], a real-time price-based demand response management model for residential appliances is proposed. This management model is embedded into smart meters, employing scenario-based stochastic optimization and robust optimization approaches to determine the optimal operation considering future electricity price uncertainties. Authors of [START_REF] Chen | Uncertainty-aware household appliance scheduling considering dynamic electricity pricing in smart home. Smart Grid[END_REF] propose a stochastic energy consumption scheduling algorithm, which takes the time-varying pricing information, distributed renewable generation, and the customer-defined target trip rate as inputs, and generates an operation schedule to minimize the monetary expense. A decisionsupport tool based on particle swarm optimization is presented in [START_REF] Pedrasa | Coordinated scheduling of residential distributed energy resources to optimize smart home energy services. Smart Grid[END_REF] to help residential consumers optimize their acquisition of electrical energy services. With this approach, the minimization of monetary expense and the maximization of user comfort are balanced based on user-defined monetary benefits for electric services.

However, all of them only think about the scheduling of electric devices in a single resident case, that is to say the smart home with these approaches merely considers to optimize its profit by reducing the usage of flexible appliances during peak-load hours or deferring them to low price periods. In general, it ignores the states of its neighbours. As indicated in [START_REF] Li | Automated residential demand response: Algorithmic implications of pricing models. Smart Grid[END_REF], it shows that when the load management of a set of homes work selfishly, new peak-loads will occur during lower electricity price periods and it is called peak rebounds [START_REF] Safdarian | A distributed algorithm for managing residential demand response in smart grids[END_REF]. This accentuates the necessity of an effective global demand response management scheme to ensure the coordination among them. In [START_REF] Kahrobaee | A multiagent modeling and investigation of smart homes with power generation, storage, and trading features. Smart Grid[END_REF], a multi-agent based approach is developed to minimize different smart homes' electricity bills in line with the grid's goal to flatten the total demand curve. Other work that considers to maximize various users' profits while avoiding severe peak-load rebounds can be found in [START_REF] Safdarian | A distributed algorithm for managing residential demand response in smart grids[END_REF][START_REF] Mohsenian-Rad | Optimal residential load control with price prediction in real-time electricity pricing environments. Smart Grid[END_REF][START_REF] Atzeni | Demand-side management via distributed energy generation and storage optimization. Smart Grid[END_REF]. But in almost all of them, user comfort is not considered adequately. For example, deferring the operation of washing machine until the midnight should be avoided, for the noise will disturb the sleeping residents.

In this work, we propose a metaheuristic algorithm based on Cooperative Particle Swarm Optimization (CPSO) to optimize scheduling and operation of time-shiftable and power-shiftable devices in a set of smart homes of a district. By employing this algorithm, not only user comfort is guaranteed and energy consumption is reduced in individual smart home, but also peak rebounds are avoided on the power grid so that the global energy efficiency is improved. The contribution of this work is threefold: firstly, an objective function that considers dissatisfaction and energy cost of individual home and electricity variances on the grid is proposed, meanwhile typical household appliances are categorized and modelled; secondly, due to the problem involves a lot of variables, a metaheuristic algorithm based on CPSO is developed to solve it; thirdly, by conducting experiments in three cases, we demonstrate the applicability and usefulness of the proposed method.

The rest of this paper is organized as follows. Section 2 mathematically formulates the problem. Section 3 presents the proposed method in detail. Experimental results and analysis are given in section 4. Finally, we conclude in section 5.

Problem Formulation

This section intends to present mathematical formulations for the problem we need to solve. The objective function is introduced firstly. Then we divide controllable household appliances into two categories: power-shiftable and time-shiftable respectively, and formulate mathematical models for each of them.

Objective Function

The system architecture is shown in fig. 1. In a District A, there are N smart homes connecting to the main grid to exchange data and buy/sell electricity. The Grid Agent and Home Agent are responsible for managing the main grid and each smart home respectively. In a smart home, different electric appliances are operated to work or planed to work in order to satisfy occupants' requirements.

When they function properly, occupants' comfort can be guaranteed, whereas running defectively or unreasonably can reduce the satisfaction, for example in case of space over or under heating or scheduling a washing machine to work during midnight. Therefore one of the system goals is to minimize the dissatisfaction caused by these appliances in each smart home. Furthermore, demand response enabled by time-varying prices can propel customers to change their consumption patterns for a more economical purpose, such as turning down some devices' working power during on-peak periods and deferring some devices' running time to off-peak periods. Hence, decreasing the household cost of electricity usage should be considered in the system. However, a non-coordinated response of customers for time-varying prices may lead to severe peak rebounds at periods with lower prices. For example, when all N customers in District A prefer to use air-conditioning, washing machine, dish washer, etc. during off-peak time selfishly, the total load on the main grid will turn to be extremely high. This will increase the burden of generating units, do harm to the main grid, and bring unforeseen disasters. Towards flattening the total load profile, the variance of energy consumption on the main grid need to be minimized. Corresponding to these three situations, the objective function of each household i is to minimize the integration of the three sub-objectives, where the dissatisfaction and the cost can be evaluated by its individual home agent while the variance is evaluated by the grid agent based on power scheduling data received from all smart homes in the district, as expressed below:

min Dissatisf action(i) + α • Cost(i) + β • V ariance (1)
where α and β are two weight factors. The three sub-objectives are calculated as:

Dissatisf action(i) = T t=1 S j=1 [I ij (t) • U ij (t)] (2)
Cost(i) = T t=1 [ρ(t) • S j=1 P ij (t)] (3)
V ariance = T t=1 [ N i=1 S j=1 P ij (t) - 1 |T | N i=1 T t=1 S j=1 P ij (t)] 2 (4)
where t, T index and set of time interval, i, N index and set of household, j, S index and set of electric appliance, I ij (t) binary variable denoting allowable working status of appliance j in household i at time t, U ij (t) dissatisfaction caused by operating appliance j in household i at time t, ρ(t) electricity sale price at time t P ij (t) working power of appliance j in household i at time t.

Household appliances can be divided as fixed devices, which have fixed load profiles such electric oven, refrigerator, TV, etc., and controllable devices including power-shiftable devices and time-shiftable devices, that will be particularly described in the next two subsections.

Power-Shiftable Devices

Power-shiftable devices represent these devices whose working power is changeable to fit variations of environmental status, and at some circumstances they can decrease the power at the range that occupants can accept to reduce load on grid. Two main power-shiftable devices considered in this work are electric space heater and electric water heater.

The dissatisfaction by using the space heater is defined as:

∆ sh (t) = T * in (t) -T in (t) (5)
U sh (t) = ∆ sh (t) × if 0 < ∆ sh (t) < ζ ∆ sh (t) 2 × 10 otherwise (6)
where T * in (t) and T in (t) are the desired temperature and the actual indoor temperature at time t respectively; U sh (t) is the dissatisfaction caused by using the space heater at time t; is a small scalar number; ζ is a temperature difference limit that occupants can tolerate. It means the farther away the indoor temperature is from the desired temperature the more dissatisfaction comes, and if the temperature difference is under occupants' tolerance, the dissatisfaction is negligible. According to [START_REF]ASHRAE: ASHRAE Handbook: Fundamentals[END_REF], the discrete time model using one-hour time step for indoor temperature variations is given by

T in (t + 1) = T in (t)e -1/τ + R • P sh (t) • (1 -e -1/τ ) + T out (t) • (1 -e -1/τ ) (7)
where R is the thermal resistance of the building shell in • C/kW , C is the heat capacity of indoor air in kW h/ • C, τ = R • C, P sh (t) is the working power of the space heater at time t, and T out (t) is the outdoor temperature at time t.

The dissatisfaction by using the water heater is defined as:

∆ wh (t) = T * hot (t) -T hot (t) (8)
U wh (t) = ∆ wh (t) × θ if 0 < ∆ sh (t) < η ∆ wh (t) 2 × 10 otherwise ( 9
)
where T * hot (t) and T hot (t) are the desired hot water temperature and the actual hot water temperature at time t respectively; U wh (t) is the dissatisfaction caused by using the water heater at time t; θ is a small scalar number; η is a temperature difference limit of hot water that occupants can tolerate.

In modelling water heating, we assume that the storage of hot water is always full, which means the equivalent of cold water enters the storage immediately when some hot water is drawn to be used. Hence, based on the heat balance, the temperature of hot water at each hour is calculated from the following equation: [START_REF] Kahrobaee | A multiagent modeling and investigation of smart homes with power generation, storage, and trading features. Smart Grid[END_REF] where C water is the specific heat of water in kW h/L • • C, P wh (t) is the heating power of the water heater at time t, V total is the total volume of the hot water storage, V outf low (t) and T cold (t) are the volume of hot water drawn to be used and the cold water temperature at time t separately. Time-shiftable devices are these appliances that can shift the power consumption time within a preferred working period and when they start to work they are in a constant power for a certain period. In this work, three timeshiftable appliances are considered: washing machine, clothes dryer, and dish washer. The dissatisfaction caused by using these devices are shown in fig. 2. It means the operation of washing machine and clothes dryer during the sleeping time will annoy the habitants due to the noise, and the dish washer should have cleaned the tableware and be off during the lunch and dinner time.

T hot (t + 1) = V outf low (t) • (T cold (t) -T hot (t)) + V total • T hot (t) V total + P wh (t) V total • C water

Time-Shiftable Devices

Scheduling Optimization Algorithm

In this research, we use Cooperative Particle Swarm Optimization (CPSO), which is based on the standard Particle Swarm Optimization (PSO), to search for the near-optimal scheduling and operation for each controllable devices in a smart home, because of its straightforward implementation and demonstrated ability of optimization.

Standard Particle Swarm Optimization

Standard Particle Swarm Optimization is derived from simulating social behavior as a stylized representation of the movement of organisms in a bird flock or fish school, which is originally proposed and developed by [START_REF] Kennedy | Particle swarm optimization[END_REF]15]. It is a metaheuristic algorithm that has been turned out to be powerful to solve complex non-linear and non-convex optimization problems [START_REF] Lin | Recurrent functional-link-based fuzzy-neural-network-controlled induction-generator system using improved particle swarm optimization[END_REF]. Moreover, it has several other advantages, such as fewer parameters to adjust, and easier to escape from local optimal solutions.

In PSO, a population of candidate solutions, here dubbed particles that include position vector x and velocity vector v, is randomly generated around the search-space initially. After that they are iteratively updated to simulate the movement around the search-space according to mathematical formulae over the particles' position and velocity, as expressed below:

v k+1 i = w • v k i + c 1 • rand() • (p best k i -x k i ) + c 2 • rand() • (g best k -x k i ) ( 11
)
x k+1 i = x k i + v k+1 i ( 12
)
where w is inertia weight, c 1 and c 2 are acceleration constants, rand() generates random value between 0 and 1, i is the particle index, k is the iteration index, p best k i is the local best position, and g best k is global best position. For Equation 11, the first part expresses particle's inertia of previous action, the second part reflects particle's cognition that stimulates the particle to diminish errors, and the last part is called social part, which indicates the information sharing and cooperation among particles.

Cooperative Particle Swarm Operation

In this work, the problem involves multiple households, and in each of them contains multiple controllable appliances. Therefore, if it employs the standard PSO, a full dimensional vector should be constructed for each particle. This will cause the "two steps forward, one step back" problem [START_REF] Van Den Bergh | A cooperative approach to particle swarm optimization[END_REF], which can make the convergence very hard. In order to solve this problem that involves a large number of decision variables, we adopt and develop the CPSO proposed in [START_REF] Van Den Bergh | A cooperative approach to particle swarm optimization[END_REF]. The original method utilizes a divide-and-conquer approach. The vector to be optimized is divided into several components. Each swarm attempts to optimize only one of the components. The fitness of a particle belonging to one of the swarm is calculated by concatenating it with the present global best particles of the other swarms. In our study, we make trade-off between the swarm number and the particle vector dimension of each swarm.

In detail, nine swarms are designed for each smart home, of which four swarms optimize the space heater power, four swarms optimize the water heater power, and one swarm optimizes the starting time of the three time-shiftable devices, as shown in fig. 3. Hence, each particle of the space heater swarm has 6 dimensions instead of only one dimension, and each dimension expresses the heating power for one hour. It's the same setting for water heater swarms. For the timeshiftable device swarm, each particle has 3 dimensions indicating the starting time of the three devices. As presented in subsection 2.1, the objective function of each household is to minimize the aggregation of the three sub-objectives: dissatisfaction, cost, and variance. The dissatisfaction and the cost of each home can be evaluated by its home agent after it has received the power data of powershiftable devices and the starting time data of time-shiftable devices, while the variance can be calculated only after all home agents of the district have sent their individual power scheduling data to the grid agent. Then a fitness is evaluated by summing these three parts up with different weights. In addition, in our scheduling and operation optimization problem, there exists a risk of stagnation. This is caused by dividing one day's operation of power-shiftable devices into several sequential parts, and each part is optimized by a swarm. Although these swarms need to cooperate to find a near-optimal operation solution for one day, they work successively according to the time order. Consequently, the optimizing result of the former swarm will affect the fitness function of the latter one. Note that generally particles in a swarm tend to fly to the global best position, so if the fitness function changes a lot after a number of iterations, the algorithm will be stagnated at a wrong place.

In order to solve this stagnation problem, we develop the original CPSO algorithm with particle reinitialization. The novel algorithm is called CPSO-R. In this algorithm, after a certain number of iterations the position and velocity of all particles will be reinitialized, but the global best position will be recorded. Based on this method, the stochastic initialization can easily make the algorithm jump out from the stagnation, because new random particles have the ability to explore the whole search-space.

Algorithm 1 CPSO-R algorithm define: c(j, z) ≡ (s1.g best , s2.g best , ..., sj-1.g best , z, sj+1.g best , ..., sm.g best ) Initialize each n particles of m swarms with random positions and zero velocities Set the reinitialization interval N ; Set Index = 0 while any stopping criterion is fulfilled do Index = Index + 1 for each swarm j ∈ {1, 2, ..., m} do for

each particle i ∈ {1, 2, ..., n} do if f (c(j, sj.xi)) < f (c(j, sj.p best i )) then sj.p best i = sj.xi end if if f (c(j, sj.p best i )) < f (c(j,

Experimentation

In this section, we show experimental results and provide analyses to demonstrate the performance of the presented algorithm. Before conducting the experiments, the parameters are set as: in CPSO-R, w = 0.72, c 1 = 1.49, and c 2 = 1.49 [START_REF] Eberhart | Comparing inertia weights and constriction factors in particle swarm optimization[END_REF]; in mathematical models of smart home, α = 10 -8 , β = 10 -3 , = 10 -5 , ζ = 2, θ = 10 -5 , η = 5, R = 21 • C/kW , C = 0.125kW h/ • C, C water = 0.001167kW h/L • • C, V total = 60L, P sh (t) ∈ [0, 2]kW , P wh (t) ∈ [0, 1.2]kW , Washing Machine:{2hour, 1.4kW }, Clothes Dryer:{1hour, 1.3kW },and Dish Washer:{3hour, 1.7kW }. The time-varying electricity price and one day's ambient temperature are shown in fig. 4 and fig. 5. In order to show the impacts of demand response as well as coordinated actions of neighbouring smart homes in a district to avoid peak rebounds, three case studies are simulated and compared. In each simulation, five smart homes with the same settings in a district are taken into account and occupants are assumed at home except for 8 : 00 ∼ 12 : 00 and 14 : 00 ∼ 18 : 00 during which time they are assumed to be out for working. Fig. 6 shows the experimental results of the basic case. In this case, demand response and coordination are not considered and the algorithm is only used to control the space heater and the water heater, so there is no intelligent scheduling and operation for controllable appliances. When people turn them on, they work immediately and maximize people's comfort. Fig. 6(c) shows that all the three time-shiftable devices of the five households are used after occupants return home in the evening. This not only causes peak loads on the main grid, as can be seen in fig. 6(a), but also costs more money for the electricity prices during these periods are higher. Fig. 6(b) and fig. 6(d) show that the mean indoor temperature and the mean hot water temperature of the 5 families reach the desired degrees when people are at home.

In the second case, demand response is taken into account without coordination among different homes. Therefore, each individual home try to maximize its habitants' comfort as well as minimize the electrical bill. The performance of the space heater and the water heater is the same as in case 1. Moreover, from fig. 7 ever, peak rebounds happen during the lowest price periods when dish washers of all homes work, as shown in fig. 7(a).

In order to solve the peak rebounds and make trade-off between electrical budget and comfort, all homes in the district are assumed to be able to coordinate by aggregating their electricity information on the main grid, and a 2-degree of indoor temperature and a 5-degree of hot water temperature from the desired temperatures are assumed to be tolerated for people. The experimental results of the case 3 are shown in fig. 8. From fig. 8(a), we can see that the variance of total load on the main grid is minimized, and peak loads and peak rebounds are avoided. Fig. 8(b-d) show that by employing our proposed method, powershiftable devices and time-shiftable devices can work properly to guarantee their service quality. The comparison of the three cases is listed in tab.1, and we can find that with demand response and coordination our method can both save money and balance loads on the grid.

Conclusion

This paper advocated the use of a metaheuristic algorithm based on cooperative particle swarm optimization to optimize scheduling and operation of timeshiftable and power-shiftable devices in a set of smart homes of a district. This method takes advantage of demand response and coordination of neighbouring homes to not only guarantee user comfort and reduce electrical bill of individual home but also avoid peak loads and peak rebounds on the main grid. The performance of the proposed algorithm was demonstrated by conducting and comparing three case study simulations. The experimental results indicate that selfish operation and optimization of individual home will make peak rebounds happen, which may damage the power grid, cause unforeseen disasters, and do harm to the global profit, while a coordinated management of a set of smart homes can get rid of this negative situation and increase the benefits bringing from demand response. In future work, the multi-agent based technique will be studied, and proactive and reactive agent properties will be capitalized on to construct a more intelligent system.

Fig. 1 .

 1 Fig. 1. System Architecture

Fig. 2 .

 2 Fig. 2. Dissatisfaction of Time-Shiftable Devices

Fig. 4 .

 4 Fig.4. Time-Varying Price[START_REF] Safdarian | A distributed algorithm for managing residential demand response in smart grids[END_REF]

Fig. 5 .

 5 Fig. 5. Outdoor Temperature of One Day

Fig. 6 .

 6 Fig.6shows the experimental results of the basic case. In this case, demand response and coordination are not considered and the algorithm is only used to control the space heater and the water heater, so there is no intelligent scheduling and operation for controllable appliances. When people turn them on, they work immediately and maximize people's comfort. Fig.6(c) shows that all the three time-shiftable devices of the five households are used after occupants return home in the evening. This not only causes peak loads on the main grid, as can be seen in fig.6(a), but also costs more money for the electricity prices during these periods are higher. Fig.6(b) and fig.6(d)show that the mean indoor temperature and the mean hot water temperature of the 5 families reach the desired degrees when people are at home.In the second case, demand response is taken into account without coordination among different homes. Therefore, each individual home try to maximize its habitants' comfort as well as minimize the electrical bill. The performance of the space heater and the water heater is the same as in case 1. Moreover, from fig.7(b) we can see that time-shiftable devices are deferred to work during low price periods: dish washers are set to work during the middle of the night; while washing machines and clothes dryers are set to work during the afternoon, because they can annoy people if they work during the middle of the night. How-

Fig. 7 .

 7 Fig. 7. Experimental Results of Case 2

Fig. 8 .

 8 Fig. 8. Experimental Results of Case 3

  sj.g best i )) then sj.g best i = sj.p best i end if Update velocity vi and position xi according to Equations 11 and 12 Return the global best solution (s1.g best , s2.g best , ..., sm.g best ).

	end for
	end for
	if mod(Index, N ) == 0 then
	Reinitialize each n particles of m swarms with random positions and zero ve-
	locities
	end if
	end while

Table 1 .

 1 Comparison of The Three Cases

		mean cost mean power	std.
		(cent euro)	(kW)	(power on grid)
	Case 1	228.1	5.764	5.293
	Case 2	219.6	5.764	3.615
	Case 3	213.5	5.465	0.469


David Antón

Idoia Berges

Jesús Bermúdez

Alfredo Goñi

Arantza Illarramendi

Knowledge-Based Telerehabilitation Monitoring

Keywords: Telerehabilitation, Telemedicine, Knowledge representation, Ontology, Kinect

Introduction

People's higher survival to diseases and traumas which leave physical sequels has generated an increase in demand for rehabilitation processes. Rehabilitation is a critical component for resuming normal activities of daily living. For example, Maire et al. [START_REF] Maire | A Specific Arm-Interval Exercise Program could Improve the Health Status and Walking Ability of Elderly Patients After Total Hip Arthroplasty: A Pilot Study[END_REF] indicate that the improvement in physical fitness and functional status as a result of rehabilitation is associated with better health status after hip replacement.

Traditional rehabilitation takes place in rehabilitation centers or hospitals which many times get saturated and which requires patients to travel to appointments. This travel is often associated with both time and financial costs [START_REF] Dávalos | Economic Evaluation of Telemedicine: Review of the Literature and Research Guidelines for Benefitcost Analysis[END_REF]. An alternate rehabilitation method is using telerehabilitation technologies where rehabilitation services are delivered directly into patients' home [START_REF] Botsis | Home Telecare Technologies for the Elderly[END_REF], reducing so the congestions of the centers and letting physiotherapists intervene effectively especially for those patients who have difficulty with transportation to rehabilitation centers [START_REF] Piqueras | Effectiveness of an Interactive Virtual Telerehabilitation System in Patients After Total Knee Arthroplasty: A Randomized Controlled Trial[END_REF].

A common component in recent telerehabilitation systems is motion capture technology [START_REF] Metcalf | Markerless Motion Capture and Measurement of Hand Kinematics: Validation and Application to Home-Based Upper Limb Rehabilitation[END_REF], [START_REF] Cuervo | Biomechanical Motion Capture Methods Focused on Tele-Physiotherapy[END_REF]. The use of telerehabilitation systems with motion capture has been shown to increase the intensity of rehabilitation and enhance user experience [START_REF] Hanif | Investigating the Possible Role and Usefulness of Video Capture Virtual Reality in Motor Impairment Rehabilitation[END_REF], [START_REF] Lee | Telerehabilitation as a Means of Health-Care Delivery[END_REF].

Another consideration to attract users' attention and interest in the system is the incorporation of avatars. According to Ortiz et al. [START_REF] Ortiz | Elderly Users in Ambient Intelligence: Does an Avatar Improve the Interaction?. User interfaces for all[END_REF] there are many potential advantages in the use of avatars, rather than other conventional methods.

In this paper we show KiReS, a telerehabilitation system that makes use of Kinect's technology to analyze patients' exercises through the monitoring of the position of the body in space. Microsoft Kinect® (Microsoft Corp., Redmond) is a markerless camera based visual tracking system that extracts information about the three dimensional position of 20 different body joints. Developed primarily for gaming purposes, the device lets the users interact without carrying any wearable devices on their body [START_REF]Kinect for Windows[END_REF], [START_REF] Andersen | Kinect Depth Sensor Evaluation for Computer Vision Applications[END_REF]. Currently, telerehabilitation systems can be found both in an academic setting as in a commercial environment. However, KiReS provides some novel features that we summarize in the following:

• Friendly and helpful interaction with the system. This means that KiReS combines the use of a non-wearable motion control device with motivational interfaces based on avatars, since successful rehabilitation depends largely on the user's motivation and compliance with therapy. Furthermore, KiReS facilitates physiotherapists an interface that is based on the therapy protocols they typically use with the added value that it provides an easy way to define new exercises. • Provision of actionable information. KiReS uses different techniques to provide actionable information. On the one hand, it manages a novel domain specific ontology that we have built, that supports physiotherapists in the therapy design process by: assuring the maintenance of appropriate constraints and selecting for them a set of exercises that are recommended for the user. This type of information is not provided by current systems and it has been recognized as very interesting by the consulted physiotherapists. On the other hand, it is able to convert low-level recorded Kinect data into high-level knowledge. • Monitoring of rehabilitation sessions. KiReS incorporates an algorithm that evaluates online performed exercises and sets if they have been properly executed by comparing the obtained results with the recorded reference data. Automatic exercise evaluation is a key feature of our proposal, taking into account that, in home oriented telerehabilitation systems, it is crucial that the user is autonomously evaluated without the direct intervention of the physiotherapist during rehabilitation sessions.

The rest of the paper is organized as follows: In section 2 we introduce some related works. Next, in section 3 we briefly introduce the KiReS workflow. Then, in sections 4 and 5 we address the therapy planning and the therapy execution and controlling processes, respectively. Finally in section 6 we present some conclusions.

Related Works

Existing home telerehabilitation systems make use of different types of interaction devices and are oriented to the treatment of many physical pathologies. In a first ap-proximation we can classify them into two main groups. In the first group those works that propose to wear devices are included. Among them we can mention ArmAssist [START_REF] Perry | ArmAssist: An Integrated Solution for Telerehabilitation of Post-stroke Arm Impairment[END_REF]. The proposed system evaluates online performed arm exercises by using a lowcost device and a table mat that communicates with a PC. The second group includes those systems that advocate that users do not wear devices but they only use them. In [START_REF] Lockery | Store-and-Feedforward Adaptive Gaming System for Hand-Finger Motion Tracking in Telerehabilitation[END_REF] Lockery et al. present a system that uses a webcam and adaptive gaming for tracking finger-hand movement. They attached trackers to some objects and a webcam captures user's hand and generates some metrics that provide information about the quality, efficiency, and skill of the user. More recently, in the context of hand evaluation, Iosa et al. [START_REF] Iosa | Leap Motion Controlled Videogame-Based Therapy for Rehabilitation of Elderly Patients with Subacute Stroke: A Feasibility Pilot Study[END_REF] present a Leap Motion based rehabilitation system for elderly people that have suffered subacute stroke. This pilot study uses Leap Motion for conducting a videogame-based therapy that evaluates hand's ability and grasp force. However, a great number of works advocate for using Kinect, a motion capture device that tracks user full body movements without physical contact. In this line, Gotsis et al. [START_REF] Gotsis | Mixed Reality Game Prototypes for Upper Body Exercise and Rehabilitation[END_REF] present 21 game concept prototypes which receive and process data sent by Kinect, but the evaluation model is not dealt with by the authors. Pastor et al. [START_REF] Pastor | A Feasibility Study of an Upper Limb Rehabilitation System using Kinect and Computer Games[END_REF] and Chang et al. [START_REF] Chang | A Kinect-Based Upper Limb Rehabilitation System to Assist People with Cerebral Palsy[END_REF] have studied the feasibility of Kinect oriented to upper limb rehabilitation. In both works patients improved their motion ranges and systems acceptability was high. Su et al. [START_REF] Su | Kinect-Enabled Home-Based Rehabilitation System using Dynamic Time Warping and Fuzzy Logic[END_REF] present a system to assist patients in conducting home-based rehabilitation. System's evaluation matched that of the therapist in 80% of the cases, and users' usability and readability evaluation of the system was positive. Gabel et al. [START_REF] Gabel | Full Body Gait Analysis with Kinect[END_REF] developed a method focused on full body gait analysis using Kinect.

Results showed accurate and robust gait analysis using Kinect and its viability for diagnosis and monitoring of treatments in domestic environments. Galna et al. [START_REF] Galna | Retraining Function in People with Parkinson's Disease using the Microsoft Kinect: Game Design and Pilot Testing[END_REF] developed a game aimed at training dynamic postural control for people with Parkinson Disease. Finally, some proposals which are commercial products such as [21],

[22], [23] can be found but they do not show many technical details concerning their internal behavior and are oriented to specific pathologies.

In our case, we have paid special attention in developing motivational interfaces and providing an accurate online evaluation of exercises. Although these features are also considered somehow in some of the previous works, we have not found any that additionally provides the type of actionable information that KiReS provides.

KiReS workflow

The use of KiReS involves the performing of the activities shown in the UML activity diagram of Fig. 1, which are executed by three different actors: the physiotherapists, the users and the knowledge manager of the system. Some of these activities correspond to the therapy planning (purple) and others to the therapy execution and controlling (green).

With respect to the therapy planning, first of all, the physiotherapist makes an initial evaluation of the user, which includes what it is known as anamnesis. As a result of this evaluation some knowledge about the user is asserted in the Telerehabilitation Ontology (TrhOnt). After that, the physiotherapist assigns appropriate exercises to the user taking into account those recommended by TrhOnt (the ontology includes exercise descriptions and the exact details of all joints and movements involved in the exercises are stored in the KiReSdb database). If the physiotherapist wants to assign a particular exercise that does not exist yet, then the physiotherapist can create it by using the "Create New Exercise" activity.

Concerning the therapy execution and controlling process, once the exercises have been assigned, the user can perform them by using KiReS. Those exercises are monitored and the results are stored in KiReSdb. After the exercises have been performed and monitored two different activities can take place: 1) the physiotherapist can make a user reevaluation in order to finish the rehabilitation process or to assign new exercises to the user; and 2) a knowledge extraction process is performed in order to find new knowledge to add to the ontology.

The telerehabilitation ontology (TrhOnt)

TrhOnt is an OWL ontology composed of four interrelated parts of knowledge. We have designed it as a service artifact; therefore, OWL reasoners' capabilities play a crucial role. In the following we explain more about each type of knowledge.

─ Patient knowledge: This part consists of classes and properties for representing information such as personal and family data, goals, symptoms, results of physical examination, diagnoses, reported value in the Visual Analogue Scale (VAS) [START_REF] Miller | Measurement of Subjective Phenomena in Primary Care Research: The Visual Analogue Scale[END_REF] and everything captured at the anamnesis. ─ Anatomy knowledge: The Foundational Model of Anatomy1 (FMA) is a domain ontology that represents a coherent body of explicit declarative knowledge about human anatomy. We have extracted a module from FMA-OWL [START_REF] Golbreich | The Foundational Model of Anatomy in OWL 2 and its Use[END_REF] that is useful for the desired telerehabilitation process based on Kinect. FMA-OWL in its version 4.0 contains more than 100000 classes, 156 object properties connecting the classes, and more than 700000 axioms. ─ Movements and exercises knowledge: Classes and properties have been defined to represent atomic movement and complex movement (i.e. those compose of atomic). Basically, a movement is characterized by its type, its associated joint and its amplitude (min and max range of movement). Furthermore exercise classes are defined as compositions of movements. ─ Experts' domain knowledge: TrhOnt includes axioms that reflect specific knowledge about characteristics of recommended (and contraindicated) exercises depending on patient's state. This knowledge will be useful to the therapist during the "Assign Exercises" activity. Due to the information recorded in the Patient knowledge part, inference services (such as class subsumption and instance realization) applied on expert's domain knowledge are able to offer a list of recommended exercises for that patient.

The TrhOnt ontology takes part in the activities that evaluate and reevaluate users, the activity that assigns exercises to users and in the knowledge extraction activity. The ontology has been implemented using Protégé [26]. In Fig. 2 we show a snapshot of the class GlenohumeralJoint.

Creation of new exercises

KiReS offers an interface for the physiotherapist that provides assistance to create exercises step by step, this way it is guaranteed that the exercise structure is respected and our recognition algorithm is able to evaluate them. A posture is the simplest element of an exercise and therefore necessary for the definition of any other structure. The physiotherapist performs the posture in front of the system and records it (see Fig. 3). Then, a recording player tool allows the physiotherapist to select frame by frame which postures to store from the recording. Before storing postures, the posture recognition algorithm analyzes them in order to guarantee that they are similar enough. This type of similarity verification avoids adding very different postures with the same name and, at the same time, with well labeled postures the accuracy of the recognition algorithm is higher.

Movements have a name associated to identify them and are defined with two postures (initial and final) and with the recording of the transition between those postures (Fig. 4). Once both postures are selected, the system analyzes them. The relevant joints that best represent the transition from initial posture to final posture are selected and these joints are recorded and stored. Movement recording makes use of the same features as posture recording. The physiotherapist selects the movement to record and visualizes the initial and final postures of the movement. The posture recognition algorithm checks when the therapist makes both the initial and final posture and in the meantime the trajectories of the relevant joints are recorded. After reaching the final posture the recording player tool shows the movement and the therapist can replay it and decide whether to store it in the KiReS database or to repeat the recording. The information concerning the name, the initial and final postures, the type, the joint of the movement and the range of motion involved is added to the ontology to allow reasoning over movements.

Fig. 3. Posture edition

Lastly, exercises are defined by assigning movements to them. Simple exercises can consist of just one atomic movement but complex exercises are a combination of atomic movements, which create a sequence of movements. The only restriction when combining movements is that the final posture of a movement must match the initial posture of the next one. The exercise creation interface allows the therapist to define the composition of an exercise. It shows a form to fulfill data about the exercise and two lists with the movements assigned to that exercise and with the available movements to add. Once this is done the exercise will be stored in the system (in the database and also in the ontology for reasoning) and will be available to be added to a therapy session.

Fig. 4. Movement definition

For the implementation of the interfaces Unity 4 [27] was used and all the scripts that control the behavior of the interface were developed in C#. The avatars and the rest of the 3D models were modeled in 3Ds Max and exported to Unity. However, official Kinect drivers are not directly compatible with Unity, for this reason, some open source C# scripts [START_REF]Kinect SDK into Unity[END_REF] were used for interaction. This library provides basic functionality for Kinect for Windows in Unity.

Therapy execution and controlling

Users are monitored at the same time they are performing the exercises and all captured data are recorded in the KiReSdb. After that the physiotherapist can make a user reevaluation in order to finish the rehabilitation process or to assign new exercises to the user. Moreover, the knowledge extraction activity is performed in order to find new knowledge to add to the TrhOnt ontology.

Performing exercises

When users are performing exercises at home the interface must meet two requirements. It has to be easy to understand and at the same time attractive enough to encourage users to participate in therapy. The exercise interface of KiReS presents two 3D avatars that guide the user (see Fig. 5). The avatar on the right shows the movements of the user in real time, while the avatar on the left acts as a instructor, showing the exercise the user has to do. The four boxes below (see Fig. 5) provide information about the ongoing therapy session to the user. The two boxes on the right show the number of series and repetitions left 2 . When the user has done all the series the session is finished. The box on the left shows the name of the next posture the user has to reach. The box in the middle shows the "state" of the current movement, it is continuously updated by the exercise recognition algorithm and it displays information to guide the user in real time.

Besides, when the user is close to reaching a posture, the box indicates with a three level color scale (red, yellow and green) how close s/he is from reaching the posture.

In the upper center of the screen there is a ribbon that shows the exercise as a list of postures that have to be reached in the current execution. This ribbon is updated as the user completes exercises. Under this ribbon a textual explanation of the exercise is displayed. When a session is finished a new screen shows the results of the session: the execution accuracy of all exercises execution, the time taken to finish the session and the final evaluation of the session. In summary, the avatars and the informative boxes provide information to the user. This way, the system empowers and keeps the user aware of his/her therapy, but also provides a game-like immersive experience that motivates and makes the therapy more enjoyable.

Exercise Monitoring

While the user is performing the exercises, the system have been properly executed by comparing the result reference data.

As mentioned in section movement is composed of an initial posture, a final posture and the angular traject ries of the joints involved in the movement (the relevant ture and the final posture of a movement are identified.

• Identification of the initial posture the user to get into the initial posture. The posture classification method checks the user's current posture until it identifies it as the initial one. These checks are pe formed in real-time at a rate of about 30 checks per second with which Kinect provides data. When the initial posture is identified the system starts the trajectory recognition.

• Trajectory recognition

recognize if the movement itself is well performed. During the recognition, the tr jectory of each relevant ry of the same joint culated between those two trajectories the trajectory path is considered to be correct, and incorrect in opposite case. In o der to calculate the distances among trajectories, we use a variant of the Time Warping (DTW) • Identification of the final posture recognition method also checks the posture of the user. When the final posture is identified the movement is finished. If an exercise has more movements the met od tries to identify the initial posture of the next movement. Identifying the final posture of a movement

Exercise Monitoring

While the user is performing the exercises, the system evaluates them and sets if they have been properly executed by comparing the results obtained with the As mentioned in section 4, exercises usually consist of series of movemen movement is composed of an initial posture, a final posture and the angular traject involved in the movement (the relevant joints). Both the initial po ture and the final posture of a movement are identified.

the initial posture: When starting a movement the system waits for the user to get into the initial posture. The posture classification method checks the user's current posture until it identifies it as the initial one. These checks are pe ime at a rate of about 30 checks per second, which is the frequency with which Kinect provides data. When the initial posture is identified the system starts the trajectory recognition. Trajectory recognition: The trajectory recognition method has as a main purpose to recognize if the movement itself is well performed. During the recognition, the tr jectory of each relevant joint involved in the movement is compared to the traject joint recorded as reference for that movement and a distanc culated between those two trajectories. If the distance is less than a threshold value the trajectory path is considered to be correct, and incorrect in opposite case. In o der to calculate the distances among trajectories, we use a variant of the Time Warping (DTW) algorithm [START_REF] Salvador | Toward Accurate Dynamic Time Warping in Linear Time and Space[END_REF]. Identification of the final posture: While analyzing the trajectories, the exercise recognition method also checks the posture of the user. When the final posture is identified the movement is finished. If an exercise has more movements the met od tries to identify the initial posture of the next movement. Identifying the final of a movement has a peculiarity given the context of rehabilitation. In and sets if they s obtained with the recorded , exercises usually consist of series of movements. Each movement is composed of an initial posture, a final posture and the angular trajecto-

). Both the initial pos-When starting a movement the system waits for the user to get into the initial posture. The posture classification method checks the user's current posture until it identifies it as the initial one. These checks are perwhich is the frequency with which Kinect provides data. When the initial posture is identified the system main purpose to recognize if the movement itself is well performed. During the recognition, the trainvolved in the movement is compared to the trajectoa distance is cal-. If the distance is less than a threshold value the trajectory path is considered to be correct, and incorrect in opposite case. In order to calculate the distances among trajectories, we use a variant of the Dynamic While analyzing the trajectories, the exercise recognition method also checks the posture of the user. When the final posture is identified the movement is finished. If an exercise has more movements the method tries to identify the initial posture of the next movement. Identifying the final has a peculiarity given the context of rehabilitation. In some stages of therapy what is expected from the user is to try to reach that position or to at least make the physical effort to reach it. Thus, the method adapts the threshold depending on the time spent performing the movement. The threshold is multiplied by a flexibility factor that makes the algorithm be less rigid in posture classification. Finally, the overall exercise rating is the average of the rates of all the movements that compose the exercise. Detailed information of the algorithm can be found in [START_REF] Anton | Exercise Recognition for Kinect-Based Telerehabilitation[END_REF].

As said before, the flexibility factor is a very important concept in the evaluation of exercises. Several implementations of the exercise monitoring algorithm have been analyzed in order to adapt an adequate strategy. That strategy has been selected and validated, as statically significant, by applying Friedman and Nemenyi tests.

Knowledge Extraction

The data obtained during exercise execution and evaluation that are stored in the database of KiReS can be analyzed to extract knowledge in order to provide more information to the physiotherapists. This new knowledge is added to the ontology and will be available to the physiotherapist in the activities "Assign New Exercises" and "User Reevaluation". For example, a statistical analysis of raw data obtained from the telerehabilitation session of a user can find relevant information for the therapist. In Fig. 6 we show a shoulder exercise execution with a symmetric movement in which both arms are moved at the same time. The user raises up both arms to the head and then moves them down. The raw data analyzed consists on the results of evaluating the trajectories of several body joints during a session. A statistical analysis allows obtaining the correlation among these data that can be of interest for the physiotherapist. The conclusion is that "The left arm is progressing, both elbow and shoulder are recovering, but the recovery of the right arm might not be uniform and the patient may need a check". New assertions will be added to the ontology that will be used to notify the physiotherapist.

User Reevaluation

After the user has performed the exercises and the knowledge extraction process has been made, the physiotherapist must decide if the user has achieved the rehabilitation goals, or if new exercises must be assigned to the user. For that, the new extracted knowledge about user's medical condition (obtained ROM, accuracy, speed…) will be available in the ontology ready to be checked by the physiotherapist.

Conclusions

KiReS provides home-based telerehabilitation with a natural form of interaction. The interface for the users includes two avatars, one with which the user can see the exercise s/he must do and another one with which s/he can see how s/he is actually doing it. From the point of view of physiotherapists, KiReS allows them to define customized therapies for the users. Moreover, physiotherapists can also analyze the knowledge extracted from the data recorded from the users' executions in order to track the users' evolution, obtain new knowledge about exercise performance or use the data to identify and correct undesired situations. Another relevant aspect to highlight is that KiReS is not designed for a specific pathology; the system can be loaded with a broad spectrum of exercises as opposed to the majority of proposals that consider fixed exercises to specific physical pathologies. KiReS was tested in a trial we performed in a rehabilitation center. Moreover, for this trial we prepared questionnaire that all the users agreed to fill at the end of the session. This early trial results showed some aspects that we consider relevant about the users' interaction and experience with the system. First, we found that the interaction with Kinect was easy to learn for the users and they found the system comfortable to interact with. Second, they see the system as a complement to their therapy that can improve medical attention but not as effective as the ordinary session. Third, they showed a predisposition to using the system again and felt satisfied with the experience. Finally, the overall impression of the interface content was positive and users found the information 3D avatars gave to them helpful.

As future work we are considering the possibility of enhancing the information KiReS retrieves by adding biosignal tracking devices such as pulse oximeters. This would require updating ThrOnt to incorporate these new relevant data. Also in the future we expect to adapt the system to work with the new version of Kinect.

Fig. 1 .

 1 Fig. 1. KiReS activity diagram

Fig. 2 .

 2 Fig. 2. Axioms about Glenohumeral Joint in Protégé

Fig. 5 .

 5 Fig. 5. User exercise interface

Fig. 6 .

 6 Fig. 6. Knowledge extraction example

http://sig.biostr.washington.edu/projects/fm/FME/index.html

A series is the list of exercises to be done on a session and the repetitions is the number of times an exercise has to be done in each series.

Acknowledgements

We want to thank Dr. Jon Torres and Dr. Jesús Seco for their help with the physiotherapy-related aspects. This work was supported by the Spanish Ministry of Economy and Competitiveness [grant number FEDER/TIN2013-46238-C4-1-R] and by the Basque Country Government [grant number IT797-13]. The work of David Antón was supported by a grant of the Spanish Government (Subprograma de Formación de Personal Investigador).


Emilie M D Jean-Baptiste

Pia Rotshtein
email: p.rotshtein@bham.ac.uk

Martin Russell
email: m.j.russell@bham.ac.uk

POMDP based Action Planning and Human Error Detection

Keywords: POMDP, Planning Under Uncertainty, Assistive Technology

This paper presents a Partially Observable Markov Decision Process (POMDP) model for action planning and human errors detection, during Activities of Daily Living (ADLs). This model is integrated into a sub-component of an assistive system designed for stroke survivors; it is called the Artificial Intelligent Planning System (AIPS). Its main goal is to monitor the user's history of actions during a specific task, and to provide meaningful assistance when an error is detected in his/her sequence of actions. To do so, the AIPS must cope with the ambiguity in the outputs of the other system's components. In this paper, we first give an overview of the global assistive system where the AIPS is implemented, and explain how it interacts with the user to guide him/her during tea-making. We then define the POMDP models and the Monte Carlo Algorithm used to learn how to retrieve optimal prompts, and detect human errors under uncertainty.

Introduction

The need to support the population suffering from cognitive impairments led to an increase number of research on assistive technology. Such systems generally focus on one specific task, and aim to provide prompts to the user in order to help him/her to complete a sequence of actions. For example, Peters et al. [START_REF] Peters | TEBRA -an automatic prompting system for persons with cognitive disabilities in brushing teeth[END_REF] developed the TEBRA system that supports mildly impaired people during teeth-brushing. Mihailidis et al. [START_REF] Mihailidis | The coach prompting system to assist older adults with dementia through handwashing: An efficacy study[END_REF] implemented the COACH system that helps patients with Alzheimer's during hand-washing. Jean-Baptiste et al. [START_REF] Jean-Baptiste | Intelligent assistive system using real-time action recognition for stroke survivors[END_REF] engineered an assistive system named CogWatch, which guides apraxic people during tea-making. The notion of task can also be understood as a daily plan composed of different Activities of Daily Living (ADLs). This can be seen in the Autominder project [START_REF] Rudary | Adaptive cognitive orthotics: Combining reinforcement learning and constraint-based temporal reasoning[END_REF], which was implemented as a scheduling system for the care of the elderly, using reinforcement learning. Each system monitors the user in a sensorized environment, tracks their behaviour and infers their potential need for guidance in order to retrieve adequate prompts. Indeed, in the context of rehabilitation, guidance should be provided via errorful learning technique [START_REF] Middleton | Errorless learning in cognitive rehabilitation: a critical review[END_REF]. Hence, we consider a prompt to be adequate when [START_REF] Peters | TEBRA -an automatic prompting system for persons with cognitive disabilities in brushing teeth[END_REF] it is a valid continuation of what the user has achieved so far (i.e., the user state), and (2) it is retrieved because an incorrect pattern in the user's behaviour (i.e., user's error) is detected by the system. To be able to achieve this, the system relies on an Artificial Intelligent Planning System (AIPS) that is composed of at least two modules: [START_REF] Peters | TEBRA -an automatic prompting system for persons with cognitive disabilities in brushing teeth[END_REF] the Action Policy Module (APM), which insures that prompts are a valid continuation of the user state, and (2) an Error Recognition Module (ERM), whose goal is to detect potential user's errors during the task. While the authors of Autominder, TEBRA and COACH all defined the probability models implemented in their AIPS's APM, no information is given about how the systems automatically detect, under uncertainty, if an error has been made by the user. Partially Observable Markov Decision Processes (POMDPs) provide a natural decision-theoric model for this problem [START_REF] Astrom | Optimal control of Markov decision processes with incomplete state estimation[END_REF][START_REF] Smallwood | The optimal control of partially observable Markov processes over a finite horizon[END_REF].

The contributions of this paper are as follows. First, we define two POMDP frameworks, and how we factorize them for ADLs management. We explain how the AIPS is trained to enable action planning via the APM, and human error detection via the ERM. Note that both APM and ERM should output their strategies based on their understanding of the user state. However, under uncertainty, the user state is represented by a belief state, which is a probability distribution over user states. Hence, the Belief State Space is infinite, and another challenge is to correctly associate a strategy to a belief state the system has not been trained for. In such a case, Nearest Neighbor Search (NNS) techniques can be applied by the AIPS to select the closest neighbor of the current belief state, and retrieve its corresponding strategy. Thus, in this paper, a specific NNS technique for ADLs management is also defined.

The paper is structured as follows. First we give an overview of the assistive system (i.e., CogWatch) in which our AIPS is implemented, then explain the technique used for its APM and ERM to fulfil their respective goals. Finally, we evaluate the POMDP-based AIPS performance via user simulation, compare it with a Markov Decision Process (MDP)-based AIPS, and discuss our future plans.

Assistive System Architecture

Figure 1 depicts the main modules composing CogWatch. It works as follows. First, the patient chooses the type of tea (e.g., black tea with sugar) (s)he would like to receive training for. Immersed in an instrumented environment, the patient moves sensorized objects at his/her disposal to perform actions (i.e., a u in the figure) related to the task. When moved, these sensorized objects generate data that are passed to a module called the Action Recognition System (ARS) whose aim is to recognise the patient's actions. Processing the data in real-time, the ARS outputs an observation o that it communicates to the POMDP-based AIPS. In a realistic environment, the ARS is prone to errors, and its observation o may differ from the real action a u made by the user. For example, when the patient adds a teabag into the cup, the ARS may believe that sugar has just been added, and pass this incorrect information to the AIPS.

Taking into account the uncertainty related to the ARS observations, the POMDPbased AIPS maintains probability distributions b s and b e , that it uses to select a prompt a * m which is composed of: (1) the best next action a * s the system considers the patient should follow to successfully continue the task, and (2) its best understanding of whether the patient has just made an error or not e * s . The AIPS prompt a * m is then passed to the Cue Selector, which displays, when necessary, the AIPS output in a way the patient can easily understand ãm (e.g., still images, video, recorded message). At this point, the patient can make new actions and enter in new cycles with the system until the task is completed.

Overview of POMDPs

Artificial Intelligent (AI) planning has often focused on MDPs where the environment is fully observable [START_REF] Russell | Artificial Intelligence: A modern approach[END_REF]. From the AIPS point of view, the environment corresponds to the user's behaviour and his/her progression through the task when observed via the ARS outputs. Thus, when based on a MDP, the AIPS makes the assumption that there is no uncertainty related to the actions a u made by the user; it considers that the observation o received from the ARS directly corresponds to the user's action a u . However, as explained previously, the ARS outputs do not always correspond to the real user's behaviours. Various reasons can explain this fact, for example: (i) The sensors used to gather information about the user's environment can be noisy; (ii) Many events beyond the system's control can arise, such as exogenous events that can cause the environment to change in an unexpected way. Thus, we must consider the environment to be only partially observable. POMDPs provide a rich framework for acting optimally in partially observable domains.

Formally, a POMDP is a tuple S, A, T , C, O, Z where S is a set of AIPS states; A is a set of prompts the AIPS can output; T defines a transition probability P (s | s, a m );

C defines the immediate cost c(s, a m ); O is a set of observations made by the ARS; Z defines an observation probability P (o | s , a m ).

The POMDP operates as follows. At each time step, the AIPS is in an unobservable state s ∈ S. As s is only partially known, the AIPS maintains a probability distribution over states, called the belief state b, with b(s) being the probability to be in state s. Taking into account the current belief state b, the AIPS selects a prompt a m ∈ A, then receives a cost c(s, a m ) for doing so. As the ARS detects another action a u from the user, the AIPS receives a new observation o ∈ O, which depends on the new unobserved state s and a m . This new observation allows the belief state b to be updated as follows:

b (s ) = P (o | s , a m ) s∈S P (s | a m , b, s)P (s | a m , b) P (o | a m , b) (1)
With Equation (1), and k being a normalization constant [START_REF] Kaelbling | Planning and acting in partially observable stochastic domains[END_REF], we obtain:

b (s ) = k • P (o | s , a m ) s∈S P (s | a m , s)b(s) . (2)

POMDP-based APM and ERM

In this work, the output of the AIPS is the optimal prompt a * m , with a * m = a * s , e * s , where a * s and e * s are respectively the optimal outputs the POMDP-based APM and ERM can select. As shown in Figure 1, the modules are distinct and base their planning on different representations of the belief state. In this section, we explain how the POMDP is adapted, applied and solved, for the APM and ERM to respectively estimate their belief states b s and b e , and to select their respective optimal outputs a * s and e * s .

POMDP's Adaptation

CogWatch is currently used to give assistance during tea-making. When interacting with the system, the user can choose between four types of tea for which training can be given. A part of our approach consists in adapting some parameters of the POMDP to fit the system's specifications: The Machine's Action Space A: The AIPS prompt is of the form a * m = a * s , e * s (i.e., a * m ∈ A), with a * s being the optimal action the APM can send to the user to guide him/her during the task, and e * s being the optimal ERM understanding of whether the user has made an error, each time an observation is received. Let A s be the user's Action Space, with A s = "Fill Kettle", "Boil Water", "Pour Kettle", "Add Teabag", "Add Sugar", "Add Milk", "Stir", "Remove Teabag", "Pour Cold Water from Jug into Mug", "Toy with the Kettle" and E the Error Space with

E = True, False . Hence, a * s ∈ A s , e * s ∈ E, and A = A s ∪ E.
The Machine's State Space S: This is a finite set of machine's states s. Taking into account CogWatch's specificities, these machine's states correspond to the user states. One of our contributions is to define the machine's states as sequences of actions a u . Thus s is defined by the tuple a u , s d , with:

-The user's action a u (i.e., a u ∈ A s ), being the most recent action really made by the user. -The history of the user state s d (i.e., s d ∈ S), being a sequence of actions representing what the user has really achieved. For example, s d = ["Add Teabag", "Fill Kettle", "Boil Water"]). Thus, if we consider the current user state to be s t , and the previous user state to be s t-1 , then s t-1 = s d and s t = s d .

The Cost Function C: C(b, a) is the cost for taking action a when in belief state b. As b is a probability distribution over the MDP states, we define C(b, a) such as:

C(b, a) = s∈S c(s, a)b(s), (3)
where c(s, a) is the cost for taking action a when in state s. As explained in [START_REF] Jean-Baptiste | Intelligent assistive system using real-time action recognition for stroke survivors[END_REF], c(s, a) is a mechanism to incorporate human judgment about the importance of different types of behaviour into the MDP.

The Observation Space O: In CogWatch, the Observation Space is defined such as O = A s . Indeed, the user's actions space A s has been defined based on the actions the ARS can detect.

Note that, both a u and s d are unobservable from the AIPS perspective. Moreover, in the rest of this paper, we refer to the machine's state space as the State Space; as the latter and the user state space are the same.

Belief State Representation

In the literature, when solving ADLs with a POMDP, the POMDP belief state is generally composed of multiple probability distributions over different parameters; the latter being related to variables such as users attitude (dementia level, responsiveness, awareness), and user's behavior [START_REF] Hoey | Assisting persons with dementia during hand-washing using a partially observable Markov decision process[END_REF]. For COACH [START_REF] Boger | A planning system based on Markov Decision Processes to guide people with dementia through activities of daily living[END_REF], this parametrization lead to over 22 million states; rendering explicit representation of the model infeasible. Here, we show how the belief state can directly be defined as a probability distribution over the underlying MDP state space.

Taking into account the Observation Space O, psychologists from our team defined a set of common errors cognitively impaired participants make during tea-making (e.g, sequence error, repetition error). This information is then used to restrict the State Space S, so it only contains a finite set of valid user states. We consider a user state to be valid if it does not contain any of the errors defined by psychologists. This technique allows us to maintain the MDP State Space and the belief states dimensionality at a manageable size.

From the APM's perspective The State Space S for "Black tea" contains 33 valid user states. Hence, the belief state b s that the APM uses to plan the best next action the user should perform, is a probability distribution over 33 states. From the ERM's perspective For the ERM to detect if the user makes mistakes during the task, we add an error state s e in S, with s e being an encapsulation of all user states that psychologists consider to not be valid. Let S e be this new State Space. In this case, the belief state b e used by the ERM is a probability distribution over the 34 states.

Typically, both belief states can be represented such as: (5)

b s = [b s (s 0 ), ..., b s (s 32 )], with 32 k=0 b s (s k ) = 1, (4)

Belief State Update

In the literature (e.g., COACH system), the ADL is solved using the standard definitions of the MDP or POMDP. Here, a factored POMDP is implemented. The factorization of the POMDP is a technique used in dialogue systems [START_REF] Young | The hidden information state model: a practical framework for POMDP-based spoken dialogue management[END_REF][START_REF] Williams | Factored partially observable Markov decision processes for dialogue management[END_REF]. In order to apply this technique to ADLs management, we modified the formulation of the factorization, and the formulation of the equation used to update the belief state. Given s = a u , s d , each part of Equation ( 2) can be expressed by Equations ( 6) and [START_REF] Smallwood | The optimal control of partially observable Markov processes over a finite horizon[END_REF]. Observation Function

P (o | s , a m ) = P (o | a u , s d , a m ) ≈ P (o | a u ) (6)
The observation function is the probability that the ARS makes the observation o given that the AIPS is in state s and previously selected the prompt a m . It is simplified, taking into account the fact that, in our system, the ARS observations only depend on the actions made by the user. In this form, the observation function corresponds to the ARS confusion matrix.

Transition Probability

P (s | a m , s) = P (a u , s d | a m , a u , s d ) ≈ P (a u | a m )P (s d | s d , a u ) (7)
The transition probability is the probability for the AIPS to be in state s , given that it was previously in s and selected prompt a m . After decomposition of s, we can simplify it by making some assumptions:

-The probability for the user to make a new action a u only depends on his/her compliance to the AIPS's prompt a m : P (a u | a m ). -The probability for the user to be in state s d only depends on his/her current action a u and state s d (s)he used to be in:

P (s d | s d , a u ).
The approximations ( 6) and ( 7), when substituted in (2), give respectively equations ( 8) and ( 9), for the APM and the ERM:

b s (s d ) = k • P (o | a u )P (a u | a m ) s d ∈S P (s d | s d , a u )b s (s d ) , (8)
and

b e (s d ) = k • P (o | a u )P (a u | a m ) s d ∈Se P (s d | s d , a u )b e (s d ) . (9)

Offline Training and POMDPs Solving

To solve a POMDP we need to find a policy (π * : B → A) between the Belief State Space B and the Action Space A, such that π * (b) minimizes the expected cost of task completion given belief state b.

In the case of the APM, the states of the MDP are sequences of actions that can lead to successful task completion, with the Belief State Space being B s and the Action Space A s . In the case of the ERM, an additional state s e is added to the MDP State Space, with the Belief State Space being B e and the Error Space E.

Both APM and ERM are trained offline with a user simulator in order to obtain their respective policy. A Monte Carlo (MC) Algorithm [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] is used for policy optimisation.

In the next sub-section, we give the general description of the optimisation algorithm implemented for our system.

Policy Optimisation

During training, a belief-state-action value function Q(b, a m ) records the immediate cost for taking prompt a m in belief state b, for all belief states contained in a Belief State Space B. The Q function is defined as the expected cost of a trial starting in belief state b, the AIPS taking action a m , and thereafter proceeding according to the current policy π until the trial ends and a final belief state is reached.

During each tea trial, a sequence of belief points-action pairs b, a m is recorded, and used to update the estimate Q(b, a m ).

At the end of the training, when the Q values are estimated, the optimal policy π * is obtained with:

π * (b) = arg min am Q(b, a m ), ∀b ∈ B, ∀a m ∈ A. (10)
The policy optimisation algorithm implemented is shown in Algorithm (1) for the APM training. Taking into account the specificities of the belief state used by the ERM, the algorithm used for its training can easily be inferred from Algorithm [START_REF] Peters | TEBRA -an automatic prompting system for persons with cognitive disabilities in brushing teeth[END_REF]. Note that, when respectively applied to the APM and ERM, equation [START_REF] Hoey | Assisting persons with dementia during hand-washing using a partially observable Markov decision process[END_REF] becomes:

π * (b) = arg min a Q(b, a), ∀b ∈ B s , ∀a ∈ A s , (11)
Training via User Simulation As many trials are necessary to robustly estimate the Q values, it is common practice that a simulated user is implemented to interact with the system during policy optimisation [START_REF] Levin | A stochastic model of human-machine interaction for learning dialog strategies[END_REF][START_REF] Thomson | Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems[END_REF][START_REF] Scheffler | Corpus-based dialogue simulation for automatic strategy learning and evaluation[END_REF][START_REF] Georgila | Learning User Simulations for information state update dialog systems[END_REF][START_REF] Schatzmann | A survey of statistical User Simulation techniques for reinforcement-learning of dialogue management strategies[END_REF][START_REF] Rieser | Cluster-based User Simulations for learning dialogue strategies[END_REF][START_REF] Quateroni | Combining user intention and error modelling for statistical dialogue simulators[END_REF]. Hence, we designed a Simulated User (SimU ) based on real patients data, able to generate actions based on the type of tea selected. The SimU 's structure is based on data from 52 control and cognitively impaired participants, aged between 21 and 82, who completed four types of tea (black tea, black tea with sugar, white tea, white tea with sugar). Its complete architecture has already been defined in [START_REF] Jean-Baptiste | Intelligent assistive system using real-time action recognition for stroke survivors[END_REF].

Algorithm 1 MC policy optimisation algorithm -APM

Evaluation

Once the policies π * : B s → A s and ε * : B e → E are obtained, they are used online to evaluate the POMDP-based AIPS performance. We can define the AIPS performance as its ability to retrieve semantically optimal actions when necessary. In such a case, an action is considered to be semantically correct when it is a valid and non redundant continuation of the current user state s. It means that from the user's point of view, a * s should be one more step toward a successful completion of the tea (s)he is trying to make. Moreover, this prompt should only be given when necessary. This means that the AIPS should output a * m only when it "believes" the participant has just made an error, (i.e., when e * s =T rue). Evaluating a system such as CogWatch requires many parametrizations, and is therefore complicated. Moreover, as the system is composed of distinct modules (i.e., the ARS, the AIPS, the Cue Selector); joint evaluation of the complete system is a challenge. Even focusing on the evaluation of the AIPS only would not decrease the complexity of the task. Indeed, the tea trials can be performed in various ways; which means that there is a vast space of user states to potentially evaluate. One approach to evaluation is to let patients interact with the system. This has been done with the MDP-based system [START_REF] Jean-Baptiste | Cogwatch: Intelligent agent-based system to assist stroke survivors during tea-making[END_REF]. However, it is not reasonable to have cognitively impaired patients intensively interacting with the system to solely evaluate the impact of different parameters on the system's performance. Hence, an alternative is to let a simulated user interact with the AIPS immersed in a simulation of the other components. This will enable a wider coverage of user space, and the possibility to analyse the main elements in the AIPS structure that have an impact of its ability to fulfil its goals. User simulation is a common way to evaluate POMDP techniques [START_REF] Thomson | Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems[END_REF]; it is known to provide a useful basis for comparing systems [START_REF] Young | The hidden information state model: a practical framework for POMDP-based spoken dialogue management[END_REF]. The risk with using a Simulated User is that it does not properly capture real users' behaviours. However, our SimU is based on real user data, and in experiments, the task completion rate for the SimU and real users is similar [START_REF] Pflugler | Using human-computer interface for rehabilitation of activities of daily living (ADL) in stroke patients. lessons from the first prototype[END_REF].

As one can see in Figure 2, the evaluation works as follows: At the beginning of each trial (i.e., at t = 0), the SimU is in state s 0 =∅ (the empty set). From the APM and ERM's perspective, it means that when the SimU begins a trial, there is no action contained in its history; the state s 0 is empty. The SimU then selects an action a u that is passed to a virtual ARS. The latter is based on the behaviour of the real ARS. When the virtual ARS outputs an observation o, both b s and b e are automatically updated. The policies obtained after training are then used, allowing the AIPS to output a * m = a * s , e * s . If the belief states updated b s and b e are already contained in their respective policies M s and M e , their optimal actions are already known. On the other hand, if a belief state updated has not been visited during training and is not contained in the policy, a Nearest Neighbor Search (NNS) technique is applied to identify which neighbor should be selected (i.e., see b in M e ). The APM or ERM then associates the policy of this neighbor to the current belief state.

For evaluation, two main scenarios are implemented. In the first scenario, if e * s = T rue, a m is passed to the SimU , which outputs another action a u such as a u = a * s ; if e * s = F alse, a m is not passed to the SimU , which outputs another action a u based on its intrinsic settings. The AIPS performance is then calculated with:

P hits = P CR + P (a SC |e u , e s ) , (13)
with P CR the probability for the ERM to correctly detect that the user made no error; P (a SC |e u , e s ), the probability that the APM retrieves a semantically correct action a SC (a SC ∈ A s ) given that the user made an error e u and that the ERM detects that an error has been made e s (e s ∈ E). In the second scenario, the ERM is ignored, and the ability of the APM to correctly guide a 100% compliant SimU is measured. In this case, the APM constantly tells the user what to do. Before reporting on the POMDP-based AIPS performance, and analysing the potential impact of NNS techniques on the latter, a short overview of the NNS techniques used during evaluation is given in the next section.

Nearest Neighbor Search

During evaluation, the system may generate a belief state that it has never seen during training. In such a case, it will have to select a neighbor of this belief state in order to retrieve its corresponding optimal output a * m . This means that the choice of the metric used to select neighbors during evaluation may have an impact on the quality of the output. There are many metrics available to measure the distance between probability distributions. Some are not true metrics, but provide a notion of distance which have been proven useful to consider [START_REF] Gibbs | On choosing and bounding probability metrics[END_REF][START_REF] Chung | Measures of distance between probability distributions[END_REF]. In this paper, we will show that the AIPS performance does not only depends on the ARS error rate, but also on the metric used to select neighbors during evaluation. Hence, we will compare the impact of the correlation distance [START_REF] Pearson | Notes on the history of correlation[END_REF], euclidean distance [START_REF] Marie | Encyclopedia of Distances[END_REF] and a technique we developed and named SciMK.

SciMK measures the similarity of two belief states by comparing the underlying patterns of their MDP strategies. As explained previously, in our system, a belief state is a probability distribution over the MDP states. As each of these states can be associated to a strategy a * s after solving the MDP [START_REF] Jean-Baptiste | Intelligent assistive system using real-time action recognition for stroke survivors[END_REF], each belief state can be transformed into its corresponding vector of MDP strategies. An invariant numerical label (NL) can then associated to each MDP strategy to facilitate the comparison between vectors of MDP strategies. The process is described in Figure 3: -Step (1), the belief state is represented as a probability distribution over the MDP states. -Step (2), the belief state is represented as a probability distribution over the MDP strategies corresponding to each MDP states from step (1). -Step (3), the belief state is represented as a vector of numerical labels (i.e., weights), each of the latter corresponding to a specific MDP strategy.

For example, in step (3), π * (state 2 ) = π * (state N ) and their corresponding numerical label is 10. It means that, based on the results found after solving the MDP, both states Fig. 3. Belief state transformation state 2 and state N are similar to the point where the same next best action should be performed after them. Contrary to the euclidean or correlation distance, when using SciMK, we do not compare the current vector of NLs with all the elements contained in the Belief State Space. First, each belief state contained in the Belief State Space is transformed into its corresponding vector of NLs, then the Belief State Space is separated in different clusters, with each cluster being a group of vectors of NLs which had their highest probability associated to the same MDP strategy before transformation. For example, let's consider that π * (state 1 ) ="add teabag". As one can see, state 1 has the highest probability in the belief state -step (2) in Figure 3. Thus, the vector of NLs in step (3) will only be compared to other vectors that had, in step (2), their highest probability also associated to the same strategy as the one found for state 1 -"add teabag". Note that after solving the MDP, different states can be related to the same strategy.

Results and discussion

In the first scenario, we let the SimU performs the task (i.e., black tea) 300 times at varying ARS error rate. Recall that the SimU complies to the AIPS's APM best next action only when the ERM detects that an error is made. As expected, Figure 4 shows that the POMDP outperforms the MDP-based AIPS, and the ability of the AIPS (both MDP and POMDP) to output correct prompts at the right time decreases when the ARS error rate increases. Indeed, the more the ARS outputs wrong observations, the more difficult it is for the AIPS to output correct prompts. Note that at high ARS error rate (i.e., 30%), it is more advantageous to have an ERM using the correlation distance to select neighbors than the euclidean distance. On the same figure, it also seems that the percentage of hits is independent from the NNS technique used by the POMDP-based APM. However, this can be explained by our definition of P hits . The latter is intrinsically dependent on the ERM ability to detect that an error has been made or not. If the ERM does not succeed to properly detect when errors really occur, there will be few occasions to measure P (a SC |e u , e s ) from Equation [START_REF] Williams | Factored partially observable Markov decision processes for dialogue management[END_REF]. If we only focus on the ability of the ERM to correctly detect whether errors have been made or not, we can define:

P ERM hits = P T P + P CR , (14)
with P T P the probability for the POMDP-based ERM to detect that errors have really been made (True Positive), and P CR same than in Equation [START_REF] Williams | Factored partially observable Markov decision processes for dialogue management[END_REF]. In this case, one can see in Figure 5 -(1) that the POMDP-based ERM barely outperforms the ability of the MDP-based ERM to detect errors under uncertainties. Moreover, at high ARS error rate (i.e., 30%), when the ERM uses the euclidean distance to select neighbors, the POMDP-based ERM makes more mistakes than the MDP-based one. These results highlight the fact that the POMDP framework implemented for the ERM needs to be improved. Instead of having only one state s e that encapsulates all types of errors made, one improvement would be to extend the belief state b e used by the ERM for it to take into account more error states. (N.B., Recall that, based on the MDP-based AIPS definition, the latter is not dependent on any NNS techniques [START_REF] Jean-Baptiste | Intelligent assistive system using real-time action recognition for stroke survivors[END_REF].)

To have a better understanding of the APM performance only, the second scenario is implemented . In the latter, the AIPS always delivers the best next action it believes the SimU should perform, whether an error has been made or not. Moreover, the SimU is set to always comply to the AIPS prompts. Thus, we measured the ability of the APM to help the SimU successfully complete the tasks at varying ARS error rate. Figure 6 shows the SimU success rate when performing black tea (1), black tea with sugar (2) and white tea with sugar (3). The horizontal line labeled "0% compliance" corresponds to the SimU success rate when performing the task totally by itself, so when ignoring the whole system. The results show the impact of the NNS techniques applied by the APM to choose the right neighbors during evaluation. When using SciMK to select neighbors, the APM retrieves correct prompts to the user more often than when using the correlation or euclidean distance. One can also see that the POMDP-based system always outperforms the MDP's ability to guide the SimU ; except in (3), when the APM uses the euclidean distance at 30% ARS error rate. In this specific case, it is also interesting to note that the POMDP-based APM using the euclidean distance makes the SimU fails the task more often than when the latter performs it without guidance (i.e., 0% compliance). Indeed, when performing white tea with sugar without guidance, the SimU 's success rate is 10%, while it is close to 0% when the APM selects its neighbors with the euclidean distance, at 30% ARS error rate.

From a general point of view, Figure 6 also shows the impact of the complexity of the task on the ability of the APM (MDP and POMDP) to guide the SimU properly. As explained previously, the dimensionality of the belief state b s for black tea is 33. As this dimensionality increases with the number of mandatory steps required to complete a task, the dimensionality of b s is 205 for black tea with sugar, and 1539 for white tea with sugar. To have a belief state being a probability distribution over 1539 MDP states means that, each time the SimU selects an action, the POMDP-based APM potentially considers that the SimU is in 1539 different states at the same time. The more the belief state dimensionality increases, the more complicated it is for the APM to output correct prompts. One solution to this issue would be to focus on size reduction of the MDP State Space, by merging similar MDP states based on the pattern of actions they are composed of. This dimensionality leverage of the MDP State Space will then decrease the computational burden from the POMDP-based AIPS side.

It would be interesting to compare the performance of the CogWatch AIPS with other different systems' AIPS. However, it is a very complicated task. Indeed, even if CogWatch, TEBRA and COACH focus on sequential ADLs, none shares the same POMDP framework (Action Space, State Space); all systems have been implemented to deal with different tasks; and all these tasks have been assessed in a different way by all authors; and no information is given about the NNS techniques applied by COACH or TEBRA's AIPS. Moreover, the AIPS performance highly depends on the system's ARS's error rate, which is also different in all systems. This is partially due to the fact that TEBRA, COACH and CogWatch implement their own ARS's algorithms: our system integrates a Hidden Markov Model-based action detector, TEBRA -a Bayesian network classifier, COACH -a color based flocking tracker.

Conclusion

In this paper, we presented the Partially Observable Markov Decision Process (POMDP) based Artificial Intelligent Planning System (AIPS) integrated in an assistive system designed for cognitively impaired participants. This system's aim is to automatically guide and assist them during Activities of Daily Living (ADLs), such as tea-making. We explained that the AIPS's goal is to provide optimal guidance, and help the user during the task when necessary. The structure of the AIPS is given, as well as the framework of its two main modules : the Error Recognition Module (ERM), whose aim is to detect human's error under uncertainty, and the Action Policy Module (APM), which has to plan optimal prompts to help the user succeed the task. Deployed in a sensorized environment, we described how the POMDP-based AIPS copes with uncertainties, and compared its performance with a Markov Decision Process (MDP)-based AIPS in the same scenarios. We reported that the POMDP-based AIPS outperforms the MDP under uncertainty if parametrised properly. We also compared different POMDP-based AIPS with the MDP, to have a better understanding of our framework strengths and limitations. The advantage of the POMDP is its ability to model uncertainties. Each time the POMDP-based AIPS receives an observation, it updates a belief state which is a probability distribution over states. This allows the AIPS to select a prompt to guide the user that will cost the less considering all the potential states the user might be in. However, we showed that the performance of a POMDP-based AIPS and a MDP-based one depends on the intrinsic parameters of its APM and ERM, and on the complexity of the task. Our future plan is to modify our current POMDP model in order to enable our AIPS to not only detect when the user needs assistance, but what exact type of errors that may have been made during the task. We will also work on state space leverage to improve the system's performance.

Fig. 1 .

 1 Fig. 1. CogWatch's architecture.

  b e = [b e (s 0 ), ..., b e (s 32 ), b e (s e )], with 32 k=0 b e (s k ) + b e (s e ) = 1.

  and ε * (b) = arg min e Q(b, e), ∀b ∈ B e , ∀e ∈ E.

  set of machine's actions a Q(b, a) : expected cost for selecting a when in b N (b, a) : number of times a is selected when in b Bs : initial set of belief states b π : initial policy, with π : Bs → A repeat k ← 0 b = b0 (probability 1 to be in the initial state s0 = ∅) Generate a tea trial with the Simulated User: repeat k ← k + 1 Get Simulated User's output a u,k and update b. a k ← π(b) or π(bn) with bn ← nearest neighbor of b in Bs. record b k , a k , c(b k , a k ) , K ← k until the Simulated User terminates the trial Scan trial, update B and Q for r ∈ [0, K] do C ← sum of the costs c(br, ar) from br to bK . if ∃br ∈ Bs and | b k -br | < then Q(br, ar) ← Q(br ,ar ) * N (br ,ar )+C N (br ,ar )+1 N (br, ar) ← N (br, ar) + 1 else create a new point bw in Bs initialise Q(bw, ar) ← c(bw, a k ) end if end for Update the policy π(br) = arg mina Q(br, a), ∀br ∈ Bs, ∀a ∈ A. until converged

Fig. 2 .

 2 Fig. 2. Diagram of the system used to evaluate the AIPS.

Fig. 4 .

 4 Fig. 4. Percentage of AIPS Hits at varying ARS error rate and NNS techniques. (1) ERM using correlation distance, (2) ERM using euclidean distance.

Fig. 5 .

 5 Fig. 5. Percentage of ERM Hits at varying ARS error rate and NNS techniques. (1) ERM using correlation distance, (2) ERM using euclidean distance.

Fig. 6 .

 6 Fig. 6. SimU success rate at varying ARS error rate and NNS techniques. (1) Black tea, (2) Black tea with sugar, (3) White tea with sugar.