Benoit Leduc

Keywords: ADL Activity of Daily Lining, AI Artificial Intelligence, BN Batch Normalization, BPE Byte Pair Encoding, CAM Class Activation Map CLA Cortical Learning Algorithm, CNN Convolution Neural Network, CRF Conditional Random Fields, DBN Dynamic Bayesian Networks GRU Gated Recurrent Unit, GUI Graphical User Interface, HAR Human Activity Recognition, HHMM Hierarchical Hidden Markov Model PIR Passive Infrared, ReLU Rectified Linear Unit, ResNet Residual Networks, RNN Recurrent Neural Networks, SDAE Stacked Denoising Autoencoder

J'aimerais tout d'abord remercier mon encadrement en commençant par mon directeur de thèse, Ioannis Kanellos, Professeur à

RÉSUMÉ EN FRANÇAIS

La maison intelligente est, de nos jours, au centre d'attentions importantes en matière de recherche et de développement, notamment en ce qui concerne les nombreuses possibilités d'applications et de services innovants qu'elle peut offrir en termes de sécurité, d'économie d'énergie, d'amélioration du confort et d'aide à la santé des habitants. Ces dernières années ont ainsi vu émerger une multitude de techniques et d'approches qui prennent systématiquement appui sur le paradigme de l'intelligence artificielle afin de doter les maisons d'aujourd'hui de ces applications et ces services. La capacité à comprendre le mode de vie et les activités de ses résidents, au travers des capteurs domotiques, est devenu un domaine stratégique. Il stimule désormais d'intenses recherches. En effet, pour pouvoir proposer à ses occupants des services adaptés à leurs situations de vie et répondre à leurs besoins, comprendre le contexte ambiant d'une maison est essentiel.

Cette thèse se propose de contribuer au domaine scientifique et technique actuel de la reconnaissance de l'activité humaine dans une maison intelligente au travers de capteurs domotiques.

Elle tente, plus spécifiquement, à apporter une réponse à la question suivante :

Comment reconnaître précisément les activités quotidiennes des habitants dans une maison au moyen de capteurs domotiques non-intrusifs ?

Dans le premier chapitre nous proposons une courte introduction à cette thématique et aux défis qu'elle procure. Nous y définissons et montrons notamment la complexité que peut avoir l'activité humaine.

Dans un second chapitre nous proposons un état de l'art plus ciblé sur les systèmes de reconnaissance d'activité humaine pour en relever les enjeux tant scientifiques que techniques et ainsi donner une claire assise à nos propositions. Nous nous y attachons, en particulier, à présenter comment le flux de capteurs est segmenté, représenté et codé. Nous discutons aussi des méthodes de reconnaissance des formes et d'interprétation des données temporelles pour reconnaître des activités à partir de données de capteurs, ainsi que des jeux de données pour la reconnaissance d'activités. Enfin, nous cherchons à identifier certains problèmes ouverts et des questions liées à ce sujet.

Dans un troisième chapitre, nous décrivons l'approche préconisée, le contexte et la méthodologie expérimentale utilisée dans les travaux menés. Nous présentons le codage du flux de capteurs utilisé et les raisons de son choix et nous détaillons les modèles de base qui servent pour nos comparaisons, la préparation des jeux de données, les métriques de comparaison et, enfin, la méthode d'évaluation.

V Dans le quatrième chapitre, nous explorons les questions de reconnaissance des formes en ce qui concerne la reconnaissance des activités. Nous évaluons une première approche de la représentation et de l'extraction de caractéristiques qui utilise des méthodes issues des domaines du Traitement Automatique du Langage Naturel et de la Classification de Séries Temporelles.

Nous expliquons la nécessité de modéliser la reconnaissance d'activité dans une maison comme un problème plus complexe qu'un simple problème de reconnaissance de formes et la valeur ajoutée espérée par ces idées.

Dans le cinquième chapitre, nous développons des méthodes pour améliorer la représentation des données de capteurs par des techniques de projection dans un espace latent avancé. Le contexte dans lequel un capteur est activé fournit des informations précieuses sur l'activité en cours. Tout comme un mot peut avoir un sens différent selon les autres mots de son contexte proche (dans une phrase, par exemple), l'activation d'un capteur peut avoir un sens différent selon les autres capteurs de la même trace. L'inspiration de notre approche par des méthodes avancées du Traitement Automatique du Langage Naturel vise, justement, à extraire une représentation contextualisée de ces capteurs afin d'aider les modèles à comprendre dans quel contexte d'activité l'activation d'un capteur se produit. Dans cet ordre d'idées, la méthode implémentée adresse une technique de modélisation du langage capable de contextualiser les mots dans une phrase.

Dans le sixième chapitre, nous présentons une série d'études utilisant soit, le contexte temporel, ou le contexte provenant d'activités précédentes, pour améliorer les modèles de reconnaissance d'activité. Les activités humaines sont souvent liées à l'heure de la journée et aux activités précédentes et s'organisent en patterns plus globaux. Nous proposons d'ajouter une représentation de ces connaissances pour aider les modèles à mieux discriminer les activités qui sont similaires en termes d'activation de capteurs ou qui se produisent à des moments particuliers de la journée. Due to cost reduction and recent advances in sensors, Internet of Things (IoT), and communication technologies have transformed our environment into an ecosystem of interconnected objects that can communicate, collaborate, and capture data. These technologies have stimulated the development of intelligent environments. They are making our transportation systems, cars, factories, hospitals, offices, homes, cities, and personal devices smarter, more efficient, more eco-friendly, and more comfortable.

LIST OF FIGURES

A great deal of research into these intelligent environments is underway for their potential to provide a wide range of innovative applications and services. Smart homes are part of these intelligent environments and have been increasingly studied and developed these last few years for the many possible applications that they can offer in terms of security, energy saving, comfort improvement, and health support.

The Smart Home Concept

The house of tomorrow is already designed as a "smart house" but smart home technologies have a much longer history than one might imagine. The idea of smarter homes in terms of comfort and convenience dates back to at least the 1890s and early 1900s, when wealthy individuals took advantage of the advent of electricity to create more automated, luxurious, relaxing, and comfortable homes [15]. Nowadays, a smart home is a house containing home automation devices, such as sensors and actuators, that monitor and control its features. These devices create a technological ecosystem interconnected by means of specific communication protocols [24] on which programmed algorithms or scenarios can rely.

Originally, smart home technology was used to control environmental systems such as lighting and heating, but recently, the technology has expanded so that almost every electrical component in the home can be included in the system. Appliances such as dishwashers, washing machines, refrigerators, etc. are becoming connected and equipped with a variety of sensors making them more intelligent. They can thus send back information or be controlled by the system. Moreover, today our various entertainment objects such as televisions, audio systems, etc. and external house objects such as cars, can also be integrated into the system allowing new interactions and possibilities. Today every object of our daily life could be integrated into this ecosystem.

Introduction

Smart home technology not only turns on and off lights, appliances, or objects on demand.

Today thanks to Artificial Intelligence (AI), it can monitor the internal environment and any activities undertaken when the house is occupied. Moreover it can make decisions in regards to observations taken by the system. As a result, a smart home could be able to "perceive" and "understand" its environment as well as the activities and changes that occur within it, thanks to sensors of various kinds (temperature sensor, motion sensor, magnetic sensor, pressure sensor, etc), installed in strategic places (motion sensor above a bed, magnetic sensors on medicine cabinet door, etc) or in everyday appliances and objects.

Once synonymous with complicated installations and inconsiderate expenses, home automation has changed to be a far simpler and easier addition to the modern home. It is increasingly democratized today by technological advancements such as voice assistants (Alexa, Google, Siri, ...), and new solutions have put comfort, security, and energy efficient advancements within the reach of more people than ever before. The home of tomorrow should not only be a place of residence, but a true platform of services and experiences for the user.

A Smart Home For Saving Energy

One of the great challenges facing the world today is global warming; which, could be partly solved by reducing and better using energy resources. A recent report released in 2020 by Energy Outlook [34], an energy intelligence and consulting firm that collects and publishes data and reports on the state of global energy consumption, shows a 2.9% increase in global energy consumption in 2018. This is considered the fastest rate of increase in decades from its usual primary annual average of 1.5% and is expected to rise rapidly in the coming years. After industries, we see that cities, residential areas, and houses are important points of energy consumption.

Today's houses are increasingly designed to be more insulated in order to easily regulate and limit the energy consumption of temperature management systems. Moreover, household appliances are less and less energy consuming. However, despite these efforts. Unfortunately, energy consumption in homes is still high, partly due to poor thermal and behavioral management on the part of residents.

The smart home has become a new option to mitigate this consumption. Thanks Artificial Intelligence (AI) it is not only able to automatically regulate the temperature but also to advise and guide the residents on better consumption. A lot of research is underway to allow the house of tomorrow to be less energy consuming through the use of smart home technologies.

A Smart Home For Better Health

Nowadays, the issue of personal dependence has become one of the great global challenges for public health due to the aging of the population and the increasing cost of medical institutions [12]. However, the loss of dependence does not only affect the elder population, but any person facing -temporary or permanent -reduction in capacities (physical, functional, cognitive, social, emotional,...) [32]. Most often, it translates into a difficulty in carrying out simple tasks, such as cooking, taking medication, going to the toilet, etc., tasks that can quickly make daily life painful, even costly, both in terms of personal or familial economy and, on a larger scale, in terms of national medical institutions. In France, for example, a new part of the Social Security system was created in 2021 to cover the personal dependence cost. The cost of caring for dependent persons represents 30 billion euros per year, or 1.4% of the Gross Domestic Product (GDP) in France [22].

Providing automated services to enable dependent people to live as independently, comfortably, and healthily as possible in their own homes, while limiting the risks associated with their in domo activities, is a necessity. It has already opened up an unprecedented economic field which is trying to find solutions through intelligent environments [3,31]. Today, the issue of personal dependence is looking for technical solutions by hopefully drawing from the paradigm of Artificial Intelligence (AI) and smart homes to turn future houses into Ambient Assisted Living (AAL) places.

Recognition of Human Activities: Concepts, and Approaches

In order to provide adequate services and applications, a smart home must "understand" and "interpret" the daily routines of its residents. It is therefore necessary to develop ad hoc human activity recognition techniques, models, and algorithms to track and analyze the behavior of one or more people to deduce their activities. However, these techniques face many challenges, linked to not only the very complexity of human life, but also of the human environment. interweaving, or even overlapping. It can be performed by one or more persons and is longer than an action (Figure 1.1a). The more complex the activity is, the greater the number of actions and operations that make it up. Moreover, it is possible that an activity is composed of only one action. For example, we can consider the activity or action of "walking", which makes it difficult to define the boundaries between these two terms.

Humans can performs a very wide range of possible activities that can be grouped in many concepts or families of thought. The concept of Activity of Daily Lining (ADL) was proposed by Katz [18] and then extended by Lawton and Brody [19] to add the concept of instrumented ADLs. This concept refers to human activities that are performed at home daily such as, "bathing", "dressing", "grooming", "work", "eating", etc. This term is used in particular in healthcare to refer to people's daily self-care activities. Healthcare professionals often use a person's ability or inability to perform ADLs as a measurement of their functional status. Eat The way in which one imagines, or conceives an activity influences the description and /

or the division of it. The more generalized the description or vision, the greater the number of actions or operations that comprise it. The activity "watching television" for example, can be viewed in a very general way as "being seated in front of the television" but also, can be divided in more elementary actions like "taking the remote control", "turning on the television", "sitting down", "selecting the program". Even from there it is still possible to go down in level of granularity of the description of the activity by dividing certain actions even more which then become operations. For example, "taking the remote control" which can be cut down to

Introduction

"seek the remote control with a glance", "walk towards the remote control","pick up the remote control", etc. Describing an activity based on actions, or even operations, can be very complex and depends very much on the level of abstraction or granularity that we choose to describe it.

Once the level of granularity is chosen , it must be taken into account that a person can perform one or more activities in a sequential (Figure 1.1b), intertwined (Figure 1.1c), concurrent, or in a parallel manner. Concurrent and parallel activities seem very similar, but there exists small differences between the two descriptions. Concurrency means executing multiple activity at the same time but not simultaneously, e.g. wash dishes while cooking, see Figure 1.1d. In general, it refers to activities that a resident wants to perform simultaneously but cannot and must alternate between them. Parallel activities means executing multiple activities simultaneously, e.g. watching a TV while eating, see Figure 1.1e.

Moreover, many people do not necessarily live or spend all their time alone. A dwelling may have several people engaged in one or more activities independently, collaboratively, or in parallel (Figure 1.1f Another factor that must be taken into consideration is the complexity of human activity and that there are a large number of variations for performing the same activity. The same person will not necessarily perform an activity in the same way, nor will it necessarily be performed in the same way by another person. The possibilities become even more numerous when several people collaborate on the same activity. All this shows that human activity is very difficult to describe, interpret, model, and recognize.

Human Activity Recognition Approaches

Monitoring a resident's behavior in a home is a crucial task and involves the use of particular human activity recognition approaches. Generally, human activity recognition systems must capture contextually relevant information to infer an actor's activity, based on the data and data type. Depending on the manner and type of data from these monitoring facilities [17],

there are currently two main approaches to recognizing activities [5,10]: vision-based systems and sensor-based systems (see Figure 1.3).

Human Activity Recognition Vision Based Sensor Based

Wearable Sensor on Objects Ambient Sensors

Figure 1.3 -Human Activity Recognition approaches

Vision Based

The vision based human activity recognition uses cameras to track human behaviour and changes in the environment. This approach uses computer vision techniques, e.g. marker extraction, structure model, human body detection, motion segmentation, action extraction, and motion tracking. Researchers use a wide variety of cameras, from simple RGB cameras, to more complex systems with a fusion of several cameras for stereo vision, or depth cameras able to detect the depth of a scene with infrared lights. This domain has been an important research axis for a long time, due to the importance of fields such as human-machine interactions [16], robotics [26],

and security [29]. Several survey papers on vision based activity recognition have been published [1,10]. Beddiar et al. [1] aims to provide an up-to-date analysis of vision based human activity recognition literature and recent progress.

However, these systems pose the question of acceptability. A recent study [30] shows that the acceptability of these systems depends on users' perception of the benefits that such a smart home can provide. It also affects their concerns on the monitoring and sharing of their data.

This study shows that older adults (ages 36 to 70) are more open to tracking and sharing data, especially if it is useful to their doctors and caregivers. Meanwhile, younger adults (up to age [START_REF] Ali Hamad | Dilated causal convolution with multi-head self attention for sensor human activity recognition[END_REF] are rather reluctant to share information. Based on this observation we argue for the use of less intrusive systems, such as smart homes based on IoT sensors.

Sensor Based

human activity recognition from sensors consists of using a network of sensors and connected devices to track a person's activity. The sensors produce data in the form of a time series of state changes or parameter values. A wide range of sensors -magnetic contact detectors, RFID, accelerometers, motion sensors, noise sensors, radar, etc.-can be placed directly on a person, on objects, or in the environment. Thus, the sensor-based solutions can be divided into three respective categories: Wearable [27], Sensor on Objects [21] and Ambient Sensor [14].

Given the privacy concerns of residents and the reluctance of both individuals and legislators to install cameras in what is considered private space, ambient sensor-based human activity recognition systems seem to dominate the smart home research field [8,17,2]. Indeed, ambient sensors are generally considered less intrusive and, therefore, are better accepted.

Figure 1.4 -The richness of the sensors versus a user's perceived privacy.

The Figure 1.4 taken from Debes et al. [11] illustrates the richness of the sensors versus the perceived privacy of a person using these sensors. Clearly, sensors that provide rich information about a person's activities are not generally perceived as privacy-preserving. Sensors providing the least amount of information are therefore generally preferred to equip smart homes. This adds additional difficulties and challenges for human activity recognition systems, as we will see in the next section.

Introduction

The Challenges Regarding Human Activity Recognition in Smart Homes

In a smart home, the sensors capture into traces the operations, actions, and interactions of the resident(s) within the house, see Figure 1.5. For example, the presence or entrance of a resident into a room can be captured by a motion sensor switching from an "off" state to an "on" state.

human activity recognition in smart homes thus consists of translating sensor traces into ADLs.

Although simple in its formulation, the recognition of ADLs remains a task with many challenges, both for hardware and the system's algorithms, in terms of design and generalization.

... The number of residents living at the same time in the same house adds other difficulties.

The more residents there are, the more the activation traces of the sensors corresponding to different activities are intertwined. For example, if one resident is "cooking" while another is "showering" at the same time, the changes in sensor states that correspond to each activity by one and the other tend to mix. In addition, even if a resident lives alone, other residents on temporary visits may perform activities while present.

Furthermore other challenges arise from the very choice of sensors. As explained in the previous section, ambient sensors, unlike cameras, provide little information. A motion sensor, as its name indicates, will only give the binary information of the existence or nonexistence of a movement in the space it monitors. The activation of each sensor, independently, therefore gives little information about the activity in progress. For example, the activation of the motion sensor in the kitchen may indicate activities such as "cooking", "washing the dishes", "cleaning", etc.

Thus, the information offered by this sensor is not very precise, it can only be used in conjunction with the activation of other sensors to make sense aka by the context of the data. Indeed, it is the context in which a sensor is activated that gives the information meaning.

Moreover, human activity recognition models for smart homes should also look for the context related to the room, the objects, the devices of the house used, and even the type of interactions during ADLs to understand and discriminate them. For example, opening the front door to enter the house and opening the front door to leave the house activates the same type of magnetic sensor, on the same type of object, with the same interaction, but transcribes a different activity.

Furthermore, these sensors are in general triggered by events related to the action of the resident, or an environment change. This generates temporal series that are not regularly sampled (temporally sparse), which makes temporal interpretation difficult and requires, a different treatment from classical temporal series. The activation of two consecutive sensors at an interval of one second or one hour can transmit very different information.

Finally, there is a problem related to labeling sensor activations as activities. Indeed, in order to train a human activity recognition system in a smart home, it is necessary to assign to each sensor activation a label of the activity during which it appears. However, it is often difficult to define a clear cut boundary between the end of an activity and the beginning of a new one.

Moreover, the choice of the name of the activity is very dependent on the person in charge of the annotation. For example, the activities "sleeping in a bed at night", or "sleeping on the couch for a nap", can be differentiated by similar labels ("Sleeping" and "Resting") or not ("Sleeping").

In general, a set of activity labels is predefined, such as the ADLs list from Katz [18]'s work for example. Which implies that some activities performed by the resident(s) remain unknown and could be very similar in terms of transferring a sensor's activation traces to labeled activities.

It is therefore challenging for human activity recognition systems to find the correct features to discriminate not only all labeled activities between them, but also all unknown ones.

In view of these problems of adaptability, imprecision, robustness, normalization, reliability, etc. of the data, human activity recognition systems in smart homes are continually challenged in terms of pattern recognition and temporal sequence analysis [2]. In an attempt to address all of these challenges and difficulties, researchers are studying different methods for human activity recognition in smart homes.

Knowledge Driven Activity Recognition

In KDA methods, an activity model is built through the incorporation of rich prior knowledge gleaned from the application domain, using knowledge engineering and knowledge management techniques. KDA methods are motivated by real-world observations that involve ADLs and lists of objects required for performing such activities. In real life situations, even if the activity is performed in different ways, the number of objects and types of objects involved do not vary significantly. For example, the activity "brush teeth" may contain actions involving a "toothbrush", "toothpaste", "water tap", "cup", and a "towel". Even if a version of the activity does not include all of the actions such as "cup" or a "towel" it will still include most actions and several key actions such as "toothbrush" and, "toothpaste" will occur every time.

KDA are founded upon the observation that most activities, specifically, routine ADLs, take place in a relative circumstance of time, location, and space. For example, "brushing teeth" is normally undertaken twice a day in a bathroom in the morning and before going to bed and involves the use of "toothpaste" and "toothbrush". Thin implicit relationships between activities, related to temporal and spatial context and the entities involved, provide a diversity of hints and heuristics for inferring activities. The knowledge structure is modeled and represented through forms such as schemas, rules, or networks. KDA modeling and recognition intends to make use of rich domain knowledge and heuristics for activity modeling and pattern recognition. Three sub approaches to KDA exist: mining based approach [28], logic-based approach [9] and ontology based approach [4].

Ontology based approaches are the most commonly used, as ontological activity models do not depend on algorithmic choices. They have been utilized to construct reliable activity models.

Chen and Nugent [5] have proposed an overview of this in their work. Yamada et al. [33] use ontologies to represent objects in an activity space. Their work exploits the semantic relationship between objects and activities. A "teapot" is primarily used in the activity of "tea preparation"

for example. This approach can automatically detect possible activities related to an object. It can also link an object to several representations or variability of an activity. Chen et al. [6,4,7] constructed context and activity ontologies for explicit domain modeling.

KDA provide the advantage to formalize activities and propose semantic and logical approaches. Moreover, these representations try to be as complete as possible to overcome activity diversity. However, some of the key limitations of these approaches are the complete domain knowledge requirement needed to build activity models, and the approaches' weakness when it come to handling uncertainty and adaptability to changes and new settings. The approaches need domain experts to generate knowledge and design rules that can break or bypass previous ones. These limitations are partially solved with the DDA approaches. Supervised learning requires labeled data on which an algorithm is trained. The algorithm uses the labeled data to learn the correlation between the label and the data input. It creates rules that infer the occurrence probability of the label knowing the input. In general, rules correspond to boundaries between data (Support Vector Machines [13]) and discrimination criteria (Decision Trees [23]). After training, the algorithm is then able to classify new unknown data, based on the learned rules. The more variety in examples available, the more the algorithm will be able to generalize and easily handle different inputs.

Data Driven Activity Recognition

Unsupervised learning methods extract the knowledge from the data itself without labels.

They do not use prior class knowledge and attempt to find and discover logic relations between the data. They need a large amount of data as the number of examples is important. Many unsupervised methods exist, some use distance rules to cluster data that are similar, others attempt to learn these rules by themselves. For strategies to train these types of algorithms the input representation is very important, because it can generate bias. The main advantage of unsupervised learning is that these methods can discover rules and correlations not imagined by an expert.

The DDA's strength is its probabilistic modelling capacity. These models are capable of handling noisy, uncertain, and incomplete sensor data. They can capture domain heuristics, e.g., some activities are more likely to occur than others. They don't require a predefined domain knowledge, are more adaptive to evolution and new situations. However, DDA requires a great deal of data and in the case of supervised learning, clean and correctly labelled data.

Introduction

Outline and Research Questions

The progress made in sensors, IoT, communication technologies, and their integration in our daily objects has allowed the development of so-called "intelligent" environments. These technologies open the door to home automation, allowing us to transform and improve our homes into smart homes. Smart homes capable of perceiving (sensors) and acting (actuators) to, for example, save energy and improve our comfort, health and safety. Smart home technology is now more accessible than ever and is becoming more and more popular. However, in order to advance and produce more enhanced services better adapted to each individual, we must overcome a major challenge in development, the ability to "understand" human life, especially ADLs. In this thesis, we are interested in the recognition of human activities in smart homes, and we address the following question:

How to recognize the activities performed by inhabitants in a house through home automation sensors?

Sensors used to equip smart homes must be as non-intrusive as possible to preserve privacy, which implies that they provide poor amounts of information. Each sensor has also been optimised to minimize their energy consumption. Therefore, they output information on their own timing, and are often event-triggered. The data streams of each sensor are unsynchronized between them. Moreover, sensors provide different output value types (binary, scalar, etc). This leads us to our first key research question:

(i) How to efficiently encode, represent, and extract information, features, and patterns from non-intrusive home automation sensors of different natures to perform recognition of the ADLs?

Furthermore, Humans do not always live alone. Different residents performing different activities in parallel mean that a sensor's activation trace that corresponds to an activity could be mixed with another one: this is called the problem of parallel activities. This leads us to ask the following question:

(ii) How can we help recognition algorithms to better understand the sequences of sensor activations and sensor relatedness to separate the sequences and better understand captured sensor traces?

Typologies of houses are different, resulting in many different sensor configurations. The DDA is based on data mining, machine learning, and deep learning techniques, which seems promising as this approach allows for better adaptability and generalization. The increase in the number of datasets and thus access to rich data with multiple activity labels, as well as an increase in computing power, allows designers today to generate more complex models. But, even if a deep learning model can optimize generalization on a specific dataset, its performance drops when tested on another dataset, which brings us to the next question:

(iii) Can we transfer the knowledge gained from one house to another house?

Finaly, from an industrial point of view, while models can be trained on any dataset, it is still impossible to obtain a large amount of labeled data from a client. It is not realistic to collect many months of data and ask a customer to label it. This leads to the last question:

(iv) How could we apply activity recognition methods to a actual real house?

Thesis Organisation

Chapter 2 provides an overview of activity recognition systems in the literature. We present how the sensor stream is segmented, represented, and encoded. We discuss pattern recognition and temporal data interpretation methods for recognizing activities from sensor data as well as datasets for activity recognition. We describe the state of the art developments in the field and discuss open problems and issues related to these topics.

Chapter 3 presents the proposed approach, background, and experimental methodology used in this thesis. We present the sensor stream encoding used in this work. We detail the basic models used for the comparison, the preparation of the datasets, the comparison metrics, and the evaluation method.

Chapter 4 addresses pattern recognition issues with respect to activity recognition. We evaluate the first approach to representation and feature extraction that uses methods from the fields of Natural Language Processing (NLP) and Time Series Classification (TSC). We explain the need to model activity recognition in a house as a more complex problem than just a pattern recognition problem.

Chapter 5 discusses methods for improving the representation of sensor data through advanced integration methods. The context in which a sensor is activated provides valuable information about the current activity. Just as a word can have a different meaning depending on the other words in a sentence, the activation of a sensor can have a different meaning depending on the other sensors in the same trace. We propose the use of advanced Natural Language Processing (NLP) methods to extract a contextualized representation of these sensors to help models understand in which activity context the activation of a sensor occurs. In particular, we present a method inspired by a language modeling technique capable of contextualizing words in a sentence.

Chapter 6 presents studies using previous temporal and activity context to improve activity recognition models. Human activities are often related to the time of day and to previous activities. We propose to add a representation of this knowledge to help models better discriminate between activities that are similar in terms of activation sensors or occur at particular times of day.

Chapter 7 presents a method for applying activity recognition models in real houses using synthetic data from a digital twin house. Models used in activity recognition require labeled Introduction data to learn the model parameters. A model trained for one house cannot automatically be used in another house due to differences in house topology and inhabitant behavior. We present a transfer learning method that allows us to use synthetically generated labeled data from the digital twin of a real house to learn the model parameters for the real house.

Chapter 8 summarizes the results of this thesis and discusses future directions for activity recognition research.

Chapter 2

DATA DRIVEN-BASED HUMAN ACTIVITY RECOGNITION IN SMART HOMES LITERATURE REVIEW

Introduction

Recognizing human activity in smart homes offers many challenges, as discussed above. Activity recognition systems based on DDA, in particular deep learning based models, are nowadays a mainly studied and promising approach to try to address these different challenges. DDA are indeed able to generalize, show adaptability and do not require a predefined knowledge of the domain.

This chapter provides an overview and discussion of the state of the art of activity recognition systems based on DDA. We will see through the research questions ,recall below, which methods and works exist and try to answer them. We will propose a discussion around previous work limitations and the potential solutions envisaged to remedy them.

• How to efficiently encode, represent, and extract information, features, and patterns from non-intrusive home automation sensors of different natures to perform recognition of the ADLs?

• How can we help recognition algorithms to better understand the sequences of sensor activations and sensor relatedness to separate the sequences and better understand captured sensor traces?

• Can we transfer the knowledge gained from one house to another house?

•

How could we apply activity recognition methods to an actual real house?

Based on these questions, we first present and discuss how the sensor stream is represented, segmented. Then we discuss pattern recognition and temporal data interpretation methods for recognizing activities from smart home sensor data. We discuss about complex human activity recognition and data variability. We present datasets for activity recognition and discuss their limitations. Finally, we discuss open problems and issues related to this topic

Representation of Sensor Data Flow

Smart home sensors record the actions and interactions of the residents in their environment over time. These recordings are event logs that capture the actions and activities of daily life in the home. Most sensors are binary or stateful, and only send their status when there is a change, to save battery power and to not overload wireless communications. Indeed, most of the time, to increase ease of installation or to accommodate installation needs in older houses, the sensors run on batteries and communicate over a wireless network. In addition, sensors may have different triggering times. This results in scattered time series samplings and irregular sampling.

In general, binary sensors are coded with values of zero and one, while non-binary sensors are normalized and coded with a numeric value between zero and one. the sensor events. In the first case, each sensor records its activity on an independent track over time, see Figure 2.1a. The set of these time series can then be treated as a multivariate time series. This is the most commonly used representation because each sensor is considered as a feature. However, as these time series are made of state changes, they are not continuous (event 20

Segmentation of Sensor Data Flow

according to a resident's action) nor synchronized between themselves (time lag due to wireless communications and sensor reactivity in case of simultaneous activation of sensors).

In general, these time series are resampled in order to generate regular and synchronized time series. Nevertheless, event if resampled time series can gives and encode some information, the sensor duration time activation for example, due to this practice it is possible to alter or modify the data stream. While it is easy to complete the time series by repeating the last state until a new change occurs (see Figure 2.2) the decision over the resampling step may be less obvious.

Indeed, the resampling can make certain changes of state disappear if the granularity becomes too large. In contrast, if the sampling is too small, the amount of data becomes very large and more difficult to segment and process. Moreover, this representation makes it difficult to interpret the order of activation relationships between the sensors (order of actions), but allows to easily highlight which sensors are activated at the same time in the house and activation duration time, during an activity. In the second representation, the set of sensors is perceived as a temporal series of events ordered in time, see Figure 2.1b. This representation has the advantage of not requiring resampling or filling, and thus limiting preprocessing and potential data degradation. It provides the system the ability to use the data in raw form and to exploit the relationships between the sensors and their activation, as well as the order in which they evolve. It also allows the system to compress the information and thus to potentially process longer information sequences. However, it does not allow systems to consider the evolution of time between state changes nor their durations.

Segmentation of Sensor Data Flow

Once the sensor log is transformed into a univariate or multivariate time series, and as in many areas of activity recognition, a common approach to handling the data is to first segment the data Part, Chapter 2 -Data Driven-Based Human Activity Recognition in Smart Homes Literature Review stream, and then generate features to identify the activity in each of these segments. Depending on the type of activity recognition system, "online" or "offline", some types of segmentation are more suitable than others. A online recognition is a necessity for reactive systems. In some situations, it is not suitable to recognise activities several minutes or hours after they occur, such as in case of an emergency or for danger detection. Online recognition typically uses a sliding window version of the various windowing methods described in the following subsections.

Quigley et al. [START_REF] Quigley | A comparative analysis of windowing approaches in dense sensing environments[END_REF] have studied and compared different segmentation strategies also known as windowing approaches. The Table 2.1 summarizes and categorizes the different segmentation techniques detailed below. An example of three of these segmentation methods applied to a univariate sensors' representation is provided in Figure 2.3.

Explicit Windowing (EW)

Explicit Windowing (EW) consists of parsing the data flow per activity [19,79]. Each of these segments corresponds to one window that contains a succession of sensor events belonging to the same activity. This window segmentation depends on the knowledge of the data labeling and the size of each window is directly related to the duration of the activity. In case of the absence of labels, it is necessary to find the points of activity change. The algorithms will then classify these windows by assigning the right activity label. This approach has some drawbacks.

First, it is necessary to find the segments corresponding to each activity in case of unlabelled data. Then, the algorithm must use the whole segment to predict the activity. It is therefore not possible to use this method for real-time ADLs recognition. Indeed, a real-time activity recognition system tries to determine the activity at the moment it is performed and therefore uses only a fragment of the sequence. However, the EWs contain the entire sensor sequence corresponding to the activity. The system must therefore wait until the entire sensor activation sequence corresponding to the activity has appeared before deciding.

Time Windows (TW)

The use of Time Windows (TW) consists of dividing the data stream into time segments with a regular time interval. This approach is intuitive, but usually only favorable for the time series of sensors with regular sampling over time. This is a common technique with wearable sensors such as accelerometers and gyroscopes. One of the problems with this approach is the selection of the optimal duration for the time interval. If the window is too small, it may not contain any relevant information. If it is too large, then the information may be related to several activities, and the dominant activity in the window will have a greater influence on the choice of the label.

van Kasteren et al. [START_REF] Leonhard | Activity recognition for health monitoring elderly using temporal probabilistic models[END_REF] determined that a window of 60s is a time step that often grants a good classification score. This value is used as a reference in many recent works [START_REF] Medina-Quero | Ensemble classifier of long short-term memory with fuzzy temporal windows on binary sensors for activity recognition[END_REF]33,34,[START_REF] Ali Hamad | Dilated causal convolution with multi-head self attention for sensor human activity recognition[END_REF].

Quigley et al. [START_REF] Quigley | A comparative analysis of windowing approaches in dense sensing environments[END_REF] shows that TW achieve a high level of accuracy but do not often find all possible classes. Indeed, a TW can contain different pieces of activity, especially if the time step is large. The most represented activity within the TW will be used by the system to determine the activity label of the TW. The elements of a very short activity can then be absorbed by the elements of the most present class in the TW. Moreover, in the case of a too small time step, the current TW can be very similar to others which allows the system to make more errors.

Sensor Event Windows (SEW)

Contrary to a TW which divides the sensors stream according to time, a Sensor Event Windows (SEW) divides the stream into segments containing an equal number of sensor events. This type of segmentation is used in the context of a univariate representation of the sensor stream. Thus, in this segmentation the sensor events are considered as a succession of events, in other words as a sequence of events. Each window is usually labeled with the label of the last event in the window. The sensor events that precede the last event in the window define the context of the last event. This type of window varies in terms of duration. Some Sensor Event Windows (SEW)s can be as short as 5 minutes or as long as 5 hours. The duration of a Sensor Event Windows (SEW) is determined by the delta of time between the first and last event.

This method is simple but has some drawbacks. The size of the window for a number of events, as with any type of window, is a difficult parameter to determine. This parameter defines the size of the context of the last event. If the context is too small, there will be a lack of information to characterize the last event. However, if it is too large, it will be difficult to interpret. A window of 20 to 30 events is usually selected in the literature [5].

Moreover, just like TW, a SEW can contain events that belong to different activities. It is quite possible that the last element of the SEW is an event that belongs to an activity "B" while all previous events belong to an activity "A". In this case, it is possible that the recognition system associates this SEW with activity "A" or instead associates it with activity "B".

Dynamic Windows (DW)

Dynamic Windows (DW) use a non-fixed window size, contrary to the previous methods. It is a two-stage approach that uses an offline phase and an online phase [1]. During the offline phase, the data stream is split into EW. From the EW, the "best-fit sensor group" is extracted based on rules and thresholds define by an expert, in other words, the characteristics, namely, number of activations of each sensor, sensor location, activation duration of each sensor and number of activated sensors. Then, during the online phase, the dataset is streamed to the classification algorithm. When it identifies the "best-fit sensor group" in the stream, the classifier associates the corresponding label with the given input segment.

The DW approach is more reactive in online activity recognition as it may have less instances of sensor events in order to find an activity. For example, the TW approach must run through 60s of sensor data before producing a window, the SEW approach must run through 20 sensor events before producing its first window, but, the DW approach may not always have to wait this long before producing its activity windows. Problems can arise if the source dataset is not properly annotated such that it uses a EW in the first step. Thus, DW inherits the problems of EW. Quigley et al. [START_REF] Quigley | A comparative analysis of windowing approaches in dense sensing environments[END_REF] have shown that this approach is inefficient for modeling similar activities as the "best-fit sensor group" could be also similar. Furthermore, rules and thresholds are manually designed by experts which is time-consuming.

Fuzzy Time Windows (FTW)

Fuzzy Time Windows (FTW) were introduced in the work of Medina-Quero et al. [START_REF] Medina-Quero | Ensemble classifier of long short-term memory with fuzzy temporal windows on binary sensors for activity recognition[END_REF]. Unlike previous segmentation methods, FTW not only segments but also extracts features. FTW can be considered as an extension of TW applied to multivariate binary sensor sequences. This segmentation approach uses a succession of FTW of different time sizes defined by the Fibonacci number sequence. For each sensor, the FTW sequences encode, as a numerical value, the activations, more or less long term according to the Fibonacci number sequence. Each FTW evaluates the activation of the sensor in question over a period of time defined by its size and produces a score. These scores are grouped into vectors by sensors, then assembled to form a feature matrix, where each row is a sensor and each column its activation score over time. The column number is related to the number of FTW. The strength of this representation is that it can take into account distant activations of the same sensor in a window.

However, this method has a main drawback, FTW are designed to be only applicable to binary sensors' data. Nevertheless in a smart home the sensors are not necessarily binary, e.g. humidity sensors.

Discussion

The segmentation of the sensor data stream, independently of its representation, is an important step in a human activity recognition system. It depends on the reactivity that one wishes to give to the system. Indeed, EW are more adapted to offline systems while TW, SEW, DW and FTW are more adapted to an online recognition system. We have seen that each segmentation has its advantages and disadvantages. But also that some segmentation methods can only be used with a certain type of representation. Moreover, some segmentation methods like DW or FTW integrate another important step of an activity recognition system, the definition of features. In the next section we will see in more detail this primordial step of a human activity recognition system.

Feature Extraction

Once the data stream has been segmented, it is necessary to extract features from these segments and give them to a classification algorithm to assign the correct activity label. As seen in the previous chapter, the topology of the houses, their equipment, and the way an inhabitant performs the activities all have a great amount of variability, that increases the feature extraction difficulty. Feature extraction can be done manually following established rules or automatically.

It is a decisive step, which must be sufficiently discriminating to help human activity recognition methods to classify the ADLs.

Handcrafted Features

The researchers first defined (see Table 2.2) and studied characteristics [13]. Initially Cook et al. [19] and Yala et al.

[79] proposed several feature vector extraction methods, described below.

Baseline Method

This consists of extracting a feature vector from each window that contains the time of the first and last sensor events in the window, the duration of the window and a simple count of the different sensor events within the window. The size of the feature vector depends on the number of sensors in the datasets. For instance, if the dataset contains 34 sensors, the vector size will be 34 + 3. From this baseline, researchers upgrade the method to overcome different problems or challenges.

Time Depend Method

This tries to overcome the problems associated with sensor event vanishing influence through time. In most datasets, sensor events are not sampled regularly and the temporal distance of an event from the last event in the segment has to be taken into account. To do this, the sensors are weighted according to their temporal distance. The more distant the sensor is in time, the less important it is.

Sensor Depend Method

This has been proposed to address problems affecting the relationships between the events in a segment. The idea is to weight the sensor events in relation to the last sensor event in the segment. The weights are based on a matrix of mutual information between sensors based on mutual sensor occurrence and calculated offline. If the sensor pairs with the last sensor of the segment in other parts of the sensor flow, then the weight is high and respectably low otherwise.

Sensor Depend Extension Method

This method proposes adding the frequency of the sensor pair to the mutual information matrix.

The more frequently a pair of sensors appears together in the dataset, the greater their weight.

Past Contextual Information Method

This is an extension of the previous approaches that takes into account information from past sessions. The classifier does not know the activity of the previous segment. For example, the activity "enter home" can only appear after the activity "leave home". Normally, the previous activity cannot be added to the feature vector because the algorithm might not be able to generalize enough and focus on that particular feature. Therefore, Cook et al. [19] propose a two-part learning process. First, the model is trained without knowing the previous activity.

Then, the activity prediction from the previous segment is added to the input feature vector of the classifier when classifying the current segment.

Temporal, Spatial and Frequency Features

Inspired by Cook et al. [19] and Yala et al.

[79]'s work, Aminikhanghahi and Cook [5] evaluated different types of sensors flow segmentation and also listed different handcrafted features.

Temporal features such as day of the week, time of day, number of seconds since midnight, or time between sensor transitions have been studied. Spatial features were also evaluated such as location and also metrics such as, the number of events in the window or the identifier of the sensor appearing most frequently in the previous segments.

Latent Knowledge Method

This method was proposed recently by Yan et al. [80] to improve the baseline method proposed by Cook et al. [19] and Yala et al. [79]. Their proposal is to learn probability features on sensor occurrence from EWs and combine these probability features with the basic feature vectors that are generated from SEW. To generate the probability features, an HMM was trained to learn the probabilities between sensors in each EW. The authors conclude that the unsupervised learning features significantly improves the performance of hand-created features.

Part, Chapter 2 -Data Driven-Based Human Activity Recognition in Smart Homes Literature Review

Automatic Features

In the aforementioned works, features are extracted manually. They are carefully designed and heuristic. However, creating handcrafted features is a tedious task that requires expert knowledge. Moreover, handcrafted features lack adaptability.

Unsupervised Machine Learning

A few years ago Cook et al. [19] introduced an unsupervised method for discovering activities from sensor data based on a machine learning algorithm. The algorithm searches for a sequence pattern that best compresses the input dataset. After many iterations, the algorithm reports the best patterns. These patterns are then clustered and given to a classifier to perform the final classification. This approach was a first step for unsupervised feature extraction, but did not produce high performance rates.

Autoencoder

Wang et al. [START_REF] Wang | Human activity recognition in a smart home environment with stacked denoising autoencoders[END_REF] shows that automatic features can outperform handcrafted features. However, in this approach there is still a first step of manual generation of characteristics.

Seq2Seq Model

Recently, Ghods and Cook [28] proposed a method also based on a deep learning technique, "Ac-tivity2Vec", to generate an activity latent projection from sensor data. They used a Sequenceto-Sequence (Seq2Seq) model [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF] to encode and extract automatic features from the sensor's stream. They used EW from an univariate time series representation where each EW is a sequence of sensor ID. The model is trained, to reconstruct each EW in input into output. Their

End-to-End Deep Models

experiment shows that the "Activity2Vec" method seems generates good automatic features.

However, they observed that the intra-class similarity scores were lowest with the handcrafted features, meaning that the handcrafted features appear to perform better than the "Activ-ity2Vec" features.

Discussion

We have just seen different methods and approaches to perform one of the most important steps of an activity recognition system, the feature extraction. Handcrafted feature extraction approaches have proven their worth, but they do not allow for sufficient generalization to address the adaptability issues of the activity recognition system, which is one of the major challenges of this domain. Indeed, once these features are generated by an expert, they are adapted to the environment for which they were made.

Approaches for automatic extraction of these features have also been considered, notably via deep learning methods. Indeed, the automatic extraction of features is one of the challenges taken up by deep learning approaches. However, the envisaged methods do not seem to be sufficiently efficient yet.

End-to-End Deep Models

In the previous section, we saw that feature extraction is an important step for a human activity recognition system. From these features, an algorithm attempts to identify patterns in order to associate the correct activity label. Therefore, human activity recognition can be reduced to a pattern recognition and classification problem. However, thanks to deep learning, in addition to patterns, these models are able to automatically extract features and perform the final classification task. Deep learning allows us to consider end-to-end models, i.e. models able to extract features from raw data and to perform the classification task at the same time.

CNN models have proven themselves in terms of automatic feature extraction, pattern recognition and classification, especially in the video, image and sound domains.

They have three advantages for human activity recognition: (1) they can capture local dependencies, i.e., the importance of neighboring observations correlated to the current event;

(2) they are scale-invariant in terms of step difference and event frequency; and finally (3) they can learn a hierarchical representation of the data.

Therefore, in order to extract features and patterns in activities, Gochoo et al. [30] and Mohmed et al. [START_REF] Mohmed | Employing a deep convolutional neural network for human activity recognition based on binary ambient sensor data[END_REF] and Brenon et al. [12] have transformed sensor activity sequences into images and then used 2D CNN.

Gochoo et al. [30] first turned activity sequences into binary images to perform this task. The y axis of the image represents the sensors, which are ordered and grouped by location, and the

x axis corresponds to time. The researchers then designed a Deep 2D CNN to extract features, patterns and finally make the classification. Their work showed that this type of structure could be applied to the ADLs recognition problem. However, this work uses only binary sensors. It could be useful to take into account sensors of other natures. Moreover, other information such as previous activities' labels could also be worthwhile.

In an extension, Tan et al. [START_REF] Tan | Multi-resident activity recognition in a smart home using rgb activity image and dcnn[END_REF] proposed the use of colored pixels to encode the previous activity and temperature sensors. This work obtained better results than the previous one but neglected other potentially useful information.

Mohmed et al. [START_REF] Mohmed | Employing a deep convolutional neural network for human activity recognition based on binary ambient sensor data[END_REF] adopted the same strategy but converted activities into grayscale images.

Of note is the fact that previous works did not take into account the activation duration of sensors' activations. They used a gray value to encode the duration of each sensor activation.

Then they applied the AlexNet structure [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] to extract patterns and features from images.

Finally, they used the features with classifiers to perform the final ADLs recognition.

The previous approach achieved good classification results on EW and a custom dataset.

However, the temporal order of sensor activations was not taken into account. Another work of Tan et al. [START_REF] Tan | Multi-resident activity recognition in a smart home using rgb activity image and dcnn[END_REF] used an image representation with a sliding window on EW to try to tackle this problem. This approach showed good results but appears to not be robust enough to deal with unbalanced datasets (i.e., when some classes have more occurrences than others), or with unlabeled sequences of sensors. Moreover, the temporal dependencies are still not token into account.

Discussion

All these works have shown that CNN are applicable to the ADLs recognition problem, thanks to their ability to extract relevant patterns and features. However, this representation in the form of an image makes it difficult to represent and encode non-binary sensors. Moreover, the temporal aspects and temporal dependencies are not interpretable by a 2D CNN and the order of appearance and evolution of the state changes over time is important information. This is why other methods capable of taking into account this temporal aspect have been and continue to be studied today.

Sequence Models

The human activity recognition in a smart home, more precisely ADLs recognition is a problem of classification and pattern recognition in time series with irregular sampling. However, more specific machine learning for sequential data analysis have proven efficient to tackle this problem.

The Table 2.3 shows a summary and comparison of recent deep learning works.

First, statistical Markov models such as HMMs [17,16] and their generalisation, probabilistic A RNN is designed to take a series of inputs with no predetermined limit on size. It remembers the past and its decisions are influenced by what it has learned from the past. A RNN can take one or more input vectors and produce one or more output vectors and the output(s) are influenced not just by weights applied on inputs like a regular neural network, but also by a hidden state vector representing the context based on prior input(s)/output(s). So, the same input could produce a different output depending on previous inputs in the series. But the use of a RNN suffers from the long-term dependency problem [9]. To avoid this problem, two RNN variations have been proposed: the Long Short Term Memory (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF] and Gated Recurrent Unit (GRU) [14] that is a simplification of the LSTM.

Liciotti et al. [START_REF] Liciotti | A sequential deep learning application for recognising human activities in smart homes[END_REF] studied different LSTM structures on activity recognition with EW (presegmented activity sequences). They showed that the LSTM approach outperforms "Shalow" machine learning-based human activity recognition approaches in terms of classification score without using handcrafted features, as LSTM can automatically generate features that encode the temporal pattern. LSTM approaches and, in particular, bidirectional LSTM, achieve the best results, due to their ability to exploit their internal memories to capture long-term dependencies Hamad et al. [33] have proposed an extension of the LSTM and FTW structure, by adding FTW using the future data of the sensor in addition to the past information. The purpose of this complement is to introduce a delay in the decision-making of the classifier. The implication is that relying only on the past is not enough to predict the right label for an activity, and that in some cases delaying the recognition time allows the system to make a better decision. To illustrate with an example, if a binary sensor deployed on the front door generates an opening activation, the chosen activity could be "the inhabitant has left the house". However, it may happen that the inhabitant opens the front door only to talk to another person at the entrance of the house and comes back home without leaving. Therefore, the accuracy could be improved by using the activation of the following sensors. It is therefore useful to introduce a time delay in decision-making. The longer the delay, the greater the accuracy, but if this delay is unbounded, the algorithm can no longer be considered real time. While a long delay may be acceptable for some types of activity, others require an extremely short decision time in case of an emergency, i.g the fall of a resident.

The FTW approach allowed systems to obtain excellent classification results. Nevertheless, FTW are designed to be only applicable to binary sensors' data and in a smart home the sensors are not necessarily binary, e.g. humidity sensors.

As the 2D CNN showed capabilities at improving the ADLs recognition problem, the researchers studied a form of CNN more adapted to time series: the 1D CNN. 1D CNN seem to be a competitive solution for time series classification problems as reported in [77,26]. They allow

Sequence Models

good pattern extraction and at the same time interpret the temporal aspect.

The work of [START_REF] Singh | Convolutional and recurrent neural networks for activity recognition in smart environment[END_REF] shows that 1D CNN can equally be well applied to the ADLs recognition problem and have some advantages against LSTM models. As for all CNN models, 1D CNN have the advantage of being fast to train. They achieve high accuracy, capture patterns, but suffer from the lack of interpretation of long term dependencies, and deal poorly with variable size sequences.

A comparison between a 1D CNN and LSTM approach was performed in the work of Singh et al. [START_REF] Singh | Human activity recognition using recurrent neural networks[END_REF], Liciotti et al. [START_REF] Liciotti | A sequential deep learning application for recognising human activities in smart homes[END_REF], and Wang et al. [75] . They showed that LSTM perform better in the classification task, as they allow learning long term dependencies from sensor data.

Recently Hamad et al. [34] have combined a LSTM and a 1D CNN with a FTW segmentation.

This work has demonstrated that the combination of the two models can help to increase the classification score in general. In addition, this proposed model is capable of achieving better performance for the recognition of minority classes (classes with the least number of occurrences).

Nevertheless, the score for the minority classes was still insufficient to handle imbalanced human activity problems.

LSTMs appear to be a viable solution to significantly improve the human activity recognition task in the smart home, despite a longer learning time than approaches based on CNNs. The interpretation of time dependency seems to be the key to human activity recognition systems in smart homes. The combination of an LSTMs based structure with a FTW shows very good performance especially through the idea of using information from the future by embedding a lag in the decision making. However, this approach remains limited to the use of binary sensors It seems that taking into account the order of occurrence of sensor events (LSTMs), the more or less long term dependencies (LSTMs and FTWs), and the use of more contextual information, especially coming from the future (FTW + lag), increases the performance. As well as the dependencies and relations between sensor activation, the past and future context allow a better interpretation of the current sensor activation.

Like human activity recognition, the field of Natural Language Processing (NLP) deals with sequence analysis problems. Similarly, it seems that word sequence analysis can be related to sensor activation sequence analysis. Sensor activations as well as words have more or less long term dependencies, require past and future contextual information and must respect a certain order.

Syntactic analysis and context modeling have been, very early and naturally, at the center of research in NLP. The latest advances in this field have proposed various unsupervised pre-training methods of projecting words into a vector space, called embedding, to create word representations and language models. These embeddings capture information about the construction of words, sentences and texts. They are also able to capture the context of a word in a document, semantic or syntactic similarities, relations with other words, etc.

Part, Chapter 2 -Data Driven-Based Human Activity Recognition in Smart Homes Literature Review

Several training structures and methods have allowed advances in the field of NLP. They can be classified into two categories. Static embedding approaches, such as Word2ve [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF], GloVe [START_REF] Pennington | Glove: Global vectors for word representation[END_REF], FastText [10], etc. And contextualized approaches, such as ELMo [START_REF] Matthew E Peters | Deep contextualized word representations[END_REF], BERT [23], GPT [START_REF] Radford | Improving language understanding by generative pre-training[END_REF], etc. These models have proven to be effective in the context of text classification, and could be considered for the field of human activity recognition in smart homes.

Complex Human Activity Recognition

One key issue for acticvity recognition is that most sequential data analysis algorithms can only process simple, primitive activities, and can not as of yet deal with complex activities. A simple activity is an activity that consists of a single or few actions caried out by a single user, such as turning on the light, opening a drawer. A complex activity is an activity that involves a sequence of actions potentially involving different interactions with objects, equipment, or other people, for example, cooking. Furthermore, current deep learning models for recognizing human activity in smart homes do not define relations between different activities.

Sequences of Sub-activities

Indeed, ADLs are not micro actions such as gestures that are carried out the same way by all individuals. ADLs that our smart home want to recognize are sequences of micro actions, which we can call compound actions. These sequences of micro actions generally follow a certain pattern, but there are no strict constraints on their compositions or the order of micro actions.

This idea of compositionality was implemented by an ontology hierarchy of context-aware activities: a tree hierarchy of activities links each activity to its sub-activities [START_REF] Hong | Evidential fusion of sensor data for activity recognition in smart homes[END_REF]. Another work proposed a method for teaching this hierarchy: as the HMM approach is not well suited to process long sequences, an extension of HMM called Hierarchical Hidden Markov Model (HHMM) was proposed in [8] to encode multilevel dependencies in terms of time and follow a hierarchical structure in their context. human activity recognition in smart home(there have not been extensions of such hierarchical systems using deep learning) uses hierarchical LSTM using two-layers of LSTM to tackle the varying composition of actions. Covering everything from human activity recognition-based on videos proposing [22] to using tow hidden layers in the LSTM for human activity recognition using wearables [76]. This approach can constitute multiple inspirations for human activity recognition in smart home applications. Other works in video-based human activity recognition proposed automatically teaching systems stochastic grammar describing the hierarchical structure of complex activities from annotations acquired from multiple annotators [START_REF] Tayyub | Learning hierarchical models of complex daily activities from annotated videos[END_REF].

The idea of these human activity recognition algorithms is to use the context of a sensor activation, either by introducing multi-timescale representation to take into account longer term

Complex Human Activity Recognition

dependencies or by introducing context-sensitive information to channel the attention in the stream of sensor activations.

Interleave and Concurrent Activities

Human activities are often carried out in a complex manner. Activities can be carried out in an interleaved or concurrent manner. An individual may alternately cook and wash dishes, or cook and listen to music simultaneously, but could just as easily cook and wash dishes alternately while listening to music. The possibilities are infinite in terms of activity sequence. However, there are some activities whose appearances would be impossible to witness in the dataset as they would be anomalous, such as and individual cooking while the same individual sleeps in their room.

Researchers are working on this issue. Modeling these types of activities is complex, but it could be modeled as a multi label classification problem. Safyan et al. [START_REF] Safyan | Ontology-driven semantic unified modelling for concurrent activity recognition (oscar)[END_REF] have explored this problem using ontology. Their approach uses a semantic segmentation of sensors and activities.

This allows the model to relate the possibility that certain activities may or may not occur at the same time for the same resident. Li et al. [START_REF] Li | Concurrent activity recognition with multimodal cnn-lstm structure[END_REF] exploit a CNN-LSTM structure to recognize concurrent activity with multimodal sensors.

Multi-user Activities

Monitoring the ADLs performed by a single resident is already a complex task. The complexity increases with several residents. The same activities become more difficult to recognize. On the one hand, a group and a resident may interact to perform common activities. In this case, the activation of the sensors reflects the same activity for each resident in the group. On the other hand, everyone could perform different activities simultaneously. This produces a simultaneous activation of the sensors for different activities. These activations are then merged and mixed in the activity sequences. An activity performed by one resident in this case is noise for the activities of another resident.

Some researchers are interested in this problem. As with the problem of recognizing competing activities. The multi-resident activity recognition problem is a multi-label classification problem [3]. Tran et al. [START_REF] Son N Tran | Multi-resident activity monitoring in smart homes: A case study[END_REF] tackled the problem using a multi-label RNN. Natani et al. [START_REF] Natani | Sequential neural networks for multi-resident activity recognition in ambient sensing smart homes[END_REF] studied different neural network architectures such as MLP, CNN, LSTM, GRU, or hybrid structures to evaluate which structure is the most efficient. The hybrid structure that combines a CNN 1D and a LSTM is the best performing one.

Part, Chapter 2 -Data Driven-Based Human Activity Recognition in Smart Homes Literature Review

Datasets and Real-World Applications

Datasets are necessary to train, test, and validate activity recognition systems. Datasets were first generated in laboratories, but these records didn't allow enough variety or complexity of activities and were not real enough. To overcome these issues, public datasets were created from recordings in real homes with volunteer residents. In parallel to the development of practices for being able to allow systems to compare the same condition or train on the same data, competitions were created, such as Evaluating AAL Systems Through Competitive Benchmarking -AR (EvAAL-AR) [29] or UCAmI Cup [24] to improve dataset training.

Real Smart Home Datasets

A variety of public real home datasets exist [START_REF] Tapia | Activity recognition in the home using simple and ubiquitous sensors[END_REF][START_REF] Van Kasteren | Human activity recognition from wireless sensor network data: Benchmark and software[END_REF]18,2,20]. De-La-Hoz-Franco et al. [21] provide an overview of sensor-based datasets used in human activity recognition for smart home.

They compiled documentation and analysis of a wide range of datasets, with a list of results and applied algorithms. The Table 2.4 details some datasets from the literature, they are the results generated from the hard work of the smart home community. Nevertheless, datasets are not perfect, and dataset production includes several problems such as: sensors type and placement, variability in terms of user profile or typology of dwelling, and the annotation strategy.

Sensor Type and Positioning Problem

When acquiring data in a house, it is difficult to choose the sensors and their numbers and locations. It is important to select sensors that are as minimally invasive as possible in order to respect the privacy of the volunteers [2]. Because of this no cameras or video recordings are used. The majority of sensor-oriented smart home datasets use so-called low-level sensors.

These include infrared motion sensors (PIR), magnetic sensors for items that open and close, pressure sensors placed on sofas or beds, and sensors for temperature, brightness, monitoring of electricity, water consumption, etc.

The location of these sensors is critical for properly capturing activity. Strategic positioning allows a system to accurately capture certain activities, e.g. a water level sensor in the toilet to capture toilet usage or a pressure sensor under a mattress to know if a person is in bed. There is no precise method or strategy for positioning and installing sensors in homes. CASAS [18] researchers have proposed and recommended a number of strategic positions. However, some of these strategic placements can be problematic in terms of long term use. It is possible to imagine that during the life of a house the organization of or use of its rooms may change. For example, if a motion sensor is placed above a bed to capture its use but the bed is moved to a different place in the room, then the sensor will no longer be able to capture this information. In the context of a dataset and its use to validate algorithms, this constraint is not important. But it becomes important in the context of real applications to evaluate the resilience of algorithms, which must continue to function in case of loss of information.

In addition to sensors positioning, it is important to choose enough sensors to cover maximum capture of all possible activities. The number of sensors can be very different from one dataset to another. For example, the MIT dataset [START_REF] Tapia | Activity recognition in the home using simple and ubiquitous sensors[END_REF] uses 77 and 84 sensors for each of its apartments.

The Kasteren dataset [START_REF] Van Kasteren | Human activity recognition from wireless sensor network data: Benchmark and software[END_REF] uses between 14 and 21 sensors. ARAS [2] has apartments with 20 sensors. Orange4Home [20] is based on an apartment equipped with 236 sensors. This difference can be explained by the different types of dwellings, but also by the number and granularity of the activities that we want to recognize. Moreover, some datasets are voluntarily over-equipped.

There is still no method or strategy to define the number of sensors installed according to an activity list.

Profile and Typology Problem

It is important to take into account that there are different typologies of houses: apartment vs house, dwellings with gardens, multiple floors vs single floor, dwellings with one or more bathrooms, dwellings with one or more bedrooms, etc. These different types and variability of houses create multiple difficulties; for example, the investment in terms of number of sensors can be more or less important for activity recognition. In addition, the network coverage of the sensors can be problematic. For example, Alemdar et al. [2] faced a problem of data synchronization. One of their houses required two sensor networks to cover the whole house. They had to synchronize the data for their dataset needs.

Also, because each house is different, the data from one house to another is very different.

Thus a model learned in one house is hardly applicable in another because of differences in house configuration; sensor equipment; and family compositions and habits. Indeed, the location, the number and the sensors type of smart homes can influence activity recognition systems performances. This issue could be solved by collecting a new dataset for each new household to train the models anew, however this is costly.

Part, Chapter 2 -Data Driven-Based Human Activity Recognition in Smart Homes Literature Review

Another solution is to adapt the models learned in a household to another. Transfer learning methods have recently been developed to allow pre-trained deep learning models to be used with different data distributions, as reviewed in [78]. Transfer learning using deep learning has been successfully applied to time series classification, as reviewed in [25]. For activity recognition, Cook et al. [15] reviewed the different types of knowledge that could be transferred into traditional machine learning. These methods can be updated with deep learning algorithms and by benefiting from recent advances in transfer learning for deep learning. Furthermore, adaptation to new settings have recently been improved by the development of meta-learning algorithms.

Their goal is to train a model on a variety of learning tasks, so it can solve new learning tasks using only a small number of training samples. This field has seen recent breakthroughs, as reviewed in [START_REF] Hospedales | Meta-learning in neural networks: A survey[END_REF], which as of yet have never been applied to human activity recognition. Yet, the peculiar variability of human activity recognition data in smart home can only benefit from such algorithms.

It is therefore necessary that the datasets be able to handle different house configurations in order to train and evaluate the algorithms with a high generality of configurations. Fortunately, there are several datasets with several houses. [START_REF] Tapia | Activity recognition in the home using simple and ubiquitous sensors[END_REF][START_REF] Van Kasteren | Human activity recognition from wireless sensor network data: Benchmark and software[END_REF][START_REF] Van Kasteren | Human activity recognition from wireless sensor network data: Benchmark and software[END_REF]2]. CASAS [18] is one such dataset, with about 30 different house configurations. These datasets are often used in the literature [21]. However, the volunteers are mainly elderly people, and coverage of several age groups is vital for developers and researchers. After all, a young resident likely does not display the same behavior as an older one. The Orange4Home dataset [20] covers the activities of a young resident.

However, the number of residents is also important. Activity recognition is more complex in the case of multiple residents. This is why several datasets also cover this field of research [18,2,[START_REF] Van Kasteren | Human activity recognition from wireless sensor network data: Benchmark and software[END_REF].

Annotation Problem

Dataset annotation is something essential for supervised algorithm training. When creating these datasets, it is necessary to deploy strategies to enable annotation. Such as journals [START_REF] Van Kasteren | Human activity recognition from wireless sensor network data: Benchmark and software[END_REF],

smartphone applications [20], Personal Digital Assistant (PDA) [START_REF] Tapia | Activity recognition in the home using simple and ubiquitous sensors[END_REF], Graphical User Interface (GUI) [2] or voice records to annotate datasets [START_REF] Van Kasteren | Human activity recognition from wireless sensor network data: Benchmark and software[END_REF].

As these recordings are made directly by volunteers, they were asked to annotate their own activities. For the MIT dataset [START_REF] Tapia | Activity recognition in the home using simple and ubiquitous sensors[END_REF], residents used a PDA to annotate their activities.

Every 15 minutes, the PDA beeped to prompt residents to answer a series of questions to annotate their activities, however, several problems were encountered with this method of user self-annotation. Such as some short activities not being entered, errors in label selection, or omissions. A post-annotation based on the study of a posteriori activations was necessary to overcome these problems, thus potentially introducing new errors. In addition, this annotation strategy is cumbersome and stressful because of the frequency of inquiries. It requires great rigor from the volunteer and at the same time interrupts activity execution by pausing it when the information is being given. These interruptions reduce the fluidity and natural flow of activities.

van Kasteren et al. [START_REF] Van Kasteren | Human activity recognition from wireless sensor network data: Benchmark and software[END_REF] proposed another way of annotating their data. The annotation was also done by the volunteers themselves, but using voice through a Bluetooth headset and a journal. This strategy allowed the volunteers to be free to move around and not need to create breaks in the activities. This allowed for more fluid and natural sequences of activities. The Diary allowed the volunteers to complete some additional information when wearing a helmet was not possible. However, wearing a helmet all day long was a constraint.

The volunteers of the ARAS dataset [2] used a simple GUI on tablets to annotate their activities. Several devices were placed in homes to minimize interruptions in activities and avoid wearing an object such as a helmet all day long. Volunteers were asked to indicate only the beginning of each activity. It was assumed that residents will perform the same activity until the start of the next activity. This assumption reflects a bias that sees human activity as a continuous stream of known activities.

Synthetic Smart Home Datasets

As seen above, the creation of real datasets is a tedious task and recording campaigns are difficult to manage. It requires hardware, volunteers, and post-processing. The cost to build real smart home and the collection of datasets for such scenarios is expensive and sometimes infeasible for many projects. Measurement campaigns must include a wide variety of activities and actors. They should be done with substantial rigor to obtain plenty of qualitative data.

Moreover, difficulties with finding the optimal placement for sensors [START_REF] Helal | Scalable approaches to activity recognition research[END_REF], finding appropriate participants [START_REF] Sumi Helal | Persim-simulator for human activities in pervasive spaces[END_REF][START_REF] Mendez-Vazquez | Simulating events to generate synthetic data for pervasive spaces[END_REF] and the lack of flexibility [7,27] makes dataset collection difficult. Even if fully equipped and ready-to-use environments, such as Amiqual4Home [START_REF] Lago | The contextact@ a4h real-life dataset of daily-living activities[END_REF], can be used on an ad hoc basis, recruiting volunteers and collecting data is still very time-consuming. For these reasons, researchers created smart home simulation tools. The Table 2.5 present and compare some simulation tools from the literature. These simulation tools can be categorized into two main approaches, model-based [START_REF] Lee | Persim 3d: Context-driven simulation and modeling of human activities in smart spaces[END_REF] and interactive [START_REF] Synnott | The creation of simulated activity datasets using a graphical intelligent environment simulation tool[END_REF], according to Synnott et al. [START_REF] Synnott | Simulation of smart home activity datasets[END_REF]. The model-based approach uses predefined Part, Chapter 2 -Data Driven-Based Human Activity Recognition in Smart Homes Literature Review models of activities to generate synthetic data. [11], Lee et al. [START_REF] Lee | Persim 3d: Context-driven simulation and modeling of human activities in smart spaces[END_REF] and Ho et al. [START_REF] Ho | A smart home simulation tool to support the recognition of activities of daily living[END_REF] use predefined models of the agent behavior, describing different possible events and activities with their probability of occurrence, their order and their duration. With this first approach the quality of the data relies heavily on the quality of the modeling. These simulators usually consider big granularities of events and activities, as it is complicated to model fine-grained events and activities. Finally, these approaches are usually entirely scripted. If a user wishes to simulate several days (or more) of data, he needs to script each day independently.

In contrast, the interactive approach relies on having an avatar that can be controlled by a researcher, human participant, or simulated participant. [START_REF] Park | The user activity reasoning model in a virtual living space simulator[END_REF] and Ariani et al. [6] assume a human controlling an avatar to perform activities in the simulated environment. These approaches allow more coherent and realistic behavior as there is a real human involved. However, they require near-real time interaction between the controlling human and the simulated environment. Thus, producing large amounts of data is time consuming. The usage of these kinds of simulation tools are more restricted to the generation of data during a short period, focusing on specific activities or situations.

Some recent work focusing on hybrid simulators such as OpenSH [4], can combine advantages from both interactive and model-based approaches. The smart home simulation tools can focus on dataset generation or data visualization. Some simulation tools provide multi-resident options or fast forwarding to accelerate the time during execution.

These tools allow researchers and developers to quickly generate data and visualize it. But capturing activities can be unnatural and far too clean, as in they do not contain the messiness of real human actions which are far from perfect in their execution. Systems need the noise generated from human behavior; without it some of the uncertainty found in real human actions may be missing.

Discussion

Real datasets such as Orange4Home [20] provide a large sensor set. That can help to determine which sensors can be useful for which activities. CASAS [18] include many house and apartment configurations, plus multiple topologies with elderly residents. Which allows systems to evaluate their adaptability to different house topologies. ARAS [2] has younger aged residents and multiresidents' living in a single dwelling. These are all useful for validating the noise resilience and segmentation ability of an activity recognition system.

The strength of datasets from real houses is their realism. They are real playgrounds that allow to evaluate algorithms in situations close to reality. But sensors in these datasets may ironically be placed too strategically, as they are focused on covering certain specific kinds of activities. In some datasets, PIR sensors are placed in a grid or installed at checkpoints to track a resident's trajectory. Strategic placement, a large number of sensors, and/or the choice to use a particular sensor is great for helping algorithms to infer knowledge, but are not indicative of how real house sensor set-up would happen.

Moreover, the current publicly available datasets from these real-world environments are not sufficient to cover the probability distribution of any possible activities, lifestyle or configuration variability. These datasets are indeed limited in the number of activities and occupants they take into account, as well as the types of sensors and the type of resident. Therefore, algorithms trained on data from one particular environment will be difficult or impossible to reuse in others as they are adapted to a particular configuration of house, inhabitants and life habits.

Collecting new datasets is a very expensive and difficult task. The cost of sensors, the difficulty of accessing multiple types of homes, recruitment, collecting data from actual residents also takes time, and cannot be expedited. Moreover, the data requires labeling, i.e., segmentation of the data stream (start and end time of activities) as well as their class labels, which implies a significant investment on the part of the user and presents a high risk of error, as these annotations are performed manually and usually by the user himself.

Synthetic datasets generation tools allow systems to quickly evaluate different configuration sensors and topologies. In addition, they can produce large amounts of data without the setting up of real sensors or need for volunteer subjects. Also, their annotations are more precise compared to real dataset methods (diary, smartphone apps, voice records).

Nevertheless, there are no synthetic datasets which are publicly available. Only some dataset generation tools such as OpenSH [4] are available to the public, but they require the user of the tool to maintain the simulation themselves. Moreover, these simulation environments often lack realism, due to a lack of variability, which does not allow the algorithms generalize well.

In addition, virtual smart homes are not related to real smart homes. In general, these virtual smart homes are simplified versions of typical homes (small square homes, no floors), which also simplifies the selection, number and placement of sensors and the recording of sensor data.

In accordance with the dataset limitations described above, a human activity algorithm trained on a dataset corresponding to a particular configuration of house, inhabitants and life habits, will not be adapted to another house. From an industrial/commercial point of view, it is not feasible to ask the inhabitants to self-tag their activities for several months in order to collect enough data to train or adapt the final system. On the contrary, the system should be able to apply previously learned models and adapt them immediately to the new situation.

However, generating data from simulation environments close to the final environment to adapt human activity recognition models, bridging the gap between training and test data, could be a solution.

Part, Chapter 2 -Data Driven-Based Human Activity Recognition in Smart Homes Literature Review

Contribution Proposal

In this chapter, we have seen state of the art research on human activity recognition in smart homes based on home automation sensors through DDA. A significant amount of work has already been done. However, there are still many challenges to be met in order to allow these techniques to fully integrate into our daily lives and our homes.

In this thesis we focus on offline pre-segmented ADLs recognition and tackle the following:

• encoding, feature extraction, pattern recognition and relevant knowledge to better interpret sensor data;

• encode relevant time and previous context information;

• knowledge transfer from one home to another;

• data generation for application into the real world;

Encoding, Feature Extraction, Pattern Recognition and Relevant Knowledge

Just like words, sensor activations carry meaning thanks to their context and nature. They follow a certain syntax directly related to the syntax induced by the residents' actions. In other words, the syntax represents the logical sequence of the residents' actions during their activities.

Moreover, similar to human activity recognition in smart homes, the NLP domain works on sequence analysis and classification problems. This is why, in this work, in order to answer the first problem of feature extraction and context understanding, we propose to transfer methods used for coding and representation of words and language coming from the world of NLP to the domain of human activity recognition in smart homes. Indeed, in the NLP, many deep learning-based methods have been developed

to encode, interpret, and understand words, texts, language, and sequences. These methods are able to understand semantics and syntax by making logical links and by understanding the various levels of dependency between words.

Time and Previous Context Information

Previous works consider human activities as sequences of sensor activations which reflect the actions of residents. They do not take into account temporal information. Time precision and understanding offers the possibility to create specific services or scenarios, as for instance, reminders for morning and evening medication intake. In addition, taking into account time information might improve the ADLs recognition.

The consideration of relations between activities is another interesting task. Throughout the day, we perform activities that are often linked. Being able to interpret these relations could help systems recognize the current activity, especially when similarities between sensor activation sequences lead to confusions.

Transfer Learning

Second, despite the ability of current human activity recognition models to automatically extract features and patterns to fit any dataset thanks to deep learning, these models remain fixed to the dataset on which they are trained. Once trained on the data of one house, it is not possible for them to work on another house. Moreover, it is not currently possible for an algorithm to take advantage of knowledge acquired on a larger dataset to generalize and increase its performance on a more precise dataset. The knowledge thus acquired to recognize ADLs on a dataset is not transferable. However, there would be an advantage in taking advantage of more general knowledge to improve performance in another context where the data are in smaller quantities.

We therefore propose in this work to draw inspiration once again from the methods of the NLP world, where models are able to represent and store general knowledge in a latent space through embeddings. Thanks to these embeddings, the knowledge becomes transferable to other tasks, allowing the algorithms to train faster but also to reach higher performances on the final task they are intended for.

Data Generation

As we have seen over the past decades, more and more smart homes and real-world test beds are emerging. The current publicly available datasets from these real-world environments are not sufficient to cover the probability distribution of any possible activities, lifestyle or configuration variability and thus allow algorithms to generalize. These datasets are indeed limited in the number of activities and occupants they take into account, as well as the types of sensors and the type of resident.

Collecting new datasets is a very expensive and difficult task. The cost of sensors, the difficulty of accessing multiple types of homes, recruitment, collecting data from actual residents also takes time, and cannot be expedited. Moreover, the data requires labeling, i.e., segmentation of the data stream (start and end time of activities) as well as their class labels, which implies a significant investment on the part of the user and presents a high risk of error, as these annotations are performed manually and usually by the user himself.

In accordance with the dataset limitations described above, a human activity recognition algorithm trained on a dataset corresponding to a particular configuration of house, inhabitants and life habits, will not be adapted to another house. From an industrial/commercial point of view, it is not feasible to ask the inhabitants to self-tag their activities for several months in order to collect enough data to train or adapt the final system. On the contrary, the system should be able to apply previously learned models and adapt them immediately to the new situation, based on only a few data from the test case: this is the principle of learning by transfer. It is therefore not necessary to have a large amount of data similar to the test case.

To address this challenge of the gap between the training data and the use case data, data generation could be a solution, thanks to the Digital Twin concept [31,32]

BACKGROUND, APPROACH AND EXPERIMENTAL METHODOLOGY

Introduction

A significant amount of work has already been done for HAR in smart homes. Deep Learning techniques have shown exemplary improvement in this area, especially through the application of recurrent networks such as LSTM. However, there are still many improvements to envisage until these techniques become fully integrated into our daily lives and homes.

Despite the ability of LSTM to interpret temporal links and patterns within sequences of activity, classification scores are not always successful, especially when datasets become more complex. In the case of similar activities, in terms of sensor activations, when we deal with large numbers of activities or datasets containing the activities of several residents, LSTM-based approaches demonstrate great difficulties. Indeed, it seems that complementary knowledge and information must be extracted from the data to improve these models.

This model enrichment can be achieved by using new approaches to represent sensor events.

The combination of FTW, to encode and extract features, and some LSTM showed good performances. The ability of FTW to encode more or less temporally distant sensor activations seems to be the key in such forms of segmentation and encoding methods. Nevertheless, the FTW can only be used with binary sensors although activations of smart homes sensors are not always binary, as temperature, humidity, CO2, etc. Techniques able to use multi-modal sensors are therefore needed. Moreover, FTW are built following predefined manual rules. It should be interesting to find an automatic way of avoiding their loss of generality.

Building models with a part for feature extraction (FTW) and a part for classification and pattern recognition (LSTM), seems to be a promising alternative. Mixing models adapted to certain sub-tasks, such as feature extraction or pattern extraction, could in fact be an opportunity for improvements. Classically, some parts of the model could be trained for a specific task, for example a first part for input coding and feature extraction, and a second part for the pattern recognition task from the previously generated features.

This chapter describes the ideas and the proposed approach likely to improve the recognition Part, Chapter 3 -Background, Approach and Experimental Methodology algorithms of human activities in a smart home. Firstly, we detail the idea and the proposed approach through a step by step workflow for a better understanding. Secondly, We will describe the reference (i.e. baseline) models used for the experiments of this work. We will then present the datasets and the evaluation method used. Finally, we will try to explain the metrics that allow us to compare the proposed approach to the reference models.

Proposed Approach

Key Idea

This thesis proposes to consider the traces of different sensors as a single time series, in which the sensor activations follow each other, building, therefore, a sequence of sensor activations. This long sequence can then be divided into sub-sequences, each one corresponding to an activity.

Within these sequences, the sensor activations define a pattern that allows to recognize the corresponding activity. This pattern is built up taking into account which sensor activations are present, their order of appearance and their frequency. The NLP domain, especially as far as the text or sentence classification tasks are concerned, also deals with sequences of words that define patterns. The motivations are for similar reasons with our concerns. Therefore, we can get fruitful inspiration from the language paradigm and draw technical advantages from the NLP domain methods to try to solve the HAR problem in smart homes. The goal by using this paradigm is to find means to represent and extract some information concerning part of the rationality of sensor activation relations. By analogy, just as language can describe activities with words, this work assumes that "the house has its own language to describe what is happening in it".

One could say that the activation of a sensor is similar to a word "spoken by the house" and a sequence of activations is similar to a sentence describing one or more activities. Sensor activations, like words in a sentence, are organized into a "syntax". Indeed, each sensor activation translates an operation or an action of a resident. Therefore, the order of the activation of the sensors (order of operations, order of actions), as well as the links and the relations between them, give information about the way the activity is performed. As some words can only appear after a particular one, sensor activations follow a similar logic. For example, the detection of movement in a room usually could occur after someone has opened or passed through a door. The opening of the faucet in the bathroom can only appear if a resident has entered the bathroom, and cannot appear following an entry into the living room. This syntax can then be used as a pattern to recognize.

Moreover, in the case where several residents live under the same roof, interpreting that certain orders of actions are possible or not, makes it easier to distinguish what each resident does. For example, let us consider two residents, each one performing a different activity. The first one cooks while the second takes a shower. In this case, the sensors of the house could

Proposed Approach

potentially capture the actions of each resident in the following order: entering the kitchen, entering the bathroom, opening the fridge door, turn on the bathroom light, turn on the kitchen faucet, turn on the hotplate, turn on the shower faucet, turn off the kitchen faucet, etc. It is therefore useful to be able to interpret the logic of the possible actions sequences.

In this work, the problem of recognizing human activities in a smart home is envisaged as a language modeling task and a text classification problem. Our approach consists of four parts.

First, we segment the dataset into sub-sequences of activities using an EW. This step, turns the dataset into "sentences". Second, the sensor activations and, at the same time, the sequences of activations are encoded. Third, sensors sequences are turned into index in order to be usable by models. Fourth, a classifier, based on a neural network structure captures the patterns, as well as the best features, to classify the activation sequence into ADLs. In order to facilitate the understanding of our general approach and some of its details, we will follow the step by step workflow, described in Figure 3.1.

Bed To Toilet (t)

M004ON M005OFF M007OFF M004OFF M004ON M004OFF M007ON M004ON M007OFF M007ON M005ON M004OFF M003ON M003OFF M002ON M003ON M002OFF M003OFF M003ON M003OFF
... ...

...

Bed to Toilet

Sleeping

Recordings of sensors' activations Encoding of sensors' activations ("words") and segmentation into activities ("sentences") Classification of the sequence of sensors' activations

Step 1 and Step 2: Sensors' Stream Segmentation and Encoding

From the stream of activations recorded by the sensors, Step 1 performs a EW to segment the dataset into event sequences. Each sequence is an activity and the event timestamp order inside the sequences is kept. These event sequences are then encoded in Step 2.

The goal of the encoding is to obtain "words" that can be used as inputs of the model. To be able to represent all possible activations of the different sensors, we create a corpus of sensors' activations and consider these activations as categorical values, i.e. each sensor activation is represented by a unique value. By transforming the sensors' activations into categorical values, we allow the model to learn the notion of frequency of occurrence of an activation in a sequence.

Moreover, the model, via such a representation, can capture the relation of an activation to another. All these categorical variables ("words") define the smart home vocabulary that describes activities.

More concretely, a set of sensors S = {s 1 , ..., s |S| } produces events e i ∈ E, E being the set of events. An event e i is composed of the sensor ID s i , the value v i and the timestamp t i . We concatenate the sensor ID s i and its value v i and ignore the timestamp t i to create the "word" associated to this event, e.g., s i = M 001 and the binary value v i = ON becomes M001ON. For instance, the sequence of events for the activity bed_to_toilet in Figure 3.

1 at Step 1, is turned into the sequence [M004ON M005OFF M007OFF M004OFF M004ON M004OFF M007ON M004ON M007OFF M007ON M005ON M004OFF].
Sensors with numerical values are represented in the same way, as a concatenation of the sensor label and the measurement. For instance, if the activation of the temperature sensor named T004 gives a value of 24.5, the input to the system is T00424.5.

Step 3: Words to Indexes

Each sensor activation in sequences is transformed into an index to be used as the input to a neural network. The index starts at 1 while the 0 value is reserved for the sequence padding.

As in NLP domain, indexes are assigned following the "word" frequency (categorical value of the sensor activation frequency), e.g., if the "word" M004ON has the highest occurrence in the dataset, the assigned index is the lowest one, i.e. 1. This ordinal encoding encodes the frequency of a sensor activation. Thus, a sequence of "words" such as

[M005OFF M007OFF M004OFF M004ON M004OFF M007ON M004ON M007OFF M007ON M005ON M004OFF] becomes the sequence of indexes [1 4 8 2 1 2 7 1 8 7 3 2],
according to the sensor activation frequency.

Step 4: Classification

Once the dataset is segmented into encoded sequences, these sequences are used by a neural network to perform classification. The model is composed of an embedding layer and a neural Moreover, this work also studies advanced particular embedding methods inspired by the NLP domain to attempt to improve the ADL's recognition.

The Baseline Models

Thus, two of the currently best models in the field, defined by Liciotti et al. [6]'s work, are taken as baseline models in order to position our work. Liciotti et al. [6] study uses a similar sensors' sequence representation, as the one we advocate (sequential representation), and compare pitted several LSTM-based models against state-of-the-art machine learning methods (HMM, CRF, NB, etc) and CNN-based approaches. More in detail, their approach consists in encoding, under the form of categorical variables, the sequences of events of sensors before they pass them through the models. Liciotti et al. [6]'s work shows that models with the best classification results are the LSTM with embedding and the bidirectional LSTM with embedding (see Figure 3.2). These two models have shown exemplary results on five CASAS datasets (Milan, Cairo, Kyoto2 Kyoto3, Kyoto4)

The Experiments Datasets

This thesis work was conducted thought several experiments on three CASAS datasets: Aruba, Milan and Cairo, created by the Washington State University [3]. The data collected came from real houses equipped with smart sensors and with real inhabitants. The sensor events log, see example in Figure 3.3, are generated from motion sensors (sensor IDs begin with "M"), door closure sensors (sensor IDs begin with "D"), and temperature sensors (sensor IDs begin with "T"). The log is defined by one event by row and six columns that correspond respectively to the date of the event, the timestamp of the event, the ID of the sensor, the value taken by the sensor, the activity during which the sensor was triggered and finally the status of the activity (it's begin / it's end). The three selected datasets are different in terms of housing structure, see floor maps in Figure 3.4, number of inhabitants and contain several months of labeled activities, details in Table 3 exactly the sames activities, it is therefore not easy to compare them directly. However, some of the activities could be grouped under a meta label or relabeled, in order to get a better comparison, e.g "Lunch", "Dinner" and "Breakfast" from the Cairo dataset could become the activity "Cook". Although each of the datasets originally included a large number of activities, for a part of this study we are interested in learning abstract models for 11 ADL activities as the works of Liciotti et al. [6] and Cook [2]. As seen before, a few number of activities are common in datasets, thus activities are grouped under new generic labels, detailed in the Table 3.2. The purpose of this relabeling, according to the authors of [6] and [2], is to allow a fairer comparison between datastets. More than just allowing us to make a consistent comparison of the results, this relabeling allows us to be comparable with the previous cited works. The

The Training and Evaluation Method

choice of the new labels in works of Liciotti et al. [6] and Cook [2] is also motivated by the fact that these labels are typically used to discriminate the functional health of an individual within a clinical scenario [12]. The previous cited works have already grouped the activity for the Milan and Cairo datasets, thus we used the same categorization for these two datasets and have grouped by our own the activities labels of the Aruba dataset.

It should be noted that this relabeling also has an effect of partially rebalancing the datasets, but also, paradoxically, increases the number of examples of a particular class called "Other".

This class corresponds to unidentified sensor activations or unidentified activation sequences, i.e. sensors events between the end of an activity and the beginning of another. As this class often represents more than 50% of the dataset, it introduces a bias which should be emphasized and kept in mind when analyzing the results. For example, if the classification algorithm is able to classify correctly all of the items in this class as "Other", then the global accuracy of this algorithm will be formally reported to be at least 50%, minimizing its actual accuracy for classes with meaningful labels.

Anomaly cleaning of datasets

After a detailed analysis of the datasets, we noticed that they can contain anomalies, especially in the "Milan" dataset: (1) they may contain duplicated data, full or partial days; (2) some row in the file contains characters errors, e.g "Oc" instead a "ON" value; (3) sensor activation traces in the dataset are not correctly ordered temporarily, i.e., within the time series, events with a later timestamp may be recorded before events with an earlier timestamp.

Activities in datasets are labeled with a "start" or "end" keyword to set when an activity begins and when it ends. However, activities can be nested, i.e., start1 start2 end2 end1. Moreover, it appends that they are interleaved, i.e., start1 start2 end1 end2. According to these observations, it is therefore important to pay attention to these particular cases when pre-segmenting the dataset into sequences of activities or to associate a label to a sensor event.

We observed, by reproducing the work of Liciotti et al. [6], that this cleaning has clearly an impact on the final results. We have noticed a loss of 5 points of accuracy on the "Milan" dataset using the bi-directional LSTM model proposed by authors. This loss can be explained in particular by a decrease in the number of occurrences of the "Other" class.

The Training and Evaluation Method

The Stratified K-fold Cross Validation

In this thesis, the stratified K-fold cross validation method [7] was chosen to evaluate and compare our proposed approach against the two selected baseline models. In particular, we used for our experiments the stratified (over class) threefold cross-validation procedure whith shuffling, see Figure 3.9.

This choice was motivated by our intention to apply the evaluation strategy used in Liciotti et al. [6]'s work. Moreover, it seems to be the best strategy as we work with pre-segmented activity sequences. Indeed, we want to recognise and classify sequence of activities independently.

In addition, to help models to be more general, it is better to shuffle all sequences and avoid time drift problem (if an activity is performed differently as the dataset progresses). The main idea of this evaluation method is to first shuffle all data (activity sequences) and then split them into, here, three folds. The split is applied in a so-called "stratified" manner, whose goal is to make folds by preserving the percentage of samples for each class. In other words, each fold contains the same percentage of samples for each class. Finally, each fold is used as the test set, while the two others are used for training.

The Process

To perform the training, the validation and the test of algorithms, we apply the procedure detailed in Figure 3

Metrics

In order to stop the model training and avoid overfitting, we used a method so-called "early stop". It consists in stopping the model training if the validation loss doesn't decrease since a specified number of epochs.

When the training is finished, the "best" model is finally evaluated with the test fold. Results

are saved and all the processes restart with another fold as test set. At the end, the reported results are the average of scores obtained on test sets.

Model Training Validation

Validation Results

Test Results

Stratified Three Fold Cross Validation Test

Metrics

Background

Using a meaningful measure of generalizability is an essential requirement for evaluating the performance of a classification algorithm on a given dataset. Since the actual ability of an algorithm to correctly predict class labels of unseen data could only be determined if an infinite amount of test data were available, generalizability must be approximated by an estimate. The ADLs recognition problem is a multi-class classification problem. The researchers have designed and used different metrics for classification tasks [11].

In the literature on recognition of ADLs metrics such as "Accuracy", "Precision", "Recall", and "F-Score" are mainly used to evaluate the algorithms [8,10,[START_REF] Medina-Quero | Ensemble classifier of long short-term memory with fuzzy temporal windows on binary sensors for activity recognition[END_REF]. These metrics are defined classes, but also to differentiate them from majority classes. Thus, it is important to evaluate whether a method or an algorithm is able to recognize this type of class or not. This type of evaluation demonstrates the strength of the method to not only look for features that separate the majority classes from the others, but features that seek to discriminate all classes. Therefore, it would be more appropriate to use measures able to deal with unbalanced datasets to properly evaluate the performance of the algorithms [4,5].

Balanced Accuracy

The Balanced Accuracy [1] expressed by Equation 3.1, avoids inflated performance estimates on imbalanced datasets. It is the macro-average of Recall scores per class or, equivalently, raw accuracy where each sample is weighted according to the inverse prevalence of its true class.

Thus, for balanced datasets, the score is equal to accuracy.

Balanced_Accuracy = 1 2 ( T P T P + F N + T N T N + F P ) (3.1)

Weighted Metrics

The "macro-average" scores simply calculates the mean of the metrics, giving equal weight to each class. In problems where infrequent classes are nonetheless important, macro-averaging is a means of highlighting their performance. The "weighted" accounts for class imbalance by

computing

Library and Report Tools

In order to evaluate and compare the performances of the algorithms, we have chosen to use the "classification_report" method provided by the "scikit-learn" library [9]. This method allows us to generate detailed reports that contain all the previous mentioned metrics excepted the Balanced Accuracy.

However, as complement to the measures provided by the report function, we used the Balanced Accuracy computation method also provided by the "scikit-learn" library. Thanks to all these metrics, we will be able to compare more finely the proposed methods to other algorithms and highlight their performances or their handicap, in particular for the recognition of minority classes.

Summary

This chapter introduced the approach developed in this thesis to attempt to improve ADLs recognition algorithms. The proposed approach is inspired by the language paradigm in order to better understand sensors' activation links, relations and meanings. The main hypothesis is, such as words in a sentence, sensor's activation in an activity observes a particular syntax, meaning and context according to the performed activity. Understanding this kind of information and knowledge should help to improve ADLs recognition algorithms. This chapter also describes the two Liciotti et al. [6]'s baseline models used for all experiments. It introduced the datasets exploited for the different studies and explained their prepossessing. It also detailed the evaluation methodology and the metrics used in the following chapters.

The next chapter addresses pattern recognition issues with respect to activity recognition.

An evaluation of a first approach using a CNN-based classifier as pattern recognition algorithm combined with an embedding as automatic features extraction algorithm is provided.

Chapter 4

THE TIME SERIES CLASSIFICATION APPROACH

Introduction

During the last two decades, Time Series Classification (TSC) has been considered one of the most challenging problems in data mining [3]. Due to its natural temporal ordering, time series data is present in almost every task that requires human cognitive processing [6]. Any classification problem using data registered by taking into account some notion of ordering can be cast as a TSC problem [4]. Time series are encountered in many real-world applications such as electronic health records [12], acoustic scene classification [10], cybersecurity [15] and in particular HAR [11,18].

In smart homes, human actions during activities are in fact logged by sensors as state changes along minutes, hours, and days and then ordered by the system. In other words, these logs contains sequences of state changes along time that translate the ADLs. Thus, the ADLs recognition problem can be modeled as can be modeled as a TSC problem. Through this modeling, the objective is to extract significant characteristics and sufficiently descriptive patterns from a time series.

Recently Wang et al. [19] Originally, the FCN was designed to aid in semantic image segmentation [9], where it has shown compelling quality and efficiency at this task. The main idea of this model is to consider each output pixel as a classifier corresponding to the receptive field, and thus the network can be trained pixel by pixel according to the semantic segmentation annotation by category, see The architecture proposed by Wang et al. [19], see Figure 4.2, follows the original architecture almost exactly. One of the main characteristics of this architecture is the replacement of the final fully connected layer, found in traditional convolutional networks, and the convolutional layer of the original architecture by a Global Average Pooling (GAP) layer [8] and a softmax layer.

The Fully Convolutional Network

The Fully Convolutional Network

The GAP layer drastically reduces the number of parameters while enabling the use of the Class Activation Map (CAM), as proposed by Zhou et al. [22]. The CAM is a method that highlights which parts of the input contributed the most to a certain classification. In greater detail, the architecture is composed of three blocks described by EQ 4.1. Where

x is the input, W the weight matrix, b the bias, ⊗ the convolution operator, and h the hidden representation. Each block consists of a 1D convolutional layer with Batch Normalization (BN) [5] and a Rectified Linear Unit (ReLU) activation to speed up the convergence and help improve generalizations. After the three convolution blocks, features are fed into the GAP layer. The idea is to generate one feature map for each corresponding category of the classification task.

The resulting vector is fed directly into the softmax layer to create the final classification.

y = W ⊗ x + b z = BN (y) h = ReLU (z) (4.1)
One advantage of GAP over the use of fully connected layers is that GAP is more native to the convolution structure thanks to how it enforces correspondences between feature maps and categories. Thus, the feature maps can be easily interpreted as category confidence maps.

Another advantage is that there is no parameter to optimize in the GAP; thus, over fitting is avoided at this layer. Furthermore, GAP sums out the spatial information, which makes it more robust to spatial translations of the input. Meanwhile, the main advantage of FCN is the invariance in the number of parameters across time series of different lengths. This invariance due to using a GAP layer enables the use of a transfer learning approach, where one can train a model on a certain source dataset and fine-tune it on the target dataset [2].

The Fully Convolutional Network for Activity of Daily Lining Recognition in Smart Homes

The Model

To increase the FCN performance, we propose to add an embedding layer between the input and the FCN classifier, see Figure 4

The Evaluation

In order to evaluate the method, the unidirectional LSTM and our FCN were compared with the widely used CASAS [1] benchmark datasets, describe earlier (cf. chapter 3), for two experiments. The first experiment evaluates: I. the abilities of both models to extract patterns and features from pieces of activity sequences; II. the contribution of the use of an embedding for the classification of ADLs; III. compare training times. The second experiment is a comparison between both models using the full activity sequences for classification instead of sequence pieces. It evaluates the performance of both algorithms to use short and long dependencies and patterns to recognize the ADLs.

Part, Chapter 4 -The Time Series Classification Approach

FCN and LSTM Comparison: Pattern Recognition Abilities

Table 4.1 show the performances of two FCN and two LSTM on raw sensor data for the two datasets (Aruba, Milan). Vanilla unidirectional LSTM / FCN and unidirectional LSTM / FCN with an embedding layer on different window sizes were evaluated. The average balanced accuracy and the average F1-score of were computed from the 3-fold cross validation results. FCN appears to obtain the best F1-score with and without the embedding applied to both datasets.

The LSTM is close to or equal to the FCN on a large SEW (windows greater than 50 sensors' activations). Compared to the FCN the LSTM seems to need more events to generate the classification.

From a balanced accuracy point of view, FCN often get the best values. Except when using the Milan dataset with a window size higher than 50. This decrease in performance seems due to the zero padding. Indeed, the average sequence length on the Milan dataset is around 88 events. When the window size is similar or greater than this average, the performance of the FCN decreases. Some small sequences like "Bed to Toilet" are so not classified. This results in a drop in the balanced accuracy score.

The FCN obtained the best values with SEW of sizes 50 and 25. Performance decreased as the SEW size decreased, but the FCN maintained a high score for balanced accuracy and the F1-score. Performance dropped less with FCN than with LSTM. It seems that FCN can generate more relevant automatic features than LSTM when using smaller sequences which contain less information. Moreover, look at results on both datasets, using an embedding layer increase the classification performances of both models. The embedding layer is thus, a relevant layer to extract meaningful features for both models and should be more studied.

Temporal Dependencies Evaluation

Method

Previous experiments show that, compared to the LSTM, FCN is faster to train and seems better at extracting and recognizing patterns in SEWs. Moreover, the experiments show that the use of an embedding layer increases the performance of both models. Nevertheless, it also shows that when the SEW is padded with zeros, in the case the SEW is larger than the activity sequence, the FCN loses performance. This suggests that the FCN is experiencing padding difficulties.

The previous experiment was performed using small windows, which does not allow us to see whether FCN can interpret temporal dependencies between sensor events like LSTM. In order to evaluate this, we will use the FCN with embedding model and the bidirectional LSTM, directly on the EWs. In other words, we now will use the entire activity sequence as input.

Thus, the models will have access to the whole sequence, thereby allowing them to evaluate the whole activity. In doing so, they can extract patterns and interpret the temporal dependencies within the sequence. This information is important because some events that are more or less

Part, Chapter 4 -The Time Series Classification Approach spaced temporally in a sequence can be correlated or even dependent. Which thus allows for better discrimination of the sequences between them. On contrary to the previous experiment, we used the bidirectional LSTM with embedding proposed in Liciotti et al. [7]'s work and not the unidirectional LSTM. This choice is motivated by the fact that the input is now longer, but also because this model has shown the best performance on the sequence classification problem for HAR in smart homes.

Some activities last longer than others, or contain more or fewer actions than others. Therefore, the sequences captured by sensor activations are often of different sizes, as they can contain greater or fewer sensor events. To train the algorithms, it is necessary for the sequences to be the same size. Sequences are either truncated, because they are too long for memory / learning efficiency reasons, or filled in with zeros, because they are too short.

To train the algorithms, as before, we used the Stratified 3-fold cross validation strategy (see chapter 3). Here, all EWs are distributed in three sets of data randomly but with equal distribution in terms of class. The results presented in the Table 4.3 summarize scores obtained after training the algorithms as described above. By looking at this table, we can see that the FCN have a very strong degradation of its performance compared to the LSTM in the previous experiment. This drop in performance could be explained by the zero-padding used to obtain sequence of same length.

FCN and LSTM Comparison: padding trouble

Where the LSTMs is able to ignore the padding, the FCN might interprets it as useful data.

Other works have shown that the processing of time series of different lengths is problematic [17] and methods to re-scale or re-sample could be applied. However, in our case of time series in to a sensor activation. As can be seen in Figure 4.5a, the FCN places great importance on padding, which distorts the result. Indeed, the padding should not be part of the pattern being recognized. We can see through the zoom in on the beginning of the sequence in Figure 4.5b, that the padding seems to be more important than the useful data located at the beginning of the sequence. The FCN therefore learns where the padding is located and the size of the padding, but does not extract a useful pattern. This interpretation of the padding by the FCN is caused by the representation of the padding generated by the embedding. In fact, the embedding assigns a representation vector to this value 0. To attempt to solve the zero-padding problem, we replaced this type of padding with others, as illustrated in Figure 4.6: specifically symmetric-padding [20] and circular-padding [13]. Instead of adding to the end of the sequence of zeros, the symmetric-padding instead repeats the sensors' activations according to an axis of symmetry. The circular padding, on the other hand, repeats the complete value sequence infinitely. These types of padding act as data augmentation and force the FCN to focus on patterns.

Padding Methods Comparison

By using a Grad-CAM on a sequence completed with a circular padding for example, see If we zoom in as we did previously on the beginning of the sequence, see Figure 4.7, we can clearly see the pattern the FCN is focused on. As we can observe, the FCN is always focused on the same pieces of sensors that seems to be the beginning and the end of the sequence. Some activations in the middle seem also important for this activity.

As visible in Table 4.4 the symmetric padding allows FCN to obtain the highest scores on the Aruba and Milan datasets. However, it degrades the performance of the FCN over the Cairo dataset. Meanwhile, we can see that the scores are more balanced with the use of circular padding, with the only main concern being that this technique does not significantly improve results for the Cairo dataset. We can potentially explain the poor scores on the latter by the fact that it is a dataset containing two residents and is therefore more complex. The patterns are less easily identifiable because the sequences of one resident are affected by the actions of the second resident. Padding is certainly a problem for FCN but it seems that pattern recognition is not enough to recognize activities.

FCN and LSTM Final Comparison

The strength of FCN is its ability to recognize patterns while LSTM uses temporal relationships in addition to patterns. Comparing the best results, or at least the most balanced ones, obtained with the circular padding against those obtained with the bidirectional LSTM (see Table 4.5), we can see that the scores of the bidirectional LSTM are largely superior. In particular, the scores of the bidirectional LSTM are most effective for not only the Cairo dataset, but also the Milan. We can clearly see that pattern recognition alone is not sufficient for full recognition of activities. Rather, it is necessary to interpret the temporal relationships within the sequence.

In comparison, despite FCN being a structure known for its ability to extract relevant features and its performance in terms of time series classification, is not sufficient for activity sequence classification.

Summary

This chapter presented a hybrid approach that combines an input representation from the NLP domain and a classifier algorithm from the TSC domain to create an end-to-end model. Our assessment on two datasets with SEW over activity sequences shows that the embedding significantly improves the performance of LSTM and FCN in all cases. This means that the domain knowledge incorporated in the embedding can improve the understanding of the sensors' activation by LSTM and FCN.

By using SEWs we saw that the FCN obtained the same or greater performance than the LSTM, with the exception of two configurations only. We also found that by in large the FCN was quicker to train. Moreover, the FCN outperformed the LSTM when the window size decreased.

Summary

This seems means that the FCN can extract meaningful patterns.

Nevertheless, when the FCN is compared to the LSTM over full activity sequences with zeropadding, the LSTM outperforms drastically. We tried to solve this padding problem by using two other padding methods, circular-padding and symmetric-padding. The circular-padding method improved the FCN results to hell the LSTM ones. This padding approach showed the highest performance and most balanced results on the three studied CASAS datasets. Further, the method demonstrated that pattern recognition can be used to get high ADLs recognition scores over two datasets Aruba and Milan. Unfortunately, the FCN approach combined with the circular padding did not outperform the bidirectional LSTM approach. This gap is even more visible on the Cairo dataset, where the FCN approach is still very far behind the LSTM method.

In the first experiment as all SEW were shuffled and split into train and test sets, the data of subsets was similar, thus only the pattern recognition was necessary. However, the second experiment focuses on the ability of models to extract the information over all the activity sequences. In addition, in this second experiment the data of the train and test subsets were less similar.

We saw with this last experiment, that temporal dependencies were also important, which allowed the LSTM to outperform the FCN. Indeed, patterns in datasets with pets, such as Milan, or with multiple residents, such as Cairo, are less easy to extract, due to the potential noisy sequences. These datasets need more than just pattern recognition. They need sequential and temporal relations understanding, to extract the actually important information, because some sensor activations are temporally correlated and linked with the current activity while others are not. In the next chapter we will explore how to improve the embedding layer to encode, understand, and tackle the question of long temporal dependencies and sensor's activation context.

[22] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921-2929, 2016.

Chapter 5

LANGUAGE MODELING FOR HUMAN ACTIVITY RECOGNITION

Introduction

The common points of previous approaches using deep learning algorithms for the human activity recognition in smart homes is that they extract automatically features and patterns from raw data. LSTM-based approaches have demonstrates the best results as they can capture patterns and temporal dependencies. However, they ignore some long dependencies, the semantics, the sensor type, the syntax and the context in which the sensor is activated.

Semantic analysis, context modeling and words relations have been, very early and naturally, at the center of NLP research. The latest advances in this field have proposed various unsupervised pre-training methods of projecting words into a vector space, called embedding, to create word representations and language models [5,7]. These embeddings capture information about the construction of words, sentences and texts. They are also able to capture the context of a word in a document, the semantic or the syntactic similarities, relations with other words, etc. They have proven to be effective in transfer learning on translation, text generation and classification tasks. Embedding in the NLP field can be classified into two categories. Static embedding approaches, such as Word2vec [11], GloVe [12], FastText [2], etc. And contextualized approaches, such as ELMo [13], BERT [4], GPT [14], etc.

In the context of human activity recognition, previous work has already investigated the use of embedding methods. The training of a Word2Vec model was used to cluster and create a semantic relations map of habits of a population in a city [3]. The use of a public pre-trained, on texts, Word2Vec model was used to associate a label with unknown activities in the context of worn sensors [9,16]. The same approach has been studied to annotate unknown activities and perform "zero-shot" learning [6] in a smart home [1].

In the previous chapter we saw that an embedding layer improve models classification results.

Unsupervised embedding training, as used in the NLP domain, but for sensor activations, has not been exploited. Capturing the context and semantics of sensor activation is not exploited either.

However, thanks to the NLP embeddings' generation methods, it would be possible to capture Part, Chapter 5 -Language Modeling for Human Activity Recognition this information of sensor activations. This type of information would enrich the knowledge on the activation in question and would allow distinguishing it according to the cases of appearance.

In previous chapters, we have already drawn attention to the fact that home automation sensors or sensors embedded in everyday objects provide little information. It is the type of sensor, the type of activation, the order and the context composed of sensor activations before and after a given sensor, that provide richer information that can be better exploited. This why bidirectional LSTM approaches seems to better perform than unidirectional LSTM ones, in addition to the ability of bidirectional LSTM to better encode long sequences.

This last remark already indicates that it would be necessary to study some notion of semantics, syntax and context related to each activation. Indeed, the encoding of the ith activation data should not depend only on its value and be expressed as a function f (activation i ). Instead, it should capture more information on sensor behavior, incorporating relevant neighboring activations while smoothing out variability and noise. In other words, the sensor encoding at ith activation should also depend on the other activations in the sequence, and be expressed as a function f (activation i , {activation 1 , activation i-1 }, {activation i+1 , activation n }), where n is the number of activations in the sequence. This could improve the performance of ADL classification algorithms. Moreover, the creation of a pretrained embedding could allow, by transfer learning, to perform the human activity recognition task in new houses.

This chapter focuses on using advanced embedding methods inspired by NLP domain to obtain a better sensor representation in order to improve a LSTM-based classifier. First, we will dive into the world of semantic embedding with Word2Vec and see what kind of information and functionality such a model can provide. Second, we will explore the language model principle and contextualized embedding domain with ELMo. Third, we will evaluate and observe the potential improvement of using advanced pre-train embeddings. Finally, we are going to evolve if it is possible to transfer knowledge from one house to another via an embedding.

Sensor Events: The Semantic and Context Embedding

Word Embedding : Word2Vec

Word2Vec is one of the most popular techniques to learn word embeddings using shallow neural networks. Proposed by Mikolov et al. [11], Word2Vec is an unsupervised learning technique to learn continuous representations of words. In many ways, Word2Vec builds on a bag of words, but, instead of assigning discrete values to words, it learns a continuous multi-dimensional vector representation for each word in the training corpus. There exist two training modes for learning representations of words with the Word2Vec algorithm, the Continuous Bag Of Words (CBOW) and the Skip-Gram modes.

As seen in Figure 5.1, these two methods are essentially the reverse of each other. In both methods, all the words in the vocabulary are one-hot encoded and a window size is defined to consider a fixed number of words. The CBOW is trained by learning to predict the center word in each window based on its surrounding words (context words), as in Figure 5.1a. Once a prediction is made, the difference between the predicted vector and the ground truth is backpropagated to update the dense layer that define the embedding. On the other hand, the Skip-Gram method is trained to predict, given a target word, the most probable words in a fixed sized window around it (context words), as in Figure 5.1b. Generally, Skip-gram is better with a larger corpus but is slower to train, while CBOW is more suitable if the corpus is small or speed is a factor. By design, Word2Vec captures the similarity of words in a corpus. Moreover, thanks to the distance of a word from other words it calculates, Word2Vec also captures part of the meaning of the word. CBOW, for example, learn better syntactic relations between words while Skip-gram is better in capturing better semantic relations. In practice, this means that for the word "cat", CBOW would retrive as closest vectors morphologically similar words like plurals, i.e. "cats".

On the other hand, Skip-gram would consider morphologically different words (but semantically relevant) like "dog" much closer to "cat" in comparison.

Moreover, because Skip-gram rely on single words input, it is less sensitive to overfit frequent words. Indeed even if frequent words are presented more times that rare words during training, Part, Chapter 5 -Language Modeling for Human Activity Recognition they still appear individually. CBOW is prone to overfit frequent words because they appear several time along with the same context. This advantage over frequent words overfitting leads Skip-gram to be also more efficient in term of documents required to achieve good performances, much less than CBOW. It's also the reason of the better performances of Skip-gram in capturing semantical relations.

Word2Vec for Sensor Event Embedding

As it is made for words, we have trained a Word2Vec model for sensors in order to capture sensors and sensors' activation relations. To do so, we used the Skip-Gram method, described above, to predict surround sensor activations of a middle one in a window. This choice was motivated by the strong ability of the Skip-Gram methods to be more semantically relevant. Indeed, we don't want to learn only if activations from a same sensor are close or not, but we also expect to capture relations between sensors' activation.

M004ON M005OFF M007OFF M004OFF M004ON M004OFF M007ON M004ON M007OFF M007ON M005ON M004OFF M004ON M005OFF M007OFF M004OFF M004ON M004OFF M007ON M004ON M007OFF M007ON M005ON M004OFF M004ON M005OFF M007OFF M004OFF M004ON M004OFF M007ON M004ON M007OFF M007ON M005ON M004OFF M004ON M005OFF M007OFF M004OFF M004ON M004OFF M007ON M004ON M007OFF M007ON M005ON M004OFF M004ON M005OFF M007OFF M004OFF M004ON M004OFF M007ON M004ON M007OFF M007ON M005ON M004OFF M004ON M005OFF M007OFF M004OFF M004ON M004OFF M007ON M004ON M007OFF M007ON M005ON M004OFF M004ON M005OFF M007OFF M004OFF M004ON M004OFF M007ON M004ON M007OFF M007ON M005ON M004OFF M004ON M005OFF M007OFF M004OFF M004ON M004OFF M007ON M004ON M007OFF M007ON M005ON M004OFF
Input Prediction A zoom on the grey cluster, "kitchen", on the top of the room colored picture, is provided in Figure 5.5a and can be compared to the floor map of the house in Figure 5.5b. All the sensors associated with the kitchen on the plan are in the zoomed cluster. Two sensors' activation, "M014ON" and "M014OFF", that aren't in the "kitchen" but belong to the "dining room" appear close to some kitchen sensors (M018 and M017 activations). This close distance is quite coherent as they are physically close and are correlated to zones of passage between these two rooms. After examining the different clusters, they all seem to correspond to a room in the house. This grouping into rooms is quite consistent because ADLs are usually located in one room of the house. It is important to note that, in an unsupervised way, the Word2Vec method has captured the notion of sensor localization by grouping the sensors belonging to the same room. This sensor localization clustering was also observed by Singla and Bose [17] in their work on IoT devices identification with binary sensors.

Concerning the activation of the temperature sensors, it seems that they are also grouped in sub clusters belonging to the same room. If we zoom in the main office temperature sensor cluster, we can observe that temperature are ordered according to temperature values, see Figure Generally, in the Aruba dataset a high temperature in the kitchen means that the resident is cooking. This sensor correlation between the kitchen, the office and the living room temperature sensors should help the classifier to better understand when the resident is cooking.

Here, we have illustrate only the case of the Aruba dataset. However, we can observed similar results on all the evaluated datasets, see Appendix B.

Sensor2Vec: Activity Sequences Embedding Visualization

In the previous section, we have visualized and attempted to interpret the features learned by the Sensor2Vec model. We will now plot the embedding of each activity sequence in order to see if some activity clusters appear thanks to the features provided by the Sensor2Vec model.

Aiming at visualizing this embedding of activities sequences, we created a model composed of the Sensor2Vec layer and a GAP layer [8] (see Figure 5.7).

The Sensor2Vec layer provides a feature vector representation for each sensor activation in the input sequence. The GAP layer summarizes these feature vectors into one vector which is the activity sequence embedded vector. Once all activation sequences are turned into a vector, as for Sensor2Vec visualization we use the UMAP to reduce each activity vector to a two-dimensional vector. Each activity vector is then displayed and tagged with a color corresponding to the activity label.

Sensor2Vec

Global Average Pooling activity sequence activity sequence embedded vector On the Aruba dataset figure, we can see the "Sleeping" activity cluster that is almost isolated, as the "Bed to Toilet" and "Work" activities clusters. It seems that an appropriate classifier should easily find boundaries between these clusters. However, this embedding method does not seem to be efficient enough to isolate in individual clusters all the activity classes. The majority of the clusters are connected by the points corresponding to the activity sequences labeled "Other". Similar activities such as "Enter Home" and "Leave Home" are still mixed. The more complex the dataset, the less the activities are grouped into clusters. In the Milan figure, activity clusters are in the periphery but not completely isolated. The "Bed to Toilet" activity is mainly mixed with the "Bathing" activity. Two clusters corresponding to the activity "Bathing" appear, however it seems possible as the Milan house has two bathrooms. Only the "Work" activity cluster looks mainly isolated. This lets us think that this activity will be well recognized compared to the others. Generally for this dataset activities seem mainly recognizable. However, it seems that activities will have a high confusion with the "Other" class. For the Cairo dataset the both activities "Sleep" and "Work" are in majority isolated. However, all "Bed to toilet" dots are mixed with a subset of "Sleep" dots. The activity "Take Medicine" is completely absorbed in the "Eat" dots activities. We will expect the baddest classification results on this dataset with this embedding method. 98

Sensor Events: The Syntax and Context Embedding

This pre-trained embedding method seems promising to improve a ADLs recognition algorithm. Coupled with the UMAP algorithm, in an unsupervised way, let's generate almost isolated activity clusters. However, even if the Word2Vec method is a powerful embedding technique and allows us to create a meaningful Sensor2Vec embedding, this embedding method has a main drawback. It provides a single representation for each word / sensor, regardless of the context.

Put differently in the NLP field, the word "orange" in syntagms, such as "the orange juice" or "the orange car", garners the same vector representation, although "orange" does not have the same meaning in the two sequences of words. This lack of context understanding is an important issue to capture the sentence's meaning and introduce the well-known polysemy problem.

As said earlier, sensors meaning information depends on surrounding sensors' activation such as words in the sentence, cf. above example. The NLP field provides advanced embedding methods that can understand word context, and can represent the same word according to its context. In the next section, we will study one of them for the ADLs recognition problem.

Sensor Events: The Syntax and Context Embedding

The Language Model Principe

Before diving into the world of advanced embedding methods, it is important to understand the language model technique used in the NLP field. To illustrate what is the intuition behind a language model, let's take the following sentence example, "Project A has an elevated risk of ...".

By reading this incomplete sentence, we are able to predict the next word after "of". Between the two words "delay" and "water" the most probable word seems "delay". Indeed, we have some knowledge about how a sentence is built according to rules and the sentence context. This knowledge is called a language model. More formally, the NLP field define a language model, basically as a probability distribution over words or word sequences. Thus, to generate language models, NLP researchers train deep learning models on the task, which is simply to assign a probability to any sequence of words (see Eq 5.1).

P (w i |w 1 , w 2 , ..., w i-1 ) (5.1)
The Word2Vec method is a kind of simple language model method as it uses surrounding words to predict the middle one, or vice versa, from a window of word. Thus, it computes a probability between words. However, it is built over a "bag of word" technique. Therefore, when the model is trained, the surrounding words (context words) are used without any order consideration. Based on this assumption, modern deep learning-based language models use structures able to learn this order but also the dependencies between words.

Part, Chapter 5 -Language Modeling for Human Activity Recognition

The Perplexity Metric

In the NLP domain, language models are trained and evaluated using a particular metric, the "Perpexity" [18]. It indicates the variability of a prediction model. In other words, perplexity is a measurement of how well a probability model predicts a sample, a word in the NLP context.

It may be used to compare probability models. A low perplexity indicates the probability distribution is good at predicting the sample. Lower perplexity corresponds to lower entropy and, thus, better performance. The perplexity for a language model can be defined as the inverse probability of the test set, normalised by the number of words, EQ 5.2.

P P (W ) = N 1 P (w 1 , w 2 , ..., w i-1 ) (5.2)
W is the test set. It contains the sequence of words of all sentences one after the other, including the start-of-sentence and end-of-sentence special words, <SOS> and <EOS>, usually used in the NLP field. For example, a test set with two sentences would look like this:

W = (< SOS >, F irst, sentence, ., < EOS >, < SOS >, Second, one, ., < EOS >)
N is the count of all words in the test set, including special words and punctuation. In the example above N = 10. It can also be defined as the exponential of the cross-entropy. This last definition is generally used for the model training.

Contextualized Word Embedding: ELMo

Generally, word embeddings, such as Word2Vec, fail to deal efficiently with lexical polysemy.

These word embeddings cannot take advantage of information of the context in which the word was used. Instead of using a fixed embedding for each word, ELMo [13] looks at the entire sentence before assigning each word in its embedding. The core idea behind contextual word embeddings is to provide more than one representation for a word, based on the context in which this word appears. ELMo acquires its understanding of language by being trained to: (1) predict the next word in a sequence of words ("language modeling", Eq 5.1), and (2) also predict the previous word ("reverse language modeling", Eq 5.3). This association defines the "bidirectional language modeling" and the two tasks are performed in the same time by the model.

P (w i |w i+1 , w i+2 , ..., w N ) (5.

ELMo for Sensor Event Embedding

Based on the ELMo model, we have trained a sensor contextualized embedding. Our objective is to embed a sensor vector activation according to its context, such as words. To do so, we used the first version of the ELMo model (without the CNN layer). This choice was motivated by the Part, Chapter 5 -Language Modeling for Human Activity Recognition fact that the "smart home vocabular" is less extensive than a real language, so unknown values should be rarer. In addition, the datasets provide a large amount of example sensor activations which should cover all possible variations.

The ELMo model inputs are activity sequences. As activity sequences can be long and in order to accelerate the training of the model, we limit the past and the future context length as it is done in the original ELMo implementation.

As in the NLP domain, two special words, "<start>" and "<end>", are added into the vocabulary to mark the beginning and end of activity sequences. These special words allow us to use the last and the first sensor activation of an activity sequence. In other words, the last sensor activation of the sequence will predict the special word "<end>" in the forward network and the first sensor activation will predict the special word "<start>" in the backward network (see Figure 5.12).

Moreover, as it is usually made in the NLP field we also add to the vocabulary another special word, "<UNK>", to prevent out of vocabulary words. Out of vocabulary words refer to the new words which are encountered at testing. These new words do not exist in the vocabulary.

Hence, these methods fail in handling these words. In our case, thanks to this special word, an unknown sensor's activation in the test dataset will be replaced by the word "<UNK>", allowing the model to still encode and represent the sequence.

Sensor Events: The Syntax and Context Embedding

To train the sensor ELMo version, we respect the same procedure as in the NLP domain for language model generation. The corpus is divided into three subsets for: training (80%), validation (10%) and testing (10%). ELMo uses only the context contained in the sentence. In other words, the context of the previous and following sentences is not taken into account. Therefore, to realize this division, we proceeded as follows. Each sentence is considered independent. This allows to shuffle the whole corpus (set of activity sequences) and to divide it according to the above proportions. Each subset contains a percentage of the activity sequences without order or dependency relations. During training, we used perplexity to track the learning performance of the model in place of accuracy. All ELMo models were trained following the parameters detailed in Table 5.2

ELMo Sensor: Activity Sequences Embedding Visualization

The main advantage of ELMo is to provide more than one representation for each word based on the context in which it appears. Therefore, as the word vector provided by this embedding depends on surrounding words, it is not appropriate to visualize ELMo embedding of isolated words. However, the visualization of the sentence embedding garners the appearance of some interesting cues.

In order to visualize the embedding of the activity sequences, we proceeded in the same way as for the Sensor2Vec method. A GAP layer is added to the ELMo embedding to generate a unique activity sequence embedded vector (see Figure 5.13). Then, these vectors are reduced via the UMAP algorithm to be displayed into a 2D-dimensional space. Finally, each dot is colored according to the activity label. The Figures 5.14 Globally, compared to the Sensor2Vec method, the clusters proposed by the association of ELMo and UMAP are more isolated from each other. This seems means that ELMo is able to extract more revealing features than Sensor2Vec. Moreover, within the activity class "Other", isolated clusters appear. We assume that these clusters of activation sequences, labeled "Other", are in fact potential new activity classes. ELMo mixed with UMAP seems a possible way to discover new activities. This visualization demonstrates the strong ability of the ELMo embedding to capture relevant features from raw sensors activations. However, this clustering is not perfect, when looking closely at the clusters. Some points of a different color than the cluster majority color can appear. This mix can be explained in two ways. First, some sequences belonging to two different classes may be very similar in terms of sensor activation because there is not enough sensor to distinguish them. Second, it is potentially a labeling error in the dataset. Indeed, it is difficult to create and label datasets about human activities in houses [2].

Looking at the Aruba ELMo embedding projection (Figures 5.14) we can observe that sequences that belong to the same class seems well grouped. However, we can note an important class confusion between "Enter Home" and "Leave Home". This confusion seems normal as these two activities are very similar. Sequence that belong to the "Eat" activity are totally mixed into clusters that belong in majority to the class "Other". We can observe similar clustering errors on the two other datasets. For Milan "Bathing" and "Bed to toilet" activities are mixed. We can see the same things for the activities class of "Cook" and "Take Medicine". For the dataset Cairo, the "Eat" and "Take Medicine" activities seem the most mixed. In addition, the "Bed to toilet" activity is often mixed with the activity "Other".

The combination of the ELMo pretrained embedding and a dimension reduction algorithm, such as UMAP, seems able to make unsupervised clustering of similar activity sequences. Indeed, the training of the ELMo embedding and UMAP are done in an unsupervised way, i.e., by using no labels. Therefore, it seems possible to group pre-segmented activity sequences completely unsupervised. In other words, if it is possible to split a dataset into unlabeled activity sequences and, by using ELMo, and UMAP it seems possible to group sequences belonging to almost the same class.

ELMo Outputs Comparison

Usually, the ELMo representation is the weighted sum of the outputs of the different layers level of the model, where the weights are trained according to the final task (see Figure 5.17a).

However, three other output forms were evaluated, as depicted in Figure 5.17. First, the sum of outputs by level (Figure 5.17b). Secondly, the last output layer of the model (Figure 5.17c).

Finally, the concatenation output of the different layers of the model (Figure 5.17d).

Embedding Output The Table 5.3, shows the comparative results for the three studied datasets. To perform this experiment, we replaced the embedding layer of the bi-directional LSTM neural network structure proposed by Liciotti et al. [8] with the various ELMo outputs. From the Table 5.3, we can see that the "Concat" output outperforms all others except the Milan dataset in terms of recall, F1-score and balanced accuracy. We assume that this performance comes from the fact that the "Concat" output gives to the classifier the choice to select which feature is the most relevant from each level of representation. However, except the "Last" output version, the "Weighed Sum" and the "Sum" have very close scores. The "Weighted Sum" output has a slight advantage over the "Sum" output. Based on this result all the next experiments use the "Concat" output to train the classifier part.

Forward LSTM L1 Output Backward LSTM L1 Output Forward LSTM L2 Output Backward LSTM L2 Output Embedding Output S1 S2 S3 x x x + + [ [ [ ] ] ] = ELMo

Comparison with Context-Free Embeddings

To evaluate our proposed methods using pre-trained embbeding, we compare them to the baseline models define in Chapter 3, which does not use contextual information. We also add two models for comparison, a LSTM and a bidirectional LSTM without embedding layer, to validate the improvement of this layer.

The experiment results in Tables 5.4 and 5.5 show that the bidirectional LSTM obtained better results, as shown in reference [8]. In addition, using an embedding layer improved the global classification performance, thanks to the ability of this layer to capture meaningful features between sensors' activations. This last confirm the utility of using a projection layer.

Against all odds, we notice that the Sensor2Vec embedding did not improve the classification performance compared to baseline models. On the contrary, using Sensor2Vec decreases the performance. This seems means that the Sensor2Vec embedding did not accurately capture some important features by itself. The knowledge of some sensors semantic and localisation seems not enough.

The ELMo embedding performs better than all other methods on every dataset except on the Milan dataset but shows a high improvement on the multi-user dataset Cairo. It increases the F1-score and the Balanced Accuracy by 10 points. Multi-user dataset seems requires the understanding of the context in order to differentiate users' activities. For the Milan dataset ELMo is slightly behind the Liciotti-baseline model, but ELMo win in precision. Concerning the training time, we noted a negligible gain of about 30 s to 2 min, depending on the dataset, by using a pretrained embedding.

In order to evaluate if the ELMo model could improve a less efficient classifier, we also evaluated the ELMo embedding coupled with the FCN studied in Chapter 4. For this experiment we used the FCN with the circular padding as it demonstrated the best performances. The experiment results in Table 5.6 show that the FCN combined with ELMo help to reduce the gap between the standard FCN model and the Liciotti-baseline model. One again the most important improvement can be observed over the Cairo dataset.

Part, Chapter 5 -Language Modeling for Human Activity Recognition

Comparison Against Staked Bidirectional LSTM

The ELMo model can be approximated by a bidirectional LSTM layer with an embedding layer.

In this experiment, we compare the ELMo method to two stacked layers of bidirectional LSTM with an embedding layer. Table 5.7 shows the results of this comparison. The results demonstrate that the ELMo structure obtains a non-negligible gain on the three datasets, except on the Milan dataset. On this one, gains are less, but not negligible, on the Precision and F1-score. ELMo still allows recognizing more classes and is more precise than the other structures. The stacking of bidirectional LSTM does not allow obtaining better performance. This type of structure can even degrade the performance in some cases, as on the Cairo dataset, or Aruba. It seems that the method to train ELMo helps to capture more useful features.

Confusion Matrices Analysis

The confusion matrices in Appendix C presents the confusion and classification scores activities for the three-folds experiments over the three studied datasets for the Liciotti-baseline (embedding + bidirectional LSTM) and ELMo + bidirectional LSTM approaches. We observe that the misclassification rate (1 -accuracy score) is lower with ELMo method in general except over the Milan dataset.

The performance of both models are very close for the dataset Aruba. We can observe more differences for the dataset Milan. Depending on the folds, the Liccioni-baseline model seems to perform a little bit better than the ELMo approach. However it seems that the ELMo model can recognize better the activity "Eat" 2 folds over 3.

Part, Chapter 5 -Language Modeling for Human Activity Recognition

Concerning the dataset Cairo, the ELMo model confirms its performance. It recognizes all activity classes independently of the fold. Moreover, the ELMo model shows a better percentage of recognition in classes.

However, these matrices show a simplified version of the problem. Indeed, we have grouped some activities under a "meta" label in order to compare fairer the datasets between them. Thus, we retrained models over the three datasets according to their original activity label and results are a little bit different (see confusion matrices in Appendix D).

For the Aruba dataset, we observe that the medical activity "Respirate" is not correctly recognized for the both methods. This activity has very few examples (5 examples) and, therefore, is difficult to recognize. The ELMo approach demonstrates the ability to recognize this activity over the three folds while the Liciotti-baseline model recognized it in only one fold.

The activity "Wash Dishes" is poorly recognized and is completely confused with the activity "Meal Preparation". This mixing is explained by the fact that these two activities activate the same sensors. This misclassification was already observable on the previous Aruba's confusion matrices where a small part of the "Work" activity is mixed with the "Cook" activity. In order to differentiate these two activities, the algorithms should be able to take into account the time of day, or the activity preceding the current activity.

In the case of the Milan dataset principally three activities seems poorly recognized ("Chores", "Dining_Rm_Activity" and "Eve_Meds"). The activity "Dining_Rm_Activity" is poorly recognized because it mainly activates only the motion sensor located in the dining room, which is often activated during resident travel between the living room and the kitchen. However, the ELMo approach is able to find a part of the occurrences of this activity. This performance is explained by the fact that ELMo captures in which context the motion sensor in the dining room is activated. In other words, it is able to differentiate between the resident's passage in the dining room to join the kitchen or the living room, from a real activation of the sensor for this activity. The "Chores" activity is very similar to the activity "Other" as during this activity the resident moves in all the house and activates random sensors. The activity "Eve_Meds" is mainly mixed with the activity "Morning_Meds" which seems rational as it is the same activity but at two particular moments of the day.

Cario's confusion matrices show the difficulty of both models to correctly identify the activities that belong to the "Cooking" activity ("Breakfast", "Dinner" and "Lunch"). Indeed, these three activities depend on the time of the day. We can see that the ELMo approach is able to almost distinguish them. Moreover, ELMo demonstrates to in general better recognize all activities.

5.5. Pre-Trained Embedding: From One House to Another 5.5 Pre-Trained Embedding: From One House to Another

Help Classifiers with Embedding Transfer Learning

In the field of NLP, embeddings are pretrained on large corpora and then used on a more specific corpus to perform particular tasks, such as text classification. This is the principle of transfer learning. The objective is to use the very generic features of the large corpus on a smaller and specific corpus to gain in learning time but also in genericity. In the case of ADLs recognition, this practice would allow for transfer the knowledge from a smart house to another one so that the latter can recognize ADLs without further training. This transferred model could then be refined for the context of this new house. We experimented with this practice using Aruba's ELMo embedding on the Cairo and Milan datasets.

To evaluate the transfer learning ability of a ELMo pre-trained sensor embedding we ex- Looking at the result in Table 5.8, the generic features learned from the Aruba dataset allowed the classification of activities in the Cairo dataset with scores a little bit less, but close, compared to the ELMo embedding trained on Cairo. Nevertheless, the ELMo from Aruba outperform the Liccioti-baseline which is train on the target data. We can observe on contrary on the Milan dataset that using the ELMo from Aruba improve sightly results compared to the ELMo originally trained on Milan. We can also see that over this dataset the ELMo from Aruba-based model shows better performance against the Liciotti-baseline in term of precision, recall and F1-score.

Part, Chapter 5 -Language Modeling for Human Activity Recognition

But, why the embedding transfer learning works ? Indeed, the two target datasets do not have the same number of sensors and can have different sensors values, in particular for the temperature sensors, compared to Aruba. However, the three datasets belong to the CASAS benchmark datasets, which use the same sensor type and the same denomination structure. The sensors follow the structure: "sensor type" + index. Moreover, Aruba have a larger vocabulary which theoretically avoid out of vocabulary words problem. The Aruba vocabulary have a size of 259 possible words, while Milan have 136 different words, and Cairo have 197 possible words.

Nevertheless, the house typologies are completely different, and sensors with the same names not sharing the same location or function between datasets. In other words, a sensor with the name ID "M019" which is a kitchen motion sensor into Aruba is motion sensor for the bedroom door in Milan and a motion sensor for the kitchen cabinet in Cairo.

However, we observe that, even if a word does not have the same "meaning" from one dataset to another the encoding provided by the ELMo embedding generate patterns still allows the classifier to reach performance rates equivalent to a model fully trained on the destination dataset. We conjecture this good performance comes from the fact that ELMo takes into account the word order and generate a similar "syntax" for activity sequence that belong to same classes.

Thus, even if the input words have changed, the syntax is captured by the classification part of the model which compensate the gap, i.e., the order of each word, as well as the patterns of their recurrence in the sequence. These results indicate the importance of contextualized embeddings. It seems that particular patterns belong to activity classes. Nevertheless, the observation of these patterns is not always obvious. Even if our hypothesis seems to be validated, it would be interesting to be able to evaluate or use more adapted and advanced methods of explicability [15].

Embedding Output Analysis

Summary

Summary

Human activity recognition in smart homes is not only a problem of pattern recognition but also a temporal sequence analysis problem, where the semantics and context of each sensor trigger can change the meaning of a sensor activation. Moreover, the nature of the activated sensor can give a certain amount of information about the current activity.

In this chapter, we proposed a new approach to the field of recognition of ADLs in smart homes. We used techniques from the domain of NLP to capture the context and the semantics of sensor activations in an embedding. This approach allows recognition of a larger number of activity classes, despite the fact that datasets remain unbalanced. Indeed, fewer activation sequences are confused with the "Other" class, which, nevertheless, represents more than 50% of the datasets.

The visualization of the Sensor2Vec embedding allowed us to realize that this method has captured some relations between the activations of sensors. It appears that sensors of the same nature have a close distance in this embedding space. Moreover, the clusters that appear in the space represent different rooms in the house. Nevertheless, this semantic embedding does not provide better performance than the Liciotti-baseline models.

However, our experimentation shows that capturing the context of a sensor activation allows for improvement of the classification of activity sequences, particularly on datasets containing activities performed by several residents. Moreover, we have seen that the contextual ELMo embedding allows a less efficient classifier, especially in our case the FCN, to gains drastically in performance.

Finally, we were able to evaluate that an embedding trained in a house seems to be reused in a new environment containing another denomination of sensors and allow a high rate of classification of activities. It should be noted that these methods are capable of extracting generic information, transferable to other datasets. This last observation suggests that transfer learning between environments is possible through these methods, as it is possible today in the NLP domain.

Through our proposition, that combines a sensor activations model based on language model with a sequence model, our experimental results on real smart home data highlight the importance of a dynamic contextualized semantic representation in ADL recognition. However, we have seen with the original datasets labels, that some activities are not correctly recognized.

These activities seem to rely on the time of the day or to potential previous activities. In the next chapter we propose to explore the idea to improve ADLs recognition algorithm thanks to time information or previous activities context.

Chapter 6

TAKING ADVANTAGE OF DEPENDENCE ON PREVIOUS ACTIVITIES AND TIME OF THE DAY

Introduction

We have seen in the previous chapters that human activity recognition implies a series of complex modeling. Previous works consider human activities as sequences of sensor activations which reflect the actions of residents. They do not take into account temporal information nor do they focus on the recognition of meta activities such as "Cooking", "Eating", "Sleeping", etc., that may be useful for some health monitoring applications [18,19]. Furthermore, it also appears to be useful for systems to recognize more particular activities such as "Preparing Breakfast", "Eating Dinner", "Morning Medication", "Evening Medication", etc. Indeed, precision offers the possibility to create specific services or scenarios, as for instance, reminders for morning and evening medication intake. In addition, taking into account time information might improve the recognition of meta activities.

The consideration of relations between activities is another interesting task. Throughout the day, we perform activities that are often linked. For example, "Washing Dishes" is linked to "Cooking" or "Eating" because the activity "Washing Dishes" is generally performed just before or after "Cooking" and "Eating". Being able to interpret these relations could help systems recognize the current activity, especially when similarities between sensor activation sequences lead to confusions. For example, "Washing Dishes" can be confused with the activity "Cooking" because they are performed in the same room and therefore may activate the same sensors.

Knowing that the previous activity was "Eating", could remove this confusion. The recognition of the activity does not only depend on a local context, but also mainly on a more global context of the activity.

We have seen that LSTM-based models are suitable structures for the human activity recognition problem in smart homes. They are able to consider the time through the timesteps interpretation, but proposed models do not use the temporal evolution along the day. In addition, the In this chapter, we propose to improve the previously shown method (the Liciotti et al. [3]'s bidirectional LSTM) and our previous presented method ELMo embedding based. We have done this by using temporal information or pieces of extra-activity information from a more extended context. We firstly provided a brief recall description of the previously seen two models used for baseline. Then in the next three sections, we detail and investigate each proposed extension against the baseline. In the fifth section, we will make a global comparison of the proposed improvement approaches. Finally, we will conclude a summary detailing the main elements of this contribution.

Methods and Baselines

Methods

Two experiments are conducted to evaluate baseline models and the proposed extensions with the three studied datasets (Aruba, Milan, Cairo). In the first, similar to the experiments in previous chapters, we grouped dataset activities into 11 ADLs, (cf. Chapter 3), in order to make a consistent comparison of the results between datasets. In the second experiment, we used the original datasets' class labels, due to the fact that they are more precise. Moreover, these activity labels could be similar but not realised at the same moment, such as "Breakfast", "Lunch" and "Dinner". This second experiment aims at evaluating models' capacities to recognise a larger number of low-level activities, similar activities, or activities performed at particular moments during the day.

To compare the models, we primarily examine the accuracy and F1-score. We also look at the Balanced Accuracy and weighted F1-score. As a reminder, the first two will guide us on how the models perform on the datasets overall. The second two are studied because the datasets are unbalanced, they will inform us if the models find all activity classes with the same performance or if they focus on the classes with the highest number of occurrences (cf. Chapter 3). 122 6.2. Methods and Baselines

Baselines

For baseline models, we used :

1. the bidirectional LSTM-based model of [3] (see Figure 6.1a). Henceforward, we note this baseline LB for Liciotti-baseline.

2. our proposed model using an ELMo-like pretrained embedding defined in the previous chapter (see Figure 6.1b). We'll use the abreviation EB for ELMo-baseline. The Figure 6.2 presents, Accuracy, Balanced Accuracy, F1-Score and Weighted F1-Score for the three datasets with grouped labels and original labels. We can observe firstly that scores are lower with original dataset labels. This could be explained by the fact that similar activities, e.g. "Breakfast", "Lunch" and "Dinner" must be recognized individually while they are packed into the same label in the "Grouped" activity dataset version. We can notice that the EB model

ELMo

Previous Activities as an Extension of the Context

As successive activities are interrelated, they are poorly modelled by Markov processes. The information about what the person did before seems useful to recognize the current activity. To capture these relations, we propose to add to the current activity sequence, the two previous activity sequences in input, as in Figure 6.3. In the selected datasets, activity sequences are generally separated by sequences labeled as "Other". The latter corresponds either to unknown activities or to transitions between two activities. That's why we chose to add at least two and not one previous activity sequence. We The Figure 6.4 presents comparison metric results of baseline and previous activities models, for both datasets versions. We can observe that this proposed extension (LPA and EPA) decreases the classification performances for both datasets versions. It seems that the knowledge provided by the extended context does not help. The first possible explanation is that sequences become most similar by the addition of previous activities. Another factor could be the sequence length. In our example, the average sequence length increases from ∼ 600 to ∼ 1800 elements.

Yet, a well known problem on long sequences with recurrent neural networks, in part decreased by LSTM, is the vanishing gradient problem [2] that implies that long-term relationships are hardly encoded by recurrent neural networks. The longer the sequence, the harder it is to find time relationships. Moreover, originally, activity sequences had different lengths and activities belonging to the same class had a similar size. For example, the activity "Bed to Toilet" is a very short activity in terms of event numbers compared to "Sleep". It appears that this length information that can be used by the LSTM is destroyed by the addition of previous sequences. Many of our activities are related to a particular time in the day. Thus, it seems valuable to add this time information to recognize daily activities. For example, "Eating" occurs in general three times a day, in the morning ("Breakfast"), in the middle of the day ("Lunch") and in the evening ("Dinning").

We propose to add a second input that is the time value of when the activity sequence occurs, see Figure 6.5. The time value is the hour of the first element in the sequence. We chose to use the hour granularly because using minutes could be too precise as sensor activations don't usually appear at the exact same minute every days. This inclusion seems to allow the model to understand that particular times of the day are more conducive to the occurrence of given activities.

This second input is given to a projection layer, which is concatenated in the result of the bidirectional LSTM output, before the softmax layer. Such a configuration allows the softmax layer to use the activity sequence and the time representation to conclude the task classification.

We note LT for Liciotti + time and ET for ELMo + time. The time information seems to help the model to better recover class (F1-score) and to be more balanced between classes (Balanced Accuracy). Results show a high improvement of classification performance, in particular for the LT model. As for the ET model, it seems to perform better on the Cairo dataset.

The Time of Sensor Event Occurrence

As discussed above, time may be important for activity recognition. Here we propose to extend the previous structure by using a sequence of time values in addition to the sequence of sensor activations as input into the model, see For both models the gap between the F1-scores and Weighted F1-sores, similar for Accuracy and Balanced Accuracy, becomes smaller. This means that despite the unbalanced factor of the dataset both models recognize all activities in a balanced way.

Grouped Activities: Class-by-Class Analysis

In order to deepen our analysis, we studied class-by-class the performance of each model via the F1-score metric. Figure 6.10 is a visual overview of how each model recognized each activity for the three grouped dataset versions, while Table 6.1 contains the detailed F1-score values.

Thanks to these two representations, we can conduct the following analysis. On the Aruba dataset, at first glance of Figure 6.10, all the models seem to be similar to each other during each activity. The LPA and LTS models were not able to find the "Take medicine" class which only contained the "Respirate" activity and only occurred six times in the dataset.

Nevertheless, the LPA and EPA models achieved a very large improvement on the "Enter Home"

and "Leave Home" activities, about 20% compared to the baseline model for the EPA model. It seems that past activity information is important for recognizing these two activities.

The Aruba dataset house contains several entries into and out of the house and it is not uncommon for the resident to exit by one entry and enter by another. Moreover, we notice that the resident sometimes makes several entries and exits in a short period of time. In this case, the model observes, thanks to the addition of the sequences of preceding activities, that one must leave before entering and vice versa. This helps the model to differentiate these two activities.

However, even though the EPA model is close to the other models during the other tests, it loses about 50% of its performance on the "Take medicine" activity.

With the Milan dataset, all the models have similar performance in eight out of ten cases, and the LPA and EPA scores are always under the other models. We noticed that the most difficult activities to recognize were "Eat" and "Bed to toilet", all models performed poorly on these two classes. The model with the best results for the class "Eat" is the EB model. Against all expectations, it was not the models which used time that had the best performances for this activity, but the ET and ETS models which closely compared to Liciotti's-based models.

This could be explained by the fact that the sensors' activation context is more important, since the sensor used to recognize this activity can be used when a resident is traveling due to their location. In other words, the PIR above the dining table may be activated when the resident passes by since the table is on the way between the kitchen and the living room. While the absence of information on time and previous activities seems to give better performances than the EB model, especially for the "Eat" activity, the models using the above-mentioned information give the best performances on the "Bed to toilet" activity. The LTS and ETS models increase the performance significantly for the "Bed to toilet" activity. The ETS model improves 97% over the EB model and the LTS model improves 31.28% over the LB model. In addition, it seems that using information from the previous activities also increases the performance compared to the baseline models, but less so than the time under sequence form.

For the Cairo dataset, two activities seem difficult to recognize, "Bed to Toilet" and "Sleep".

They are indeed very close to each other as these two activities occur during the same time of day (Night). In addition, as the dataset contains the activities of two resident, one resident can go to toilet while the second one is sleeping. The ETS model obtains the best performance for the activity "Sleep", it results in a significant increase in performance over the EB model (23%).

For the "Bed to Toilet" activity, the ETS model is slightly better than the EB model, but the best model is the LT model. In the original Milan dataset some activities, such as "Eve Meds", "Morning meds", "Dining RM Activity", suggest by their denomination that it is necessary to consider temporal information. Indeed, we noticed that the best scores for this dataset were obtained by methods using time. While the baseline models were unable to recognize the "Eve Meds" activity, the LT, LTS, ET and ETS approaches obtained an F1-score higher than 65%. The LTS approach obtained the best score with 81.12%. The "Morning Meds" activity which was previously recognized with a F1-score between 58% and 65% on the baseline models obtained an F1-score between 72% and 84% with the models integrating time.

Moreover, we noticed that the ETS model could improve the F1-score of 64.60% with the EB model to 78.05%. This same model managed to have the highest F1-score of 21.64% on the "Chores" activity. The "Dining Rm Activity" was better recognized by the EB and ETS approaches. The ETS model appears to get the best scores globally.

The Cairo dataset also offers a number of time-dependent activities, "Breakfast", "Dinner", "Lunch", "Night Wandering", etc. Although the EB model already performs well on these activity classes, it seems that the use of time improves the recognition scores of these classes. The ETS and LTS models give the best results for all activities with high F1-score. For the activity "Lunch" the best baseline model is EB with a score of 27.64%, while the LTS approach obtained the overall best score of 91.42% and the ETS got 88.75%, which correspond to an increase of times 3.3 and times 3.2, respectively. We notice that in this dataset the ETS method obtained the best score for 50% of the classes and the second-best score for the remaining 50%.

Summary

In this chapter we have proposed to extend our model, the bidirectional LSTM with ELMo pretrained embedding, and the Liciotti et al. [3]'s model, the bidirectional LSTM with embedding, using time information or previous activities' context. The results presented in this chapter show that the proposed approaches lead to a viable solution that significantly improves the task of recognizing ADLs in a smart home scenario. After a full comparison of the two basic models, we find that the extended models, applied to three CASAS datasets, clearly perform better.

This improvement is particularly strong for the ELMo model combined with sequential time information.

We have mainly explored two enhancement strategies, one that uses a larger contextual sequence as input and one that uses temporal information. For the temporal approach we first used the time of occurrence of the activity sequence, i.e. when the activity occurs, and then a sequence of temporal events, i.e a sequence containing the time of occurrence of each event in the sensor activation sequence. We have observed that the contribution of time allows a better classification rate, but also a better equilibrium in terms of number of recognized classes, which is visible thanks to the Balanced Accuracy and F1-score metrics.

However, while the addition of temporal information seems to improve the results, the use of a larger context does not. On the contrary, the use of more contextual information seems to decrease the classification performance. We suggest that this loss of performance occurs because sequences become more similar to each other with the addition of previous activities.

Another hypothesis could be the sequence length, as the input models can become too large enough for the LSTM structure. The average sequence length for the Cairo dataset, for example, increases from ∼ 600 to ∼ 1800 elements, which suggests to us the possibility of a potential vanishing gradient problem [2]. In a future work, it could be interesting to investigate the proposed approach by using Transformer Structures [6] in order to avoid such a limitation.

Moreover, originally, activity sequences had different lengths and activities belonging to the same class had a similar length. By adding previous activities, we destroy this potentially 6.7. Summary important knowledge that seems useful for the LSTM structure. Nevertheless, we can observe exceptions in the recognition of activities such as "Leaving Home", "Enter Home" and "Wash Dishes" on the Aruba dataset. The use of a wider context allows recognizing an activity that was not recognized before, "Wash Dishes", but also improves the F1-score by about 20% for the activities "Leaving Home" and "Enter Home". It appears that previous activities' knowledge seems to help in some cases.

In conclusion, we have seen that the approach combining ELMo and time sequence obtains the best performance results in general on every dataset. This revelation seems to confirm the effectiveness of using an ELMo pretrained embedding. We achieved the highest improvement with the Cairo dataset, which is the most challenging dataset in this study, as the activities of two residents were monitored. The Licotti's model significantly improved in all test cases by using the time sequence and could even challenge more ELMo based models.

However, trying to imagine and create the best possible approach to perform the complex task of human activity recognition in a smart home environment is not enough. It is necessary to be able to apply these methods in a real-world setting. This task is very difficult to implement since these models and methods need a large amount of labeled training data from the real final system. This is difficult to obtain, as it is hardly possible to ask an end user to label this data over a long period of time. We will see in the next chapter a proposal to consider the application of a human activity recognition method in real homes.

GENERATING DATA FOR HUMAN ACTIVITY RECOGNITION IN REAL-WORLD APPLICATIONS

Introduction

In the previous chapters, we have seen human activity recognition approaches. We have proposed some improvements that constitute steps towards the deployment of real-world applications.

Nevertheless, there is a gap between the application of HAR algorithms on the existing datasets and their deployment on real-world data.

Over the past few decades, the number of real smart home projects and test beds has been growing around the world. These multiple real-world environments produce datasets over periods ranging from days, weeks, to months with the help of volunteer residents. Some of these datasets are made public and allow evaluation of ADLs recognition algorithms and others. However, these datasets are not sufficient to cover the distribution of possibilities in terms of activities, lifestyles or house configurations. Indeed, they are limited in the number of activities, occupants they take into account, as well as in the types of sensors and residents' category. Therefore, algorithms trained on data from one particular environment will be difficult or impossible to reuse in others as they are adapted to a particular configuration of house, inhabitants and life habits.

Collecting new datasets to increase the number of examples is a very costly task in terms of equipment and difficult to accomplish. Even if fully equipped and ready-to-use environments, such as Amiqual4Home [7], can be used on an ad hoc basis, recruiting volunteers and collecting data is very time-consuming [4]. It is necessary to ask residents to live in the environment and to label their activities over several weeks or months, i.e., to track the start and end times of activities and the label of the activity. This involves a significant investment on the part of volunteers and presents a high risk of error in labeling. Moreover, there are very few such environments accessible, again limiting the diversity of environments. To overcome this, new data on households can be collected from customers by companies offering smart home solutions.

However, this solution poses problems in terms of sufficient sensor equipment, anonymization, privacy, security and regulation. Retraining a deep learning model on these new data should require to collect and annotate a large amount of data from the test case, which can be costly in terms of means and time.

The use of transfer learning techniques seems to be a solution to reducing the required amount of data in the test case. We have seen in the previous chapters that the use of pretrained embeddings could allow this. Nevertheless, it is necessary to have a large amount of data for pre-training but also labeled data of the target environment to adjust the algorithms so that they are perfectly adapted. Assuming we have access to a large amount of data for the pre-training, from the point of view of a commercial real-world application, it is not possible to ask end-users or customers to label their activities themselves for several days, weeks or months.

A digital copy of a target smart home, that having the same virtual sensors, could be a way to generate data to refine the algorithms for the physical target smart home.

The concept of Digital Twin consists in creating a complete virtual representation of a physical object, process or system. To address the challenges of data scarcity and, in particular, the gap between training data and use case data, data generation through the concept of Digital Twin seems to be a viable solution. In this chapter, we propose to bridge the gap between test and training data using a smart home simulator based on the Digital Twin concept. In the next section, we briefly explain the Digital Twin concept and its potential application for the ADLs recognition problem. In the third section, we will present the Virtual Home simulator and the proposed extension, Virtual Smart Home simulator, for the generation of Digital Twin smart homes. Then we will detail our process to generate valuable data from a Digital Twin smart home and how we evaluated them. Finally, we will evaluate an algorithm of ADLs recognition trained through this Digital Twin smart homes concept.

Digital Twin Smart Homes for Human Activity Recognition:

Background and Concept

The Digital Twin concept is a particular simulation method. The objective of a Digital Twin is to be a complete virtual representation of a physical object, process or system. The digital representation provides both the elements and the dynamics of the operation of a device throughout its life cycle. Digital Twin models are gaining more and more interest for their potential and strong impact in application fields such as manufacturing, aerospace, healthcare, and medicine [2]. Nasaruddin et al. [8] defines the Digital Twin of a building as the interaction between the interior environment of a real building and a digital but realistic virtual representation model of the building environment. The latter enables real-time monitoring and data acquisition. For instance, a Digital Twin of a building have been used in [6] to find the strategic locations of sensors to collect data efficiently.

Real House

Digital Twin House In the context of ADLs recognition, a Digital Twin could be a digital copy of a target smart house equipped with the same virtual sensors (see Figure 7.1). In this digital environment, one or more avatars could perform ADLs by modeling complex resident behaviors. Thus, the virtual sensor logs could then be used to refine the final algorithms before deployment in the target house.

Besides, the Digital Twin concept could also be used to generate data from a large amount of houses and household configurations and residents' habits, with the possibility of accelerating the simulation, of automatic labelling, and without the cost of the sensors. For example, common data from a large number of Digital Twins homes could be used to generate a richer dataset.

This large dataset could then be used for algorithm pre-training and finally refined on a target Digital Twin. Furthermore, a Digital Twin home may be used to evaluate the correct positioning and choice of sensors to recognize a predefined list of activities.

Virtual Smart Home: a Digital Twin Simulator

In order to develop this concept of Digital Twin for the smart home it is necessary to have a simulation environment. As we have seen in Section 2.8.2, the literature provides us with a number of smart home simulators with their advantages but also their disadvantages. The main problem of these simulation environments is their complexity of implementation but also their maintenance. These simulation environments often require 3D modeling and programming skills and the projects are often abandoned or not maintained. This makes it difficult to select a viable simulation environment for a Digital Twin approach.

However, in order to create a Digital Twin of a smart home, we have developed the Virtual Smart Home simulator. It is an extension of the Virtual Home simulator developed by Puig et al. [9]. The choice to use and extend this simulator was motivated by the fact that: 1) it is fully open source, therefore editable and extensible; 2) under development and has an active community; 3) it is designed and implemented to simulate human activities in a house.

Virtual Home

Virtual Home is a multiagent platform to simulate activities in a household. Agents are represented as humanoid avatars, which can interact with the environment through high-level instructions. These interactions can be chained under scripts to perform complex tasks and activities

Virtual Smart Home

In summary, Virtual Home, enables the creation of scenarios where 3D avatars carry out activities of daily life and can be filmed via a virtual camera. Moreover, a database of several scripted activities is provided and can be applied on the avatars. The Virtual Home environment allows us to add and create new complete or piece of house via the Unity3D editor environment. Therefore, it allows us to create the digital reproduction of a home. Unfortunately, several features are missing to reproduce a smart house and generate sensor data. To be able to produce scenarios and collect sensor data, we have implemented the following new features into Virtual Home:

Virtual Sensors

We have implemented the possibility to log sensors and the five following virtual sensors: (1) opening/closing sensors; (2) pressure sensors; (3) lights (4); power consumption and (5) a floor presence sensor.

All sensors except the sensitive floor sensor were simulated using the environment graph of the scene. This graph lists all the objects in the scene with an ID and associates them with the states corresponding to their nature (e.g. closed/open, on/off). Power consumption sensor translates if an appliance is turned "on" or "off".

The floor presence sensor is simulated using a downwards raycast from the avatar that checks the collision with the floor. As the floor of the digital house is divided by rooms, the intersection of the raycast with the floor rooms allows us to identify the room in which the avatar is located.

The floor presence sensor does not directly give which avatar is in which room, it just gives the information of presence. In other words, it outputs if someone is or not in a room.

Interactive Objects and Avatar Actions

Some animations and interactions are not possible. Virtual Home proposes a large number of objects that can be included in the apartments, but many are passive and can not be interacted with by the agent Thus, we added new interactive objects to increase the number of possible interactions. We have added the possibility to open and close a drawer, turn on and off faucets on some object, open and close door of cabinets, etc. Moreover, the avatar can't perform some actions or activities that would be interesting for some ADLs. Therefore, we have implemented new avatar actions and animation such as "Lying" or "Eat".

Simulation Acceleration

Virtual Home not provide the possibility to change the speed of time flow. Thus, we have added the possibility to accelerate the time of the simulation. However, it is not possible to speed up time faster than the Unity game engine. Indeed, as Unity3D is a real-time render engine, the time factor cannot exceed the time required by the 3D engine that generates the rendering of the 3D scene. It takes a minimal amount of time for this engine to calculate the movements, interactions, and changes in the scene in addition to the visual rendering. We were able to accelerate the simulation time with a factor 4.

Configuration User Interface

In order to configure the simulation and start the data generation, we designed a graphical user interface (see Annex E). This interface is build to follow a three-step workflow. A first window allows the creation of an experiment configuration file specifying: (1) the simulation house; and

(2) the set of sensors that we wish to monitor and log. A second window allows the creation of a scenario configuration file:

The Smart Home Digital Twin

We conducted our experiments via the Experiment'Haal smart apartment, the IMT's living laboratory for health and autonomy [1], which aims to develop and host experimental assistive devices for user testing. This smart apartment consists of a main living space, a kitchen, a sleeping area and a bathroom (see Figure 7.4a). It allows simulating a domestic environment in order to validate the technological solutions proposed by the end-users in their daily life context, before they can be deployed on a larger scale.

We reproduced the apartment into the Virtual Smart Home Simulator to create its Digital Twin, (see Figure 7.4). All the elements of the apartment were measured and placed to be as close as possible to reality. However, due to the lack of a 3D model for some objects, we had to replace them with the most suitable model possible. We did not notice any difference or particular impact due to these adjustments at the time of the simulations. The majority of sensors' apartments were reproduced to generate activities logs into the Digital Twin. Only motion, temperature and humidity sensors were not copied. The motion sensors were avoided as the apartment does not have walls, which allow a motion sensor to be we can see on the contrary more difference between results of synthetic and real data for the two other subject. The performance are better for the subject "S7" and "S3" with synthetic data according to the F1-score and the Balanced Accuracy.

One to One

In this experiment, we train the model with the synthetic data of a subject and test the trained algorithm with the real data of the same subject. The result should allow us to have a first evaluation on the usability of the synthetic data. The results are presented in Table 7.4.

At first sight, the synthetic data generated for each subject allows to train and recognize activity sequences for each real dataset (one subject step). Subjects "S9" and "S7" obtain good performances in terms of Accuracy, Balanced Accuracy, and F1-score. We notice that subject "S7" presents the best performances. For these two subjects the synthetic data seems to be realistic enough to allow a recognition of more than 70% in terms of accuracy for both subjects.

On the contrary, the "S3" subject presents the worst performances. It seems that the synthetic data generated for this subject, are not able to allow a recognition of the activities of this subject.

We notice for the subject S9 that the activity "Bathe" is not recognized, while for the subject "S7" it is recognized at 100%. The subject "S3" presents 4 classes of activity out of 10 not recognized. From these results we can deduce that it is possible to use synthetic data to train an algorithm and recognize activities from real data. However, it seems that using only activity data from a single subject is not always sufficient. It is therefore necessary to bring more data and variability in the training dataset.

Many to One

In this experiment, the algorithm is trained on all synthetic data. Once trained, it is tested independently with the real data of each subject. The purpose of this experiment is to evaluate to be able to interpret the logic of the possible sequences of actions. Moreover, in the case of a resident living alone, this knowledge should allow to better contextualize and interpret similar activities in terms of sensor sequence as involving the same sensors.

In order to better interpret the sequences of sensor activations, their relations and the logic of their sequencing, we once again relied on the language paradigm and NLP techniques, in particular on language modeling. We trained and used two advanced forms of embeddings based on Word2Vec and ELMo. By confronting these methods, we were able to observe and demonstrate that the use of a contextualized embedding such as ELMo increases the classification performance of a model based on a bidirectional LSTM. We have noted that taking into account the context and modeling the probability of occurrence of a sensor activation, has particularly improved the performance of the classifier on a dataset containing the activities of several people.

Another important aspect of human activity to consider is time. Human life is very often correlated with the wall-clock-time or periods of the day.

To account for the time of the day, we added an encoding and an interpretation of time (cf.

Chapter 6). We could observe that the addition of the time in the form of sequences allowed to increase the performances of the model coupling the contextual embedding ELMo and a classifier based on a bidirectional LSTM. We also noticed that this approach drastically increased the performance of the Liciotti et al. [3] model that was used as a baseline, especially on the Cairo multi-resident dataset.

In addition to often having a temporal aspect, ADLs are often correlated with each other.

Some activities are expected to appear before or after others. For instance, the activity of cooking calls for the activity of eating. This is why we tested the extension of the context of an activity sequence by adding the previous activities. However, the approach using the previous activities did not lead to any improvement. We even noticed a performance degradation except on some particular activities depending on the datasets. We suppose that the degradation comes from the size of the input sequences, which have become potentially too long in number of events for the LSTM based model.

Can we transfer the knowledge gained from one house to another house?

Houses have very different typologies, resulting in even more sensor configurations. The advantages of deep learning-based methods are their adaptability and generalizability. The increase in the number of datasets and thus access to rich data with multiple activity labels, as well as the increase in computational power, make it possible to design and generate more complex models today. But while a deep learning model can interpolate a model on a specific dataset, its performance drops when tested on another dataset. Thus, due to the variability of houses and configurations in each dataset, interpolation shows it limit, and we would need a mechanism Part, Chapter 8 -Conclusion such as transfer learning to enable better generalisation of models.

The NLP domain uses in particular pre-trained embedding models for generalizing on a set of data and transferring its knowledge to a new context. We were inspired by this method to transfer the knowledge acquired on the Aruba dataset to reuse it on a second dataset, Cairo, through the ELMo embedding. We trained a new classifier on the Cairo dataset using the ELMo embedding of Aruba. We observed that the knowledge and representation of the embedding from Aruba allowed us to obtain very similar results to training a classifier with an embedding trained on the destination dataset. The use of pre-trained embedding allows to transfer knowledge and representation from a house A to a house B in the same way as in the NLP domain for text. It would therefore be possible to have smaller destination datasets but to take advantage of the generalization from larger datasets via pre-trained embedding.

How could we apply activity recognition methods to a actual real house?

The use and application of activity recognition approaches in a real-world, industrial and commercial context remains a major challenge due to the lack of labeled data from the destination environment. Although activity recognition models can be trained on any dataset, it is still impossible to obtain a large amount of labeled data from a client. It is not realistic to collect several months of data and ask a client to label it.

One idea, increasingly common in manufacturing or industry, and now in the medical field, is the creation and use of digital twins. The intuition is to be able to apply, study, observe on a digital double certain changes or effects. This intuition extends to real time changes that can come from the physical or virtual world. Inspired by this idea of digital replica, we tried to answer the problem of transferability of activity recognition methods to the real world.

We created a digital copy (Digital Twin like), containing virtual sensors, of a real intelligent apartment that we used to generate a synthetic data set. For this we used an avatar, simulating a resident, to perform activities of daily living in this simulation environment. The data from the virtual sensor activations recorded during the execution of the avatar's activities were used to train an activity recognition algorithm. We showed that the generated data was similar to the real data of the apartment. At the same time, we showed that the algorithm trained on synthetic data obtains high classification performances ( 80%) on the real apartment data. This work has shown that it is possible to train an algorithm from synthetic data and label it without acquiring and labeling real data.

Limitations and Perspectives

Limitations and Perspectives

In this thesis, we have sought to improve the performance of recognition algorithms through the language paradigm and techniques from the NLP domain. We have shown that contextualization and the consideration of time allow us to push back certain limits. We have seen that it was possible to transfer knowledge from one house to another, but also to transfer knowledge from a virtual house to a real one. However, limitations remain in the proposed approach and further work could be considered to address them. Furthermore, based on this preliminary work, new perspectives can be explored.

New Datasets Comparison

A limitation of this work is the evaluation of the proposed method on only three CASAS datasets.

An interesting perspective would be to apply the method in a first step on a larger panel of datasets provided by the CASAS benchmark [3]. We could think of the series of datasets "HH" containing thirty similar apartments. Then in a second step, on other datasets available in the literature such as, [5], [13], [1], etc. It could be observed the generability of the method proposed in this thesis, as well as potential new limitations.

No Pre-Segmented Activity Recognition and Activity Sequences Auto-Segmentation

This work is based on the classification of EW, i.e. of pre-segmented activity sequences. The model thus has as input all the sensor activations corresponding to the activity to decide. It knows the beginning, the end and the length (in terms of number of events to estimate the duration). However, in order to create more dynamic applications and services, it is necessary to recognize the ADLs during and as the residents perform them.

A first perspective of work would be to transpose and evaluate the proposed method in a "online" activity recognition context. A first approach would be to train the method as realized here, on EWs. Then to use TWs for the final recognition application. A second possibility would be to train the embedding on EW but to train the classifier on TW. The classifier could then exploit the knowledge learned on the dataset of activity sequences to try to recognize each TW.

Concerning the ELMo embedding training, it is necessary to have labeled dataset. This embedding is trained with independent sentences and thus, need segmentation in EW. This cutting into EW avoid the understanding of the relations between activity sequences. However, it could be considered not to use EW for embedding training and to exploit an approach similar to GPT [14]. GPT follows the basic principle of language modeling by predicting the next word from the previous words. But unlike ELMo or BERT, the preceding words are potentially words belonging to the previous sentence. This allows the interpretation of a more global context, but

Dealing with Unknown Sensors Values

In this work, we used an encoding of sensor events in the form of a categorical variable sequence (concatenation of the sensor ID and its value). This encoding method is limited by a certain size of "vocabulary" or, in other words, the possible sensor activation from the training data. It is currently impossible to obtain a representation of the values of sensors that have never been observed in training. For example, a temperature sensor that never exceeded 30 degrees in the training dataset but has a higher temperature in real-world model use will not be represented.

Even though we have prevented this case by using, as it is done in the NLP domain, a special word ,"<UNK>" to replace the unknown values, too many unknown values in the data would lead to a degradation of the classification performance. The simplest solution is to obtain the largest possible volume of data to cover the maximum possible values, but getting a large amount of data with a variety of examples is not always easy.

The NLP field has already considered and proposed solutions that could be exploited. We could exploit a character-based representation [16], using the version of the ELMo model based on word embedding from a CNN. But also, methods such as Byte Pair Encoding (BPE) [10] or WordPiece [15] could be considered to split words representing a sensor activation, into character compositions or subwords. The use of WordPiece seems a promising option as we could consider a sensor activation in two words, "T001" and "30", which will bring several advantages. Firstly, using the example above, only the value of 31 degrees would be replaced by the special word "<UNK>" and not the entire sensor activation. This leaves the model with the ability to interpret which known sensor activates, regardless of its value.

Secondly, as the training of an embedding based on language modeling seeks to predict the next token (word), in the case of using subwords, the tokens would represent subwords. The model would then be trained to predict the subwords which would add the possibility for the model to learn that, a value of a temperature sensor is necessarily scalar and can only evolve between -40 and +50 degrees for example. This would add an additional level of knowledge, potentially useful for the detection of anomalies or impossible values.

Transfer Learning, Universal Smart Home Language and Translation

We have shown that the use of a pre-trained embedding allows to transfer knowledge and representation of sensors from one house to another. However, in our experiment, we have realized this transfer between two datasets belonging to the CASAS benchmark. The two datasets share a very similar vocabulary as the naming of the sensors and their values follows the same structure. Thus, a word representing the activation of a sensor present in dataset "A" has a very high Part, Chapter 8 -Conclusion it would be to predict whether two activities follow each other. This would make it possible to exploit the logic of the succession of activities to better recognize them.

Digital Twin and Synthetic Data Generation

In order to realize the digital twin of a connected apartment, we augmented the Virtual Home A possible evolution path for the creation of a new version of VirtualSmartHome could be the exploitation of video games environments. Indeed, video games are a multi-billion dollar industry, where developers are striving to build very realistic worlds. Always associated with entertainment, the impressive amount of detail and diversity of objects, environments, appearances, human body movements and character traits, as well as the power to control the virtual world make it an excellent source of training data.

This idea has already been considered as a data source for semantic segmentation in autonomous vehicle applications [7], [8] but also ADLs video generation for training robots [9].

This same idea could be considered for the generation of sensor data, through life simulation sets, and thus take advantage of the richness of these environments. Moreover, today some video sets have APIs to allow the creation of personal extensions.

"Progress in knowledge is essentially the transformation of previous knowledge."

-Karl Popper, On the Sources of Knowledge and of Ignorance

1. 1

 1 Representation of activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Categorization of human activity . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Human Activity Recognition approaches . . . . . . . . . . . . . . . . . . . . . . . 1.4 The richness of the sensors versus a user's perceived privacy. . . . . . . . . . . . . 1.5 Activities to sensors logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Supervised and Unsupervised approaches . . . . . . . . . . . . . . . . . . . . . . 2.1 Representation of data flow of binary sensors (each bar is a state change of a sensor) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Time series completion of binary sensors example . . . . . . . . . . . . . . . . . . 2.3 Example of segmentation methods on the univariate sensor representation. In this example, the Sensor Event Windows have a size of 6 events . . . . . . . . . . . . 3.1 Proposed model architecture and global workflow . . . . . . . . . . . . . . . . . . 3.2 Liciotti's baseline models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Extract of a CASAS dataset log . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Floor maps of Aruba, Milan and Cairo Datasets. . . . . . . . . . . . . . . . . . . 3.5 Aruba's activities labels and occurrences . . . . . . . . . . . . . . . . . . . . . . . 3.6 Milan's activities labels and occurrences . . . . . . . . . . . . . . . . . . . . . . . 3.7 Cairo's activities labels and occurrences . . . . . . . . . . . . . . . . . . . . . . . 3.8 Global occurrences and activities overlapping . . . . . . . . . . . . . . . . . . . . 3.9 K-fold cross validation principle. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.10 Classification Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Fully Convolutional Network (FCN) for segmentation . . . . . . . . . . . . . . . 4.2 Fully Convolutional Network (FCN) for Time Series Classification (TSC) . . . . 4.3 Fully Convolutional Network (FCN) for Activity of Daily Lining (ADL) recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Fully Convolutional Network (FCN) and SEW framework . . . . . . . . . . . . . 4.5 Exemple of a Gradient-weighted Class Activation Map (Grad-CAM) of an activity sequence. Red means strong focus. . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Example of zero padding, symmetric padding and circular padding . . . . . . . .

Figure 1 . 1 -

 11 Figure 1.1 -Representation of activities

Figure 1 . 2 -

 12 Figure 1.2 -Categorization of human activity

Figure 1 . 5 -

 15 Figure 1.5 -Activities to sensors logs

Figure 1 . 6 -

 16 Figure 1.6 -Supervised and Unsupervised approaches

Figure 2 . 1 -Figure 2 . 1

 2121 Figure 2.1 -Representation of data flow of binary sensors (each bar is a state change of a sensor)

3 Figure 2 . 2 -

 322 Figure 2.2 -Time series completion of binary sensors example

Figure 3 . 1 -

 31 Figure 3.1 -Proposed model architecture and global workflow

3. 3 .

 3 The Baseline Models network-based classification algorithm. The embedding is a projection layer that projects, in a d-dimensional space, each sensor activation. The role of embedding is to extract meaningful features that encode each sensor activation to help the classifier to perform its task. Thus, the classifier must extract from the embedding output, a list of meaningful vectors, patterns and features to recognize them. Via these patterns, it attributes the correct activity class label. In this work, three types of classifiers have been studied: a CNN-based classifier and two LSTM-based classifiers. The CNN-based classifier is motivated by the strong ability of CNN to recognize patterns, while LSTM-based classifiers are inspired by the Liciotti et al. [6]'s work.

Figure 3 . 2 -Figure 3 . 3 - 58 3Figure 3 . 4 -

 32335834 Figure 3.2 -Liciotti's baseline models

, 3 . 6 , 3 . 7 .

 3637 This imbalance is explained by the fact that these datasets come from real homes with real inhabitants, and the activities are related to the lifestyle of these inhabitants. Moreover, as it can be seen on Figure3.8 the three datasets doesn't cover Part, Chapter 3 -Background, Approach and Experimental Methodology

Figure 3 . 5 -Figure 3 . 6 - 60 3Figure 3 . 7 -Figure 3 . 8 -

 3536603738 Figure 3.5 -Aruba's activities labels and occurrences

Figure 3 . 9 -

 39 Figure 3.9 -K-fold cross validation principle.

  .10. First, one fold is reserved for the final test (33% of the dataset). Then the two other folds are concatenated, shuffled and 20 % is kept for the validation step. These 20% validation data are not considered for training the model. The parameters' model are trained with the 80% data left thanks to the sparse categorical cross-entropy loss. At the end of each training epoch, when all training data was seen by the model, the trained model is validated on the 20% reserved data. This step allows us to observe the model performance thanks to the validation loss and to check if it is starting to overfit (the model memorizes the training data). During the training, we save and keep only the best model. To select the best model at each epoch, we check if the current model has a better accuracy than the previous one (the one at the previous epoch). If it is the case, the model is saved, otherwise a new epoch of training begins.

Figure 3 .

 3 Figure 3.10 -Classification Stage

  using four features, True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). The F-score, also called the F1-score, is a measure of the precision of a model on a dataset. The F-score is a way of combining the precision and recall of the model, and it is Part, Chapter 3 -Background, Approach and Experimental Methodology defined as the harmonic mean of the precision and recall of the model. This metric is often the most relevant one to evaluate a model. These measures are fine for balanced datasets, but real house datasets for recognition of ADLs are generally unbalanced. In other words, some activities have more examples than others. In an unbalanced data set, a minority class is more difficult to predict because there are few examples of that class, by definition. It is more difficult for a model to learn the characteristics of minority

  provide a standard baseline to exploit deep neural networks for endto-end TSC without any crafting in feature engineering or data pre-processing. They proposed in their work an adaptation of two models from the field of computer vision, the Residual Networks (ResNet) and the Fully Convolutional Network (FCN). They showed that through pure end-toend training of the raw time series data, both models achieve the best performance compared to other state-of-the-art models. In particular, the FCN wins 18 out of 44 problems and is in the top 5 of the other 26. Two years later Fawaz et al.[3] determined via a comparative study of state-of-the-art TSC algorithms with more datasets (96 datasets) that FCN could also achieve good results on many of the datasets they used. These works show that the FCN has great capabilities for feature extraction and classification.However, even if FCN has great capabilities for feature extraction, works by Tahir et al.[16] and Yan et al.[21], have shown the importance of good feature representation for HAR Part, Chapter 4 -The Time Series Classification Approach applications. Designing these features is a tedious task. Through deep learning algorithms and to avoid manual features representation, researchers in the NLP field have devised many useful algorithms and methods such as embeddings to represent word meanings and model language to increase the performance of deep learning classifiers. This chapter presents a deep learning methodology for recognizing ADLs inspired by the NLP and TSC domains. This approach combines the projection of the input in an embedding with the FCN classifier algorithm. The embedding layer extracts a level of features and representations from the sensors, while the FCN layer performs pattern extraction with another level of extracted features and performs the final classification. The objective of this approach is to address the challenge of pattern recognition and classification generated by the classification of ADLs.

Figure 4 . 1 -

 41 Figure 4.1 -Fully Convolutional Network (FCN) for segmentation

Figure 4 . 1 .

 41 Figure 4.1. The FCN is a convolutional network that does not contain any local pooling layers or fully connected layers but, uses a convolutional layer as the final layer. Due to its performance on feature extraction, Wang et al. [19] transferred the FCN to TSC problems.

Figure 4 . 2 -

 42 Figure 4.2 -Fully Convolutional Network (FCN) for Time Series Classification (TSC)

Figure 4 . 3 -

 43 Figure 4.3 -Fully Convolutional Network (FCN) for Activity of Daily Lining (ADL) recognition

4. 5 .

 5 Temporal Dependencies Evaluationthe form of events, re-sampling is not possible because the length of the sequences of activities is information that may be important.By using the Gradient-weighted Class Activation Map (Grad-CAM) method proposed by Selvaraju et al.[14] we can visualize what the FCN focuses on to determine the label of the sequence. Grad-CAM is used for the deciphering of convolutional neural networks and consists, in the context of an image, of searching for which parts of the image led the network to its final decision. It produces heatmaps representing the activation classes on the images received as input. An activation class is associated with a specific output class. These classes will indicate the importance of each pixel in relation to the class concerned by increasing or decreasing the intensity of the pixel. Here, we have applied this method to the FCN for a sequence problem. Padding (a) Zero padding visualization: full sequence Padding Sensors' Activations (b) Zero padding visualization: zoom on the beginning of the sequence

Figure 4 . 5 -

 45 Figure 4.5 -Exemple of a Gradient-weighted Class Activation Map (Grad-CAM) of an activity sequence. Red means strong focus.

Figure 4 .

 4 Figure 4.5 represents the Grad-CAM of an activity sequence and a zoom in on the beginning of the sequence with the area containing the sensor activations. Each colored column corresponds

Figure 4 . 6 -

 46 Figure 4.6 -Example of zero padding, symmetric padding and circular padding

Figure 4 .

 4 Figure 4.7a, we can see the repetition of the pattern and thus the sensor activations repetition.

4. 5 .

 5 Temporal Dependencies Evaluation (a) Circular padding visualization: full sequence (b) Circular padding visualization: zoom on the beginning of the sequence

Figure 4 . 7 -

 47 Figure 4.7 -Exemple of a Gradient-weighted Class Activation Map (Grad-CAM) of an activity sequence with a circular padding. Red means strong focus.

Figure 5 . 1 -

 51 Figure 5.1 -Word2Vec methods.

Figure 5 . 2 - 1 Part, Chapter 5 -Figure 5 . 4 -

 521554 Figure 5.2 -Windowing example with a window of size 5

  Figure 5.5 -Room cluster

5 -

 5 Figure 5.6 -Room cluster

Figure 5 . 7 -

 57 Figure 5.7 -Sensor2Vec activity sequence embedding model

Figure 5 . 8 -

 58 Figure 5.8 -Activity sequences of Aruba embedded by Word2Vec

Figure 5 . 9 -

 59 Figure 5.9 -Activity sequences of Milan embedded by Word2Vec

Figure 5 . 10 -

 510 Figure 5.10 -Activity sequences of Cairo embedded by Word2Vec

3 )Figure 5 .

 35 Figure 5.11 -ELMo architecture

Figure 5 . 12 -

 512 Figure 5.12 -ELMo sensor activation architecture

, 5 .

 5 15 and 5.16 show the activity sequence embedding via ELMo of the Aruba, Milan and Cairo datasets respectively.

Figure 5 . 13 -

 513 Figure 5.13 -ELMo activity sequence embedding model

Figure 5 . 14 -Figure 5 . 16 - 5 . 4

 51451654 Figure 5.14 -Activity sequences of Aruba embedded by ELMo

Figure 5 . 17 -

 517 Figure 5.17 -ELMo outputs.

  ploited the same previous model structure, with the exception that we use the ELMo embedding of Aruba for the training of classifiers. In this model, the Aruba ELMo embedding extract and encode the activity sequence of the Cairo or Milan datasets. Then the output representation of the embedding is used by the classifier. The classifier extract patterns and features to perform the final task. During training, the weights of the ELMo embedding were frozen, and only the bidirectional LSTM and the softmax were trained to classify the activities of the second dataset.

Figure 5 . 18 -

 518 Figure 5.18 -ELMo output vizualisation

Figure 5 . 19 -

 519 Figure 5.19 -ELMo output heat-map of two random Cairo's "Eat" activity sequences

Figure 5 .

 5 Figure 5.20 -"Eat" and "Sleep" heat-maps comparison

Part, Chapter 6 -

 6 Taking Advantage of Dependence on Previous Activities and Time of the Daycontext for the LSTM models consists of the current sequence and part of the training batch, but sequences in the batches are not necessarily correlated, as it is preferable to shuffle the sequences to help models generalization. Nevertheless, few deep learning works have attempted to exploit time or previous activities' information,[5, 4, 1]. For example, Tan et al.[5] have encoded activities sequences into images and used a pixels color line to encode the previous activity label, in order to help the model to recognize the ongoing activity. Then they used a CNN-based architecture for the classification and not a LSTM-based one. In another example, Ramos et al.[4] have encoded a time feature evolution under sine and cosine values and concatenated this feature with the input sensor vector, to allow a LSTM model to use the time of the current activity window. Devanne et al.[1] uses two layers of LSTM to recognise activities of daily living in videos, by concatenating the previous label to the input features.

6 -

 6 Figure 6.1 -Baseline models

  performs better than the LB model on all datasets, independently of the dataset version. In addition, the EB model have the best improvement, compared to the LB model, on the Cairo dataset for both datasets versions. Nevertheless, for the grouped Milan dataset version, the LB model seems to perform slightly better than the EB model in terms of Balanced Accuracy and F1-score.

Figure 6 . 2 - 6 . 3 .

 6263 Figure 6.2 -Scores for baseline models on Aruba, Milan and Cairo datasets

Figure 6 . 3 -

 63 Figure 6.3 -Architecture of ELMo with previous activities

Figure 6 . 5 -

 65 Figure 6.5 -Activity sequence and time input

Figure 6 . 6 -

 66 Figure 6.6 -Metrics comparison between baseline and time extension models on Aruba, Milan and Cairo datasets

Figure 6 . 7 .

 67 Each time value corresponds to the time at which the sensor event occurred in the sensor sequence. This temporal representation should allow the model to understand how time changes over the course of the activity, which can help it encode the fact that certain activities occur at a particular time of day and have a certain duration. Although duration can be interpreted by the length of the sequence, i.e., the number of elements in the sequence, this temporal representation should help the model to understand that some activities only take place during a specific time of day. We note LTS for Liciotti + time sequence and ETS for ELMo + time sequence.

Figure 6 . 8 -

 68 Figure 6.8 -Metrics comparison between baseline and time sequence models on Aruba, Milan and Cairo datasets

Figure 6 . 10 -

 610 Figure 6.10 -Algorithms performances for each grouped activity on Aruba, Milan and Cairo

Figure 6 . 12 -

 612 Figure 6.12 -Algorithms performances for each activity of origin on Aruba, Milan and Cairo

Figure 7 . 1 -

 71 Figure 7.1 -Digital Twin of Smart Home Principe

  with multiple agents. Moreover, it includes a Knowledge Base, providing instructions to perform a large set of activities. It also can render videos of human activities, or train agents to perform complex tasks. It allows generating a large-scale video dataset of complex activities that are rich and diverse by simply recording the agent executing programs. Recordings provide dense ground-truth information, e.g. RGB, pose, segmentation, etc. (see Figure7.2).

Figure 7 . 2 -

 72 Figure 7.2 -Virtual Home cameras types

Figure 7 . 3 -

 73 Figure 7.3 -Apartment and avatar example

( 1 )

 1 the start date of the experiment log; (2) the acceleration factor of the simulation; and (3) the different activities to play and their duration. The last window interface allows checking the configuration files, and to start the simulation.

7. 4 .

 4 The Smart Home Digital Twin

Figure 7 . 4 -

 74 Figure 7.4 -The real apartment and the Digital Twin

7. 5 .Figure 7 . 7 -

 577 Figure 7.7 -Number of sensor activations per scenarios

Figure 7 . 8 -

 78 Figure 7.8 -Number of unique sensor per scenarios

Part, Chapter 8 -

 8 Conclusionpotentially allows finding breaks between sentences thanks to context changes. It could be the same for activity sequences.

[ 6 ]

 6 simulation environment to create VirtualSmartHome. Although this simulator allowed us to set up a proof of concept, there remains a main limitation. Indeed, Virtual Home is built on top of the Unity3D set engine and does not offer the possibility to accelerate the simulation environment. Unity3D is used as a real time rendering engine. It is therefore not suitable for such an operation. Moreover, the version of VirtualSmartHome remains limited in terms of actions, interactions and behavior modeling.

Figure B. 2 -

 2 Figure B.2 -Milan's cluster colorized by sensors' localisation

Table 2 .

 2

								1 -Summary of segmentation methods
	Segmentation type	Usable for Real Time	Require resamplig	Time representation	Usable on raw data	Capture long term dependencies		Capture dependence between sensors	# steps
	EW			No			No	No		Yes		only inside the sequence			Yes	1
	SEW			Yes			No	No		Yes		depends of the size			Yes	1
	TW			Yes		Yes	Yes	Yes		depends of the size			No	1
	DW			Yes			No	No		Yes	only inside the pre segmented sequence		Yes	2
	FTW			Yes		Yes	Yes	Yes				Yes				No	2
	t0		t1	t2		t3	t4	t5	t6	t7	t8	t9 t10		t11 t12 t13 t14 t15	t16
	True labels	Activity 1		Activity 2	Activity 3	Activity 4				Activity 5
	Sensors'
	activation
	sequence
	Explicit
	Windows	Activity 1		Activity 2	Activity 3	Activity 4				Activity 5
	Time
	Windows	TW		TW	TW	TW	TW	TW	TW	TW	TW	TW	TW	TW	TW	TW	TW	TW
	Sensor Event Windows	SEW SEW	SEW SEW SEW		…		SEW	SEW			…				SEW
	Figure 2.3 -Example of segmentation methods on the univariate sensor representation. In this
	example, the Sensor Event Windows have a size of 6 events
											22

Table 2

 2

	.2 -Summary of handcrafted features for smart homes sensor based human activity
	recognition
	Domain	Feature name
		day of the week
	Time Features	hour of day second past midnight
		time of transition
		number of sensor events
		windows duration (time)
		most recent sensor
	Windows Features	last sensor location
		most frequent sensor
		dominant location
		previous dominant location
	Sensor Features	count of events elapse time of each sensor
		relation between sensors
		sensors localisation

Table 2 .

 2

		No	Eldery	12-22 days	Binary	14-21	8	3	2011
	Alemdar et al. [2]	Yes	Young	2 months	Binary	20	27	3	2013
	Cumin et al. [20]	No	Young	2 weeks	Binary, Scalar	236	20	1	2017
	Cook et al. [18]	Yes	Eldery	2-8 months Binary, Scalar	14-30	10-15	>30	2012
	Ordóñez et al. [52]	No	Eldery	14-21 days	Binary	12	11	2	2013
	Tapia et al. [69]	No	Eldery	2 weeks	Binary, Scalar	77-84	9-13	2	2004

4 -Example of real datasets of the literature

Ref

Multi resident Resident type Duration Sensor type # of Sensors # of Activity # of Houses Year

van Kasteren et al.

[START_REF] Van Kasteren | Human activity recognition from wireless sensor network data: Benchmark and software[END_REF]

Table 2 .

 2 5 -Overview of smart home simulation tools.

	Simulator	Open-Source 3D Cross-Platform	Approach	Focus	Multi-Inhabitants Fast-Forwarding
	OpenSHS [4]	Yes	Yes	Yes	Hybrid	Dataset generation	Partially	Yes
	SIMACT [11]	Yes	Yes	Yes	Model-based	Visualisation	No	Not applicable
	PerSim 3D [44]	No	Yes	Yes	Model-based Dataset generation	No	Not applicable
	Park et al. [53]	No	Yes	Yes	Interactive	Visualisation	No	Yes
	Ariani et al. [6]	No	No	No	Interactive	Dataset generation	Yes	No
	SESim [38]	Yes	Yes	No	Model-based Dataset generation	No	Yes

  . In this thesis, we propose to bridge the gap between test and training data using a smart home simulator that can generate training data closer to the test cases by replicating real apartments in a Digital Twin and modeling complex ADLs in that Digital Twin.

	Part, BIBLIOGRAPHY
	Chapter 3
	[75] Aiguo Wang, Shenghui Zhao, Chundi Zheng, Jing Yang, Guilin Chen, and Chih-Yung
	Chang. Activities of daily living recognition with binary environment sensors using deep
	learning: A comparative study. IEEE Sensors Journal, 21(4):5423-5433, 2020.
	[76] LuKun Wang and RuYue Liu. Human activity recognition based on wearable sensor using
	hierarchical deep lstm networks. Circuits, Systems, and Signal Processing, 39(2):837-856,
	2020. doi: 10.1007/s00034-019-01116-y.
	[77] Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch
	with deep neural networks: A strong baseline. In 2017 International joint conference on
	neural networks (IJCNN), pages 1578-1585. IEEE, 2017.

[78] Karl Weiss, Taghi M. Khoshgoftaar, and DingDing Wang. A survey of transfer learning.

Journal of Big Data, 3(1):9, May 2016. doi: 10.1186/s40537-016-0043-6.

[79] Nawel Yala, Belkacem Fergani, and Anthony Fleury. Feature extraction for human activity recognition on streaming data. In 2015 International Symposium on Innovations in Intelligent SysTems and Applications (INISTA), pages 1-6. IEEE, 2015.

[80] Surong Yan, Kwei-Jay Lin, Xiaolin Zheng, and Wenyu Zhang. Using latent knowledge to improve real-time activity recognition for smart iot. IEEE Transactions on Knowledge and Data Engineering, 2019.

  .1.

	Part, Chapter 3 -Background, Approach and Experimental Methodology
						activity sequence
	M004ON	M005OFF	M007OFF	M004OFF	M004ON	M004OFF	M007ON	M004ON	M007OFF	M007ON	M005ON	M004OFF	M004ON	M005OFF	M007OFF	M004OFF
	8	11	15	9	8	9	14	8	15	14	10	9	8	11	15	9
						Embedding
					Unidirectional LSTM / Bidirectional LSTM
						Softmax
						activity label

Table 3 .

 3

	1 -Details des Datasets
		Aruba Milan Cairo
	Residents	1	1+pet 2+pet
	Number of sensors	39	33	27
	Type of sensors	M,T,D M,T,D M,T
	Number of activities	12	16	13
	Number of days	219	82	56
	others see, the Figures 3.5

Table 3 .

 3

		2 -List of meta activities groups
		Milan	Cairo	Aruba
	Bathing	Master Bathroom Guest Bathroom
	Bed to toilet	Bed to toilet	Bed to toilet	Bed to toilet
			Lunch
	Cook	Kitchen Activity	Dinner	Meal Preparation
			Breakfast
	Eat	Dining Room Activity		Eating
	Enter home			Enter home
	Leave home	Leave home	Leave Home	Leave Home
	Personal hygiene
	Relax	Read Watch Tv		Relax
	Sleep	Sleep	R1 sleep R2 sleep	Sleeping
	Take medicine	Eve Meds Morning Meds	R2 take medecine	Resperate
	Work	Desk Activity Chores	Laundry R1 work in office	Work Housekeeping Wash Dishes
	Other	Meditate Master Bedroom Activity Other	Night wandering Other R2 wake R1 wake	Other

  the average of metrics in which each class's score is weighted by its presence in the true data sample. Weighted measure allows us to validate if a model focuses on majorities classes or not. A high weighted score and low macro-average average score for the same metric means the model focuses on most representative classes and vice versa. Weighted Precision, Weighted

				3.8. Summary
	W eighted_Recall =	Recall c1 * support c1 + ... + Recall cn * support cn T otal_support	(3.3)
	W eighted_F 1_score =	F 1_score c1 * support c1 + ... + F 1_score cn * support cn T otal_support	(3.4)
	Recall and Weighted F1-score are defined by Equations 3.2, 3.3, 3.4 respectively
			T otal_support	(3.2)
			66

W eighted_P recision =

P recision c1 * support c1 + ... + P recision cn * support cn

  .3. In this method, as explained in the previous chapter, we encode the input sensors' activation sequences into sequences of categorical variables which represent the sensors' activation sequences. Then, we project these sequences into the embedding to extract the meaningful features. The embedding turns the univariate time series into a multivariate time series where each channel is a feature. The FCN then uses the embedding output to extract patterns and new features to classify each sequence.

						activity sequence
	M004ON	M005OFF	M007OFF	M004OFF	M004ON	M004OFF	M007ON	M004ON	M007OFF	M007ON	M005ON	M004OFF	M004ON	M005OFF	M007OFF	M004OFF	M00
	1	4	8	2	1	2	7	1	8	7	3	2	1	4	8	2
						Embedding
					Fully Convolutional Network
						activity label

Table 4 .

 4 1 -F1 Score and Balanced Accuracy for Aruba and Milan datasets

			Aruba			Milan
	Model	100	75	50	25	100	75	50	25
					F1 Score (%)
	LSTM	96.67	94.67	90.67	85.00 84.00 85.67 75.33 64.00
	FCN	99.00	98.00	97.67	92.33 77.33 93.67 88.33 83.67
	LSTM + Embedding 100.00	99.67	98.00	90.00 98.00 97.00 93.00 73.67
	FCN + Embedding	100.00 100.00 100.00 99.00 99.00 98.00 97.00 94.33
				Balanced Accuracy (%)
	LSTM	81.45	76.09	71.05	83.30 62.15 64.95 55.70 43.29
	FCN	88.85	87.41	87.08	80.32 42.24 76.41 71.82 71.34
	LSTM + Embedding	94.55	93.61	90.20	74.81 88.52 86.77 82.05 59.35
	FCN + Embedding	95.37	95.07	94.89 92.44 84.23 86.64 87.83 90.86
				76

Table 4 .

 4

	3 -Comparison between embedding + bidirectional LSTM and embedding + FCN with
	zero padding
		embedding + bidirectional LSTM	embedding + FCN
		Aruba Milan	Cairo	Aruba Milan Cairo
	Accuracy	96.60 91.47	85.32	65.25 43.10 53.05
	Precision	92.5 83.49	74.49	44.68	4.27	17.1
	Recall	88.68 75.58	69.86	28.68	8.45	15.2
	F1-score	89.92 78.62	71.10	28.00	5.63	11.35
	Balance Accuracy	88.67 75.57	69.85	23.31	9.49	17.08
	Weighted Precision 96.73 91.05	83.17	77.05 21.24 36.69
	Weighted Recall	96.61 91.48	84.29	71.97 43.99 52.46
	Weighted F1-score 96.55 91.09	84.43	67.93 28.45 39.40

Table 4 . 4

 44

		Aruba Milan Cairo Aruba Milan	Cairo	Aruba Milan Cairo
	Accuracy	65.25 43.10 53.05 95.01 82.18	36.46	94.09 80.12	53.3
	Precision	44.68	4.27	17.1	78.94 69.87	15.61	77.86 65.07 21.29
	Recall	28.68	8.45	15.2	77.54 66.43 16.16	73.73 66.48 14.19
	F1-score	28.0	5.63	11.35 77.41 65.42	10.5	73.84 64.62 13.67
	Balance Accuracy	23.31	9.49	17.08 76.79 66.36 19.57	72.47 65.11 17.15
	Weighted Precision 77.05 21.24 36.69 94.46 81.85 42.95	93.61 80.35 35.01
	Weighted Recall	71.97 43.99 52.46 94.85 82.15	43.26	93.93 79.87 51.83
	Weighted F1-score	67.93 28.45	39.4 94.55 81.42	37.35	93.58 79.41 37.56

-Comparison between zero padding, symmetric padding and circular padding FCN zero-padding FCN symmetric-padding FCN circular-padding

Table 4 .

 4

	5 -Comparison between embedding + bidirectional LSTM and embedding + FCN with
	circular padding
		embedding + bidirectional LSTM	embedding + FCN
		Aruba Milan	Cairo	Aruba Milan Cairo
	Accuracy	96.60 91.47	85.32	94.09 80.12	53.3
	Precision	92.5 83.49	74.49	77.86 65.07 21.29
	Recall	88.68 75.58	69.86	73.73 66.48 14.19
	F1-score	89.92 78.62	71.10	73.84 64.62 13.67
	Balance Accuracy	88.67 75.57	69.85	72.47 65.11 17.15
	Weighted Precision 96.73 91.05	83.17	93.61 80.35 35.01
	Weighted Recall	96.61 91.48	84.29	93.93 79.87 51.83
	Weighted F1-score 96.55 91.09	84.43	93.58 79.41 37.56

Table 5 .

 5

	2 -ELMo embeddings hyperparameters
	Embedding size	64
	Context windows size 60
	Max epochs number	400
	Batch size	512

Table 5 .

 5 3 -ELMo outputs types comparison

			Aruba		Milan		Cairo
		Weighted Sum Sum Last Concat Weighted Sum Sum Last Concat Weighted Sum Sum Last Concat
	Accuracy	96,80	96,77 96,25 96,86	89,90	89,74 89,58 90,25	88,70	88,52 86,68 89,36
	Precision	94,50	94,47 93,5 94,63	84,00	82,91 84,09 85,91	81,00	80,24 81,15 84,30
	Recall	88,60	88,4 87,59 88,73	76,00	75,93 75,05 74,85	78,30	76,31 78,06 80,21
	F1-score	90,50	90,46 89,59 90,71	78,70	78,54 78,34 77,86	78,50	77,78 79,02 81,34
	Balanced Accuracy	88,56	88,39 87,59 88,72	76,00	75,92 75,04 74,84	77,00	76,3 78,05 80,20
	Weighted Precision	96,90	96,89 96,35 96,98	89,80	89,43 89,38 90,06	88,05	87,64 86,77 89,24
	Weighted Recall	96,82	96,77 96,25 96,86	89,80	89,74 89,58 90,25	88,80	88,52 86,68 89,36
	Weighted F1-score	96,72	96,7 96,18 96,79	89,60	89,43 89,26 89,74	88,10	87,88 86,45 89,02

Table 5 .

 5 4 -Without and with embedding coupled with a LSTM Classifier.

			Aruba				Milan				Cairo
		No Embedding	Liciotti Sensor2Vec ELMo	No Embedding	Liciotti Sensor2Vec ELMo	No Embedding	Liciotti Sensor2Vec ELMo
	Accuracy	94.63	96.50	96.6	96.61	79.18	89.88	86.31	87.29	74.82	81.44	79.31	82.62
	Precision	71.84	80.50	79.39	86.24	76.22	79.17	81.75	86.16	60.73	68.88	55.31	78.10
	Recall	70.00	79.52	80.07	79.39	61.89	74.26	69.33	75.35	60.46	69.98	62.90	72.358
	F1 score	70.55	79.49	79.34	80.73	65.94	76.28	72.07	78.90	59.11	68.80	58.92	73.78
	Balance Accuracy	70.00	79.52	80.11	79.39	61.88	74.26	69.33	75.35	60.46	69.98	62.91	72.58
	Weighted Precision	93.56	96.11	96.21	96.41	78.84	88.51	86.08	87.26	72.08	79.64	73.6	82.77
	Weighted Recall	94.06	95.5	96.6	96.61	79.18	88.99	86.31	87.29	74.82	81.44	79.31	82.62
	Weighted F1-score	93.77	96.88	96.32	96.43	78.42	88.63	85.79	87.07	72.73	80.09	75.79	82.05

Table 5 .

 5 5 -Without and with embedding coupled with a Bi-LSTM Classifier.

			Aruba				Milan				Cairo
		No Embedding	Liciotti Sensor2Vec ELMo	No Embedding	Liciotti Sensor2Vec ELMo	No Embedding	Liciotti Sensor2Vec ELMo
	Accuracy	95.01	96,61	96.59	96,86	82.24	90,82	88.33	90,25	81.68	85,33	82.27	89,36
	Precision	79.75	92,5	82.97	94,63	79.6	79,94	84.42	85,91	68.45	74,49	77.56	84,3
	Recall	77.73	88,68	81.06	88,73	67.77	74,91	73.62	74,85	70.09	69,86	69.38	80,21
	F1-score	77.92	89,92	81.43	90,71	71.81	76,86	76.59	77,86	68.47	71,1	70.95	81,34
	Balance Accuracy	77.73	88,67	81.06	88,72	67.77	74,90	73.61	74,84	70.09	69,85	69.38	80,2
	Weighted Precision	94.69	96,73	96.23	96,98	82.28	90,31	88.28	90,06	80.22	84,29	82.04	89,24
	Weighted Recall	95.01	96,61	96.59	96,86	82.24	90,82	88.33	90,25	81.68	85,33	82.27	89,36
	Weighted F1-score	94.74	96,55	96.32	96,79	81.97	90,41	87.98	89,74	80.49	84,43	81.14	89,02

Table 5 .

 5 6 -Liciotti Bi-LSTM and ELMo + Bi-LSTM and ELMo + FCN(Circular Padding) comparison

			Aruba			Milan			Cairo
		Liciotti ELMo	ELMo+ FCN	FCN Liciotti ELMo	ELMo+ FCN	FCN Liciotti ELMo	ELMo+ FCN	FCN
	Accuracy	96,61	96,86	94.59	94,09 90,82	90,25	85.33	80,12	85,33	89,36	84.39	53,3
	Precision	92,5	94,63	73.23	77,86	79,94	85,91	83.74	65,07	74,49	84,3	79.28	21,29
	Recall	88,68	88,73	69.27	73,73 74,91	74,85	70.49	66,48	69,86	80,21	80.08	14,19
	F1-score	89,92	90,71	69.30	73,84 76,86 77,86	74.79	64,62	71,1	81,34	78.13	13,67
	Balance Accuracy	88,67	88,72	69.57	72,47 74,90	74,84	70.86	65,11	69,85	80,2	80.29	17.15
	Weighted Precision	96,73	96,98	94.09	93,61 90,31	90,06	85.31	80,35	84,29	89,24	83.91	35.01
	Weighted Recall	96,61	96,86	94.53	93,93 90,82	90,25	85.36	79,87	85,33	89,36	82.27	51.83
	Weighted F1-score	96,55	96,79	94.10	93,58 90,41	89,74	85.02	79,41	84,43	89,02	82.27	37.56

Table 5 .

 5 7 -Comparison with two layers of Bi-LSTM.

			Aruba		Milan		Cairo
		1L	2L	ELMo 1L	2L	ELMo 1L	2L	ELMo
	Accuracy	96,61 96.46 96,86 90,82 90.03 90,25 85,33 84.99 89,36
	Precision	92,5 82.01 94,63 79,94 84.29 85,91 74,49 80.87 84,30
	Recall	88,68 79.05 88,73 74,91 77.31 74,85 69,86 76.6	80,21
	F1-score	89,92 79.97 90,71 76,86 79.29 77,86 71,1 77.44 81,34
	Balance Accuracy	88,67 78.74 88,72 74,90 75.51 74,84 69,85 76.52 80,20
	Weighted Precision 96,73 96.04 96,98 90,31 90.22 90,06 84,29 85.04 89,24
	Weighted Recall	96,61 96.41 96,86 90,82 90.28 90,25 85,33 84.4	89,36
	Weight F1-score	96,55 96.13 96,79 90,41 90.07 89,74 84,43 84.08 89,02

Table 5 .

 5 8 -ELMo from Aruba applied to Cairo and Milan VS original ELMo and Liciotti-baseline

			Cairo			Milan
		ELMo from Cairo ELMo from Aruba Liciotti ELMo from Milan ELMo from Aruba Liciotti
	Accuracy	89,36	88,1	85,33	90,25	90,36	90,82
	Precision	84,3	79,39	74,49	85,91	87,82	79,94
	Recall	80,21	73,44	69,86	74,85	75,61	74,91
	F1-score	81,34	75,61	71,1	77,86	79,52	76,86
	Balance Accuracy	80,2	73.43	69,85	74,84	75,6	74,90
	Weighted Precision	89,24	87,1	84,29	90,06	90,29	90,31
	Weighted Recall	89,36	88,1	85,33	90,25	90,36	90,82
	Weighted F1-score	89,02	87,29	84,43	89,74	90,03	90,41

  will note LPA for Liciotti + previous activities and EPA for ELMo + previous activities.

			activity sequence n-2				activity sequence n-1							activity sequence n
	T00216	T00218	D001OPEN	D001CLOSE	T00123	M015ON	M010OFF	M010ON	M015OFF	M007OFF	T00124	M007ON	M004ON	M005OFF	M007OFF	M004OFF	M004ON	M004OFF	M007ON	M004ON	M007OFF	M007ON	M005ON	M004OFF
	40	42	22	23	37	18	11	12	20	8	35	8	1	4	8	2	1	2	7	1	8	7	3	2
												Embedding / ELMo
												Bi LSTM
												Softmax
												activity label of
												activity sequence n

The Time of Activity Sequence Appearance

  Part, Chapter 6 -Taking Advantage of Dependence on Previous Activities and Time of the Day

									100
									90
									80
									70
									60
									50
	Aruba		Milan			Cairo						Aruba	Milan	Cairo
	Liciotti-baseline (LB)	ELMo-baseline (EB)	Liciotti + previous activities (LPA)					Liciotti-baseline (LB)	ELMo-baseline (EB)	Liciotti + previous activities (LPA)
		ELMo + previous activities (EPA)									ELMo + previous activities (EPA)
	(a) Grouped activities Accuracy					(b) Original activities Accuracy
									100
									90
									80
									70
									60
									50
	Aruba		Milan			Cairo						Aruba	Milan	Cairo
	Liciotti-baseline (LB)	ELMo-baseline (EB)	Liciotti + previous activities (LPA)					Liciotti-baseline (LB)	ELMo-baseline (EB)	Liciotti + previous activities (LPA)
		ELMo + previous activities (EPA)									ELMo + previous activities (EPA)
	(c) Grouped activities Balanced Accuracy		(d) Original activities Balanced Accuracy
									100
									90
									80
									70
									60
									50
	Aruba		Milan			Cairo						Aruba	Milan	Cairo
	Liciotti-baseline (LB)	ELMo-baseline (EB)	Liciotti + previous activities (LPA)					Liciotti-baseline (LB)	ELMo-baseline (EB)	Liciotti + previous activities (LPA)
		ELMo + previous activities (EPA)									ELMo + previous activities (EPA)
	(e) Grouped activities F1-scores					(f) Original activities F1-scores
									100
									90
									80
									70
									60
									50
	Aruba		Milan			Cairo						Aruba	Milan	Cairo
	Liciotti-baseline (LB)	ELMo-baseline (EB)	Liciotti + previous activities (LPA)					Liciotti-baseline (LB)	ELMo-baseline (EB)	Liciotti + previous activities (LPA)
		ELMo + previous activities (EPA)									ELMo + previous activities (EPA)
	(g) Grouped activities Weighted F1-scores		(h) Original activities Weighted F1-scores
	Figure 6.4 -Metrics comparison between baseline and previous activities models on Aruba,
	Milan and Cairo datasets
	6.4 M004ON	M005OFF	M007OFF	M004OFF	M004ON	M004OFF activity sequence M007ON	M004ON	M007OFF	M007ON	M005ON	M004OFF	12 time when the sequence appears
	1	4	8	2	1	2	7	1	8	7	3	2
						Embedding / ELMo						Projection
						Bi LSTM
											Softmax
										activity label

  The Figure6.8 presents the results between baseline and time sequence extension models.We can observe on it that the ETS model outperforms all other models on every dataset, but in particular shows the best improvements with the Milan and Cairo datasets. The Balanced Accuracy and the F1-score rise for both dataset versions, except for the Aruba dataset where performance is still almost the same. The LTS model performed better on the Milan and Cairo datasets than the LB model, but decreased in performance on the Aruba dataset. For both models the best improvements can be seen on the Cairo dataset. The LTS improves 15.7% for the grouped labels version and 28.58% for the original labels version in Balanced Accuracy. It rose also to 14% for the grouped labels version and 31.28% for the original labels version in F1-score.In the meantime, the ETS model improved 3% and 17.27 % on the grouped and original dataset versions in Balanced Accuracy. In addition, this model increased the F1-score by 4.7% and 16.45% respectively for grouped and original dataset versions.

																6.5. The Time of Sensor Event Occurrence
						activity sequence											time sequence
	M004ON	M005OFF	M007OFF	M004OFF	M004ON	M004OFF	M007ON	M004ON	M007OFF	M007ON	M005ON	M004OFF
	1	4	8	2	1	2	7	1	8	7	3	2	1	1	1	1	2	2	2	3	4	5	6	6
						Embedding / ELMo											Embedding
						Bi LSTM											Bi LSTM
												Softmax
												activity label
						Figure 6.7 -Activity sequence and time sequence
												128

Table 6 .

 6 1 -F1-score by models and by relabeled activities of each dataset Taking Advantage of Dependence on Previous Activities and Time of the Day

			Liciotti	ELMo	Liciotti +	Liciotti +	Liciotti +	ELMo +	ELMo +	ELMo +
			Baseline	Baseline	previous activities	time	time sequence	previous activities	time	time sequence
			(LB)	(EB)	(LPA)	(LT)	(LTS)	(EPA)	(ET)	(ETS)
		Bed_to_toilet	99,37	98,72	97,81	99,69	99,36	97,48	99,05	98,72
		Cook	96,89	96,92	95,89	96,88	97,29	95,53	96,84	97,14
		Eat	93,46	95,58	88,21	93,52	92,50	90,68	93,82	94,47
		Enter_Home	78,07	77,67	97,67	81,40	78,37	98,23	77,36	81,24
	Aruba	Leave_Home Other	81,02 98,49	81,99 98,74	96,48 97,83	80,45 98,50	80,87 98,58	98,01 97,89	81,13 98,58	82,16 98,62
		Relax	98,45	98,72	98,24	98,66	98,61	97,96	98,84	98,91
		Sleep	98,88	99,38	99,50	98,74	99,13	98,99	99,12	98,85
		Take_medicine	72,23	77,78	0,00	71,12	0,00	38,89	77,78	72,23
		Work	82,34	81,62	83,09	82,79	81,39	80,65	80,91	80,63
		Bathing	94,13	92,32	93,51	93,56	96,10	91,73	92,67	95,31
		Bed_to_toilet	61,94	41,55	73,12	58,30	81,32	71,65	56,89	81,91
		Cook	90,42	89,19	82,98	90,85	89,77	82,32	89,95	89,22
		Eat	15,39	32,64	19,97	19,53	0,00	6,07	26,41	23,24
	Milan	Leave_Home Other	96,53 92,88	97,40 91,93	90,24 87,20	95,79 92,69	95,32 92,35	87,17 85,95	96,29 92,09	96,46 91,68
		Relax	93,20	90,98	84,75	92,53	92,55	81,44	91,06	91,04
		Sleep	88,24	89,74	80,27	90,22	91,30	78,99	90,22	89,88
		Take_medicine	84,57	82,04	72,01	86,55	82,50	67,66	81,80	85,60
		Work	68,93	70,79	61,07	69,57	69,83	61,85	71,47	68,20
		Bed_to_toilet	14,02	44,71	15,02	56,43	52,23	19,53	36,71	48,58
		Eat	83,45	87,23	74,85	85,15	88,13	73,18	84,55	89,00
		Leave_Home	94,36	97,11	88,15	96,23	96,40	83,61	96,46	97,88
	Cairo	Other	90,88	92,92	86,85	91,83	92,25	86,71	92,95	94,30
		Sleep	59,42	72,15	69,30	75,99	82,41	69,26	77,61	88,76
		Take_medicine	76,48	87,64	64,75	83,27	83,00	83,80	81,75	89,07
		Work	79,08	87,59	55,99	79,10	72,66	52,82	86,40	88,33

  Part, Chapter 6 -Taking Advantage of Dependence on Previous Activities and Time of the Day

	100
	75
	50
	25
	0
	B e d _ t o _ T o il e t	E a t in g	E n t e r _ H o m e	H o u s e k e e p in g	L e a v e _ H o m e	M e a l_ P r e p a r a	O t h e r	R e la x	R e s p ir a t e	S le e p in g	W a s h _ D is h e s	W o r k
	Liciotti-baseline (LB)	ELMo-baseline (EB)	Liciotti + previous activities (LPA)	Liciotti + time (LT)	Liciotti + time sequence (LTS)	ELMo + previous activities (EPA)
						ELMo + time (ET)	ELMo + time sequence (ETS)

Table 6 .

 6 2 -F1-score by models and by activities of each dataset

			Liciotti	ELMo	Liciotti +	Liciotti +	Liciotti +	ELMo +	ELMo +	ELMo +
			Baseline	Baseline	previous activities	time	time sequence	previous activities	time	time sequence
			(LB)	(EB)	(LPA)	(LT)	(LTS)	(EPA)	(ET)	(ETS)
		Bed_to_Toilet	99,37	99,05	98,77	99,36	99,04	97,48	98,72	98,41
		Eating	92,19	95,65	91,30	94,26	91,43	90,39	93,68	93,51
		Enter_Home	77,53	76,27	96,97	79,49	79,94	97,76	78,29	80,18
		Housekeeping	78,40	81,05	83,81	81,33	69,94	76,41	78,20	84,74
		Leave_Home	80,97	81,17	96,01	81,15	80,43	97,31	81,56	80,70
	Aruba	Meal_Preparation Other	96,67 98,33	96,65 98,58	95,93 97,93	96,59 98,54	95,94 98,06	95,66 97,88	96,61 98,48	96,64 98,49
		Relax	98,24	98,73	98,00	98,54	98,18	98,05	98,74	98,76
		Respirate	55,56	77,78	0,00	55,56	22,23	0,00	44,45	55,56
		Sleeping	98,88	99,12	99,00	99,00	98,88	99,11	98,99	98,34
		Wash_Dishes	0,00	4,31	22,95	0,00	0,00	25,23	9,37	4,05
		Work	98,02	98,00	96,54	98,54	98,00	95,98	98,30	98,85
	Milan	Bed_to_Toilet Bed_to_toilet	28,32	63,16	44,28	50,71	64,20	26,99	61,68	62,43
		Breakfast	47,90	47,76	70,91	71,27	75,99	80,67	55,89	71,12
		Dinner	28,44	52,18	18,37	74,45	95,32	24,42	53,30	89,50
		Laundry	86,67	100,00	77,47	88,58	75,56	72,23	100,00	100,00
		Leave_home	95,83	97,17	88,14	95,68	95,78	74,09	97,17	97,88
		Lunch	7,85	27,65	24,04	22,86	91,42	32,01	37,79	88,75
		Night_wandering	70,54	69,79	65,88	73,24	78,16	72,14	75,66	77,73
		Other	88,42	91,15	83,24	89,75	90,13	82,05	90,88	91,95
		R1_sleep	51,92	65,78	64,18	69,85	76,76	58,21	68,06	73,71
		R1_wake	70,86	86,11	64,65	80,10	84,05	70,31	87,72	90,25
		R1_work_in_office	73,11	74,79	58,32	72,86	60,77	48,39	72,63	83,00
		R2_sleep	67,92	69,86	69,45	71,39	81,47	64,34	78,27	83,76
		R2_take_medicine	76,59	85,10	73,91	82,98	84,72	82,78	81,60	75,86
		R2_wake	70,50	76,64	59,53	76,30	81,16	73,13	83,16	86,93

Table 7 .

 7 4 -Results of one to one experiment

			S9				S7				S3
		Precision Recall	F1-score Support Precision Recall	F1-score Support Precision Recall	F1-score Support
	Bathe	0,00%	0,00%	0,00%	2 100,00% 100,00% 100,00%	2	50,00% 100,00%	66,67%	4
	Cook	90,00%	90,00%	90,00%	10	54,55% 100,00%	70,59%	6	75,00%	75,00%	75,00%	4
	Dress	75,00% 100,00%	85,71%	3 100,00% 100,00% 100,00%	3 100,00%	50,00%	66,67%	2
	Eat	50,00%	50,00%	50,00%	6 100,00%	75,00%	85,71%	4	0,00%	0,00%	0,00%	3
	Enter Home	50,00%	75,00%	60,00%	4 100,00%	33,33%	50,00%	3	60,00% 100,00%	75,00%	3
	Go To Toilets	66,67%	80,00%	72,73%	5	75,00% 100,00%	85,71%	3	0,00%	0,00%	0,00%	2
	Leave Home	40,00%	50,00%	44,44%	4	60,00% 100,00%	75,00%	3	0,00%	0,00%	0,00%	2
	Sleep	50,00%	33,33%	40,00%	3 100,00%	75,00%	85,71%	4 100,00%	50,00%	66,67%	2
	Wash Dishes	100,00%	80,00%	88,89%	5 100,00%	25,00%	40,00%	4	44,44% 100,00%	61,54%	4
	Watch TV	100,00%	85,71%	92,31%	7 100,00%	80,00%	88,89%	5	0,00%	0,00%	0,00%	2
	Accuracy	71,43%				78,38%				57,14%
	Balanced Accuracy	64,40%				78,83%				47,50%
	Macro Avg	62,17%	64,40%	62,41%	49	88,95%	78,83%	78,16%	37	42,94%	47,50%	41,15%	28
	Weighted Avg	70,78%	71,43%	70,39%	49	87,36%	78,38%	76,91%	37	44,92%	57,14%	46,59%	28

5.4. Activity recognition with Pre-trained Embedding

48,68 46,60 63,96 59,44 71,47 68,37 53,74 78,05 Chores 3,93 4,77 0,00 5,56 0,00 0,00 13,34 21,64 Desk_Activity 89,73 93,99 87,89 94,53 93,76 83,33 96,19 95,39 Dining_Rm_Activity 20,00 29,90 16,67 12,73 0,00 12,13 5,56 26,28 Eve_Meds 0,00 0,00 15,75 78,79 81,12 8,34 65,86 70,91 Guest_Bathroom 96,60 95,72 94,19 96,76 96,43 91,24 95,99 96,14 Kitchen_Activity 90,79 89,79 82,80 90,52 89,79 80,52 88,98 89,46 Leave_Home 96,19 97,39 87,16 96,03 96,24 85,05 96,21 95,76 Master_Bathroom 87,77 89,25 91,58 90,36 92,55 90,75 89,00 93,95 Master_Bedroom_Activity 77,73 78,72 65,03 80,07 75,35 63,37 75,72 75,24 Meditate 85,19 94,45 79,66 84,65 93,34 60,24 94,45 93,27 Morning_Meds 65,28 58,72 53,80 84,93 72,90 44,49 74,78 72,47 Other 89,14 89,55 83,43 89,79 89,14 81,44 88,85 88,92 Read 93,88 93,72 88,92 95,82 94,94 86,91 93,88 93,52 Sleep 85,64 89,18 76,94 85,62 91,59 77,42 89,14 90,37 Watch_TV 82,38 80,94 75,14 82,28 82,77 72,43 80,67 81,93 Cairo

In this experimentation, we evaluate the abilities of both models to extract patterns and features. To conduct this evaluation, we followed the framework presented in Figure 4.4. The input of our models is a sliding window of SEW that contain a piece of an encoded activity sequence. Depending on the SEW size and the activity length, some windows are padded with zero values to maintain a constant input length, this process is called "zero padding". Once all the activity sequences have been cut into smaller windows and annotated with the label of the activity from which it originates, we distribute them into subsets for training and testing the algorithms. This distribution is done using the Stratified 3-fold cross validation method detailed in chapter 3.

Sensor Events: The Semantic and Context Embedding

Sensor2Vec: Sensors' Activations Visualization

Once the embedding of the sensors is trained, it is possible to interpret the features generated by the model by visualizing the embedding vector of each sensor activation. To do this, each embedding vector is projected into a 2D or 3D space. Therefore, as embedding vectors have a d-dimensions size, they should be reduced. After extracting the vector representation of each sensor activation, the Uniform Manifold Approximation and Projection (UMAP) algorithm [10] is used to reduce vectors' dimension.

UMAP is a recent dimension reduction technique that can be used for visualisation, but also for general non-linear dimension reduction. The algorithm is founded on three assumptions about the data: 1) The data is uniformly distributed on Riemannian manifold; 2) The Riemannian metric is locally constant (or can be approximated as such); 3) The manifold is locally connected.

Once reduced to 2D vectors, each of them can be plotted. For example, the Figure 5.4a is the sensors' activation projection of vectors from Aruba dataset. At a first look, several clusters can be seen. Coloring each point with a color that corresponds to the type of sensor, as in Figure 5.4b, outlines that temperature sensors (light blue) are clustered all together, and other clusters seems to be composed mainly of motion sensors. The model has learnt that temperature sensors have not particular links with others sensors. It seems that the model learned the sensor nature. We first studied the performance of the different models on the three re-labeled datasets. Figure 6.9 contains a comparison of the scores obtained by each model. On the Aruba dataset, all the models had a similar accuracy and F1-score. A slight improvement can be noted for the LPA and EPA models in terms of accuracy. The LT model had the best balanced accuracy score, followed very closely by the ET sequence and EB models. If we look at the F1-score for this dataset, the EB model remains the best, followed by the ET and ETS models. This last metric allows us to conclude that the models using the ELMo embedding seem to be more balanced in terms of class recognition for this dataset. Indeed, a high F1-score means that recall and precision are also high.

On the Milan dataset, the LTS and LB models come second and third in terms of accuracy after the EB model, while the other ELMo based models are not far behind. In terms of balanced In summary, we can see that using information from previous activities does not seem to improve the performance of the two baseline models, except for the recognition of the activity classes "Enter Home" and "Leave Home" in the Aruba dataset. The addition of time and its interpretation in sequential form seems to produce performance improvements for the dataset. This increment allows us to improve the performance of the algorithms for the Cairo dataset, in particular for the Liciotti model. In general, the ETS model produces the best performances for the three datasets. The addition, time sequence inclusion, combined with ELMo embedding, seems to be a powerful method for the activity sequence classification problem. equivalent to the score from the previous experiment. However, the balanced accuracy rating has decreased, which may be caused by the increase in the number of classes, with very similar classes, e.g. "Meal preparation" and "Wash dishes". The Balanced Accuracy allows us to determine that these approaches can recognize a greater number of different classes of activities. The EB model The EB model already had a slight lead over the LB model, which was solidified by the Liciotti and ELMo extended approaches. Indeed, the ETS approach outperforms all the others in terms of balanced accuracy, F1-score and weighted F1-score. The latter seems to be the most efficient and balanced approach for this dataset. Nevertheless, the LT and LTS approaches are not outdone as they perform better than the LB, and are close to the performance of the ETS approach. Once again, the LPA and EPA approaches drop in performance, as with the previous experiment.

Activities of Origin: Global Analysis

For the Cairo dataset, the EB approach performs better than the LB approach in accuracy (81.22% versus 75.77%), balanced accuracy (70.57% versus 62.06%) and in F1-score (71.94% versus 61.78%). This gap is reduced by the LT and LTS approaches, which catch up and largely exceed, in particular the LTS model, the performances of the EB approach. However, the ETS approach keeps a significant lead, especially in balanced accuracy and F1-score. The "previous activities" approaches do not improve the performances for this dataset.

Activities of Origin: Class by Class Analysis

As before, we studied in more detail the performance of each model for each class of datasets (see Figure 6.12 and Table 6.2).

Concerning the Aruba dataset, the hardly recognizable classes are "Respirate" and "Wash dishes". "Respirate" corresponds to an activity using a medical breathing apparatus and appears only six times in the dataset. In terms of sensor activation, the activity "Wash dishes" is similar to "Meal preparation". The "Wash dishes" class of activity is poorly recognized or completely unrecognized by baseline models. Looking at the results of the approaches using time, it seems that the temporal information does not grant better recognition of this activity. However, the LPA and EPA models received the best scores for this activity with a F1-score around 25%, it seems that only these two approaches worked. For the activity "Respirate" the EB performed the best with a F1-score of 77.78%, followed by the ETS, LT and LB models. We noticed that the use of information from previous activities significantly increases the performance for recognizing the classes "Enter Home" and "Leave Home", as was the case previously. Globally the two best models on this dataset and configuration, labels of origin, are EB and ETS.

Data Generation Methodology

Our goal is to generate data from a Digital Twin apartment in order to train an algorithm, and then to evaluate the performance of the trained algorithm on real data from the apartment. To do this, we defined three scenarios that we called: "Morning", "Middle of the day" and "Evening".

The morning scenario consists of simulating the activities from the time a resident wakes up until he or she leaves the apartment to go to work. The midday scenario is a simulation of the resident returning home, having lunch, and resting before returning to work. The last scenario, evening, represents the activities from the time the resident returns home until he goes to bed.

For these scenarios we have defined a list of possible activities labels : "Bathe", "Cook breakfast", "Cook dinner", "Cook lunch", "Dress", "Eat breakfast", "Eat dinner", "Eat lunch", "Enter home", "Go to toilet", "Leave home", "Read", "Sleep", "Wash dishes", "Watch TV".

These labels were inspired from labels of literature dataset such as CASAS [3], but principally by the ADLs definitions define by Katz [5] We recorded real-world data from ten volunteers who played all the three scenarios in the smart apartment. Each volunteer performed each scenario at least once. Some volunteers performed the same scenario several times in different ways to add variability. The volunteers were recruited without any particular selection criteria. All of them were in good health and did not present any particular disorder because we did not aim at a medical framework. The main objective was to collect activity data without any particular anomalies to more easily reproduce them in the simulation environment.

Using a camera, we annotated the different activities as the volunteers performed them and describes them aloud. We asked the volunteers to express, just before the end of the current activity, the next activity that they perform. A new activity always starts at the end of the current activity. In this way, all sensor activations have an activity label.

Once all scenarios were recorded, we replicated the scenarios performed by the volunteers in the digital twin of the apartment. To do this, we used the video recordings of each volunteer's scenarios to define activity scripts that exactly copy the volunteers' actions. Each script contained the list and order of actions that the avatar had to perform to replicate the activities performed in the different scenarios. The avatar then executes the different scripts in the digital twin to generate the virtual sensor logs.

Synthetic and Real Data Post-Processing and Comparison

Unfortunately, because the transcription of the videos into a script is time consuming, we were only able to reproduce the scenarios of 3 volunteers in the Digital Twin which resulting in 15 reproduced scenarios over 62. We will compare and use the data of these 3 volunteers in the following, details in the Table 7.1.

Post-Processing

Before using the generated data for the algorithm training, we cleaned, analysed and compared the two datasets. Since the synthetic dataset only contains binary sensor activations and the smart apartment contains more sensors than the digital twin, due to the lack of implementation of more complex sensors such as the CO2 sensor, we had to remove all sensors activations that we haven't implemented. In addition, as smart outlets were used to monitor some appliance such as the television for example, we turn power consumption's into binary states. If the consumption of the television exceeds a certain threshold, then it is considered turned on, and vice versa.

For the synthetic data, we have removed the bad room transition. Indeed, we did not control the path finding algorithm of the avatar in the simulator. Thus, the avatar, in some cases, crosses multiple rooms before it reaches its room destination. For example, in the smart apartment, volunteers that start in the entrance and want to reach the kitchen cross rooms in this order: entrance, living-room, dining-room and then, kitchen. In some cases the avatar adds the cross of the bedroom before or after the dining-room. These particular cases could disturb the algorithm of ADLs recognition as during the recording, the volunteers have respected some passage points between rooms to simulate doors.

Basic Comparison

A first comparison consists in looking at the number of sensor activations in synthetic and real scenarios' data (see Figure 7.7). We can see that we have more sensor activations into real data than into synthetic data. This is explained as synthetic data doesn't have trouble and are "perfect" compared to real data. Indeed, the connected floor sensor sometimes loses and recovers the position of the resident, which increases the number of activations. Another example is the armchair pressure sensor, which alternates between "pressure detected" and "not pressure detect" in function of the position of the resident in the armchair.

Comparison with Cross-Correlation Method

In the context of a digital twin, it is important to evaluate whether the data generated by the simulator are equivalent to the real data. Moreover the similarity between synthetic and real data must be respected so that an algorithm can be trained under the same conditions as if it had been trained on real data. In order to compare more precisely the synthetic and real data, we used the Cross-Correlation Method [12].

Cross-Correlation is used in signal processing, which measures the similarity of two series as a function of the displacement of one relative to the other. This is also known as a sliding dot product or sliding inner-product. The cross-correlation is similar in nature to the convolution of two functions. In an autocorrelation, which is the Cross-Correlation of a signal with itself, there will always be a peak at a lag of zero, and its size will be the signal energy. The Cross-Correlation for discrete function [10] is expressed as in Equation 7.1. Where f and g are two functions, and n is the displacement or lag between the two functions.

Table 7.2 contains the cross correlation score for a lag of 0 between each synthetic and real scenarios. The results show that synthetic dataset seem close, which means that the simulator can provide sensor logs close to reality (between 70.00% and 97.00%). We can observe the worst similarity for the scenario 14. In the next section, we will use these data to train an ADLs recognition algorithm and conclude if Digital Twin data can be used.

The Evaluations

To validate the generated data, we trained the algorithm following four methods and experiments explained in the following subsections. To perform the training we proceeded as in the previous chapters, i.e. we segmented the data in the form of activity sequence. We also grouped under the same label "Cook" all activities related to cooking, i.e. "Cook breakfast", "Cook dinner"

and "Cook lunch". We did the same for activities related to the more generic activity "Eat", i.e. "Eat breakfast", "Eat dinner", "Eat lunch". This grouping is motivated by the fact that the real experiments were carried out during the day and do not respect the wall clock time of appearance of each scenario. In other words, the "evening" scenario was not realized in the evening, and the same is true for the others. This prevents us from using temporal information to differentiate these activities, which are very strongly linked to the time of day. For the training of the algorithms, we used each time 20% of the training set as validation data to stop the training in case of overfitting (see Chapter 3).

Leave One Subject Out Cross Validations

The first two approaches apply the Leave One Out cross validation method [11] on the one hand on the synthetic data only and on the other hand on the real data only. This type of methods is usually used for datasets containing small amounts of data or when the data are linked to whole entities. As we have three subjects, the principle is to use one subject for the test and the two others for the training. Then a rotation on which subject becomes the test set is performed which allow us to validate the algorithm on each subject data. Finally, we make a comparison between results of the synthetic and real Leave One Out cross validation. The Table 7.3 provides the result of the two Leave One Out cross validations. The results demonstrate that the algorithm can be trained with real and synthetic data and obtain almost similar results. We can observe a high similarity between the results of the subject "S9". However, 7.7. Summary whether increasing the amount of data provided by the synthetic data of the different subjects improves the capabilities of the algorithm for recognizing the ADLs. Table 7.5 shows the results of this experiment. We can observe that the performances, compared to the One to One method, are improved.

Except for the subject "S7" which keeps very similar performances to the previous experiment.

However, for subjects "S9" and "S3" we observe an improvement, especially for subject "S3", who until now showed the worst results. These results show us that it seems conceivable and possible to generate fully synthetic data, to use them to train an algorithm and to recognize activity sequences from real data.

Summary

The datasets are not sufficient to cover the distribution of possibilities in terms of activities, lifestyles, or house configurations. They are limited in the number of activities, occupants they consider, as well as the types of sensors and the category of residents. Therefore, algorithms trained on data from one particular environment will be difficult or impossible to reuse in others, as they are tailored to a particular home configuration, inhabitants, and living habits.

Collecting new datasets to increase the number of examples is a very expensive task in terms of equipment and difficult to achieve. It is necessary to ask residents to live in the environment and label their activities over several weeks or months. This involves a significant investment on the part of volunteers and presents a high risk of mislabeling.

The use of transfer learning techniques seems to be a solution. Nevertheless, it is necessary to have labeled data from the target environment to adjust the algorithms to fit perfectly. From the perspective of a real-world business application, it is not feasible to ask end users or customers to label their activities themselves for several days, weeks, or months.

Part, Chapter 7 -Generating Data for Human Activity Recognition in Real-World Applications

To address the challenges of data scarcity and, in particular, the gap between training data and use case data, in this chapter, we proposed data generation through the digital twin concept.

We improved and extended the Virtual Home simulation environment to allow it to simulate an intelligent environment equipped with home automation sensors. We reproduced a digital version of a real intelligent apartment in this simulation environment. Then, with the help of volunteers, we produced data in the real intelligent apartment through activity scenarios. We finally reproduced the volunteers' activities, via an avatar, in the Digital Twin of the apartment.

We were able to see through analysis and use of similarity metrics between time series, that the data generated by the simulator and that generated by the volunteers are relatively similar.

Thanks to this, we were able to generate synthetic sensor logs, which allowed us to train a classification algorithm for ADLs. This algorithm could be validated on the real data created by the volunteers. We were able to demonstrate that the algorithm was able to recognize activities on real data with good performances (around 80% on average according to the F1-score metric).

However, the experiments were performed on a small amount of data. It would be necessary to validate the results through a larger amount of data. But also from data generated with residents living perpetually in the environment in question. We can nevertheless conclude that the use of a Digital Twin to generate data similar to the target environment, in order to train an activity recognition algorithm seems promising.

CONCLUSIONS AND PERSPECTIVES

Conclusion and Contributions

Today, due to the reduction of manufacturing costs, the progress of sensors and communication systems, our environment and our daily life is increasingly equipped with sensors and intelligent interconnected objects. This technology allows our environment to become more intelligent and capable of providing multiple new services. The smart home is at the center of attention for the many innovative applications and services it can offer in terms of security, energy saving, comfort improvement and health support.

The last few years have seen the emergence of a multitude of techniques and approaches through artificial intelligence to equip homes with the cornerstone to all these services; the ability to understand the lifestyle and activities of its residents through home automation sensors.

Despite advances in deep learning and the increase in the amount of data, these methods do not allow for sufficient generalization and full treatment of the complexity and variability of human activity. This observation allowed us to question these limitations and to try to answer them in order to propose new ways of improvement and possible research directions. We have therefore tried to answer the following questions:

•

How to efficiently encode, represent, and extract information, features, and patterns from non-intrusive home automation sensors of different natures to perform recognition of the ADLs?

•

How can we help recognition algorithms to better understand the sequences of sensor activations and sensor relatedness to separate the sequences and better understand captured sensor traces?

• Can we transfer the knowledge gained from one house to another house?

•

How could we apply activity recognition methods to an actual real house?

Part, Chapter 8 -Conclusion

How to efficiently encode, represent, and extract information, features, and patterns from non-intrusive home automation sensors of different natures to perform recognition of the ADLs?

The actual human activity recognition approaches have difficulties because home automation sensors are voluntarily simple and not very rich in order to information to respect the privacy of the resident(s) and to save energy. The values they provide can also be of very different nature (binary, scalar, states, etc.). Moreover, the time series generated by these sensors are event-driven because they are triggered by the actions of the residents or the environment. This makes these time series not sampled in a regular way, requiring a particular treatment.

For this purpose, we proposed to treat these time series as a single sequence of events and to be inspired by methods of representation and sequence processing such as those used in the field of Natural Language Processing. We used embedding methods that allowed us to obtain rich representations of these events and their relations. We have tried to couple this representation with a time series classification method. We have compared a LSTM approach, designed to analyse sequential data and which have demonstrated exemplary performance, against the FCN algorithm, which has also shown good performance for classical time series classification.

Despite the ability of FCN to extract features and patterns, our results (cf. Chapter 4) have allowed us to demonstrate: 1) that the sequential representation and the use of an embedding allows to improve the classification performances of the models 2) that the problem of activity recognition is not only a pattern recognition problem and requires in addition to interpret temporal and potentially contextual dependencies. Indeed, the patterns in the activity sequences are not exactly similar between sequences belonging to the same label due to sampling in the form of events. This form of sampling imposed by the rhythm of life of the habitats distorts the patterns because the events do not appear exactly with the same temporal spacing or regularity.

How can we help recognition algorithms to better understand the sequences of sensor activations and sensor relatedness to separate the sequences and better understand captured sensor traces?

Human beings do not always live alone. Because of actions done by several residents in parallel, the activation trace of a sensor corresponding to an activity can be mixed with another trace.

By interpreting that certain orders of actions are possible or not, it is easier to distinguish what each resident is doing. For example, let us consider two residents, each one performing a different activity. The first one cooks while the second has a shower. In this case, the sensors of the house could potentially capture the actions of each resident in the following order: presence in the kitchen, presence in the bathroom, the fridge door open, the bathroom light on, the kitchen faucet, the hotplate on, the shower faucet on , the kitchen faucet off, etc. It is therefore useful

Limitations and Perspectives

chance of appearing in the vocabulary of dataset "B".

If this same experiment were performed between two datasets using different naming conventions, the transfer learning would need another step, indeed each of the two houses has different "vocabularies". To echo the NLP domain, it would be like trying to transfer the knowledge learned on an English corpus to a French corpus.

Moreover, the proposed encoding lacks of universality. Even in the case where the "vocabulary" is similar, the name or ID of a motion sensor located in the kitchen for dataset "A" can designate a completely different sensor in dataset "B", a motion sensor in the bathroom for example. Even if we have observed during our experimentation that this limitation does not seem to have a strong impact, being able to keep the "sense" and the context of activation of a sensor learned in one dataset, could allow to guarantee better performance in the transfer of knowledge to another dataset. At least two solutions could be considered to overcome these limitations.

First, by creating a universal vocabulary so that a sensor, such as a motion sensor in the kitchen, always has the same name, or naming strategy, regardless of the dataset.

Secondly, it would be possible to draw inspiration from translation models and methods based on Sequence to Sequence (Seq2Seq) models [11], [2]. Intuitively, considering that each house has its own "language", it might be possible to translate the activity sequences of house A into the "language" of house B.

Transformers-Based Embedding

In this work, we used the embedding methods Word2Vec and ELMo to demonstrate the feasibility and interest of transferring these methods from the NLP field to the domain of human activity recognition in smart homes. However, new methods based on a structure called Transformers [14] have emerged in recent years in the NLP domain. Thanks to their ability to capture distant dependencies, in theory infinitely, and to focus attention on important elements in sequences, methods based on Transformers have become state of the art. They have for the same reason become state of the art and relevant solutions in other domains such as Music Generation [4] and Music Classification [12], which also deal with sequence problems.

Transformers could be a new model to exploit for the human activity recognition in smart homes, in particular by using embedding methods today state of the art, such as GPT [14] and BERT [4]. GPT would allow a better interpretation of the long and very long term dependencies of the sensors, while BERT would allow to reinforce the understanding of the local context (internal to the sequence of activity) but also inter-sequences of activity relations context.

Indeed, BERT is trained in parallel according to two tasks: the Masked Language Model 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.92 0.00 0.00 0.08 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.09 0.88 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.50 0.00 0.26 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.74 ARUBA Liciotti Bi-LSTM K=1 0.98 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.93 0.00 0.00 0.06 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.66 0.33 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.09 0.89 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.50 0.00 0.24 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.73 ARUBA ELMo Bi-LSTM K=1 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.91 0.00 0.00 0.08 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.80 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.13 0.87 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.27 0.00 0.00 0.00 0.08 0.01 0.00 0.00 0.64 ARUBA Liciotti Bi-LSTM K=2 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.01 0.00 0.00 0.94 0.00 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.77 0.21 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 0.87 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.27 0.00 0.00 0.00 0.09 0.00 0.00 0.00 0.64 ARUBA ELMo Bi-LSTM K=2 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.95 0.00 0.00 0.05 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.64 0.36 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.06 0.93 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.50 0.00 0.19 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.80 ARUBA Liciotti Bi-LSTM K=3 0.98 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.00 0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.62 0.38 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 0.96 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.50 0.00 0.19 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.80 ARUBA ELMo Bi-LSTM K=3 0.95 0.02 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.60 0.40 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.91 0.00 0.00 0.08 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.01 0.00 0.00 0.00 0.00 0.01 0.00 0.03 0.00 0.00 0.94 0.02 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.09 0.91 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.13 0.00 0.87 0.00 0.00 0.00 0.00 0.05 0.00 0.00 0.00 0.00 0.00 0.95 0.00 0.00 0.00 0.07 0.00 0.00 0.21 0.03 0.03 0.00 0.66 MILAN 0.96 0.01 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.93 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.91 0.00 0.00 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.22 0.00 0.78 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.96 0.04 0.00 0.00 0.00 0.00 0.01 0.00 0.05 0.00 0.00 0.92 0.02 0.01 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.12 0.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.09 0.00 0.91 0.00 0.00 0.00 0.00 0.25 0.00 0.00 0.05 0.00 0.00 0.70 0.00 0.00 0.00 0.07 0.00 0.00 0.21 0.03 0.03 0.00 0.66 MILAN ELMo Bi-LSTM K=1 0.94 0.02 0.00 0.00 0.00 0.03 0.00 0.00 0.00 0.00 0.28 0.72 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.89 0.00 0.00 0.11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.96 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.10 0.90 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.15 0.00 0.85 0.00 0.00 0.00 0.00 0.20 0.00 0.00 0.05 0.00 0.00 0.75 0.00 0.00 0.00 0.03 0.00 0.00 0.48 0.00 0.00 0.00 0.48 MILAN 0.99 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.59 0.41 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.87 0.00 0.00 0.12 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.02 0.00 0.00 0.94 0.02 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.10 0.90 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.13 0.00 0.87 0.00 0.00 0.05 0.00 0.10 0.00 0.00 0.00 0.00 0.00 0.85 0.00 0.00 0.00 0.03 0.00 0.00 0. 0.93 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.38 0.59 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.85 0.00 0.00 0.13 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.96 0.04 0.00 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.94 0.01 0.01 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.06 0.94 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17 0.00 0.81 0.00 0.02 0.00 0.00 0.15 0.00 0.00 0.05 0.00 0.00 0.80 0.00 0.00 0.00 0.04 0.00 0.00 0.25 0.00 0.11 0.00 0.61 MILAN 0.92 0.07 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.38 0.59 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.00 0.00 0.93 0.00 0.00 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.03 0.00 0.00 0.00 0.00 0.01 0.00 0.04 0.00 0.00 0.93 0.02 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.01 0.06 0.91 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.06 0.00 0.94 0.00 0.00 0.00 0.00 0.20 0.00 0.00 0.05 0.00 0.00 0.75 0.00 0.07 0.00 0.04 0.00 0.00 0.25 0.00 0.07 0.00 0.57 MILAN ELMo Bi-LSTM K=3 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.71 0.00 0.29 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.91 0.00 0.00 0.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.10 0.00 0.87 0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.98 ARUBA (original activity labels) Liciotti Bi-LSTM K=1 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.71 0.00 0.29 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.91 0.00 0.00 0.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.08 0.00 0.90 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.97 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.98 ARUBA (original activity labels) ELMo Bi-LSTM K=1 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.93 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.69 0.00 0.31 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.09 0.09 0.45 0.00 0.00 0.27 0.09 0.00 0.00 0.00 0.00 0.00 0.00 0.06 0.00 0.94 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 ARUBA (original activity labels) Liciotti Bi-LSTM K=2 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.61 0.00 0.39 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.73 0.00 0.00 0.27 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.04 0.00 0.96 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.01 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.50 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.91 0.00 0.00 0.00 0.00 0.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 ARUBA (original activity labels) ELMo Bi-LSTM K=2 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.84 0.00 0.00 0.00 0.00 0.16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.75 0.00 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 0.00 0.87 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 ARUBA (original activity labels) Liciotti Bi-LSTM K=3 0.98 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.94 0.00 0.00 0.00 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.68 0.00 0.31 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.10 0.00 0.88 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 ARUBA (original activity labels) ELMo Bi-LSTM K=3 0.45 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.45 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.10 0.00 0.00 0.00 0.00 0.10 0.00 0.00 0.10 0.00 0.00 0.60 0.00 0.00 0.10 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.89 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.91 0.00 0.00 0.00 0.00 0.02 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.00 0.09 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.87 0.01 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.87 0.00 0.00 0.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.14 0.00 0.00 0.00 0.00 0.86 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.04 0.00 0.00 0.03 0.00 0.00 0.89 0.00 0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.16 0.84 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.09 0.00 0.00 0.08 0.00 0.83 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.12 0.00 0.00 0.86 MILAN (original activity labels) Liciotti Bi-LSTM K=1 0.28 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.59 0.00 0.00 0.00 0.03 0.00 0.03 0.00 0.00 0.10 0.00 0.00 0.00 0.00 0.10 0.00 0.00 0.00 0.00 0.00 0.70 0.00 0.00 0.10 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.78 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.14 0.00 0.00 0.00 0.00 0.00 0.71 0.14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.00 0.00 0.00 0.01 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.92 0.00 0.00 0.00 0.00 0.01 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.00 0.05 0.00 0.00 0.00 0.00 0.01 0.01 0.00 0.90 0.02 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.84 0.00 0.00 0.15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.14 0.00 0.21 0.00 0.00 0.00 0.00 0.64 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.02 0.00 0.00 0.91 0.01 0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.11 0.87 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.09 0.00 0.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.27 0.00 0.00 0.71 MILAN (original activity labels) ELMo Bi-LSTM K=1 0.57 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 0.00 0.00 0.09 0.00 0.00 0.55 0.00 0.18 0.00 0.00 0.00 0.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.22 0.00 0.00 0.11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.96 0.01 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.91 0.00 0.00 0.00 0.00 0.00 0.08 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.93 0.00 0.00 0.00 0.00 0.06 0.00 0.00 0.00 0.10 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.90 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.72 0.00 0.00 0.28 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.23 0.00 0.00 0.00 0.00 0.69 0.08 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.03 0.01 0.00 0.02 0.00 0.00 0.91 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.07 0.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.19 0.00 0.79 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.24 0.00 0.00 0.76 MILAN (original activity labels) Liciotti Bi-LSTM K=2 0.43 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 0.00 0.00 0.09 0.00 0.00 0.55 0.00 0.18 0.00 0.00 0.00 0.78 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17 0.00 0.00 0.06 0.00 0.00 0.00 0.44 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.56 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.96 0.00 0.00 0.00 0.00 0.00 0.00 0.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.91 0.00 0.00 0.00 0.00 0.00 0.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.96 0.00 0.00 0.00 0.00 0.04 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.75 0.00 0.00 0.21 0.00 0.03 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.23 0.00 0.00 0.00 0.00 0.69 0.08 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.03 0.00 0.00 0.91 0.01 0.01 0.01 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05 0.94 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.06 0.00 0.92 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 0.00 0.00 0.80 MILAN (original activity labels) ELMo Bi-LSTM K=2 True label 0.28 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.66 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.09 0.00 0.00 0.18 0.00 0.00 0.55 0.00 0.18 0.00 0.00 0.00 0.94 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.83 0.17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.89 0.00 0.00 0.00 0.00 0.00 0.11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.00 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.92 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.74 0.00 0.00 0.16 0.00 0.09 0.00 0.00 0.00 0.00 0.00 0.00 0.17 0.00 0.00 0.00 0.00 0.67 0.00 0.17 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.07 0.00 0.21 0.00 0.00 0.00 0.00 0.71 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.01 0.00 0.00 0.94 0.00 0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.04 0.95 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.06 0.00 0.94 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.22 0.00 0.00 0.76 MILAN (original activity labels) Liciotti Bi-LSTM K=3 True label 0.31 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.62 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.09 0.09 0.00 0.00 0.27 0.00 0.00 0.45 0.09 0.00 0.00 0.00 0.00 0.94 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17 0.00 0.00 0.00 0.00 0.83 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.97 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.01 0.86 0.00 0.00 0.00 0.00 0.00 0.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.03 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.96 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.78 0.00 0.00 0.22 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17 0.00 0.00 0.00 0.83 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.21 0.00 0.21 0.00 0.00 0.00 0.00 0.50 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.03 0.00 0.00 0.01 0.00 0.00 0.94 0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.01 0.00 0.00 0.00 0.00 0.03 0.94 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.06 0.00 0.00 0.08 0.00 0.85 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.22 0.00 0.00 0.78 MILAN (original activity labels) ELMo Bi-LSTM K=3 True label 0.40 0.00 0.00 0.00 0.00 0.00 0.20 0.00 0.30 0.00 0.00 0.10 0.00 0.00 0.00 0.38 0.19 0.00 0.00 0.12 0.00 0.25 0.00 0.00 0.00 0.00 0.06 0.00 0.00 0.29 0.36 0.00 0.00 0.07 0.00 0.29 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.42 0.17 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.08 0.00 0.18 0.00 0.00 0.00 0.00 0.00 0.73 0.09 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.01 0.01 0.01 0.91 0.01 0.02 0.01 0.00 0.02 0.01 0.17 0.00 0.00 0.00 0.00 0.00 0.39 0.06 0.33 0.00 0.00 0.06 0.00 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.06 0.06 0.72 0.00 0.00 0.00 0.11 0.00 0.00 0.06 0.00 0.00 0.00 0.00 0.25 0.00 0.00 0.69 0.00 0.00 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.11 0.06 0.00 0.00 0.67 0.00 0.11 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.13 0.00 0.00 0.00 0.00 0.80 0.00 0.05 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05 0.00 0.00 0.00 0.90 CAIRO (original activity labels) Liciotti Bi-LSTM K=1 True label 0.60 0.00 0.00 0.00 0.00 0.00 0.10 0.00 0.20 0.00 0.00 0.10 0.00 0.00 0.00 0.25 0.12 0.00 0.00 0.38 0.00 0.19 0.00 0.00 0.00 0.00 0.06 0.00 0.00 0.21 0.64 0.00 0.00 0.00 0.00 0.14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.25 0.25 0.00 0.00 0.17 0.00 0.25 0.00 0.00 0.00 0.00 0.08 0.00 0.09 0.00 0.00 0.00 0.00 0.00 0.50 0.18 0.09 0.00 0.00 0.14 0.00 0.00 0.00 0.00 0.01 0.00 0.01 0.01 0.00 0.94 0.02 0.01 0.00 0.01 0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.17 0.11 0.72 0.00 0.00 0.00 0.00 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.06 0.06 0.00 0.78 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.25 0.00 0.00 0.75 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.06 0.11 0.06 0.00 0.72 0.00 0.06 0.00 0.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05 0.00 0.00 0.00 0.05 0.00 0.90 CAIRO (original activity labels) ELMo Bi-LSTM K=1 True label 0.30 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.60 0.00 0.00 0.00 0.00 0.10 0.00 0.59 0.18 0.00 0.00 0.06 0.00 0.12 0.00 0.00 0.00 0.00 0.06 0.00 0.00 0.29 0.29 0.00 0.00 0.07 0.00 0.29 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.96 0.00 0.00 0.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.17 0.00 0.00 0.17 0.00 0.25 0.00 0.00 0.00 0.00 0.08 0.00 0.04 0.00 0.00 0.00 0.00 0.00 0.74 0.00 0.17 0.04 0.00 0.00 0.00 0.00 0.00 0.01 0.00 0.00 0.00 0.01 0.01 0.90 0.01 0.04 0.01 0.00 0.00 0.03 0.06 0.00 0.00 0.00 0.00 0.00 0.06 0.00 0.83 0.00 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.12 0.00 0.76 0.00 0.00 0.00 0.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.27 0.00 0.00 0.73 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 0.11 0.17 0.06 0.00 0.50 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.00 0.00 0.86 0.07 0.05 0.00 0.00 0.00 0.00 0.00 0.00 0.23 0.00 0.09 0.00 0.05 0.00 0.59 CAIRO (original activity labels) Liciotti Bi-LSTM K=2 True label 0.70 0.00 0.00 0.00 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.10 0.00 0.65 0.06 0.00 0.00 0.12 0.00 0.12 0.00 0.00 0.00 0.00 0.06 0.00 0.00 0.00 0.43 0.00 0.00 0.14 0.00 0.36 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.96 0.00 0.00 0.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.17 0.17 0.00 0.00 0.42 0.00 0.17 0.00 0.00 0.00 0.00 0.08 0.00 0.04 0.00 0.00 0.00 0.00 0.00 0.83 0.00 0.09 0.00 0.00 0.04 0.00 0.00 0.01 0.01 0.00 0.00 0.00 0.02 0.01 0.94 0.01 0.00 0.00 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.11 0.00 0.78 0.06 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 0.00 0.76 0.00 0.00 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.60 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.11 0.06 0.00 0.00 0.00 0.72 0.00 0.11 0.00 0.14 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.79 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.27 0.05 0.00 0.00 0.09 0.00 0.59 CAIRO (original activity labels) ELMo Bi-LSTM K=2 True label 0.10 0.00 0.00 0.00 0.00 0.00 0.30 0.00 0.20 0.00 0.00 0.00 0.00 0.40 0.00 0.88 0.06 0.00 0.00 0.00 0.00 0.06 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.79 0.07 0.00 0.00 0.00 0.00 0.14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.62 0.15 0.00 0.00 0.00 0.00 0.15 0.00 0.00 0.00 0.00 0.08 0.00 0.05 0.00 0.00 0.00 0.00 0.00 0.77 0.00 0.18 0.00 0.00 0.00 0.00 0.00 0.00 0.01 0.01 0.00 0.01 0.00 0.02 0.87 0.02 0.03 0.02 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.16 0.16 0.63 0.00 0.00 0.05 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.06 0.00 0.83 0.00 0.00 0.00 0.11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.47 0.00 0.00 0.53 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.24 0.18 0.00 0.00 0.59 0.00 0.00 0.00 0.27 0.00 0.00 0.00 0.07 0.00 0.00 0.00 0.00 0.00 0.07 0.60 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05 0.19 0.00 0.00 0.00 0.00 0.00 0.76 CAIRO (original activity labels) Liciotti Bi-LSTM K=3 True label 0.50 0.00 0.00 0.00 0.00 0.00 0.30 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.00 0.62 0.19 0.00 0.00 0.06 0.00 0.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.36 0.43 0.00 0.00 0.07 0.00 0.14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.46 0.31 0.00 0.00 0.15 0.00 0.08 0.00 0.00 0.00 0.00 0.00 0.00 0.05 0.00 0.00 0.00 0.00 0.00 0.77 0.14 0.05 0.00 0.00 0.00 0.00 0.00 0.01 0.01 0.00 0.00 0.01 0.00 0.00 0.94 0.02 0.01 0.00 0.01 0.00 0.02 0.05 0.00 0.00 0.00 0.00 0.00 0.16 0.16 0.47 0.00 0.00 0.16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.06 0.00 0.94 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.47 0.00 0.00 0.47 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.18 0.06 0.00 0.00 0.71 0.00 0.06 0.00 0.07 0.00 0.00 0.00 0.07 0.00 0.00 0.00 0.00 0.00 0.00 0.87 0.00 0.05 0.00 0.00 0.00 0.00 0.00 0.05 0.14 0.00 0.00 0.00 0.00 0.00 0.76 CAIRO (original activity labels) ELMo Bi-LSTM K=3

EXPERIMENTS AND MODELS HYPERPARAMETERS

SENSOR2VEC EMBEDDING VIZUALISATION