Le script est une structure qui décrit une séquence stéréotypée d'événements ou d'actions survenant dans notre vie quotidienne. Les histoires utilisent des scripts, avec une ou plusieurs déviations intéressantes, qui nous permettent de mieux saisir les situations quotidiennes rapportées et les faits saillants du récit. Ainsi, la notion de script est très utile dans de nombreuses applications d'intelligence ambiante telles que la surveillance de la santé et les services d'urgence. Ces dernières années, l'avancement des technologies de détection et des systèmes intégrés permettent aux systèmes de santé de collecter en permanence les activités des êtres humains, en intégrant des capteurs dans des dispositifs portables (par exemple smart-phone ou smart-watch). La reconnaissance de l'activité humaine (HAR) a ainsi connue un essor important grâce notamment à des approches d'apprentissage automatique telles que le réseau neuronal ou le réseau bayésien.

Ces avancées ouvre des perspectives qui vont au delà de la simple reconnaissance d'activités. Ce démarche proposée dans la dernière section. Elle prend appui sur les 2 domaines de l'apprentissage automatique et du traitement du langage naturel.

Résumé du Chapter 4

Ce chapitre présente le cadre de notre proposition pour traiter les données déséquilibrées provenant des données de capteurs de téléphones mobiles. Nous donnons une introduction sur la question du déséquilibre dans la première section, puis nous présentons les travaux connexes qui traitent de ce problème. Ensuite, dans la section 4.3 nous présentons notre nouveau cadre, qui utilise une technique d'apprentissage actif et de suréchantillonnage pour résoudre les problèmes de données de déséquilibre, basé sur le perceptron Multicouche. Ensuite, les résultats de l'expérience sont illustrés dans la section 4.4 sur 2 jeux de données comportant des données audio et d'accéléromètre, toutes 2 comportant des données déséquilibrées. L'apport de notre méthode est clairement démontré. Le chapitre se termine par une conclusion.

Résumé du Chapter 5

Dans ce chapitre, nous présentons notre nouvelle approche pour générer des scripts à partir d'événements, qui sont détectés à l'aide d'une méthode d'apprentissage en profondeur sur des données de capteurs portables. Après une introduction en section 5.1, les travaux associés sont présentés dans la section 5.2. La méthode proposée est détaillée dans la section 5.3. Elle comporte 2 étapes: génération de concepts sémantiques à partir des données capteur, génération de scripts à partir des concepts. Les concepts sémantiques sont définis comme des triplets associant geste, activité, contexte. Ils sont générés via des réseaux de neurones profonds et ordonnés temporellement.

La génération de scripts s'effectue via une méthode hybride associant une phase de détection à base de patrons et une phase de traduction de ce "langage source" sous la forme de phrases en langage cible (scripts).

Les résultats expérimentaux sont présentés dans la section 5.4, sur un ensemble de données d'activités journalières impliquant plusieurs sujets. Les performances des 2 phases de traitement sont évaluées séparément. Le chapitre se termine par une brève discussion et un aperçu des travaux futurs dans la section 5.5.

Résumé du Chapter 6

Nous présentons dans ce chapitre, une approche pour apprendre des scripts à partir de textes en langage naturel. Dans la section 6.1, nous présentons une introduction sur la notion de script et son intérêt pour la reconnaissance d'activités humaines. Nous proposons un bref panorama des travaux dans le domaine de l'apprentissage de script et donnons les bases de notre contribution.

La section 6.2 est dédiée à la présentation des travaux connexes: modèles de représentation des événements, en particulier multi-arguments et compositionnels, modélisation et apprentissage des séquences d'événements. Dans la section 6.3, nous détaillons les méthodes proposées et présentons notre modèle NESS, fondé sur l'exploitation des réseaux de neurones récursifs et une représentation des événements comme composition de leurs arguments sémantiques. Les résultats expérimentaux sont présentés dans la section 6.4. Nous comparons les performances de notre approche avec celles d'autres méthodes de la littérature sur une base de données publique et montrons son apport avant de conclure en section 6.5.

Résumé du Chapter 7

Ce chapitre est dédié à la présentation de nos conclusions et perspectives. Il rassemble nos principales propositions dans les différents domaines scientifiques que nous avons couvert: reconnaissance de l'activité humaine, génération et apprentissage de scripts via les traitements du langage naturel.

Nous revenons sur la variété et la nouveauté des outils et modèles utilisés et insistons sur leur complémentarité pour faire face aux difficultés (i) du monde numérique des données capteur et (ii) du monde sémantique de la représentation de l'activité en langue naturelle. Nous revenons sur le rôle pivot de la notion de script et sur l'apport de l'apprentissage profond pour affronter ces difficultés.

Nous abordons ensuite les limitations de ce travail: pauvreté des données, caractère répétitifs des événements, manque de représentativité des activités, manque de richesse des ressources langagières. Les principales perspectives concernent l'introduction d'autres données, comme l'émotion ou la physiologie. Enfin, nous rappelons l'apport de cette thèse à différentes applications du monde réel.

permettant de les structurer et de les transformer sous une forme clairement accessible (par exemple, une histoire ou des scripts). Nous insistons aussi dans ce chapitre sur le rôle joué par l'apprentissage automatique dans ce processus qui met en jeu une chaîne complexe de traitement, et sur les limites des techniques conventionnelles, centrées sur une problématique d'abstraction des données. Par l'apprentissage profond, nous centrons notre attention sur les corrélations entre informations et éléments cachés de contexte, au sein et entre niveaux d'abstraction. Trois enjeux majeurs sont finalement identifiés: prise en compte du déséquilibre des données pour la reconnaissance d'événements, modèles d'apprentissage profonds pour la génération flexible de concepts sémantiques et de scripts à partir d'événements, apprentissage de scripts à partir de textes en langage naturel capables d'exploiter l'information syntaxique et sémantique sur le contexte textuel des événements. Enfin, la dernière section présente un aperçu des différents chapitres de la thèse.

Résumé du Chapter 2

Ce chapitre présente l'état de l'art en le structurant en deux grandes sections : (i) transformation de données en textes et génération de scripts à partir de données, (ii) reconnaissance de l'activité humaine à partir de données ambiantes. Dans la première section, nous présentons tout d'abord quelques grandes tendances de la transformation de données en texte. Dans ces méthodes, centrées sur la génération automatique de textes en langue naturelle (NLG), l'accent est peu mis sur la problématique de l'apprentissage automatique. Nous discutons ensuite la problématique de la génération de scripts utilisant le traitement de langage naturel (NLP) et les méthodes conventionnelles d'apprentissage automatique mises en jeu. Les aspects liés à la modélisation des événements et de leur structure temporelle sont particulièrement mis en avant. On notera néanmoins que ces travaux s'appliquent non pas à des données ambiantes mais à des textes en langue naturelle.

La section suivante du chapitre présente l'état de l'art de la reconnaissance de l'activité humaine (HAR) à partir de données de capteurs portables en utilisant l'apprentissage automatique. Premièrement, nous introduisons la problématique de la reconnaissance de l'activité humaine. Nous présentons ensuite les méthodes pour réaliser la reconnaissance de l'activité, en insistant d'abord sur les questions de l'extraction de caractéristiques et des métriques pour l'évaluation de ces méthodes. Nous consacrons ensuite deux sections à la problématique de l'apprentissage automatique, la première dédiée aux techniques conventionnelles, la seconde aux techniques de l'apprentissage profond.

Résumé du Chapter 3

Ce court chapitre donne un aperçu de la problématique abordée dans la thèse, de ses dimensions, et de la démarche proposée. La problématique est présentée comme associant une dimension d'abstraction des données et une dimension cognitive de communication. L'abstraction des données met en jeu la qualité des données (introduction de la notion de déséquilibre), la prise en compte du contexte (what, when and where) mais aussi celle de la traduction entre univers de discours (du numérique au langagier) différents. La communication des données d'activité met en jeu la notion de script véhiculant une information temporelle. Elle implique la prise en compte d'une dimension linguistique. Des questions difficiles d'apprentissage sont impliquées dans les 2 dimensions et nous mettons en avant l'apport des techniques récentes de l'apprentissage profond. Nous présentons la

Abstract

Script is a structure describes an appropriate sequence of events or actions in our daily life. A story has invoked a script with one or more interesting deviations, which allows us to deeper understand what was happened in the routine behavior of our daily life. Therefore, it is essential in many ambient intelligence applications such as healthmonitoring and emergency services. Fortunately, in recent years, with the advancement of sensing technologies and embedded systems, which make the health-care system possible to collect activities of human beings continuously, by integrating sensors into wearable devices (e.g., smart-phone, smart-watch, etc.). Hence, human activity recognition (HAR) has become a hot topic interest in research over the past decades. In order to do HAR, most researches used machine learning approaches such as Neural networks, Bayesian networks, etc. Therefore, the ultimate goal of our thesis is to generate such kind of stories or scripts from activity data of wearable sensors using a machine learning approach.

However, to the best of our knowledge, it is not a trivial task due to the very limitation of information on wearable sensors activity data. Hence, there is still no approach to generate script/story using machine learning, even though many machine learning approaches were proposed for HAR in recent years (e.g., convolutional neural network, deep neural network, etc.) to enhance the activity recognition accuracy.

In order to achieve our goal, first of all in this thesis we proposed a novel framework, which solved for the problem of imbalanced data, based on active learning combined with oversampling technique so as to enhance the recognition accuracy of conventional machine learning models i.e., Multilayer Perceptron. Secondly, we introduce a novel scheme to automatically generate scripts from wearable sensor human activity data using deep learning models, evaluate the generated method performance. Finally, we proposed a neural event embedding approach that is able to benefit from semantic and syntactic information about the textual context of events. The approach is able to learn the stereotypical order of events from sets of narrative describing typical situations of everyday life. [START_REF] Regneri | Learning script knowledge with web experiments[END_REF] after pre-processing by using TimeML framework and our own preprocessing library. . . . . . If the rise of big data has opened many opportunities, making sense of this mass of information has become one of the major challenges of this century. In this thesis, we are focusing on an original approach to deal with a subset of big data, namely personal ambient data (e.g., smartphones, GPS, wearable sensors, etc.), by presenting the collected data in the form of a personal story. Indeed, human has developed cognitive capabilities specifically tuned to tell and understand the information in the form of stories [START_REF] Reiter | Building Natural Language Generation Systems[END_REF]; [START_REF] Portet | Automatic generation of textual summaries from neonatal intensive care data[END_REF]; [START_REF] Hunter | Automatic generation of natural language nursing shift summaries in neonatal intensive care: Btnurse[END_REF]].

Excerpt of the dataset provided by

List of Tables

Automatic story generation (SG) has been studied in the domain of Natural Language Generation (NLG). NLG is concerned with the automatic generation of written texts from temporal data to support various domains of daily life such as: the generation of weather report from weather simulations [START_REF] Reiter | Building Natural Language Generation Systems[END_REF]; [START_REF] Reiter | Choosing words in computer-generated weather forecasts[END_REF]]; helping the patients understand their complex medical data [START_REF] Portet | Automatic generation of textual summaries from neonatal intensive care data[END_REF]; [START_REF] Hunter | Automatic generation of natural language nursing shift summaries in neonatal intensive care: Btnurse[END_REF]]; or supporting social communication for disable people [START_REF] Williams | Generating basic skills reports for low-skilled readers*[END_REF]. Many story generators have been proposed [START_REF] Andersen | Automatic extraction of facts from press releases to generate news stories[END_REF]; [START_REF] Jacobs | Scisor: Extracting information from on-line news[END_REF]; [START_REF] Poibeau | Multi-source, Multilingual Information Extraction and Summarization[END_REF]]. However, most of these approaches were not interested in generating a story from real data but rather from virtual environments where all the information is complete and known. Although NLG has a long history of generative narratives from data or knowledge, most of these narratives do not concern personal ambient data. In Ambient Intelligence, data is captured by the sensor and thus filled with uncertainty, incomplete and very low level (e.g., the measure of the skin temperature but not whether the person feels too warm or cold). Therefore, our ultimate goal is to generate automatically stories from ambient data, and the Ph.D. subject is focused on Story Generation from Smart-phone Data. As will be described in this thesis, story generation from smart-phone data concern the definition of a method to extract meaning information from real sensor At 10:00, she sat on a sofa in the living room and took a coffee with crackers because she was not fully awake.

After that, she turned on television and watched a musical game-show, while her mother was cleaning up the floor. situation in which the data is collected. The structured information must be organized to respect the conventional human-understandable stories and such a story must be expressed via a textual representation.

Being able to solve such problems involve many challenges. First, each of the above steps are related to different research domains (data analysis, knowledge reasoning, Natural Language Generation) and are generally addressed as a pipeline of processing steps (reprocessing of data, segmentation, event extraction, script recognition, textual generation). In the state of the art, such processing is performed using a model acquired through machine learning, however, this application suffers from the sparsity of information. Indeed, if it is possible to find data-set related to smartphones and activity, it is extremely hard to find a data-set containing both annotated smartphone data and their representation textual stories.

Second, there is a wide semantic gap between the raw signal and the final story. To deal with this gap, each step has to consider the overall objective of the task of story generation. This is very difficult to perform in a pipeline of successive steps. Furthermore, with such a chain of abstraction steps, each optimized separately, hidden relationships between information across the abstraction levels that may not be captured. Current methods still rely on a pipeline approach and lack to consider the correlation between information inside and across the various abstraction levels. Going further in these directions raises still unsolved issues in modeling and learning. We explore some of them in this thesis, based on the recent deep learning approaches.

In this thesis, we present the work performed to move toward story generation from sensor data by using deep learning neural networks. In particular, the thesis work was focused on Human Activity Recognition (HAR) from sensors, Concept-to-Text generation from abstract events, and scripts learning to extract a typical sequence of events (e.g., sub-stories) that can be used to model the structure of a story. Hence, this approach is seen as a bridge that would allow connecting between the domain of HAR, scripts modeling and NLG in the NLP domain.

Research Overview and Contributions

In this thesis, Human Activity Recognition (HAR) is the first target of the research orientation. In HAR, several issues were addressed in [START_REF] He | Learning from imbalanced data[END_REF]; Lara et al. (2012)] such as imbalanced data, flexibility, etc. Second, the HAR system only allows one to recognize discrete human daily activities, hence it is not sufficient to build or generate a story as described in [START_REF] Miyanishi | Generating an event timeline about daily activities from a semantic concept stream[END_REF], which is defined as a problem in NLG domain. Finally, we would like to predict or infer missing events on scripts modeling from the narrative text, which is a hot topic in NLP [START_REF] Modi | Inscript: Narrative texts annotated with script information[END_REF][START_REF] Frermann | A hierarchical bayesian model for unsupervised induction of script knowledge[END_REF]; [START_REF] Regneri | Learning script knowledge with web experiments[END_REF]]. Therefore, considering all of these problems in different domains, we summarize the contributions of the thesis is the following:

• Dealing with Imbalanced Datasets for Human Activity Recognition using Mobile Phone Sensors: In the recent years, the wide spreading of smart-phones that are daily carried by humans and fit with tens of sensors triggered an intense research activity in human activity recognition (HAR). HAR in smart-phone is seen as essential not only to better understand human behavior in daily life but also for context provision to other applications in smart-phones. Many statistical and logical based models for on-line or off-line HAR have been designed, however, the current trend is to use deep-learning with neural networks. These models need a high amount of data and, as most discriminative models, they are very sensitive to the imbalanced class problem [START_REF] He | Learning from imbalanced data[END_REF]. In this work, we study different ways to deal with imbalanced data sets to improve the accuracy of HAR with neural networks and introduce a new over-sampling method, called Border Limited Link SMOTE (BLL SMOTE) that improves the classification accuracy of Multi-Layer Perceptron (MLP) performances, which is presented in Chapter 4.

• Automatic Scripts Generation from Human Activity Recognition using Wearable Sensors Data: Human activity recognition (HAR) plays an important role in the real world, which allows us to recognize activities of daily living (ADLs) and understand human life. In spite of this task being an active field in the past decade, most existing researches concentrate on predicting individual activity labels from internal and external sensors data. However, our daily life scripts are composed of chronological ADLs with the corresponding locations and higher activities (e.g., shopping in the supermarket after eating at the kitchen and leaving home). Therefore, in this work, we proposed a novel approach that generates scripts from human activity recognition using wearable sensors data. The proposed method combines semantic concepts such as gesture, activity, and location detected by sensors for producing a script with the above three real-world properties. First, we used deep learning neural networks (DNN) to recognize concepts in terms of a sequence (e.g., gesture, activity, and location) from wearable sensors data. Second, we combine these concepts by using manually detection templates in order to glue the concepts in terms of a sentence. Finally, we formulated the natural language generation step as a machine translation approach, by applying the sequence to sequence model Sutskever et al. (2014a) to enrich the target generated sentence from their semantic concepts, which is considered as the source language. This work is presented in Chapter 5.

• Neural Networks Modeling for Semantic Script Induction from Narrative Text: Typical situations of everyday life such as going to a restaurant are pre-encoded common sense knowledge known as Scripts. Learning such scripts from narratives has been studied in Natural Language Processing (NLP). These script models would allow predicting likely next events and would support the natural generation of narratives.

Early work of Script learning focused on word frequency-based methods that suffered from the sparsity problem. Therefore, event embeddings has raised interest to overcome this issue [START_REF] Modi | Event embeddings for semantic script modeling[END_REF]; [START_REF] Modi | Inducing neural models of script knowledge[END_REF]]. In this work, we proposed a neural event embeddings approach that is able to benefit from semantic and syntactic information about the textual context of events. The approach is able to learn the stereotypical order of events from sets of narratives describing typical situations of everyday life. This work is described in Chapter 6.

Thesis Outline

Chapter 2 gives a short overview of the state-of-the-art on both data-to-text generation and script learning and generation. Then, it presents human activity recognition (HAR) from mobile phone sensors, which focuses on the definition of the problem, and the way this task is generally evaluated. It also introduces a review of state-of-the-art on human activity recognition, which describes different methods and works, using for HAR on both machine learning approaches such as conventional and deep learning.

Chapter 3 describes the scientific challenges, the overall approach of the thesis and the design of the framework to generate stories from raw smartphone sensor data.

In Chapter 4 we introduce our novel framework, based on Multilayer Perceptron (MLP), which uses active learning and oversampling technique so that it can solve the problems of imbalanced data.

In Chapter 5, we proposed a novel approach allows us to generate a story in terms of Scripts from human activity recognition using wearable sensor data and evaluation for

Scripts generation.

Chapter 6 presents a novel technique to classify the order of events from the sequence of events or scripts from natural language texts. This technique permits to predict what event happens next from learning ordered of the sequence of events.

Chapter 7 provides an overall conclusion about the work performed during the thesis and discusses the numerous perspectives for further research it opens. Scripts is a structure that describes an appropriate sequence of events in a particular context, was first introduced by [START_REF] Schank | Scripts, plans and knowledge[END_REF]. The structure is an interconnected whole and what in one slot affects to what can be in another. As described in [START_REF] Schank | Scripts, plans and knowledge[END_REF] story is a script with various interesting deviations. Such numerous scripts describe different sequence of events or actions in our daily life such as bus taking script, working day script and laundry script, and so on. For instance, Table 2.1 shows an example of laundry scripts in different scenarios.

The above scripts enable us an insight into the sequence of events happening in the laundry context with different scenarios, which are providing in the form of human-readable texts. For instance, the sequence of events in scenario 2 lets us deduce that the person is washing clothes using a washing machine, or scenario 3 shows us that the person is drying his/her clothes. In general, the overall script would represent the prototypical sequence of events that human would imagine being part of this task. Hence script is used by humans to interpret scenes (she doing the laundry) predict the next event and do common sense reasoning. Since real-life stories are intended to report human activities in a way that is tuned to the human capability of the understanding script, we introduce the state of the art in reasoning with the script and human activity recognition from a computer science perspective.

State of the art of Data-to-text and Script Generation

In the several past decades have witnessed a significant researches on data-to-text generation and script generation. As our literature surveys on data-to-text and script generation, we can divide it into two domains as follows:

For data-to-text using Natural Language Generation, [START_REF] Reiter | Choosing words in computer-generated weather forecasts[END_REF] introduced a SUMTIME-MOUSAM system to automatically generate textual weather forecast by choosing the right word to communicate numeric weather prediction data. The input to SUMTIME-MOUSAM system is numerical weather parameters and the output is the weather forecast. However, the input parameters of SUMTIME-MOUSAM system are produced by simulation. In addition, story generation is not only interested in weather forecast but also studied in health-care such as helping the patients understand their complex medical data in [START_REF] Portet | Automatic generation of textual summaries from neonatal intensive care data[END_REF]; [START_REF] Hunter | Automatic generation of natural language nursing shift summaries in neonatal intensive care: Btnurse[END_REF]] presented BT-45 system which generates textual summaries of about 45 minutes of continuous physiological signal (e.g., heart rate, pressure of oxygen and carbon dioxide in blood, oxygen saturation, mean blood pressure, and peripheral and temperature of the baby) and discrete events (e.g., equipment settings and drug administrations). As a result, an experiment on clinical data of BT-45 system shows that generated texts are inferior to human expert texts in a number of ways. Likewise, [START_REF] Williams | Generating basic skills reports for low-skilled readers*[END_REF] proposed the SKILLSUM system that generates personalized feedback reports for people with limited reading skills. In order to do this, there are two approaches to the SKILLSUM system which the first approach is to determine the content and structure (document-planning) by using pilot experiments and interview with domain experts. The second approach is choosing linguistic expressions (micro-planning) by using constraints based on corpus analysis and preference rules. The evaluation of the SKILLSUM system illustrated that the generated text was more effective than canned text at enhancing users' knowledge of their skills. Moreover, many story generators have been proposed [START_REF] Andersen | Automatic extraction of facts from press releases to generate news stories[END_REF]; [START_REF] Jacobs | Scisor: Extracting information from on-line news[END_REF]; [START_REF] Poibeau | Multi-source, Multilingual Information Extraction and Summarization[END_REF]]. [START_REF] Andersen | Automatic extraction of facts from press releases to generate news stories[END_REF] introduced JASPER which is a fact extraction system.

The system used a template-driven approach, partial understanding technique and heuristic procedures to extract certain important pieces of information from a limited range of text.

Nevertheless, these approaches were not interested in generating a story from real data but rather from virtual environments where all the information is complete and known. In more recent research, [START_REF] Miranda | Génération de récits à partir de données ambiantes[END_REF] works on story generation from real sensor data, which allows generating a sequence of human activities during their ski-touring. In order to generate the sequence of activities, the proposed system used signal segmentation with threshold from altitude, which was collected from the global positioning system (GPS) of the smart-phone to extract information and knowledge about human activities such as moving, standing as well as the interest location. Then, it is mapped into an ontology tool with data-to-text processes (e.g., micro-planning, etc.) to generate the story. However, different from this work, our thesis only focuses on the machine learning approach for recognizing human activities.

For script generation, early works conducted in the 1970s defined a script as a sequence of structured events organized in temporal order [START_REF] Schank | Scripts, plans and knowledge[END_REF].

Scripts learning was revived by [START_REF] Chambers | Unsupervised learning of narrative event chains[END_REF] work, which presented a statistical approach to capture script knowledge. They proposed an unsupervised induction framework, called narrative event chains to infer event and rich understanding from raw news-wire text. Narrative event chains were automatically extracted, by following mentions of an entity (e.g., protagonist) through the narrative text, which is detected on the outputs of a coreference resolution system and a dependency parser. The relationship between events was computed using Pairwise Mutual Information (PMI) score. The model's ability to capture commonsense knowledge was evaluated using the Narrative Cloze (NC) task, in which one event is removed from the event chain and the model is evaluated by ranking all candidate events. Likewise, in [START_REF] Regneri | Learning script knowledge with web experiments[END_REF] work proposed approach to unsupervised learning scripts, which focused on temporal event structure of scripts by building a temporal script graph. The graph was calculated for a scenario by identifying corresponding event descriptions using a Multiple Sequence Alignment (MSA) algorithm, and converting alignment into a graph. At the evaluation phase, the script graph algorithm showed that it significantly outperforms a clustering-based baseline, and the algorithm can distinguish event descriptions that appear at different points in the script story-line. Another unsupervised learning of scripts knowledge from the natural text was proposed by [START_REF] Frermann | A hierarchical bayesian model for unsupervised induction of script knowledge[END_REF], which used a hierarchical Bayesian model by inducing jointly events and constraints on event ordering in one unified framework. They refer to two types of entities in each type of scenarios such as event types and participant types.

Then, they incorporated the Generalized Mallows Model (GMM) over orderings from sets of Event Sequence Descriptions (EDSs). In the evaluation stage, their system outperforms the MSA algorithm provided by [START_REF] Regneri | Learning script knowledge with web experiments[END_REF] on the task of event ordering and achieving comparable results in the event induction task.

In the same year, [START_REF] Pichotta | Statistical script learning with multi-argument events[END_REF] proposed a script learning approach that employs events with multiple arguments. Unlike previous works, they model the interactions between multiple entities that enable to better prediction of events in natural texts or documents. A multi-argument event is a relational atom a = v (e s , e o , e p ), where v is a verb lemma, and e s , e o , e p are a subject relation to the verb v , direct object of v , a prepositional relation to v , respectively. Their multi-argument events modeling allows to predict event is most likely happened at some point in the sequence. By counting the number of times occur event a 1 and a 2 in order to calculate the conditional probability P (a 2 |a 1 ) of seeing a 1 and a 2 in order. In order to infer events, they proposed a ranking candidate events a by maximizing S(a) = p-1 i=1 logP (a|a i ) + |A| i=p logP (a i |a), where |A| is an ordered list of events, and p is the length of A. The model demonstrates that multi-argument events improves predictive accuracy of inferring held-out events. Like the same year, [START_REF] Modi | Inducing neural models of script knowledge[END_REF] presented another approach for event ordering tasks based on distributed representation (e.g., vectors of real numbers) of the event of predicates and their arguments and then the event representation was used in a ranker to predict the ordering of events. However, in this work, they concentrated on ordering tasks rather than predict missing events given a set of events. At the evaluation phase, their approach showed improvement in the F 1 score on event ordering with respect to the graph induction approach Regneri et al. (2010) (84.1% vs. 70.6%). Unlike previous works rely on event counting-based methods, [START_REF] Modi | Event embeddings for semantic script modeling[END_REF] proposed a neural network model based on compositional representations of events, namely event embedding, in order to predict missing event e k in the prototypical ordering of events (e 1 , e 2 , ...e k-1 , ..., e n ). The model demonstrated that it can obtain statistical dependencies between events in a scenario and outperformed count-based on the narrative cloze task.

More recent researches [START_REF] Granroth-Wilding | What happens next? event prediction using a compositional neural network model[END_REF]; [START_REF] Pichotta | Learning statistical scripts with lstm recurrent neural networks[END_REF]; [START_REF] Hu | What happens next? future subevent prediction using contextual hierarchical LSTM[END_REF]] work on multiple neural networks to improve the quality of the semantic properties of events were obtained by the model. These studies present an event embedding approach in terms of dense vector representation instead of symbolic event representation using by [START_REF] Chambers | Unsupervised learning of narrative event chains[END_REF]; [START_REF] Regneri | Learning script knowledge with web experiments[END_REF]; [START_REF] Jans | Skip n-grams and ranking functions for predicting script events[END_REF]]. [START_REF] Pichotta | Learning statistical scripts with lstm recurrent neural networks[END_REF] proposed a Long Short Term Recurrent Neural Networks (LSTM-RNN) to inferring held-out events from text and inferring novel events from the text. The input to the sequence modeling task is an event represents as a tuple of (v , e s , e o , e p , p), where v is a verb lemma, e s , e o , e p , p are arguments stands in subject, direct object, prepositional relations, prepositional relating v and e p , respectively. The model coupled with Beam Search algorithm to generate event inferences on a given set of candidate events that exist in the training set. [START_REF] Granroth-Wilding | What happens next? event prediction using a compositional neural network model[END_REF] introduced a compositional neural network to automatically acquire knowledge of event sequences from text, which provides a predictive model for use in narrative generation systems, by learning event embeddings using Word2Vec [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] on narrative event chains, where each event is represented as a predicate word or an argument word. [START_REF] Hu | What happens next? future subevent prediction using contextual hierarchical LSTM[END_REF] presented a contextual hierarchical LSTM can automatically generate a short text that describes a possible future sub-event from given previous sub-events that do not exist in the training data. However, all of these researches work on NLP from natural language text. Therefore, in order to work with scripts generation from sensors data, we either need to search for other techniques or find the relevant ways to connect sensors data to this domain.

State of the art in Human Activity Recognition from

Wearable Systems

In this section, first of all, we present the definition of the problem of HAR in Section 2.2.1. In Section 2.2.2 we show HAR system including design issues and HAR methods, and the way this task is generally evaluated. In addition, conventional machine learning and deep learning on HAR systems are introduced in Section 2.2.3 and Section 2.2.4.

Human Activity Recognition Problems

In recent years, due to the significant development of micro-electronic and computer system allow people to interact with these devices as part of their daily living. An active research area was established with the main purpose is to extracting knowledge from collected data of these devices, namely Ubiquitous Sensing [START_REF] Perez | G-sense: a scalable architecture for global sensing and monitoring[END_REF]. Particularly, human activity recognition is a task of this field and applied in several daily life domains such as medical, military and security. For instance, patients with diabetes, obesity, or heart disease are often required to follow a well-defined exercise routine as part of their treatment [START_REF] Jia | Diatetic and exercise therapy against diabetes mellitus[END_REF]. Therefore, recognizing activities such as walking, running, or resting becomes quite useful to provide feedback to the caregiver about the patient's behavior.

As a consequence, one of the recent challenge approaches in our thesis applied machine learning to recognize human activities. In fact, human activity recognition (HAR) has been approached in two different ways, namely external and wearable sensors. External sensors are fixed in a predetermined point of interest, and the wearable sensors are attached to the user. My thesis is only concerned about wearable ones due to its advantages such as very comfortable for carrying, moving and allows to collect data for outdoors activities.

Consequently, we could collect the data from the sensors (e.g., accelerometer, gyroscope, heart rate, breath rate, etc.) to detect human activities in different scenarios.

HAR is commonly solved using machine learning techniques. Similar to other machine learning applications, activity recognition requires two stages, i.e., training and testing Definition 1 (Human Activity Recognition Problem (HARP)) Given a set of n time series S = {S 0 , . . . , S n-1 } , each one from a particular measured attribute, and all defined within time interval I = [t i , t j ], the goal is to find a temporal partition (I 0 , ..., I r -1 ) of I, based on the data in S, and a set of labels representing the activity performed during each interval I k (e.g., sitting, walking, etc.). This implies that time intervals I k are consecutive, non-empty, non-overlapping, and such that

r -1 k=0 I k = I.
Definition 2 (Relaxed HAR problem) Given a set W = {W 0 , . . . , W m-1 } of m equally sized time windows, totally or partially labeled, and such that each W i contains a set of time series S i = {S i,0 , . . . , S i,k-1 } from each of the k measured attributes, and a set A = {a 0 , . . . , a n-1 } of activity labels, the goal is to find a mapping function f: S i → A that can be evaluated for all possible values of S i , such that f(S i ) is as similar as possible to the actual activity performed during W i .

Definition 3 Given a classification problem with a feature space χ ∈ R n and a set of classes A = {a 0 , . . . , a n-1 } , an instance x ∈ χ to be classified, and a set of predictions P = {p 0 , . . . , p k-1 } for x, from k classifiers, the goal of a multi-classifier system is to return the correct label a * iff ∃p i ∈ P|a * = p i Definition 1 only enables one to recognize activities that are not performed simultaneously. Therefore, the definition 2 is solved for the case of co-occurrence activities, which performed more than one in a single time window. However, during transition windows is made the relaxation produces errors, thus the number of transitions is expected to be small as much as possible, which leads to deduce the relaxation errors for most applications. Moreover, there is no one algorithm can achieve the best accuracy for all activities.

Hence, the multi-classifier should be considered as a problem in HAR, which is defined in definition 3.

Human Activity Recognition System Design issue and human activity recognition methods

There are eight main challenges related to human activity recognition, namely (1) definition of the activity set, (2) selection of attributes and sensors, (3) obtrusiveness, (4) data collection protocol, (5) recognition performance, (6) energy consumption, (7) processing, and (8) flexibility [START_REF] Lara | A Survey on Human Activity Recognition using Wearable Sensors[END_REF]. In this work, we consider the primary issues mentioned below.

Firstly, the design of any HAR system depends on the activities to be recognized.

In fact, changing the activity set A immediately turns a given HARP into a completely different problem. From the literature, seven groups of activities can be distinguished.

These groups and the individual activities that belong to each group are summarized in Table 2.2. Secondly, the success of a HAR system depends also on the sensors and the attributes chosen to describe the data in a specific domain. In the literature, there are four groups of common attributes that are computed using wearable sensors in a HAR context: environmental attributes, acceleration, location and physiological signals [START_REF] Lara | A Survey on Human Activity Recognition using Wearable Sensors[END_REF]. The first place is the attributes that provide context information describes the individual's surroundings (e.g., temperature, audio level, etc.), the second place is triaxial accelerometers often used to recognize ambulation activity, the third place is Global Positioning System (GPS) currently equipped in cellular phones, the fourth place is the vital signs data (e.g., heart rate, respiration rate, skin temperature, etc.).

These attributes and sensors would give us a definition of the set of activities for the daily living domain. In fact, acceleration is one of the important attributes in our domain which would define set of activities (e.g., walking, running, lying down, sitting, standing still, etc.), while the environmental attributes could be interested (e.g., audio level and GPS enables to realize the places where users have been being can also be helpful to figure out their activities, and physiological signals are confidently a valuable attribute to improve the recognition accuracy, for instance, heart rate is at a high level could allow to observe that user might perform running rather than sitting or lying. Meanwhile, obtrusiveness and data collection protocol show that they have an effect on the accuracy of HAR systems to raise interesting questions such as how many sensors are enough and where are the places to attach sensors on a human body to improve the accuracy of human activity recognition task.

Regarding the design of a HAR system there are several aspects to deal with: (1) the features extraction methods, (2) the learning algorithm and (3) the quality of the training data.

Two approaches were introduced to extract the features from time-series data: statistical and structural are showed in along with the quality of data and learning algorithm. Moreover, it is believed that features extraction methods would take a significant effect on HAR classification accuracy, for instance, [START_REF] Khan | A triaxial accelerometer-based physical-activity recognition via augmented-signal features and a hierarchical recognizer[END_REF] proposed a group of features can improve the recognition accuracy up to 97% by using Autoregressive model coefficient, Tilt Angle, Signal Magnitude Area. This work shows that carefully choosing features has a great impact on the classification, hence the need to explore new methods to extract features.

F(t) = a 0 + a 1 * t + ... + a n-1 * t n-1 Exponential F(t) = a * b t + c Sinusoidal F(t) = a * si n(t + b) + c
For learning algorithm, there are two mainstream learning approaches, namely supervised and unsupervised learning, which deal with labeled and unlabeled data, respectively.

Supervised learning is one of the most popular approaches to activity recognition with many algorithms as presented in

Evaluation metrics

When evaluating a machine learning algorithm, the training and testing datasets should be disjoint. This is with the aim of assessing how effective the algorithm is to model unseen data. A very intuitive approach is called random split and it simply divides the entire dataset into two partitions: one for training and the other one for testing -usually, two-thirds of the data are for training and the remaining one third is for testing. However, a random split is highly biased by the dataset partition. If instances in any of the training or testing sets are concentrated in a particular feature space sub-region, the evaluation metrics would not reflect the actual performance of the classifier. Therefore, a more robust approach is cross-validation. In k-fold cross-validation, the dataset is divided into k equally-sized folds. In the first iteration, the very first fold is used as the testing set while the remaining k-1 folds constitute the training set. The process is repeated k times, using each fold as a testing set and the remaining ones for training.

In the end, the evaluation metrics (e.g., accuracy, precision, recall, etc.) are averaged out over all iterations. In general, the selection of a classification algorithm for HAR has been merely supported by empirical evidence. The vast majority of the studies use crossvalidation with statistical tests to compare classifiers' performance for a particular dataset.

The classification results for a particular method can be organized in a confusion matrix M n * n for a classification problem with n classes. This is a matrix such that the element M ij is the number of instances from class i that were wrongly classified as class j = i .

The following values can be obtained from the confusion matrix in a binary classification problem:

• True Positives (TP): The number of positive instances that were classified as positive.

• True Negatives (TN): The number of negative instances that were classified as negative.

• False Positives (FP): The number of negative instances that were classified as positive.

• False Negatives (FN): The number of positive instances that were classified as negative.

The accuracy is the most standard metric to summarize the overall classification performance for all classes and it is defined as follows:

Accur acy = T P + T N T P + T N + F P + F N

The precision -often referred to as positive predictive value -is the ratio of correctly classified positive instances to the total number of instances classified as positive: P r eci si on = T P T P + F P The recall, also called true positive rate, is the ratio of correctly classified positive instances to the total number of positive instances:

Recall = T P T P + F N
The F-measure combines precision and recall in a single value: Fmeasur e = 2 * P r eci si on * Recall P r eci si on + Recall

Conventional Machine Learning on HAR Systems

There are a lot of studies and reports about human activity recognition using wearable sensors in recent years. HAR systems can be divided up following two dimensions. By learning approach, namely supervised and semi-supervised; and by the response time constraint, namely on-line and off-line. The on-line approach provides immediate feedback on the performed activities [Lara and Labrador (2012); [START_REF] Anguita | Energy efficient smartphone-based activity recognition using fixed-point arithmetic[END_REF]]. The off-line approach either needs more time to recognize activities due to high computational demands or are intended for applications that do not require real-time feedback [START_REF] Kwapisz | Activity recognition using cell phone accelerometers[END_REF]; [START_REF] Ronao | Deep convolutional neural networks for human activity recognition with smartphone sensors[END_REF]; Lara et al. (2012)]. Figure 2.1 presents a flow diagram of using conventional machine learning approaches based on sensors for activity recognition. First of all, the raw data was obtained by some types of sensors (e.g., smart-phones, smartwatches, WiFi, Bluetooth, audio, etc.). Secondly, the features were extracted from the raw signal by statistic information such as mean and standard deviation, etc. Finally, these features were fed into machine learning models (e.g., kNN, SVM, HMM, etc.) so as to perform the recognition tasks.

Supervised Online HAR system

Applications of online activity recognition systems are often seen in health-care, where the continuous monitoring of patients who have physical or mental pathologies is crucial for their protection, safety, and recovery. Therefore, it is required for an online system to identify what the user is currently doing Lara and Labrador (2012). Likewise, interactive games or simulators may enhance the user's experience by considering activities and their locations [START_REF] Riboni | Cosar: Hybrid reasoning for context-aware activity recognition[END_REF]. Unlike online systems, offline HAR systems are not dramatically affected by processing and storage issues because the required computations are performed in a server with large computational and storage capabilities. Additionally, energy expenditures are not analyzed in detail as a number of systems require neither integration devices nor wireless communication so the application lifetime would only depend on the sensor specifications.

Study

Ambulation activities are recognized very accurately by [Lara et al. (2012); [START_REF] Khan | A triaxial accelerometer-based physical-activity recognition via augmented-signal features and a hierarchical recognizer[END_REF]]. These systems place an accelerometer on the subject's chest, which is helpful to avoid ambiguities due to abrupt corporal movements that arise when the sensor is on the wrist or hip. An additional challenge was raised in [START_REF] Bao | Activity recognition from user-annotated acceleration data[END_REF], where activities such as eating, reading, walking, and climbing stairs could happen concurrently. However, no analysis is presented to address that matter. In the following paragraph, we introduce several HAR systems that use an off-line supervised learning approach. [START_REF] Kwapisz | Activity recognition using cell phone accelerometers[END_REF] presented a HAR system under the mobile Android platform.

This system only used the accelerometer to recognize six ambulatory activities such as walking, jogging, climbing up stairs, climbing down stairs, sitting and standing from twentynine users attached in their pockets. 43 features were extracted from the time-domain method. Decision tree C4.5, Multilayer Perceptron (MLP) and Logistic Regression were applied to classify the activities. The author reported that MLP can obtain 91% accuracy outperforming the other learning algorithms. However, the activities climbing up and climbing downstairs were the most difficult to recognize.

The system proposed by [START_REF] Zhu | Multi-sensor fusion for human daily activity recognition in robot-assisted living[END_REF] and standing) and non-stationary activities (e.g., walking and running). Then, an HMM receives the ANN's output and generates a specific activity prediction. An important issue related to this system is that all the data were collected from one single individual, which does not permit to draw strong conclusions on the system flexibility. domain features (e.g., mean, root mean square, variance, correlation, and standard deviation, etc.) were extracted by a window-size 100 samples. Then, these features were selected by a genetic algorithm, so that it can gain attention on the feature set, which is appropriated for different activities for a single person and effective in representing these activities across multiple subjects. The system shows that it can achieve 93% of average accuracy for subject-independent in HAR. Sztyler and Stuckenschmidt (2016) presented an approach for activity recognition based on acceleration data, which located in different on-body positions with multiple wearable devices. This work concentrated on both subject-independent and subject-dependent in HAR. Without the information of the device position, the classifiers (e.g., Decision Tree) only can achieve 80% of F-measure. While with information on device position the classifier can obtain higher F-measure up to 89%. (2) shallow features (e.g., mean, variance, frequency, etc.) usually are only able to be used for low-level activity recognition task (e.g., walking, jumping, etc. As a result, these models outperformed the state of the art on benchmark datasets.

Deep Learning on HAR Systems

Convolutional neural network

In recent years, CNN is applied in many areas such as image classification, speech recognition, and text analysis because of its automatic feature extraction from signals and

promising results on the tasks. In time series classification like HAR, CNN has two different models: local dependency (correlated) and scale in-variance (frequencies) as mentioned in [START_REF] Wang | Deep learning for sensor-based activity recognition: A survey[END_REF]. Most researches on CNN for HAR models concentrated on several aspects as follows: input adaptation, pooling, and weight-sharing.

• Input adaptation: The input adaptation is the approach to form sensor readings (e.g., 3-axis accelerometers) in HAR to be the same as a virtual image, which is the inputs of CNN. There are two types of adaptation in HAR: (1) data-driven treats each dimension sensor readings as a channel, then applies 1D convolution on them [Zeng et al. (2014a); Jiang and Yin (2015a)]. Thereby, the drawback of this approach is the ignorance of dependencies between dimension and sensors, which may impact on the performance as mentioned in [START_REF] Wang | Deep learning for sensor-based activity recognition: A survey[END_REF]; (2) model-driven reshapes the inputs to a virtual 2D image in order to adopt 2D convolution [Ha and Choi (2016a); [START_REF] Jiang | Human activity recognition using wearable sensors by deep convolutional neural networks[END_REF]]. Hence, this approach can improve the temporal relation of the sensor. However, the formulation of the image from the time series is a non-trivial task.

• Pooling: An interest of pooling is it can accelerate of training process on large data [START_REF] Bengio | Deep learning of representations: Looking forward[END_REF]. Most approaches performed average and max pooling after convolution [Ha and Choi (2016a); [START_REF] Kim | Hand gesture recognition using micro-doppler signatures with convolutional neural network[END_REF]].

• Weight-sharing: this approach can also speed up the training process on a new task

Restricted Boltzmann machine

Restricted Boltzmann machine (RBMs) is a generative stochastic artificial neural network, which is restricted in their neuron and forms a bipartite graph. It is composed of two layers; i.e., visible and hidden [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF] where each neuron in the layer is not connected. The hidden layer assists to forward pass or classification task and the visible layer helps for backward pass/reconstruction of the original input (known as generative learning). A deep belief networks (DBNs) is defined as a stack of RBMs, in which two consecutive layers are represented as an RBM. This stack is often followed with a softmax layer to create a classifier or help to cluster unlabeled data in an unsupervised task.

In recent years, there are several researches work on HAR using RBMs and DBNs. In summary, although this state-of-the-art of the HAR domain shows that activities daily living can be recognized with conventional and deep machine learning. However, there is still no approach that allows generating a story or script from the HAR system. In addition, conversely, there is no approach for script generation on the NLP domain using sensor-based human activity recognition. In this thesis, we approach these issues in order to connect both domains together. Moreover, we will deal with existing issues on each domain such as imbalanced data and high-level activities recognition, which enhance the prediction of the HAR system. Besides, we handle issues on script generation such as event representation, the sequence of events classification so as to predict the next event, missing event in natural language text on the NLP domain.

Chapter 3

Problematic and Overall approach

Problem Statement

As stated in the introduction, generating a coherent textual story from raw sensor data is highly challenging. From a pure data abstraction and data processing point of view, the following research questions should be addressed.

1. What information must be extracted from ambient data to be able to generate a story?

2. What data processing approach would make the extraction of this information possible?

3. How this information should be structured?

4. What Natural Language Generation process must be employed to generate a story from this set of information?

However, from a cognitive point of view, the problem is not defined in terms of data processing but in terms of information communication. Hence, in this specific case, the problem of story generation can be seen as constructing a discourse that respects the following objectives.

1. Linguistically represents a set of events discernible in the data.

2. Relates them temporally and causally.

3. Integrates these chunks of the information under a common communicative purpose.

This communicative purpose will drive, among others, the below sub-objectives (a) The motivation and/or justification of decisions/actions in the story;

(b) The filtering/amplification of pieces of information. In HAR, the model is typically acquired using machine learning over datasets collected from smartphones and manually labeled. Such a model should be invariant to the users and devices on which the data is collected [START_REF] Lara | A Survey on Human Activity Recognition using Wearable Sensors[END_REF]. To permit the model to generalize, a large amount of data from several different users and devices is necessary.

A Data Abstraction Approach

Another important problem is the class imbalance. Indeed, data-sets acquired from reallife do not contains uniformly distributed examples. For instance, the jump activity might far less present in the data than the sitting activity. Such a situation biases the learning towards the majority classes. Thus, a solution must be found to make unbiased learning possible. To address the HAR problem, we will use Artificial Neural Network models learned on smartphone data and propose a meta-learning framework to deal with imbalanced data.

However, most of HAR systems in recent researches [START_REF] Anguita | Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine[END_REF]

The Cognitive Side of the Approach

Regarding the structuring aspect of a story, there is a large number of ways to perform it according to the audience, the communication goal and even the medium with which the story is being expressed (e.g., picture, video, text). Since the objective is to make the generated story highly understandable by a human, we take a cognitive approach to the story by considering the Scripts theory. Indeed, much of our common sense knowledge about the world (such as what usually happens when going to a restaurant) is thought to be represented in our mind by Scripts. Script theory was introduced by Tomkins (1978) who claimed that human behavior mostly follows patterns called Scripts which are stereotypical likely sequences of events. This theory has been popularized in AI by [START_REF] Schank | Scripts, plans and knowledge[END_REF] who introduced it as a method for representing procedural knowledge. Hence, in this thesis, we define an event/narrative as a sentence of words that describes human activities, action, and their location. We call script as a sequence of event/narrative, which is represented in the temporal order. Finally, we consider the story as a sequence of scripts.

However, script models must be acquired to be useful for story generation. If scripts are built by humans from experience, computers extract them from data. Over the past decade, there have been plenty of studies [START_REF] Modi | Inscript: Narrative texts annotated with script information[END_REF] scales. In addition, with this model, we can solve the problem of temporal variation of ADL events by removing repetitive events based on creating new segmentation using the time-slices on the raw signal. As a result, we can re-segment the events located on the appearance frequency of their labels in the raw signal. As mention in [START_REF] Miyanishi | Generating an event timeline about daily activities from a semantic concept stream[END_REF] simple ADL labels prediction such as walking, running, etc. is the shortcoming to generate a story or event time-line. Therefore, in this research, we also follow the approach in [START_REF] Miyanishi | Generating an event timeline about daily activities from a semantic concept stream[END_REF] by recognizing the sequence of combinations of semantic concepts (e.g., human physical activity, action, location).

In the second part of the Figure 3.1, the output of prediction of HAR (the sequence of combinations of semantic concepts) will be considered as the input of sequence to sequence neural network model [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF], so that the system can generate the candidate event/text descriptions from the semantic concepts. In fact, generating descriptions is a popular research in visual content (e.g., images and video) can be divided into four different approaches: (1) generating descriptions for images or videos contain some associated text [START_REF] Aker | Generating image descriptions using dependency relational patterns[END_REF]; Feng and Lapata (2010)], (2) generating descriptions by using manually defined rules or templates [START_REF] Tan | Towards textually describing complex video contents with audio-visual concept classifiers[END_REF]; [START_REF] Guadarrama | Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition[END_REF]], (3) retrieving existing descriptions from similar visual content [START_REF] Farhadi | Every picture tells a story: Generating sentences from images[END_REF]; [START_REF] Ordonez | Im2text: Describing images using 1 million captioned photographs[END_REF]], (4) learning a language model from a training corpus to generate descriptions [START_REF] Kulkarni | Baby talk: Understanding and generating simple image descriptions[END_REF]; [START_REF] Kuznetsova | Collective generation of natural image descriptions[END_REF]]. In order to generate text descriptions from semantic concepts, we adopt these approaches by combining approach (2) and (4). At first, we generate the text description corpus from semantic concepts by simple detection rules as [START_REF] Tan | Towards textually describing complex video contents with audio-visual concept classifiers[END_REF] to glue the gesture, action, and location. However, using manually defined rules limit the natural flexibility of language as mentioned in [START_REF] Kuznetsova | Collective generation of natural image descriptions[END_REF]. Therefore, in the next step, the text description corpus can be considered as a target, the semantic concepts are determined as the source for the sequence to sequence neural network (seq2seq) model. By using the seq2seq model we can generate and enrich the semantic and syntactic of the generating candidate event/text descriptions. By recognizing the sequence of combinations of semantic concepts, we can obtain a sequence of candidate event/text descriptions (Event timeline/Scripts) that were generated from the seq2seq model.

system is a problem to automatically construct the structure of some level of abstraction such as sequences of events, actions or protagonist and entities that participate in that can be told as a story.

The very first devoted work by [START_REF] Schank | Scripts, plans and knowledge[END_REF] studied script knowledge focused on manual construction of its bases. Then in the past decade, there were many researches revised this work, which interested in automatically induction of script knowledge from text such as: [START_REF] Chambers | Unsupervised learning of narrative event chains[END_REF]; [START_REF] Pichotta | Statistical script learning with multi-argument events[END_REF]] carried out studies on naturally occurring texts or [START_REF] Regneri | Learning script knowledge with web experiments[END_REF] worked on crowd-sourced data. Most of these approaches focused on count-based techniques that represent events as a graph. Each node of the graph is an event, which is modeling as a verbal predicate with tuples of its arguments, and arcs correspond to the temporal precedence relation.

Consequently, these methods usually suffer from the data sparsity because of the limitation of their model to capturing dependencies only between events have the common entities (e.g. protagonist) and the probability of predicate and argument estimates based on their co-occurrences (count-based method). Furthermore, [START_REF] Modi | Event embeddings for semantic script modeling[END_REF] proposed event embedding modeling to conquer the imperfections of the count-based methods. One of the advantages of the event embeddings is enabling to capture the semantic properties of events, it means that the events with different surface form of their constituents but are semantically similar will receive the similar event embeddings. However, one of the disadvantages of these event embeddings is the lack of understanding events with deeper syntactic and semantic analysis of language. Therefore, in this proposed method we present a novel technique to model scripts based on Recursive Neural Networks Semantic Syntactic, namely NESS to represent a sequence of events as composition of their semantic predicates and arguments. The advantage of our proposed model can effectively dealt with sparsity in semantic space by representing meaning at a higher level of abstraction than the surface forms of words.

In the fourth part of Figure 3.1, it presents the way that we enrich our source and target language, which can be considered as an event structure, and then feed into the seq2seq model as the input. In order to do this, we take the output of script modeling then combine with the source and target that we glued from semantic concepts of daily activity location, which detected by a deep neural network. However, this work still has not covered in this thesis due to time limitations.

Chapter 4

Human Activity Recognition from mobile phone sensors

In this chapter, we present our proposed method to deal with imbalanced data for conventional learning methods using a mobile phone sensor. First of all, in Section 4.1 we introduce the problem and related work in Section 4.2. Next, in Section 4.3 we introduce our novel framework, which is using active learning and oversampling technique so that it can solve for the problems of imbalanced data, based on Multilayer Perceptron (MLP) -a machine learning model. Next, the experimental results are illustrated in section 4.4. The chapter ends with a conclusion in Section 4.5.

Introduction

Human Activity Recognition from wearable sensors and in particular smartphones has been subjected to intense research and industrial activity this last decade [START_REF] Lara | A Survey on Human Activity Recognition using Wearable Sensors[END_REF]. Many learning algorithms have been used to classify physical human activities such as Running, Walking, etc. as well as interactive and social activities (chatting, talking, playing, etc.). HAR is useful for health monitoring, senior care and personal fitness training as well as for providing context to smartphone applications. Physical human activities are generally classified from recorded sensor data (e.g. accelerometers, GPS, audio, etc.)

which are embedded into wearable devices (e.g. smartphones and smartwatches).

HAR systems performances are highly dependent on the classification model (Decision Tree, Support Vector Machine, Multi-Layer Perceptron, etc.), the feature used, the number of classes and the size of the datasets available for training [START_REF] Lara | A Survey on Human Activity Recognition using Wearable Sensors[END_REF]. However, another aspect that plays an important role in this domain is the lack of a uniform collection of different activities. In fact, this is the case for most smartphone datasets (e.g. Running = 4% and Walking =40% distribution). This is called the Class Imbalance Problem that is known to have a serious influence on the performance of learning algorithms because most standard algorithms expect balanced class distributions [START_REF] He | Learning from imbalanced data[END_REF].

In the past, research on HAR based on wearable sensors did not systematically handle the class imbalance problem. Therefore, in this work, we introduce a generic framework that integrates active learning with an over-sampling method based on MLP to overcome this problem. We also introduce a new over-sampling method, called BLL SMOTE -an extension of SMOTE Chawla et al. (2002) -which can apply to non-convex spaces.

Contributions. Our contributions are summarized as follows.

• A framework integrating MLP and active learning with over-sampling.

• A new over-sampling method, BLL SMOTE.

• Experiments with 2 available datasets that show the impact of taking the class imbalance problem into account in the learning.

Related Work

Human Activity Recognition (HAR) from wearable sensors data is a very rich domain of research. We restrict here in presenting the main work regarding the classification models being used, the available ecological datasets and the techniques to deal with imbalanced class distribution in data. Regarding the classification models, there have been many

approaches to deal with HAR from wearable sensors. Over the last decade, the most common approach is to process windows of data streams to extract a vector of features that will, in turn, be fed to a classifier. wearable sensors [START_REF] Bayat | A study on human activity recognition using accelerometer data from smartphones[END_REF]; [START_REF] Arifoglu | Activity recognition and abnormal behaviour detection with recurrent neural networks[END_REF]].

ANN has also been broadly used in HAR [START_REF] Bayat | A study on human activity recognition using accelerometer data from smartphones[END_REF]; [START_REF] Kwapisz | Activity recognition using cell phone accelerometers[END_REF]; [START_REF] Khan | A triaxial accelerometer-based physical-activity recognition via augmented-signal features and a hierarchical recognizer[END_REF]] and in addition to that deep learning now is a challenging topic that many scientists are interested in HAR, for instance, [START_REF] Ronao | Deep convolutional neural networks for human activity recognition with smartphone sensors[END_REF]; [START_REF] Zeng | Convolutional neural networks for human activity recognition using mobile sensors[END_REF]] is a demonstration of applying convolution neural networks in HAR using wearable sensors. For applying the machine learning approach in our thesis, we achieved to build a framework of human activity recognition based on a supervised learning algorithm namely Multilayer Perceptron as shown in Figure 4.2. It was implemented using Tensorflow library is provided by Google Team. An advantage of this framework allows us to deal with poor data quality by using methods such as sampling and cost-sensitive. Indeed, the quality of training data is also an extreme challenge, in recent years, a survey [START_REF] He | Learning from imbalanced data[END_REF] reported that imbalanced data has a serious influence on the performance of learning algorithms. Most standard algorithms expect balanced class distributions, hence datasets contain imbalanced class distribution which makes these algorithms fail to correctly represent the distributive characteristics of the data. As a consequence, it would produce unfavorable accuracies across the classes of the data and leads to a decrease in the total accuracy of learning algorithms. In order to deal with imbalanced data, there are several methods introduced in He and Garcia ( 2009), namely sampling, cost-sensitive, kernel-based and active learning. In the first place, under-sampling (e.g., random undersampling) was evaluated is as better than over-sampling (e.g., random over-sampling), which means that reduction of the majority classes provides higher accuracy than an increase of minority classes. In addition, the strategy of under-sampling is also important, [START_REF] Yen | Cluster-based under-sampling approaches for imbalanced data distributions[END_REF] proved that cluster based under-sampling outperforms the other undersampling techniques. In the second place, cost-sensitive learning methods are concerned with the costs associated with misclassifying examples by using different cost matrices.

Several methods based on learning algorithms were introduced such as cost-sensitive dataspace weighted adaptive boosting, cost-sensitive decision tree. Although, we only interest in cost-sensitive neural networks in three ways: first, cost-sensitive modification can be applied to the probabilistic estimate; second, the neural network outputs can be made cost-sensitive; third, cost-sensitive modification can be applied to the learning rate. The idea of the probabilistic estimate is integrating cost factors into the testing stage of classification to adaptively modify the probability estimate of neural network output, while the outputs of the neural network are altered during training to bias the neural network to focus more on expensive class. The learning rate can be applied cost-sensitive to put more attention on costly examples during the learning. In the third place, we consider the active learning that has been investigated to compromise with imbalanced learning problems Ertekin et al. (2007a,b). Generally, active learning is used to solve problems related to unlabeled training data. Moreover, it integrates with sampling techniques to analyze the effect of under-sampling and over-sampling by using uncertainty sampling methodology;

the challenge is how to measure the uncertainty of an unlabeled instance in order to choose the maximally uncertain instance to augment the training data. Moreover, this framework also allows for dealing with over-fitting problems such as regularization and dropout.

Machine learning is highly dependent on datasets. It is, even more, the case with Deep Learning. If many papers report work dealing with non-accessible datasets, many others investigate datasets that are made publicly available. The survey by [START_REF] Micucci | Unimib shar: a new dataset for human activity recognition using acceleration data from smartphones[END_REF] presents a large number of publicly available datasets acquired from a smartphone.

However, it also shows a lack of uniformity in tasks, sensors, protocol, time windows, etc.

It is worth to notice that most of the datasets are restricted to inertial sensors such as accelerometers. The audio sensors are largely ignored while being among the only ones that are always found on a smartphone. It is also worth noticing that some are very imbalanced since the distribution among classes are very different. For instance, in the ExtraSensory Dataset [START_REF] Vaizman | Recognizing detailed human context in the wild from smartphones and smartwatches[END_REF], sitting represents 44.2% of the data while running only 0.3%. In this case, the learning approach should consider the class imbalance problem. [START_REF] He | Learning from imbalanced data[END_REF] reported that imbalanced data has a serious influence on the performance of learning algorithms, because most standard algorithms expect balanced class distributions. Hence, datasets exhibiting imbalanced class distribution make these algorithms fail to correctly represent the distributive characteristics of the data. As a consequence, it would produce mis-classification of minority classes higher than mis-classification of majority classes, and leads to a decrease in the overall accuracy of learning algorithms.

In fact, in HAR, a few studies coped explicitly with this problem such as [START_REF] Abidine | A new multi-class WSVM classification to imbalanced human activity dataset[END_REF] who proposed Weighted Support Vector Machines (WSVM) to improve the learning of minority classes. However, the approach is based on a scheme that puts more weight on the errors in the minority classes than on the majority classes. Therefore, this approach is highly dependent on instances of minority classes.

In general, in order to deal with imbalanced data, several other approaches were introduced in [START_REF] He | Learning from imbalanced data[END_REF] such as over-sampling, active learning. For the former approach, some methods were proposed such as SMOTE Chawla et al. (2002) or Borderline SMOTE Han et al. (2005) which work by generating new synthetic instances of minority classes. Their studies showed that over-sampling techniques succeeded to enhance the classification accuracy for imbalanced datasets. For the latter approach, Ertekin et al.

(2007a) introduced a SVM based active learning framework in which SVM starts to train on a given training dataset, then selects the most informative instances from a pool of the features extracted from the sensors, the hidden layer nodes H = {h 1 , ..., h l } and the output layer nodes Y = {y 1 , ..., y k } are computed as follows:

         h 1 . . . h l          =          w 1,1 * x 1 +... +w 1,n * x n . . . . . . . . . w l,1 * x 1 +... +w l,n * x n                   y 1 . . . y k          =          w 1,1 * h 1 +... +w 1,l * h l . . . . . . . . . w k,1 * h 1 +... +w k,l * h l          +          b 1 . . . b k         
This is shown in Figure 4.2, where {w 1,1 , ..., w l,n }, {w 1,1 , ..., w k,l } and {b 1 , ..., b k } represent weights and bias.

Active Learning

The principle of Active Learning (AL) is to learn to label unknown instances by selecting (querying) some specific instances and ask an external system (e.g., a human operator) to label them. It has become an emerging research topic with applications in many fields such as image segmentation [START_REF] Biswas | Active image clustering: Seeking constraints from humans to complement algorithms[END_REF], data clustering [START_REF] Mai | AnyDBC: An Efficient Anytime Densitybased Clustering Algorithm for Very Large Complex Datasets[END_REF] text classification [START_REF] Tong | Support vector machine active learning with applications to text classification[END_REF]. Applying AL in HAR is thus an interesting approach since it can further boost up the accuracy by involving humans in the classification task, especially for hard to classify activities. Moreover, its scheme provides a natural way to cope with data imbalance by exploring some most uncertain data spaces, as pointed out in Ertekin et al. (2007a).

Typically, an active learning algorithm chooses objects that their labels are among the most uncertain ones to query users for. Uncertain instances can be chosen in many different ways Settles (2010). Our technique is built upon the uncertainty sampling technique [START_REF] Settles | Active learning literature survey[END_REF] whose principle is that the most relevant instances to be selected for annotation are the ones for which the estimates are the less certain. Thus, after MLP training, we predict the labels of U using the training output Y of MLP. Y can be seen as a vector of the probability of labels. Then the instances in U are ranked according to their decreasing Shannon Entropy, because the higher entropy of an instance is, the more uncertainty there is on its class. Therefore, the most uncertainty instance can be picked up by maximized Shannon entropy Shannon (2001) using Equation (4.1):

x * H = ar g max

x - i P θ (y i |x) log P θ (y i |x) (4.1)
where x is an instance, P θ (y i |x) is the probability of all possible labels on the instance.

Oversampling Border Limited Link SMOTE Method

While classical active learning methods only add in the training set the uncertainty instances, that were labeled by a user, our method also performs over-sampling on queried data. This makes it possible to put new information into the training set and tackle the class imbalance problem.

Over-sampling consists of adding a new sample to a training set, whether they are synthetic or real. For instance, SMOTE Chawla et al. (2002) generates a new synthetic instance, using Equation (4.2):

x new = x i + (x θ i -x i ) * λ (4.2)
where x new is the new sample generated from x i ∈ S min , with S min is the samples of minority class, x θ i is one of the k-nearest neighbors of x i : x θ i ∈ S min , and λ ∈ [0, 1] is the random number, which allows to randomly generate the new synthetic instance x new along the line between x i and x θ i . However, this method is not relevant in the case of non-convex spaces. For instance, imagine space as represented in Figure 4.3. If x i and x θ i are two samples of the green (circle) class, a direct application of Eq. (4.2) would produce a new sample x new which would not be in the right space.

To avoid the mis-generation of synthetic instances in the case of non-convex dataset, we introduce the BLL SMOTE method described as follows. The method uses Eq. (4.2) but calculates the distance from x new to each of the k-nearest neighbors of x i , denoted as d j = d(x new , x θ i ), j = 1, ..., k, where d is the Euclidean distance. Then, the distance of the artificial instance x new with its nearest instance

x dif f ∈ S min such that x dif f ∈ S, denoted as d dif f = d(x new , x dif f ) is computed. Finally, each d j is compared to d dif f . If any d j is greater than d dif f
, then this artificial instance x new is not accepted to be generated.

Otherwise, x new is accepted.

An advantage of BLL SMOTE is to avoid the mis-generated new synthetic instance in non-convex datasets

Implemented Framework

The overall learning process is summarized in Algorithm 1. Providing a training set S, a pool of instances U and a number of query N, an MLP model Ω is firstly trained from S (line 4 ). Next, uncertainty sampling is used to query instances to be added to S (line Ω : the learned model

1 Initialize S from a subset of X 2 Initialize L ← ∅ 3 for i : 1 → N do 4 Ω = MLP _T r ai n(S) 5 L ← Uncer tai nty _Sampli ng(U, Ω) 6 U ← U -L 7 Query labels y * for all x * ∈ L 8 S ← S ∪ L 9 foreach x * ∈ L do
L_nei ghbor s = k_nearest_neighbors (x * ) foreach x θ * ∈ L_nei ghbor s do

x new = x * + (x θ * -x * ) * λ d dif f = Eucli dean_di stance(x new , x dif f ) accepted_gener ati on = T r ue foreach x θ * ∈ L_nei ghbor s do d = Eucli dean_di stance(x new , x θ * ) if d > d dif f then accepted_gener ati on = F alse break;
if accepted_gener ati on = T r ue then

S = S ∪ (x new , y * )
return Ω

BLL SMOTE also has an effect on the classification performance of each class. On Figure 4.7, the MLP performance on LIG-SPHAD at the last step of active learning with over-sampling BLL SMOTE, demonstrates that the minority classes such as Jump, Unstable can achieve nearly 0.7 and 0.65 F1 score respectively, that is much higher than the MLP performance in Figure 4.5 where F1 score are 0.6 and 0.5 respectively. On the ExtraSensory dataset, the right side of Figure 4.7 also illustrates that minority classes such as Running, Stairs can reach an F1 score of 69% and 65% respectively higher than in the right side of Figure 4.5, where the same classes achieved 60% and 50% F1 score respectively. Hence, BLL SMOTE makes it possible to increase minority class performance in a discriminative setting.

Conclusion

In this work, we introduced a generic framework that integrates active learning with the over-sampling method based on MLP to overcome the class imbalance problem. We also introduce a new over-sampling method, called BLL SMOTE -an extension of SMOTE [START_REF] Chawla | Smote: Synthetic minority over-sampling technique[END_REF] -which can be applied to non-convex spaces.

The experiments carried out on two different datasets demonstrated that using active learning with over-sampling to tackle the imbalance distribution of class can increase the global F1 score of the two datasets by about 15% absolute over the baselines. In each case, BLL SMOTE shows slightly higher performances than using SMOTE plus Active learning.

In addition, BLL SMOTE is able to increase the classification performance of minority classes. Another important point of this study is the fact that our method prevents the mis-generation of the synthetic samples, thanks to its capacity to manage non-convex datasets.

These results show two advantages over classical approaches: the method makes it possible to improve overall and local performances and does not require extra external data. This last advantage is important in a domain such as a smartphone HAR where data collection is costly and available datasets might differ too much in terms of target, features or time resolution.

Chapter 5

Scripts Generation from Events

In this chapter, we present our novel approach to generate scripts from events, which are detected using a deep learning method on wearable sensors data. Section 5.1 shows an introduction, then related works are presented in section 5.2. In section 5.3 we introduce our proposed methods, and the experiment results are illustrated in section 5.4. The chapter ends with a short discussion and an outlook of future work in Section 5.5.

Introduction

Script is a stereotypical sequence of events in prototypical scenarios, was first introduced by [START_REF] Schank | Scripts, plans and knowledge[END_REF]. As mentioned in [START_REF] Schank | Scripts, plans and knowledge[END_REF] a story has invoked a script with one or more interesting deviations, and more precisely script is also a very boring little story. There are extremely numerous scripts in our daily life such as birthday party script, working day script, and restaurant script, and so on. For instance, Table 5.1 shows an example of restaurant scripts in different scenarios.

The above scripts enable us an insight into the sequence of events happened in the restaurant context with different scenarios, which are providing in the form of human readable texts. For instance, the sequence of events in scenario 2 let us deduce that the person is ordering food, or scenario 4 shows us that the person is leaving the restaurant.

In general, scripts allow human to deeply understand everyday human life in the real world.

Fortunately, nowadays HAR systems are using wearable sensor data also permit us to recognize human activities during daily life, which are exploiting machine learning models [START_REF] Anguita | Energy efficient smartphone-based activity recognition using fixed-point arithmetic[END_REF]; [START_REF] Kwapisz | Activity recognition using cell phone accelerometers[END_REF]; [START_REF] Blachon | On-line context aware physical activity recognition from the accelerometer and audio sensors of smartphones[END_REF]] to classify and predict human activity such as walking, running, sitting, etc.. However, these current HAR systems are often tackling by recognizing discrete daily human activities, hence it is not sufficient to constitute a script. In this work, we have therefore proposed a novel approach in order to automatically generate scripts from the human activity recognition Scenario 1: Entering Self into restaurant.

Look where empty tables are.

Where to sit.

Self to table.

Sit down.

Scenario 2: Ordering Receive menu.

Read menu.

Decide what self want.

Order to waitress.

Scenario 3: Eating Receive food.

Eat food.

Scenario 4: Exiting Ask for check.

Receive check.

Tip to waitress.

Self to cashier.

Money to cashier.

Self out of restaurant.

Table 5.1: Event sequence of restaurant visiting system using wearable sensor data. Figure 5.1 depicts a process of the raw signal data (e.g., accelerometers, audio, etc.) of human activity into the generation of the scripts.

First, our method generates a rich semantic concept (e.g., gesture, activity, location) of raw signal sensor data (e.g., accelerometers) detected by using the Sequential HAR system, which will be described in section 5.3. Unlike [START_REF] Miyanishi | Generating an event timeline about daily activities from a semantic concept stream[END_REF] semantic concepts are unknown related to each other's, therefore [START_REF] Miyanishi | Generating an event timeline about daily activities from a semantic concept stream[END_REF] have to develop temporal interactions among semantic concepts to remove objects and places unrelated to each action. In contrast, our sequential HAR system allows us to predict semantic concepts over time, and present it in the form of a sequence. And second, we propose to formulate the generation of scripts as a hybrid combination of semantic concepts detection templates and a machine translation problem. In more detail, detection rules provide several templates to combine semantic concepts and generate sentences as a target language for machine translation problems, which is using semantic concepts as source language and the generated sentences as the target language, will be detailed in focus on generate activity reports based on charts or graphs by predicting activity labels from sensor data. In contrast, in this work, we mainly focus on generating a sequence of events/scripts.

For recognize human activities of daily life, many approaches [START_REF] Anguita | Energy efficient smartphone-based activity recognition using fixed-point arithmetic[END_REF]; [START_REF] Kwapisz | Activity recognition using cell phone accelerometers[END_REF]; [START_REF] Bayat | A study on human activity recognition using accelerometer data from smartphones[END_REF]] were proposed used online and offline classification models, which allow recognizing indoor and outdoor activities with external and wearable sensors. Moreover, due to the evolutionary development of wearable sensors, it can be easy to integrate on wearable devices (e.g., smart-phone, smartwatch, etc.) to record attributes (e.g., acceleration, location, etc.). Recently, many vision-based activity recognition methods have been proposed using wearable cameras [START_REF] Ma | Going deeper into first-person activity recognition[END_REF]; [START_REF] Pirsiavash | Detecting activities of daily living in first-person camera views[END_REF]; [START_REF] Ohnishi | Recognizing activities of daily living with a wrist-mounted camera[END_REF]] for predicting daily activities and their related objects in home environments. With this motivation in our current work, we are also using wearable sensors to predict daily activities by developing a deep learning model. However, we still have not considered the wearable camera for our HAR system.

For context-aware of human activities of daily life, some studies were carried out [START_REF] Lee | Activity and location recognition using wearable sensors[END_REF]; [START_REF] Riboni | Cosar: Hybrid reasoning for context-aware activity recognition[END_REF]] to recognize daily activities, and simultaneously the places where the activities were performed. According to [START_REF] Lee | Activity and location recognition using wearable sensors[END_REF], the transition of locations is detected by integrating the subject's motor activities using a dead-reckoning method. Similarly, [START_REF] Riboni | Cosar: Hybrid reasoning for context-aware activity recognition[END_REF] proposed a system that combines between ontological reasoning and statistical inferencing to recognize more accurately activities based on contextual conditions (e.g., location, surrounding environments, used objects). Nevertheless, these researches did not consider on the sequence of events or scripts generation from daily activities recognition.

There have been much extensive researches work on language description generation [START_REF] Miyanishi | Generating an event timeline about daily activities from a semantic concept stream[END_REF]; [START_REF] Rohrbach | Translating video content to natural language descriptions[END_REF]] from visual content (e.g., image, video, etc.). In general, there are four main research directions according to [START_REF] Rohrbach | Translating video content to natural language descriptions[END_REF] (1) generating descriptions for images and videos which already contain some associated texts [START_REF] Aker | Generating image descriptions using dependency relational patterns[END_REF]; [START_REF] Feng | How many words is a picture worth? automatic caption generation for news images[END_REF]];

(2) generating descriptions with manually defined rules or templates [START_REF] Tan | Towards textually describing complex video contents with audio-visual concept classifiers[END_REF][START_REF] Guadarrama | Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition[END_REF]];

(3) retrieving existing descriptions from similar visual content [START_REF] Farhadi | Every picture tells a story: Generating sentences from images[END_REF]; [START_REF] Ordonez | Im2text: Describing images using 1 million captioned photographs[END_REF]]; (4) learning a language model from training corpus to generate descriptions [START_REF] Kulkarni | Baby talk: Understanding and generating simple image descriptions[END_REF]; [START_REF] Kuznetsova | Collective generation of natural image descriptions[END_REF]]. However, these current work did not focus on description generation from wearable sensor data (e.g., accelerometers, audio, etc.), which is much more limited information than images or videos. each step t is defined by a time-sliced that segments raw signal from start time t i to end time t j has an input vector x t ij (x t ij is composed by 3-axis signal of sensor data), and a label of sensor data y t ij (one-hot vector) is the most frequent label that is represented in the time duration t i to t j , and produce a hidden state h t ij . The function of LSTM is defined as below: , where W o ∈ R n L Xn H . The dimension n L are the size of labels. Afterward, we combine simultaneously labels into semantic concepts with its start and end times. Consequently, we successfully capture a chronological semantic concept that composes of three conceptual components gesture, activity, the location from sensor data.

g t ij = si gmoi d(W g x t ij + U g h t ij -1 + b g ) s t ij = tanh(W s x t ij + U s h t ij -1 + b s ) f t ij = si gmoi d(W f x t ij + U f h t ij -1 + b f ) ŝt ij = f t ij * s t ij -1 + g t ij * s t ij o t ij = si gmoi d(W o x t ij + U o h t ij -1 + b o ) h t ij = o t ij * r elu( ŝt ij ) where W g , W s , W f , W o ∈ R n H Xn I and U g , U s , U f , U o ∈ R n H

Scripts Generation from Semantic Concepts

In this section, we will introduce the technique to generate scripts from the semantic concepts that we detected by using deep learning neural network model, which is described in section 5.3.1. First, we will generate target sentences/sentence descriptions from the semantic concepts by manually defined detection rules. Second, we will enrich the semantic and syntactic of the target sentence by learning sequence to sequence model.

Scripts Generation from Manually Detection Templates

In recent years, manually defined rules and templates were used in several studies [START_REF] Tan | Towards textually describing complex video contents with audio-visual concept classifiers[END_REF]; [START_REF] Kulkarni | Baby talk: Understanding and generating simple image descriptions[END_REF]; [START_REF] Kuznetsova | Collective generation of natural image descriptions[END_REF]], which allowed to generate language from extracted semantic concepts from visual content (i.e., images, videos). In this section, we follow this technique in order to generate target language/sentences from the semantic concepts, which are extracted from outputs of recognition by using deep learning neural networks as presented in the section 5.3.1. To generate target sentences/ language, we consider a triplet of concepts: human gesture concept (e.g., walking), human action concept (e.g., meal preparation) and location concept (e.g., home) the places where the action was done. Simple rule-based methods rely on these concepts. Our first template is based on the subject who is known as first-person wore the wearable sensors, therefore to be more generic the subject phrase is "the person". Then, it is concatenated with the human gesture, action and location concepts according to identified gesture/action/location concepts. For instance, if "sitting" and "meal preparation" are detected simultaneously, then we form a sentence likes "the person sits and prepares a meal". If the location "home" is identified, then we output "the person sits and prepares the meal at home". Sometimes, the location concepts might be identified but gesture and action concepts can be missing. In this case, we use a phrase to present location setting only such as "the person is at home". The figure 5.3 presents the simple rules for the detection templates. As a result, the target sentences/language was created by using target sentences named as (s 1 , s 2 ..., sm). The function of RNNs defined as follow:

h t = δ h (W h x t + U h h t-1 + b h ) y t = δ y (W y h t + b y )
However, with the simple decoder seq2seq might not be efficient to generate corrected words for the sequence of words. Therefore, in this seq2seq model, we also apply attention decoder [START_REF] Qader | Generation of company descriptions using concept-to-text and text-to-text deep models: dataset collection and systems evaluation[END_REF]; [START_REF] Britz | Efficient attention using a fixed-size memory representation[END_REF]] by multiplied attention weights with encoder output vectors to produce the weighted combination. This result obtains a specific part of the input sequence; therefore, decoder attention can choose the right words to generate. We follow [START_REF] Qader | Generation of company descriptions using concept-to-text and text-to-text deep models: dataset collection and systems evaluation[END_REF] to produce attention weights in decoder by calculating the following Equation 5.1.

α ij = exp (e ij ) Tx k=1 exp (e ik ) (5.1)
where e ij is computed as follow: e ij = f (s i-1 , h j ), e ij represents an alignment model that decoder at step i which parts of hidden state of the input sequence to be attended.

The alignment model f can be a simple feed-forward neural network. Then, the context vector c at step i should be updated on the sequence length T x as Equation 5.2.

c ij = Tx j=1 α ij h j (5.2)
The entire process of the seq2seq model using for automatic scripts generation is shown in Figure 5.4. We evaluate our proposed method using datasets of activity daily livings from Sztyler et al. (2016), which are collected from seven subjects, that recorded manually their daily routine including posture, activity, and location for several days. .2: List of names of semantic concepts used for gesture, activity and location our DNN using Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], with a learning rate of 0.001 and a batch size of 256. We used the cross-entropy loss for a multi-class label of gesture, activity, and location. Then, we predicted labels of semantic concepts with leave-one cross-days training, which trains a model on several days and tested it with another day of data. For the preliminary experiment, we only chose subject 2 from seven subjects in the datasets Sztyler et al. (2016) for our experiment. In subject 2 datasets, the data was collected for thirteen days. In order to test the semantic concepts and scripts generation in one day, we train our DNN on several days (e.g., day 1, 3-9, 12, 13, 15, 16) and test on day 2. The classification performance (F1-score) of gesture, activity, location labels were 0.95, 0.76, 0.94, respectively. As a result, we obtained the labels of 29075 gestures, 29075 activities, 29075 of locations with a time-sliced of 40 and no-overlap between segments from the test set. Afterward, we merged these labels to create semantic concepts. There are two types of concepts that we defined after semantic generation: manually labeled that was collected by the subject and predicted ones in test set, where we consider as TrueConcepts and PredConcepts, respectively. In the next section, we will introduce scripts generation performance from the semantic concepts.

Experiments

Scripts Generation Performance

We generated a sequence of events in terms of scripts using semantic concepts made from sensor data as presented in the proposed method in section 5.3.2. First, we obtained the manually semantic concepts as described above with 29075 semantic concepts, we consider it as the source language for the encoder of the seq2seq model. Second, we made the target language (sc 1 , sc 2 , ..., sc m ) for creating a given input decoder of the seq2seq model by using the manual detection templates on extracting semantic concepts. For the training seq2seq model, we used both semantic concepts and target language (sc 1 , sc 2 , ..., sc m )

of TrueConcepts for encoder and decoder, respectively. For testing seq2seq model, both semantic concepts and target language (sc 1 , sc 2 , ..., sc m ) of PredConcepts were used.

As a result, we obtained target outputs (sc 1 , sc 2 , ..., sc m ′ ) from test set. Finally, we evaluated target outputs that were generated from the seq2seq model, which is using common metrics: BLEU-1, BLEU-2, BLEU-3 and BLEU-4 in both PredConcepts and TrueConcepts. BLEU is a metric for evaluating a generated sentence to a reference sentence. It is one of the first metrics to claim a high correlation with human judgments of quality. BLEU's output is always a number between 0 and 1. This value indicates how Table 5.3 shows an excerpt from the generation of scripts using the seq2seq model. As we can see from the figure, although the seq2seq model can achieve to generate scripts in terms of human-readable text. However, there are repetitive sentences or events in the scripts, hence it is still needed to consider the technique to merge the events (sentences) together.

Conclusion

In this work, we proposed a novel technique that can generate scripts from human activity recognition using wearable sensor data. As we have known, the HAR systems only allow

Chapter 6

Learning Scripts from Natural Language

Texts

There are difficulties because of a lack of data to extract information from sensor data and it is also very hard to learn events from sensor data. Therefore, in this chapter, we present an approach to learn events from natural language text. In section 6.1 we present an introduction about the scripts learning in Natural Language Processing domain, and its related works in section 6.2. In section 6.3 we show our proposed methods, and the experimental results are illustrated in section 6.4. The chapter ends with a short discussion and an outlook of future work in Section 6.5.

Introduction

Scripts are thought to be represented in our minds, which present much of our common sense knowledge about the world (such as what usually happens when going to a restaurant). Script theory was first introduced by [START_REF] Tomkins | Script theory: differential magnification of affects[END_REF] who claims that human behavior mostly follows patterns called Scripts which are stereotypical likely sequences of events. Automatically acquiring script models would be useful not only to get a better insight about the human mind but also to build systems which could communicate with human more naturally.

Take for instance the situation where someone says "I am going to a restaurant". The human recipient of this message will naturally infer that this will take some time and that this person will have to choose dishes. This knowledge about the fact that, at a restaurant, people read the menu and wait for their order, is naturally shared between the protagonists of the conversation. However, this implicit knowledge must be made explicit to machines to interpret the situation. Such a script could be learned from the text. Let's take the Early work on script learning focused on manual construction of script knowledge bases [START_REF] Schank | Scripts, plans and knowledge[END_REF], but in recent years there was a strong research interest in the automatic induction of script knowledge from text, for example [START_REF] Chambers | Unsupervised learning of narrative event chains[END_REF]; [START_REF] Pichotta | Statistical script learning with multi-argument events[END_REF]] investigated studies on naturally occurring texts or crowd-sourced data [START_REF] Regneri | Learning script knowledge with web experiments[END_REF]; [START_REF] Li | Scheherazade: Crowd-powered interactive narrative generation[END_REF]]. Most of these approaches represent the sequence of events as a graph. Each node of the graph is an event, which is modeled as a verbal predicate with its arguments, and arcs correspond to the sequential relation. A graph models common-sense knowledge about the stereotypical sequence of events more than their temporal order 1 . However, most of the recent approaches extracted events from texts using count-based techniques which are usually limited in their modeling due to their poor handling of data sparsity. Indeed, such approaches can capture dependencies only between events that have shared entities (e.g. protagonist) based on probability models acquired from their co-occurrences (count-based method). To avoid this problem, different researchers [START_REF] Modi | Event embeddings for semantic script modeling[END_REF]; [START_REF] Dasigi | Modeling newswire events using neural networks for anomaly detection[END_REF]] have proposed an event predicate-argument representation based on embedding modeling. The advantage of the event embeddings is to capture the semantic properties of events, i.e., the events with different textual surface forms but which are used in the same context will receive close embeddings. However, one disadvantage of this approach is that it requires prepossessing of the semantics content of the text which is still a challenging task. Furthermore, few approaches exploit the syntactic structure of the event text. For instance, the distance between "John eats the food" and "John is on the table" should be large. Hence when learning the embeddings of "John eats the food on the table" far less weight must be given to the prepositional phrase "on the table".

In this chapter, we present a new technique to model the embeddings of events that is resistant to syntactic variations. This learning is based on Recursive Neural Network (RNN) and arguments. An advantage of our proposed model is that it can effectively deal with sparsity in semantic space by representing meaning at a higher level of abstraction than the surface forms of words. Moreover, our approach does not rely on extra domain knowledge such as wordnet but only on predefined word embeddings. Results on a publicly available dataset provided by [START_REF] Regneri | Learning script knowledge with web experiments[END_REF] shows that one version of NESS (only verbs) can provide performance superior to the state-of-the-art baseline systems.

Our approach is focused on learning the stereotypical order of events from a set of texts. In that sense, it is a different task than reconstructing temporal ordering (cf.

SEM EVAL Bethard et al. (2017)). Furthermore, this modeling does not capture all the information of the script such as the one that can be useful for predicting the next events (see the narrative Cloze Task [START_REF] Chambers | Unsupervised learning of narrative event chains[END_REF]). However, our model can be extended to model other aspects such as sequence generation, event prediction, and story generation. But we leave these extensions for further work.

Contributions. Our contributions are summarized as follows.

• A technique based on Recursive Neural Network (RNN) for Scripts learning, namely NESS;

• An experiment with a comparison with baseline models [START_REF] Regneri | Learning script knowledge with web experiments[END_REF]; [START_REF] Modi | Inducing neural models of script knowledge[END_REF]] that shows the impact of event embeddings based on syntactic and semantic features.

The chapter is organized as follows. Section 6.2 presents a summary of the state of the art in learning techniques with Scripts. NESS modeling techniques are detailed in Section 6.3. Experiments are reported in Section 6.4. The chapter ends with a short conclusion and an outlook on future work.

Related work

Scripts have been popularized in AI by [START_REF] Schank | Scripts, plans and knowledge[END_REF] who introduced it as a method for representing procedural knowledge. At that time, event sequences representing scripts were manually encoded in knowledge bases to perform tasks such as event inference, event generation, etc. Nowadays there are a lot of researches on script induction For the event representation phase, the system from [START_REF] Chambers | Unsupervised learning of narrative event chains[END_REF] presents narrative events as pairs of the form (ev ent, dependency ), where the event is represented by a verb and the dependency represents typed dependency relations between event and a protagonist such as subject and object. The event chain is formed by collecting events sharing a common protagonist from texts using a syntactic parser and a co-reference system. The system achieves impressive performance to classify temporal relations between given event descriptions. Another technique introduced by Regneri et al.

(2010) considers the whole predicate-argument structure as an atomic unit. But these representations suffer from the sparsity issue which would limit its applicability to machine learning. Another approach from the system of [START_REF] Balasubramanian | Generating coherent event schemas at scale[END_REF] proposed

Rel-grams to represent events as a triples (ar g 1 , r elati on, ar g 2 ), where ar g 1 and ar g 2 represent the subject and object respectively, to overcome the lack of coherence of protagonist representation of event chains from the system of [START_REF] Chambers | Unsupervised learning of narrative event chains[END_REF].

In [START_REF] Pichotta | Statistical script learning with multi-argument events[END_REF], an event is represented as a tuples of v (e s , e o , e p ), where v is a verb lemma, e s is the subject, e o is the object, and e p is an entity with prepositional relation to v . An advantage of multi-arguments event model is that it allows encoding a richer representation of events in comparison with previous methods. Therefore, it has been used in recent works such as [START_REF] Modi | Event embeddings for semantic script modeling[END_REF]; [START_REF] Granroth-Wilding | What happens next? event prediction using a compositional neural network model[END_REF]]. Another approach of event representation is that one can use a compositional model based on recursive neural networks [START_REF] Socher | Semantic compositionality through recursive matrix-vector spaces[END_REF], which learns compositional vector representations for phrases and sentences of arbitrary syntactic type and length and which has been shown successfully in detecting atypical events in news [START_REF] Dasigi | Modeling newswire events using neural networks for anomaly detection[END_REF].

For the sequence modeling phase, existing models can be considered into two main methods: weak-order, and strong-order. The former studied in the relations between pairs of events, and the latter investigated the temporal order of events in a full sequence.

Event-pair models used discrete event representations and estimated event relations by statistical counting as [START_REF] Chambers | Unsupervised learning of narrative event chains[END_REF] used Pairwise Mutual Information (PMI) to calculate event relations, and while most sub-sequence models followed Jans

Learning Model

The model used for learning is based on the Recurrent Neural Network architecture which encodes the input sequence into the fixed-length vector. This model is able to treat a sequence of variable size and has become the standard approach for many tasks in particular in Natural Language Processing tasks Sutskever et al. (2014a). Briefly, a recurrent unit, at each step t takes an input x t and a previously hidden state h t-1 and compute its hidden state and the output using:

h t = σ h (W h x t + U h h t-1 + b h ), y t = σ y (W y h t + b y ),
where y t is the output vector at each step; W, U, b are the parameters of the neural layer and σ h and σ y the activation functions of the neural layers. Numerous improvements have been made to this architecture such as using mono or multi layer of Long Short-Term Memory (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF] to prevent the exploding/vanishing gradient problem and to model long dependencies in the sequence.

An LSTM is used to map a sequences of existing events {e 1 , ..., e n } into hidden vectors {h 1 , ..., h n }, which encode the order. Pair of hidden vector linked to 2 events are merged to form u is computed as a composition of arguments (cf. section 6.3.2). This is then passed to a feed forward layer which in turn feed the output layer y which is composed of a single neuron and softmax activation function to make a binary classification (follow or not follow).

Event Representation

Input events are a sequence of natural language sentences. In order to represent an event in a numerical space, a sentence is parsed as a syntactic tree where each node is labeled with its linguistic constituent and the leave by their POS tag (part-Of-Speech). For instance, as exemplified in Figure 6.2, the first event e 0 is "John eats the food on the table". All the word will be attributed their POS tag: "John (NN) eats (VB) the (DT ) food (NN) on To transform this syntactic structure into a numerical vector representation, we take an argument representation approach. Using this approach, the embeddings of the event can be represented as a single vector e, which is composed of representations of individual words that are guided by the POS tagging.

same context is a negative training example.

s = S T u (6.2)

The optimization of θ ar g is obtained by Equation 6.3: ar g min θarg max(0, 1s e i + s e j + s u ) (6.3)

where s e i and s e j are the scores of the composition of the entire argument produce by the root node of argument as the respectively to events e i and e j , and s u is the score produced by randomly replacing one of the words in the argument at a time. where α is the reference binary label determining whether the event e i follows the event e j or not, and the h(e i ) is the output of hyperbolic tangent function define as following:

h(e i ) = We implement the functions and using mini-batch (size = 128) stochastic gradient descent with adam learning [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] schedule.

6.4 Experimental results

Data

To evaluate the approach, we selected the crowd-sourced data provided by [START_REF] Regneri | Learning script knowledge with web experiments[END_REF]. They collected events sequence descriptions (ESDs) of different kinds of human daily activities (e.g. visit a restaurant, cooking eggs, etc.). Although this dataset is small (only 30 ESDs per activity) it is the only one available to test script learning. The events were extracted from the original dataset using an automatic method for marking up temporal relations in natural language texts [START_REF] Verhagen | Automating temporal annotation with TARSQI[END_REF]. These temporal relations follow TimeML (www.timeml.org). The TimeML scheme flags tensed verbs, adjectives, and nominals with EVENT tags with various attributes such as the class of event, tense, grammatical aspect, part-of-speech, and cardinality of the event if it has appeared more than one. For temporal relation annotation, there are 14 temporal relations in TLINK Chapter 7

Conclusion and Perspectives

Conclusion

In this thesis, story generation from ambient sensors is approached by a pipeline of abstraction: from raw data to a sequence of sentences summarizing the 'story' of this data.

The first step of the pipeline focused on Human Activity Recognition from smartphone data. Indeed, daily life story being about what people do, being able to extract human activity is a necessary step to reach this goal. We approached HAR by machine learning.

However, conventional machine learning approaches are sensitive to the problem of imbalanced data. This why our first contribution was a framework for solving the imbalanced data issue. This method enhanced the performance of the conventional machine learning model (e.g., Multilayer Perceptron) for activity classification tasks. However, conventional machine learning models rely on feature engineering to extract parameters from raw data.

Therefore, we subsequently used a deep learning approach in order to recognize high-level human activities from raw data. Furthermore, the final text was also generated using a sequence-to-sequence deep model using a concept-to-text approach [START_REF] Qader | Generation of company descriptions using concept-to-text and text-to-text deep models: dataset collection and systems evaluation[END_REF].

These two models used together gave an initial system that can automatically generate scripted texts from wearable sensor data using HAR. Secondly, we introduce an event presentation of script modeling from natural language text on the Natural Language Processing (NLP) domain, which enables us to classify the order of sequence of events so as to predict what event happened next in natural language text. Moreover, with this approach, the model can also predict missing event [START_REF] Modi | Event embeddings for semantic script modeling[END_REF], and generate sub next event in natural text [START_REF] Hu | What happens next? future subevent prediction using contextual hierarchical LSTM[END_REF]. However, we let this work for future research. Finally, most current state-of-the-art on both domains of the HAR system and script generation (SG) on NLP has not provided any approach to generate story/script from sensors data.

Therefore, different from currently state-of-the-art, we propose a novel approach to invent a system, which allows generating script from HAR using wearable sensors data.

The two main reasons why we have to connect from the HAR domain to script generation on the NLP domain: (1) firstly, due to the segmentation of raw sensors data in order to build the input, which then is fed into the input layer of deep learning for HAR.

It leads to the issue of an event would be missing within the sequence of events. As a result, prediction of missing events in the sequence of events by using script modeling will assist to solve this problem; (2) secondly, since the limitation of raw signals recording using sensors, the training, and test data are also limited for prediction tasks on HAR system.

Therefore, the prediction of an event using script learning from the natural text would replace the limited raw sensor data, and it also provides a new approach for the HAR task.

In summary, this thesis contributes novel approaches to support for limitations of both HAR and NLP domains. For the HAR domain, script learning can assist the HAR task by predicting missing events and the next event in the sequence of events. For script learning, the HAR task provides an approach to generate a script using deep learning from sensors data. Moreover, the thesis also provides an approach to connect from HAR to the NLP domain by generating script from sensors data.

Further work and open challenges

Although the thesis has achieved preliminary results to automatically generate scripts/story from wearable sensor data using HAR. Nevertheless, there are issues and open challenges we would like to address for future work.

Firstly, as mentioned in the conclusion of chapter 5, our script generation method using deep learning from wearable sensor data has still suffered from repetitive events.

This problem happened because of the segmentation method of raw signals from sensor data. Consequently, we have to find a relevant way to deal with this issue.

Secondly, although we can achieve to use seq2seq with an attention model, which allows the generating model to choose the right words to generate. In addition, it also enriches the semantic and vocabulary of the targeted sentence. However, the corpus of source and target languages are very limited in this thesis, especially in the case of HAR from sensor data. Therefore, we need to find a relevant approach to build richer vocabulary for both source and target language.

Thirdly, it is also interesting to engage other information into our overall approaches such as emotion (e.g., sad, happy, etc.) and health status (e.g., fever, hypothermia, etc.).

Obviously, this information would combine with human daily activity and their location so as to make audiences deeply understand the story. Moreover, with this combined information our current semantic concepts daily activity location also can be naturally enriched, which would solve for the limitation of semantic concepts in our current issues.

Therefore, emotion recognition [START_REF] Tarnowski | Emotion recognition using facial expressions[END_REF]; [START_REF] Chaparro | Emotion recognition from EEG and facial expressions: a multimodal approach[END_REF]] from facial expression or [START_REF] Batbaatar | Semantic-emotion neural network for emotion recognition from text[END_REF]; [START_REF] Shaheen | Emotion recognition from text based on automatically generated rules[END_REF]] from the text are extensions for our future works. Besides, health status detection [START_REF] Chen | Health status detection for patients in physiological monitoring[END_REF] from physiological monitoring also can be considered as another open challenge for our future works. Furthermore, the health status may assist us to deeper understand human decision-making, which relates to the human context. For instance, the person forgot his jacket during the mountain climbing, his body temperature went down quickly. Therefore, he decided to come back instead of going up to the peak.

Finally, we would like to point out the usefulness of our thesis to the real world. First of all, the thesis gives a novel approach to translate sensors data (e.g., numeric and binary)

to text in term of script, which allows audiences to an insight deeply about the sequence of events in the daily activity domain (e.g., smart-home, outdoor environment). It can be applied to health-monitoring in the home environment for elderly people, in which the scripts would be summarized human activities. From the analysis of historical script or summarized activities, it can assist to detect routine behaviour for elderly people. From routine behaviour analysis, we can develop a recommended system to encourage human activity changing. Moreover, we also can investigate emergency alarming system in order to warn unusual changing activity.

  4.3.3 Active Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.4 Oversampling Border Limited Link SMOTE Method . . . . . . . 4.3.5 Implemented Framework . . . . . . . . . . . . . . . . . . . . . . 4.4 Experimental Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.2 Baseline results with the MLP . . . . . . . . . . . . . . . . . . . 4.4.3 MLP learning with BLL SMOTE . . . . . . . . . . . . . . . . . . 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Scripts Generation from Events 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Proposed Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Semantic Concepts Generation with Deep Learning Neural Network Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 Scripts Generation from Semantic Concepts . . . . . . . . . . . 5.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Experimental Settings and Results . . . . . . . . . . . . . . . . . 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Learning Scripts from Natural Language Texts 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Proposed Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.1 Learning Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.2 Event Representation . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 Baseline methods . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Conclusion and Perspectives 7.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Further work and open challenges . . . . . . . . . . . . . . . . . . . . . List of Figures 1.1 Relationship between data and their representation modality. . . . . . . . 1.2 Example of story written by a user (right side) and the smart-phone data was collected during an experiment (left side). . . . . . . . . . . . . . . 2.1 An Illustration of sensor-based activity recognition using conventional machine learning approaches Wang et al. (2019). . . . . . . . . . . . . . . 2.2 An Illustration of sensor-based activity recognition using deep learning approaches Wang et al. (2019). . . . . . . . . . . . . . . . . . . . . . . . 2.3 Architecture of an auto-encoder. . . . . . . . . . . . . . . . . . . . . . 2.4 Architecture of a stacked auto-encoder Chen et al. (2018) . . . . . . . . 3.1 An overview of the story generation framework from sensor data with 4 main parts including Activity Recognition; Structured event generation; Text generation and Script learning. Structured event generation is not covered in this thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Events sequence of bus taking . . . . . . . . . . . . . . . . . . . . . . . 4.1 The active oversampling framework . . . . . . . . . . . . . . . . . . . . 4.2 Multilayer Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Example of mis-generation synthetic instance in non-convex dataset . . 4.4 Distribution of the activity labels over the datasets . . . . . . . . . . . . 4.5 Baseline learning curve of the MLP on LIG-SPHAD (left) and ExtraSensory Dataset (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 MLP + Active Learning on LIG-SPHAD (left) and ExtraSensory Dataset (right). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Last step of Multilayer Perceptron after last query of Active Learning and Over-sampling BLL SMOTE on LIG-SPHAD (left) and ExtraSensory Dataset (right) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Illustration of generating scripts from human activity using wearable sensor data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Illustration of generating semantic concepts (e.g., gesture) from human activity using deep neural network. . . . . . . . . . . . . . . . . . . . . 5.3 Illustration of scripts generation from manually detection templates. . . 5.4 Illustration of seq2seq model for enrich semantic and syntactic of scripts generation from semantic concepts. . . . . . . . . . . . . . . . . . . . . 5.5 Collector and labeling framework: Wear (smart-watch) Hand (smart-phone). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Sample rows of original dataset. . . . . . . . . . . . . . . . . . . . . . . 5.7 Performances of scripts generation using seq2seq model for TrueConcepts and PredConcepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Events sequence of restaurant visiting . . . . . . . . . . . . . . . . . . . 6.2 NESS model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Composition of argument backed by a binary tree . . . . . . . . . . . .

2. 1

 1 Event sequence of laundry . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Types of activities recognized by state-of-the-art HAR systems . . . . . 2.3 Group of features extraction methods . . . . . . . . . . . . . . . . . . . 2.4 Categories of attributes and features extraction methods . . . . . . . . . 2.5 Classification algorithms used by state of the art HAR systems . . . . . . 2.6 Summary of state-of-the-art in online HAR systems . . . . . . . . . . . . 2.7 Summary of state-of-the-art in offline HAR systems . . . . . . . . . . . . 2.8 Deep learning models for HAR systems surveyed by Wang et al. (2019) . 5.1 Event sequence of restaurant visiting . . . . . . . . . . . . . . . . . . . 5.2 List of names of semantic concepts used for gesture, activity and location 5.3 An excerpt of scripts generation from the model . . . . . . . . . . . . . 6.1 Results on the datasets Regneri et al. (2010) for the verb-frequency baseline (BL), the verb-only embedding model (EE v er b ), Regneri et al. (2010) (MSA), Frermann et al. (2014) (BS), Modi and Titov (2014) full model (EE), the full model NESS 1 and verb-only embedding model (NESS 2 ). Results with models other than NESS are extracted from Modi and Titov (2014) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Generation at the Age of Big Data

  At 10:30 her mother told her to go to supermarket. Then, she seemed very exciting with it, and went to prepare her dress quickly.At 10:50 They left home and went shopping in the supermarket.

Figure 1 . 2 :

 12 Figure 1.2: Example of story written by a user (right side) and the smart-phone data was collected during an experiment (left side).

(

  or evaluation). The training stage initially requires a time series dataset of measured attributes from individuals performing each activity. The time series are divided into time windows to apply feature extraction in order to filter relevant information in the raw signals and define metrics to compare them. Later, learning methods are used to generate an activity recognition model from the dataset of extracted features. Likewise, for testing, data are collected during a time window and a feature vector -also called feature set -is calculated. Such a feature set is evaluated in the a priori trained learning model, generating a predicted activity label. The data collected from wearable sensors are naturally indexed over the time dimension, which allows us to define the human activity recognition problem as follows.

Figure 2

 2 Figure 2.1: An Illustration of sensor-based activity recognition using conventional machine learning approaches Wang et al. (2019).

2

  AMB: Ambulation activities, DA: Daily activities, TR: Transition between activities.3 C4.5: Decision Tree, LR: Logistic Regression, ALR: Additive Logistic Regression, MLP: Multilayer Perceptron, NB: Naive Bayes, SVM: Support Vector Machines, RF: Random Forest, LB: LogitBoost, HMM: Hidden Markov Model.

  Lara et al. (2012) proposed a system that combines acceleration data with vital signs to achieve accurate activity recognition. Centinela recognizes five ambulation activities and includes a portable and unobtrusive real-time data collection platform, which only requires a single sensing device and a mobile phone. Time-and frequency-domain features are extracted from acceleration signals while polynomial regression and transient features are applied to physiological signals. After evaluating eight different classifiers and three different time window sizes, and six feature subsets, Centinela achieves an overall accuracy of over 95%. The results also indicate that incorporating physiological signals allows for a significant improvement of the classification accuracy. As a trade-off, Centinela relies on ensembles of classifiers accounting for higher computational cost, and it requires wireless communication with an external sensor, which increases energy expenditures. Bayat et al. (2014) introduced a recognition system based on acceleration data, which used a new low-pass filter to detach the component of gravity acceleration from body acceleration in raw data. There are 18 features were selected from 24 features extracted along with a window size of 128 samples and 50% overlap. Six daily activities were classified in several classification models such as MLP, LB, SVM, RF. The achieved accuracy by an average of these models was 91.15%. Saputri et al. (2014) proposed a neural network with a three-stage genetic algorithmbased feature selection to solve for the flexibility or user-independent issue in HAR. Time-

Figure 2 . 2 :

 22 Figure 2.2: An Illustration of sensor-based activity recognition using deep learning approaches Wang et al. (2019).

[

  [START_REF] Zebin | Inertial sensor based modelling of human activity classes: Feature extraction and multi-sensor data fusion using machine learning algorithms[END_REF];Zeng et al. (2014a);[START_REF] Ha | Convolutional neural networks for human activity recognition using multiple accelerometer and gyroscope sensors[END_REF]].

Figure 2

 2 Figure 2.4: Architecture of a stacked auto-encoder Chen et al. (2018)

  [START_REF] Zhang | Real-time activity recognition on smartphones using deep neural networks[END_REF] introduced a real-time application for activity recognition on smartphones using DBNs. In pre-train, the input to DBNs is acceleration data and forms a Gaussian-binary RBMs between input layer and first hidden layer. After pre-train, a soft-max layer is added to classify seven activities(e.g., walking, running, lying , etc.). The DBNs classification model achieved the highest accuracy in comparison with other learning models such as Bayesian nets, SVM, LR, J48, RF, and KNN.[START_REF] Hammerla | PD disease state assessment in naturalistic environments using deep learning[END_REF] proposed a system evaluates Parkinson's Disease (PD) in naturalistic environments (e.g., the daily life of individual affected by PD). A total of approximately 5500 hours of accelerometer data was collected by 34 participants in a home environment, in order to detect for 4 classes: asleep, on , off and dyskinesia. From the raw signal 91 hand-crafted features were extracted in each minute. Then, in the first step of training, the features are normalized and fed into RMBs to reconstruct the input features. In the second step of training, a next Gaussian-binary RBMs layer was taken the activation probabilities of the first RBMs as its input data. Finally, a softmax layer was added as a top-layer to classify the four classes.Recurrent neural networkRecurrent neural network (RNN) is a class of artificial neural networks, where each neuron connected to create a directed graph along a temporal sequence. Long-Short-Term-Memory (LSTM) is an artificial RNN architecture, which deals with explode and vanishing gradient problems encountered in RNN. In recent years, there are few researches on RNN and LSTM for HAR tasks[START_REF] Hammerla | Deep, convolutional, and recurrent models for human activity recognition using wearables[END_REF];[START_REF] Edel | Binarized-blstm-rnn based human activity recognition[END_REF];[START_REF] Murad | Deep recurrent neural networks for human activity recognition[END_REF]]. For instance,[START_REF] Edel | Binarized-blstm-rnn based human activity recognition[END_REF] presented a binarized BLSTM-RNN system, which re-defined all parameters (e.g., weight, input, output, hidden layers, etc.) in the network in term of binary values. Hence, the model only computed on bite-wise operation instead of using arithmetic operations. As a result, the system reduced the high computational cost of traditional RNN-LSTM, while it still remains a good accuracy for the HAR task.[START_REF] Murad | Deep recurrent neural networks for human activity recognition[END_REF] proposed a deep recurrent neural network (DRNN), which is able to capture long-range dependencies in variables-length input sequences. Their experiments showed that DRNN outperforms other classifications such as DBNs, CNN, kNN, SVM in benchmark datasets.Hybrid modelA hybrid model is the connection of different types of deep learning models together such as CNN and RNN[START_REF] Morales | Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition[END_REF];[START_REF] Yao | Deepsense: A unified deep learning framework for time-series mobile sensing data processing[END_REF]]. For instance,[START_REF] Yao | Deepsense: A unified deep learning framework for time-series mobile sensing data processing[END_REF] proposed a deep learning framework, namely DeepSense, which combines convolutional neural network and recurrent neural network so as to perform three tasks: car tracking with motion sensors, human activity recognition and user identification with biometric motion analysis. It is shown that CNN is responsible for capturing the spa-tial relationship, and RNN can achieve to obtain for the temporal relationship. Hence, integration of CNN and RNN can boost the activity recognition task that have varied time span and signal distribution. Similarly,[START_REF] Morales | Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition[END_REF] also proposed a framework (DeepConvLSTM) based on convolutional and LSTM neural networks, which allowed to perform sensor fusion naturally and automatically extracted features using CNN, after that LSTM was used for learning the temporal dynamic of feature activation. As a result, DeepConvLSTM was compared with deep non-recurrent networks and previous learning models on two well-known public HAR datasets (OPPORTUNITY, Skoda). The result demonstrated that DeepConvLSTM outperformed these learning models. Other researches integrate CNN with other models such as SAE[START_REF] Zheng | Exploiting multi-channels deep convolutional neural networks for multivariate time series classification[END_REF] and RBM[START_REF] Liu | Lasagna: towards deep hierarchical understanding and searching over mobile sensing data[END_REF]. In these works, CNN performed features extraction and generative models assist to adjust the input data for the training process. There is an expectation to expanded researches for this area in the future.

From a data abstraction

  point of view, the story is about what people do and what they feel during daily life. Hence, in order to generate a story, we need to know what kinds of activity are of interest, and what are the methods to automatically extract them.In recent years with the advanced technology of smart-phone sensors (e.g., capacity increase, cost efficiency, power efficiency, etc.) it became possible to use it for human activity recognition (HAR) purposes[START_REF] Bayat | A study on human activity recognition using accelerometer data from smartphones[END_REF];[START_REF] Kwapisz | Activity recognition using cell phone accelerometers[END_REF][START_REF] Anguita | Energy efficient smartphone-based activity recognition using fixed-point arithmetic[END_REF]]. Indeed, HAR plays an important role in automatically story generation that allows us to automatically identify and discover what and when people are doing during their daily life. Moreover, Recognizing Human Activity from sensors data can assist human to predict physical activity in everyday life such as walking, running and working at a computer, etc.[START_REF] Lara | A Survey on Human Activity Recognition using Wearable Sensors[END_REF];[START_REF] Bayat | A study on human activity recognition using accelerometer data from smartphones[END_REF];[START_REF] Blachon | On-line context aware physical activity recognition from the accelerometer and audio sensors of smartphones[END_REF];[START_REF] Kwapisz | Activity recognition using cell phone accelerometers[END_REF]]. Therefore, HAR is a core requirement in this thesis in order to generate a story.

  Many instance-based classifiers have been used in the field, such as Bayesian Network Lara et al. (2012), Decision Trees Lara et al. (2012); Blachon et al. (2014), Random Forest Blachon et al. (2014), Artificial Neural Network (ANN) [Khan et al. (2010); Kwapisz et al. (2011); Lara et al. (2012)], Support Vector Machines (SVM) [Lara et al. (2012); Anguita et al. (2012)], etc. Since human activities can be seen as a sequence of smaller sub-activities, sequential models such as Conditional Random Fields Blachon et al. (2014), Hidden Markov Model Zhu and Sheng (2009) or Markov Logic Network Chahuara et al. (2016) have also been applied. However, since the advent of Deep Learning, ANN has become of the most popular model in HAR from

Algorithm 1 :

 1 Algorithm MLP AL OS Border Limited Link SMOTE Input: (a) X = {x 1 , ..., x n } : training instances (b) S: pool of training instances for MLP (c) U: pool of unlabeled instances (d) N: number of query Output:

Figure 5 . 2 :

 52 Figure 5.2: Illustration of generating semantic concepts (e.g., gesture) from human activity using deep neural network.

Figure 5

 5 Figure 5.4: Illustration of seq2seq model for enrich semantic and syntactic of scripts generation from semantic concepts.

Figure 5

 5 Figure 5.5: Collector and labeling framework: Wear (smart-watch) Hand (smart-phone).

  Figure 5.6 is depicted for the sample rows of the original dataset. Each line of the raw data corresponds to one measurement, in which the first element is the row index; the second element is a

  similar the candidate text is to the reference texts, with values closer to 1 representing more similar texts. Cumulative scores refer to the calculation of individual n-gram scores at all orders from 1 to n and weighting them by calculating the weighted geometric mean.The cumulative and individual 1-gram BLEU use the same weights, e.g. (1, 0, 0, 0). The 2-gram weights assign a 50% to each of 1-gram and 2-gram and the 3-gram weights are 33% for each of the 1, 2 and 3-gram scores. The weights for the BLEU-4 are 1/4 (25%) or 0.25 for each of the 1-gram, 2-gram, 3-gram, and 4-gram scores. Figure 5.7 shows our evaluation of PredConcepts and TrueConcepts. As we can see from the Figure, the performances of the seq2seq model on predict labels are high with the BLEU approximate 0.96, while the performances of the seq2seq model on true labels are less than predict ones, especially in case BLEU-4 is 0.76.

Figure 5 . 7 :

 57 Figure 5.7: Performances of scripts generation using seq2seq model for TrueConcepts and PredConcepts.

  example of Figure6.1 where two texts (scenarios 1 and 2 on the right side) describe a natural language texts[START_REF] Granroth-Wilding | What happens next? event prediction using a compositional neural network model[END_REF];[START_REF] Modi | Event embeddings for semantic script modeling[END_REF];[START_REF] Modi | Modelling semantic expectation: Using script knowledge for referent prediction[END_REF]]. Moreover, this kind of knowledge can be applied to many domains of artificial intelligence, more specifically on automatic narrative generations[START_REF] Baez Miranda | Task based model for récit generation from sensor data: an early experiment[END_REF];[START_REF] Martin | Event representations for automated story generation with deep neural nets[END_REF]], information extraction[START_REF] Dasigi | Modeling newswire events using neural networks for anomaly detection[END_REF]; Pichotta and Mooney (2016)], and dialogue[START_REF] Harrison | Learning from stories: using natural communication to train believable agents[END_REF].

[

  [START_REF] Chambers | Unsupervised learning of narrative event chains[END_REF];[START_REF] Regneri | Learning script knowledge with web experiments[END_REF];[START_REF] Modi | Inducing neural models of script knowledge[END_REF]; Pichotta and Mooney (2014); Jans et al. (2012); Granroth-Wilding and Clark (2016); Pichotta and Mooney (2016); Modi (2016); Hu et al. (2017); Martin et al. (2018)] from texts. Each research developed different approaches which are generally based on two phases: event representation and modeling to learning the scripts.

  et al. (2012) used skip n-gram. However, counting-based methods suffer from data or event sparsity, therefore more recent work developed embeddings system to tackle this issue. According to the system of Granroth-Wilding and Clark (2016) leveraged the skipgram model of[START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] for training embedding of events and arguments by ordering them into a pseudo sentence.[START_REF] Modi | Event embeddings for semantic script modeling[END_REF] applied word embedding to verbs and arguments directly and automatically unified event embedding into a single structured event embeddings by using a neural network. Another approach is presented by[START_REF] Frermann | A hierarchical bayesian model for unsupervised induction of script knowledge[END_REF] which is based on a hierarchical Bayesian model to build a generative model for joint learning of event types and ordering constraints.

  [START_REF] Socher | Recursive deep models for semantic compositionality over a sentiment treebank[END_REF] used a supervised objective that is based on the label error at the topmost node in the RNN. Event composition takes argument representation and produces the event representation and label indicating whether the event is in order or not. Therefore, the event composition node's parameters θ ev ent = {W ev ent ∈ R n×kn , b ev ent , LB ev ent ∈ R n×1 } where k is the number of arguments per event. LB ev ent is the label operator. The objective of this phase is as Equation6.4: ar g min θevent (-α log h(e i ) × h(e j ) + ((1α) log(1h(e i ) × h(e j ))) (6.4)

Table 2

 2

	.1: Event sequence of laundry

Table 2

 2

	Group	Activities
		Walking, running, sitting, standing still, lying,
	Ambulation	climbing stairs, descending stairs, riding escalator,
		and riding elevator
	Transportation	Riding a bus, cycling, and driving
	Phone usage	Text messaging, making a call
		Eating, drinking, working at the PC, watching TV,
	Daily activities	reading, brushing teeth, stretching, scrubbing, and
		vacuuming
	Exercise/fitness	Rowing, lifting weights, spinning, Nordic walking, and doing push ups
	Military	Crawling, kneeling, situation assessment, and opening a door
	Upper body	Chewing, speaking, swallowing, sighing, and mov-ing the head

.2: Types of activities recognized by state-of-the-art HAR systems light intensity are fairly low allow to recognize that user might be sleeping), location or

Table 2

 2

.3. It depends on the given signal to choose relevant methods were illustrated in Table

2

.4. For instance,

[START_REF] Chen | Online classifier construction algorithm for human activity detection using a tri-axial accelerometer[END_REF]

;

[START_REF] He | Learning from imbalanced data[END_REF]

] the acceleration signals are highly fluctuating and varying, therefore the statistical feature extraction -either time or frequency domain -is the best method to handle with these signals. Besides that,

Lara et al. (2012)

proposed that structural feature extraction -polynomial -is the best fit for the physiological signals such as heart rate, respiration rate, breath amplitude, and skin temperature.

Table 2

 2

	Attributes	Features
	Altitude	Time domain
	Audio	Speech recognizer
	Barometric pressure Time domain and frequency domain
	Humidity	Time domain
	Light	Time domain and frequency domain
	Temperature	Time domain

.3: Group of features extraction methods

Table 2

 2

.4: Categories of attributes and features extraction methods In general, these features extraction methods can influence on recognition accuracy

Table 2

 2 Table 2.5.

	Type	Classifiers
	Bayesian	Naive Bayes and Bayesian Networks Lara et al. (2012)
	Clustering	K-nearest neighbor Lara et al. (2012)
		Multilayer	Perceptron
	Neural Networks	[Kwapisz et al. (2011);
		Bayat et al. (2014)]
		Support Vector Machine
	Kernel Method	[Bayat et al. (2014); An-guita et al. (2012); Khan
		et al. (2014)]
		Fuzzy Basis Function and
	Fuzzy Logic	Fuzzy Inference System
		Berchtold et al. (2010a)
		Multiclass Logistic Regres-
	Regression methods	sion Riboni and Bettini (2011), Additive Logistic
		Regression Lara et al. (2012)
		Hidden Markov Models and
	Markov Models	Conditional Random Fields
		Blachon et al. (2014)
	Classifier ensembles	Boosting and Bagging Lara et al. (2012)

.5: Classification algorithms used by state of the art HAR systems

Table 2

 2 predict the most probable activities. Then, symbolic reasoning is applied to refine the results of statistical inferencing by selecting the set of possible activities performed by a user based on her/his current context. COSAR gathers data from two accelerometers, one in the phone and another on the individual's wrist, as well as from the cellphone's GPS. COSAR uses an interesting concept of potential activity matrix to filter activities based upon the user's location. For instance, if the individual is in the office, he or she is probably not cycling. Another contribution is the statistical classification of activities with a historical variant. For example, if the predictions for the last five time windows were jogging, jogging, walking, jogging, jogging, the third window was likely a misclassification (e.g., due to the user performing some atypical movement) and the algorithm should automatically correct it. However, this introduces an additional delay to the system's response, according to the number of windows analyzed for the historical variant. The time and frequency domain were used to extract 74 features. Later, learning algorithms such as C45 decision tree, Random forest, and Conditional Random Fields were applied to classify the activities. The author reported that the Random Forest was the most performance with an overall F-measure of 74%.

		Sensors 1	Activities 2 Algos 3	Performance
	Berchtold et al. (2010b) ACC	AMB,	RFIS	71-98% accu-
		(phone)	PHO (10)		racy
	Riboni and Bettini (2011) ACC	AMB, DA	COSAR 93% accuracy
		(watch,	(10)
		phone),
		GPS
	Lara and Labrador (2012) ACC, VS	AMB (3)	C4.5	92.6% accu-
		(Chest)			racy
	Blachon et al. (2014)	ACC, Au-	DA (5)	RF,	74%	F-
		dio		CRF	measure

.6: Summary of state-of-the-art in online HAR systems 1 ACC: Accelerometers, VS: Various Sensors, HAR: Heart rate. 2 AMB: Ambulation activities, DA: Daily activities, PHO: Activities related to phone usage. 3 C4.5: Decision Tree, RF: Random Forest, CRF: Conditional Random Fields, RFIS: Recurrent Fuzzy Inference System. Berchtold et al. (2010b) investigated a system, namely ActiServ, for smart-phone based activity recognition. The system enables to integrate labeled annotation with new training data directly on smart-phone at run-time, and using fuzzy inference to classify for ten ambulation activities based on phone's accelerometer feature extraction. The method to recognize ten daily activities such as brushingTeeth, hikingUp, hikingDown, rid-ingBycycile, jogging, standingStill, strolling, WalkingDownstairs, WalkingUpstairs, Writin-gonBlackBoard that statistical methods can not classify alone. The idea is statistical inferencing is performed based on raw data retrieved from body-worn sensors (e.g., ac-celerometers) to

The overall accuracy was roughly 93% using Multiclass Logistic Regression though, in some cases, standing still was confused with writing on a blackboard, as well as hiking up with hiking down.

Lara and Labrador (2012)

proposed Vigilante, a mobile application for real-time human activity recognition under the Android platform. This application used accelerometers and physiological signals (e.g., heart rate, respiration rate, etc.) in order to recognize five ambulation activities. The feature extraction methods statistical and time, frequency domain was used to extract 84 features. Then, several learning algorithms were applied to classify activities such as Bayes Network, C45, MLP, SMO, etc. The author reported that C45 decision tree had the best performance by achieving up to 96.8% accuracy. However, the author also noticed that the models were extremely influenced by the dataset. When they tried to train with a given user and tested it on another user data, the classification accuracy significantly decreased to 64%.

[START_REF] Blachon | On-line context aware physical activity recognition from the accelerometer and audio sensors of smartphones[END_REF]

introduced another online HAR system using the mobile Android platform. They reported that the audio signal can play an important role with accelerometers to recognize nine ambulation activities. in case of large activities recognition. Moreover, Vigilante is only approach to integrate vital signals with accelerometers to allow recognize activities related to the health-care purpose.

Table 2

 2

.7: Summary of state-of-the-art in offline HAR systems 1 ACC: Accelerometers, VS: Various Sensors.

  uses Hidden Markov Models (HMM) to recognize ambulatory activities. Two accelerometers, placed on the subject's wrist and waist, were connected to a PDA via a serial port. The PDA sent the raw data via Bluetooth to a computer that processed the data. This configuration is obtrusive and uncomfortable because the user has to wear wired links that may interfere with the normal course of activities. The extracted features are the angular velocity and the 3D deviation of the acceleration signals. The classification of activities operates in two stages. In the first place, an Artificial Neural Network discriminates among stationary (e.g., sitting

  ). Nevertheless, it is hard to use for recognizing high-level or context-aware activities[START_REF] Yang | Activity recognition: Linking low-level sensors to high-level intelligence[END_REF] (e.g., drinking coffee, have breakfast, etc.); (3) traditional learning approaches often need a large number of labeled data for training model. To tackling these limitations, in recent years there are many types of researches interest in deep learning for HAR issues[Jiang and Yin (2015a);[START_REF] Ronao | Deep convolutional neural networks for human activity recognition with smartphone sensors[END_REF];[START_REF] Murad | Deep recurrent neural networks for human activity recognition[END_REF];[START_REF] Ignatov | Real-time human activity recognition from accelerometer data using convolutional neural networks[END_REF][START_REF] Bevilacqua | Human activity recognition with convolutional neural networks[END_REF]]. Figure2.2 shows a demonstration of using deep learning approaches based on sensors for activity recognition. As we can see from the flowchart of Figure2.2, a difference compares to Figure2.1 (conventional machine learning) is the feature extraction and model building procedures are usually carried out at the same time, it is also an advantage of deep learning approaches in comparison with conventional machine learning approaches. Therefore, manually designed features are replaced by features learning automatically through the network. Moreover, deep generative models[START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF] allows exploiting the unlabeled samples for the training model. In addition, training on a large-scale labeled dataset using deep learning models can be transferred to new tasks where there are few or none labels. In the following paragraph, we investigate the deep learning models used in HAR tasks (cf. Table2.8 ) thus DNN is able to learn from a large amount of data.[START_REF] Vepakomma | A-wristocracy: Deep learning on wrist-worn sensing for recognition of user complex activities[END_REF] presented a wristocracy framework that allows recognizing 22 fine-grained activity contexts with various activity classes (e.g., sitting on a sofa, use the refrigerator, walk indoor, etc.).The hand-engineered features were extracted by a sliding windows of 2 seconds from accelerometers, gyroscope and location context (Bluetooth beacon message), then these features are fed into a DNN model.[START_REF] Hammerla | Deep, convolutional, and recurrent models for human activity recognition using wearables[END_REF] explored a deep feed-forward network (DNN) and compare with other networks such as Convolutional neural network (CNN) and Recurrent neural network (RNN). These networks performed HAR task on three representative datasets that contain movement data captured with wearable sensors.

	Model	Description
	DNN	Deep fully-connected network, artificial neural net-work with deep layers
	CNN	Convolutional neural network, multiple convolu-tional operations for feature extraction
	RNN	Recurrent neural network, network with time cor-relations and LSTM
	DBN/RBM	Deep belief network, restricted Boltzmann machine
	SAE	Stacked auto-encoder, feature learning by decoding-encoding auto-encoder
	Hybrid	Combination of some deep models
	Table 2.8: Deep learning models for HAR systems surveyed by Wang et al. (2019)
	Deep neural network

Artificial neural network (ANN) is a fundamental network for developing the deep neural network (DNN). It is designed to contain more layers than ANN (with very few hidden layers),

  Moreover, as reported in[START_REF] Yang | Deep convolutional neural networks on multichannel time series for human activity recognition[END_REF] most existing work on HAR relies on heuristic hand-crafted feature design and shallow feature learning architectures, which cannot find those distinguishing features to accurately classify different activities. Therefore, Yang et al. (2015) motivate to develop an automate systematic feature learning from the raw signals as inputs to be fed into a deep convolution neural network, so that the feature learning and classification are mutually enhanced by unifying in one model. Moreover, deep architecture can obtain specific variance or salient patterns of signals at different

	; Modi (2016); Modi and
	Titov (2014); Frermann et al. (2014); Regneri et al. (2010); Granroth-Wilding and Clark
	(2016); Hu et al. (2017)] carried out on learning scripts using Natural Language Processing

(NLP) but there is a lack of research regarding script learning from sensor data. Hence, to extract the script model, we address the issue by learning the script of daily life from the text with the hope they can be transferable to story generation from data. Once again, we will use a deep learning approach to this problem.

  Xn H are weighted matrices. The dimension n I is the size of the input vector, n H is the size of hidden vector. b g , b s , b f , b oare bias vectors, * is the element-wise multiplication. g t ij , s t ij , f t ij , ŝt ij , o t ij are the input gate, states of memory cells, forget gates of the memory cells, new states of memory cells and output gates of memory cells, respectively. We refer to the above function as h t ij = LST M(x t ij , h t ij -1 ). Then, the output of hidden states h t ij are fully connected to an output layer with activation softmax that produces the probability of all possible labels. Next, we predict labels of sensor data x t ij at each time-sliced window t ij by maximizing the conditional probability p(y t ij |x t ij ) = exp (Wo ht ij +bo )

k (Wo h k +bo )

(

  IN) the (DT ) table (NN)" where NN is a noun, DT is determiner, V B is a verb and IN is a preposition. Higher-level nodes are constituents, for instance (PP on table) means that "on table" is a prepositional phrase. Hence the complete syntactic structure of the sentence is fleshed out.

manuscrit défend la thèse selon laquelle ces données de capteurs portables peuvent être utilisées pour générer des récits articulés autour de scripts en utilisant l'apprentissage automatique. Il ne s'agit pas d'une tâche triviale en raison du grand écart sémantique entre les informations brutes de capteurs et les abstractions de haut niveau présente dans les récits. A notre connaissance, il n'existe toujours pas d'approche pour générer un une histoire à partir de données de capteurs en utilisant l'apprentissage automatique, même si de nombreuses approches d'apprentissage automatique (réseaux de neurones convolutifs, réseaux de neurones profonds) ont été proposées pour la reconnaissance de l'activité humaine au cours des dernières années. Afin d'atteindre notre objectif, nous proposons premièrement dans cette thèse un nouveau cadre qui traite le problème des données non uniformément distribuées (problème du biais induit par des classes majoritaires par rapport aux classes minoritaires) basé sur un apprentissage actif associé à une technique de suréchantillonnage afin d'améliorer la macro-exactitude de classification des modèles d'apprentissage classiques comme le perception multi-couche. Deuxièmement, nous présentons un nouveau système permettant de générer automatiquement des scripts à partir de données d'activité humaine à l'aide de l'apprentissage profond. Enfin, nous proposons une approche pour l'apprentissage de scripts à partir de textes en langage naturel capable d'exploiter l'information syntaxique et sémantique sur le contexte textuel des événements. Cette approche permet l'apprentissage de l'ordonnancement d'événements à partir d'histoires décrivant des situations typiques de vie quotidienne. Les performances des méthodes proposées sont systématiquement discutées sur une base expérimentale.Résumé du Chapter 1Ce chapitre introduit la thèse par une présentation des motivations qui nous ont conduit à aborder la génération d'histoires à partir de données d'activité issues d'un outil de type smartphone. Il en discute les principales difficultés, et donne un aperçu de notre recherche et de nos principales contributions pour les aborder. En fait, de nos jours, les données des capteurs portables sont collectées partout. Cependant, ces données sont souvent représentées sous une forme binaire et numérique, ce qui est difficile à comprendre pour l'homme, et il n'existe toujours pas d'approche

. This approach is called NESS for recursive neural Network Event Semantic Syntactic, to represent events as a hierarchical composition of their semantic and syntactic predicates 1 Stereotypical order denotes the order in which events are told in texts not their real temporal order. E.g "I've been to London yesterday. I took the bus" is a pair of events not told in chronological order though perfectly natural.2 Recursive Neural Network (RNN) is not to be confused with Recurrent Neural Network (also RNN for short.

Acknowledgments I would like to express my deepest gratitude to my thesis supervisors, Dr. Catherine GARBAY and Dr. Francois PORTET, for providing me with extensive support, patience throughout the duration of my Ph.D. This thesis would be impossible without their guidance and support during difficult times. I would like to thank my core jury members, Dr. Dan ISTRATE, Dr. Anthony FLEURY, Professor Norbert NOURY, Professor Paule-annick D'AVOINE, and Dr. Sylvie CHAR-BONNIER for providing the most constructive and invaluable comments and suggestions. I would like to express my sincere gratefulness to my colleagues (Dr. Raheel Qader, Dr. Nguyen Van Bao, To Thanh Hai, Tran Thi Phuong Thao, Nguyen Thi Thanh Quynh) who have been generous with their help, inspiration, and encouragement. Especially, I would like to thank Dr. Mai Thai Son for all the "what"s, "why"s and "how"s and for all the collaboration we could achieve. I would like to express my sincere thanks to the Ministry of Education and Training of Vietnam in partnership with the French government (Campus France), who awarded me a doctoral scholarship, and with which I devoted myself to the establishment of the thesis. I also would like to thank my supervisors for their role in partial funding support for me throughout the duration of my Ph.D. I would like to thank my family for making this happen with their unconditional love and support all the time of my life. Especially, I would like to take the opportunity for an offcial acknowledgment for my wife (DANG Thi Hong Hue) and our children for their support and motivation during the whole of my Ph.D.

List of Algorithms

1 Algorithm MLP AL OS Border Limited Link SMOTE . . . . . . . . . . . 59

At 10:00, the person walks in the building. Then, he/she sits down in the living room at 10:02.

After that, he/she puts coffee in the coffee machine in the living room at 10:03. At 10:05 he/she put the cracker on the table in the living room.

At 10:06 he/she eats cracker in the living room. Then, he/she drinks coffee in the living room at 10:10.

At 10:11 he/she leaves the building. Then, he/she goes shopping in the street at 10:12.

Figure 5.1: Illustration of generating scripts from human activity using wearable sensor data.

section 5.3.

We evaluate our proposed method on a daily-activity dataset collected by several subjects on several days Sztyler et al. (2016). The experimental results present that the proposed method can automatically generate scripts in unseen locations with high relevant accuracy corresponding to the actual texts. Furthermore, we found that our method is the first approach that enables us to generate automatically scripts from the HAR system using wearable sensor data.

The remainder of the work is organized as follows. Section 5.2 presents a summary of the state of the art in text generation from the HAR system. The proposed methods are detailed in Section 5.3. Several experiments are reported in Section 5.4. The work ends with a short discussion and an outlook of future work in Section 5.5.

Related Work

In recent years, there have been some researches interested in activity report generation in health care [START_REF] Kashimoto | Sensing activities and locations of senior citizens toward automatic daycare report generation[END_REF]; [START_REF] Inoue | Mobile activity recognition for a whole day: recognizing real nursing activities with big dataset[END_REF]]. However, these existing studies

Proposed Method

In order to automatically generate scripts, we design the proposed system illustrated in Figure 5.2. In the proposed system, multiple semantic concepts were produced from sensor data based on learning Sequential Deep Learning Model so that it can classify and predict three real-world properties (gesture, activity, location) be considered as semantic concepts and events. We will detail these processes in the next section 5.3.1.

Semantic Concepts Generation with Deep Learning Neural Network Model Semantic Concepts Representation and Events

In the first step, we represent the real-world states as a sequence of events e = [e 1 , e 2 , ...e n ],

where each event e is denoted as a tuple of semantic concepts (sc 1 , sc 2 , ..., sc m ). Each semantic concepts is represented by concept form of < g, a, l >, where g is the gesture of the subject (i.e., Walking, Running, Sitting, etc.), a is the activity of the subject does (i.e., Mealpreparation, Deskwork, Housework, Shopping, etc.), l is the location (i.e., Street, Home, Building, etc.) the subject stands.

Generating Semantic Concepts using Deep Neural Network Model

In this section, we present the deep learning Neural Network Model with a sequence-labeling approach that is used to generate semantic concepts of gesture, activity, and location by predicting labels from sensor data as illustrated in Figure 5.2. The person stands and chats at street or road or pasture.

The person stands and chats at street or road or pasture.

The person stands and chats at street or road or pasture.

The person is at the building.

The person sits and eats or drinks at home.

The person stands and cleans up at home.

The person sits and works at office.

The person sits and works at office.

The person is in the building.

The person is in the building.

The person is in the building.

The person is in the building.

The person is in the building.

The person stands and works at desk in the building.

The person stands and works at desk in the building.

The person stands and works at desk in the building.

The person stands and works at desk in the building.

The person stands and works at desk in the building. is a penalty for the divergence d(π, σ) between an observed ordering π and the canonical ordering σ. The Generalized Mallows Model is defined as GMM(π, σ) ≈ Π i e -ρ i v i where ρ i is the item specific dispersion parameter and v i is a vector of inversion counts. The authors then assumed that for each ESD the event e is realized or not by drawing from Bi nomi al(Θ e ) 2 . Then, drawing an event ordering π based on the GMM(ρ).

Finally, in [START_REF] Modi | Inducing neural models of script knowledge[END_REF] Event Embeddings (EE) are computed based on their predicates and arguments. Given an event, e = (v , d, a 1 , a 2 ), (here v is the predicate lemma, d the dependency and a 1 , a 2 are corresponding argument lemmas), each lemma (and dependency) is mapped to a vector using a lookup matrix C. Then each event representation is learned by A * tanh(T * C a 1 ,: + R * C v ,: + T * C a 2 ,: ) + b. then the event representations are used in a linear ranker to predict the expected ordering of events. Both the parameters of the compositional process for computing the event representation ((R, T, A and lookup matrix C) and the ranking component of the model are estimated from data.

Results

NESS was also compared to a version of our model which uses only verbs (NESS 2 ) in a similar way to BL and EE v er bs in [START_REF] Modi | Inducing neural models of script knowledge[END_REF]. The training sets are the dev set (doorbell, laundry, omelet, restaurant), the test sets are 10 sets (bus, coffee, fast-food, return, iron, microwave, eggs, shower, phone, vending). The results are presented in Table 6.1. From these results we can see that NESS is competitive in classifying the order of events. NESS 2 with a verb only reaches an F1-score of 83.2% which outperforms the BL and EE v er bs models with an F1-score of 68.0% and 79.1% respectively. In addition, regarding the full model including predicate and arguments, NESS 1 achieved 82.0% F1score higher than the MSA and BS model that reached 70.6% and 77.6% of F1-score.

Yet NESS 1 is slightly lower than EE which obtained 84.1% of F1-score.

Conclusion

In this work, we introduced a novel technique based on Recursive Neural Network to modeling the Scripts learning in order to capture the order of sequence of events from natural text. We also presented experimental results of our model on the public dataset provided by [START_REF] Regneri | Learning script knowledge with web experiments[END_REF], and compare our model with baseline models such as MSA, BL, BS, EE v er bs , full model EE that presented the results in [START_REF] Modi | Inducing neural models of script knowledge[END_REF].

From the results, we can see that our model can achieve with high accuracy (82.0% and 83.2% in F1-score for both model NESS 1 and NESS 2 respectively) to learning the order of sequence of events by using RNN model. Moreover, in order to overcome the sparsity issue by using count-based methods we presented the binary tree event embeddings. In future work, we can develop and apply other techniques to learn Scripts in order to improve the semantic representation by using some kind of Tree-Structured Long Short Term Memory Networks [START_REF] Tai | Improved semantic representations from tree-structured long short-term memory networks[END_REF].