Alexis Brenon
email: alexis.brenon@univ-grenoble-alpes.fr

François Portet
email: francois.portet@univ-grenoble-alpes.fr

Michel Vacher
email: michel.vacher@univ-grenoble-alpes.fr

Arcades: A deep model for adaptive decision making in voice controlled smart-home

Keywords: Smart-home, Decision system, Context-aware, Reinforcement learning, Deep learning

In a voice controlled smart-home, a controller must respond not only to user's requests but also according to the interaction context. This paper describes Arcades, a system which uses deep reinforcement learning to extract context from a graphical representation of home automation system and to update continuously its behavior to the user's one. This system is robust to changes in the environment (sensor breakdown or addition) through its graphical representation (scale well) and the reinforcement mechanism (adapt well).

The experiments on realistic data demonstrate that this method promises to reach long life context-aware control of smart-home.

Introduction

One of the applications in which Ubiquitous Computing [START_REF] Weiser | The computer for the 21 st century[END_REF] generates great expectations is smart-home. Smart-home is an application domain which brings together home automation and ambient intelligence to ease life of dwellers and to provide support to people in loss of autonomy. The development of smarthomes is not only a cultural and technological evolution but is also recognized as one way to address the challenges created by an aging population in developed countries [START_REF] Peetoom | Literature review on monitoring technologies and their outcomes in independently living elderly people[END_REF]. If home automation is concerned with sensing (sensors, actuators, middle-ware) and low-level automation (heating control, lighting control), Ambient Intelligence should provide perception and reasoning capabilities into the smart-home ecosystem.

However, although the development of smart-homes is supported by a large amount of research and industrial projects, it has not reached a large public since many challenges are still to be addressed. One of the main challenges is due to the complexity of setting up the smart-home system in case of new situations (devices, house, dwellers, after an accident, etc.). Furthermore, smart-homes often need to perform inferences from data in order to make decision so as to provide services that enhance users' experience and control over their environment [START_REF] Augusto | Intelligent environments: a manifesto[END_REF]. To make this enriched control possible, these systems perceive their environment and decide which actions to apply to the environment. This decision can be made after some specific events such as a user's request, a scheduled event (e.g., programmed task) or a detected risky situation. This perception is not only useful to trigger a decision but also to adapt the decision making to the current circumstances in which such decision must be performed. These circumstances are called the context and systems that explicitly take the context into account, are said context-aware [START_REF] Loke | Context-Aware Pervasive Systems[END_REF]. However, defining explicitly what pieces of information is playing an important role in the context of a decision is a difficult and tedious task. Indeed, if a general definition of context exists in the field [START_REF] Dey | Understanding and using context[END_REF], the large amount of existing models [START_REF] Bolchini | A data-oriented survey of context models[END_REF][START_REF] Ibarra | Engineering context-aware systems and applications: A survey[END_REF] demonstrates not only the importance of this notion for research but also the lack of a unified way of modelling context in intelligent environments.

Even though a model could be defined for a specific home, successful interaction must match users' preferences. To do so, explicit user configuration is a way to handle preferences but this have been proven to reduce the acceptance of the system [START_REF] Zaidenberg | Learning user preferences in ubiquitous systems: A user study and a reinforcement learning approach[END_REF]. More subtle ways of adapting system to users' preferences must be found. Moreover, as shown by Zaidenberg et al. [START_REF] Zaidenberg | Learning user preferences in ubiquitous systems: A user study and a reinforcement learning approach[END_REF], users ask for a way to monitor the system to better understand what it does and why. Indeed, smart-home systems must let the user keep control of the house even when she is in loss of autonomy [START_REF] Portet | Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects[END_REF]. All these factors make the development of an end-to-end smart-home system challenging.

In this paper, we propose a new system called Arcades (Adaptive Reinforced Context-Aware Deep Decision System) to perform adaptive context-aware decision for voice command in a smart home that does not use an explicitly defined context. Rather, the context information is directly learned from data using deep neural network [START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF][START_REF] Lecun | Deep learning[END_REF]. Indeed, one of the main advantages of Deep Learning (DL) is to learn attributes from raw data at the same time as the classification model is trained. This ability of DL has been demonstrated in numerous classification tasks reaching superhuman accuracy in the case of image classification [START_REF] Lecun | Deep learning[END_REF].

Another aspect of context awareness is to be able to capture a changing environment over time. Indeed, sensors, activity places, dwellers, etc. can change and any system that targets a lifelong support must be able to take these changes into account. Scenario 1 illustrates this need. In this simple scenario, it is impossible to know how to act without considering the context since the two voice commands are semantically the same. Here the activity or the location influence the choice of the lamp to light. Changes in the daily living routine (changing a lamp of place, moving the table, etc) must be accounted for in the configuration of the system. If static rule based systems are employed, any change must be impacted in the rule based system, making this impossible to modify by non-technical person such as the elderly users. To adapt continuously to a changing environment, the decision model we present in this paper uses reinforcement learning to update its decision policy.

Scenario 1 The inhabitant is washing up dishes and utters the voice command "Turn on the light". The lamp above the sink is lighted. Then she goes to the kitchen table and utters "Give me some light". The lamp above the table is lighted.

Finally, to make the smart home system easy to interact with we rely on speech interaction as well as on the sensor data capturing the interaction of the user with her own home. This make the set-up accessible since the user does not have to learn a complex jargon or to deeply understand the internal mechanisms of the system. Relying on machine learning paradigms, Arcades is able to build a model relevant for the given task, and then to adapt itself, in a non-intrusive manner. Moreover, we advocate the use of a new graphical approach to represent the environment state that brings two advantages: it handles multiple heterogeneous sensor data (making it resilient to smart-home evolution) and it allows users to check what the system perceives (cf. [START_REF] Zaidenberg | Learning user preferences in ubiquitous systems: A user study and a reinforcement learning approach[END_REF]). Thus, the work presented in this paper brings the following contributions.

• A system which represents the decision-making task as a reinforcement learning one, to adapt a decision making to the user's behaviour and her environment in an on-line setting (i.e., data are processed only with a limited history of past and no information about the future) in which the task is to perform voice command execution.

• A unified representation of the smart-home heterogeneous data sources, namely image, which is readable by machine and human ; hence, the model does not depend on intermediate representation layers but only on the input data. This is a strong strength of our approach as well as an original aspect of the research presented here (we are not aware of any work using this original data representation in this field).

• A deep learning model able to build and extract relevant contextual features from (semi-)raw sensors input (i.e. images) as well as solving the reinforcement learning task.

• An evaluation of the system on a realistic smart-home data set available to the community.

The paper is organised as follows. The next section (section 2) summarises the related work. The method is detailed in section 3 which describes the overall approach and the data representation. In section 4 we present an experiment to train and evaluate Arcades on realistic data. Complementary experiments are summed up in section 5. Finally, the results and some possible improvements are discussed in section 6.

Related works

Automatic adaptive decision-making in smart-homes has received far less attention than more widespread research on context or situation recognition. However, it had recently regained interest, particularly in the domain of energy management [START_REF] Li | Reinforcement learning aided smart-home decision-making in an interactive smart grid[END_REF][START_REF] Berlink | Batch reinforcement learning for smart home energy management[END_REF]. In the literature, a decision-making system depends on two main abilities: (1) the ability to represent knowledge perceived from the environment and (2) the ability to make decision from this perception given a specific user and situation.

With respect to the first ability, research on context aware systems had generated a vast amount of knowledge representations. Logical approaches are generally applied for representation in pervasive environments because they can easily model relations among entities and perform automatic inference. For instance, Mileo et al. [START_REF] Mileo | Reasoning support for risk prediction and prevention in independent living[END_REF] applied a logic programming approach, Answer Set Programming, in order to implement a system of prediction of risky situations. Event calculus, a method to model situations using first order logic, has also been applied in the development of smart-homes [START_REF] Chen | A logical framework for behaviour reasoning and assistance in a smart home[END_REF][START_REF] Katzouris | Event recognition for unobtrusive assisted living[END_REF].

The main trend for knowledge representation in smart environments has become ontology, in particular those based on Description Logic (DL). The Ontology Web Language (OWL) is the main implementation of DL, and it has been largely applied in pervasive environments [START_REF] Rodríguez | A survey on ontologies for human behavior recognition[END_REF][START_REF] Liao | A rdf and owl-based temporal context reasoning model for smart home[END_REF]. Liao and Tu [START_REF] Liao | A rdf and owl-based temporal context reasoning model for smart home[END_REF] employed an ontology to organize temporal events and the concepts about the smart-home. Context-aware systems have also been developed using ontologies to model the current context for mobile applications [START_REF] Attard | Ontology-based situation recognition for context-aware systems[END_REF][START_REF] Yilmaz | iconawa -an intelligent context-aware system[END_REF]. Fuzzy ontologies also received attention because of their ability to model imprecision and allow inference with vague concepts such as "hot weather". Rodríguez et al. [START_REF] Rodríguez | A fuzzy ontology for semantic modelling and recognition of human behaviour[END_REF] presented an application of fuzzy ontologies for activity recognition in pervasive environments. Another important contribution to make ontologies capable of dealing with uncertainty is the use of probabilistic models in ontology inference. A notable application of such approach has been presented by Helaoui et al. [START_REF] Helaoui | A probabilistic ontological framework for the recognition of multilevel human activities[END_REF]. In Chahuara et al. [START_REF] Chahuara | On-line human activity recognition from audio and home automation sensors: comparison of sequential and non-sequential models in realistic smart homes[END_REF] a Markov Logic Network together with an ontology in Description Logic (DL) is employed to perform context recognition. However, the diversity of approaches also shows the lack of consensus regarding the different concepts to be considered in a smarthome and their organization. Hence, there is a need for alternative approaches to handle information being inferred from the sensor network of the smart-homes.

In all of these approaches, context modelling is considered from a top-down perspective, where elements of the context are identified by expertise. In this paper, we propose a different approach: a data-driven constitution of the context where all the sensor data are unified into a common representation: an image. The closest work to this approach is the recent one by Singh et al. [START_REF] Singh | Transforming sensor data to the image domain for deep learning -an application to footstep detection[END_REF] which transforms a pressure sensor matrix image to perform gait identification using deep neural network. However, in their work the data were already sequences of 2-dimensional values while in the case of smart home we must dealt with a set of monodimensional sensor data of variable type (Boolean, categorical, numerical) whose position is only loosely known and changing.

Regarding the decision-making process, several logic-based approaches are presented in the literature. Moore et al. [START_REF] Moore | Rule strategies for intelligent context-aware systems: The application of conditional relationships in decision-support[END_REF] developed a system that exploit a set of fuzzy rules in order to find the most appropriate action under a certain condition given by the context. Similarly, Kofler et al. [START_REF] Kofler | A semantic representation of energy-related information in future smart homes[END_REF] and Gómez-Romero et al. [START_REF] Gómez-Romero | Context-based scene recognition from visual data in smart homes: an information fusion approach[END_REF] used Description Logic to define the behavior of a context-aware system which models context elements (activities) in an ontology. There are several works applying ECA (Event-Condition-Action) rules based systems into pervasive environments [START_REF] Leong | A rule-based framework for heterogeneous subsystems management in smart home environment[END_REF][START_REF] Yau | Hierarchical situation modeling and reasoning for pervasive computing[END_REF]. However, in these proposals the system is set a priori to execute an action given a specific configuration (the condition) and consequently the system does not adapt its behavior to a changing situation. Bayesian networks, for instance, are among the most important methods for decision-making, as exemplified by Lee and Cho [START_REF] Lee | Fusion of modular bayesian networks for context-aware decision making[END_REF]. They have modeled uncertainty on context-aware system by means of Bayesian networks but the method does not include formal elements of decision-making. Influence Diagrams (ID, [START_REF] Howard | Readings on The Principles and Applications of Decision Analysis[END_REF]) is a method based on Bayesian networks that includes special variables to appropriately model the decision process. Some research work on context aware systems [START_REF] De Carolis | C@sa: Intelligent home control and simulation[END_REF][START_REF] Nishiyama | Development of agent system based on decision model for creating an ambient space[END_REF] have relied on ID in order to model the decision process, treating the uncertainty and measuring the expected utility of possible actions. However, the expressiveness of this probabilistic approach is limited since only propositional variables and conditional dependencies among variables can be represented in the model, besides the fact that it is less human readable than logical approaches. In Chahuara et al. [START_REF] Chahuara | Context-aware decision making under uncertainty for voice-based control of smart home[END_REF] a Markov Logic Network (MLN) is proposed to perform automatic decision-making in smart-home. MLN make it possible to benefit from a formal representation of the decision process using first order logic while modeling uncertainty by weighting the logic formulae. The other advantage is that weights can be learned from data. The approach has been tested in a realistic setting and showed the strength of the approach. However, it does not include mechanism to adapt the decision model to new situations. Furthermore, if part of the model can be learned from data, the logic part is acquired by human expertise which has the same drawback as the logic based knowledge representations in smart-home.

To tackle the problem of adaptation, some researches in smart-home have designed the decision problem as a reinforced decision-making problem. The most famous of these works is The Neural Network House [START_REF] Mozer | The neural network house: An environment that adapts to its inhabitants[END_REF]. It is a house equipped with more than 75 sensors and actuators monitoring the environmental conditionssuch as temperature, light and sound levels, motions, door and windows openings -and controlling some devices of the house such as heaters or lights. The devices were controlled by the ACHE system [START_REF] Mozer | Smart Environments: Technology, Protocols and Applications[END_REF], which commanded the lights in order to reduce the overall energy consumption without impairing the inhabitant comfort. Reinforcement Learning (RL) is the base of the ACHE system. A state estimator builds a high-level environmental state from the sensor data. This state contains the current activity of the user and the level of natural light at the user's location. Then, a controller makes a decision about which light to act on, and decides on the intensity level to set. To evaluate this decision, two kinds of cost are defined and returned to the controller for its learning. On the one hand, the energy cost, directly indexed on the actual energy cost of electricity in the area of the smart-home. On the other hand, the comfort cost, which is induced when the inhabitant have to modify the decision made by the system. Although this project showed promising results, it seems to be no longer active but has given rise to a number of researches by other groups [START_REF] Zaidenberg | Reinforcement learning of user preferences for a ubiquitous personal assistant[END_REF][START_REF] Karami | User in the loop: Adaptive smart homes exploiting user feedback -state of the art and future directions[END_REF] which showed that RL allows some flexibility that traditional machine learning paradigms do not. Nevertheless, if RL is particularly suitable for restricted environment (with a few possible states and actions), it poorly scales to large real cases sets. This drawback, is mainly caused by the classical implementations used for reinforcement learning tasks, based on an associative array whose size depends on the number of variables and their domain values to represent the state of the environment. A way to solve this, is to change the data structure, relying for example on a deep neural network [START_REF] Lecun | Deep learning[END_REF][START_REF] Schmidhuber | Deep learning in neural networks: An overview[END_REF]. A notable example of deep reinforcement learning for decisionmaking is provided by Mnih et al. [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] who proposed a new approach to train a deep neural network using reinforcement learning. The proposed model, the Deep Q-Network (DQN), has been applied to a decisionmaking task regarding classic Atari2600 games. Roughly speaking, given a pre-processed image frame from the Atari display, the model chooses the best action out of 18 (17 combinations of a joystick plus 1 no action) to maximize its reward based on the game score. A convolutional neural network (CNN) processes the frames, while a fully connected network approximates the value of the states of the RL problem. After few millions training iterations, the model is compared to random and human decision-making and shows that the DQN reaches or surpasses human performances in the majority of tasks. This work proves that RL and a deep convolutional network can be combined. Thus, RL can be used in situations approaching real-world complexity. This inspired us to use a similar agent to make context-aware decision in a smart-home.

Table 1 lists the main attributes of the papers considered in this state of the art. It shows the variety of models that have been applied for decision-making in contrast to the lack of experimental real-time tests in real environments. Even when several approaches have been proposed to handle knowledge representation, inference or adaption, most of these works have been applied to a particular problem of context awareness, such as activity recognition, and did not consider the problem as a decision-making one. In this work, we propose an approach using deep-learning to learn the features/concepts that comes into play during the decision at the same time as the decision model. This learning is performed using the reinforcement learning paradigm so that on-line adaptation is made possible. Furthermore, we propose an original knowledge presentation, raw image, to benefit from the recent improvement in image processing using deep-learning. The remaining of this paper presents the method and its application to decision-making in a smart-home.

Methods

Typical smart-homes considered in the study are the ones that permit voice based interaction. Voice-User Interfaces (VUIs) in domestic environments recently gained interest in the speech processing community. The rising number of smart-home projects that consider Automatic Speech Recognition (ASR) in their design [START_REF] Istrate | Embedded implementation of distress situation identification through sound analysis[END_REF][START_REF] Filho | From science fiction to science fact: a smart-house interface using speech technology and a photorealistic avatar[END_REF][START_REF] Lecouteux | Distant speech recognition in a smart home: Comparison of several multisource asrs in realistic conditions[END_REF][START_REF] Christensen | homeservice: Voice-enabled assistive technology in the home using cloud-based automatic speech recognition[END_REF] exemplifies this trend. Adapted to homes containing multiple rooms which are fitted with sensors and actuators -such as infra-red presence detectors, consumption meters, multimedia servers, etc. -this kind of interface is particularly suitable to people in loss of autonomy [START_REF] Portet | Design and evaluation of a smart home voice interface for the elderly: acceptability and objection aspects[END_REF][START_REF] Vacher | Evaluation of a context-aware voice interface for ambient assisted living[END_REF]. Smart-homes aim at providing daily living context-aware decision using the perception of the situation of the user, as previously illustrated in Scenario 1.

The interactions between the user, her home and the control system is depicted in Figure 1. An underlying home automation network, composed of typical home automation sensors and actuators (switches, lights, blinds, etc.) and several microphones per room allows a controller to interact with the home and the user.

As emphasized in section 1, the main challenges in such a context are to be able to handle heterogeneous information in a unique representation space and to adapt the decision-making to an evolving environment. Thus, the remaining of this section focuses on the description of Arcades, the Adaptive Reinforced Context-Aware Deep Decision System, and its components. We start with the reinforced adaptation loop (subsection 3.1), then the data representation (subsection 3.2) and finally the decision model acquired using deep learning (subsection 3.3). Last section (subsection 3.4) explains how all these components interact with each other to allow the system to learn.

Adaptation of the decision model through reinforcement learning

A Reinforcement Learning (RL) task relies on the Markov Decision Process (MDP) formalism which considers two components: the environment and the agent. Figure 2 shows the interactions between each component: the agent (here the decision-making system) is fed with the current environment state s t ∈ S. It chooses an action a t ∈ A based on this state. The chosen action affects the environment (here the house and the user) whose state changes (from s t to s t+1 ) and which generates a reward. This reward R(s t ) = r t can be positive if the action is appropriate, negative if the action is not adequate, or null. Based on this reward, the agent adapts its decision policy. In Scenario 1, the voice command and context of the interaction compose the state, while the decision made by the system is the action to execute. Then the user will be able to evaluate/reward this decision by different means not discussed here.

In a fully qualified MDP, dynamic programming [START_REF] Bellman | Dynamic Programming, 1st Edition[END_REF] is used to compute the optimal policy. However, it cannot be used in real world problem due to missing information about the environment model and the reward function. In that case, Q-Learning [START_REF] Watkins | Learning from delayed rewards[END_REF][START_REF] Sutton | Reinforcement Learning: An Introduction, 2nd Edition[END_REF] can be used to compute and update a policy which will converge toward the optimal one. In Q-Learning, a function, named Q-function, associates a value, named Q-value, to each possible pair of state-action, named Q-state: Q : S × A → R, where S is the set of possible states of the environment, and A is the set of possible actions to perform. Beginning with an initial Q-function (Q 0 ), it can be updated after each interaction ⟨s t , a t , r t , s t+1 ⟩, following Equation 1 (where γ is the discounting factor of the reward function and α is a learning rate). This updating rule has been proven to converge toward the actual Q-value of each Q-state, allowing to compute the optimal policy to solve the task.

Q k+1 (s t , a t ) = α ( r t + γmax a Q k (s t+1 , a) ) + (1 -α)Q k (s t , a t ) (1)
The following sections describe how the states of the environment can be acquired, and the model used for the Q-function, before presenting how both components interact during the learning process.

A common representation for heterogeneous information: a raw image approach

One challenge in reasoning with raw data is to define a referential space where all kinds of information can be represented and manipulated. Indeed, real smart-home data are multi-modal [START_REF] Vacher | The sweet-home speech and multimodal corpus for home automation interaction[END_REF]: from continuous time series, such as the overall water consumption or temperature, to event based semantic information such as voice commands or the state change of a device. Hence, finding a representation that can accommodate low-level sensor events (presence in a room) and high-level events (user current activity) is a problem that has not found a consensual solution. Here, we are proposing the use of a graphical representation, projecting any information on a two-dimensional map of the smart-home any time the environment state changes. Then, we will rely on the ability of Convolutional Neural Networks (CNN) to build relevant features from this raw representation. This approach provides some advantages listed below.

1. The images generated can be used by the user to monitor what the system sees or understands. 2. The image size is only loosely linked to the number or type of sensors (adding, removing, or changing some sensors will not change the size of the image), hence images accommodate a variable number of data sources (sensors can come and go) a property that classical input vector machine learning approaches do not naturally handle. 3. Images naturally convey spatial information (here an a priory information about the home room organisation). 4. The high redundancy of sensor values can be efficiently used by deep learning.

Example of generated image for the smart-home considered in the study are provided in Figure 3. In this image, as in the Scenario 1, the user asks to turn on the light while she is washing up dishes. The smart home walls are represented by black lines while icons represent raw sensor data. For instance usage of light is represented by a black lamp on white background while a turned off lamp is a white lamp on dark background. The highest part presents some gauges for electricity and water consumption. The big icon at the bottom left represents the uttered command. Finally, other icons match the last known state of each information provider (sensor or inference system) of the home, using a different icon for different information type. It exists a dedicated icon for various kinds of information such as:

• Binary sensors symbols: presence detector, door/window detector, blind state, light state, etc.; • Continuous sensors symbols: light level, sound level, temperature, humidity;

• Continuous sensors gauges: electricity consumption, water consumption, etc.;

• High level information symbols: uttered command.

The image and icons have been chosen to maximise the contrast and redundancy for the input of a CNN, to be understandable by human and to be easily generated on the fly. In fact, the implementation relies on the SVG1 technology, to make it possible to update the image on-line to reflect the state of the smart-home.

Decision making model through deep-learning

As explained in subsection 3.1, the decision policy rely on the Q-function of the agent (the simplest policy being: "Given s, choose the action with the highest Q-value") and we are going to present two ways to represent it.

Classical implementations choose a tabular representation. A two-dimensional array, indexed by states in one dimension and by actions in the other, records the Q-values for each Q-state. However, this approach does not scale to big or continuous sets of states or actions and cannot capture redundancy in states.

Another recent implementation relies on a deep neural network to model the Q-function. This representation has been introduced by Mnih et al. [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] who introduced the Deep Q-Network (DQN), which has been successfully tested on decision-making task related to Atari2600 games (cf. related work section 2). Although, their application was different from our, the voice command problem can be seen as a general problem of decision-making in a changing environment. We thus adapted the approach to smart-home decision-making.

The decision model is based on a Convolutional Neural Network (CNN) to process the images followed by a fully connected neural network to choose the best action to perform. Figure 4 presents the architecture of our Q-network. While not discussed here, different architectures has been tested, taking into account base architecture proposed by Mnih et al. [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], technical limitation like memory consumption as well as some a priori knowledge like the 6 px based grid of the images. Finally, the neural network is composed of four Thus, this model provides a way to process the generated images presented in subsection 3.2. After some preprocessing (scaling and reducing the depth of the image), the image is passed forward in the Qnetwork. The network extracts relevant features used to classify it in the range of possible actions. It provides, as output, the Q-values for each action given the input state (and thus can be seen as a function Q-network : [0, 255] 256×256 → R 33 ). Then, Equation 1 detailed earlier can be used as a loss function to update the weights of the Q-network, as in typical deep learning approaches.

C 16 [8 + 2/4] C 32 [6 + 1/3] C 64 [3 + 0/2] C 64 [3 + 0/1] F C 512 F C 33 Q(s, ) Q(s, ) Q(s, ) Q(s, ) Q(s, ) Q(s, ) s Figure 4: Architecture of the Q-network. Cn[f + p/k]
While the work from Mnih et al. [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] is not the first attempt to use neural network as Q-function [START_REF] Tsitsiklis | An analysis if temporal-difference learning with function approximation[END_REF], it seems to be the first successful one. This is due to different optimization techniques as the use of a target network, experience replay [START_REF] Lin | Reinforcement learning for robots using neural networks[END_REF], mini-batch learning or RMSProp gradient optimization [START_REF] Hinton | Lecture 6e -rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF]. All these techniques have been kept in our implementation and the reader is referred to the original paper [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] for detailed explanations.

Learning method

Algorithm 1 Logic used to allow the agent and the environment to interact. The overall learning of the decision-making module (the agent) is decomposed into two phases: a pretraining phase (i.e., initialization of the agent), during which the model is learned from scratch and an adaptation phase during which the model is tuned toward the specific inhabitant and environment. In practice, the pre-training phase is performed off-line and is mostly devoted to the learning of the CNN and the subsequent neural network. It is well-known that CNN needs a high amount of data that is why, in our work, we used artificially generated data to feed the CNN with raw input to learn the CNN weights. In the adaptation phase the CNN is not supposed to evolve but the last part of the neural network (i.e., the decision part) is biased towards the adaptation data. This data can be either acquired on-line or performed on corpus if available. In any of these phases, the reinforcement learning method is the same and is described in the Algorithm 1. Starting from an initial agent, the system makes maxSteps interactions, in which the observable state of the environment is passed to the agent, before it chooses an action applied back to the environment which releases a reward. All these interactions are recorded by the agent which uses them during a learning procedure. This training epoch (interactions plus training) is realized a number maxEpoch of times.

Algorithm 2 Function used to simulate the evolution of the environment for synthetic data.

function performAction(predictedAction) if predictedAction is trueAction∨ numberOf T ries > triesT hreshold then moveToNextDataSample numberOf T ries ← 0 else numberOf T ries ← numberOf T ries + 1 end if end function Since during the pre-training phase the agent explores artificially created data, the environment is not able to evolve consequently to a wrong action. Thus, to multiply the number of situations and explore wrong decisions, the function described in Algorithm 2 simulates the environment evolution for synthetic data. In this function, as long as the action chosen is not the correct one, up to triesT hreshold actions can be successively tried to simulate the patience of the user, repeating the same command until the system eventually behaves correctly.

Similarly, all the experiments presented in this work are realised on a previously recorded corpus. Thus, it is not possible for the user to interactively generate a reward. We defined a simple reward function able to simulate this reward, based on a ground truth (obtained from annotations of the corpus) and the predicted action: if the chosen action is correct emit +1, else emit -1.

Algorithm 3 Agent policy

Require: ϵ ∈ [0; 1] function getAction x ∼ U([0, 1]) if x ≤ ϵ then action ∼ U(A) ▷ Random action else Q-values ← Q-Network.forward(state) action ← argmax(Q-values) end if return action end function
Fetching the action to perform requires the agent to execute its policy described in Algorithm 3. We use a near greedy policy named ϵ-greedy policy [START_REF] Sutton | Reinforcement Learning: An Introduction, 2nd Edition[END_REF]. Given ϵ ∈ [0; 1], this policy returns a random action with probability ϵ or the action with the highest Q-value given the current state. To compute the Q-value of each action, the agent uses its approximated Q-function, modeled by the deep neural network Q-Network.

Algorithm 4 Agent learning procedure Notation: B st is the vector composed of the s t component of each sample.

function learn B ← getSamples(minibatchSize)

▷ Each sample is ⟨s t , a t , r t , s t+1 ⟩ Q-values ← ( Q-Network.forward(B st ) )[ B at ] ▷ Vector of Q-values of each Q-state in B ▷ (0 for all Q-states not in the batch). targets-values ← B rt + γ max ( target-Q.forward( B st+1 ) ) ▷ Target values computed with the target network loss ← targets-values -Q-values gradient ← Q-Network.backward( B st , loss ) updatedGradient ← RMSprop(gradient) Q-Network.parameters ← Q-Network.parameters + updatedGradient end function
The learning of the agent follows the work of Mnih et al. [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF]. In particular, to estimate the Q-value the learning does not use the latest experienced interactions. Instead, each interaction ⟨s t , a t , r t , s t+1 ⟩ is recorded in a pool. Then, a set B of interactions is randomly drawn from this pool and used as a (mini-)batch to train the network. An updated Q-value, more accurate than the current one (taking account of the reward provided by the user) and named target-value, is computed for each Q-state involved in B according to: target-value = r t + γ max a Q(s t+1 , a). The difference between current Q-values and target-values is used as a loss and back-propagated through the Q-Network to compute the gradient. The gradient obtained is optimized using the RMSprop optimization algorithms [START_REF] Hinton | Lecture 6e -rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF] before being applied to the weights of the neural network. This whole process is described in Algorithm 4.

Experiment

The experiment conducted in this study was decomposed in two phases as illustrated in Figure 5. The first phase, the pre-training phase, uses a generated corpus to train an agent and a similar corpus to test it. The validation phase then, performs a cross-validation on a real corpus. As data recorded for each participant is comparable, a Leave One Subject Out Cross-Validation (LOSOCV) method is used to avoid bias. It uses fourteen of fifteen participants' data for adapting and tests Arcades with the left out participant's data.

Input data

Sweet-Home Corpus

The data corpus was recorded in the Domus smart-home designed by Laboratoire d'Informatique de Grenoble (LIG) [START_REF] Gallissot | Une plateforme usage pour l'intégration de l'informatique ambiante dans l'habitat. l'appartement domus[END_REF]. This 30 m 2 home includes a bathroom, a kitchen, a bedroom, and a study as show in Figure 6. All these rooms are equipped with sensors and actuators such as infrared motion sensors, tactil sensors, temperature sensors, power meter, etc. In addition, seven microphones in the ceiling capture the audio. The apartment is fully usable and can accommodate a resident for several days. More than 150 devices are managed in the apartment to provide different services (e.g. light, opening/closing shutters, media management, etc.).

To collect decision data in context, several participants were recruited to play scenarios of the daily life in the apartment. While enacting daily life scenarios, participants had to issue voice commands to activate the actuators in the smart home. The objective of this experiment was to test a smart controller in real situations corresponding to voice commands spoken by the user. This study considered the following situations: to cleanup the apartment; to prepare and eat a meal; to converse via video conference; to do leisure activities (reading); to take a nap.

To guide the participants, the grammar voice command was provided with a scenario of everyday life. This scenario was designed to last about 45 minutes, however, there was no constraint on the execution time. Each participant received a list of actions to make and voice commands to utter that was interpreted by an intelligent system acting accordingly. Each participant had to use a voice command, repeating up to 3 times in case of system failure, before a Wizard of Oz was used.

A total of 15 people (9 women, 6 men) participated in the experiment to record 11 hours of data. The average age of participants was 38 ± 13.6 (19 -62). All experiments were video recorded but only for annotation purposes. This data set is part of the publicly available Sweet-Home corpus [START_REF] Vacher | The sweet-home speech and multimodal corpus for home automation interaction[END_REF].

State and action sets

The state space can be seen from different points of view. If the state space is represented only by the location and activity of the user (i.e., classical context variables), the number of states is 324 [START_REF] Brenon | Preliminary study of adaptive decision-making system for vocal command in smart home[END_REF]. But these contextual information necessitate inference models and our aim is to directly make decision from raw data. When state space is represented by raw data issued by the 81 available sensors (49 binary sensors and 32 continuous sensors which can take values in different ranges -from [0; 100] up to [0; 2 32 -1]), this leads to a big number (more than 10 12 ) of possible states. This shows again that decision making from raw data cannot be performed using explicit representation and that approximation of the kind presented here with neural network is the only way to make this decision making tractable.

The action set depends on the number of actuators and commands concerning them. Even if some actions were never realized during the corpus record, we chose to handle a large set of actions. Thus, any device used at least once has been incorporated, with all its possible commands even if some of them are not used. This ended up with a set of 33 possible actions listed in Table 2.

Command

Actuator devices turn on the light all lights in the kitchen, light above the sink in the kitchen, ceiling light of the kitchen, all lights in the bedroom, bedside lamp of the bedroom, ceiling light of the bedroom, ceiling light of the study turn off the light all lights in the kitchen, light above the sink in the kitchen, ceiling light of the kitchen, all lights in the bedroom, bedside lamp of the bedroom, ceiling light of the bedroom, ceiling light of the study

. Generated corpus

From a deep-learning point of view, the Sweet-Home corpus does not contain enough relevant samples of voice commands. Furthermore, it contains unpredictable samples or missing data, which is not advisable for a first proof of concept. As a workaround, a data generation system was developed based on a few generation rules. This system can be tuned to generate deterministic data (which always associate the same output to the same input), restricted data (focusing only on most significant events), etc.

The generated corpus follows the same format as the real raw data corpus. Thus, the subsequent translation of the annotations to SVG image (cf. subsection 3.2) can be applied seemingly to the real or generated data. The generation system producing samples on-demand, this corpus is theoretically of infinite size, a new sample always being available if necessary.

The corpus is generated to use the ability of the deep learning to build its own features from raw data. Thus, the generation consisted in generating annotations for the voice command and the expected action as well as simulating values for the raw sensor data. To achieve this, we generate an "annotated" state, similar to the ones used in our preliminary study [START_REF] Brenon | Preliminary study of adaptive decision-making system for vocal command in smart home[END_REF], which is converted using logical rules. For example, if the annotated location is the kitchen, then the presence detector sensors of the kitchen are constrained to be active. In the meantime, the bedroom being the room next to the kitchen, its presence detector sensors are constrained to be active with a defined probability. This mechanism is applied for all relevant sensors, while random values are applied to irrelevant ones. The real Sweet-Home corpus is a set of CSV files, one for each sensor plus two for the uttered command annotation and the expected action (the target of the learning). These files contain the value for the data source for each time-stamp which can be parsed to generate a data structure similar to the one generated by our generation system.

Experiment: learning on raw data

First attempts to train a model from images generated with raw data lead to poor performances (oscillations, long convergence time, etc.). To address this problem, we choose to investigate the hyper-parameter choices, conducting an exhaustive search for three of them which are target_q, minibatch_size and update_freq. Result of this research is summed up in AppendixA. These parameters are used by the agent, minibatch_size and update_freq being directly related to the learning procedure described in Algorithm 4. The former is the number of samples used for the training, that is to say the size of B. The latter is the number of steps done before calling the learn procedure. As explained, the learning does not happen after each interaction but after each update_freq interactions. The last parameter, target_q, is related to the target network trick introduced by Mnih et al. [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], and the whole details of its use is out of the scope of this paper. However, these three parameters are tightly linked. A high target_q and minibatch_size must improve the overall stability of the Q-network, but a high minibatch_size leads to a longer training time. Thus, increasing the update_freq must reduce this learning time.

While the data streams of synthetic generated corpus are perfectly synchronized, this is not the case for the real data. Despite our efforts to synchronize sensor streams and annotations, the real smart-home corpus does not provide a high quality of synchronization. To overcome this problem, we choose to provide to the Q-network a wider window of the last 3 seconds of the state of the environment. However, this mechanism generate many samples where no action is required by the agent (2 samples of no action for one sample of any of the 32 other actions), and thus heavily unbalance our classes. To overcome this, the reward function was updated to heavily penalize an error about this majority class. Instead of receiving a reward of (-1), the agent receives a reward of (-192) if it chooses to do nothing while an action is expected.

The parameters of this experiment are detailed in Table 3. The pre-training phase used synthetic data generating images similar to the one presented in Figure 3. The model was trained using up to 10 • 10 6 interactions, evaluating the system on 5000 independent interactions every 100 • 10 3 interactions. Once a model is learned from scratch, the validation phase adapts it to new data. Each fold of the LOSOCV method runs the system for 400 • 10 3 interactions on the real data. Test of Arcades happens every 4000 interactions, running 5000 monitored interactions.

To compare against a classical approach a baseline system using a tabular Q-learning has been learned and tested by LOSOCV. However, since the classical approach uses an array to represent the state, this Data real, raw totalLearningSteps 400 • 10 3 learningSteps 4000 evaluationSteps 5000 approach would not have been tractable if it was applied directly to the large amount of sensors values (cf. subsection 3.3). Hence, the baseline was trained on the ground truth humanly annotated location and activity labels. An except of the ground truth data is showed below. It shows that a voice command to open the blinds in the kitchen should be followed by the opening of all the blinds of the kitchen. The second line is related to Scenario 1, when the light is asked to be turned on while cooking in the kitchen, the only possible action is to light on the lamp above the sink.

open -blind kitchen none -> open blind kitchen on -light kitchen cook -> on light kitchen -sink

Thus this kind of ground truth dataset favours the baseline performances.

Table 4 summarizes the result of the learning. Even though the reinforcement learning objective is to maximise the reward, in this paper, standard classification measures such an F-Measure have been chosen. Indeed, the task can be seen as a classification one, classifying a sample (the environment state) between 33 classes (the actions). In our experiment, a correct classification is considered only when the chosen action is composed of the right action (e.g., turn on) on the right device (e.g., light) at the right place (e.g., study). Any other choice is a wrong classification. While the baseline tabular Q-learning only deals with humanly annotated data, it presents a low F-Measure of 46%. Arcades on raw sensor data does not lead to very high performances, but it demonstrates that a context aware decision-making model without an explicit representation of context is possible and lead to a very promising result of 67.5% of F-Measure.

Detailed results of Arcades are presented Figure 7 and Figure 8. These are presented using the following metrics:

• Reward per episode: This value is the objective of the agent. It is the average sum of rewards obtain during each episode (an independent set of interactions). Given the reward function and the maximum length of an episode, the reward function cannot be greater than 1.

• Precision, Recall, F1-Score: standard performance measures for classification. Arcades can be seen as a classifier, classifying a sample (the environment state) between 33 classes (the actions).

First, it is interesting to note that the evaluation performances are better than the learning ones. This is explained with the exploration factor (the ϵ of the ϵ-greedy policy), which is decreased from 0.99 to 0.5 during the first 200 000 learning interactions (after that, 1 action over 2 is selected randomly) while it is set to 0 during the evaluation phase (greedy policy, no random decision).

Second, while being very long, the pre-training phase finally reaches a very high plateau. With more than 4 days and 16 hours of training Arcades acquires a coherent behavior as of the 8 • 10 6th interaction (about 100 days with one decision per second), as shown in Figure 7. Then, during the validation phase the system slightly adapts to the real data, starting with low performances, to finally reach more than 65 % of F1-Score (Figure 8). Compared to the pre-training phase, this adaptation happens in a short time, comparable to the one obtained with the baseline system.

Complementary experiments

While in the case of a classical tabular Q-learning, the associative table can be interpreted, this is more complicated for the parameters of a neural network. This is a recurrent grief addressed to neuronal models, and thus methods have been developed to better understand how they work [START_REF] Karpathy | Cs231n: Convolutional neural networks for visual recognition[END_REF]. In this section, we present some complementary experiments aiming at understanding Arcades mechanisms.

In subsection 5.1 we test the ability of the system not to adapt to user preferences, but to changes in the sensor topology. Then, we investigate, in subsection 5.2, which features the neural network builds, and if they match the ones usually found in the literature.

Adaptation to the hardware

One of the main goals of Arcades is to adapt itself to the user's preferences, but also to the sensors. To verify this hypothesis, we made experiments in which some sensors disappear (because of faulty sensors or if the user uninstall them). Four experiments were performed, based on a pre-trained (with synthetic raw data) model, in which different kinds of sensor were removed.

In the first experiment, the kitchen motion sensors were removed. Figure 9 shows that the performances are impacted but that the system quickly adapts its behavior to reach back its initial performances.

In the next experiments, the acoustic noise from the kitchen microphone was removed, then door switches for kitchen cupboard and fridge. The latter are particularly active when the user tidy the home, and then we expected a dip of the performances. However, for these two experiments, no impact can be measured (Figure 10 and Figure 11) suggesting that the information obtained from these sensors are of little use to Arcades.

The last experiment simulates the break of the automatic speech recognition system. While we could expect a drastic fall of the performances, near the random agent which would get about 3 % of F1-Score, the system still acts not so badly, with an F1-Score oscillating around (20 ± 10) %. Noticing that the average reward is not so far below (0) is another hint that its behavior is far from random. One way to explain this, is that the system still gets images 2 seconds before an action is expected. This information seems to be grabbed by the system during pre-training, and used to know when an action is expected (even if it is not the right one). However, the oscillations of the score shows that Arcades is not able to infer the user intention only from the sensor values (i.e. from contextual information).

These results pushed us to ask which features the system does extract to make its decision.

State features

During its forward pass through the network, the input image is transformed to extract relevant features, resulting in high-dimensional vectors. One way to study these vectors is to project them in a two-dimensional space while preserving some inter-vectors relations such as the euclidean distance. One of the most used projection method is the t-SNE (t-distributed Stochastic Neighbor Embedding) projection [START_REF] Van Der Maaten | Visualizing high-dimensional data using t-sne[END_REF] which perform very well to project vectors issued by a convolutional neural network.

In the following experiment, we provide some states (images) to a pre-trained model, while recording vectors after the last convolutional layer, and the one before the last fully connected layer. These vectors are also linked to the information characterizing the input state (location, activity, uttered command, expected action, sensors activity). Projecting the vectors provide a bi-dimensional coordinate (0, 0) ≤ (x, y) ≤ (1, 1) where to represent any of the linked information.

Results presented in Figure 13(c,b), show that some information are identified and extracted by the convolutional part of the network. For example, coherent blobs can be spotted for the uttered command and the user location. By contrast, no group can be made for the activity (Figure 13a), which does not seem to be a relevant feature to extract from the image for the DQN. These results confirm the usual use of the user location as a contextual information. However, the user's activity is not exploited by the network. It might be possible that a lower level annotations, such as agitation level, could be extracted.

The last projection, presented in Figure 13d, provides information about the way the neural network transforms the space to make each class linearly separable. While there is a majority of "Do Nothing" action, we can identify some groups of common action type (icon type) and action place (icon color), proving that given states contain enough information to classify correctly most of them.

Discussion

The results of the experiments of a reinforcement learning method with a deep model for automatic adaptive decision-making in smart-homes show promises. In particular, the experiments show that:

• the system is able to use trivial user's interactions (rewards) to adapt a decision-making process;

• graphical representation of multi-modal heterogeneous data can be interpreted to make decisions;

• the system can adapt its behavior to a user and to faulty or uninstalled sensors;

• the deep model can learn relevant contextual information;

• learning can be done on-line, using a limited history of the past and no information about the future;

• the method can scale to big input space (raw sensor data).

All the work has been conceived to be reproducible and all the written source code as well as the graphic components are available on-line2 , as a set of Lua source code and SVG files. All these sources have been put under version control system (Git). This allows the reproduction of the experiments, freezing and tagging the source for each experiment. The source code follows a modular approach for improving re-usability of each component. Regarding the data used for the experiments, it can be retrieved separately 3 .

If Arcades exhibits interesting results, it opens new research questions and possible improvements. At the RL level, the reward function plays a very important role in driving the learning in an acceptable solution. In our previous study [START_REF] Brenon | Preliminary study of adaptive decision-making system for vocal command in smart home[END_REF] different reward functions were tried, taking account of the semantic distance between two actions (e.g. lighting a lamp instead of another) or the real distance between two action locations (lighting in the kitchen rather than the study). Nevertheless, these more sophisticated functions did not bring a significant improvement in the agent behavior that is why we decided to stick to a more simple one. Though this function led to better learning, it is worth investigating other reward function that would be closer to human evaluation (e.g. a confusion between a lamp in a room should be less problematic than a confusion of room). Similarly, in a real interactive setup, microphones can be used to acquire a more nuanced evaluation from the user, for example using a set of keywords like "No!", "Very good.", "Keep going.", etc. Another research direction would be to take into account implicit rewards from the user as done by Karami et al. [START_REF] Karami | User in the loop: Adaptive smart homes exploiting user feedback -state of the art and future directions[END_REF].

At a deep learning level, it is well-known that these models require an unreasonable quantity of data as well as a long time before reaching the convergence point. These are important factors for acceptability of the system by naive user and should be reduced as much as possible. If the pre-training phase is a first solution, others should be studied, such as the use of an expert system during the first interactions, waiting for Arcades to learn an appropriate model. Recent works propose different methods to leverage these parameters. For instance, Pritzel et al. [START_REF] Pritzel | Neural episodic control[END_REF] make an intensive use of few relevant samples to reduce the number of the overall required interactions. In parallel, Usunier et al. [START_REF] Usunier | Episodic exploration for deep deterministic policies: An application to starcraft micromanagement tasks[END_REF] propose a different strategy than the ϵ-greedy one, which speeds up the exploration of the different actions. Knowledge transfer could be another alternative which need more experiments to be validated.

Finally, paradigm shifts can also be interesting. The use of MDP as the underlying model of RL tasks is related to the computation complexity induced by more generic models like the POMDP [START_REF] Cassandra | Pomdp for dummies[END_REF]. However, these models are more suitable to handle the unknown and variable behavior of the user. The DQN approach being able to scale to big state space, it can be interesting to try to use more generic models like POMDP or MOMDP [START_REF] Araya-López | A closer look at MOMDPs[END_REF] as the underlying one of our RL task. More complex neural network architectures can also be implemented such as Recurrent Neural Networks (RNN, [START_REF] Lipton | A critical review of recurrent neural networks for sequence learning[END_REF]), allowing the system to better handle time dependency as well as implementing some attention mechanisms [START_REF] Xu | Show, attend and tell: Neural image caption generation with visual attention[END_REF].

Despite this need for improvements, this study show that deep reinforcement learning is feasible and shows promises for a life long adaptive and robust context aware decision making.

AppendixA. Hyper-parameters optimization

Through the whole available hyper-parameters of the system, we choose to make an exhaustive search for two of them. The target_q parameter defines the number of interactions during which a frozen network is used to compute the target Q-values before being updated with the current Q-network. The minibatch_size parameter defines the number of samples drawn to build a mini-batch used for training the network.

Mnih et al. [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] state that these two parameters are very important to provide a good stability of the performances. Nevertheless, as shown on the bottom left plot of the Figure A.14, the minibatch_size parameter heavily impact the execution time. In spite of improving the average reward per episode and F1-score, using a high minibatch_size make the system slower to process the same number of interactions. The bottom right plot provides an intuitive view of a compromise, relating the F1-score to the execution time. This lead us to choose a target_q of 4096 with a minibatch_size of 16.

However, another hyper-parameter, called update_freq, allows to reduce the execution time. The learning step of the agent is indeed made each update_freq interactions. Thus, increasing this number will execute the time-spending step less often, and yet need bigger mini-batches to reach the same level of performance. The Figure A.15 presents the same plots but with different hyper-parameters. As expected, a low update_freq increases the execution time. The plot of the score-time ratio lead us to choose a bigger minibatch_size parameter of 32, increasing the update_freq parameter to 12.

The old and new values of hyper-parameters are summed up in the Table A.5.

Table 1 :

 1 Summary of the studies related to context representation and decision-making in smart homes

Figure 1 :

 1 Figure1: A typical smart-home system equipped with a voice-user interface: use case of a user asking to turn on the light.

Figure 2 :

 2 Figure 2: The main components of Arcades and their interactions

Figure 3 :

 3 Figure 3: Examples of image generated for the agent input. From left to right : kitchen, bathroom, bedroom and study.

Figure 5 :

 5 Figure 5: Experiment protocol. After pre-training with synthetic data, the agent is adapted with data from 14 of 15 subjects and evaluated on the left out participant's data.

Figure 6 :

 6 Figure 6: The Domus smart-home

  tag real-raw-full-1

Figure 7 :

 7 Figure 7: Pre-training results

Figure 8 :

 8 Figure 8: Validation results

Figure 9 : 3 -Figure 10 : 3 -Figure 12 :

 9310312 Figure 9: Performances without motion sensors

Figure 13 :

 13 Figure 13: t-SNE projections

Figure A. 14 :

 14 Figure A.14: Metrics related to different choices of hyperparameters

  denotes a convolution layer (using n filters of size f , with a padding of p, and using a stride of k) and F Cn a fully connected linear layer (of width n).convolutional layers, reducing a 256 × 256 input image to features maps of sizes: 16 × 64 × 64, 32 × 21 × 21, 64 × 10 × 10, and 64 × 8 × 8. The features maps of the last layer are then rearranged in a vector of dimension 64 × 8 × 8 = 4096. This vector passes through fully connected linear layers of width 512 and 33 which is the number of possible actions. A rectified linear unit separates each layer.

Table 2 :

 2 Available actions for the agent to act on the environment.

Table 3 :

 3 Parameters of the experiment.

Table 4 :

 4 F-Measure of the pre-training and evaluation phases for the decision making task.

		Pre-training F-M (%) Validation F-M (%)
		synthetic data	real corpus
	Baseline 1	NC	46.0
	Arcades 2	100	67.5
	1. Learned and evaluated on humanly annotated data.	2. Learned and evaluated on raw data.

Table A . 5

 A5

	target_q	1024	4096	4096
	minibatch_size	64	16	32
	update_freq	3	3	12

: Evolution of the hyper-parameters Initial value Intermediate value Final value

Scalable Vector Graphics (SVG) is a widely-deployed, royalty-free, XML-based vector image format developed and maintained by the World Wide Web Consortium (W3C) (https://www.w3.org/Graphics/SVG/)

brenona.gricad-pages.univ-grenoble-alpes.fr/arcades/

http://sweet-home-data.imag.fr/


Fabien Acher

Volvo Gtt

Kerem Ege

Céline Sandier

Youssef Gerges

Vibro-Acoustic Modeling of a Trimmed Truck Cabin in Low Frequency Range to Tackle the Challenge of Weight Reduction

Vibro-acoustic modeling of a trimmed truck cabin in low frequency range to tackle the challenge of weight reduction

Introduction

In 2012-2015, the CLIC project (City Lightweight and Innovative Cab), a scientific collaboration gathering public and private organizations, was carried out to propose an innovative lighten truck cabin with the use of high-tensile steels. During this project, a particular focus was put on the acoustic analysis of the cabin. An important part of the project was dedicated to develop and validate numerical methods to assess the vibroacoustic behavior of a structure including the acoustic treatments (both for low and mid/high frequency ranges [START_REF] Maxit | SmEdA vibro-acoustic modeling of a trimmed truck cab in the midfrequency range[END_REF]).

In the low frequency range (typically below 300Hz), simulations of body-in-white structures coupled to cavities have been performed for several decades using the finite element method in modal coordinates. These simulations generally do not take into account the effect of the acoustic treatments. However, as shown in this paper, their influence can be significant on the noise level.

To close this gap, a numerical method for the vibro-acoustic response [0-200Hz] of the trucks cabin taking into account the effect of the acoustic treatments has been implemented. This paper summarizes the works done to correlate this method.

Possible numerical strategies are shortly reviewed. The strategy chosen for implementation is detailed. The measurement campaign performed to validate the method is introduced as well as the details of the FEM models. Finally, results are shown and comparisons between tests and simulations are discussed for various indicators (VTF/NTF/Noise reduction).

Numerical Strategy

In the automotive/truck industry, the FEM simulation a structure below 300Hz (body-in-white for example), coupled or not with a cavity, is generally done using modal methods (a modal extraction and a modal frequency response). Considering the number of degrees of freedom of the models (several millions), a direct approach, inverting the matrix in physical coordinates, is not realistic with today's computational resources. The modal approach is now well established and has demonstrated for several decades its ability to simulate accurately this kind of lightly damped problems.

A typical truck cabin structure is trimmed with acoustic treatments whose main function is to bring acoustic absorption in the cabin (trim panels), insulate from exterior noise (floormats and trim panels) or reduce the vibrations of the cabin floor (floormats). The materials of these treatments have frequency dependant properties and are generally heavily damped. For these reasons, it is difficult to represent them with modal methods.

To overcome this limitation several methods have been proposed:  A first group of methods based on super-elements reduction techniques is mentioned [START_REF] Coyette | From Body in White to Trimmed Body Models in the Low Frequency Range: a New Modeling Approach[END_REF]. The important computational resources associated (time and storage space) are a strong limitation for these methods.

10/19/2016  A second method, based on the energetic behaviour is proposed [START_REF] Van Den Nieuwenhof | Efficient analysis of large trimmed configurations using modal approaches[END_REF]. In this method, each mode can be affected by the acoustic components but the effect on the coupling between the modes is neglected.  A third method, described in [START_REF] Coyette | From Body in White to Trimmed Body Models in the Low Frequency Range: a New Modeling Approach[END_REF] and [START_REF] Bertolini | FE Analysis of a Partially Trimmed Vehicle using Poroelastic Finite Elements Based on Biot's Theory[END_REF] consists in condensing the impedance matrix of the acoustic treatments to its degrees of freedom interfaced with the structure or the cavity. The condensed impedance matrix is then projected into the modal space and injected in the modal impedance matrix of the complete system. This 3 rd method introduces coupling terms between the modes and overcomes the limitation of the 2 nd one. This 3 rd method, based on reduced impedance matrices of the trim components is chosen and implemented in the present work.

Theoretical Framework

The theoretical development of the approach based on reduced impedance matrices is presented.

The dynamic behaviour of a system made of a structure (such as the body-in-white) and its cavity, represented in modal coordinates can be written as:

( 𝑍 𝑆 (𝜔) 𝐶 𝑆𝐹 𝐶 𝑆𝐹 𝑇 𝑍 𝐹 (𝜔) ) ⋅ ( 𝛼 𝑆 (𝜔) 𝛼 𝐹 (𝜔) ) = ( Φ 𝑆 𝑇 ⋅ 𝐹 𝑆 Φ 𝐹 𝑇 ⋅𝐹 𝐹 𝜔 2 ) (1)
With

{ 𝑍 𝑆 (𝜔) = 𝐾 𝑆 + 𝑖 𝜔 𝐶 𝑆 -𝜔 2 𝑀 𝑆 𝑍 𝐹 (𝜔) = 1 𝜔 2 𝐾 𝐹 + 𝑖 𝜔 𝐶 𝐹 -𝜔 2 𝑀 𝐹 (2)
ω is the pulsation; K, C and M are the modal stiffness, modal damping and modal mass matrices for the structure and cavity. C SF is the modal structure-fluid coupling matrix. F are the excitations applied on the structure and the cavity.

An acoustic treatment T, such as the floormat represented Figure 1 is coupled to the structure through its lower surface (Γ S ) and to the cavity through its upper surface (Γ F ). The degrees of freedom of its finite element model can be separated into inner degrees of freedom (D inner ) and coupling degrees of freedom (D Γ ), located on the surfaces Γ s and Γ F . Its impedance matrix in physical coordinates can be written:

𝑍 𝑇 (𝜔) = ( 𝑍 𝑖𝑛𝑛𝑒𝑟 (𝜔) 𝑍 𝑖𝑛𝑛𝑒𝑟,Γ (𝜔) 𝑍 Γ,𝑖𝑛𝑛𝑒𝑟 (𝜔) 𝑍 Γ (𝜔) ) (3)
In physical coordinates, its dynamic behaviour is described by:

( 𝑍 𝑖𝑛𝑛𝑒𝑟 (𝜔) 𝑍 𝑖𝑛𝑛𝑒𝑟,Γ (𝜔) 𝑍 Γ,𝑖𝑛𝑛𝑒𝑟 (𝜔) 𝑍 Γ (𝜔) ) ⋅ ( 𝐷 𝑖𝑛𝑛𝑒𝑟 (𝜔) 𝐷 Γ (𝜔) ) = ( 0 ℎ Γ ) (4)
Where h Γ are the excitation applied on the coupling surfaces (there is no excitation on the inner degrees of freedom).

Developing this system, it is possible to describe the inner degrees of freedom in function the degrees of freedom of its interfaces with the structure and the cavity. The dynamic behaviour of the treatment can then be reduced to its interface degrees of freedom:

(𝑍 Γ (𝜔) -𝑍 Γ,𝑖𝑛𝑛𝑒𝑟 (𝜔) ⋅ 𝑍 𝑖𝑛𝑛𝑒𝑟 -1 (𝜔) ⋅ 𝑍 𝑖𝑛𝑛𝑒𝑟,Γ (𝜔)) ⏟ 𝑍 𝑟𝑒𝑑 ⋅ 𝐷 Γ (𝜔) = ℎ Γ (5)
This reduced impedance matrix Z red is then projected into the structure and cavity modes and injected into equation [START_REF] Maxit | SmEdA vibro-acoustic modeling of a trimmed truck cab in the midfrequency range[END_REF].

𝒵 𝑟𝑒𝑑 (𝜔) = ( Φ 𝑆 𝑇 0 0 Φ 𝐹 𝑇 ) ⋅ 𝑍 𝑟𝑒𝑑 (𝜔) ⋅ ( Φ 𝑆 0 0 Φ 𝐹 ) (6)
( 𝑍 𝑆 (𝜔) + 𝒵 𝑟𝑒𝑑,𝑆 (𝜔) 𝐶 𝑆𝐹 + 𝒵 𝑟𝑒𝑑,𝐶 (𝜔) 𝐶 𝑆𝐹 𝑇 + 𝒵 𝑟𝑒𝑑,𝐶 𝑇 (𝜔) 𝑍 𝐹 (𝜔) + 𝒵 𝑟𝑒𝑑,𝐹 (𝜔) ) ⋅ ( 𝛼 𝑆 (𝜔) 𝛼 𝐹 (𝜔) ) = ⋯ = ( Φ 𝑆 𝑇 ⋅ 𝐹 𝑆 Φ 𝐹 𝑇 ⋅𝐹 𝐹 𝜔 2 ) (7)
The "updated" system is then solved to retrieve the vibro-acoustic response including the effect of the acoustic treatment.

The computation of the reduced impedance matrix is the "costly" step since it needs the inversion of a dense matrix. The computational cost will increase with the size of the treatment.

In this method, it is considered that the presence of the acoustic treatment will not change the mode shapes or, in other words, that those mode shapes can be used as a mathematical basis for the representation of the system response.

Simulation Model

The structure, the cavity and the acoustic treatments of the truck cabin are modelled with FEM method.

Structure

The structure model corresponds to the body-in-white of the structure with the windows. The system that decouples the cabin from the ground as well as the adaptations performed on the cab to close it completely are included in the model (see Figure 5 Error! Reference source not found.). Only the roof hatch is simplified to a local mass applied on its connection points, its vibro-acoustic coupling with the cavity is not considered. The damping treatments made of bituminous pads are included in this model. A view of the finite element mesh of the cabin structure can be seen Figure 2.

10/19/2016

Cavity

The volume of the cavity varies depending on the acoustic treatments present. For each configuration of treatments, the exact cavity volume is meshed. An average mesh size of 35mm is used to ensure the quality of the modal extraction up to 900Hz. The cavity mesh without any acoustic treatment is shown Figure 3.

Acoustic treatments

The acoustic treatments considered are the floormats (made of one layer of polyurethane foam and one heavy layer of rubber material) and the trim panels (made of compressed felt, covered with a nonwoven scrim and partially reinforced with an absorbent felt). They are divided in five different parts (see Figure 4): They are meshed with standard meshing requirements. To avoid continuity of displacement between the structure and the treatments (which does not happen in the reality) an air gap is modelled between them. The porous and fibrous components models rely on the Biot theory. The Johnson-Champoux-Allard model is used [START_REF] Johnson | Theory of dynamic permeability and tortuosity in fluid saturated porous media[END_REF], [START_REF] Champoux | Dynamic tortuosity and bulk modulus in air-saturated porous media[END_REF].

This model requires specific material properties, including the resistivity, the porosity, the tortuosity, the viscous length and the thermal length. Their mechanical properties (Young modulus, Poisson's ratio, loss factor) are also needed for an accurate modelling.

A proper characterization of all the materials is performed. The air flow resistivity is measured following the standard ISO-9053. The porosity is measured with a "porosimeter" following the method described in [START_REF] Beranek | Acoustic impedance of porous materials[END_REF]. A third measurement in a Kundt tube allows determining the remaining acoustic parameters from their analytical expression in the Johnson-Champoux-Allard model [START_REF] Johnson | Theory of dynamic permeability and tortuosity in fluid saturated porous media[END_REF], [START_REF] Champoux | Dynamic tortuosity and bulk modulus in air-saturated porous media[END_REF].

10/19/2016

Simulation Process:

The calculation needs several steps that must be executed sequentially:

1. In a first step, the modes of the cavity and the modes of the structure are extracted. The maximum frequency of extraction is taken high enough to avoid modal truncature effect. The modes shapes are stored on disk. The modal damping matrix of the structure, including the effect of the bituminous damping pads, is also extracted and stored on disk. The MSC Nastran software package is used for this step. The Actran software package is used to perform steps 2 to 4.

2. In a second step, for each treatment the impedance matrices are reduced to the degrees of freedom coupled with the structure and cavity . They are immediately projected into the modal spaces and stored on disk.

3. In a third step, the fluid / structure coupling in areas where no acoustic components are present is computed. When acoustic components are present, the fluid/structure coupling is contained on the reduced impedance matrices.

4. As a final step, the modal frequency response of the complete system is assembled and solved for the given excitations.

Measurements Campaign

The measurement campaign is performed to acquire data to be compared with simulation results.

In the measurement set-up the cabin is decoupled from its support using air springs fixed at cab anchorages (see Figure 5Error! Reference source not found.). The air springs provide a decoupling frequency below 5Hz. All holes (cables pass-through, steering column hole...) were capped to ensure a proper acoustic sealing of the cavity (see Figure 5Error! Reference source not found.). One of the main remarks from the measurement is that the effect of the acoustic treatments occurs above 50Hz (see Figure 8Error! Reference source not found.).

10/19/2016

Results and Discussions

Different indicators are looked at. They are listed with an increasing complexity:  The inertance at the anchorage points of the cabin on the chassis or at the centre of the panels of the cabin; 

The VTF between the anchorage points and the panel points; 

The NTF between the panel points and the driver's ears; 

The NTF between the cab anchorage points and the driver's ears.

The results are analysed up to 230Hz which corresponds to the maximum frequency excited by the 2 nd harmonic of the firing order of the engine (3 rd order for a 6-cylinders engine). Among all the results available, a few are presented to illustrate the conclusions issued from their analysis. Three excitation points are considered as shown Figure 10.

A first one close to the front left anchorage point of the cabin on the chassis (excitation in transversal direction), a second one on the tunnel area below the tunnel floormat (excitation in vertical direction) and a third one on the rear panel behind the rear trim panel (excitation in longitudinal direction). The NTF are given to the driver's left ear.

Correlation of the bare model

The vibro-acoustic response of the bare model, without any acoustic treatment, is simulated and compared to the measurements.

The correlation work does not aim at artificially modifying local parameters or setting non-physical materials to exactly match the measurement on the whole frequency range. It rather consists in modifying global parameters (junction modelling, material young modulus...) to identify and bring closer the global or panel modes of the structure. The goal is to assess that the simulation represents well the measurements. At the end of the correlation, the remaining shifts between the simulation and experimental results shall be known and understood.

Examples of final comparisons for inertance results are shown in Figure 11 for the front left anchorage point and Figure 12 for the rear wall point.

Correlation is satisfying both in term of spectrum content and levels.

10/19/2016 The VTF between the anchorage points and the centres of panels is then analysed to assess the correlation. Example of VTF correlation between the front anchorage point and the rear wall point is displayed Figure 13. The simulated and measured VTFs show similar spectrum and levels.

The differences are mainly inherited from the differences observed on the local inertances.

Finally the measured and simulated NTFs are compared. Figure 14 and Figure 15 show NTFs to driver's ear from the front left anchorage point and from the rear wall point respectively.

The good agreement on the structure response is similarly observed on the acoustic response. Results from the mean NTF (average of NTF from 18 microphones) have similar level of correlation. Main discrepancies can be observed below 50Hz. This is below the first cavity mode and corresponds to the frequency range where the acoustic treatments have no effect (see Figure 8). A possible explanation lies in the handle of rigid body modes but this is not verified since correlation efforts are put on frequencies above 50Hz.

Results with a single acoustic treatment

In a first step, the effect of the floormat and the rear trim panel taken independently is observed.

Floormats only

For a cabin equipped with the floormats only, the NTF from the tunnel point to the driver's ear is plot Figure 16. On the same plot, the noise reduction brought by the floormats is added both as measured 10/19/2016 and simulated. This allows freeing the results from the discrepancies of the bare model.

Measured and simulated curves show similar trends and levels. When looking at the noise reduction between the bare cabin and the cabin equipped with floormats, the noise reduction is correctly simulated. Levels of noise reduction are close. They are sometimes shifted in frequency. This is inherited from the frequency shift between measured and simulated modes in the bare model as the floormats mainly damp the panel modes. Such floormats made of a thick layer of polyurethane and a heavy layer of rubber have a resonance frequency. Noise from the panel is generally amplified at this frequency. For the floormats considered in this study, the resonance occurs around 160Hz. The drop of noise reduction around this frequency both in measurement and simulation shows that the simulation properly models this phenomenon.

Rear trim panel only

For a cabin equipped with the rear trim panel, the NTFs from the rear wall point to the driver's ear are shown Figure 17.

Measured and simulated curves show similar trend and levels. The noise reduction given by the rear trim panel only is correctly simulated. This effect can be mainly observed when frequency increases (> 180Hz). The over-estimated sound reduction at some frequencies (135Hz/145Hz) is due to anti-resonance phenomena.

Below 50Hz, although the effect of the panel is negligible, simulation shows noise reduction. This difference is not explained so far but the possible interaction of rigid body modes with the treatment is not excluded.

Results with all acoustic treatments

For a cabin equipped with all the acoustic treatments (headliner, rear trim panel and all floormats), the NTFs from the anchorage points of the cabin to the driver's ear are analysed. Figure 18 and Figure 19 show comparison of the NTF and the Mean NTF (average from 18 microphones). The correlation between the measurements and the simulation is satisfying. The noise reduction given by the treatments is properly simulated which is the main objective of the works.

By comparing with the results from the configurations without treatments and with a single treatment, it is possible to analyse the treatment responsible for a particular effect or that is the cause of a discrepancies at a given frequency.

This level of correlation of the noise reduction between measurement and simulation validates the method and the modelling. One can be confident that the behaviour of the treatments is correctly modelled. However, the quality of the results below 50Hz remains low, need to be explained and improved.

Conclusion

The challenge of vehicle weight reduction requires accurate methods to predict its vibro-acoustic behavior including the acoustic treatments.

The method presented in this paper aims at predicting the vibroacoustic response of the cabin including the acoustic treatments while using a modal scheme (0-300Hz).

A measurement campaign is performed to get reference data to compare with numerical results.

The FEM model without any treatment is first correlated to the measurements. This is an important step to limit the errors on the models considering the treatments, as they are based on this first model. A good level of correlation is obtained both on VTF and NTF indicators.

As final result, the FEM model with all the treatments is compared to the measurements. Satisfying correlation is obtained on the VTF and NTF and very similar levels of noise reduction are obtained above 50Hz.

Although the modelling needs to be improved below 50Hz, the method is considered as validated for the analysis of future weight reductions of truck cabins.

Figure 1 :

 1 Figure 1: View of a floormat

Figure 2 :

 2 Figure 2: View of the cabin structural model

Figure 3 :

 3 Figure 3: View of the cavity mesh for the bare configuration



  the headliner;  the rear trim panel;  the driver floormat;  the tunnel floormat;  the passenger floormat.

Figure 4 :

 4 Figure 4: View of the 5 acoustic treatments

Figure 5 :

 5 Figure 5: View of the decoupling system and the caps to close the cavity

Figure 6 :

 6 Figure 6: View of the microphones positionned inside the cabinThe cabin is excited using an impact hammer in the normal direction to the surface close to all accelerometers. For each excitation, all transfer functions are measured including:Local inertance;  Vibration Transfer Functions to all accelerometers;  Noise Transfer Functions to all microphones.

Figure 7 :

 7 Figure 7: View of the accelerometers positionned on the rear wall of the cabin Different configurations are measured in order to determine the influence of the acoustic treatments, both individually and grouped:  Configuration 1: cabin with all acoustic treatments (floormats, rear trim panel and headliner);  Configuration 2: cabin with floormats and rear trim panel (without headliner);  Configuration 3: cabin with floormats only;  Configuration 4: cabin with rear trim panel only;  Configuration 5: cabin without any treatment.

Figure 8 :

 8 Figure 8: Effect of the acoustic treatment on the NTF.The use of 18 microphones allows measuring the mean pressure within the cavity. To validate that 18 microphones are enough to measure the mean pressure a simulation is performed on the configuration without treatment. The mean pressure is computed and compared using all the cavity volume and the 18 microphones. Results are presented Figure9Error! Reference source not found.. Results are very close and around 1dB difference maximum is observed. The measurement of the mean NTF from 18 microphone is validated.

Figure 9 :

 9 Figure 9: Comparison of mean NTF obtained from the cavity volume and from 18 microphones

Figure 10 :

 10 Figure 10: View of the excitation points for the results presented

Figure 11 :

 11 Figure 11: Measured and simulated inertance at the front left anchorage point

Figure 13

 13 Figure 13 Measured and simulated VTF between the front left anchorage point and a point on the rear wall

Figure 14 :

 14 Figure 14: Measured and simulated NTF between the front left anchorage point and the driver's ear

Figure 16 :

 16 Figure 16: Measured and simulated NTF and noise reduction in presence of the floormats. Excitation from the tunnel point and delta

Figure 17 :

 17 Figure 17: Measured and simulated NTF and noise reduction in presence of the rear trim panel. Excitation from a rear wall point

Figure 18 :

 18 Figure 18: Measured and simulated NTF and noise reduction in presence of all the acoustic treatments. Excitation from the front left anchorage point.

Figure 19 :

 19 Figure 19: Measured and simulated Mean NTF (average 18 microphones) and noise reduction in presence of all the acoustic treatments. Excitation from the front left anchorage point

Acknowledgments

The authors acknowledge Oleg Danilov (Volvo GTT), for his valuable advices, the FFT / MSC Software experts for their support in development of the model and the Matelys Company for the collaboration in the characterization of the porous materials.

The measurement works were co-funded by the French government (FUI 12 -Fonds Unique Interministériel) and European Union (FEDER -Fonds européen de développement régional). It was carried out in the framework of the LabEx CeLyA ("Centre Lyonnais d'Acoustique", ANR-10-LABX-60) and the research project CLIC ("City Lightweight Innovative Cab") labelled by LUTB cluster (Lyon Urban Truck and Bus), in partnership with Renault Trucks, Arcelor-Mittal, ACOEM, ALTRAN, FEMTO-ST (Univ. de Franche-Comté) and LVA (INSA de Lyon).

Contact Information

Fabien Acherfabien.acher@volvo.com Volvo GTT, Lyon, France

Definitions/Abbreviations