Timothée Lesort
email: timothee.lesort@ensta-paristech.fr

Natalia Díaz-Rodríguez
email: natalia.diaz@ensta-paristech.fr

Jean-François Goudou
email: jean-francois.goudou@thalesgroup.com

David Filliat
email: david.filliat@ensta-paristech.fr

State Representation Learning for Control: An Overview

Keywords: State Representation Learning, Low Dimensional Embedding Learning, Learning Disentangled Representations, Disentanglement of control factors, Robotics, Reinforcement Learning

Representation learning algorithms are designed to learn abstract features that characterize data. State representation learning (SRL) focuses on a particular kind of representation learning where learned features are in low dimension, evolve through time, and are influenced by actions of an agent. The representation is learned to capture the variation in the environment generated by the agent's actions; this kind of representation is particularly suitable for robotics and control scenarios. In particular, the low dimension characteristic of the representation helps to overcome the curse of dimensionality, provides easier interpretation and utilization by humans and can help improve performance and speed in policy learning algorithms such as reinforcement learning.

This survey aims at covering the state-of-the-art on state representation learning in the most recent years. It reviews different SRL methods that involve interaction with the environment, their implementations and their applications in robotics control tasks (simulated or real). In particular, it highlights how generic learning objectives are differently exploited in the reviewed algorithms. Finally, it discusses evaluation methods to assess the representation learned and summarizes current and future lines of research.

Introduction

Robotics control and artificial intelligence (AI) in a broader perspective heavily rely on the availability of compact and expressive representations of the sensor data. Designing such representations has long been performed manually by the designer, but deep learning now provides a general framework to learn such representations from data. This is particularly interesting for robotics where multiple sensors (such as cameras) can provide very high dimensional data, while the robot objective can often be expressed in a much lower dimensional space (such as the 3D position of an object in a manipulation task). This low dimensional representation, frequently called the state of the system, has the crucial role of encoding essential information (for a given task) while discarding the many irrelevant aspects of the original data. By Low dimensional, we mean that the learned state dimension is significantly smaller than the dimensionality of the observation space.

Such state representation is at the basis of the classical reinforcement learning (RL) framework [Sutton, 1998] in which an agent interacts with its environment by choosing an action as a function of the environment state in order to maximize an expected (discounted) reward. Following this framework, we call observation the raw information provided by one or several of the robot sensors, and we call state a compact depiction of this observation that retains the information necessary for the robot to choose its actions.

While deep reinforcement learning algorithms have shown that it is possible to learn controllers directly from observations [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], reinforcement learning (or other control algorithms) can take advantage of low dimensional and informative representations, instead of raw data, to solve tasks more efficiently [START_REF] Munk | Learning state representation for deep actor-critic control[END_REF]. Such efficiency is critical in robotic applications where experimenting an action is a costly operation. In robotics, as well as in machine learning, finding and defining interesting states (or features) for control tasks usually requires a considerable amount of manual engineering. It is therefore interesting to learn these features with as little supervision as possible. The goal is thus to avoid direct supervision using a true state, but instead use information about the actions performed by the agent, their consequences in the observation space, and rewards (even if sparse, and when available). Along with this information, one can also set generic constraints on what a good state representation should be [Jonschkowski andBrock, 2015, Lesort et al., 2017].

Feature learning in general is a wide domain which aims at decomposing data into different features that can faithfully characterize it. It has been a particular motivation for deep learning to automatically learn a large range of specific feature detectors in high dimensional problems. State representation learning (SRL) is a particular case of feature learning in which the features to learn are low dimensional, evolve through time, and are influenced by actions or interactions. SRL is generally framed in a control setup constrained to favor small dimensions to characterize an instance of an environment or an object, often with a semantic meaning that correlates with some physical feature. The physical feature can be, for instance, a position, distance, angle or an orientation. The objective of SRL is to take advantage of time steps, actions, and optionally rewards, to transform observations into states: a vector of a reduced set of the most representative features that is sufficient for efficient policy learning. It is also worth distinguishing between feature learning on a process that is only observed, and learning the state representation of a process in which the learning agent possesses embodiment and acts. The first one considers learning directly from observations, e.g., pixels, and leaves no room for the agent to act. The latter gives opportunity to exploit more possibilities to learn better representations by balancing between exploration and exploitation, e.g., active learning, or artificial curiosity [START_REF] Péré | Unsupervised learning of goal spaces for intrinsically motivated goal exploration[END_REF], Pathak et al., 2017].

As stated above, learning in this context should be performed without explicit supervision. In this article we therefore focus on SRL where learning does not have the pattern recognition regression or classification sense, but rather the sense of the process of model building [START_REF] Lake | Building Machines That Learn and Think Like People[END_REF]. Building such models can then exploit a large set of objectives or constraints, possibly taking inspiration from human learning. As an example, infants expect inertial objects to follow principles of persistence, continuity, cohesion and solidity before appearance-based elements such as color, texture and perceptual goodness. At the same time, these principles help guide later learnings such as object' rigidity, softness and liquids properties. Later, adults will reconstruct perceptual scenes using internal representations of the objects and their physically relevant properties (mass, elasticity, friction, gravity, collision, etc.) [START_REF] Lake | Building Machines That Learn and Think Like People[END_REF]. In the same way, the SRL literature may make use of knowledge about the physics of the world, interactions and rewards whenever possible as a semi-supervision or self-supervision that aids the challenge of learning state representations without explicit supervision. Recently, several different approaches have been proposed to learn such state representation. In this review paper, our objective is to present and analyze those different approaches, highlight their commonalities and differences, and to propose further research directions. We extend a previously published review [START_REF] Böhmer | Autonomous learning of state representations for control: An emerging field aims to autonomously learn state representations for reinforcement learning agents from their real-world sensor observations[END_REF] with the most recent and rapidly evolving literature of the past years and focus on approaches that learn low dimensional Markovian representations without direct supervision, i.e., exploiting sequences of observations, actions, rewards and generic learning objectives. The works selected in this survey mostly evaluate their algorithms in simulations where agents interact with an environment. More marginally, some SRL algorithms are tested on real settings such as robotics tasks, e.g., manipulation or exploration as detailed in Section 4.4.

In the remainder of the paper, we first introduce the formal framework and notation, then present the objectives that can be used to learn state representations, and discuss the implementation aspects of these approaches before summarizing some current and future lines of research.

Formalism and definitions

SRL Formalism

The nomenclature we use is very close to the one used in reinforcement learning literature [Sutton, 1998] and is illustrated in Fig. 1. We define an environment E where an agent performs actions a t ∈ A at time step t and where A is the action space (continuous or discrete). Each action makes the agent transition from a true state st to st+1 which is unknown but assumed to exist. We call the true state space S. The agent obtains an observation of E from its sensors, denoted o t ∈ O where O is the observation space.

Optionally, the agent may receive a reward r t . The reward is given at st by a reward function designed to lead the agent to a certain behavior that solves a task. The reward is optional as learning a state representation don't aim to solve a task, but is often present as one of the goal of SRL may be to improve task learning performance.

The SRL task is to learn a representation s t ∈ S of dimension K with characteristics similar to those of st without using st . More formally, SRL learns a mapping φ of the history of observation to the current state s t = φ(o 1:t ). Note that actions a 1:t and rewards r 1:t can also be added to the parameters of φ [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF]. In this paper, we are specifically interested in the particular setting in which this mapping is learned through proxy objectives without access to the true state st . This family of approaches is called unsupervised or self-supervised.

Finally, we note ôt the reconstruction of o t (similarly for ât and rt ), that will be used in various SRL approaches.

SRL approaches

Based on the previously defined notations, we can briefly summarize the common strategies used in state representation learning that are detailed in the next sections. In the following, θ represents the parameters optimized by minimizing the model's loss function. In most of the approaches we present, this model is implemented with a neural network. • Reconstructing the observation: learning the function φ (Eq. 1) so that it is possible to reconstruct the observation with a decoder φ -1 (Eq. 2) by minimizing the reconstruction error between the original observation and its predicted reconstruction. The reconstruction is learned under different constraints that give to s t specific characteristics (e.g., dimensionality constraints, local denoising criterion [START_REF] Vincent | Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion[END_REF], sparse encoding constraints [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF], etc.) (Fig. 2).

s t = φ(o t ; θ φ ) (1) ôt = φ -1 (s t ; θ φ -1 ) (2)
where θ φ and θ φ -1 are the parameters learned for the encoder and decoder, respectively.

• Learning a forward model:

A forward model predicts s t+1 from o t or s t and a t (Fig. 3). In this context, we want to learn the mapping φ from o t to s t using the model that predicts s t+1 from o t . Hence the prediction work in two steps, first encoding from o t to s t then transition from s t to ŝt+1 . We can not compute any error on s t , however at t + 1 the model can learn from the error between ŝt+1 and s t+1 . The error is back propagated through the transition model and the encoding model. Consequently the methods allows to learn a model φ.

ŝt+1 = f (s t , a t ; θ f wd ) (3)
Learning such a model makes it possible to impose structural constraints on the model for state representation learning. For example, the forward model can be constrained to be linear between s t and s t+1 , imposing that the system in the learned state space follows simple linear dynamics.

Learning such model enforces that the state encodes enough information to recover the action that modified the state (Fig. 4).

• Using prior knowledge to constrain the state space: A last approach is to handle SRL by using specific constraints or prior knowledge about the functioning, dynamics or physics of the world (besides the constraints of forward and inverse models) such as the temporal continuity or the causality principles that generally reflect the interaction of the agent with objects or in the environment [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF]. Priors are defined as objective or loss functions L, applied on a set of states s 1:n (Fig. 5), to be minimized (or maximized) under specific condition c. An example of condition can be enforcing locality or time proximity within the set of states.

Loss = L prior (s 1:n ; θ φ |c) (5)
All these approaches are detailed in Section 3.

State representation characteristics

Besides the general idea that the state representation has the role of encoding essential information (for a given task) while discarding irrelevant aspects of the original data, let us detail what the characteristics of a good state representation are.

In a reinforcement learning framework, the authors of [START_REF] Böhmer | Autonomous learning of state representations for control: An emerging field aims to autonomously learn state representations for reinforcement learning agents from their real-world sensor observations[END_REF] defines a good state representation as a representation that is:

• Markovian, i.e. it summarizes all the necessary information to be able to choose an action within the policy, by looking only at the current state.

• Able to represent the true value of the current state well enough for policy improvement.

• Able to generalize the learned value-function to unseen states with similar futures.

• Low dimensional for efficient estimation.

Note that these are some characteristics expected of the state representation, but they cannot be used for learning this representation. Instead, they can later be verified by assessing the task performance for a controller based on the learned state. Note also that multiple state representations can verify these properties for a given problem and that therefore, there is no unique solution to the state representation learning problem. We detail this problem when discussing the evaluation of the learned state space in Section 4.3.

State representation learning can also be linked with the idea of learning disentangled representations that clearly separate the different factors of variation with different semantics. Following [START_REF] Achille | A Separation Principle for Control in the Age of Deep Learning[END_REF], a good representation must be sufficient, as efficient as possible (i.e., easy to work with, e.g., factorizing the data-generating factors), and minimal (from all possible representations, take the most efficient one).

The minimal assumption is comparable to the simplicity prior [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF]. It assumes that only a small number of world properties are relevant, and that there exists a low dimensional state representation of a higher level input observation. Related to Occam's razor, this prior favors state representations that exclude irrelevant information to encourage a lower dimensionality.

The efficiency aspect of the representation means that there should be no overlapping between dimensions of the learned state features. Unfortunately, independence of features alone may not be enough to assure a good quality of representations and guarantee a disentanglement of factors of variation [START_REF] Thomas | Independently controllable factors[END_REF]. Higher level abstractions can, however, allow to improve this disentanglement and permit easier generalization and transfer. Cues to disentangle the underlying factors can include spatial and temporal scales, marginal independence of variables, and controllable factors [START_REF] Thomas | Independently controllable factors[END_REF].

State representation learning applications

The main interest of SRL is to produce a low dimensional state space in which learning a control policy will be more efficient. Indeed, deep reinforcement learning in the observation space has shown spectacular results in control policy learning [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], Lillicrap et al., 2015, Mnih et al., 2016] but is known to be computationally difficult and requires a large amount of data [START_REF] Rusu | Progressive neural networks[END_REF]. Separation of representation learning and policy learning is a way to lighten the complete process. As described in most of the reviewed papers [START_REF] Mattner | Learn to swing up and balance a real pole based on raw visual input data[END_REF], Watter et al., 2015, van Hoof et al., 2016, Munk et al., 2016, Curran et al., 2016, Wahlström et al., 2015, Shelhamer et al., 2017, Oh et al., 2017], this approach is used to make reinforcement learning faster in time and/or lighter in computation.

SRL can be particularly relevant with multimodal observations that are produced by several complementary sensors with high dimensionality as is, for example, the case of autonomous vehicles. Low dimensional representations are then key to make an algorithm able to take decisions from hidden factors extracted from these complementary sensors. This is for instance the case of representation learning from different temporal signals in [Duan, 2017, Bohg et al., 2017]. Audio and images are blended in [START_REF] Yang | Deep multimodal representation learning from temporal data[END_REF] while RGB and depth are combined in [Duan, 2017]. SRL can also be used in a transfer learning setting by taking advantage of a state space learned on a given task to rapidly learn a related task. This is for example the case in [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF] where a state space related to a robot position is learned in a given navigation task and reused to quickly learn another navigation task. SRL is also used as pretraining for transfer to other applications afterwards such as reinforcement learning [START_REF] Munk | Learning state representation for deep actor-critic control[END_REF], Oh et al., 2017].

For concrete examples on SRL application scenarios see Section 4.4.

Another case where SRL could be useful is in the application of Evolution Strategies (ES) for robot control learning [START_REF] Stulp | Robot Skill Learning: From Reinforcement Learning to Evolution Strategies[END_REF]. Evolution strategies are a family of black box optimization algorithms that do not rely on gradient descent and can be an alternative to RL techniques (such as Q-learning and policy gradients) but are less adapted to high-dimensional problems. Indeed, the convergence time of ES algorithm depends on the dimension of the input: the larger the dimension is, the larger amount of solutions ES has to explore [START_REF] Stulp | Robot Skill Learning: From Reinforcement Learning to Evolution Strategies[END_REF]. ES optimization methods have shown to be efficient for deep reinforcement learning [START_REF] Conti | Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents[END_REF] but they could then take a clear advantage of a lower dimension input to explore faster the parameter space than using raw data [START_REF] Alvernaz | Autoencoder-augmented Neuroevolution for Visual Doom Playing[END_REF].

Learning objectives

In this section, we review what objectives can be used to learn a relevant state representation. A schema detailing the core elements involved in each model's loss function was introduced in Fig. 2345, which highlights the main approaches to be described here. This section touches upon machine learning tools used in SRL such as auto-encoders or siamese networks. A more detailed description of these is later addressed in Section 4.

Reconstructing the observation

A first idea that can be exploited is the fact that a true state, along with some noise, was used to generate the observation. Under the hypothesis that the noise is not too large, compressing the observation should retain the important information contained in the true state. While this idea is very often exploited with dimensionality reduction algorithms [Fodor, 2002] such as Principal Component Analysis (PCA), we focus here on the recent approaches specifically dealing with state representation.

The PCA algorithm is a linear transformation able to compress and decompress observations with minimal reconstruction error. PCA have been exploited to reduce the dimensionality of the state space during learning [START_REF] Curran | Dimensionality reduced reinforcement learning for assistive robots[END_REF]. By projecting images into a 3-or 4-dimensional space, it is possible to produce a state that is used by a reinforcement learning algorithm and that reduces the convergence time in Super Mario games [START_REF] Karakovskiy | [END_REF] and different simulations such as Swimmers or Mountain Car.

Auto-encoders are models that learn to reproduce their input under constraints on their internal representations such as dimensionality constraints (Fig. 2). Their architecture can therefore be used to learn a particular representation in low dimensions s t by reconstructing o t .

Simple auto-encoders can be used to learn 2D representation of a real pole from raw images [START_REF] Mattner | Learn to swing up and balance a real pole based on raw visual input data[END_REF] (see Section 4.4 on evaluation scenarios). After training, the encoding vector from the AE is used to learn a controller to balance the pole. An auto-encoder whose internal representation is constrained to represent a position that serves as input to a controller is also presented in [START_REF] Finn | Deep spatial autoencoders for visuomotor learning[END_REF] and [START_REF] Alvernaz | Autoencoder-augmented Neuroevolution for Visual Doom Playing[END_REF]. The proposed model learns a state representation from raw pixels of respectively a PR2 robot's hand and an agent in the VizDoom environment.

These models, based on auto-encoders that reconstruct the observation at the same time step, can however learn only if the factors of variations are only linked to the actual state, or if very prominent features exists [START_REF] Lesort | Unsupervised state representation learning with robotic priors: a robustness benchmark[END_REF]. In order to relax this assumption, it is possible to reconstruct observations from other time steps or to use constraints on the evolution of the state (as will be more detailed in Section 3.2) to focus reconstruction on features relevant to the system dynamics.

An example of auto-encoder tuned to take into account the system dynamics is proposed in [START_REF] Goroshin | Learning to linearize under uncertainty[END_REF] where an AE with a siamese encoder projects sequences of images into a state representation space S with constraints on the transition between s t and s t+1 to be linear. They use observations at several time steps in order to take time into account in the representation, and predict future observations through a single decoder that reconstructs ôt+1 . This makes the model able to learn representations that are related to several time steps and filter out random features of the environment.

The idea of using an auto-encoder to learn a projection into a state space where transitions are assumed to be linear has also been used by [START_REF] Watter | Embed to control: A locally linear latent dynamics model for control from raw images[END_REF]. The model presented, "Embed To Control" (E2C), consists of a deep generative model that learns to generate image trajectories from a linear latent space.

Extending [START_REF] Watter | Embed to control: A locally linear latent dynamics model for control from raw images[END_REF], the representation with dynamic constraints can be learned on policy at the same time as a reinforcement learning algorithm learns a task [van Hoof et al., 2016]. They compared different types of auto-encoders to learn visual and tactile state representations and use this representation to learn manipulation task policies for a robot. Sharing the same idea of embedding dynamic constraints into auto-encoders, Deep Variational Bayes Filters (DVBF) are an extension of Kalman filters which learn to reconstruct the observation based on a nonlinear state space using variational inference [START_REF] Karl | Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data[END_REF]. The reconstruction from a non linear state space based on a model inspired by a Deep Dynamical Model (DDM) [START_REF] Wahlström | From Pixels to Torques: Policy Learning with Deep Dynamical Models[END_REF] and E2C [START_REF] Watter | Embed to control: A locally linear latent dynamics model for control from raw images[END_REF] is proposed in [START_REF] Assael | Data-efficient learning of feedback policies from image pixels using deep dynamical models[END_REF]. It is argued that the model is adapted for better training efficiency and it can learn tasks with complex non-linear dynamics [START_REF] Assael | Data-efficient learning of feedback policies from image pixels using deep dynamical models[END_REF]. Their result shows improvements over the PILCO model [START_REF] Deisenroth | Pilco: A model-based and data-efficient approach to policy search[END_REF], which learns a state representation by only minimizing the reconstruction error without constraining the latent space.

Learning a forward model

Last subsection reviewed how reconstructing an observation is useful to learn state representations. Now we will review how temporal dynamics of the system can also help the same purpose. Therefore we present approaches that rely on learning a forward model to learn a state space. The general idea is to force states to efficiently encode the information necessary to predict the next state (Fig. 3).

As described in Section 2, in the case of the forward models we study here, the model is used as a proxy for learning s t . The model firstly makes a projection from the observation space to the state space to obtain s t and applies a transition to predict ŝt+1 . The error is computed by comparing the estimated next state ŝt+1 with the value of s t+1 derived from the next observation o t+1 at the next time step.

Note that forward models can benefit from the observation reconstruction objective presented in Section 3.1. As an example, the works previously presented in Section 3.1 [START_REF] Goroshin | Learning to linearize under uncertainty[END_REF], van Hoof et al., 2016, Watter et al., 2015, Assael et al., 2015, Karl et al., 2016] belong to the auto-encoder category of models. However, they all predict future observations to learn representations and therefore, they as well belong to the family of forward models.

The method these works use to combine forward models and auto-encoders consists in mapping o t to s t , and then compute the transition, with the help of a t , to obtain ŝt+1 . ŝt+1 is then remapped onto the pixel space in form of a vector ôt+1 . The error is then computed pixel-wise between ôt+1 and o t+1 . One common assumption is that the forward model in the learned state space is linear [START_REF] Goroshin | Learning to linearize under uncertainty[END_REF], van Hoof et al., 2016]. The transition is then just a linear combination of s t and a t as in Eq. 6. W, U and V are either fixed or learned parameters [van Hoof et al., 2016].

ŝt+1 = W * ŝt + U * a t + V (6)
In a similar way, the Embed to Control model (E2C) based on variational auto-encoders uses Eq. 6 to compute the mean µ of a distribution and learn supplementary parameters for the variance σ of the distribution [START_REF] Watter | Embed to control: A locally linear latent dynamics model for control from raw images[END_REF]. Then, ŝt+1 is computed with Eq. 7:

ŝt+1 ∼ N (µ = W * ŝt + U * a t + V, σ) (7)
Using distributions to compute ŝt+1 allows to use the KL-divergence to train the forward model. This method is also used in [START_REF] Karl | Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data[END_REF] and [START_REF] Krishnan | Deep Kalman Filters[END_REF]. However, the transition model in [START_REF] Krishnan | Deep Kalman Filters[END_REF] considers the KL-divergence between P (s t+1 ) and P (ŝ t+1 ) and does not use the loss of the reconstruction based on o t+1 and ôt+1 .

The use of a t is a common feature in most forward models based approaches. In fact, as several future states s t+1 from a given state are possible, s t alone does not contain enough information to predict s t+1 . The only approach that gets rid of the need for actions assumes that the transition from s t-1 to s t allows to deduce the transition from s t to s t+1 and uses several past states to predict s t+1 [START_REF] Goroshin | Learning to linearize under uncertainty[END_REF]. Actions are therefore implicit in this approach.

Another use of a forward model, connected to an intrinsic curiosity model (ICM) which helps agents explore and discover the environment out of curiosity when extrinsic rewards are sparse or not present at all, is proposed in [START_REF] Pathak | Curiositydriven exploration by self-supervised prediction[END_REF]. In this model, an intrinsic reward signal is computed from the forward model's loss function L f wd ( f is the forward function learned by the model, φ is the encoding model):

L f wd ( φ(o t+1 ), f ( φ(o t ), a t )) = 1 2 f ( φ(o t ), a t ) -φ(o t+1 ) 2 2 (8)
It is argued that there is no incentive in this model for s t to learn to encode any environmental features that cannot influence or are not influenced by the agent's actions. The learned exploration strategy of the agent is therefore robust to uncontrollable aspects of the environment such as the presence of distractor objects, changes in illumination, or other sources of noise in the environment [START_REF] Pathak | Curiositydriven exploration by self-supervised prediction[END_REF].

Forward models are therefore able to learn representations of controllable factors: in order to predict next state, the model must understand the object being controlled. This kind of representation can also be learned through a controllability prior [START_REF] Jonschkowski | PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations[END_REF].

If a robot acts by applying forces, controllable things should be those whose accelerations correlate with the actions of the robot. Accordingly, a loss function can be defined to minimize the covariance between an action dimension i and the accelerations in the state dimension i. The following formula from [START_REF] Jonschkowski | PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations[END_REF]] makes it explicit:

Controllability i = e -cov(at,i,st+1,i) , (9)
where cov(a t,i , s t+1,i ) is the covariance between the state s t+1,i at dimension i and time t and a t,i (the action at dimension i that led to such state). Note that here the learned state is assumed to represent an acceleration. Related with this prior also is the notion of empowerment [START_REF] Klyubin | Empowerment: A universal agent-centric measure of control[END_REF], defined as an information-theoretic capacity of an agent's actuation channel to influence its own evolution. The concept of empowerment is related to accountability or agency, i.e., recognizing when an agent is responsible for originating the change of state in the environment.

Learning an inverse model

The forward model approach can be turned around and, instead of learning to predict next state (given previous state and action), use current and next states to predict the action between them. The inverse model framework is used in SRL by firstly performing a projection of o t and o t+1 onto learned states s t and s t+1 , and secondly, by predicting the action ât that would explain the transition of s t into s t+1 (Fig. 4). As before, learning this model can impose constraints on the state representation to be able to efficiently predict actions.

An example using inverse models to learn state representations is the Intrinsic Curiosity Module (ICM) [START_REF] Pathak | Curiositydriven exploration by self-supervised prediction[END_REF]. It integrates both an inverse and forward model and the authors argument that using an inverse model is a way to bypass the hard problem of predicting original observations (e.g., pixels in images), since actions have much lower dimension.

A different kind of inverse model is used in [START_REF] Shelhamer | Loss is its own reward: Self-supervision for reinforcement learning[END_REF], where the policy gradient algorithm used to learn a controller is augmented with auxiliary gradients from what is called self-supervised tasks. In this case, in lack of external supervision, the prediction error resulting from interactions with the environment acts as a self-supervision. They learned a inverse dynamics model to retrieve from o t and o t+1 the action a t performed between the two successive time steps.

Note that connections among forward and inverse models are important: inverse models can provide supervision to learn representations that the forward model regularizes by learning to predict s t+1 [START_REF] Agrawal | Learning to poke by poking: Experiential learning of intuitive physics[END_REF]. In practice, this is implemented by decomposing the joint loss function as a sum of the inverse model loss plus the forward model loss [START_REF] Agrawal | Learning to poke by poking: Experiential learning of intuitive physics[END_REF]. Conversely, [START_REF] Zhang | Decoupling dynamics and reward for transfer learning[END_REF] shows in an ablation study that using an inverse model (along with a forward model and an auto-encoder) is the factor that contributes the most to learning a good state representation. Another approach including forward and inverse models, as well as a reconstruction of the observation including multimodal inputs is [Duan, 2017].

Using feature adversarial learning

Adversarial networks [START_REF] Goodfellow | Generative adversarial nets[END_REF] can also be used for unsupervised learning of state representations. The use of the Generative Adversarial Network (GAN) framework to learn state representations is proposed in [START_REF] Chen | Infogan: Interpretable representation learning by information maximizing generative adversarial nets[END_REF]. They present a model named InfoGAN that achieves the disentanglement of latent variables on 3D poses of objects. As described in [START_REF] Chen | Infogan: Interpretable representation learning by information maximizing generative adversarial nets[END_REF], the goal is to learn a generator distribution P G (o) that matches the real distribution P data (o). Instead of trying to explicitly assign a probability to every o in the data distribution, GANs learn a generator network G that samples from the generator distribution P G by transforming a noise variable z ∼ P noise (z) into a sample G(z). The noise variable has two components. A first one, z G , randomly sampled from a Gaussian distribution, and a second one with smaller dimension, z U , sampled from a uniform distribution. The latter is used during training so that the G(z) has a high mutual information with z U . Then, the sample from z U has a high correlation with G(z) and can thus be considered as a state representation. This generator is trained by playing against an adversarial discriminator network D that aims at distinguishing between samples from the true distribution P data and the generator distribution P G . The authors succeed to learn states corresponding to object orientations from sequences of images.

Another example of SRL with Generative Adversarial Networks is presented by [START_REF] Donahue | Adversarial feature learning[END_REF], Dumoulin et al., 2016]. BiGAN and ALI are extensions of regular GANs to learn the double mapping from image space to latent space, and from latent space to image space. They allow the learned feature representation to be useful for auxiliary supervised discrimination tasks, and competitive with unsupervised and self-supervised feature learning. The BiGAN has also been experimented in [START_REF] Shelhamer | Loss is its own reward: Self-supervision for reinforcement learning[END_REF] to learn state representations used for reinforcement learning, but lead to lower performance than their own approach (Section 3.2).

Exploiting rewards

As opposed to RL, the use of a reward value in SRL is not compulsory. However, it can be used as supplementary information to help differentiating states and to learn task related representations. Rewards are helpful to disentangle meaningful information from a noisy or distracting one, and to tie the representation to a particular task. However, in a multi-task setting, this approach can also be used to learn a generic state representation that is relevant to different tasks.

A predictable reward prior which estimates rt+1 given a state s t and an action a t is implemented in [START_REF] Munk | Learning state representation for deep actor-critic control[END_REF] (along with a forward model) to learn a state for reinforcement learning. Another approach that exploits reward is presented in [START_REF] Oh | Value Prediction Network[END_REF]. Besides predicting the reward similarly to [START_REF] Munk | Learning state representation for deep actor-critic control[END_REF], they also learn to predict the value (discounted sum of future reward) of the next state and exploit this capacity to rely on planning multiple steps for policy learning. The author state that predicting rewards for multiple steps is much easier that predicting observations, while giving the important information for learning a policy.

A dimensionality reduction model called reward weighted principal component analysis (rw-PCA), as another way of using rewards for state representation was proposed in [START_REF] Parisi | Goal-driven dimensionality reduction for reinforcement learning[END_REF]. rwPCA uses data collected by an RL algorithm and operates a dimensionality reduction strategy which takes reward into account to keep the information into a compressed form. The compressed data is afterwards used to learn a policy.

On the same idea of constructing a task-related representation, [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF]] and [START_REF] Lesort | Unsupervised state representation learning with robotic priors: a robustness benchmark[END_REF] use rewards as supplementary information to impose constraints on the state space topology. One of these constraints makes the space more suited to discriminate between states with different rewards. The state space is then particularly adapted to solve a given task. This constraint is called causality prior in [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF] and [START_REF] Lesort | Unsupervised state representation learning with robotic priors: a robustness benchmark[END_REF]. It assumes that if we have two different rewards after performing the same action in two different time steps, then the two corresponding states should be differentiated and far away in the representation space (Equation 10).

L Caus (D, φ) = E[e -ŝt 2 -ŝt 1 2 | a t1 = a t2 , r t1+1 = r t2+1 ] (10)

Other objective functions

In this section, we present other approaches assuming various specific constraints for state representation learning. Following [START_REF] Lake | Building Machines That Learn and Think Like People[END_REF], the learning process can be constrained by prior knowledge (either initially provided by the designer or acquired via learning) to allow the agent to leverage existing common sense, intuitive physics, physical laws, mental states of others, as well as other abstract regularities such as compositionality and causality. This kind of a priori knowledge is called prior [START_REF] Bengio | Unsupervised feature learning and deep learning: A review and new perspectives[END_REF], [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF], [START_REF] Lesort | Unsupervised state representation learning with robotic priors: a robustness benchmark[END_REF], [START_REF] Jonschkowski | PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations[END_REF], and is defined through cost functions. These loss functions are applied in the state space in order to impose the required constraints to construct the model projecting the observations in the state space. In the following, ∆s t = s t+1s t is the difference in between states at times t and t + 1, and D is a set of observations.

• Slowness Principle

The slowness principle assumes that interesting features fluctuate slowly and continuously through time and that a radical change inside the environment has low probability [START_REF] Wiskott | [END_REF]Sejnowski, 2002, Kompella et al., 2011].

L Slowness (D, φ) = E[ ∆ s t 2 ] (11)
This assumption can have other naming depending on the unit of s t , e.g., prior of time coherence (time) [Jonschkowski andBrock, 2015, Lesort et al., 2017] or inertia (velocity) [START_REF] Jonschkowski | PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations[END_REF].

• Variability

The assumption of this prior is that positions of relevant objects vary, and learning state representations should then focus on moving objects [START_REF] Jonschkowski | PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations[END_REF].

L V ariability (D, φ) = E[e -st1-st2 ] (12)
e -distance is used as a similarity measure that is 1 if the distance among states is 0 and goes to 0 with increasing distance between states. Note that this prior is counter-balancing the slowness prior introduced above as the slowness alone would lead to constant values.

• Proportionality

The proportionality prior introduced in [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF] assumes that for the same action in different states, the reactions to this action will have proportional amplitude or effect. The representation then vary in the same amount for two equal actions in different situations.

L P rop (D, φ) = E[( ∆s t2 -∆s t1 ) 2 |a t1 = a t2 ] (13)
• Repeatability This prior states that two identical actions applied at similar states should provide similar state variations, not only in magnitude but also in direction [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF].

L Rep (D, φ) = E[e -st 2 -st 1 2 ∆s t2 -∆s t1 2 | a t1 = a t2 ] (14)
• Dynamic verification Dynamic verification [START_REF] Shelhamer | Loss is its own reward: Self-supervision for reinforcement learning[END_REF] consists in identifying the corrupted observation o tc in a history of K observations o t where t ∈ 0, K . Observations are first encoded into states and the sequence is classified by a learned function f to output the corrupted time step. Negative samples are produced by incorporating observations from a wrong time step into the sequence of images. This discriminative approach forces SRL to encode the dynamics in the states.

• Selectivity

States can be learned by using the idea that factors such as objects correspond to 'independently controllable' aspects of the world that can be discovered by interacting with the environment [START_REF] Thomas | Independently controllable factors[END_REF]. Knowing the dimension K of the state space, the aim is to train K policies π k with k ∈ 1, K . The goal is that the policy π k causes a change in s when actions are taken according to π k , the selectivity of a feature k is:

L sel (D, φ, k) = E s (k) t+1 -s (k) t k ′ s (k ′ ) t+1 -s (k ′ ) t |s t+1 ∼ P a st,st+1 (15)
where P a st,st+1 is the environment transition distribution from s t to s t+1 under action a. The selectivity of s (k) t is maximal when only that single feature changes as a result of some action. Maximizing the selectivity improves the disentanglement of controllable factors in order to learn a good state representation.

Using hybrid objectives

Reconstruction of data in the observation space, forward models, inverse models, exploitation of rewards, and other objective functions presented in the previous sections are different approaches to tackle the state representation learning challenge. However, these approaches are not incompatible, and models often take advantage of several objective functions at the same time.

For instance, interactively learning to poke by poking [START_REF] Agrawal | Learning to poke by poking: Experiential learning of intuitive physics[END_REF] is an example of empirical learning of intuitive physics using o t and o g as current and goal images, respectively, in order to predict the poke action. The latter is composed by the location, angle and length of the action that sets the object in the state of goal image o g . Simulations shows that using the inverse model or jointly the inverse and forward models improve performance at pushing objects and that when the training data available is reduced, the joint model outperforms the inverse model with a performance comparable to using a considerably larger amount of data.

The authors in [START_REF] Finn | Deep spatial autoencoders for visuomotor learning[END_REF], Goroshin et al., 2015] use the reconstruction of the observation and the slowness principle in their SRL approach. [START_REF] Goroshin | Learning to linearize under uncertainty[END_REF], van Hoof et al., 2016, Watter et al., 2015, Assael et al., 2015, Karl et al., 2016, Ha and Schmidhuber, 2018] combine the reconstruction of observation and forward models. [Jonschkowski andBrock, 2015, Lesort et al., 2017] take advantage of rewards with a causality prior (Eq. 10) and several other objective functions such as the slowness principle, proportionality, and repeatability to learn state representations. We illustrate, as an example, the combination of objective functions from [START_REF] Watter | Embed to control: A locally linear latent dynamics model for control from raw images[END_REF] in Figure 6. Figure 6: Example of hybrid model (E2C [START_REF] Watter | Embed to control: A locally linear latent dynamics model for control from raw images[END_REF]) combining a set of previously described loss term blocks of previous figures. Several errors are used to learn the state representation using at the same time a forward model error, a reconstruction error and a constraint on the representation that enforces the transition to be linear.

Table 1 summarizes all the reviewed models by showing, for each one, which proxies or surrogate functions have been used for learning: reconstruction of observation, prediction of the future (forward model) and/or retrieving actions (inverse model), and what kind of information is used: action and/or rewards.

Building blocks of State Representation Learning

In this section, we cover various implementation aspects relevant to state representation learning and its evaluation. We refer to specific surrogate models, loss function specification tools or strategies that help constraining the information bottleneck and generalizing when learning lowdimensional state representations,

Learning tools

We first detail a set of models that through an auxiliary objective function, help learning a state representation. One or several of these learning tools can be integrated in broader SRL approaches as was previously described.

Auto-encoders (AE)

Auto-encoders (AE) are a common tool used to learn state representations that are widely used for dimensionality reduction [START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF], Wang et al., 2012, Wang et al., 2016]. Their objective is to output a reproduction of the input. Its architecture is composed by an encoder and a decoder. The encoder projects the input to a latent space representation (often in lower dimension than the input), which is re-projected to the output afterwards by the decoder. In our problem setting, o t is the input, s t the latent representation, and ôt is the output. The dimensionality of the latent representation can be chosen depending on the dimension of the state representation we want to learn and enforcing it in such case. The AE will then automatically learn a compact representation by minimizing the reconstruction error between input and output. The usual loss function L to measure the reconstruction error is the mean squared error (MSE) between input and output, computed pixel-wise. However, it can be any norm.

Loss = L(x, x) (16)
Auto-encoders are used in different SRL settings [START_REF] Finn | Deep spatial autoencoders for visuomotor learning[END_REF], Mattner et al., 2012]; PCA can also be considered as a particular case of auto-encoder [START_REF] Curran | Dimensionality reduced reinforcement learning for assistive robots[END_REF].

Denoising auto-encoders (DAE)

The main issue of auto-encoders is the risk of finding an easy but not satisfying solution to minimize the pixel reconstruction error. This occurs when the decoder reconstructs a kind of average looking dataset. To make the training more robust to this kind of mean optimization solution, denoising auto-encoders (DAE) [START_REF] Vincent | Extracting and composing robust features with denoising autoencoders[END_REF], Vincent et al., 2010] can be used. This architecture adds noise to the input and makes the "average" image a more corrupted solution than the original AE. The DAE architecture is used in [van Hoof et al., 2016] to learn visual and tactile state representations. The authors compared state representations learned by a DAE and a variational auto-encoder (VAE) by using the learned states in a reinforcement learning setting. They found that, in most cases, DAE state representation models gather less rewards than those with VAE state representations.

Variational auto-encoders (VAE)

The SRL literature has also benefited from the variational inference used in variational autoencoders (VAE) [START_REF] Kingma | [END_REF]Welling, 2013, Jimenez Rezende et al., 2014] to learn a mapping from observations to state representations. A VAE is an auto-encoder with probabilistic hidden cells: it interprets S as a set sampled from distribution P (s t |o t ). It then approximates P (s t |o t ) with a model q θ (s t |o t ) called the approximate posterior or recognition model. θ represents the parameters of the model, which, for instance, can be a neural network. The VAE also provides a generator which approximates P (o t |s t ) with a model p φ . φ represents the parameters of the generator. Both models p and q are then trained by minimizing the error between o t and ôt and the KL divergence between q θ (s t |o t ) and the normal distribution N (µ = 0, σ = I) (where µ is the mean of the distribution, σ its covariance matrix and I the identity matrix). VAE-related models that do not use exactly the original VAE, but variations of it, are [START_REF] Watter | Embed to control: A locally linear latent dynamics model for control from raw images[END_REF], Assael et al., 2015, Krishnan et al., 2015, van Hoof et al., 2016, Karl et al., 2016, Higgins et al., 2016].

Siamese networks

Siamese networks [START_REF] Chopra | Learning a similarity metric discriminatively, with application to face verification[END_REF] consist of two or more identical networks that share their parameters, i.e., have the exact same weights. The objective of the siamese architecture is not to classify input data, but to differentiate between the inputs (same versus different class or condition, for example). This kind or architecture is useful to impose constraints in the latent space of a neural network. For example it can be used to learn similarity metrics or time dependencies, as it is done in time-contrastive networks [START_REF] Sermanet | Time-contrastive networks: Self-supervised learning from multi-view observation[END_REF].

In the context of SRL, siamese networks can be employed to implement some priors previously presented in Section 3.6. For example, two siamese networks can be used to compute a similarity loss and optimize the slowness principle (or temporality prior) between s t and s t+1 as in [START_REF] Lesort | Unsupervised state representation learning with robotic priors: a robustness benchmark[END_REF]. In [START_REF] Goroshin | Learning to linearize under uncertainty[END_REF] they use three siamese networks to compute three consecutive states at the same time that are fed into another model that predicts the next state.

Observation/action spaces

This section presents a summary of the dimensionality of the observation, state and action spaces, as well as the applications in which the reviewed papers are evaluated (Table 2). The continuity or discreetness of the action space is also shown. These are good proxies to assess the complexity of the problem tackled: the higher the dimensionality of observation and action, as well as the smaller we want the dimension of the state to be, the harder is the task of learning a state representation because much more information will need to be processed and filtered in order to keep only the information that is substantial.

We note also that the reviewed literature often presents results with presumably higher dimensionality of learned states than theoretically needed (e.g. using state of dimension 6 for a 2 joints robotic arm). The dimensionality of the state may seem obvious when we are learning a state that should (according to the task) correlate with a clear dimension (position, distance, angle) in the environment. However, deciding the dimensionality of the state space is not always trivial when we are learning more abstract states with no clear dimension associated to it. For instance, visually representing the state associated to an Atari game scene in a complex situation is not as easy to interpret nor assess in comparison to the dimensionality of states associated to a position of an arm, its angle or velocity. Indeed, since the learning objective we reviewed are just proxies to Encoder Encoder

s t o t s t+1 o t+1

Loss function

Figure 7: Representation of siamese encoders: networks with tied (shared) parameters. In SRL, siamese networks allow to set constraints via loss functions among several states, e.g. in robotic priors.

guide state representation learning, they can lead to something different that the ideal and minimal state representation. In particular, increasing the capacity of the model by augmenting the dimensionality of the state above the dimension of the true state can lead to a better optimization of the learning objectives.

Evaluating learned state representations

This section provides a review of validation metrics and embedding quality evaluation techniques used across the literature. These are summarized in Table 3.

The most common way of evaluating the quality of the learned state space is by letting an agent use the states to learn a control task, and thus assessing whether the representation is general enough to be transferable. This method is for example applied to evaluate the performance of an SRL algorithm using reinforcement learning [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF], Jonschkowski et al., 2017, Munk et al., 2016, van Hoof et al., 2016, Finn et al., 2015, Pathak et al., 2017, Shelhamer et al., 2017, Oh et al., 2017, Parisi et al., 2017, Assael et al., 2015].

Table 3: Evaluation methods for state representation learning and their respective objective.

Metric

Evaluation objective Task performance [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF], Jonschkowski et al., 2017, Munk et al., 2016, van Hoof et al., 2016, Finn et al., 2015, Pathak et al., 2017, Shelhamer et al., 2017, Oh et al., 2017, Parisi et al., 2017, Assael et al., 2015, Higgins et al., 2016, Zhang et al., 2018, Alvernaz and Togelius, 2017] Assesses that the information needed to solve a task is contained in st. The evaluation is done in reinforcement learning algorithms.

Disentanglement metric score [START_REF] Higgins | beta-vae: Learning basic visual concepts with a constrained variational framework[END_REF] Measures the disentanglement of the latent factors. It assumes that generative factors are known and interpretable. Used in transfer learning, object recognition. Distortion [Indyk, 2001] Measures the preservation of local and global geometry coherence in unsupervised representation learning. NIEQA (Normalization Independent Embedding Quality Assessment) [START_REF] Zhang | A new embedding quality assessment method for manifold learning[END_REF] Measures the local & global neighborhood embedding quality assessment; not limited to isometric embeddings. Used in manifold learning. KNN-MSE [START_REF] Lesort | Unsupervised state representation learning with robotic priors: a robustness benchmark[END_REF] Measures the degree of preservation of the same neighbors in between the latent space and the ground truth. Supervised regression [START_REF] Jonschkowski | PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations[END_REF] Evaluates the performance of a supervised regression between the learned states and the ground truth.

However, this approach is often very costly and inefficient in terms of time, computation and data. Also, various state-of-the-art RL algorithms may be applied to learn a policy and may result in very different performances for a given state representation. The uncertainty inherent to RL therefore makes RL algorithms sufficient but not practical nor appropriate to be a necessary condition to validate a particular state representation. In consequence, it would be desirable to have an intermediate manner to assess the representation that is independent of the algorithm applied to complete the task and there are, indeed, several more direct ways to assess the learned state space. For example, visual assessment of the representation's quality can be done using a Nearest-Neighbors approach as in [START_REF] Sermanet | Time-contrastive networks: Self-supervised learning from multi-view observation[END_REF], Pinto et al., 2016]. The idea is to look at the nearest neighbors in the learned state space, and for each neighbor, retrieve their corresponding observation. Visual inspection can then reveal if these two observations indeed correspond to nearest neighbors in the ground truth state space s we intend to learn.

While the nearest neighbor coherence can be assessed visually, KNN-MSE is a quantitative metric derived from this qualitative information [START_REF] Lesort | Unsupervised state representation learning with robotic priors: a robustness benchmark[END_REF]. Using the ground truth state value for every observation, KNN-MSE measures the distance between the value of an observation and the value of the nearest neighbor observations retrieved in the learned state space. A low distance means that a neighbor in the ground truth is still a neighbor in the learned representation, and thus, local coherence is conserved.

For an observation o, KNN-MSE is computed using its associated learned state s = φ(o) as follows:

KNN-MSE(s) = 1 k

s ′ ∈KN N (s,k) ||s -s′ || 2 (17)
where KNN(s, k) returns the k nearest neighbors of s (chosen with the Euclidean distance) in the learned state space S, s is the ground truth associated to s, and s′ is the ground truth associated to s ′ . One of the characteristics that a good representation should possess is to produce a disentangled representation of variation factors. The evaluation of these characteristics can be done using the selectivity prior (see Section 3.6 and Eq. 15) from [START_REF] Thomas | Independently controllable factors[END_REF]. This prior cares about the independence among variations of the representation under each action. However, it is applicable mainly if actions are known to be independent.

Another way to quantitatively compare the degree of disentanglement reached by a model is using the disentanglement metric score [START_REF] Higgins | beta-vae: Learning basic visual concepts with a constrained variational framework[END_REF]. It assumes that the data is generated by a process in which the generative factors are known, interpretable, and that some are conditionally independent. In order to measure the disentanglement, it uses a simple low-capacity and low VC-dimension linear classifier's accuracy (reported as disentanglement metric score). The classifier's goal is to predict the generative factor that was kept fixed for a given difference between pairs of representations from the same latent factor.

Other metrics from the area of manifold learning can be used, such as distortion [Indyk, 2001] and NIEQA [START_REF] Zhang | A new embedding quality assessment method for manifold learning[END_REF]; both share the same principle as two quantitative measures of the global quality of a representation: the representation space should, as much as possible, be an undistorted version of the original space.

Distortion [Indyk, 2001] gives insight of the quality of a representation by measuring how the local and global geometry coherence of the representation changes with respect to the ground truth. It was designed in the embeddings context as a natural and versatile paradigm for solving problems over metric spaces.

NIEQA (Normalization Independent Embedding Quality Assessment) [START_REF] Zhang | A new embedding quality assessment method for manifold learning[END_REF]] is a more complex evaluation than distortion that measures the local geometry quality and the global topology quality of a representation. NIEQA local part checks if the representation is locally equivalent to an Euclidean subspace that preserves the structure of local neighborhoods. NIEQA objectives are aligned with KNN-MSE [START_REF] Lesort | Unsupervised state representation learning with robotic priors: a robustness benchmark[END_REF], as a measure to assess the quality of the representation, especially locally. The global NIEQA measure is also based on the idea of preserving original structure in the representation space, but instead of looking at the neighbors, it samples "representative" points in the whole state space. Then, it considers the preservation of the geodesic distance between those points in the state space.

One last mechanism to assess SRL methods is to use supervised learning to learn a regression from the learned representation to its ground-truth [START_REF] Jonschkowski | PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations[END_REF]. The training and test sets are separated into two datasets to evaluate if the regression can generalize to unseen states. The assumption is that this regression measures how well meaningful features are encoded in the state. A good generalization would show a good encoding.

Evaluation scenarios

Datasets used to validate state representation learning include varied, but mainly simulated, environments because they are easier to reproduce and generate. Unlike in image recognition challenges where MNIST digits or ImageNet datasets prevail, in state representation learning, a varied set of regular video games or visuomotor tasks in robotics can be found as a test suite for robotics control. Examples of simulated environments include, among others:

• Pendulum (Inverted or classical): The goal is to represent the state of the pendulum [START_REF] Watter | Embed to control: A locally linear latent dynamics model for control from raw images[END_REF], Jonschkowski et al., 2017, van Hoof et al., 2016, Mattner et al., 2012]. The pendulum starts in a random position, and the objective is to swing it up so it stays upright (there is no specified reward threshold at which the task is considered solved).

• Cart-Pole: consists of an inverted pendulum attached to a cart which moves along a frictionless track; the system is controlled applying +1 or -1 force to the cart, and a reward of +1 is provided for every time step that the pole remains upright. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center1 ( [START_REF] Watter | Embed to control: A locally linear latent dynamics model for control from raw images[END_REF], [START_REF] Jonschkowski | PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations[END_REF]).

• Atari games [START_REF] Bellemare | The arcade learning environment: An evaluation platform for general agents[END_REF]: mostly low (2D) dimensional simulated environments with different agents and goals. In these games, states can be represented through different variables (time in achieving a task, amount of bonus, keeping alive, etc.) [START_REF] Shelhamer | Loss is its own reward: Self-supervision for reinforcement learning[END_REF], Oh et al., 2017].

• More advanced test games include VizDoom, where the levels passed, reward accumulated and exploration levels are used as evaluation metrics [Pathak et al., 2017, Alvernaz and[START_REF] Alvernaz | [END_REF]. Likewise, Mario Benchmark [START_REF] Karakovskiy | [END_REF] is a platform designed for reinforcement learning based on the "Super Mario Bros" video game. This test suite is for example experimented in [START_REF] Curran | Dimensionality reduced reinforcement learning for assistive robots[END_REF], Pathak et al., 2017].

• Other evaluation benchmarks tested in the reviewed works in this survey include simulated octopus arms [START_REF] Engel | Learning to control an octopus arm with gaussian process temporal difference methods[END_REF], Munk et al., 2016], labyrinths [START_REF] Thomas | Independently controllable factors[END_REF], navigation grids [Magrans de [START_REF] De | Curiosity-driven reinforcement learning with homeostatic regulation[END_REF]Kanai, 2018, Oh et al., 2017], driving cars [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF], or mountain car scenarios [START_REF] Curran | Dimensionality reduced reinforcement learning for assistive robots[END_REF]. Another example is the bouncing ball, where the goal is to learn a representation of one bouncing ball position in 2D (x,y) [START_REF] Karl | Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data[END_REF].

• In the robotics domain we can find benchmarks on robot manipulation skills [START_REF] Finn | Deep spatial autoencoders for visuomotor learning[END_REF], van Hoof et al., 2016] such as Baxter pushing a button [START_REF] Lesort | Unsupervised state representation learning with robotic priors: a robustness benchmark[END_REF], grasping [START_REF] Finn | Deep spatial autoencoders for visuomotor learning[END_REF], stabilizing [van Hoof et al., 2016], poking objects [START_REF] Agrawal | Learning to poke by poking: Experiential learning of intuitive physics[END_REF], Duan, 2017] or balancing a real pendulum [START_REF] Mattner | Learn to swing up and balance a real pole based on raw visual input data[END_REF]. Nevertheless, some approaches achieve to learn in real environment scenarios, for instance, with mobile robots that explore an arena [START_REF] Jonschkowski | Learning state representations with robotic priors[END_REF].

Many of the latter simulated scenarios are part of Universe and OpenAI Gym [START_REF] Brockman | Openai gym[END_REF] or DeepMind Labs [START_REF] Beattie | Deepmind lab[END_REF]. These benchmarking tasks used in the most prominent state representation learning literature are summarized in Table 2.

Discussion and future trends

In this section, we first discuss the implications of SRL for autonomous agents and the assessment, comparison and reproducibility of the representation learned. Finally, we explore the consequences of SRL on the interpretability of machine learning algorithms.

SRL models for autonomous agents

SRL methods provide unsupervised tools for autonomous agents to learn representations about the environment without extra annotations. They need, however, that the agent gathers data to learn. Therefore, the role of environment exploration is an important dimension to investigate in SRL. If the space is not sufficiently explored by the agent, acquisition of varied observations and exposure to actions that lead to optimal performance can be hindered [START_REF] Pathak | Curiositydriven exploration by self-supervised prediction[END_REF].

One way to incorporate exploration in SRL is to integrate curiosity or intrinsic motivations [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF] in the algorithm that collects data. The overall idea of this approach is to complement the extrinsic reward by an intrinsic reward that favors states where SRL makes the most progress. This is done for example in the Intrinsic Curiosity Module (ICM) [START_REF] Pathak | Curiositydriven exploration by self-supervised prediction[END_REF] by defining an intrinsic reward linked to the forward model error which encourages exploration. This approach is improved in [Magrans de [START_REF] De | Curiosity-driven reinforcement learning with homeostatic regulation[END_REF] by balancing this exploratory behavior with an homeostatic drive to also favor actions that lead to familiar stateaction pairs. The reverse question of how the learned state space can influence the performance of intrinsic motivation approaches [START_REF] Péré | Unsupervised learning of goal spaces for intrinsically motivated goal exploration[END_REF] is also relevant. The automatic exploration, designed to maximize the quality of a learned state representation, is a field to be further explored in order to build high quality representations.

Another approach to gather enough relevant data could be to perform data augmentation by adding data from simulation; however, the problem is to make the model benefit from simulation data for real life applications, a problem that is known as the reality gap [START_REF] Mouret | Crossing the reality gap: a short introduction to the transferability approach[END_REF]. Nevertheless, using both kinds of data was shown to improve results in particular applications [START_REF] Bousmalis | Using simulation and domain adaptation to improve efficiency of deep robotic grasping[END_REF]]. An interesting research direction is therefore to study how to exploit simulation data to improve SRL for real world applications.

Another problem to ultimately perform SRL autonomously (i.e., without manual parameter tuning) is the choice of the state representation dimension, which is made empirically in most reviewed approaches. The challenge of deciding the dimensionality automatically can be related to a bias-variance trade-off, as the dimensionality of the representation constrains the capacity of the model. Indeed, increasing the states dimension augments the capacity of the model, which, as a result, will be better at reducing the training error, but also leads to overfitting. As discussed in Section 4.2, learning criteria can be better optimized by models with large capacity, and thus, an automatic process is prone to over estimate the dimension needed.

To avoid choosing manually the state dimension, it is possible to choose automatically a number of features from a larger set such that they have a certain variance and are orthogonal [START_REF] Parisi | Goal-driven dimensionality reduction for reinforcement learning[END_REF]. It can be done by using PCA to produce a set of features in which the most significant ones are selected with respect to the chosen variance. PCA can also be modified to select reward related components [START_REF] Parisi | Goal-driven dimensionality reduction for reinforcement learning[END_REF]. Although the variance has to be fixed a priori, the authors claim that this is usually easier than choosing the state dimension. Extending this technique to other state representation approach could be an interesting research direction.

Assessment, comparison and reproducibility in SRL

The assessment challenge of SRL is two-sided. First, there is no easy nor certified way for validating a learned representation. Secondly, the lack of common evaluation frameworks makes a fair comparison between approaches difficult.

As mentioned in Section 4.3, the most objective method to evaluate the quality of representations is to check if the state representation learned can be used by an RL algorithm to solve a task more efficiently. However, this assessment is uncertain and unstable given the stochasticity of reinforcement learning algorithms [START_REF] Henderson | Deep reinforcement learning that matters[END_REF]. Moreover, it is not obvious which RL algorithm is the best choice, and thus, several should be used in the comparison. In practice, a large amount of policy evaluation runs is therefore required in order to provide a robust assessment, which is possible in simulation but is seldom applicable on real robots, given the robots fragility and experimentation time involved. In this case, it is therefore interesting to use several of the other measures presented in section 4.3, that only give partial information on the state representation quality, but are possible to apply for a comparison with a ground truth state.

Comparing approaches from published results is also particularly hard because of the high variability of the environments and data used in the different approaches (as illustrated in Table 2). This points to the need of an evaluation framework incorporating several tasks and several evaluation metrics similar to the ones proposed for reinforcement learning such as the DeepMind Control Suite [START_REF] Tassa | DeepMind Control Suite[END_REF]. Reproducibility guidelines with proper experimental techniques and reporting procedures, as pointed in [START_REF] Henderson | Deep reinforcement learning that matters[END_REF] for RL, should also be defined for SRL. In the mean time, as there is not yet an ideal method for state representation assessment, researchers should at least make public their simulation environment (with possible ground truth), and use simulation settings from other approaches to provide fairer comparisons and facilitate the method reproducibility. Furthermore, we strongly encourage authors to entirely describe their experiments, in particular, report their data and models' characteristics.

Providing interpretable systems

In 2018, European Union regulations on algorithmic decision-making include a "right to explanation", "right to opt-out" and "non discrimination" of models [START_REF] Goodman | European union regulations on algorithmic decision-making and a" right to explanation[END_REF]. Artificial intelligence research is thus granted with an opportunity to further provide meaningful explanations to why algorithms work the way they do. The interpretability of results in machine learning is however a challenging problem that needs proper definition [Lipton, 2016]. In any case, monitoring the degree to which AI systems show the same thinking flows as humans is invaluable and crucial; not only to explain how human cognition works, but also to help AI make better and more fair decisions [START_REF] Lake | Building Machines That Learn and Think Like People[END_REF].

We define interpretability in the SRL context as the capacity for a human to be able to link a variation in the representation to a variation in the environment, and be able to know why the representation was sensitive to this variation. As SRL is designed to be able to give this level of interpretability, it could help improving the understanding we have about learning algorithms' output. Indeed, the higher the dimension, the less interpretable the result is for humans. Therefore, the dimensionality reduction induced by SRL, coupled with the link to the control and possible disentanglement of variation factors, could be highly beneficial to improve our understanding capacity of the decisions made by algorithms using this state representation.

Conclusion

State Representation Learning algorithms are designed to find a way to compress high-dimensional observation data into a low and meaningful dimensional space for controlled systems. These models only require the agent's observations, its performed actions and, optionally, the reward of the associated task.

This work aims at presenting an accessible guide to learn about SRL approaches, the existing tools for evaluation and the common simulation settings used as benchmark. We presented the learning objectives of the state of the art approaches on SRL and their resemblances and differences. We discussed afterwards the use of SRL for autonomous agents, the difficulties for comparing existing approaches and the interpretability of results.

A general advice when building SRL models would be to integrate as many learning objectives as possible, depending on the available data. As an example, one could use a reconstruction objective for linking the state space to the observations, combined with a predictive objective (forward model) to capture dynamics, and a reward-based objective to apprehend the effects of actions performed. More general priors could also be added to force the state space to be coherent and understandable for humans. While many models integrate several of these objectives, no proposed model currently includes all of them together.

As SRL is designed to automatically learn representations from a set of unlabeled observations, it could be used in future work to learn from evolving environments and could be a step towards continual or lifelong learning. Another area to explore in the future is the integration of exploration strategies for data collection specifically designed to be able to improve the state representation learned.

Figure 1 :

 1 Figure 1: General model : circle are observable and square are the latent state variables.

  not in any other feature. To quantify the change in s (k) t

Table 1 :

 1 Classification of the reviewed SRL models with regards to the learning objectives they implement and the information they use (actions or rewards)

	Model	Actions/Next	state constraints	Forward model	Inverse model	Reconstruct	observation	Predicts next	observation

Table 2 :

 2 Settings of each approach: characteristics of each environment and state representation dimensions

	Reference	Observa-	State	Action	Environment	Data
			tion	Di-	Dimen-
			Dimen-	men-	sion
			sion	sion
	Priors [Jon-	16*16*3	2	25	dis-	Slot cars, mobile	Raw images
	schkowski			crete		robot localization
	and Brock,
	2015]
	PVE[Jon-	Unavailable 5	Discrete	Inverted pendulum,	Raw images
	schkowski					ball in cup, cart-
	et al., 2017]					pole
	E2C [Watter	40*40*3	8	Discrete	Agent with obstacle Raw images
	et al., 2015]
	E2C [Watter	48*48*3	8	Discrete	Inverted pendulum Raw images
	et al., 2015]
	E2C [Watter	80*80*3	8	Discrete	Cart-pole	Raw images
	et al., 2015]
	E2C [Watter	128*128*3 8	Discrete	3 link arm	Raw images
	et al., 2015]
	[van	Hoof	20*20*3	3	Continuous Pendulum swing-	Raw images
	et al., 2016]					up
	[van	Hoof	228	3	Continuous Real-robot manipu-	Tactile
	et al., 2016]					lation task	data
	ML-DDPG	18 or 24	6	2 discrete 2 link arm	Joint posi-
	[Munk et al.,						tion
	2016]
	ML-DDPG	192 or 308 96	36	dis-	Octopus	Joint posi-
	[Munk et al.,			crete			tion
	2016]

https://github.com/openai/gym/wiki/Leaderboard#pendulum-v0

Acknowledgements

This research is funded by the DREAM project under the European Union's Horizon 2020 research and innovation program under grant agreement No 640891. We acknowledge Olivier Sigaud, Antonin Raffin, Cynthia Liem and other colleagues for insightful and detailed feedback.