Yin Jun Phua
email: phua@il.c.titech.ac.jp

Tony Ribeiro
email: tony.ribeiro@ls2n.fr

Sophie Tourret
email: tourret@nii.ac.jp

Katsumi Inoue
email: inoue@il.c.titech.ac.jp

Learning Logic Program Representation for Delayed Systems With Limited Training Data

Keywords: dynamical systems, Boolean networks, attractors, learning from interpretation transition, delayed systems

Understanding the influences between components of dynamical systems such as biological networks, cellular automata or social networks provides insights to their dynamics. Influences of such dynamical systems can be represented by logic programs with delays. Logical methods that learn logic programs from observations have been developed, but their practical use is limited since they cannot handle noisy input and need a huge amount of data to give accurate results. In this paper, we present a method that learns to distinguish different dynamical systems with delays based on Recurrent Neural Network (RNN). This method relies on Long Short-Term Memory (LSTM) to extract and encode features from input sequences of time series data. We show that the produced high dimensional encoding can be used to distinguish different dynamical systems and reproduce their specific behaviors.

Introduction

Being able to learn the dynamics of an environment purely by observing has many applications. For example, in multi-agent systems where learning other agents' behavior without direct access to their internal state can be crucial for decision making [START_REF] Jennings | A roadmap of agent research and development[END_REF]. In system biology, learning the interaction between genes can greatly help in the creation of drugs to treat sicknesses [START_REF] Ribeiro | Learning multi-valued biological models with delayed influence from time-series observations[END_REF].

Problem Statement

Having an understanding of the dynamics of a system allows us to produce predictions of the system's behavior. Being able to produce predictions means that we can weigh between different options and evaluate their outcome from a given state without taking any action. In this way, learning about the dynamics of a system can aid in planning [START_REF] Martínez | Relational reinforcement learning for planning with exogenous effects[END_REF].

In most real world systems, we do not have direct access to the rules that govern the systems. What we do have, however, is the observation of the systems' state at a certain time step, or a series of observations if we look long enough. Therefore, the problem is to learn the dynamics of systems purely from the observations that we are able to obtain.

Several learning algorithms have been proposed, that learn rules for a system, provided that the observations given cover every case that can happen within the system. However, most real world systems, particularly in system biology, obtaining data for even a short amount of time is difficult, time consuming and expensive. Therefore most current learning algorithms, while complete, are not practical in the biology setting. In addition to that, most real world observations that can be obtained are often full of random noise. Therefore, dealing with noise is also an integral part in solving this problem. The focus of this paper is therefore on being able to learn the rules despite some of the rules not having manifested in the observation. We also consider the setting in which actions from past states are able to have a delayed influence on the current state. In addition, our proposed model can also deal with noise within the data, that no previous approaches dealt with, as shown in the experiments section.

Proposed Approach

In this paper, we propose an approach to this problem utilizing Recurrent Neural Networks (RNN) to learn a logic program representation from a series of boolean state transitions. Our method is based on a framework called Learning from Interpretation Transition (LFIT) [START_REF] Inoue | Learning from interpretation transition[END_REF]. LFIT is an unsupervised learning algorithm, that can learn logic programs describing fully the dynamics of the system, purely by observing state transitions. In our approach, we construct two neural networks, one for encoding the observed state transitions, and another one of which to produce the logic program representation for the system. The idea behind this is that given a series of state transitions with a large enough length, it should be possible to uniquely identify the system. Therefore we can transform this into a classification problem, in which we attempt to classify which logic program a specific series of state transition belongs to. Neural networks are known to be good at performing classification, which makes them suitable tools for our proposed approach.

Our proposed approach works well even with a limited amount of data. This is possible because the neural network used in our model is not trained to model the dynamical system itself, but rather to output a classification of different systems. Therefore, it can be trained on artificial data prior to being applied to real data. Thus it is easy to see that the amount of data obtained has no direct relation with the performance of our model.

The rest of the paper is organized as follows. We cover some of the prior researches in Section 2, following by introducing the logical and neural network background required in Section 3. Then we present the RNN-LFIT approach in Section 4. We pursue by presenting an experimental evaluation demonstrating the validity of an approach in Section 5 before concluding the paper in Section 6.

Related Work

Standard LFIT

One way of implementing the LFIT algorithm is by relying on a purely logical method. In [START_REF] Inoue | Learning from interpretation transition[END_REF], such an algorithm is introduced. It constructs an NLP by doing bottom-up generalization for all the positive examples provided in the input state transition. An improved version of this algorithm, utilizing binary decision diagrams as internal data structures, was introduced in [START_REF] Ribeiro | A BDD-based algorithm for learning from interpretation transition[END_REF]. These methods, while proven to be theoritically correct, generate rules from every positive examples. The resulting NLP has been proven to be non-minimal, and thus not very humanfriendly. To allow practical use of the resulting NLP, a method for learning minimal NLP was introduced in [START_REF] Ribeiro | Learning prime implicant conditions from interpretation transition[END_REF]. In [START_REF] Ribeiro | Learning delayed influences of biological systems[END_REF], an algorithm that learns delayed influences, that is cause/effect relationship that may be dependent on the previous k time steps, is introduced. Another recent development in the prolongation of the logical approach to LFIT is the introduction of an algorithm which deals with continuous values [START_REF] Ribeiro | Inductive learning from state transitions over continuous domains[END_REF].

This class of algorithms that utilizes logical methods, are proven to be complete and sound, however a huge disadvantage with these methods is that the resulting NLP is only representable of the observations that have been fed to the algorithm thus far. Any observations that did not appear in the input, will be predicted as either to be always true or always false depending on the algorithm used.

NN-LFIT

To deal with the shortcomings stated in the previous paragraph, an algorithm that utilizes neural networks (NN) was proposed [START_REF] Gentet | Learning from interpretation transition using feed-forward neural network[END_REF]. This method starts by training a feed-forward NN to model the system that is being observed. The NN, when fully trained, should predict the next state of the system when provided with the current state observation. Then, there is a pruning phase where weak connections inside the NN are removed in a manner that doesn't affect the prediction accuracy. After the pruning phase, the algorithm extracts rules from the network based on the remaining connections within the NN. To do so, a truth table is constructed for each variable. The truth table contains variables only based on observing the connections from the outputs to the inputs of the trained and pruned NN. A simplified rule is then constructed from each truth table. In [START_REF] Gentet | Learning from interpretation transition using feed-forward neural network[END_REF], it is shown that despite reducing the amount of training data, the resulting NLP is still surprisingly accurate and representative of the observed system. However, this approach does not deal with systems that have inherent delays.

Other NN-based Approaches

There are also several other approaches attempting to tie NNs with logic programming [START_REF] Garcez | Symbolic knowledge extraction from trained neural networks: A sound approach[END_REF][START_REF] Garcez | The connectionist inductive learning and logic programming system[END_REF]. In [START_REF] Garcez | Symbolic knowledge extraction from trained neural networks: A sound approach[END_REF], the authors propose a method to extract logical rules from trained NNs. The method proposed deals directly with the NN model, and thus imposes some restrictions on the NN architecture. In particular, it was not made to handle delayed influences in the system. In [START_REF] Garcez | The connectionist inductive learning and logic programming system[END_REF], a method for constructing NNs from logic program is proposed, along with a method for constructing RNNs. However this approach requires background knowledge, or a certain level of knowledge about the observed system (such as an initial NLP to improve on) before being applicable.

In [START_REF] Khan | Construction of gene regulatory networks using recurrent neural networks and swarm intelligence[END_REF], the authors proposed a method for constructing models of dynamical systems using RNNs. However, this approach suffers from its important need of training data, which increases exponentially as the number of variables grow. This is a well-known computational problem called the curse of dimensionality [START_REF] Donoho | High-dimensional data analysis: The curses and blessings of dimensionality[END_REF].

In contrast to these methods, the method proposed in this paper does not assume there exists a direct relation between the trained RNN model and the observed system. Our model aims at classifying a series of state transition to the system that generated it, whereas each of the NN based approaches listed above aims to train a NN model that predicts the next state of the observed system.

Background

LFIT

The main goal of LFIT is to learn a normal logic program (NLP) describing the dynamics of the observed system. NLP is a set of rules of the form

A ← A 1 ∧ A 2 • • • ∧ A m ∧ ¬A m+1 ∧ • • • ∧ ¬A n (1)
where A and A i are propositional atoms, n ≥ m ≥ 0. ¬ and ∧ are the symbols for logical negation and conjunction. For any rule R of the form 1, the atom A is called the head of R and is denoted as h(R). The conjunction to the right of ← is called the body of R. We represent the set of literals in the body of R as b(R) = {A 1 , . . . , A m , ¬A m+1 , . . . , ¬A n }. The set of all propositional atoms that appear in a particular Boolean system is denoted as the Herbrand base B.

An Herbrand interpretation I is a subset of B. For a logic program P and an Herbrand interpretation I, the immediate consequence operator (or T P operator) is the mapping T P : 2 B → 2 B :

T P (I) = {h(R) | R ∈ P, b + (R) ⊆ I, b -(R) ∩ I = ∅}. (2)
Given a set of Herbrand interpretations E and {T P (I) | I ∈ E}, the LFIT algorithm outputs a logic program P which completely represents the dynamics of E.

In the case of Markov(k) systems (i.e. systems with delayed effects of at most k time steps), we can define the timed Herbrand base of a logic program P , denoted by B k , as follows:

B k = k i=1 {v t-i | v ∈ B} ( 3
)
where t is a constant term which represents the current time step. Given a Markov(k) system S, if all rules R ∈ S are such that h(R) ∈ B and b(R) ∈ B k , then we represent S as a logic program P with Herbrand base B k . A trace of execution T of S is a finite sequence of states of S. We can define T as T = (x 0 , . . . ,

x n ), n ≥ 1, x i ∈ 2 B .
Thus a k-step interpretation transition is (I, J) where I ⊆ B k , J ⊆ B.

Neural Network

A multi-layer perceptron (MLP) is a type of feed-forward neural network. An MLP usually consists of one input layer, one or more hidden layer and an output layer. Each layer is fully connected, and the output layer is activated by a non-linear function. MLPs can be trained using backpropagation by gradient descent. The other neural network that we use to learn the system's dynamics is Long Short-Term Memory (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF]. LSTM is a form of RNN that, contrary to earlier RNNs, can learn long term dependencies and do not suffer from the vanishing gradient problem. It has been popular in many sequence to sequence mapping application such as machine translation [START_REF] Sutskever | Sequence to sequence learning with neural networks[END_REF]. An LSTM consists of a memory cell for each time step, and each memory cell has an input gate i t , an output gate o t and a forget gate f t . When a sequence of n X time steps X = {x 1 , x 2 , . . . , x n X } is given as input, LSTM calculates the following for each time step:

    i t f t o t l t     =     σ σ σ tanh     W • h t-1 x t c t = f t • c t-1 + i t • l t h t = o t • c t
where W is a weight matrix, h t is the output of each memory cell, c t is the hidden state of each memory cell and l t is the input to each memory cell. σ is the sigmoid function. The input gate decides how much of the input influences the hidden state. The forget gate decides how much of the past hidden state influences the current hidden state. The output gate is responsible for deciding how much of the current hidden state influences the output. A visual illustration of a single LSTM memory cell is shown in Figure 1.

LSTM networks can be trained by performing backpropagation through time (BPTT) [START_REF] Graves | Framewise phoneme classification with bidirectional lstm and other neural network architectures[END_REF]. In BPTT, the LSTM is trained by unfolding across time steps, and then performing gradient descent to update the weights, as illustrated in Figure 2. A direct consequence of BPTT is that the LSTM can only be trained on fixed-length data. One way of overcoming this is by using truncated BPTT [START_REF] Williams | An efficient gradient-based algorithm for on-line training of recurrent network trajectories[END_REF]. In truncated BPTT, the sequence is truncated into subsequences, and backpropagation is performed on the subsequences.

It can easily be seen that the connections in an LSTM model are complex, and it can be very complicated to attempt to extract or derive relations from the inner architecture of the network. Therefore we forgo the approach of extracting rules from the model, and propose a different method which instead utilizes the LSTM to classify the different inputs depending on the system that generated them.

Model

In this section, we propose an architecture for performing LFIT. It consists of an encoder and decoder for the state transitions, and a neural network for performing LFIT. A visualization of the architecture is shown in Figure 3 and4. The input for the whole model is the sequence of state transitions obtained from observing the target system. The output of the model is an encoding of an approximation of the logic program representation in ... , where an encoder LSTM will receive a series of state vectors and encode them into a single vector, then we will multiply the logic program representation matrix and produce a vector, which will be decoded by an MLP to produce the predicted state. matrix form. However, we will be performing evaluation of the performance of the model based on the predicted state.

Given a series of state transitions X T = (x 1 , x 2 , . . . , x T ), where x t ∈ [0, 1] represents the state of the system at time t, our goal is to predict x T +1 . Note that to be able to deal with noise and continuous values, we are not restricting the domain of x t to Z 2 . If we obtain a representation of X T in the form of a vector x, we can learn a matrix P, with which we can perform matrix multiplication as Px = x T +1 . This can be thought of as performing the T P operator in algebraic space.

The training objective function of the model is defined as:

min W 1 n n i=1 (x (i) T +1 -y (i) T +1 ) 2 + λ W 2 2 ( 4
)
where W is the set of neural network weights, x T +1 is the prediction of the model, y T +1 is the truth state, W 2 2 is the weight decay regularization [START_REF] Krogh | A simple weight decay can improve generalization[END_REF] with hyperparameter λ.

The input state transition is fed to both the encoder and the LFIT model, as can be seen in the figure. We describe the responsibilities of the three neural network models in the following sections. 4: A visualization of the LSTM model that is responsible for performing the LFIT. It receives as input a matrix p 0 , a series of state vectors x 1 , . . . , x t and outputs a matrix.

Autoencoder

The autoencoder for the input sequences is responsible for encoding discrete time series into a feature vector that can later be manipulated by the neural network. This sequence of vectors is then encoded into one feature vector of dimension 2 × k a × l a , where k a denotes the number of memory cell units in the autoencoder LSTM and l a denotes the number of LSTM layers. This amount is doubled because both c and h, which represent the state of the memory cell, are considered.

LFIT Network

This LSTM network can be thought of as performing LFIT. This network takes as input the state transitions and an initial program encoding and outputs a program encoding that is consistent with the observations, which is the same as the definition of the LFIT algorithm. Although in practice, this network is responsible for classifying the series of state transition to the corresponding logic program representation.

The produced output is the representation of the normal logic program. The observations are the same input sequence as that given to the autoencoder. The dimensions of the matrix output by this network is

(2 × l l × k l , 2 × l a × k a ),
where k l denotes the number of memory cell units in this network and l l denotes the number of layers.

In this work, the initial program is always set to ∅ and the LSTM network is trained to produce the complete normal logic program representation. In future work, it could be easily extended so as to accept background knowledge.

Decoder

The decoder is responsible for mapping the product of the NLP matrix and the state transition vector into a state vector that represents the predicted following state. The decoder can theoritically be any function that maps a continuous vector into a binary vector. We detail the model used in Section 4.4.

The goal of the architecture is to produce an encoding of past states, and an encoding of a normal logic program, that can then be multiplied together to predict the next state transition. This multiplication is a matrix × vector multiplication and produces a vector of R n where n is the number of features in the logic program representation. This can be thought of as performing the T p operator within linear geometric space. A MLP then decodes this vector into the desired boolean state vector.

With the encoding of the state transition and an initial program, the LFIT network learns to produce an encoded program based on the observed state transitions. This encoded program can then be used for prediction, and in future work we plan to decode it into a normal logic program thus making it possible to reason with it.

Model Details

In our experiment, the autoencoder takes a series of 10 state transitions, where each state is a 10 dimensional vector which represents the state of each variable within the system. The autoencoder LSTM model we trained has 2 layers, each with 512 memory cell units. The produced state representation is then multiplied by a (2 × 2 × 512, 128) matrix, to produce a 128 dimension feature vector that represents the series of state transitions.

The LFIT model takes the same input as the encoder model, but the LSTM model has 4 layers, 4 being the dimension of the resulting feature vector for the predicted state, and has 1,024 hidden units which is twice the number of hidden units of the autoencoder model. The produced logic program representation is then transformed into (4, 128) matrix by multiplying it with a (2 × 4 × 1024, 4 × 128) matrix and then reshaping.

The decoder model takes the resulting feature vector for the predicted state, which is a vector of 4 dimensions, and outputs a vector of 10 dimensions with each dimension representing the state of the variables within the system. The decoder model consists of a MLP with 1 hidden layer, and each layer has 8 hidden units. Each hidden layer is activated by ReLU (Rectified Linear Unit), which is a function that outputs 0 for all input less than 0, and is linear when the input is larger than 0. The final output layer is activated by a sigmoid function, which is defined as σ(x) = 1/(1 + exp(-x)). The sigmoid function has a range of [0, 1], which is suitable for our use where we want the MLP to output a boolean vector, with noise. The decoder model is simple, this is to avoid the decoder overfitting and thus preventing the LFIT model and the encoder model from learning.

Evaluation

We applied our model to learn the dynamics of Boolean networks from continuous time series. The Boolean network used in this experiment is adapted from Dubrova and Teslenko [START_REF] Dubrova | A sat-based algorithm for finding attractors in synchronous boolean networks[END_REF] and represents the cell cycle regulation of mammalians. The Boolean network is first encoded as a logic program. Each dataset represents a time series generated from an initial state vector of continuous values. The performance of the model is measured by taking the root mean-squared error (RMSE) between the predicted state and the true subsequent state. RMSE is defined as following:

RMSE = 1 n n i=1 (ŷ i -y i ) 2 (5)
where ŷi denotes the predicted value and y i is the actual value.

The initial state vector is generated by giving each of the 10 variables a random value between 0 and 1. Generated states are then mapped back to real values: 0 becomes 0.25 + and 1 becomes 0.75 + , where ∈ (-0.25, 0.25), chosen randomly simulates the measurement noise. We used the following training parameters for our experiment:

-Training steps: 10 4 -Batch size: 100 -Gradient descent optimizer: Adam, learning rate and various other parameters are left with the defaults for Tensorflow r1.2 -Dropout: probability of 0.3 per training step -Regularization hyperparameter λ of 0.2

The model was implemented on Tensorflow r1.2 [START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF], and all experiments were done on Intel Xeon E5-2630 with 64 GiB of RAM and GTX 1080 Ti.

Training data is generated randomly by first randomly generating logic rules and grouping them together as NLPs. Then the initial state is set as the zero vector, and we continuously perform the T P operator to generate all the consequent states. Variables referring to delays before the initial state is assumed to be 0. In order to assure effective training of the model, we only train on data that varies a lot. We do so by calculating the standard deviation of all states that are generated from a certain NLP, and only keeping those with standard deviation greater than or equal 0.4. We show some of the accepted NLPs in table 3.

Here, we consider two methods for training the model. One by training the model with data without noise, that is the training data is strictly Z 2 . Another way of training the model is by training on data with added noise. Each model is trained with 50 acceptable NLPs, generating 500 data points from each NLP, and training for a total of 4 hours. We evaluate each method in the following section. 1 shows the RMSE of the prediction made by the proposed model. Each dataset represents 50 datapoints from the same NLP, generated from different initial states. The results show that there is little difference in the accuracy between dataset with noise and without noise, which shows that the robustness of our model regarding the presence of noise in the input. In table 2, we show the performance of the model trained with data with noise. Comparing with table 1, the presence of noise in the training data doesn't affect the performance. Both models are equally robust in dealing with noise in the test data. The results obtained appear a little bit skewed due to being produced from the same system. We are planning to test the model in various other dataset when we can get access to them.

Results

Dataset

Figure 5 shows the graph of the learned representation for 8 different randomly generated NLPs based on principal component analysis (PCA). PCA is a popular technique for visualizing high dimensional embeddings [START_REF] Wold | Principal component analysis[END_REF]. As with the previous experiment, the model is fed with state transitions that were generated from the NLPs. The logic representation obtained from our model is a 4 × 128 matrix. We obtain the graph shown in Figure 5 by applying PCA on this matrix which extracts 3 of the dimensions that separate the data the most. Each dot in the graph is a representation learned separately from various state transitions from the logic program. Note that learned representations that are from different logic programs are clearly separated. For each dot plotted on the graph, they are actually multiple dots, representing different initial state generated from the same NLPs. The overlap for NLP 7 and NLP 8 that can be seen in the plot is due to the 2D projection of a 3D graph.

In this experiment, we observe that the model is able to identify the dynamics of the system solely based on a sequence of state transitions. We further expect that the accuracy of the predictions can be improved more by tweaking the neural network architecture. In this paper we propose a method for learning a matrix representation of dynamical systems with delays. One of the interesting aspects of this approach is that it produces a logic program representation in matrix form, which when multiplied with a feature vector of the past states, is able to compute a vector that represents the predicted state. This could lead to future works such as reasoning and performing induction purely in the algebraic space.

The main contribution of this work is to devise a method of modeling systems where only limited amounts of data can be collected. Without sufficient amount of data, purely logical methods cannot provide useful information, and attempts at training neural networks to model the system will result in overfitting. Therefore we speculate that generating artificial data in order to train a more generalized neural network may be a more successful approach in such cases. We also managed to show that the devised method is resilience to noise, where purely logical methods are not able to deal with.

As future work, we are planning to adapt the current method to take as input a partial program as background knowledge to the network and to decode the NLP representation into logical form to allow humans to reason with. We also hope to evaluate the predictions made by this model with other similar models. a t ← f t-5 ∧ ¬d t-4 ∧ ¬i t-1 ∧ ¬g t-1 ∧ ¬g t-4 ∧ ¬d t-1 b t ← ¬d t-1 ∧ ¬d t-5 c t ← ¬b t-1 d t ←¬c t-1 ∧ ¬i t-5 ∧ ¬f t-3 ∧ ¬c t-2 ∧ ¬i t-1 ∧ ¬h t-1 ∧ ¬a t-1 ∧ ¬d t-3 ∧ ¬d t-5 e t ← e t-2 ∧ ¬e t-1 ∧ ¬a t-3 ∧ ¬f t-4 ∧ ¬j t-5 f t ← b t-2 ∧ g t-1 ∧ h t-5 ∧ ¬i t-2 ∧ ¬f t-2 g t ← ¬d t-1 ∧ ¬g t-1 h t ← ¬i t-1 i t ← ¬e t-4 ∧ ¬j t-1 ∧ ¬d t-2 ∧ ¬g t-5 ∧ ¬c t-2 ∧ ¬i t-5 ∧ ¬g t-3 ∧ ¬j t-2 ∧ ¬i t-1 j t ← b t-3 ∧ c t-4 ∧ ¬j t-2 ∧ ¬c t-3 a t ←b t-3 ∧ g t-2 ∧ f t-4 ∧ j t-4 ∧ ¬c t-5 ∧ ¬e t-2 ∧ ¬a t-4 ∧ ¬h t-3 ∧ ¬i t-3 ∧ ¬h t-1 ∧ ¬e t-3 ∧ ¬c t-1 ∧ ¬c t-2 ∧ ¬a t-5 b t ←¬e t-4 ∧ ¬c t-3 ∧ ¬i t-3 ∧ ¬f t-3 ∧ ¬b t-2 ∧ ¬i t-5 ∧ ¬i t-3 ∧ ¬a t-5 ∧ ¬f t-5 c t ← ¬c t-3 ∧ ¬b t-1 ∧ ¬c t-5 ∧ ¬j t-2 ∧ ¬b t-5 ∧ ¬i t-2 ∧ ¬a t-5 ∧ ¬b t-3 d t ← ¬e t-4 ∧ ¬a t-5 ∧ ¬e t-4 e t ← ¬g t-1 f t ← h t-1 ∧ e t-3 ∧ c t-3 ∧ ¬a t-2 ∧ ¬g t-4 g t ← ¬f t-5 h t ← ¬e t-5 i t ← ¬j t-3 ∧ ¬a t-5 ∧ ¬i t-4 j t ← ¬g t-5 ∧ ¬e t-5 ∧ ¬d t-1 a t ← ¬e t-2 b t ← ¬b t-4 c t ← j t-1 ∧ f t-1 ∧ f t-2 ∧ d t-1 ∧ h t-5 ∧ ¬g t-3 ∧ ¬c t-5 d t ←¬g t-3 ∧ ¬b t-5 ∧ ¬c t-3 ∧ ¬b t-5 ∧ ¬j t-3 ∧ ¬h t-2 ∧ ¬f t-5 ∧ ¬d t-2 ∧ ¬c t-5 e t ←g t-2 ∧ g t-4 ∧ f t-5 ∧ j t-3 ∧ e t-1 ∧ ¬j t-1 ∧ ¬a t-1 ∧ ¬f t-1 ∧ ¬e t-4 f t ←f t-5 ∧ b t-5 ∧ g t-5 ∧ ¬j t-2 ∧ ¬c t-5 ∧ ¬i t-5 ∧ ¬g t-4 ∧ ¬g t-5 ∧ ¬f t-2 ∧ ¬f t-3 ∧ ¬h t-4 g t ← a t-2 ∧ d t-3 ∧ ¬g t-2 ∧ ¬c t-3 h t ← ¬j t-5 ∧ ¬e t-4 ∧ ¬g t-5 ∧ ¬f t-1 i t ← ¬e t-4 j t ← ¬i t-5 Table 3: Example NLPs that are randomly generated and used for training

Fig. 1 :

 1 Fig. 1: An LSTM memory cell

Fig. 2 :Fig. 3 :

 23 Fig. 2: Unfolding of an LSTM network for BPTT training

  Fig.4: A visualization of the LSTM model that is responsible for performing the LFIT. It receives as input a matrix p 0 , a series of state vectors x 1 , . . . , x t and outputs a matrix.

Fig. 5 :

 5 Fig. 5: PCA plot of the learned representation for NLPs based on input time series 6 Conclusion and Future Work

Table 1 :

 1 Results of the RMSE of the prediction made by the proposed model trained on non-noisy data on various datasets

		RMSE (Original) RMSE (Noisy)
	1	0.27	0.28
	2	0.27	0.28
	3	0.26	0.26
	4	0.27	0.26
	5	0.27	0.28
	6	0.27	0.27
	7	0.27	0.28
	8	0.27	0.28
	9	0.27	0.27
	10	0.27	0.27
	Dataset RMSE (Original) RMSE (Noisy)
	1	0.27	0.28
	2	0.27	0.27
	3	0.27	0.28
	4	0.27	0.28
	5	0.28	0.28
	6	0.27	0.28
	7	0.28	0.28
	8	0.27	0.27
	9	0.27	0.27
	10	0.27	0.27

Table 2 :

 2 Results of the RMSE of the prediction made by the proposed model trained on noisy data on various datasets

Table