Mrs Nesrine Chehata

Mr Aymen Yahyaoui

Mr Khalil

Mr Hassan

Mrs Takoua

Keywords: Network Traffic Prediction, Time Series, Machine Learning, Recurrent Neural Networks, Edge Computing, Distributed Computation, Convolutional Neural Network, Deep Q-Network. i ANN Artificial Neural Networks. AP Access Point. AR Augmented Reality. CC Cloud Computing. CNN Convolutional Neural Network Recurrent Unit. ICMP Internet Control Message Protocol. IoT Intenet of Things. IP Internet Protocol PCAP Packet Capture. viii Recurrent Neural Network. SVM Support Vector Machine. SVR Support Vector Regression. TCP Transmission Control Protocol. UDP User Datagram Protocol

The emergence of Machine Learning (ML) has increased exponentially in numerous applications, including computer vision, speech recognition and natural language processing. This is due to the intersection of the massive data generated by IoT devices (e.g. sensors, mobile phones and smart devices) and the computing power available in cloud and data centers. Hence, complex systems have been designed to improve the quality of life such as health monitoring, smart homes and intelligent transport systems. However, the cloud computing has a number of issues related to the trade-off between efficiency and latency. Indeed, the former cannot ensure the performance for real-time applications due to the long-range network latency between the cloud and end devices. For that reason, we propose to leverage these issues in this work by proposing alternative solutions. First, we present a real-time network traffic prediction using different DL models and compare their performance to a classic ML model named SVR by using the Hyperparameters tuning process. In fact, traffic prediction is a tool for Internet service providers to serve the customers better QoS, i.e. lessen the latency for sensitive applications. Second, we study the edge computing paradigm that attempts to unleash DL services by bringing the cloud capabilities close to end users by deploying edge servers. Thus, we propose a distributed inference framework for DL models. The latter intends to attain a high performance while minimizing the computation latency for sensitive services and the energy consumption of IoT devices. Extensive evaluations on the standard CNN "ResNet20" for object recognition task using CIFAR10 dataset shows the effectiveness of the proposed framework in controlling latency versus inference accuracy. To further enhance the distributed inference framework, we propose automating the offloading decision and resource allocation of computing tasks for IoT devices using Deep Q-Network (DQN), a well-known DRL algorithm. Several Simulations show that DQN has been successful in managing execution of tasks belongs to IoT devices in order to make the best use of available resources.

Introduction

Thanks to advances in cloud computing capabilities and data center storage, Deep Learning (DL) has been very effective in ML across a wide range of application domains, including computer vision, virtual/augmented reality, speech recognition and natural language processing. For instance, the worldwide translator "Google Translate" uses Long-Short Term Memory (LSTM) network, a DL model dedicated for natural language processing, in order to mine the semantics of the sentences instead of mapping sentence to sentence [START_REF] Wu | Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation[END_REF]. However, DL's high performance comes at the cost of a long training of a deep network that encapsulates millions of parameters. Additionally, DL inference is computationally expensive due to the high dimensionality of the input data and the millions of computations that need to be performed over multiple layers. Furthermore, it is becoming very difficult for the cloud to meet the demands of computing tasks and real-time response due to the enormous expansion of IoT devices connected to the Internet. According to Statista, the number of IoT connected devices has increased by 200% from 2015 to 2020 [START_REF]The Statistics Portal. Internet of things (iot): number of connecteddevices worldwide from 2012 to 2020[END_REF]. The study also found that it would reach approximately 75 billion connected devices in 2025 as depicted in fig. 1. Moreover, Cisco Annual Internet report reveals that IoT devices will produce around 850 ZB of data each year starting from 2021 [START_REF]Cisco Global Cloud Index: Forecast and Methodology[END_REF]. Thus, actual cloud computing capabilities are unable to manage this massive data generated from billions of devices. Indeed, the former cannot guarantee the delay requirements for realtime applications of end devices (e.g. sensors, mobile users, cameras). Besides, offloading the 1 Acronyms computation from end devices to the cloud causes privacy issues related to sensitive data such as sending personal information or indoor image capture [START_REF] Ledig | Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network[END_REF].

A possible alternative is to analyze locally the massive flow of data generated by the end devices . Yet the problem of computing resources and energy limitations arises. In fact, IoT devices are known as resource-constrained devices because they were unable to meet the DL needs. Hence, edge computing paradigm [START_REF]Mobile-Edge Computing Introductory Technical White Paper[END_REF] appear to be an attractive solution that meets the issues related to latency, scalability and privacy. The latter consists of migrating the DL computations near the end devices by deploying local edge servers so that cloud resources becomes available and with a promising response time. As a result, edge servers unburden the huge computation demands from end devices to the cloud. For instance, a possible usecase is to deploy edge servers to enable interactive experience at home by creating augmented reality service for online shopping [START_REF] Adib | Edge computing -what is edge computing?[END_REF]. Even Edge Computing (EC) has a number of issues related to the computing power. In fact, edge servers have not been able to cover computing capabilities offered by cloud servers. As a result, edge servers cannot respond to the high resource requirements that DL applications need.

Therefore, EC and CC should work together in order to respond to the IoT needs. In fact, while EC can provide low latency and privacy, CC offers high power resources and scalability. Actually, tremendous reseach that is going around in the thematic of collaboration between EC and CC. It is in this perspective, that we want to implement, during this graduation project, an alternative solutions that responds to the high demand of end devices using the collaboration between cloud computing and edge computing. This report summarizes the work we have achieved and the results that we have reached since the month of February.

The outline of this report as follows:

• General Framework of The Project

Introduction

In this chapter, we start by representing the host laboratory LAAS-CNRS and the general context of this internship.

Presentation of the host organization: LAAS-CNRS

Presentation of CNRS

The CNRS (Centre National de la Recherche Scientifique) is the french national research organization and the main basic science institution in Europe. It is founded on several reseach units. Each research unit has a specific mission in specific domains. Therefore, CNRS carries out his activity in all fields of knowledge.

Presentation of LAAS-CNRS

The LAAS-CNRS (Laboratory of Analysis and Architecture of Systems) is a CNRS research unit, established in 1968, based in Toulouse.

The research at LAAS-CNRS aims at fundamental understanding of complex systems while considering the use that can result from it. Actually, The laboratory holds 8 scientific departments defines the directions for the coming years and coordinate the activities of 22 research teams. In fact, the names of the scientific departments are as follows:

• Crucial Computing Avancés) which his main thematic lies in networking and communication. In fact, the team's contributions relate in particular to the development of methods, models as well as to the proposal of architectures, protocols and services.

Chapter 2 Mathematical Background

Introduction

In this chapter, we provide an overview of the theoretical background of ML models used in this project. The first section focuses on the presentation of ML and explains the different categories of ML algorithms. A second part explains Deep Learning (DL) concepts and its main components. Then, we present DL models used in this project. The third section introduces the basic concepts in RL with a formulation of problems solved by RL models. In the last part, we turn our focus to Deep Reinforcement Learning (DRL), a combination of RL and DL, and we discuss the different class of RL algorithms.

Machine Learning (ML)

Definition

Machine Learning is a subfield of Artificial Intelligence, which involves the development of self learning algorithms in order to make forecasts on new data. Because of the vast amounts of data generated in our real world, taking advantage of these algorithms is becoming extremely potential. Additionally, ML algorithms allow data patterns to be captured autonomously in order to make data driven decisions.

Types of ML

In the literature, ML algorithms are classified into three categories:

• Supervised ML: In this class, we want to learn a predictive function f that maps inputs X to the desired outputs Y . A training process will be applied until the model accomplishes a desired level of performance on a training data.

• Unsupervised ML: Unlike supervised ML, this type of algorithms doesn't impose labels for each row of data. Therefore, it learns hidden patterns in the data in order to make clustering and feature reduction. For instance, it's widely used for segmenting customers in different groups for specific intervention.

• Reinforcement Learning (RL): RL is a type of learning where a machine is trained to make optimal decisions autonomously by exposing it to an external environment. Hence, this project focuses on both supervised ML and RL.

Deep Learning (DL)

Introduction

Deep Learning is a new subfield of ML research. In fact, it was inspired by the structure and the function of the brain in processing data and creating patterns for use in decision making. Back through time, DL paradigm was explored since 1950 [START_REF] Pitts | Handwritten digit recognition with a back-propagation network[END_REF]. Then, a lot of research was carried out until 1980, when they were struggling to train their proposed models and to provide the expected storage for their data sets due to the limited hardware capacities. In 1989, the first handwritten digits recognition model was build by Y. Le Cun using backpropagation networks [START_REF] Pitts | Handwritten digit recognition with a back-propagation network[END_REF]. Nowadays, DL surpasses the human expert knowledge in several fields such as computer vision, natural language processing and speech recognition by dint of the high computing performance and the incredible available storage in cloud data centers. For instance, Google's DeepMind developed a system capable of beating a professional Go player in 2015 using deep reinforcement learning [START_REF]Google AI algorithm masters ancient game of Go[END_REF].

Artificial Neural Networks (ANN)

Definition

Neural Network is a series of algorithms that tried to mimic the function of the brain in order to recognize the relationships in a set of data. It was discovered in the 80s and the early 90s and was utilized later in many applications such as face recognition, object detection and many other applications that are very difficult to solve unless the neural networks are existed.

An ANN is based on a collection of connected units where each neuron can transmit an information to another neuron. A neuron could contain a weight that varies as learning proceeds in order to minimize the loss function which we will refer to it in the next parts.

In the theoretical perspective, it is simply a non-linear function that maps the inputs to the outputs by finding correlations and could be written in a formula:

Y = f (X) (2.1)
where Y represents the Output, X is the Input and f is a non-linear function.

ANN Architecture

In ANN, each node presents a neuron, and each column represent a layer. The first layer is called the input layer. The rightmost or output layer contains the output neurons, the number of neurons represents the number of classes we are classifying. Any intermediate layer is called a hidden layer. In addition, each neuron has a propagation function that transforms the outputs of the connected neurons often with a weighted sum followed by an activation function. The example depicted in fig. 2.2 of an ANN is composed of 6 neurons as input layers, 1 st hidden layer with 4 layers, 2 nb hidden layer with 3 neurons, and the output layer contains one single neuron.

y = f (b + m i=1 w i .x i ) (2.2)
where:

• b: represent the bias which avoid the model from the overfitting problem where the NN adopts its parameters to the training set and achieve low performance on the test set.

• f : represents a nonlinear activation function applied to the weighted sum of inputs. It is one of the key knob that make it possible to model non-linear problems. Actually, we need to use nonlinear model to make sense of a complicated and a high dimensional data.

The most used activation functions are shown in the table. 2.1: As the name reveals, the input data will be forwarded through the network. Each hidden layer accepts input data, processes the data by calculating the activation function of each received signals for each neuron, and then continues to extract the information until it reaches the output layer where we have the prediction result.

Activation Function Formula ReLU f (x) = max(0, x) Tanh f (x) = 2 1+exp(-2x) -1 Sigmoid f (x) = 1 1+exp(-x)

Backward Propagation

Backward propagation is the essence of neural network training. It is the practice of finetuning the weights of a neural network based on error rate induced by the loss function. Backward propagation uses the gradient of the error function with respect to the weights and biases of the model to find the correct direction to minimize the error. The optimization algorithm and the learning rate variable are those who control the correction. This mechanism is based on loss function and optimization functions that I did not mention in my report so as not to go deeper into the concept of the neural network. For those who are interested in understanding this concept, I recommend them viewing the courses of DeepLearning.ai offered by Andrew Ng in Coursera platform [START_REF] Ng | Spécialisation Deep Learning[END_REF].

Convolutional Neural Network (CNN)

Introduction

CNN is a class of ANN and the most used type in Computer Vision and Augmented Reality. It was proposed by Yann Lecun in order to diminish the complexity of the calculations in a complex neural network i.e. a fully connected neural network.

Definition

The CNN inherits the same architecture of an artificial neural networks i.e. he possessed an input layer, a series of hidden layers, and an output layer. Unless, he has specific hidden layers, where each hidden layer consists of a convolutional layer that convolves using multiplication or dot product. Hence, we present the most used types of layers in CNN as demonstrated in the next section.

CNN Layers

• Convolutional Layer: The convolutional layer is the core of building block of a CNN that does most of the computational heavy lifting. This latter is a collection of filters that are convoluted with the previous layer that are designed to extract features and information. Indeed, each filter has its own hyperparameters such as: Dimensions, Stride, Padding. • Pooling Layer: Pooling Layers are used after convolutional layers in a CNN in order to reduce the dimensions of the data by combining the outputs of neuron clusters at one layer into a single layer. There are many types of Pooling layer such as:

-Max-Pooling: Compute the maximum value in each cluster in order to extract the most important pixel in the patch.

-Average-Pooling: Compute the average value in each cluster in order to extract the mean value in the patch. RNN is a subclass of ANN that contains feedback loops as opposed to FFNN. As depicted in fig. 2.6, these recurrent connections allow the information about previous time step to persist in the internal state of the network and hence influence the output efficiency. For instance, RNN have recieved huge success in dealing with data such as time series which have a sequential order that needs to be followed in order to be understandable.

ŷt = f Y (h t ) (2.3)
Where:

• h t : represents the hidden state of the network. In fact, it stores the actual input signal each time we make a forward pass and apply a non linear function as demonstrated below:

h t = f W (x t , h t-1 ) (2.4)
• f W : represents a non linear function that merge the signals of both the input vector and the previous hidden state.

• f Y : represents a non linear function applied to the actual hidden state in order to produce the predicted vector.

As a result, the RNN model updates both its hidden state and the predicted vector within each forward pass.

For example, one of the most used nonlinear functions for the RNN update is represented as follows:

h t = tanh(W T hh h t-1 + W T xh x t ) (2.5) y t = W T hy h t (2.6)
Where: W xh , W hh and W hy represents respectively the weight matrices of the input, the hidden state and the output.

Definition

RL is a subfield of ML that allows machines to learn automatically by interacting with an external environment without the need to acquire a data set for training and evaluation.

More formally, RL is a computational approach to learn from interaction where an agent pick out an action and receives a numerical signal, named reward, that informs the agent of the relevance of the action chosen. Instead of drawing on the experience of experts to learn from, RL allows an agent to learn on its own by focusing on maximizing the cumulative rewards received from the environment. Actually, the concept of learning by interacting with an external environment is an imitation from the natural learning process of human being.

In this section, we present the elements of RL. Then, we turn our focus to explain in advance the deployed algorithm in this project.

RL Components

Reinforcement Learning is based on learning behavior through interactions with the environment in a discrete time steps. As depicted in figure 2.8, the interaction has two main stages, observation and action. The interaction is basically between the agent and the environment. Hence, the environment begins by sending the current state s t to the agent. Based on the knowledge acquired by the agent, he will take an action a t in response to that state. As a result, the environment emits a feedback based on the chosen action. The feedback contains a reward signal r t and a new state s t+1 . The former represents the influence of the action on the environment and measures the success or failure of the agent's action while the latter is the new state of the agent. This process is repeated between the agent and the environment until a stop condition has been validated. The agent attempts to maximize the discounted cumulative reward R t instead of the current reward. Indeed, the cumulative reward is expressed as follows:

R t = ∞ i=t γ i r i = r t+1 + γr t+2 + ... + γ t+n r t+n + ... (2.7)
Where:

• γ represents the discount factor. It is always comprised between 0 and 1. It was designed to give more importance to earlier rewards. From a mathematical perspective, it ensures the convergence of the time series.

Policy

The agent perceive the environment and decides the optimal action according to a policy function π. Actually, the policy represents the core of RL in the sense that it is sufficient to determine behavior [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. It informs the agent what action to be selected in a given state. It could be deterministic or stochastic depending on the problematic and the choice of the model.

In the case of a deterministic policy, it is simply a mapping function from perceiving states of the environment to the actions to be taken in those states. Hence, we have the following expression:

π(s t ) = a t s t ∈ S, a t ∈ A (2.8)
Otherwise, the policy becomes a probability distribution in case of a stochastic policy and it could be expressed as follows:

π(a t |s t ) = pi s t ∈ S, a t ∈ A, pi ∈ [0, 1] (2.9)
During the interaction between the RL agent and the environment, the agent tries to converge to the optimal policy π * . In essence, there are several learning algorithms that exists in the literature that aim to converge the policy using different strategies.

Value Function

Whereas the reward signal indicates what is good in an immediate sense, a value function specifies what is good in the long term. For instance, a state might always provide a low immediate reward but still have a high value because it is regularly followed by other states that yield high rewards. Indeed, value functions estimates the discounted cumulative reward R t for every possible state. It could be formalized as follows:

V π (s) = E π (R t |s t = s) = E π [ ∞ i=t γ i r i |s t = s]
(2.10)

Q-Function

A Q-function defines how good an action is for each state. It represents the expected cumulative reward for every action in each state. Hence, it could be expressed as follow:

Q π (s, a) = E π (R t |s t = s, a t = a) = E π [ ∞ i=t γ i r i |s t = s, a t = a] (2.11)
The use of Q-function is more appropriate than Value function since it evaluates every possible action that could be token in a given state. In contrast, value function takes into account every possible actions and observes all possibles next states and selects the action based on the highest state value which appears too complicated and impractical. Hence, the majority of RL research studies focuses on estimating Q-Function rather than value function.

Furthermore, there is a relationship between Value function and Q-function which could be formalized as follows:

V π (s) = max a∈A Q π (s, a) (2.12)
Ultimately, the agent needs a policy to infer the best action in its state s t . Indeed, the strategy is to choose the best action that maximizes future rewards. The optimal policy could be expressed as follows:

π * (s t ) = arg max a∈A Q(s t , a) (2.13)

Exploration vs. Exploitation Trade-Off

Imagine that you are going out to eat with your friends tonight. When you get to the restaurant, you will have to decide what to order. You've been there a couple of times before and you always order the same thing. So you know you are going to be pretty happy if you order it again. Many of the other items though look really tasty. How do you decide when to order the same good meal again, or try something new.

Actually, the same issue arises for RL agent when it comes to selecting actions based on the actual optimal policy or exploring unvisited states by selecting random action. Indeed, exploration is needed because there is always uncertainty about the actual estimated Q-values. However, the agent risks to end up in a bad state and accumulates negative rewards. Hence, this is so called exploration exploitation trade-off.

To solve this dilemma, there are several approaches that attempts to balance exploration and exploitation. For instance, ε-Greedy is the most widely used strategy in the literature due to its simplicity. Indeed, the agent selects an action randomly with a certain probability ε. Otherwise, the latter exploits the current knowledge by choosing the optimal action with the highest Q-value. As a result, ε-Greedy strategy could be formalized as follows:

a t =    arg max a∈A Q(s t , a)
with a probability 1pull random action with a probability

(2.14)
Besides, it is ε-Greedy strategy that we have adopted in our project.

Deep Reinforcement Learning (DRL)

Definition

In RL, a large amount of memory is usually required to store value functions and Q-functions. Actually, the easiest way to represent an RL problem is to store each state-action pair in the form of table. This is known as Tabular Method. In fact, most of RL uses cases have an infinite state space, which makes it impossible to store value functions or Q-functions in tables. Therefore, the trial and error interaction between the agent and the environment is hard to be learned due to the huge computation and the storage requirements for real world problems. This is where Deep Learning (DL) comes into place in order to estimate Q-functions and value functions. As a consequence, the fusion of DL and RL becomes a promising solution in order to leverage complex real world use cases. For instance, the estimator of Q-function could be expressed as follows:

Q(s, a, θ) Q π (s, a) (2.15)
where:

• θ : represents a real vector that contains the parameters of the estimator. These weights are updated throughout the learning of the agent in the aim to converge to the actual discounted total reward.

• Q : represents the estimator of Q-function.

Q-Function could be approximated using Artificial Neural Networks (ANN). Indeed, ANN could be trained using gradient descent to minimize the error between q and the discount cumulative reward R.

Deep Q-Network (DQN)

Based on estimating Q-Function using neural networks, Mnih et al. [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] introduce Deep Q-Network algorithm in 2015 which have achieved high performance in learning ATARI games.

Chapter 3 Network Traffic Prediction

Introduction

We start this chapter by a general description of network traffic prediction. Then, we turn our focus on exploratory data analysis and the modelling phase. Finally, we end up with performance evaluation of the fitted models and the Hyper-Parameters tuning.

General Description

Problem Statement

With the rapid growth of Internet Technology, network traffic is growing exponentially. As a result, the resource management is becoming more difficult and more complex for Internet service providers. Hence, current traditional models cannot forecast network traffic that acts as a nonlinear system. In this chapter, we propose to leverage the emerging deep learning techniques to predict network traffic. Extensive experiments were conducted to evaluate the performance of the proposed models using real-world data and we compare the accuracy of the chosen model with a baseline model named Support Vector Regression (SVR).

Project Goals

By improving the precision of network traffic prediction, this yields to significant gains for network providers. In fact, traffic prediction has become the engine for many interesting applications such as resource allocation, short-term traffic scheduling, Long-Term capacity planning, network design and anomaly detection.

Categories of Network Traffic Prediction

We can distinguish four types of prediction according to [2] based on the forecasting time scale:

• Real-Time Forecasting: This type requires the shortest time interval between data that do not exceed a few minutes, which can be used to establish an online real-time forecasting system.

• Short-Term Prediction: It often linked with a duration of one day to few hours. Hence, its application is mainly anomaly detection and decision making in critical situations.

• Medium-Term Prediction: Its duration is from several days to one month. In fact, it can be used to guide resource planning.

• Long-Term Prediction: Long-Term prediction is usually one year. Hence, it can be used for economic investment and designing infrastructure.

Thus, based on the duration of collected Internet traces which is about several days, we have been interested in Real-Time forecasting class. Specifically, we want to forecast the volume of traffic in the range of seconds and minutes, using as input a time series signal comprised by the previous observations of this value sampled at one-minute intervals.

Related Work

In the literature, several approaches to network traffic prediction have been proposed. Actually, A. Lazaris et al. [START_REF] Lazaris | Deep Learning Models For Aggregated Network Traffic Prediction[END_REF] propose that multiple types of LSTM be used to forecast the link throughput using real world data set. In addition, W. Shihao et al. [START_REF] Shihao | A Network Traffic Prediction Method Based on LSTM[END_REF] choose to use LTSM model to predict network traffic volume. The former considers the characteristics of the auto-correlations in order to improve the accuracy of the model. Additionally, Azzouni et al. [START_REF] Azzouni | A long short-term memory recurrent neural network framework for network traffic matrix prediction[END_REF] consider the network traffic matrix to estimate the future network traffic from the previous network traffic data using LSTM models. In a study on prediction of TCP throughput, Mizana et al. [START_REF] Mirza | A machine learning approach to tcp throughput prediction[END_REF] leverage Support Vector Regression (SVR) to predict the TCP throughput based on basic network characteristics, including available bandwidth, queuing delays, and packet loss.

Motivation and Main Contribution

The contributions of our work are summarized below:

• We present an in-depth analysis of the capabilities of Recurrent Neural Network (RNN) models in network traffic prediction.

• We evaluate our proposed models using real world network traffic.

• We compare the existing schemes to find their effectiveness for real-time applications. A detailed analysis of performance for three proposed models (LSTM, GRU, SVR).

• We propose to perform a Hyperparameter Tuning process to find out the optimal weights that provides accurate models. To the best of our knowledge, we are the first that tries to leverage the difference between RNN variants in network traffic prediction problem using Hyperparameter Tuning process.

Forecasting Models

Introduction

In this section, we present different forecasting models used to predict the volume of network traffic.

Support Vector Regression (SVR)

Definition

In 1997, a new regression model has been introduced by Vapnik called Support Vector Regression which represents the most common variant of SVM.

In fact, SVR attempts to minimize the generalization error bound in order to achieve good generalization performance as opposed to the traditional regression algorithms that aims to minimize the observed training error. Hence, it gives us the mobility of how much error is acceptable and find out the appropriate curve that fit the data.

More specifically, SVR is formulated as an optimization problem as shown in the equation below by defining a convex -insensitive loss function to be minimized and tune the margin in order to gain the desired accuracy of the trained model by finding the optimal tube that contains most of training samples.

min w 1 2 ||w|| 2 s.t. |y i -w i x i | ≤ , ∀i ∈ [1, N ] ≥ 0 (3.1)
The following figure illustrate the optimization problem that SVR is trying to solve:

Kernel SVR and Different Loss Functions

For nonlinear functions, where data could be mapped into higher dimensional space, standard SVR couldn't be applied due to its linearity. Thus, a kernelf unction was introduced that tries to make transformation from feature to kernel space. The problem is formulated in equation (3.2) :

min w 1 2 ||w|| 2 + C N i=1 ξ i + ξ i * , s.t. |y i -w T k(x i )| ≤ + ξ i , ∀i ∈ [1, N ] ξ i , ξ i * ≥ 0 (3.2)
Threfore, The following figure illustrate the nonlinear regression problem that SVR is trying to solve: Source: [START_REF] Awad | Efficient Learning Machines[END_REF] There are several types of kernel functions such as:

• Polynomial Kernel • Gaussian Kernel • Gaussian Radial Basis Function • Laplace RBF Kernel
We have chosen RBF kernel function because of its success with most of nonlinear regression problems and its wide use in the literature. In fact, the RBF kernel can be defined as follows:

k(x (i) , x (j) ) = exp( ||x (i) -x (j) || 2 2σ 2 ) (3.3)

Long-Short Term Memory (LSTM)

LSTM is one of the most well-known variant of RNN layers. It was introduced in 1997 by S. Hochreiter et al. [START_REF] Hochreiter | Long short-term memory[END_REF]. In fact, it contains computational blocks that control the flow of information within a cell. Computational blocks are called gates that selectively allow information to enter or quit the cell based on the acquired knowledge. As a result, if the information is not important, the input signal will be discarded from the LSTM unit. In contrast, the LSTM cell stores merely relevant information in its internal memory. Thus, LSTM extents the RNNs models with two major advantages: 1) Introduce memory information 2) Handle long sequences more better.

Looking at LSTM cell as black box in fig. 3.3, the input is a vector of combination of the input vector x t and the previous hidden state h t-1 , where the output at time t is denoted as the output vector y t , the actual hidden state h t . • Input Gate: The role of the input gate is to decide which information is to be renewed.

The input gate output is calculated using the weight matrix W i and the bias b i . The activation function used in this block is Sigmoid.

i t = σ(W i [h t-1 , x t ] + b i ) (3.4)
• Forget Gate: The role of the forget gate is to decide which information should be discarded.

f t = σ(W f [h t-1 , x t ] + b f ) (3.5)
• State Computation: The role of this block is to update the new memory state C t of the neuron. First, it calculate the possible combinations for the memory state:

Ĉt = tanh(W C [h t-1 , x t ] + b C ) (3.6)
Then, the new memory state is calculated using the estimated new memory state, the previous memory state multiplied by the output vector of the forget gate f t .

C t = f t C t-1 + i t Ĉt (3.7)
• Output Gate: Finally, the output gate computes the output vector for the given input signal. First, the output gate vector is computed using Sigmoid activation function:

ôt = σ(W o [h t-1 , x t ] + b o ) (3.8)
Then, the output vector is calculated using element wise multiplication between the output of the output gate and the activation signal of the actual memory state:

ŷt = o t tanh(Ct) (3.9)

Gate Recurrent Unit (GRU)

GRU is a variant of RNN layers. It was proposed by K. Cho in 2014 [START_REF] Cho | On the properties of neural machine translation: encoder-decoder approaches[END_REF]. It was designed to capture dependencies at different time scales in an adaptive manner. Unlike LSTM, this type of RNN neurons consists of only two gates and one memory state. Hence, it contains three computational blocks. As a result, GRU is becoming more computationally efficient and more promising for resource-constrained devices, as is our case. Fig. 3.4 illustrate in advance the architecture of the GRU neuron. • Reset Gate: The reset gate is used to decide whether to use the previous output or discard the input as the first symbol in a sequence. The reset output vector r t is calculated as follow:

r t = σ(W r [h t-1 , x t ]) (3.10)
Where: x t is the input vector, h t-1 is the hidden state output vector, W r is the weight matrix and σ is the Sigmoid activation function used for this gate.

• Update Gate: The update gate determines how much of the output is updated. The output of the update gate is calculated using the weight matrix W r :

z t = σ(W z [h t-1 , x t ]) (3.11)
• Output Gate: The role of this gate is to calculate the hidden state h t . Similarly to LSTM output gate, it computes the possible combinations of the hidden state ĥt :

ĥt = tanh(W o [r t h t-1 , x t ]) (3.12)
Then, the new hidden state is computed based on the possible combinations:

h t = (1 -z t ) h t-1 + z t ĥt (3.13)
According to [START_REF] Rezk | Recurrent Neural Networks: An Embedded Computing Perspective[END_REF], the number GRU operations and parameters is about 75% compared to LSTM neurons.

Experimental Methodology

In order to evaluate our forecasting models, we have used real network traces. In fact, we compare the performance of two variants of RNN with the baseline regression model SVR.

Hence, we start by presenting the software environment. Then, we turn our focus on exploratory data analysis and data preprocessing. Finally, we will discuss the modeling phase by including the performance evaluation of the fitted models.

Software Environment

Programming Language Python

Python has been the choice for Machine Learning and Artificial Intelligence developers and researchers for a long time. Hence, it contains the most advanced features and libraries that increase the productivity by abstracting complex algorithms and versatile workflows.

Libraries and Packages

Python comes with the support of a broad set of open source libraries:

• Numpy: Numpy is one of the most basic library in Python for multidimensional data manipulation.

• Matplotlib: Matplotlib is a library for producing visualizations in Python.

• Scikit-learn: Scikit-learn is a python Machine Learning library.

• TensorFlow: TensorFlow is a high-level library available for working with Deep Learning on Python.

Data Analysis

Data Collection

In order to implement a predictive model, there is always a mandatory step which is choosing the right data set for both training and evaluation. In fact, we have used a real traffic data taken from the University of Leipzig Internet access link [1]. A pair of DAG 3.2 cards were used to monitor network traffic from campus network to the German research network (G-WiN). A diagram of the configuration is illustrated in figure 3.5.

Figure 3.5: Network Configuration

After selecting the data source, we have chosen the traffic that is directed from the German research network to the campus network with a duration of 4 days and 18 hours. Thus, all outer connections pass through this measurement point.

In addition, all non-IP traffic has been removed, and only TCP, UDP and ICMP traffic are kept in the traces. Then, all IP addresses have been anonymised using one-to-one mapping into another address space.

Data Preprocessing

At first, we have converted the traces from compressed format into PCAP files using Libtrace tool [6] in order to make them readable . Thus, The collected raw data set consists of network packets in PCAP format. The network traffic captured 3429 million packets over 4 days and 18 hours. In fact, the storage of this traffic was about 1615 GB.

Hence, we processed the data in order to transform it into a time series that represents the length of packets at each minute. This resulted in a time series containing 6839 values as illustrated in the figure 3.6. This part of the process was done using Bash scripts in order to optimize the execution time.

Indeed, this data was sufficient to implement our network traffic prediction because we notice that it's about a periodic curve where the long term regularities are evident.

Data Normalization

Training of ML algorithms often performs better with standardized input data. In fact, normalizing features is not only essential for comparing features with different scales, it is a common prerequisite for many machine learning algorithms. For instance, in the absence of normalizing features for optimization algorithms, some weights may update faster than others as feature values plays a role in updating weights.

Hence, we apply the Gaussian transformation (i.e. subtracting the mean and dividing by the standard deviation) as mentioned in the equations above.

First, we need to calculate the mean:

µ = 1 n n i=1 x i (3.14)
Then, we calculate the standard deviation:

µ = 1 n n i=1 (x i -µ) 2 (3.15)
Finally, we calculate the normalized values as follows:

x i = x i -µ σ ∀i ∈ [1, N ] (3.16)

Data Transformation

Time Series prediction requires a set of past observations in order to estimate future values of the features of interest. Actually, there are mainly two categories as explained below:

• One-Step Prediction:

The most straightforward strategy is one step prediction which is about taking a vector that include the last observations as input to the model responsible for yielding the prediction. Specifically, for a set of samples observed at times t 0 , t 1 , ..., t n , the value at time t n+1 is estimated.

• Multi-Step Prediction:

Multi-Step Prediction is about predicting upper than one step ahead based on the same historical samples. Specifically, for a set of samples observed at times t 0 , t 1 , ..., t n , the values at time t n+1 , t n+2 , ..., t n+m are estimated.

For formalization, we denote:

-Step Back : The number of historical samples that will be put as input to the predictive model.

-Step Ahead: The number of future samples that will be predicted.

Indeed, Multi-Step Prediction is more challenging than One-Step Prediction due to the accumulation of errors and the lack of information needed to predict further samples in the future [START_REF] Hu | Multi-Step-Ahead Time Series Prediction using Multiple-Output Support Vector Regression[END_REF].

In order to overcome these issues, a multiple-input multiple-output (MIMO) strategy was proposed by Bontempi [START_REF] Bontempi | Long term time series prediction with multi-input multi-output local learning[END_REF] which consists of producing a vector of future predicted values instead of a single value. Therefore, this method uphold the stochastic dependency of the time series.

To wrap up, a sliding window concept is introduced to indicate a fixed number of previous samples to be learned in order to forecast the future time steps. Hence, for a given time slot t, the input vector contains the following values:

[X t-StepBack , ..., X t-2 , X t-1 ] (3.17)
On the other side, the output vector contain the following values:

[X t , X t+1 , ..., X t+StepAhead ] (3.18)

Modeling: Training and Validation

In this section, we will demonstrate how each model was fitted and evaluated.

At first, the data set was divided into two parts, as indicated below, using Test-Set Validation technique which is the most basic strategy for estimating the reliability and the robustness of the predictive models. Indeed, we split the data into two separate portions:

• Training Set: It contains 80% of the original data. Hence, this portion will be used for training the predictive models.

• Test Set: It contains 20% of the original data. Hence, the fitted models will be validated on the Test Set.

Then, we turn our focus on the model architecture of each forecasting model.

Recurrent Neural Network (RNN)

We have used two variants of RNN:

• Long Short Term Memory (LSTM)

• Gate Recurrent Unit (GRU)
The proposed architecture of the two RNN variants has the same number of layers as depicted below:

• Encoder: The encoder consists of a single layer containing 16 (LSTM / GRU) neurons. Indeed, it is responsible for encoding the time series to the state of the network.

• Decoder: The decoder consists of four consecutive layers where each layer contain by default 8 (LSTM / GRU) neurons. This will decode the encoding vector in order to generate the output sequence.

In order to set the optimal weights of the chosen RNN models and minimize the error rate between the actual output and the predicted sequence, a loss function is required. In the literature, the most used loss function for regression problems is Mean Squared Error (MSE). Therefore, it could be defined as follows:

M SE = 1 n N i=1 (Y i -Ŷi ) 2 (3.19)
where

• Y i represents the actual output.

• Ŷi represents the predicted output.

We have picked out RMSProp as an optimizer for updating the NN weights in the Gradient Descent algorithm. The selection was actually based on a rigorous benchmark of three optimizers (SGD, RMSProp and AdaGrad).

Then, The proposed models were trained on the training set with the following Hyperparameters (Randomly Chosen):

• Time Granularity: 1 minute.

• Step Back: 60

• Step Ahead: 20

• Encoder: Composed of one single (LSTM/GRU) layer that contains 16 neurons.

• Decoder: Composed of four (LSTM/GRU) layers that contains respectively 8, 8, 16, 16 neurons.

• Epochs: 1000

• Batch Size: 10

• Loss Function: Mean Squared Error (MSE)

• Optimizer: RMSProp At this stage, the model is ready for training and evaluation phase. For evalutation purpose, the mean absolute percentage error (MAPE) is used as the evaluation indicator of the proposed models:

M AP E = 1 N N i=1 | Y i -Ŷi Y i | × 100% (3.20)
Indeed, we have found accurate results from our primary test. The mean absolute percentage errors of the two RNN models on the test set are shown in the table 3 Based on this primary run, it appears that GRU model outperform the LSTM model in the test subset. An interesting finding that stimulates a study of the fine tuning process in order to determine the most reliable model for network traffic prediction task.

Support Vector Regression (SVR)

We have chosen SVR model as the baseline for the comparison of performance between DL models and classical ML models. Hence, the same metric function was used (MAPE). As far as the Hyper Parameters is concerned, we have set the default values as illustrated below:

• Time Granularity: 1 minute. Based on the default configuration, we observe that GRU model outperforms the SVR model. Therefore, the primary results represents a blurry picture. Hence, a fine tuning process is required to pick out the most accurate model.

Fine Tuning

Throughout this section, we will try to refine the hyper parameters of the forecasting models in order to improve the accuracy. Indeed, we conducted our optimization using Grid Search approach. This latter performs every conceivable combination of the Hyper Parameters. As a result, it necessitate a massive computation in order to complete the task.

Hence, we define a grid of n dimensions where each dimension represents a Hyper Parameter. As a consequence, we search for all the possible configurations. As we can see, with more dimensions, the more time-consuming the search will be. In our case, we have around 6500 combinations for the Hyper Parameters. This means that we have to train the predictive models 6500 times.

In order to compensate this huge computation, we have run the Fine Tuning process on the LAAS computing platform. This latter contains 228 cores and 1.6 TB of RAM spread over 7 machines. This cluster is primarily designed for distributed computing. Hence, all the tasks are executed using the resource manager SLURM [START_REF]Slurm Workload Manager -Documentation[END_REF].

Thanks to this platform, the Hyper Parameters Tuning process took about 24 hours to complete. Hence, we have found interesting insights as illustrated in the table 3 In the first rows, we find that GRU model has achieved the best MAPE metric for around 3%. Then, the first LSTM model is ranked 16 with a MAPE equals to 3.18%. The baseline model SVR has achieved a MAPE equals to 6.06% as the highest result from all the possible combinations and it is ranked 4281 in the Fine Tuning process.

A closer look at the variations of the RNN models demonstrate that GRU has better performance over LSTM as illustrated in the figure 3.9 below: In conclusion, GRU has better performance than both of LSTM and SVR on the network traffic prediction problem. This result couldn't be observed unless the Fine Tuning process. Hence, it shows the importance of this phase.

Conclusion

In this work, we reviewed the current state of the art for DL on the network traffic prediction. We have shown that GRU is well suited for network traffic prediction. We have proposed a data preprocessing, RNN feeding technique that achieves high predictions within few milliseconds of computation. The fine tuning process has shown that GRU outperforms classical ML algorithm and the LSTM model with several orders of magintude.

For future work, network traffic prediction leads to many interesting applications such as QoS management, resource allocation, and anomaly detection. These applications serve better quality of Internet service and enhance the actual network infrastructure. Therefore, we believe that network traffic prediction represents a major key to provide minimum latency for real-time applications (e.g. augmented reality, autonomous vehicles).

Chapter 4 Distributed DL Inference in EC scenario

Introduction

We start this chapter by presenting the general scope of the project. Then, we turn our focus on the deployment of DL task at the edge. Finally, we tackle the modelling phase by including the performance evaluation of the chosen models.

General Description

Introduction

Edge Computing has become an important solution to break the bottleneck of emerging technologies by virtue of its advantages in reducing data transmission, improving service latency and easing cloud computing pressure. The edge computing architecture will become an important complement to the cloud, even replacement the role of the cloud in some scenarios. Indeed, the architecture of Edge Computing could be organized as follows:

• End Device: refers to mobile edge devices (e.g. smartphones, smart vehicles, etc) and IoT devices (e.g. sensors, cameras, accelerators, etc) .

• Edge Device: refers to the edge servers, MEC servers, and generally servers deployed at the edge of the network.

• Cloud: represents the on-demand availability of computer system resources, especially data storage and computing power. While cloud computing is created for processing computation-intensive tasks, such as DL, it cannot guarantee the delay requirements throughout the entire process from data generation to transmission to execution. On the other side, independent processing on end or edge devices is limited by their computing capability. Therefore, collaborative end-edge-cloud computing for DL is the most suitable solution to avoid latency while satisfying the required precision.

Problem Statement

Deep neural networks are the state of the art methods for many learning tasks thanks to their potential to derive better features at each network layer. However, the increased efficiency of additional layers in a deep neural network comes at the cost of additional latency and energy consumption in feed forward inference. Thus, it becomes more challenging to deploy them in the edge with limited resources. Therefore, large-scale DL models are generally deployed in the cloud while end devices merely send input data to the cloud and then wait for the DL inference results. Specifically, it cannot guarantee the delay requirement for real-time services such as real-time object recognition with strict demands.

To address these issues, DL applications tend to resort to edge computing. In fact, the use of optimization techniques, distributed DNNs and collaborative inference between IoT devices

Related Work

End devices (e.g. mobile phones, sensors) were unable to support the computational costs of DL inference due to their limited resources. To address these issues, a lot of research is underway to optimize the inference performance on IoT devices [START_REF] Cui | A survey on application of machine learning for Internet of Things[END_REF][27] [START_REF] Li | Learning IoT in edge : Deep learning for the Internet of Things with edge computing[END_REF] . The first common approach is the optimization of DL models at the edge. For instance, network compression is one of the optimization directions that aims to compress deep networks into tiny models that achieves a slightly reduced level of accuracy and alleviate considerably the amount of computation and the inference time [START_REF] Bucilu | Model compression[END_REF]. In addition, industrial companies are trying to produce custom edge hardware that speed up the inference of DL models. Indeed, Coral [START_REF][END_REF], a company supported by both Google and TensorFlow, provides custom hardware components devoted for resource-constrained devices that allow fast ML inference. A second branch of study focuses on segmenting DL models by pushing portion of the computation from cloud to the edge. Based on the network bandwidth, the energy state of IoT devices, and the delay requirements of running applications, the system will decide how much of the computation will be processed on the cloud server. Actually, it could be classified mainly into two categories 1) Model parallelism 2) Data parallelism. The former tries to distribute the computation among edge nodes by slicing the DL model into partitions. In fact, Neurosurgeon [START_REF] Kang | Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge[END_REF] intelligently decides where to split the DNN based on preliminary constraints (e.g. network bandwidth, device energy state, delay requirements). In addition, IONN [START_REF] Jeong | IONN: Incremental Offloading of Neural Network Computations from Mobile Devices to Edge Servers[END_REF] propose an incremental offloading system which uploads the DL model to the edge server sequentially. It allows the IoT device to offload partial DNN execution even before the uploading finishes. Furthermore, S. Teerapittayanon et al. [START_REF] Teerapittayanon | BranchyNet: Fast inference via early exiting from deep neural networks[END_REF] propose to multiply the output exits by adding secondary branches. The latter could achieve primarily inference result with slightly lower performance so that they could gain in delay response and energy consumption. In [START_REF] Zhao | DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge Clusters[END_REF], there is another line of work focuses on splitting the input data and dividing CNN layers into independent tasks for distribution across edge devices in order to unburden the computation.

Motivation and Main Contribution

The main contributions of this work is represented as follows:

• Propose an EC scenario for Early Exit of Inference (EEoI) Model.

• Extensive Simulations using virtualization technique show that our distributed framework achieves a better trade-off between execution delay and model performance.

Proposed Work

Model Architecture

In order to reduce the computation time of DL inference, we add early exits branches at different stages of the network. Hence, this latter enable the inference to exit early from these additional branches based on a confidence criteria. For instance, as depicted in figure 4.2, an edge device could give primary inference results at an early stage if the confidence criteria is satisfied. Otherwise, further computation should take place on the cloud or on the edge server. Actually, EEoI mechanism has been proposed by Branchynet [START_REF] Teerapittayanon | BranchyNet: Fast inference via early exiting from deep neural networks[END_REF]. In fact, this latter is composed of one single input, and several exit points. The last output could be considered as the principal network output from which intermediate outputs are added.

Training Phase

This sort of models is barely considered as a neural network models where both Forward Propagation and Backward Propagation algorithms are still applicable.

For a classification problem, one of the most used loss function is Softmax cross entropy. In fact, it could be expressed as the following:

L(ŷ, y, θ) = - 1 |C| c∈C y c log ŷc (4.1)
where:

• x: represents the input data.

• y: represents the actual output.

• ŷ: represents the predicted output.

• θ: represents the models parameters including the multiple exit points.

• C: represents the set of classes for the classification problem

The expression of the predicted output for a given ext point n is written as follows:

ŷexitn = Sof tmax(z exitn ) = exp(z exitn ) c∈C exp(z c ) (4.2)
where:

z exitn = f exitn (x; θ)
As a result, each exit point has its corresponding loss function. In order to evaluate the entire EEoI model, we form a joint optimization problem as weighted sum of the loss functions of each exit point as demonstrated below:

L Branchynet (ŷ, y, θ) = N n=1 w n L(ŷ exitn , y, θ) (4.3)
The training algorithm is composed of:

• Forward Propagation: The data will flow from input layer into the multiple branches in order to calculate the loss function written in Eq. 4.3.

• Backward Propagation: The back-propagation algorithm computes the partial derivatives of the loss function with respect to all the weights in the EEoI model (including the intermediate exit points) and then go back through the whole network in order to adjust the model parameters using gradient descent.

Fast Inference using EEoI

The proposed model is ready for making predictions the moment we make the necessary steps of modeling and training. Therefore, the classifier starts by making predictions from the earliest exit point until reaching the main output layer. Once the model predict an output that has less confidence than a given threshold, the former will pass to the next exit point. This results in more computation and longer runtime.

The policy which measures the degree of confidence is the entropy. This latter is actually a measurement of the disorder in a given system. As a result, if the value of the entropy for a

• Exit 2: The second branch has the medium amount of computation because it's located in the middle i.e. it has the average number of layers. Hence, we assigned this exit point to the edge server. Indeed, this connection is well adapted because the ES has an average of computation resources.

• Exit 3: The main branch which has the maximum number of layers, automatically needs the most computation resources because of the massive processing it produces. Hence, we assign this exit point to the Cloud.

Distributed Workflow

During inference, the predictions will be made by the earlier exits until the main branch (last exit point) is reached. As a result, the forecast will be returned by the last exit in the worst case. To deal with the communication between different nodes, we employ a distributed workflow to take charge of the message activity and the synchronization as depicted in figure 4.4.

Simulations and Results

Modeling: EEoI Architecture

To manifest the performance of the proposed work, we have conducted several simulations using the standard CNN model "ResNet 20" [START_REF] He | Deep residual learning for image recognition[END_REF] on CIFAR10 dataset [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] for object recognition problem.

As the name is indicated, the ResNet20 model consists of 20 convolutional layers. Indeed, the location of the exit points in the proposed model is not yet automated. Hence, we should make several experiments as shown in table 4.1 in order to find out the best positions for the secondary branches. One interesting finding is that the locations of the exit points influence on the model accuracy. As far as data is concerned, CIFAR10 dataset consists of 60000 labeled images for 10 classes.

The former is composed of 50000 samples for training and 10000 samples for test. Explicitly, the classes are:

• Airplane, Automobile, Bird, Cat, Deer, Dog, Frog, Horse, Ship, Truck.

The input of EEoI model is a color image with a shape of: [START_REF]Mobile-Edge Computing Introductory Technical White Paper[END_REF][START_REF]Mobile-Edge Computing Introductory Technical White Paper[END_REF][START_REF] Amini | Deep Reinforcement Learning[END_REF].

The output of EEoI model is an embedding vector that contains the probabilities for each class with a size of: (1, 10).

Training and Evaluation of the EEoI model

Training the proposed models is carried out by Python Programming Language and Tensor-Flow 2.3 library. The hyper-parameters that defines the training phase of the proposed models are presented as follows:

• Epochs : 110 The training process was done on Google Colab platform using free access GPU card "Tesla T4". Indeed, this latter performs inference of ResNet50 model 27 times more faster than CPU [START_REF]GPU Nividia Tesla T4 à coeurs Tensor pour l'inférence IA | Nividia Data Center[END_REF]. As a result, the training of each DL model took about 1 hour and 18 minutes which is considerably fast.

Results have shown that the deeper we make the secondary branches, the better the accuracy of the entire EEoI model. In fact, the table 4 4.5b depicts the evolution of the accuracy on the test set. Indeed, we apparently notice the differentiation between the exit points in terms of accuracy during the training process. Hence, we deduce that the more layers we add , the better performance we achieve. In addition, we notice that the emplacement of the branches in the EEoI model affects the total performance. For instance, ResNet20-B achieved a better accuracy than ResNet20-A at the three exit points. Hence, there is a deeper connection between the location of the exit points and the model performance which must be given more attention by studying its behaviour. In contrast, we should always take into account of the trade-off between model performance and the latency requirements of real-time applications in an Edge Computing environment.

Distributed Inference Implementation

In order to implement the EC scenario of the EEoI model as illustrated in section 4.2.4, we need to implement the three-tier architecture i.e. cloud servers, edge servers and edge devices. Hence, due the COVID-19 Coronavirus Pandemic, we decide to make a virtualization technique instead of a real testbed by providing a set of virtual nodes built on the top of the workstation that make up the three-tier architecture.

In fact, we have implemented the scenario using "Docker" open source platform [START_REF] Turnbull | The Docker Book: Containerization is the new virtualization[END_REF] to containerize the applications on virtual nodes. Thanks to this latter, we could create containers that process application packaging to deliver edge computing applications to the network. Indeed, a container is a running process that embodies additional functionalities to keep it separated from the host and the rest of containers. In addition, Docker provide ways to control the runtime options i.e. Memory, CPU. As a result of that, we have built three containers that represents the EC environment and we have set the runtime options as shown in table 4 As far as the host machine is concerned, it is about a laptop that is equipped with 32 GB of RAM and with Intel Processor 3.4 GHz, and runs on Ubuntu 18.04. So that to limit the resources for the edge device known to be a resource-constrained device (e.g. sensors, cameras, etc.), we have only varied the RAM memory from 512 MB to 3 GB in order to perceive the variation in latency of the applications. In contrast, the docker containers could consume from the host machine as much as they could in terms of CPU cycles. In fact, the variation of CPU cycles for Docker containers induce an abnormal behaviour. This is due to intervention of the operating system which consume likewise from the CPU. As a consequence, the virtualization becomes inaccurate.

The last brick in this implementation is the communication part between the nodes. Indeed, Docker provides a powerful feature that enables the connection between different containers using a specific network. In our case, we have used the bridge network provided by Docker as depicted in figure 4.6. The latter uses a software bridge network, which is a link layer that dispatch traffic between network segments, in order to enable communication between different Docker containers. Thus, we have used ZeroMQ [START_REF]ZeroMQ[END_REF] to pass messages and synchronizations between different Docker containers using the bridge network channels. In fact, ZeroMQ is an asynchronous messaging library that focuses on developing distributed and concurrent applications. Furthermore, we have used TCP transport layer for data transmission. Furthermore, we have varied the RAM memory for edge device from 512MB to 3GB in order to observe the impact of constraining the resources on the inference time. Indeed, Fig. 4.10 illustrates in advance the increase of the execution time while we are limiting the resources for the edge device. Thus, we should be aware of the consequences on the latency requirement for real-time applications when dealing with resource-constrained devices known as IoT devices. In fact, the modeling of the EEoI model, such as the selection of optimized Hyper-Parameters using automated algorithms, becomes the most important element in order to obtain the welladapted distributed inference model for real-time applications (e.g. AR applications).

Conclusion

For an EC environment, we present a distributed inference of DL models. Preliminary implementation and evaluations shows that the EEoI framework manages to control the trade-off between model performance and execution latency by varying the entropy threshold. Interesting findings demonstrate that the more constrained-resource device we have, the more we should be sensible about the DL model size, as well as the location of secondary exit points.

Future works reveals an implementation of real testbed in order to observe the influence of varying the network bandwidth between edge devices and the cloud on the inference latency.

Chapter 5

Computation Offloading for multi-user Mobile EC

General Description

Problem Statement

The rapid growth of mobile internet services has led to a wide variety of computation intensive applications (e.g. augmented reality, object recognition). Indeed, there are sensible applications that require a short latency for the execution. On the other hand, end devices such as mobile users or IoT devices have limited energy resources. As a result, they cannot handle this enormous amount of processing locally. Thus, Edge Computing attempts to resolve these issues of delay requirements and energy consumption by bringing the cloud capabilities closer to the end user by deploying edge servers. It is in this line of research, that we aim to implement an efficient computation offloading and resource allocation for mobile edge computing.

Related Work

Sensible decisions should be made whether offload the task to a near edge server or perform the computation locally. Hence, several works have been done in this branch of research. For instance, L. Huang et al. [START_REF] Huang | Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks[END_REF] propose a system that optimizes the offloading decisions and the resource allocation in an edge computing environment. In fact, they optimize offloading decisions using Deep Reinforcement Learning (DRL) while adopting convex optimization for the allocation of network resources. Furthermore, X. Feng et al. [START_REF] Huang | Deep reinforcement learning-based joint task offloading and bandwidth allocation for multi-user mobile edge computing[END_REF] presents a joint task offloading and resource allocation optimization method in EC environment using DQN. The overall offloading cost is evaluated based on energy consumption of end devices and the delay of tasks execution. In addition, X. Liu et al. [START_REF] Liu | Resource allocation for edge computing in IoT networks via reinforcement learning[END_REF] propose to leverage a distributed Q-learning algorithm by allowing each end device to control the offloading decision on its own. Based on the channel state and the remaining resources, the former tries to optimize the energy consumption and the execution delay of the remaining tasks.

Motivation and Main Contribution

In the previous works, the implementation of a real testbed or virtual simulation has rarely been considered for offloading decision and resource allocation optimization in EC environment. Recently, P. Gawłowicz et al. [START_REF] Gawłowicz | ns-3 meets OpenAI Gym: The Playground for Machine Learning in Networking Research[END_REF] is presenting a new framework ns3-gym for RL research in networking field by interconnecting the network simulator ns-3 and the RL toolkit "OpenAI Gym" together. Thus, ns3-gym becomes the benchmark for research related to applications of RL in networking. For that occasion, we propose to leverage offloading task optimization using DRL paradigm by exploiting the new framework ns3-gym.

The major contributions of this work are:

• Proposing a joint optimization problem for offloading decision and resource allocation of edge devices in a wireless EC environment.

• Solving the optimization problem using Deep Q-Network (DQN).

• Simulation of the EC scenario using ns-3 network simulator. In fact, the RL agent interacts in real-time with ns-3 environment using the framework ns3-gym.

Scenario Description

System Model

As illustrated in fig. 5.1, the scenario consists of an IoT network with N end devices connected to an Access Point (AP) via wireless communications. The AP aggregates the data from end devices and processes it using its equipped edge server. Each end device has M independent tasks to be completed. As a result, offloading their tasks could improve the computation experience in terms of task execution latency and energy consumption. To model the task, we use S in nm to denote the input data size of the m th task for the n th user. Likewise, we denote S out nm as the output data size of the m th task for the n th user. Furthermore, each end device n could decide whether to offload or execute locally a particular task m. As a result, we denote by x nm the binary decision for offloading or local execution. For instance, the offloading occurs when x nm is set to 1. In contrast, the task will be executed locally if x nm has been set to 0.

When end devices offload their tasks to the edge server. Energy consumption includes two parts for both transmission and receiving. Hence, we denote by E t nm and E r nm respectively, the energy consumption for both transmitting and receiving. The total energy consumed for end device n when offloading the task m is denoted by E of f nm . On the other side, E loc nm represents the energy consumed by the end device n while performing the task m locally.

As far as the delay is concerned, we denote by C up n and C do n respectively, the allocated bandwidth for user n for transmission and receiving the task m. As a result, the uplink transmission delay for offloading could be expressed as follows T up nm = S in nm C up n . Contrariwise, the downlink transmission delay for local execution could be expressed as follows: T do nm = S out nm C do n . In addition, the computation latency at the edge server is denoted by T edge nm . As a result, the total delay for offloading the task m for the user n could be formalized as follow: T of f nm = x nm (T edge nm + T up nm + T do nm ). In contrast, we denote by T loc nm the computation delay for executing the task n for the user m.

Optimization Problem

The problem is formulated with the aim of minimizing both the total delay and the energy consumption of end devices in order to complete the execution of the tasks, by jointly optimizing the offloading decisions for each user n and the allocated bandwidth for transmission and reception.

In this work, we propose to design an optimal task offloading scheme to minimize the task execution latency and the energy consumption of end devices. Hence, the optimization problem is formulated as follow:

min w N n=1 [ M m=1 [E loc nm (1 -x nm ) + E of f nm x nm + w n (x nm T of f nm + (1 -x nm )T loc nm )]] s.t. N n=1 C up n ≤ C up,max N n=1 C do n ≤ C do,max C up n , C do n ≥ 0, ∀n x nm ∈ {0, 1} (5.1)
The parameter w n represents the weight between energy consumption and the execution delay. In addition, the constraints of the optimization problem describe the limitations of both the binary decision variables and the allocated bandwidths for transmission and reception.

In fact, we notice that the proposed optimization problem consists of a mixed non linear programming problem, which is typically an NP-hard problem.

RL Approach

Motivation

The aim of this work is to optimize the joint task offloading 5.1. Since the DRL have shown high efficiency in solving complex problems, it would be interesting to solve the optimization problem with DRL especially when DRL algorithms provides a promising solution to handle complicated decision-making process. Specifically, we have inspired by the success of modelling an optimal action-state Q-function with Deep Neural Network (DNN). As a result, we adopt a Deep Q-Network (DQN) algorithm to address this optimization problem. Observe the reward r(t) and the next state s(t + 1) ; Calculate the target Q-value Q target : Q target = r(t) + γ max a Q pred (s(t + 1), a , w ) ; Perform Gradient Descent step to minimize (Q target -Q pred (s(t), a(t), w)) 2 ; Update the weights for the evaluation network; end end

Simulations and Results

Software Environment

• ns-3 Network Simulator: ns-3 is a discrete-event network simulator for networking systems, developed primarily for research and educational use [START_REF]ns-3 source code[END_REF].

It is an open source simulator developed in C++ using object oriented programming. In essence, ns-3 attempts to mimic the reality as closely as possible. It actually uses several core concepts and abstractions that map efficiently the way computers and network are designed. For instance, a Node is a key entity connected to the network. It is the a container for applications, protocols and network devices. An application represent a user program that generates packet flows. In addition, a protocol represents a logic of network and transport level protocols e.g. TCP, OpenFlow. A network device is an entity connected to the channel that is the basic communication sub-network abstraction e.g. Wifi, CSMA, LTE, etc. Furthermore, ns-3 allows to model the energy consumption and the mobility of different types of Node entity.

• OpenAI Gym: OpenAI Gym [START_REF]OpenAI Gym documentation[END_REF] is a framework for developing and comparing reinforcement learning algorithms. It provides simple API that unifies interactions between RL and the environment. Specifically, the main purpose of the Gym framework is to provide a standardized interface that allows access to the state and to execute actions in the environment.

Therefore, we have used ns-3 to simulate the environment while Gym was used to implement the DQN agent. In addition, we resort to a recent work [START_REF] Gawłowicz | ns-3 meets OpenAI Gym: The Playground for Machine Learning in Networking Research[END_REF] which enables real-time communication between ns-3 and Gym framework using ZMQ sockets [START_REF]ZeroMQ[END_REF].

Implementation Details

In order to simulate the EC scenario, we have chosen the following characteristics of ns-3 environment:

• Communication Between Access Point (AP) and End Devices: We have chosen the communication to be a wireless communication. Specifically, we have used Wifi communication.

• Communication Between AP and Edge Server: We have chosen a wired communication named Point-to-Point Communication in ns-3 simulator.

In our primary simulation, we set the number of end devices N = 6 and the number of tasks for each end device M = 2. In addition, to simplify the simulation and avoid the ambiguity in studying Offloading tasks in EC environment, we have made some assumptions and simplifications related to the system parameters:

• The cost function J becomes only a function of the execution time of tasks.

• We assume that the allocated bandwidth for both transmission and reception are fixed variables. Hence, we remove the allocated bandwidth from the action vector.

• In order to simulate the execution of tasks for end devices, we decided to send a fixed number of packets to local nodes as if it is a computation.

• We suppose that the local execution is less heavier than offloading to the edge server.

Performance Analysis

In fig. 5.2, we plot different episodes during the training of the agent. The results shows that the agent could not minimize the cost function. In fact, we observe that the cost function oscillates at a local minimum. This primary simulation motivates us to carry out further implementations in order to explain the behaviour of the DQN agent. In the second run, we have tried to change the reward function. In fact, the reward function represents the signal which the agent receives while performing an action in a given sate. As a result, the reward function plays a key role for the agent to understand the environment. Therefore, we have modified the reward function as follows:

if J s(t+1) (t + 1) < J s(t) (t) then r (s(t),a(t),s(t+1)) (t) = 1 else if J s(t+1) (t + 1) > J s(t) (t) then r (s(t),a(t),s(t+1)) (t) = -1 else if J s(t+1) (t + 1) = J s(t) (t) and J s(t) (t) < then r (s(t),a(t),s(t+1)) (t) = +2 else if J s(t+1) (t + 1) = J s(t) (t) and J s(t) (t) > then r (s(t),a(t),s(t+1)) (t) = -2 end if

In order to prevent the agent from falling into local minimums as if it was the case in 5.2, we assign a negative reward when the cost function stabilizes at a local minimum. In contrast, we send a positive reward to the agent in case we have a stabilization in a global minimum. Fig. 5.3 illustrate the impact of the reward function modification on the convergence performance of the agent. Indeed, we observe that the DQN agent has achieved a good performance in minimizing the cost function. In episode 9, we notice that the agent is trying to stabilize the cost function at the global minimum until he reaches it in the episode 19. Hence, we deduce an interesting finding from these simulations that the DQN components, i.e. state, action, reward function, have an enormous impact on the agent performance.

Conclusion

In this work, we investigate DRL for task offloading and bandwidth allocation using multiple simulations on ns-3 network simulator. We formulate an optimization problem with the objective of minimizing the overall cost, including the execution delay of the running applications and the energy consumption of end devices. Primary results stimulate advanced simulations by using more features, i.e adding the energy model to end devices, implementing real tasks instead of sending packets.

Conclusion

This work presents three relevant applications that serve end devices with better computing power, energy consumption and minimum latency for sensitive real-time applications.

First, network traffic prediction is the engine for many interesting applications in networking point of view such as resource allocation and QoS management. In fact, Recurrent Neural Network (RNN) have shown high capability to forecast network traffic with a promising execution time.

Second, we have leveraged Edge Computing paradigm which works in tandem with Cloud Computing to deliver real-time applications for end devices with minimum latency and efficient energy consumption. In fact, we have proposed a distributed DL inference that controls the trade-off between the accuracy of the prediction and execution latency by deciding where the execution will take place. Actually, the model performance and the latency of application are two key knobs for an efficient DL inference at the edge. This line of research is still ongoing. For instance, a possible enhancement for the proposed framework is to intervene reinforcement learning in order to automate the control of these key knobs (i.e. DL inference accuracy and the multiple constraints of networking, communication and energy consumption).

Finally, we have investigated Deep Reinforcement Learning (DRL) for optimizing offloading decision and resource allocation for multiple mobile end users. The offloading decisions have been taken centrally. Therefore, future works reveals the need to extend this framework by implementing a distributed framework by allowing each end device to take the offloading decision on its own instead of a centralized way. Additionally, more complex scenarios could be designed by adding features such as multiplying the number of edge servers. As a result, the mobile user encounter the decision of which edge server he chooses in order to perform his tasks.

Figure 1 :

 1 Figure 1: The number of IoT connected devices from 2015 to 2025[START_REF]The Statistics Portal. Internet of things (iot): number of connecteddevices worldwide from 2012 to 2020[END_REF]

•

  Networks and Communications • Robotics • Decision and Optimization • Microwaves and Optics : from Electromagnetism to Systems • Technology & Instrumentation for the Monitoring of Complex Systems • Micro Nano Bio Technologies • Energy Management During my internship, I have worked in SARA team (Services et Architectures pour Réseaux

Figure 2 . 1 :

 21 Figure 2.1: Types of ML problems [3]

Figure 2 . 2 :

 22 Figure 2.2: ANN Architecture [37]

Figure 2 . 3 :

 23 Figure 2.3: Propagation Function for a Single Neuron

Figure 2 . 4 :

 24 Figure 2.4: Convolutional Layer Example[START_REF] Bansal | 3D Convolutions : Understanding + Use Case[END_REF]

Figure 2 . 5 :

 25 Figure 2.5: Example of Pooling Layers

Figure 2 . 6 :

 26 Figure 2.6: Example of a Simple RNN

Figure 2 .

 2 Figure 2.7 depicts a frequent form of visualization for RNNs known as unfolding. Indeed, an RNN is unfolded by developing its computational graph over time and removing the loop connections. Thus, RNN unfolding reveal the repetition in the recurrent neuron and show the number of time steps required to complete a task.

Figure 2 . 7 :

 27 Figure 2.7: Unfolding of RNN neuron

Figure 2 . 8 :

 28 Figure 2.8: Interaction Between Agent and Environment

Figure 2 . 9 :

 29 Figure 2.9: Two Possible Configurations of DQN

Figure 3 . 1 :

 31 Figure 3.1: One-Dimensional Linear SVR[START_REF] Lewandowski | Time Series Analysis and Prediction[END_REF]

Figure 3 . 2 :

 32 Figure 3.2: Non-Linear Regression

Figure 3 .

 3 Figure 3.3: LSTM Cell

Figure 3 .

 3 Figure 3.4: GRU Cell

Figure 3 .

 3 Figure 3.6: Time Series of the Network Traffic Volume for about 5 days

Figure 3 . 7 :

 37 Figure 3.7: Difference of the Test Result Between two RNN variants

• Step Back: 60 • Step Ahead: 20 •

 6020 Kernel Function: RBF • C: 1.0 • ξ: 0.01 Then, the SVR baseline model was trained with the default parameters. Hence, the predictions are made on the test subset. The MAPE was 7.53% for One-Step prediction as illustrated in the figure.

Figure 3 . 8 :

 38 Figure 3.8: Prediction Result of SVR model

Figure 3 . 9 :

 39 Figure 3.9: Comparison Between two variations of RNN

Figure 4 . 1 :

 41 Figure 4.1: Edge Computing Hierarchy[START_REF] Wang | Convergence of Edge Computing and Deep Learning: A Comprehensive Survey[END_REF]

Figure 4 . 2 :

 42 Figure 4.2: Example of Early Exit Mechanism for DNN

Figure 4 . 3 :

 43 Figure 4.3: EEoI in an EC scenario[START_REF] Wang | Convergence of Edge Computing and Deep Learning: A Comprehensive Survey[END_REF]

Figure 4 . 4 :

 44 Figure 4.4: Message Activity at the Edge

Table 4 . 1 :

 41 Different Combinations for the location of ResNet Exits

• Batch Size : 32 •

 32 Loss Function : Mean Squared Error (MSE) • Optimizer: Adam • Loss Function : Categorical Crossentropy • Metric : Accuracy

Figure 4 . 5 :

 45 Figure 4.5: Curve of the Accuracy on the Test Set

  Figures 4.5a, 4.5b depicts the evolution of the accuracy on the test set. Indeed, we apparently notice the differentiation between the exit points in terms of accuracy during the training process. Hence, we deduce that the more layers we add , the better performance we achieve. In addition, we notice that the emplacement of the branches in the EEoI model affects the total performance. For instance, ResNet20-B achieved a better accuracy than ResNet20-A at the three exit points. Hence, there is a deeper connection between the location of the exit points and the model performance which must be given more attention by studying its behaviour. In contrast, we should always take into account of the trade-off between model performance and the latency requirements of real-time applications in an Edge Computing environment.

Figure 4 .

 4 Figure 4.6: EC Virtualization using Docker Platform

Figure 4 . 8 :

 48 Figure 4.8: Effect of The Threshold on the Accuracy

Figure 4 . 9 :

 49 Figure 4.9: Effect of the The Threshold on the Exited Samples

Figure 4 . 10 :

 410 Figure 4.10: Effect of The Resource Limitation on the Model Runtime Under Different Threshold

Figure 5 .

 5 Figure 5.1: System Model

Figure 5 . 2 :

 52 Figure 5.2: Convergence Performance

Figure 5 . 3 :

 53 Figure 5.3: Convergence Performance using a second Reward Function

  Introduction 1 General Framework of the Project 2.4.2 RL Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.3 Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.4 Value Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.5 Q-Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.6 Exploration vs. Exploitation Trade-Off . . . . . . . . . . . . . . . . . . . 2.5 Deep Reinforcement Learning (DRL) . . . . . . . . . . . . . . . . . . . . . . . . 2.5.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.2 Deep Q-Network (DQN) . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . General Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Project Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.3 Categories of Network Traffic Prediction . . . . . . . . . . . . . . . . . . .2.3 Long-Short Term Memory (LSTM) . . . . . . . . . . . . . . . . . . . . . 3.2.4 Gate Recurrent Unit (GRU) . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Experimental Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3.2 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data Normalization . . . . . . . . . . . . . . . . . . . . . . . . 3.3.3 Data Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Modeling: Training and Validation . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Recurrent Neural Network (RNN) . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Support Vector Regression (SVR) . . . . . . . . . . . . . . . . . . . . . . 3.5 Fine Tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Motivation and Main Contribution . . . . . . . . . . . . . . . . . . . . . 4.2 Proposed Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.1 Model Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Training Phase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 Fast Inference using EEoI . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.4 Edge Computing Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.5 Distributed Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Simulations and Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Modeling: EEoI Architecture . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Training and Evaluation of the EEoI model . . . . . . . . . . . . . . . . 4.3.3 Distributed Inference Implementation . . . . . . . . . . . . . . . . . . . . 4.3.4 Performance Analysis of the distributed Inference . . . . . . . . . . . . . 4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3.3 DQN Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4 Simulations and Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Types of ML problems [3] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 ANN Architecture [37] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Propagation Function for a Single Neuron . . . . . . . . . . . . . . . . . . . . . 2.4 Convolutional Layer Example [39] . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Example of Pooling Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Effect of The Threshold on the Accuracy . . . . . . . . . . . . . . . . . . . . . . 4.9 Effect of the The Threshold on the Exited Samples . . . . . . . . . . . . . . . . 4.10 Effect of The Resource Limitation on the Model Runtime Under Different Threshold . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Examples of Activation Functions . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Comparison of Results Between two variants of RNN . . . . . . . . . . . . . . . 3.2 Configuration of the Hyper Parameters for Grid Search . . . . . . . . . . . . . . 3.3 Results of the Fine Tuning Process . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Different Combinations for the location of ResNet Exits . . . . . . . . . . . . . .

	List of Figures
	1	The number of IoT connected devices from 2015 to 2025 [29] . . . . . . . . . . .
	3 Network Traffic Prediction 2.1
	3.1 Conclusion
	Bibliography
		v

3.1.4 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.5 Motivation and Main Contribution . . . . . . . . . . . . . . . . . . . . . 3.2 Forecasting Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Support Vector Regression (SVR) . . . . . . . . . . . . . . . . . . . . . . 3.2.2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2.2 Kernel SVR and Different Loss Functions . . . . . . . . . . . . 33.3.1 Software Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1.1 Programming Language Python . . . . . . . . . . . . . . . . . . 3.3.1.2 Libraries and Packages . . . . . . . . . . . . . . . . . . . . . . . 33.3.2.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2.2 Data Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.2.3 4 Distributed DL Inference in EC scenario 4.1 General Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.4 5 Computation Offloading for multi-user Mobile EC 5.1 General Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.3 Motivation and Main Contribution . . . . . . . . . . . . . . . . . . . . . 5.2 Scenario Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Optimization Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 RL Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2 DQN Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2.1 State . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2.2 Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.2.3 Reward Function . . . . . . . . . . . . . . . . . . . . . . . . . . 55.4.1 Software Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.3 Performance Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Example of a Simple RNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Unfolding of RNN neuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 Interaction Between Agent and Environment . . . . . . . . . . . . . . . . . . . . 2.9 Two Possible Configurations of DQN . . . . . . . . . . . . . . . . . . . . . . . .

3.1 One-Dimensional Linear SVR [4] . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Non-Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 LSTM Cell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 GRU Cell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Network Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Time Series of the Network Traffic Volume for about 5 days . . . . . . . . . . . 3.7 Difference of the Test Result Between two RNN variants . . . . . . . . . . . . . 3.8 Prediction Result of SVR model . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Comparison Between two variations of RNN . . . . . . . . . . . . . . . . . . . . 4.1 Edge Computing Hierarchy [27] . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Example of Early Exit Mechanism for DNN . . . . . . . . . . . . . . . . . . . . 4.3 EEoI in an EC scenario [27] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Message Activity at the Edge . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Curve of the Accuracy on the Test Set . . . . . . . . . . . . . . . . . . . . . . . 4.6 EC Virtualization using Docker Platform . . . . . . . . . . . . . . . . . . . . . . 4.7 Performance Results for Distributed Inference using ResNet20-B model . . . . . 4.8 5.1 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Convergence Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Convergence Performance using a second Reward Function . . . . . . . . . . . . vi List of Tables 2.1 4.2 Performance Results of the proposed models . . . . . . . . . . . . . . . . . . . . 4.3 Runtime Options of the Three-Tier Architecture . . . . . . . . . . . . . . . . . . vii

Table 2 .

 2

	1: Examples of Activation Functions

Table 3 .

 3 1: Comparison of Results Between two variants of RNN Figures 3.7a, 3.7b show the difference between predicted values and the actual times series for both GRU and LSTM models.

		.1 below:
	LSTM GRU
	MAPE (%) 16.96	5.8

Table 3 .

 3 In our case, n = (Step Back, Step Ahead, Number of Neurons in the Encoder layers, Number of Neurons in the Decoder layers).

	The table 3.2 below demonstrate the range of possible values that the Hyper Parameters
	could receive.
		Minimum Maximum Step
	Step Back	10	200	5
	Step Ahead	20	200	5
	Encoder "Layer 1" (#Neurons) 20	200	5
	Decoder "Layer 1" (#Neurons) 20	200	5
	Decoder "Layer 2" (#Neurons) 20	200	5
	Decoder "Layer 3" (#Neurons) 20	200	5

2: Configuration of the Hyper Parameters for Grid Search

  .3.

	Id	algorithm look back look ahead neuron1 neuron2 neuron3 MAPE: 1 Step
	1	GRU	40	20	50	50	50	3.06
	2	GRU	60	20	40	70	70	3.08
	3	GRU	30	30	100	80	80	3.12
	16	LSTM	20	30	70	70	70	3.18
	17	LSTM	30	30	50	90	90	3.18
	4281 SVR	10	20				6.06
	6491 LSTM	100	100	60	10	10	33.38
	6492 LSTM	30	100	10	10	10	34.97
			Table 3.3: Results of the Fine Tuning Process

Table 4 .

 4 2: Performance Results of the proposed models

	.2 illustrate the results of the training phase of

Table 4 .

 4 3: Runtime Options of the Three-Tier Architecture

	.3

  Algorithm 2: Proposed DQN Algorithm to Solve the Optimization Problem Initialize the evaluation and target Q network parameters with w; for episode = 1 to M do

	Initialize s 1 (t) ;
	for t = 1 to T do
	Following -Greedy policy, select a t according to the uniform distribution within
	[0, 1] ;
	if p < then
	a t = arg max a Q pred (s(t), a, w) ;
	else
	Select a random action a t with probability ;
	end
	Calculate J s(t) (t) according to 5.1 ;

devices.

The National Engineering Diploma from the Ecole Polytechnique de Tunisie

Applications of ML in Networking & IoT

Conclusion

In this chapter, we had an overview on the main components about ML. We have introduced the DL paradigm and the various ANN models used in this project. Then, we presented RL as a self-learning approach and its main elements. Finally, we discussed the benefits of DL for RL and we presented the well-known algorithm Deep Q-Network (DQN). specific exit point is much higher than a threshold, then this output becomes uncertain and we pass the verdict to the next exit point for further processing.

Formally, the formula of the entropy for a given exit point is defined as follows: entropy(y) = c∈C y c log y c (4.4) where:

• y: represents the output vector.

• C: represents the set of classes for the classification problem.

Hence, the algorithm that runs inference for EEoI models could be formalized as follows:

Algorithm 1: Fast Inference of EEoI (x, T) Result: Predicted Label for the Given Input x initialization;

T then return arg max ŷn ; end return arg max ŷn ; end As a result, the prediction starts from the earliest exit point to the main branch of the EEoI. Indeed, the algorithm is trying to minimize the computation process which is directly tied to the execution time and the energy consumption by identifying the earliest exit point that satisfies the condition of confidence.

Edge Computing Scenario

The proposed architecture of EEoI could be implemented in an Edge Computing scenario as depicted in 4.3. For instance, we assigned the exit points as follows:

• Exit 1: The first branch contains the lowest number of layers. Therefore, it has the minimum computation i.e. minimum latency and efficient energy consumption. As a result, this configuration is well suited for Edge Devices. Hence, we assign the 1 st exit point to the Edge Device for primary decisions.

Performance Analysis of the distributed Inference

We deploy the distributed inference of ResNet20-B model using Docker to evaluate the performance of EEoI model. Since the entropy threshold T directly affects on the model performance, as well as on the exit point that will perform the inference, we evaluate the performance of ResNet20-B under different threshold values. In fact, we can see that both of the accuracy and the runtime become lower as the entropy threshold increases. Meaning that the higher threshold leads to a decreasing in the inference time while the accuracy is deteriorating. Fig. 4.8 shows much further the reduction of the accuracy while the entropy threshold rises. In fact, we could understand the degradation of the model performance by contemplating on Fig. 4.9. The explanation for the decrease in accuracy is that the samples are getting less and less offloaded to both the edge server and the cloud server. Therefore, the prediction becomes made by the first exit point allocated to the edge device, where we have the minimum number of layers i.e. the minimum computation, which leads to a minimum latency and a minimum accuracy by itself.

For instance, if we set the entropy threshold to its maximum value (T = 1), we get the minimum values for both the accuracy and the inference time. Hence, we have improved the latency of the real-time application by diminishing the processing at the cost of the model performance. In contrast, if we set the entropy threshold to its minimum (T = 10 -15 ), we get the maximum values for both the accuracy and the inference time. As a result, we have

DQN Specifications

In order to implement DQN to solve our optimization problem 5.1, we need to specify the algorithm components i.e. the action signal, the state, and the reward function. Hence, we describe these components in the following subsections:

State

To interpret our optimization problem as a DQN problem, we define the system state at time epoch t as follow:

In fact, the state encapsulates the offloading decisions and the allocated bandwidth for transmission and reception for all the users. As a result, the system state consists of (N M + 2N ) elements.

Action

The action allows the agent to interact with the environment by selecting decisions at each time epoch. Thus, we define the action as an index selection that induces a variation between two different states. We use a variable ν to denote the selection of index where ν = 1, 2, ..., N M + 4N . As a result, we denote by a(t) the action at time epoch t, and it could formalized as follow:

In fact, we took the corresponding decision based on the element index in the action vector. Hence, we have the following cases regarding each element of the action:

then the corresponding variation will be on the offloading decision variable x nm (t). More formally, we have the following treatment:

then the corresponding variation will be an increase in the bandwidth allocated for transmission C up n (t). Formally, we have the following process:

then the corresponding variation will be a decrease on the allocated bandwidth for transmission C up n (t). Formally, we have the following process:

then the corresponding variation will be an increase on the allocated bandwidth for reception C do n (t). Formally, we have the following process:

then the corresponding variation will be a decrease on the allocated bandwidth for reception C do n (t). Formally, we have the following process:

Reward Function

Based on the current state and the chosen action, we could determine the reward signal that the environment should send to the agent. Specifically, we define the function J(t) as the value of the objective function for our optimization problem 5.1. Indeed, by extracting the binary variables for the offloading decision and the allocated bandwidth for transmission and reception, we denote by J s(t) (t) = J {xnm(t)},{C up n (t)},{C do n (t)} (t) the cost function that we aim to be minimize. The agent is supposed to minimize the cost function J(t) during the interaction by attempting to choose the optimal decisions. Therefore, the agent receive a reward when the cost function is reduced. In contrast, when the agent increases the cost function, the environment attributes a negative reward, as if it is a retribution. More formally, we define the reward signal as follow:

if J s(t+1) (t + 1) < J s(t) (t) then r (s(t),a(t),s(t+1)) (t) = 1 else if J s(t+1) (t + 1) > J s(t) (t) then r (s(t),a(t),s(t+1)) (t) = -1 else if J s(t+1) (t + 1) = J s(t) (t) then r (s(t),a(t),s(t+1)) (t) = 0 end if

DQN Algorithm

We present in the section the DQN algorithm that will serve to optimize the agent decisions in order to minimize the execution latency of the tasks and the energy consumption of IoT