Jose Jurandir

Alves Esteves

Amina Boubendir
email: amina.boubendir@orange.com

Fabrice Guillemin
email: fabrice.guillemin@orange.com

Pierre Sens
email: pierre.sens@lip6.fr

DRL-based Slice Placement Under Non-Stationary Conditions

Keywords: Network Slicing, Deep Reinforcement Learning, Placement, Optimization

We consider online learning for optimal network slice placement under the assumption that slice requests arrive according to a non-stationary Poisson process. We propose a framework based on Deep Reinforcement Learning (DRL) combined with a heuristic to design algorithms. We specifically design two pure-DRL algorithms and two families of hybrid DRL-heuristic algorithms. To validate their performance, we perform extensive simulations in the context of a large-scale operator infrastructure. The evaluation results show that the proposed hybrid DRL-heuristic algorithms require three orders of magnitude of learning episodes less than pure-DRL to achieve convergence. This result indicates that the proposed hybrid DRLheuristic approach is more reliable than pure-DRL in a real non-stationary network scenario.

I. INTRODUCTION

The promise of network Slicing is to enable a high level of customization of network services in future networks (5G and beyond) leveraged by virtualization and software defined networking techniques. These key enablers transform telecommunications networks into programmable platforms capable of offering virtual networks enriched by Virtual Network Functions (VNFs) and IT resources tailored to the specific needs of certain customers (e.g., companies) or vertical markets (automotive, e-health, etc.) [START_REF] Gpp | Management and orchestration; 5G Network Resource Model (NRM); Stage 2 and stage 3 (Release 17)[END_REF], [START_REF] Etsi Nfv Isg | Network Functions Virtualisation (NFV); Evolution and Ecosystem; Report on Network Slicing Support[END_REF].

From an optimization theory perspective, the Network Slice Placement problem can be viewed as a specific case of Virtual Network Embedding (VNE) or VNF Forwarding Graph Embedding (VNF-FGE) problems [START_REF] Herrera | Resource allocation in NFV: A comprehensive survey[END_REF], [START_REF] Laghrissi | A survey on the placement of virtual resources and virtual network functions[END_REF]. It is then generally possible to formulate Integer Linear Programming (ILP) problems [START_REF] Esteves | Location-based data model for optimized network slice placement[END_REF], which, however, turn out to be N P-hard [START_REF] Amaldi | On the computational complexity of the virtual network embedding problem[END_REF] with very long convergence time.

With regard to network management, there are specific characteristics related to network slicing: slices are expected to share resources and coexist in a large and distributed infrastructure. Moreover, slices have a wide range of requirements in terms of resources, quality objectives and lifetime. In practice, these characteristics bring additional complexity as the placement algorithms need to be highly scalable with low response time even under varying network conditions.

As an alternative to optimization techniques and the development of heuristic methods, Deep Reinforcement Learning (DRL) has recently been used in the context of VNE and Network Slice Placement [START_REF] Yan | Automatic virtual network embedding: A deep reinforcement learning approach with graph convolutional networks[END_REF]- [START_REF] Quang | Multidomain non-cooperative VNF-FG embedding: A deep reinforcement learning approach[END_REF]. DRL techniques are seen as very promising since they allow, at least theoretically, to learn optimal decision policies only based on experience [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. However, from a practical point of view, especially in the context of non-stationary environments, ensuring that a DRL agent converges to an optimal policy is still a challenge.

As a matter of fact, when the environment is continually changing the rules, the algorithm has trouble in using the acquired knowledge to find optimal solutions. The usage of the DRL algorithm in a online fashion can then become impractical. In fact, most of the existing works applying DRL to solve the Network Slice Placement or VNE problem assume a stationary environment, i.e., with static network load. However, traffic conditions in networks are basically non-stationary with daily and weekly variations and subject to drastic changes (e.g., traffic storm due to an unpredictable event).

To cope with traffic changes, we propose in the present paper a hybrid DRL-heuristic strategy called Heuristically Assisted DRL (HA-DRL) [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF]. We apply this strategy in a fully online learning scenario with time-varying network loads to show how this strategy can be used to accelerate and stabilize the convergence of DRL techniques when applied to the Network Slice Placement problem.

The contributions of the present paper are threefold:

1) We propose a network load model to network slice infrastructure conditions with time-varying network loads; 2) We propose a framework combining Advantage Actor Critic and a Graph Convolutional Network (GCN) for conceiving DRL-based algorithms adapted to the nonstationary case; 3) We show how the DRL learning process can be accelerated by using the proposed HA-DRL technique to control the algorithms convergence.

The organization of this paper is as follows: In Section II, we review related work. In Section III, we describe the Network Slice Placement problem modeling. The learning framework for slice placement optimization is described in Section IV. The adaptation of the pure-DRL approaches and its control by using heuristic is introduced in Section IV-B. The experiments and evaluation results are presented in Section V, while conclusions and perspectives are presented in Section VI.

II. RELATED WORK ANALYSIS

We review, in this section, recent studies on DRL-based approaches for network slice placement. The reader interested in a more detailed and comprehensive discussion of those works cited in the present paper may refer to [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF].

A. On Pure-DRL approaches for slice placement

There are only a few recent works on DRL for network slice placement and VNE problems in the literature and the majority of them are pure-DRL approaches [START_REF] Yan | Automatic virtual network embedding: A deep reinforcement learning approach with graph convolutional networks[END_REF]- [START_REF] Quang | Multidomain non-cooperative VNF-FG embedding: A deep reinforcement learning approach[END_REF]. In those works, only the knowledge acquired by the learning agent via training is used as a basis for taking placement decisions. The drawback of this approach is that the learning agent needs extensive exploration of the state and action spaces to learn an appropriate policy for decision making; such a process takes a lot of time during which the agent takes bad placement decisions; this leads to rejection of slices and bad performance.

Furthermore, existing works consider static or fixed average network load regimes in which network slices arrive and exit the system always at fixed rates. In reality, however, network load conditions vary over time as the demand for network services depend on many factors (unpredictable events, day-night variations, etc.). In this kind of non-stationary environment, the learning agent might have trouble in using its knowledge acquired via training as the rules of the environment are constantly changing.

B. On Hybrid DRL-heuristic approaches for slice placement

Two recent works on network slice placement and VNE have been proposed and combine DRL with heuristic methods to speed up the convergence and to increase the reliability of DRL algorithms [START_REF] Quang | A deep reinforcement learning approach for vnf forwarding graph embedding[END_REF], [START_REF] Rkhami | Learn to improve: A novel deep reinforcement learning approach for beyond 5G network slicing[END_REF]. However, these hybrid DRL-heuristic approaches have some drawbacks explained in more details in [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF]. In particular, the approach proposed in [START_REF] Quang | A deep reinforcement learning approach for vnf forwarding graph embedding[END_REF] adopts an infinite action space formulation that adds some overhead reducing the applicability of the algorithm, and the approach proposed in [START_REF] Rkhami | Learn to improve: A novel deep reinforcement learning approach for beyond 5G network slicing[END_REF] strongly depends on the quality of the initial solution provided by a heuristic. Moreover, these two works do not consider the non-stationarity assumption discussed in the present paper.

In [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF], we proposed to adapt heuristically accelerated reinforcement learning to the network slice placement problem. The approach is inspired by [START_REF] Bianchi | Accelerating autonomous learning by using heuristic selection of actions[END_REF] and addresses the shortcomings of [START_REF] Quang | A deep reinforcement learning approach for vnf forwarding graph embedding[END_REF] and [START_REF] Rkhami | Learn to improve: A novel deep reinforcement learning approach for beyond 5G network slicing[END_REF]. In this paper, we extend this approach by including the non-stationarity assumption.

C. On AI/ML approaches considering dynamic network load

A recent body of research considers dynamic network load scenarios when applying AI/ML-based approaches to support optimization of network slice life cycle management. For instance, the authors of [START_REF] Bega | Deepcog: Cognitive network management in sliced 5g networks with deep learning[END_REF] propose a deep learning-based data analytics tool to predict dynamic network slice traffic demands to help avoid SLA violations and network over-provisioning. The paper [START_REF] Alawe | An efficient and lightweight load forecasting for proactive scaling in 5g mobile networks[END_REF] also adopts neural networks to predict network slice traffic demands but, in this case, to perform proactive resource provisioning and congestion control.

To the best of our knowledge, the only paper considering placement optimization using DRL in dynamic network load scenarios is [START_REF] Pei | Optimal vnf placement via deep reinforcement learning in sdn/nfv-enabled networks[END_REF].

The authors propose a Double Deep Q Network (DDQN) algorithm for re-optimizing an initial VNF placement. They consider that the network load changes periodically with a time cycle T . They then separate time cycle T in time intervals ∆t, and train a DDQN model to specifically take charge of VNF placement re-optimization in each time interval ∆t. Despite its originality, the approach proposed in [START_REF] Pei | Optimal vnf placement via deep reinforcement learning in sdn/nfv-enabled networks[END_REF] presents two drawbacks: 1) it depends on offline learning, which is not applicable to online optimization scenarios; 2) it does not use DRL to optimize placement directly as the DRL algorithm selects the region of the network to re-optimize and delegates the optimization to threshold policy procedure. The heuristic calculation of placement decisions can lead to sub-optimal solutions. Contrary to [START_REF] Pei | Optimal vnf placement via deep reinforcement learning in sdn/nfv-enabled networks[END_REF], the present contribution is applicable to offline and online learning and directly learns a placement optimization policy.

III. NETWORK SLICE PLACEMENT OPTIMIZATION PROBLEM

This section presents the various elements composing the model for slice placement. Slices are placed on a substrate network, referred to as Physical Network Substrate (PSN) and described in Section III-A. Slices give rise to as Network Slice Placement Requests (Section III-B), generating a network load defined in Section III-C. The optimization problem is formulated in Section III-D.

A. Physical Substrate Network Modeling

The Physical Substrate Network (PSN) is composed of the infrastructure resources, namely IT resources (CPU, RAM, disk, etc.) needed for supporting the Virtual Network Functions (VNFs) of network slices together with the transport network, in particular Virtual Links (VLs) for interconnecting the VNFs of slices. The PSN is divided into three components: the Virtualized Infrastructure (VI) corresponding to IT resources, the Access Network (AN), and the Transport Network (TN).

The Virtual Infrastructure (VI) hosting IT resources is the set of Data Centers (DCs) interconnected by network elements (switches and routers). We assume that data centers are distributed in Points of Presence (PoP) or centralized (e.g., in a big cloud platform). As in [START_REF] Slim | CLOSE: A costless service offloading strategy for distributed edge cloud[END_REF], we define three types of DCs with different capacities: Edge Data Centers (EDCs) close to end users but with small resources capacities, Core Data Centers (CDCs) as regional DCs with medium resource capacities, and Central Cloud Platforms (CCPs) as national DCs with big resource capacities.

We consider that slices are rooted so as to take into account the location of those users of a slice. We thus introduce an Access Network (AN) representing User Access Points (UAPs) such as Wi-Fi APs, antennas of cellular networks, etc. and Access Links. Users access slices via one UAP, which may change during the life time of a communication by a user (e.g., because of mobility).

The Transport Network (TN) is the set of routers and transmission links needed to interconnect the different DCs and the UAPs. The complete PSN is modeled as a weighted undirected graph G s = (N, L) with parameters described in Table I, where N is the set of physical nodes in the PSN, and

L ⊂ {(a, b) ∈ N × N ∧ a = b}

TABLE I PSN PARAMETERS

Parameter Description

Gs = (N, L) PSN graph N Network nodes S ⊂ N Set of servers DC Set of data centers S dc ⊂ S, ∀dc ∈ DC Set of servers in data center dc SW dc , ∀dc ∈ DC Switch of of data center dc L = {(a, b) ∈ N × N ∧ a = b} Set of physical links cap bw (a,b) ∈ R, ∀(a, b) ∈ L Bandwidth capacity of link (a, b) cap cpu s ∈ R, ∀s ∈ S available CPU capacity on server s M cpu s ∈ R, ∀s ∈ S maximum CPU capacity of server s cap ram s ∈ R, ∀s ∈ S available RAM capacity on server s M ram s ∈ R, ∀s ∈ S maximum RAM capacity of server s M bw s ∈ R, ∀s ∈ S
maximum outgoing bandwidth from s

B. Network Slice Placement Requests Modeling

We consider that a slice is a chain of VNFs to be placed and connected over the PSN. VNFs of a slice are grouped into a request, namely a Network Slice Placement Request (NSPRs), which has to be placed on the PSN. A NSPR is represented as a weighted undirected graph G v = (V, E), with parameters described in Table II, where V is the set of VNFs in the NSPR, and

E ⊂ {(ā, b) ∈ V × V ∧ ā = b}

TABLE II NSPR PARAMETERS

Parameter Description

Gv = (V, E) NSPR graph V Set of VNFs of the NSPR E = {(ā, b) ∈ N × N ∧ ā = b} Set of VLs of the NSPR req cpu v ∈ R CPU requirement of VNF v req ram v ∈ R RAM requirement of VNF v req bw (ā, b) ∈ R Bandwidth requirement of VL (ā, b)

C. Network Load Modeling

We consider the case when the load offered by NSPRs is time varying. We specifically assume that there are several classes of NSPRs and two sets of classes. A first set is composed of NSPR classes (referred to as static) with constant arrival rates, creating a background traffic. A second set is composed of NSPR classes, with time-varying arrival rates so as to reflect some volatility in the NSPRs. Those NSPRs are said dynamic.

1) Network Load for Static NSPR Classes: Let J be the set of resources in the network (i.e., CPU, RAM, bandwidth). Let K c ⊂ N be the set of static NSPR classes. We compute the load generated by arrivals of NSPRs of class k in K c for resource j in J as in [START_REF] Slim | Towards a dynamic adaptive placement of virtual network functions under ONAP[END_REF]:

ρ k j = 1 C j λ k µ k A k j , (1)
where C j is the total capacity of resource j, A k j is the number of resource units requested by an NSPR of class k, λ k is the average arrival rate for an NSPR of class k and 1/µ k is the average lifetime of an NSPR of class k.

2) Network Load for Dynamic NSPR Classes: Let K ∆ ⊂ N be the set of dynamic NSPR classes. We consider a periodic average arrival rate λ k (t) for class k in K ∆ given by

λ k (t) = f k sin 2 πt τ k (2)
where τ k is the period of λ k (t) in time units and f k is a parameter used to control the amplitude of λ k (t). We then adapt Eq. ( 1) to compute the network load for dynamic NSPR class k and resource j as

ρ k j (t) = 1 Cj λ k (t) µ k A k j .
It is worth noting that to preserve ρ k j (t) in the [0, 1] interval, f k must be between 0 and

Cj µ k A k j .
3) Global Network Load: Finally, we define the global network load ρ j (t) for each resource j in J in the simulated time instant t as the sum of the network loads generated by static NSPR classes (K c ) and dynamic NSPR classes (K ∆ ), that is,

ρ j (t) = k∈K∆ ρ k j (t) + k∈Kc ρ k j (3)
It is worth noting 0 ≤ ρ j (t) ≤ 1.

D. Network Slice Placement Optimization Problem Statement

• Given: We describe in this section the machine learning framework used to solve the optimization formulated in Section III. We adopt the same approach as in [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF] but to cope with the non stationary behavior of NSPR arrivals, we introduce an additional set of states to describe network load.

a NSPR graph G v = (V, E) and a PSN graph G s = (N, L), • Find: a mapping G v → Ḡs = ( N , L), N ⊂ N , L ⊂ L, • Subject to: the VNF CPU requirements req cpu v , ∀v ∈ V , the VNF RAM requirements req ram v , ∀v ∈ V , the VLs bandwidth requirements req bw (ā, b) , ∀(ā, b) ∈ E,
Other methods could be considered to deal with the cyclic nature of NSPR arrivals considered in this paper (e.g., LSTM techniques able to infer the periodic characteristics of the NSPRs arrival process). However, our goal in this paper is to set up a method of dealing with non stationary NSPR arrivals, independently of underlying periodic structures. We test the proposed for periodic NSPR arrivals to validate the approach, keeping in mind that the method has to be robust against non stationary traffic variations. This point will be addressed in further studies.

A. Learning framework 1) Policy: We reuse the framework introduced in [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF]. We denote by A the set of possible actions (namely placing VNFs on nodes) and by S the set of all states. We adopt a sequential placement strategy so that we choose a node n ∈ N where to place a specific VNF v ∈ {1, ..., |V |}. The VNFs are sequentially placed so that placement starts with the VNF v = 1 and ends for the VNF v = |V |.

At each time step t, given a state σ t , the learning agent selects an action a with probability given by the Softmax distribution given by

π θ (a t = a|σ t ) = e Z θ (σt,a), b∈N e Z θ (σt,b) , (4)
where the function Z θ : S × A → R yields a real value for each state and action calculated by a Deep Neural Network (DNN) as detailed in Section IV-B1. The notation π θ is used to indicate that policy depends on Z θ . The control parameter θ represents the weights in the DNN.

2) State representation: As in [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF], the PSN state is characterized by the occupancy of servers:

cap cpu = {cap cpu n : n ∈ N }, cap ram = {cap ram n : n ∈ N } and cap bw = {cap bw n = (n,b)∈L cap bw (n,b) : n ∈ N }.
In addition, we keep track of the placement of the outstanding NSPR (under placement) via the vector χ = {χ n ∈ {0, .., |V |} : n ∈ N }, where χ n is the number of VNFs of the current NSPR placed on node n.

Accordingly, the NSPR state is a view of the current placement and is composed of four characteristics, three related to resource requirements (see Table II To deal with the non stationary environment, we introduce the Load state that represents the network load forecast used to learn the network load variations. It is defined by a set ρ j of 100 features for each resource j calculated using the network load formula given by Eq. ( 3) as follows : ρ j = {ρ j (t) : t a ≤ t ≤ t a + 100, t ∈ N}, where t a is the simulated time instant in which the current NSPR arrives.

3) Reward function: We reuse the reward function introduced in [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF]. We precisely consider

rt+1 =    0, if t < T and at is successful T i=0 δ a i+1 δ b i+1 δ c i+1 , if t = T and at is successful δ a t+1 , otherwise (5)
where T is the number of iterations of a training episode and where the rewards δ a i+1 , δ b i+1 , and δ c i+1 are defined as follows:

• An Action a t may lead to a successful or unsuccessful placement. We then define the Acceptance Reward value due to action a t as

δ a t+1 = 100, if a t is successful, -100, otherwise. (6)
• The Resource Consumption Reward value for the placement of VNF v via action a t is defined by

δ c t+1 = req bw (v-1,v) req bw (v-1,v) |P | = 1 |P | , if |P | > 0, 1,
otherwise.

(

) 7
where P is the path used to place VL (v -1, v). Note that a maximum δ c t+1 = 1 is given when |P | = 0, that is, when VNFs v -1 and v are placed on the same server.

• The Load Balancing Reward value for the placement of VNF v via a t

δ b t+1 = cap cpu at M cpu at + cap ram at M ram at . (8)
B. Adaptation of DRL and Introduction of a Heuristic Function 1) Proposed Deep Reinforcement Learning Algorithm : As in [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF], we use a single thread version of the A3C Algorithm introduced in [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]. This algorithm relies on two DNNs that are trained in parallel: i) the Actor Network with the parameter θ, which is used to generate the policy π θ at each time step, and ii) the Critic Network with the parameter θ v which generates an estimate ν π θ θv (σ t ) for the State-value function defined by

ν π (t|σ) = E π T -t-1 k=0 γ k r t+k+1 |σ t = σ ,
for some discount parameter γ.

As depicted in Fig. 1 both Actor and Critic Networks have almost identical structure. As in [START_REF] Yan | Automatic virtual network embedding: A deep reinforcement learning approach with graph convolutional networks[END_REF], we use the GCN formulation proposed by [START_REF] Kipf | Semi-supervised classification with graph convolutional networks[END_REF] to automatically extract advanced characteristics of the PSN. The characteristics produced by the GCN represent semantics of the PSN topology by encoding and accumulating characteristics of neighbour nodes in the PSN graph. The size of the neighbourhood is defined by the orderindex parameter K. As in [START_REF] Yan | Automatic virtual network embedding: A deep reinforcement learning approach with graph convolutional networks[END_REF], we consider in the following K = 3 and perform automatic extraction of 60 characteristics per PSN node. Both the NSPR state and Network Load characteristics are separately transmitted to fully connected layers with 4 and 100 units, respectively. In the Critic Network, the outputs are forwarded to a single neuron, which is used to calculate the state-value function estimation ν π θ θv (σ t ). In the Actor Network, the outputs represent the values of the function Z θ introduced in Section IV-A. These values are injected into a Softmax layer that transforms them into a Softmax distribution that corresponds to the policy π θ .

During the training phase, at each time step t, the A3C algorithm uses the Actor Network to calculate the policy π θ (.|σ t ). An action a t is sampled using the policy and performed on the environment. The Critic Network is used to calculate the state-value function approximation ν π θ θv (σ t ). The learning agent receives then the reward r t+1 and next state σ t+1 from the environment and the placement process continues until a terminal state is reached, that is, until the Actor Network returns an unsuccessful action or until the current NSPR is completely placed. At the end of the training episode, the A3C algorithm updates parameters θ and θ v by using the same rules as in [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF].

2) Introduction of a Heuristic Function: To guide the learning process, we use as in [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF] the placement heuristic introduced in [START_REF] Alves Esteves | Heuristic for edge-enabled network slicing optimization using the "power of two choices[END_REF]. This yields the HA-DRL algorithm. More precisely, from the reference framework shown in Fig. 1, we proposed to include in the Actor Network the Heuristic layer that calculates a Heuristic Function H : S × A → R based on external information provided by the heuristic method, referred to as HEU.

Let Z θ be the function computed by the fully connected layer of the Actor Network that maps each state and action to a real value which is after converted by the Softmax layer into the selection probability of the respective action (see Section IV-A). Let āt = argmax a∈A Z θ (σ t , a) be the action with the highest Z θ value for state σ t . Let a * t = HEU (σ t ) be the action derived by the HEU method at time step t and the preferred action to be chosen. H(σ t , a * t ) is shaped to allow the value of Z θ (σ t , a * t ) to become closer to the value of Z θ (σ t , āt ).

The aim is to turn a * t into one of the likeliest actions to be chosen by the policy.

The Heuristic Function is then formulated as

H(σ t , a t ) = Z θ (σt, āt) -Z θ (σt, at) + η, if at = a * t 0, otherwise (9)
where η parameter is a small real number. During the training process the Heuristic layer calculates H(σ t , .) and updates the Z θ (σ t , .) values by using the following equation:

Z θ (σ t , .) = Z θ (σ t , .) + ξH(σ t , .) β ( 10
)
The Softmax layer then computes the policy using the modified Z θ . Note the action returned by a * t will have a higher probability to be chosen. The ξ and β are parameters used to control how much HEU influence the policy.

C. Implementation Remarks 1) Algorithms considered: We consider four learning algorithms based on the reference framework presented in Fig. 1.

• DRL: It is the pure-DRL algorithm we initially proposed in [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF]. The state representation does not include the network load state and the Actor Network does not contain the Heuristic layer.

V. IMPLEMENTATION AND EVALUATION RESULTS

In this section, we present the implementation and experiments we conducted to evaluate the proposed algorithms.

A. Implementation Details & Simulator Settings 1) Experimental setting:

We developed a simulator in Python containing: i) the elements of the Network Slice Placement Optimization problem (i.e., PSN and NSPR); ii) the DRL, eDRL, HA-DRL and HA-eDRL algorithms. We used the PyTorch framework to implement the DNNs. Experiments were run in a 2x6 cores @2.95Ghz 96GB machine. 2) Physical Substrate Network Settings: We consider a PSN that could reflect the infrastructure of an operator as discussed in [START_REF] Slim | Towards a dynamic adaptive placement of virtual network functions under ONAP[END_REF]. In this network, three types of DCs are introduced as in Section III. Each CDC is connected to three EDCs which are 100 km apart. CDCs are interconnected and connected to one CCP that is 300 km away. We consider 15 EDCs each one with 4 servers, 5 CDCs each with 10 servers and 1 CCP with 16 servers. The CPU and RAM capacities of each server are 50 and 300 units, respectively. A bandwidth capacity of 100 Gbps is given to intra-data center links inside CDCs and CCP-10Gbps being the case for intra-data center links inside EDCs. Transport links connected to EDCs have 10Gpbs of bandwidth capacity. Transport links between CDCs have 100Gpbs of bandwidth capacity as well for the ones between CDCs and the CCP.

3) Network Slice Placement Requests Settings : We consider NSPRs to have the Enhanced Mobile Broadband (eMBB) setting described in [START_REF] Alves Esteves | Heuristic for edge-enabled network slicing optimization using the "power of two choices[END_REF]. Each NSPR is composed of 5 VNFs. Each VNF requires 25 units of CPU and 150 units of RAM. Each VL requires 2 Gbps of bandwidth.

B. Algorithms & Experimental Setup 1) Training Process & Hyper-parameters:

We consider a training process with maximum duration of 55 hours for the considered algorithms. We perform seven independent runs of each algorithm to assess their average and maximal performance in terms of metrics introduced below (see Section V-C). After performing Hyper-parameter search, we set the learning rates for the Actor and Critic networks of DRL and HA-DRL algorithms to α = 5 × 10 -5 and α = 1.25 × 10 -3 , respectively. For eDRL and HA-eDRL, we set the learning rates for the Actor and Critic networks to α = 5.7 × 10 -5 and α = 1.4 × 10 -3 , respectively. We implement four versions of HA-DRL and HA-eDRL agents, each with a different value for the β parameter of the heuristic function formulation (see Section IV-B2). We set in addition the parameters ξ = 1 and η = 0.

2) Network load calculation: We implement the network load model introduced in Section V-B2 considering two NSPR classes: i) a volatile class (referred to as Volatile), with a dynamic arrival rate and ii) a static class (referred to as Long term), with a static arrival rate. Table III presents the network load models proposed for both classes. We consider that each simulation time unit corresponds to 15 minutes in reality. We set the period for the network load function to 96 simulation time units, i.e., one day. The network global network load ρ(t) varies then between 0.3 and 1.0.

C. Evaluation Metrics

To characterize the performance of the placement algorithms, we consider two performance metrics:

1) Global Acceptance Ratio (GAR): The Acceptance Ratio of the different tested algorithms during training computed after each arrival as follows: #accepted NSPRs #arrived NSPRs . This metric is used to evaluate the accumulated acceptance ratio of the different algorithms as the learning process progresses. . This metric allows us to better observe the evolution of algorithm performance over time since it measures algorithm performance in independent parts (phases) of the training process without accumulating the performance of previous training phases.

D. Global Acceptance Ratio Evaluation

Fig. 2 and3 show the progression of the Global Acceptance Ratio (GAR) over time for the considered algorithms. Fig. 2a and Fig. 3a show that after 3 simulated days, the average and maximal GARs of HA-DRL and HA-eDRL with β = 2.0 are convergent and higher than 80% while for the other algorithms, the GARs remain between 20% and 40% and are not stable. We observe that while GARs of HA-DRL and HA-eDRL with β = 2.0 remain stable after the convergence reached in the first 3 simulated days, for the other algorithms, these performance metrics start to stabilize only after 2000 simulated days. Table IV shows that both the maximal and average final GARs, i.e., maximal and average GARs achieved at the end of training, are close to 82% for both HA-DRL and HA-eDRL with β = 2.0. For the other algorithms, maximal final GARs are between 61% and 71% and average final GARs are between 52% and 65%, except for HA-DRL and HA-eDRL with β = 1.0. Table IV also shows that maximal and average final GARs of HA-DRL algorithms are generally higher than the equivalent HA-eDRL versions, the gap being never higher than 6%.

The above results show that HA-DRL and HA-eDRL with β = 2.0 exhibit more robust performance with faster convergence and handle better network load variations than the other evaluated algorithms, notably when compared with the classical DRL approach. Indeed, when setting β = 2.0, the Heuristic Function computed on the basis of the HEU algorithm has strong influence on the actions chosen by the agent. Since the HEU algorithm often indicates a good action, this strong influence of the heuristic function helps the algorithms to become stable more quickly.

The addition of network load related features to the state observed by the eDRL algorithm helps improve the maximal and average final GAR when compared with DRL. However, this improvement is less significant than the improvement brought by the Heuristic Function acceleration as the eDRL algorithm does not achieve the same fast convergence and robustness than HA-DRL and HA-eDRL with β = 2.0. Table IV shows that the only algorithms that reach a maximal and average final TAR (i.e, maximal and average TAR achieved at the end of training) higher than 80% are HA-DRL and HA-eDRL with β = 2.0. HA-DRL with β = 0.1 also attains a maximal final TAR higher than 80% but its average final TAR is less than 60%. HA-DRL with β = 0.5 has average final TAR higher than 65%, but its maximal final TAR is around 74%. In addition, Table IV shows that: HA-DRL algorithms have maximal final TAR generally higher than the equivalent HA-eDRL versions, the gap being never higher than 6%; eDRL and DRL have equivalent maximal final TAR but eDRL has average final TAR 3% higher. Fig. 6b and Fig. 6a show that all algorithms have better maximal and average TAR on volatile NSPRs than on long-term NSPRs. This difference is arguably related to the number of arrivals of each requests and, therefore, the amount of training performed on each class of requests, since many more volatile requests arrive to be placed during the simulated period than long-term requests. These results reinforce the conclusion introduced in Section V-D. The maximal TAR progression results (i.e., Fig. 4) allow us to observe that in the best case, if a large number of training phases is given, all the algorithms attain a good performance. However, the average TAR progression results (see Fig. 5) show that only HA-DRL and HA-eDRL with β = 2.0 have robust performance. These algorithms are also the only one to have quick convergence being, among those evaluated, the most adapted to be used in practice.

E. Acceptance Ratio per Training Phase Evaluation

VI. CONCLUSION

As a follow-up of [START_REF] Alves Esteves | A heuristically assisted deep reinforcement learning approach for network slice placement[END_REF], we have introduced a DRL algorithm, which considers the variations of network load in the placement of network slice requests. For this purpose, we have considered four families of algorithms, pure-DRL and eDRL techniques and their variants combined with a heuristic whose efficiency was investigated in a previous work [START_REF] Alves Esteves | Heuristic for edge-enabled network slicing optimization using the "power of two choices[END_REF]. The influence of the heuristic on the DRL algorithm can be tuned by means of a parameter (namely, parameter β introduced in Section IV-B2). We have shown that introducing network load states into the DRL algorithm together with a combination with the heuristic function yields very good results in terms of GAR and TAR, which are stable in time. This study aims at proving that coupling DRL and heuristic functions yields good and stable results even under non stationary conditions. In the present paper, we have assumed that network load has periodic fluctuations. The next step is to study the behavior of the proposed algorithm in case of an unpredictable network load disruption.

  refers to a set of substrate links. Each node has a type in the set {UAP, router, switch, server}. The available CPU and RAM capacities on each node are defined as cap cpu n ∈ R, cap ram n ∈ R for all n ∈ N , respectively. The available bandwidth on the links are defined as cap bw (a,b) ∈ R, ∀(a, b) ∈ L.

  is a set of VLs to interconnect the VNFs of the slice . The CPU and RAM requirements of each VNF of a NSPR are defined as req cpu v ∈ R and req ram v ∈ R for all v ∈ V , respectively. The bandwidth required by each VL in a NSPR is given by req bw (ā, b) ∈ R for all (ā, b) ∈ E.

  for the notation) of the current VNF v to be placed: req cpu v , req ram v and req bw v = (v, b)∈E req bw (v, b) .In addition, let m v = |V | -v + 1 be the number of VNFs of the outstanding NSPR still to be placed.

Fig. 1 .

 1 Fig. 1. Reference framework for the proposed learning algorithms.

  • eDRL: This algorithm is an enhanced version of DRL in which the state representation includes the network load state and the Actor Network does not contain the Heuristic layer; • HA-DRL: This algorithm embeds the heuristic but the state representation does not include the network load state while the Actor Network contain the Heuristic layer; • HA-eDRL: The state representation includes the network load state and the Actor Network contains the Heuristic layer. 2) Implementation details: All resource-related characteristics are normalized to be in [0, 1]. This is done by dividing cap j and req j , j ∈ {cpu, ram,bw}, by max n∈N M j n . With regard to the DNNs, we have implemented the Actor and Critic as two independent Neural Networks. Each neuron has a bias assigned. We have used the hyperbolic tangent (tanh) activation for nonoutput layers of the Actor Network and Rectified Linear Unit (ReLU) activation for all layers of the Critic Network. We have normalized positive global rewards to be in [0, 10]. During the training phase, we have considered the policy as a Categorical distribution and used it to sample the actions randomly.

  Fig.2 and 3show the progression of the Global Acceptance Ratio (GAR) over time for the considered algorithms. Fig.2aand Fig.3ashow that after 3 simulated days, the average and maximal GARs of HA-DRL and HA-eDRL with β = 2.0 are convergent and higher than 80% while for the other algorithms, the GARs remain between 20% and 40% and are not stable. Fig. 2b and Fig 3b exhibit the progression of the GARs over a longer simulated time horizon of 4500 simulated days.We observe that while GARs of HA-DRL and HA-eDRL with β = 2.0 remain stable after the convergence reached in the first 3 simulated days, for the other algorithms, these performance metrics start to stabilize only after 2000 simulated days. TableIVshows that both the maximal and average final GARs, i.e., maximal and average GARs achieved at the end of training, are close to 82% for both HA-DRL and HA-eDRL with β = 2.0. For the other algorithms, maximal final GARs are between 61% and 71% and average final GARs are between 52% and 65%, except for HA-DRL and HA-eDRL with β = 1.0. Table IV also shows that maximal and average final GARs of HA-DRL algorithms are generally higher than the equivalent HA-eDRL versions, the gap being never higher than 6%.The above results show that HA-DRL and HA-eDRL with β = 2.0 exhibit more robust performance with faster convergence and handle better network load variations than the other evaluated algorithms, notably when compared with the classical DRL approach.

  ,)=0.5 HA-DRL,)=1.0 HA-DRL,)=2.0 eDRL HA-eDRL,)=0.1 HA-eDRL,)=0.5 HA-eDRL,)=1.0 HA-eDRL,)=2.0 (b) Maximal performance in 4500 simulated days.

Fig. 2 .

 2 Fig. 2. Maximal Global Acceptance ratio results.

  ,β=0.5 HA-DRL,β=1.0 HA-DRL,β=2.0 eDRL HA-eDRL,β=0.1 HA-eDRL,β=0.5 HA-eDRL,β=1.0 HA-eDRL,β=2.0 (b) Average performance in 4500 simulated days.

Fig. 3 .

 3 Fig. 3. Average Global Acceptance ratio results.

Fig. 4 and

 4 Fig. 4 and Fig. 5 show that all algorithms need at least 16 training phases to reach a convergent maximal and average Acceptance Ratio per Training Phase (TAR), except for HA-DRL and HA-eDRL with β = 2.0 which need only one training phase (see Fig 4a and Fig 5a). The fast convergence of HA-DRL and HA-eDRL with β = 2.0 is due to the strong influence of the Heuristic Function in the choice of actions as explained in Section V-D. Fig. 4a and Fig. 4b illustrate that algorithms HA-DRL and HA-eDRL with β = 0.5, HA-DRL with β = 0.1 and DRL need around 16 phases to reach a convergent maximal TAR while Fig. 4b and Fig. 4c demonstrate that eDRL, HA-eDRL with β ∈ {0.1, 1.0} and HA-DRL with β = 1.0 need around 34 phases. Finally, Fig. 5a and Fig. 5b show that HA-DRL and HA-eDRL with β ∈ {0.1, 0.5}, eDRL, and DRL need around 16 phases to obtain a convergent average TAR while Fig. 5b and Fig. 5c show that HA-DRL and HA-eDRL with β = 1.0 need around 34 phases.

Fig. 4 .

 4 Fig. 4. Maximal Acceptance Ratio per training phase results.

Fig. 5 .

 5 Fig. 5. Average Acceptance Ratio per training phase results.

  ,)=1.0 HA-eDRL,)=2.0 (a) Maximal performance for volatile NSPRs.

  ,β=0.1 HA-DRL,β=0.5 HA-DRL,β=1.0 HA-DRL,β=2.0 eDRL HA-eDRL,β=0.1 HA-eDRL,β=0.5 HA-eDRL,β=1.0 HA-eDRL,β=2.0 (b) Maximal performance for longterm NSPRs.

Fig. 6 .

 6 Fig. 6. Acceptance Ratio per training phase results (per NSPR class).

  the server CPU available capacity cap cpu s , ∀s ∈ S, the server RAM available capacity cap ram

s , ∀s ∈ S, the physical link bandwidth available capacity cap bw (a,b) , ∀(a, b) ∈ L. • Objective: maximize the network slice placement request acceptance ratio, minimize the total resource consumption and maximize load balancing. IV. LEARNING FRAMEWORK FOR NETWORK SLICE PLACEMENT OPTIMIZATION

  2) Acceptance Ratio per training phase (TAR): The Acceptance Ratio obtained in each training phase, i.e., each part of the training process, corresponding to 10 4 NSPR arrivals or 10 4 episodes. It is calculated as follows:

	#accepted NSPRs
	10 4

ACKNOWLEDGMENT

This work has been performed in the framework of 5GPPP MON-B5G project (www.monb5g.eu). The experiments were conducted using Grid'5000, a large scale testbed by Inria and Sorbonne University (www.grid5000.fr).