Yassine Presenter

Hadjadj-Aoul

Pham Tran

Anh Quang #@

Yassine Hadjadj-Aoul #
email: yassine.hadjadj-aoul@irisa.fr@nokia-

Evolutionary Actor-Multi-Critic Model for VNF-FG Embedding

What are the challenges facing network operators today?

• Lack of agility -Equipment that can hardly be adapted to the needs and of which any update is complex and not always possible -Scaling is not always possible and oversizing is costly (unlike the Cloud)

7

Lack of agility

What are the challenges facing network operators today?

• It is very difficult to grow -Renting infrastructure is no longer as profitable

Operators' needs

• Automated network infrastructure -Self configuration, self healing, self scaling, self *, …

• Supporting current and future services within the same infrastructure What metrics are considered for placement?

• This naturally depends on the problem being addressed (SLA) ...

-Reliability

• Loss • K-connectivity, average connectivity, …

-Service requirements Services placement: A well studied problem!

• The problem of placement is an old and well-studied problem 1 -Many papers in the Cloud context -Conventional service placement comes down to a problem of bin-packing or knap-sack

• NP-hard problem
-Realistic placement of services, like the VNF-FG placement is much more complex

• Services are composite, since they include several sub-services, and multi-constrained -Must be added the multiple constraints on the links

• Less attention has been paid to the placement of VNF-FG, which is actually a fairly recent issue -There is no known polynomial-time algorithm -Even small problems may be hard to solve » Propose more efficient heuristics for solving the problem, so they fall into another category.

• Main limitation:

-Some parameters are only obtained during run-time (latency and loss) which makes these approaches sometimes ineffective in a real context.

Classification of existing Approaches for the Placement

Heuristics-based approaches

-Most of the paper fall in this category -Use generally : a two step approach -Placing VNFs using traditional algorithm (First Fit, Best Fit, nearest search procedure, …) -Then placing VLs (using Shortest path "SP", K-SP, …) -Very fast, effective and deal with very large problems » For some industrials, this is the best solution

• Main limitation:

-In systems where constraints and objectives are changing, these types of approaches are not very suitable since they generally require a total redesign. Moreover, with heuristics we have a rapid convergence at the price of the risk of sticking at a local minima.

14

Local minima Global minima

Classification of existing Approaches for the Placement • Metaheuristics make it possible to respond effectively to the problem VNF-FG placement. They can very easily integrate new objectives or constraints without reconsidering the solution, unlike heuristics.

-However, to address a new placement you almost always need to start from the beginning … as there is no real learning

Classification of existing Approaches for the Placement

Learning-based approaches

-Only very few approaches (not so few now) -Use generally : a two step approach but … the reward function concerns VNFs + VLs placement, which means that we are addressing both at the same time.

-Very slow » Unclear whether these approaches can adequately address the problem.

• Main limitation:

-Efficient approaches still need to be developed. 17

Pham Tran Anh Quang, Yassine Hadjadj-Aoul, and Abdelkader Outtagarts : "A deep reinforcement learning approach for VNF Forwarding Graph Embedding". In IEEE TNSM (October 2019)

Deep Learning Approach

-Machine Learning or Optimization-based? The output of the critic drives learning in both the actor and the critic

DRL Agent

• Weighted First Fit Algorithm (WF 2 A)(*)

For VNF embedding:

•Step 1: Sort substrate nodes in terms of their weights

••

  Figure taken from an IEEE ComMag paper

-•

  With very diverse constraints (latency, bandwidth, loss, CPU, FPGA, …) • Softwarization of the network and the services -Ability to lease infrastructure to third parties without compromising the network and its efficiency -Higher programmability (e.g. Yang, P4 …) • Being part of the service delivery value chain Network slicing (in particular) is seen as an opportunity to meet these different objectives In practice, what does slicing a network consist of? • Most simple form -Placement of services consisting in one VNF • Offline vs online problem (bin packing problem) -May consider 1 or several metrics for the placement (e.g. latency, load, reliability, …) • NP-complete problem • More advanced form (more complex) -Placement of services in a VNF-FG form • Involves not only the placement of VNFs but also addressing a routing problem • Need to consider several metrics (QoS requirements) Most advanced forms -Placement and scalability of services -Run-time placement -… 10 Substrate network Service (e.g. VNF-FG) State Action (VNF-FG placement)

•

  Difficult to be profitable when you don't decide on the rates • It's difficult to get a return on investment when having a continuous evolution of the infrastructure …

						No
						growth
	1G (1980)	2G (1990)	3G (2000)	4G (2010)	5G (2020)	...
						8

-Operators are not part of the delivery chain of the service, which is very profitable (e.g. CDN)

-

  Most of the paper address the problem of VNE, which is less complex

	Classification of existing Approaches
	for the Placement
	1. Mathematical optimisation-based approaches

1 DANTZIG, G. B., "Discrete-Variable Extremum Problems", Opns. Res. Soc.

Am. 5, 266-277 (1957)

-Most of the paper fall in this category -Use of: Integer Linear Program (ILP) or Mixed ILP (MILP) …

• Integer Programming is an NP-complete problem. So:

it comes to placing both at the same time …

  With enough time this may converges to global optimum -Comment … • As the fitness (cost) function is function of VNFs + VLs placement …

	3. Metaheuristics-based approaches

-Use generally : a two step approach -Placing VNFs using an evolutionary, greedy, … algorithm -Then placing VLs using Shortest path "SP", … or using a metaheuristic -Slow, very effective and deal with very large problems » Explore all solutions, or only feasible solutions (faster with risk of stacking at a local optimum) -15 Pham Tran Anh Quang, Jean-Michel Sanner, Cédric Morin, and Yassine Hadjadj-Aoul : "Virtual Network Function Forwarding Graph Embedding: A genetic algorithm approach". In Wiley, IJCS (August 2019)

  •Step 2: Attempt deploying VNF at the lowest weight substrate node •If the substrate node can host the VNF •Step 3a: Update remaining resources of the substrate node •If the substrate node cannot host the VNF: •Step 3b: Remove that node from the selection process and back to step 2

	For VL embedding:
	Use Dijkstra algorithm to identify the lowest cost path to connect VNFs
	à Final allocation decision

(*) This algorithm has been introduced in P. T. A. Quang, Y. Hadjadj-Aoul and A. Outtagarts, "A Deep Reinforcement Learning Approach for VNF Forwarding Graph Embedding," in IEEE Transactions on Network and Service Management, vol. 16, no. 4, pp. 1318-1331, Dec. 2019. doi: 10.1109/TNSM.2019.2947905 Evolutionary Actor-Multi-Critic Model

Metaheuristics-based approaches-Features and main limitation:

-Machine Learning or Optimization-based? -Why go deep?

Data dependency

-Why reinforcement learning? https://www.sumologic.com/blog/machine-learning-deep-learning/

Conclusions

• Adopt Enhanced Exploration framework to boost the performance of DDPG • Proposed an evolutionary algorithm to improve the learning process of critic networks • The simulation confirms that the evolutionary algorithm can improve the performance remarkably


Artemio Plana-Fattori
email: artemio.planafattori@agroparistech.fr

Christophe Doursat

Denis Flick Agroparistech

Graciela Alvarez

COUPLING FLUID FLOW, HEAT TRANSFER AND FOOD PRODUCT TRANSFORMATION: HEAT TREATMENT OF A STARCH SUSPENSION INSIDE A TUBULAR HEAT EXCHANGER

Keywords: Food product transformation, Starch granule swelling, Thermo-kinetic model, Thermal history of fluid particles, Suspension viscosity

Many liquid food processes involve coupled phenomena of fluid flow, heat transfer and product transformation. A typical example is the heat treatment of a starch suspension inside a tubular heat exchanger. Fluid flow influences heat transfer which determines temperature evolution along fluid trajectories. Temperature locally influences the product transformation. The latter influences transport properties notably the viscosity which in turn influences the fluid flow behavior. In this study a numerical model was developed in order to account for these coupled phenomena in the case of a modified waxy maize starch suspension. Velocity and temperature fields were analyzed in view of the local state of the food product properties like the mean diameter of starch granules and the suspension viscosity.

INTRODUCTION

Numerical modeling of fluid flow and heat transfer inside liquid food process equipments has been largely developed (e.g., [START_REF] Norton | Computational fluid dynamics (CFD) -An effective and efficient design and analysis tool for the food industry: A review[END_REF]Sun 2006, Kumar and[START_REF] Kumar | Fluid flow and its modeling using computational fluid dynamics[END_REF]. Transport phenomena properties, including the viscosity, are often assumed to be a function of the local temperature. Food transformation processes like starch gelatinization are more rarely taken into account.

Starch is the most widely employed thickener in the food industry. Most industrial processes for starch-thickened products involve temperatures which can be high enough for enabling starch phase transitions. The well-accepted concept of "gelatinization" refers to the destruction of the crystalline structure in starch granules, which is an irreversible process that includes, in a broad sense and in time sequence, granular swelling, native crystalline melting and molecular solubilization [START_REF] Liu | Thermal processing of starch-based polymers[END_REF]. [START_REF] Liao | Role of thermorheological behaviour in simulation of continuous sterilization of a starch dispersion[END_REF] studied the continuous flow sterilization of a starch dispersion in which gelatinization is taken into account with the help of a thermo-rheological model. In such a model, the mixture viscosity depends on the local present temperature of the starch suspension, including two effects: with heating, the apparent viscosity increases due to the granule's swelling whereas the continuous phase viscosity decreases. Implicitly, that model assumes that the starch granules swelling state (for instance, evaluated in terms of the granules' mean diameter) is a function of the local present temperature only.

We argue that, rather than a straightforward relationship with the temperature, the food product viscosity can exhibit a nonnegligible dependence on the thermal history of the fluid particle under consideration. Fluid particles running close to a heating wall evolve under thermal and kinetic conditions which are different from those associated with particles moving faster at greater distances from such wall. In the present work, the influences of the continuous phase viscosity and of the granule's volume fraction on the suspension viscosity are separated. The former depends instantaneously on temperature whereas the latter depends on the whole thermal history throughout the progressive swelling kinetic of the starch granules. Finite-element-based simulations are conducted, including a thermo-kinetic model based on the experimental work by [START_REF] Lagarrigue | Swelling kinetics of wazy maize and maize starches at high temperatures and heating rates[END_REF] about a modified waxy maize starch suspension.

MODELLING

The food product is an aqueous suspension of starch; because its high content of water, it is assumed to be a Newtonian liquid. Density and thermal properties are approximated by those of pure water. Conservation equations for the food product mass, momentum and energy can be written as:

u 0

(1)

uu  ( p I  u (u) T ) ) (2)  CP uT  ( kT ) (3)
The food product viscosity () can be obtained by multiplying the water viscosity (W) and the relative viscosity associated with the suspension of particles. A number of approximations have been proposed for estimating the latter (see a review by [START_REF] Cheng | Exponential formula for computing effective viscosity[END_REF]. We adopted the model proposed by [START_REF] Thomas | Transport characteristics of suspension. VIII: A note on the viscosity of Newtonian suspensions of uniform spherical particles[END_REF] because it provides an increase of the product viscosity with temperature which is coherent with experimental results (see Fig. 7a of [START_REF] Lagarrigue | Swelling kinetics of wazy maize and maize starches at high temperatures and heating rates[END_REF]. The product viscosity is hence expressed as a function of the volume fraction associated with starch granules ():

/ W(T) 1 2.5 10.05    0.00273e 16.6  (4)

The solid volume fraction can be estimated from the mean volume-weighted diameter associated with the granules' size distribution (D), employing also respective information characterizing the starch suspension before any heat treatment:

O  ( D / DO )  (5)
Following [START_REF] Lagarrigue | Swelling kinetics of wazy maize and maize starches at high temperatures and heating rates[END_REF], [START_REF] Rao | Granule size and rheological behavior of heated tapioca starch dispersions[END_REF] and previous contributions, we define the degree of gelatinization X of the starch suspension at a given instant in terms of the values assumed by that mean diameter before (DO), during (D), and after the heat treatment (DM):

XDDODMDO (6)
Starch granule swelling can be evaluated through a secondorder kinetics:

uX  V( 1  X ) 2 (7)
where the rate constant V can be suitably expressed in terms of the Arrhenius law:

V(T)  VO exp( EA / ( R T ) ) (8)
Our attention is here focused on the 3.1 % w/w modified waxy maize starch dispersion studied by [START_REF] Lagarrigue | Swelling kinetics of wazy maize and maize starches at high temperatures and heating rates[END_REF]; it corresponds to DO  15.5 and DM  39.6 m, O  0.03, VO  2.9e12 s 1 , and EA  84.8 kJ.mol 1 . Thermal and fluid properties of pure water are properly described as functions of the temperature.

The coupled phenomena are illustrated inside a tubular heat exchanger. Such a geometry represents typical industrial devices used, for example, in continuous flow sterilization (see a discussion by [START_REF] Liao | Role of thermorheological behaviour in simulation of continuous sterilization of a starch dispersion[END_REF]. A two-dimension axial-symmetric domain was built, L  1 m long and R  5 mm wide. Conservation equations for the fluid mass, momentum and energy (eqs. 1-3), combined with the equation of convection and swelling of starch granules (eq. 7), were solved using the finite-element-based simulation package COMSOL Multi-physics (version 3.5a). The mesh contains identical rectangular elements only: suitable results

were obtained by adopting 2000 elements along the length by 40 elements along the tube radius.

A fully developed parabolic flow profile was assumed at the entrance of the tubular exchanger. The flow rate was fixed at 10 liters per hour (roughly 0.035 m.s -1 ). According to the Reynolds number (about 600), the flow regime is laminar. Boundary conditions are summarized in Table 1.

RESULTS AND DISCUSSION

Figure 1 overviews the temperature (left), the degree of gelatinization (center) and the viscosity (right) fields resulting from simulations described above. The viscosity field was deformed according to the velocity; inlet and outlet velocity profiles can be easily appreciated at domain extremes. Dark blue and red colors correspond to minimum and maximum values respectively. The volume fraction field was omitted, being a function of the degree of gelatinization (eqs. 5-6). The food product is heated from the wall (right of domain). Fluid particles move slower and reach higher temperatures when running near it. Starch granules swell after heating; their volume fraction increases as well as the food product viscosity. In comparison with the conditions prevailing near the inlet, the viscosity increases by a factor of about 7 near the wall at the outlet, slowing down the velocity in this region of the domain.

These results follow, in a partial extent, those obtained by [START_REF] Liao | Role of thermorheological behaviour in simulation of continuous sterilization of a starch dispersion[END_REF]. They also observed in the heating section of their tubular exchanger an increase of viscosity near the heating wall due to gelatinization, leading to a velocity decrease near this wall and a velocity increase at the axis of symmetry. Nevertheless starch type and concentration were different (4 % of waxy rice starch), and heating conditions were stronger (145°C); hence a direct comparison seems difficult. Their maximum viscosity was much higher (about 6 Pa.s), which caused velocity profiles almost flat were the food product was very viscous; this was not observed in our case. Other geometries than tubular exchangers have been adopted in numerical models coupling fluid flow, heat transfer and starch gelatinization, like axially rotating cans [START_REF] Tattiyakul | Simulation of heat transfer to a canned corn starch dispersion subjected to axial rotation[END_REF]. Any tentative of comparison with such results seems far to be straightforward.

The evolution of food product properties can also be studied with the help of hypothetical trajectories of fluid particles released at the domain inlet. We can imagine each fluid particle as a suspension droplet containing an ensemble of starch granules at similar thermodynamic and kinetic conditions. Figure 2 compares selected properties associated with fluid particles released at 0.5 and at 1.0 mm from the heating wall (see Figure 1). The residence time in the domain correspond to about 89 and 36 s, respectively, because the maximum velocity occurs at the symmetry axis. Figure 2 shows that the fluid particle moving closer to the wall is associated, at the outlet, with higher values of temperature (left), of mean diameter of starch granules (centre), and of viscosity (right). Other the granule starch swelling, the product viscosity depends also on the continuous phase (pure water) viscosity. The latter slowly decreases with heating, and this fact explains why the product viscosity is slightly reduced during the first seconds after particle release.

Positions A and B in Figures 1 and 2 correspond to a same temperature value (about 67 °C) which is not experienced simultaneously in the time series of the two followed fluid particles. Position A represents the outlet for the fluid particle released at 1.0 mm from the heating wall; it reached this position after about 36 s from the inlet. Position B is far before the domain outlet; the fluid particle released at 0.5 mm from the wall moves slower and spent about 57 s to come there. Despite the same temperature, position B corresponds to a mean diameter of starch granules which is about 10 % greater, and to a product viscosity about 50 % greater, than respective values at position A. Such differences are explained by the fact that fluid particles running at different distances from the heating wall move with different velocities and hence experience different temperature time series (or histories). By analyzing these two trajectories it appears that the influence of temperature on viscosity is far to be instantaneous. The product viscosity depends on the continuous phase viscosity as well as on the starch granule volume fraction. The former depends on the present temperature and the latter depends on the whole thermal history of the respective fluid particle.

CONCLUSIONS

In this study our attention was focused on the main processes driving the starch gelatinization inside a tubular heat exchanger. Just one among the growing number of types of starch was taken into account. Starch research is a very active subject, and significant effort has been accomplished in characterizing native and modified starch, including different genotypes and various climates (e.g., [START_REF] Yeh | Kinetics of phase transition of native, cross-linked, and hydroxypropylated rice starches[END_REF], Patterson et al. 2001[START_REF] Choi | Swelling characteristics of native and chemically modified wheat starches as a function of heating temperature and time[END_REF], Xie et al. 2009).

The aim of this study was to demonstrate that coupling a gelatinization model (for a given starch type and concentration) with a fluid flow and heat transfer model can highlight what happens inside a tubular heat exchanger. The numerical model enables to follow selected food product properties along different fluid particle trajectories.

Food product properties, as simulated here, are the final result of a fully-coupled number of processes: fluid flow and heat transfer drive the temperature field; temperature influences the product transformation kinetics, which in turn modifies the product viscosity and hence the fluid flow behavior itself. Our results show that the local present temperature does not determine alone the present food product properties. The degree of transformation inside a fluid particle of food product depends on its whole thermal history. This means that the temperature evolution along the fluid particle trajectories is a major information which must be properly assessed when, for instance, the product viscosity depends on the degree of transformation.

A relatively simple food product transformation was here considered (starch granules swelling). Further work will focus on coupling fluid flow and heat transfer with more complex transformation (like amylose solubilization, mechanical granules fragmentation, aggregation).

Figure 1 :

 1 Figure 1: Distribution of Selected Product Properties across the Domain


Gerardo Rubino

Pham Tran

Anh Quang

TEAM WORKING ON SERVICE PLACEMENT

Difficulty in finding an optimal placement, excepting for very small instances (i. Safety problem

Objective

Exploring the potential of deep learning for safe placement of VNF-FG

APPLICATION OF DRL FOR VNF-FG PLACEMENT

The vanilla DDPG is not suitable for very large-scale discrete action space

In VNF-FG embedding problem, there are constraints of resources such that some discrete actions are not feasible Impact of the number of fully connected Layers

IMPROVING EXPLORATION WITH EEDDPG

Noise is added to the proto-action to create H noisy actions.

Each proto action is fed into HFA in order to determine the feasible allocation of VNFs.

Evaluation: Multi-critic

Different nets due to the arbitrary initialization The best action identified (best Q value) by the evaluator will be executed

Updates of actor:

Best critic network

Introduction 3 KEY

 3 Challenges related to network slicing On using Deep Reinfocement Learning for network slicing Conclusions CONCERNS FOR NETWORK OPERATORS The ability to support current, emerging and future services Examples: Automotive, mMTC, IoT/Factory automation, e-health Ability to lease infrastructure to third parties without compromising the network and its efficiency Consequence: Multiple stakeholders in a network Automation is big deal for Network Operators Objective: Zero touch networks WHY ARE OPERATORS INTERESTED IN NETWORK SLICING? s i c a l i n f r a s t r u c t u r e I n t e r n e t o f T h i n g s s l i c e H e a l t h c a r e s l i c e M o b i l e b r o a d b a n d s l i c e Placement of services in a VNF-FG form Involves not only the placement of VNFs but also addressing a routing problem Addressing the VNFs placement then the routing … or simultanuously Need to consider several metrics (QoS requirements) Intra-domain or multi-domain placement Extremely large number of possibilities of placement (very large action space)

  allows improving the rewardConvergence to classical heuristic performance is immediateNo random behavior at the begining of the learning Learning is slow, but once learned, execution is immediateThe proposed solution allowsImproves the heuristic performance Makes the placement safer …

  DDPG-HFA E 2 D 2 PG CONCLUSIONS DRL are very efficient in addressing the targeted issue Lack of explainability But combining a heuristic with DRL allows having some guarantees in the placement quality Efforts are still needed … Ongoing work on using Graph neural nets for services placement Combining control theory and DRL for safer placement strategies.

ADVANTAGE

Improving exploration allows to discover better action which is very efficient in congested systems With learning techniques, there is sometimes a degradation of performance the use of several CRITIC networks brings more stability

The complexity is not necessarily much more important the back-propagation is only applied on one CRITIC network, the best ranked one

ADVANTAGE

Discover new and much more interesting states/actions Effective in highly congested systems


CELLULAR NETWORK ARCHITECTURE

Method 2: Can be determined analytically.

Analysis of 𝑁 #

SOME EXISTING APPROACHES TO TACKLE THE CONGESTION AT THE ACCESS

Access planning

Limit the burden … but insufficient since some devices react to events which cannot be timed.

Grouping devices

Pull-based scheme

A paging message may also include a back-off time for the MTC

Separate RACH resources for MTC

Successful attempts

𝑵 𝒔 𝑴 M IoT device 𝑞 " = 1 - 1 𝑁 𝑋 ! 𝑋 " 𝑥 ! 𝑥 " 𝜆 𝑝 𝑥 ! 𝑞 " # # $! 𝑥 % 𝑥 !,$ 𝑥 ",! 𝜇 ! 𝑥 !,' 𝜇 % 𝑥 %,' 1 -𝑝 𝑥 ! 1 -𝑞 " # # $! 𝑥 % 𝑋 ! 𝑋 " dx 1 dt = λ -x 1 + µ 1 x 1,L , dx 2 dt = px 1 + µ 2 x 2,L -x 2 , dx 1,L dt = (1 -p)x 1 -µ 1 x 1,L , dx 2,L dt = 1 -q x 2 -1 N x 2 -µ 2 x 2,L .
EFFICIENT SUPPORT OF IOT DEVICES Estimating the access's contention Objective:

𝑥 ! 𝑥 " 𝜆 𝑝 𝑥 ! 𝑞 " # # $! 𝑥 % 𝑥 !,$ 𝑥 ",$ 𝜇 ! 𝑥 !,' 𝜇 % 𝑥 %,' 1 -𝑝 𝑥 ! 1 -𝑞 " # # $! 𝑥 %
Find the probability of blocking that maximizes the average reward.

HOW TO SOLVE THE PROBLEM?

Twin Delayed Deep Deterministic policy gradient algorithm (TD3)

  of devices ... ... but a limited number of resources (i.e., # of opportunities to connect) Random access Only way to access the network (simplest) The most critical area Complex traffic pattern Poisson (e.g., credit machine in shops), Uniform (e.g., traffic lights), Beta (e.g., event driven) Different classes of IoT (including prioritized M2M) Yassine Hadjadj-Aoul, and Tarik Taleb: "Cellular-based Machine Type Communication: Overload control". In IEEE Network, Vol. 26, Issue 6, Pages : 54 -60 (November 2012) Yassine Hadjadj-Aoul, Nawel Zangar, Sami Tabbane : "On the risk of congestion collapse in heavily congested M2M networks". In proc. of IEEE ISNCC, Hammamet, Tunis (May 2016) Even when having 54 opportunities, the risk of congestion is still high … Could be modeled using the classical « Balls into Bins » problem N I : # of idle preambles (# of bins with no ball) ~20.06) HOW TO DETERMINE THE OPTIMAL NUMBER OF CONTENDING DEVICES? Method 1: Can be determined by Monte Carlo simulations. 14 Maximized when: # of contending devices = 54 Number of opportunities N

  Hadjadj-Aoul, Nawel Zangar, Gerardo Rubino: "Estimating the number of contending IoT devices in 5G networks: revealing the invisible". In Wiley, Transactions on Emerging Telecommunications Technologies (TETT). (August 2018) PROBLEM FORMULATION MARKOV DECISION PROCESS (MDP) DEFINITION MDP: 𝑀 = (𝑆, 𝐴, 𝑝, 𝑟) State 𝑆 : State space 𝑠 ! 𝑘: time step (each new frame) 𝑠 ! reflects better the real state Action 𝐴 : Action space 𝑝: blocking factor Continuous, deterministic 𝑝(𝑠 1 |𝑠, 𝑎): transition probability Related to the environment (not known)Revenue 𝑟(𝑠, 𝑎, 𝑠 1 ) : is the reward of transition (𝑠, 𝑎, 𝑠 1 )

  Splitting the preambles into H2H group(s) and MTC group(s) or allocating PRACH occasions in time or frequency to either H2H or MTC devices.

	Backlogged IoT
	device
	Select a random q	ACB barring time	Arrival
		Access with
	Arrivals	probability p		Back logged
	q < p			No	IoT devices
	Dynamic allocation of RACH resources Starting RA and preamble transmission	x	Access with probability p
	Access Class Barring (ACB) UE individual Access Class Barring Extended Access Barring Successful transmission Back logged IoT devices No IoT devices that could attempt access	Max retransmissions	attempt access IoT devices that could
	Remaining RA steps		Failure

15 Meriam Bouzouita, Yassine Hadjadj-Aoul, Nawel Zangar, Sami Tabbane, César Viho: « A random access model for M2M communications in LTEadvanced mobile networks», In « Modeling and simulation of computer networks and systems », Elsevier/Morgan Kofmann, pp. 577 -599 (2015) x x FOCUS ON THE ACB 16 Broadcast -BCCH Per-class ACB-factor p

  (s k , a k , r k , s k+1 )

	ARRIVAL REGULATION SYSTEM
	p k
	p k
	Deterministic approach
	Deals with continuous action space	s k+1	r k
	Solves the problem of overvaluation in value
	estimation
	Performs better than DDPG, PPO, …
			p k+1
		a k+1 (p k+1 )

23 Fujimoto, S.; van Hoof, H.; Meger, D. Addressing Function Approximation Error in Actor-Critic Methods. arXiv 2018, arXiv:1802.09477.

CHALLENGES AT THE ACCESS

What is the optimal number of contending devices

WHY USING DEEP REINFORCEMENT LEARNING?

The blocking factor calculation requires a good knowledge of the number of terminals willing to attempt access But it is not available in the network the state of the network is not observable It is possible to estimate this number, but this estimate is subject to noise.

The traffic pattern is very complex

Lack of data

We cannot use supervised learning Deep reinforcement learning techniques have been shown to be effective in making predictions even when the data is very noisy.

PERFORMANCE EVALUATION

Simulator:

Discrete event simulator developed from scratch Arrival process of IoT devices:

CONCLUSIONS

We proposed a mechanism to control the congestion of IoT access networks

We proposed a fluid model of the access

Allow determining optimal objective

We exploited recent advances in deep reinforcement learning, through the use of the TD3 algorithm

Simulation results show the superiority of the proposed approach

Despite the lack of accurate data

Future work:

Improve the estimation of the number of attempts.