Ali El

Jean-Paul Chaiban

Hussein Al

Haj Hassan

Paolo Dini
email: paolo.dini@cttc.es

Loutfi Nuaymi
email: loutfi.nuaymi@imt-atlantique.fr.

Roger Achkar
email: rachkar@aust.edu.lb

A El Amine
email: aelamine@laas.fr

Energy Optimization with Multi-Sleeping Control in 5G Heterogeneous Networks using Reinforcement Learning

Keywords: 5G, cellular networks, multi-sleep mode levels, energy consumption, reinforcement learning

The massive deployment of small cells in 5G networks represents an alternative to meet the ever increasing mobile data traffic and to provide very-high throughout by bringing the users closer to the Base Stations (BSs). This large increase in the number of network elements demands a significant increase in the energy consumption and carbon footprint followed by complex interference management. In order to address these challenges, we consider multi-level Sleep Mode (SM) where BS components with similar activation/deactivation times can be put to sleep. The deeper and higher energy efficient the SM is, the longer it will take the BS to activate, which might impose degradation in the Quality of Service (QoS). While this adds operational flexibility to the BS, it brings complex management to the operator. In this paper, we consider a heterogeneous network architecture where small cells can switch to different SM levels to save energy and reduce dropping rate. We propose a reinforcement learning algorithm for small cells that adapts their activities subject to service delay constraint. In this regard, the algorithm intelligently learns from the environment based on the co-channel interference, the cell buffer size and the expected cell throughput in order to decide the best SM policy. Numerical values show that important energy savings can be obtained with an acceptable dropping rate. Moreover, we show that while offloading users to the macro cell can significantly reduce their delay, dropping rate and the cluster energy consumption, it comes at a cost of decreasing the network energy efficiency up to 5 times compared with the case of no offload.

I. INTRODUCTION

The world-wide mobile traffic is witnessing an unprecedented rise triggered by the high speed internet and the advent of smart phone technology. Based on Ericsson's Mobility Report in November 2020, the mobile data traffic is expected to grow 7 times from 2020 to 2026 [START_REF] Ericsson | Ericsson Mobility Report[END_REF]. This remarkable growth in wireless data traffic is in an upward trend with no signs of slowing down. For instance, the number of 5G subscribers is expected to reach 2.8 billion by the year 2025 [START_REF] Ericsson | Ericsson Mobility Report[END_REF]. Addressing this expected increase in mobile data traffic is one of the main challenges of future cellular networks. Today, the system design of 5G is envisioned to support 1000 times increase in capacity and 100 times more connected devices than today's 4G networks [START_REF] Li | 5G network capacity: Key elements and technologies[END_REF]. In order to meet these challenges, next generation cellular networks are expected to massively deploy Small Base Stations (SBSs) as part of future 5G. These small cells are expected to be a promising solution to increase the area capacity in macro cell sites.

One of the main challenges of this architecture is the surge in the network energy consumption, placing network Energy Efficiency (EE) as one key pillar in future 5G networks [START_REF] Masoudi | Green mobile networks for 5G and beyond[END_REF]. According to the power consumption breakdown in [START_REF] Feng | Base station on-off switching in 5G wireless networks: Approaches and challenges[END_REF], the Base Station (BS) consumes the major part of the total network energy consumption which is about 60% to 80%. As such, reducing the energy consumption at this level will amount to significant reduction in the network total energy consumption. As the mobile traffic pattern fluctuates over both time and space, the ideal energy consumption of the BS is to scale with the traffic load. This is however not the case, as the BS energy consumption at zero load is about 50 -60% of its maximum energy consumption [START_REF] Buzzi | A survey of energy-efficient techniques for 5G networks and challenges ahead[END_REF]. This results in energy wastage and low EE especially for underutilized BSs. Among the different approaches proposed to reduce the energy consumption in cellular networks and to scale the BS energy consumption to traffic variations, BS sleep scheme is considered as one of the best methods since it does not require changes to current network architecture, and it is easy to implement [START_REF] Wu | Energy-efficient basestations sleep-mode techniques in green cellular networks: A survey[END_REF]. Many studies have focused on adapting the operation of the BS to save energy during off-peak periods [START_REF] Feng | Base station on-off switching in 5G wireless networks: Approaches and challenges[END_REF]- [START_REF] El-Amine | Analysis of energy and cost savings in hybrid base stations power configurations[END_REF]. However, completely shutting down or switching to sleep a BS for a long period of time can create coverage holes in the network, and thus impact the offered Quality of Service (QoS) to the served users (e.g., delay and packet dropping). Moreover, with the emergence of new services and the development of 5G, strict requirements are imposed on some services in terms of delay or dropped packets making the classical ON/OFF sleep unattractive to such services for the reasons mentioned above.

The recent finalized 5G New Radio (NR) 3GPP Rel-15 standard is designed to enable very low energy consumption. As such, new features for energy savings are introduced. Compared to 4G, NR supports Discontinuous Transmission (DTX) duration up to 160 ms as opposed to 1 ms in 4G networks [START_REF] Gpp | Study on new radio access technology: Radio access architecture and interfaces[END_REF]. This enables the implementation of new hardware deep sleep states in which the BS can lower its total energy consumption significantly. Embracing this framework, the work in [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF] proposed four distinct Sleep Mode (SM) levels. Each SM level is characterized by an activation/deactivation time and a minimum sleep duration. Compared to the classical binary sleep scheme, multi-level SM adds network flexibility to cope with the traffic requirements and allows more energy to be saved. In this framework, understanding the impact of SMs on the network load distribution, the resulting inter-cell interference, the related radio resource management procedures and finally the end-users' QoS emerges as a new paradigm in future 5G and beyond cellular networks.

In this paper, we study the impact of SMs on the overall 5G network performance under end-users' QoS constraint while taking into account the inter-cell interference. In particular, we consider a Heterogenious Network (HetNet) architecture where a Macro Base Station (MBS) is covering a cluster of small cells. While the MBS is always active to ensure coverage, the small cells can switch to different SM levels to mitigate QoS degradation while maximizing the network energy savings. In order to decide which SM to use, we leverage the off-policy Q-learning [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF], a well known reinforcement learning algorithm, and propose a distributed Qlearning algorithm that controls the small cells activities based on their level of interference, their expected throughput and their buffer size. In contrast to previous works, we focus on the recent multi-level SM model proposed in [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF]. We handle the traffic of users connected to SBSs in SM by a handover mechanism and study its impact on the overall network-level performance. We show that while offloading users to the active MBS reduces traffic drop rate, it comes at the expense of decreasing the network EE.

The rest of the paper is organized as follows. Section II reviews the existing work and presents the paper contributions. Section III describes the system architecture and the 5G multilevel SMs. In Section IV, we formulate our optimization problem and design the states, actions and cost function. We detail our proposed Q-learning algorithm in Section V. Finally, we discuss and conclude the results in Sections VI and VII.

II. RELATED WORK

The use of SM technique in Radio Access Networks (RANs) is among the most studied approaches to reduce their energy consumption [START_REF] Feng | Base station on-off switching in 5G wireless networks: Approaches and challenges[END_REF], [START_REF] Wu | Energy-efficient basestations sleep-mode techniques in green cellular networks: A survey[END_REF], [START_REF] Salahdine | A survey on sleep mode techniques for ultra-dense networks in 5G and beyond[END_REF]. We distinguish between two types of sleep schemes: binary SM level (ON and OFF) and multilevel SM. In binary SM, the BS shuts down completely when the traffic is low to save energy. Depending on the objectives, different sleep scheme based algorithms are proposed [START_REF] Peng | Stochastic analysis of optimal base station energy saving in cellular networks with sleep mode[END_REF]- [START_REF] Luo | Load based dynamic small cell on/off strategy in ultra-dense networks[END_REF]. In [START_REF] Peng | Stochastic analysis of optimal base station energy saving in cellular networks with sleep mode[END_REF], the authors investigated the potential of saving energy by switching-off macro BSs under coverage constraints using stochastic geometry. Compared with the baseline that keeps the BSs always active, their results achieved an EE gain of around 60%. In [START_REF] Ashraf | Sleep mode techniques for small cell deployments[END_REF], the authors highlighted the energy consumption behavior of small cell deployments while emphasizing on sleep mode solutions to reduce the network energy consumption. The authors discussed three different sleep mode strategies, which lead to energy savings between 10 and 60 %. The authors in [START_REF] Feng | Boost: Base station on-off switching strategy for green massive MIMO hetnets[END_REF] opted to maximize the EE of a HetNet by dynamically switching-off under-utilized SBSs along with user association and power control. As opposed to switching-off SBSs, the authors in [START_REF] Ren | A decentralized sleep mechanism in heterogeneous cellular networks with QoS constraints[END_REF] proposed to switch off macro BSs and offload their users to low-power SBSs. In [START_REF] Guo | Delay-constrained energyoptimal base station sleeping control[END_REF], the authors proposed an optimization mechanism based on delay-constrained energy-optimal BS sleeping policies. In [START_REF] Che | Dynamic base station operation in large-scale green cellular networks[END_REF], dynamic programming is applied to find the optimal BS ON-OFF policy given the on-grid energy price in order to minimize the on-grid energy cost purchased by the operator while guaranteeing the downlink transmission quality at the same time. In [START_REF] Jiang | Data-driven cell zooming for large-scale mobile networks[END_REF], a data-driven energy-saving framework is proposed. The authors adopted a heuristic switch-off policy based on the traffic conditions to maximize the network energy savings while satisfying the minimal service requirements for users. The authors in [START_REF] Luo | Load based dynamic small cell on/off strategy in ultra-dense networks[END_REF] proposed a heuristic loadbased switching-off algorithm for SBSs to optimize the energy savings in an ultra dense network while minimizing the power costs related to switching-off.

In the literature of binary sleep schemes, several works considered combining SM with Machine Learning (ML) approaches for traffic prediction and action decisions. For example, [START_REF] Vallero | Greener RAN operation through machine learning[END_REF] proved the effectiveness of binary sleep schemes and energy savings with ML to predict traffic in a two-tier HetNet powered by renewable energy and the smart grid. Instead of depending on the traffic forecast, conventional reinforcement learning techniques such as Q-learning have been applied in recent works for cell switching [START_REF] Miozzo | Switch-on/off policies for energy harvesting small cells through distributed Q-learning[END_REF], [START_REF] Miozzo | Distributed Qlearning for energy harvesting heterogeneous networks[END_REF]- [START_REF] Li | Tact: A transfer actor-critic learning framework for energy saving in cellular radio access networks[END_REF]. By interacting with the environment, the agent (i.e., the BS) learns how to adapt to the dynamic network changes in order to decide the optimal ON-OFF policy. In [START_REF] Sayed | Reinforcement learning for radio resource management of hybridpowered cellular networks[END_REF], we proposed a Q-learning algorithm to decide the number of active resource blocks based on the traffic load in a green cellular network to save energy. Fuzzy Q-Learning (FQL), a stochastic optimization tool that combines fuzzy inference system and Q-learning, has also been investigated in the problem of cell switching. For example, [START_REF] De Domenico | Reinforcement learning for interference-aware cell dtx in heterogeneous networks[END_REF] proposed a FQL algorithm to switch SBSs based on their buffer state, cell capacity and estimated spectral efficiency loss due to interference from nearby SBSs. For high order complexity networks with large state-action space, classical reinforcement learning are combined with value function approximation to derive the optimal policy. In this regard, linear function approximation and deep learning techniques are proposed to reduce the energy consumption by switchingoff cells in ultra-dense networks [START_REF] Ozturk | Energy optimization in ultra-dense radio access networks via traffic-aware cell switching[END_REF]- [START_REF] Li | Deep Q-learning based dynamic resource allocation for self-powered ultra-dense networks[END_REF].

Unlike the previous works with binary scheme, recent works have considered mutli-level SM based on the model proposed by GreenTouch in [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF] that introduces four distinct SM levels [START_REF] Salem | Reinforcement learning approach for advanced sleep modes management in 5G networks[END_REF]- [START_REF] El-Amine | Reinforcement learning for delay-constrained energy-aware small cells with multi-sleeping control[END_REF]. The authors in [START_REF] Salem | Reinforcement learning approach for advanced sleep modes management in 5G networks[END_REF] proposed a Q-learning algorithm to determine the best duration of time a BS can spend in a particular SM level in order to minimize the network energy consumption subject to the BS (de)activation latency and users service requirements constraints. In [START_REF] El-Amine | A distributed Q-learning approach for adaptive sleep modes in 5G networks[END_REF], we proposed a distributed Q-learning algorithm to control the sleep depth of the BSs in the network to maximize the trade-off between energy savings and network latency. In [START_REF] El-Amine | Locationaware sleep strategy for energy-delay tradeoffs in 5G with reinforcement learning[END_REF], we leveraged the multi-SM levels to decide the SM depth of the BS given the location of the users in the network and their velocities towards neighboring cells. The authors in [START_REF] Masoudi | Reinforcement learning for traffic-adaptive sleep mode management in 5G networks[END_REF] proposed an online traffic-adaptive BS energy saving algorithm with Q-learning that utilizes the different SM levels subject to packets drop-ping. In [START_REF] El-Amine | Reinforcement learning for delay-constrained energy-aware small cells with multi-sleeping control[END_REF], we proposed a distributed Q-learning algorithm in a two-tier HetNet that controls the activity of the SBSs while considering the co-channel interference between neighboring cells in order to maximize the energy savings of the network subject to the service delay constraint. Different from the model of [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF], others works such as [START_REF] Jang | Base station switching and sleep mode optimization with lstm-based user prediction[END_REF] studied multi-level sleep under a different model. The authors proposed a heuristic to solve their Lyapunov optimization formulation problem with LSTM-based user predication.

Despite the exhaustive works on binary sleep scheme in [START_REF] Peng | Stochastic analysis of optimal base station energy saving in cellular networks with sleep mode[END_REF], [START_REF] Feng | Boost: Base station on-off switching strategy for green massive MIMO hetnets[END_REF], [START_REF] Guo | Delay-constrained energyoptimal base station sleeping control[END_REF]- [START_REF] Luo | Load based dynamic small cell on/off strategy in ultra-dense networks[END_REF], this scheme is undesirable for future 5G services with hard constraints. Compared to the existing works with multi-level SM in [START_REF] Salem | Reinforcement learning approach for advanced sleep modes management in 5G networks[END_REF]- [START_REF] Jang | Base station switching and sleep mode optimization with lstm-based user prediction[END_REF], this paper provides additional insights on understanding advanced SM levels of [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF]. In contrast to [START_REF] Salem | Reinforcement learning approach for advanced sleep modes management in 5G networks[END_REF], [START_REF] Masoudi | Reinforcement learning for traffic-adaptive sleep mode management in 5G networks[END_REF] that consider only one BS, we study a 3GPP scenario where we have a cluster of SBSs covered by a MBS. Different from our works in [START_REF] El-Amine | A distributed Q-learning approach for adaptive sleep modes in 5G networks[END_REF], [START_REF] El-Amine | Locationaware sleep strategy for energy-delay tradeoffs in 5G with reinforcement learning[END_REF] that consider a homogeneous network of BSs, in this paper we include the co-channel interference in the decision making process. Different from our previous work in [START_REF] El-Amine | Reinforcement learning for delay-constrained energy-aware small cells with multi-sleeping control[END_REF], in this paper we further handle traffic offloading and investigate the system performance on a network-level scale. Finally, the work in [START_REF] Jang | Base station switching and sleep mode optimization with lstm-based user prediction[END_REF] proposes a sub-optimal heuristic algorithm. We summarize in tables I and II the above literature for binary and multilevel sleep schemes, respectively. Moreover, we highlight the limitations that we addressed in multi-level SM.

A. Paper contributions

In this paper, we investigate the energy/delay trade-off problem in a HetNet architecture where a macro cell covers a cluster of small cells. We propose an intelligent cell switching mechanism using Q-learning where small cells can switch to different SM levels to save energy while maintaining a good QoS depending on the service use case. We leverage the recent multi-level SM model proposed in [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF] that allows flexibility in controlling the state of the BS. Compared to existing work, this paper provides the following key insights:

• We propose an intelligent distributed controller based on reinforcement learning that chooses the best decision of SM level in order to minimize the energy consumption and data lost in the network. Due to the limited size of the defined state and action spaces, we rely on a Q-learning model that is proven to be effective and able to manage the computational requirements needed for our problem. • In contrast to [START_REF] Salem | Reinforcement learning approach for advanced sleep modes management in 5G networks[END_REF]- [START_REF] Masoudi | Reinforcement learning for traffic-adaptive sleep mode management in 5G networks[END_REF] that consider interference-limited models, we study a more realistic 3GPP scenario composed of a cluster of SBSs covered by a MBS where the co-channel interference between SBSs is regarded in the decision making process. • In contrast to our previous work in [START_REF] El-Amine | Reinforcement learning for delay-constrained energy-aware small cells with multi-sleeping control[END_REF], we handle the traffic of users when their connected SBS is turned off by a handover technique to the MBS. In this regard, we study the impact of this offloading on the overall network EE.

MBS

Fig. 1: A HetNet architecture with a macro cell and a cluster of small cells each with a distributed local controller. focus on either network or BS level, we study the system performance on both levels. Our numerical results reveal a lower traffic drop rate when users offload from their inactive SBSs, however at the expense of decreasing the network EE.

III. SYSTEM MODEL

A. Network Model

Following the scenario described by 3GPP in [START_REF] Gpp | 3GPP TR 36.872, Technical Specification Group Radio Access Network; Small cell enhancements for E-UTRA and E-UTRAN-Physical layer aspects (Release 12)[END_REF], we consider a two-tier HetNet architecture where a MBS has four SBSs with overlapping coverage regions as shown in Fig. 1. The MBS serves as a basic constant coverage and it is responsible for control signaling and data services. The SBSs on the other hand are densely deployed in the coverage area of the macro cell to create a local hotspot to increase the capacity in the designated area, thus they serve as capacity cells. Moreover, the macro cell and the small cells are operating at different frequency bands, hence they do not interfere on each others.

In order to enhance the network EE, the SBSs can dynamically switch to different SM levels while the MBS stays active for maintaining coverage. This may not only save energy and limit the CO 2 footprint, but it may also enhance the system capacity by limiting the interference between neighboring cells. A distributed local network controller (e.g., Software Defined Network (SDN)) is in charge of managing the activity of these cells (see Fig. 1). We note that for binary SM levels (ON and OFF), finding the optimal set of active SBSs is a combinatorial problem with high computational complexity. In this work, we further add another layer of complexity by considering multi-level SMs. In order to solve this problem, a local controller manages a given SBS by means of interacting with the environment in order to map what to do (action) when faced with a current situation (state). This method is known as Q-Learning [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF]. In particular, each SBS, at every iteration, exploits the expected average interference at its users from neighboring cells and the activation decision taken in Determine the optimal proportion of sleep MBSs [START_REF] Ashraf | Sleep mode techniques for small cell deployments[END_REF] HetNet: macro and small cells Heuristic algorithms Three control algorithms: (1) small cell driven;

(2) core network driven and (3) UE driven

Investigate the network energy efficiency with sleep mode algorithms for SBSs [START_REF] Feng | Boost: Base station on-off switching strategy for green massive MIMO hetnets[END_REF] HetNet: macro and small cells Convex Optimization Switch-off underutilized SBSs and user association Maximize the network energy efficiency [START_REF] Ren | A decentralized sleep mechanism in heterogeneous cellular networks with QoS constraints[END_REF] HetNet: macro and small cells Heuristic algorithm Switch-off MBSs and offload users to SBSs Minimize the network energy consumption subject to QoS constraint [START_REF] Guo | Delay-constrained energyoptimal base station sleeping control[END_REF] Single BS Queueing theory Hysteresis sleep mode scheme Study the energy-delay tradeoff [START_REF] Che | Dynamic base station operation in large-scale green cellular networks[END_REF] Homogeneous network powered by green energy and power grid Dynamic programming and Stochastic geometry BS switching-off scheme Obtain optimal BS on/off policy that reduces the on-grid energy cost [START_REF] Jiang | Data-driven cell zooming for large-scale mobile networks[END_REF] Large scale homogeneous network Heuristic algorithm Switch-off BSs according to traffic conditions Maximize energy savings while guaranteeing minimum service requests [START_REF] Luo | Load based dynamic small cell on/off strategy in ultra-dense networks[END_REF] Ultra dense network Heuristic algorithm Load based switching-off algorithm Optimize energy savings while minimizing the energy cost from switching users to other BSs [START_REF] Vallero | Greener RAN operation through machine learning[END_REF] HetNet: macro and small cells Supervised ML Sleep action decisions based on traffic prediction Investigate the effectiveness of sleep mode combined with ML approaches for traffic forecast [START_REF] Miozzo | Switch-on/off policies for energy harvesting small cells through distributed Q-learning[END_REF], [START_REF] Miozzo | Distributed Qlearning for energy harvesting heterogeneous networks[END_REF] HetNet: macro and small cells.

Small cells are solely powered by green energy Q-learning Switch-off SBSs based on harvested energy, traffic load and battery level Propose a distributed Q-learning approach that finds the optimal policy that improves the system performance (energy efficiency, drop rate, throughput) [START_REF] Abubakar | Q-learning assisted energy-aware traffic offloading and cell switching in heterogeneous networks[END_REF] HetNet: macro and small cells Q-learning Switch-off SBSs and offload users to the macro cell Study the impact of offloading users to neighboring cells on the energy consumption [START_REF] Li | Tact: A transfer actor-critic learning framework for energy saving in cellular radio access networks[END_REF] Large scale homogeneous network Transfer learning with reinforcement learning BS switching-off scheme Propose a transfer actor-critic algorithm to speed up the learning process [START_REF] Sayed | Reinforcement learning for radio resource management of hybridpowered cellular networks[END_REF] Single BS powered by green energy and smart grid Q-learning Switching-off resource blocks Decide the number of active resource blocks based on the traffic load to save energy [START_REF] De Domenico | Reinforcement learning for interference-aware cell dtx in heterogeneous networks[END_REF] HetNet: macro and small cells Fuzzy Q-learning (FQL) Switch-off SBSs according to the cells' buffer, throughput and spectral efficiency loss Propose a FQL approach to find the optimal policy that reduces energy consumption subject to QoS constraint [START_REF] Ozturk | Energy optimization in ultra-dense radio access networks via traffic-aware cell switching[END_REF]- [START_REF] Li | Deep Q-learning based dynamic resource allocation for self-powered ultra-dense networks[END_REF] HetNet: macro and small cells Deep Q-learning (DQL) Switch-off SBSs Propose a DQL approach to find the optimal policy that reduces energy consumption subject to QoS constraint the previous iteration. Then, the local controller uses this information along with the SBS buffer and traffic load levels to control its activity, as we detail later in this section.

B. Multi-Level Sleep Modes

In [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF], GreenTouch identified four distinct SM levels by grouping sub-components with similar transition latency when being activated or deactivated. The presented model enables to quantify the power consumption of the BS in each of the four SMs. These are:

-SM 1: It considers the shortest time unit of one OFDM symbol (i.e., 71µs) comprising both deactivation and reactivation times. In this mode only the power amplifier and some processing components are deactivated. -SM 2: It corresponds to the case of sub-frame or Transmission Time Interval (TTI) (i.e., 1 ms). In this SM, more components enter the sleep state.

-SM 3: It corresponds to the frame unit of 10 ms. Most of the components are deactivated in this mode. -SM 4: This is the deepest sleep level. Its time unit corresponds to the whole radio frame of 1s. In this SM level, the BS enters a standby mode where it is out of operation but retains wake-up functionality. Higher energy savings can be achieved when switching BSs to a deeper SM, since more components will be deactivated. However, this will be associated with longer transition latency that may impact the QoS of the system. In Table III, we present the SM levels characteristics for a 2 × 2 MIMO SBS. For simplicity and convenience, in this paper we consider a SISO system model. Even though a 2×2 MIMO system can enhance the achievable rate which in turn can improve the network performance, it consumes more energy. Hence the overall gain is relatively insignificant and out of the scope of this paper. For MIMO systems with a large number of antennas, key parameters such as spectral efficiency, interference and energy consumption may be greatly affected. However, this case is not very realistic for small cells since there are likely not a lot of users in the studied hotspot.

Along with SM and users' dynamics, the BS has to wake up periodically to send signalling bursts. In contrast to Long Term Evolution (LTE) systems where each antenna must transmit every 0.2 ms a unique Cell Reference Signals (CRS) for channel quality estimates and mobility measurements among other Synchronization Signaling (SS), no CRS is required for 5G [START_REF]3GPP TR 36.814, Evolved Universal Terrestrial Radio Access (E-UTRA); Further advancements for E-UTRA physical layer aspects (Release 9)[END_REF]. Instead, SS and Physical Broadcast CHannel (PBCH) are transmitted in SS/PBCH block periodically. It has been agreed in Third Generation Partnership Project (3GPP) [START_REF]3GPP TR 36.814, Evolved Universal Terrestrial Radio Access (E-UTRA); Further advancements for E-UTRA physical layer aspects (Release 9)[END_REF] that this periodicity can be set to any value among [5, 10, 20, 40, 80, 160 ms]. With these values, SM 4 cannot be used. Hence, we limit our work to the first three SM levels.

C. Traffic Model

In our model, we consider connected users continuously requesting packets at each time slot. The users have a minimum rate requirement (R min ). We use the term data block (DB m, t ) to describe the total packets requested by the users served by BS m (MBS or SBS) at time slot t. Given a cell bandwidth B m and the noise spectral density N 0 , we can calculate the rate of user u served by cell m (r m,u ), the average cell throughput (r m ) and the number of bits served by cell m (y m ) as follows:

r m,u = B m • log 2 1 + P m • h m,u N 0 • B m + m m P m • h m ,u (1)
r m = K m u=1 r m,u K m (2)
y m = T • r m (3)
where P m is the cell transmit power, h m,u is the user channel gain served by BS m, K m is the total number of users served by cell m and T is the duration of a time slot. The average throughput of the macro cell does not include the interference term since we are considering only one macro BS. At the beginning of each time slot, a data block is buffered and stacked in the designated cell buffer. The size of the data block (number of bits) is determined by the number of packets requested by the users served by this cell. We assume that the size of each packet (λ) is related to the minimum user rate requirement R min as follows:

λ = T • R min (4)
In this work, we consider the same rate requirement for all users. Thus, the size of each packet is the same. As a result, the size of the data block depends on the number of users served by the corresponding cell. Furthermore, we characterize the data block by a Time-To-Live (TTL), L (ms), which is the lifetime of the block above which the remaining data in the block are lost. If a data block i is requested at time slot t i , then its remaining lifetime at time slot t can be obtained as follows:

d i m, t = t i + L -t (5)
When a data block i is being served to the users at time slot t, its remaining data can be obtained as follows:

DB i m, t =          max 0, λ • K m, t - t j=t i y m, j if d i m, t < L 0 otherwise (6)
where y m, j is the number of bits served by cell m at time slot j. Finally, we can express the buffer of a BS m at time slot t as the set of data blocks stacked in the buffer as follows:

B m, t = {DB i m, t |d i m, t > 0} (7)
To better understand the system traffic dynamics, we illustrate in Fig. 2 the evolution of a cell buffer with time as well as the lifetime of each arriving data block. Since the users continuously request packets, at each time slot a new DB i m arrives to the cell buffer m. We assume that all users have the same R min , hence the DB i m 's have the same size. Furthermore, we consider in this example a small cell that is switching between active mode and any SM level. During the first time slot, the first data block (DB 1 m ) arrives. Since the BS is in SM, this block stays in the buffer until the BS reactivates during the next time slot and part of DB 1 m is served while another DB m arrives (DB 2 m ). At this time slot, the buffer of this cell following eq. ( 7) is expressed as B m, t+T = {DB 1 m, t , DB 2 m, t+T }. At the end of the third time slot, the remaining of DB 1 m is lost because it reached its lifetime before being served, and the buffer is updated to B m, t+3T = {DB 2 m, t+T , DB 3 m, t+2T , DB 4 m, t+3T }. During each time slot, if the BS is active, the amount of served data depends on the cell throughput that is directly related to the level of interference with neighboring cells. Thus, the states of the SBSs have an impact on the latency added to the system. On the one hand, the deeper the SM level is, the more time the user will have to wait until the SBS reactivates. On the other hand, if all the SBSs are active, the level of interference is the highest resulting in lower throughput.

D. Users offloading

As previously mentioned, when an SBS switches into an SM level, its users experience service delay and possible data loss. In order to limit these degradations in QoS, users served by the cluster can offload to the MBS that is always active. In this regard, we propose to offload a user, u , served by the cluster to the MBS if the latter has enough radio resources (i.e., bandwidth) to guarantee the user minimum rate requirement (R min ). Hence, the following user offloading condition: where r m is the new average throughput of BS m (in this case m =MBS) that considers the offloaded user u .

r m,u = r m K m + 1 ≥ R min (8)

IV. PROBLEM FORMULATION

In this work, we consider the problem of minimizing the SBSs energy consumption and the data lost (dropped). The problem can be formulated as follows:

min a ∈ {Act.,SM1,SM2,SM3} M m=1 N t=1 (1-w)• p m, t (a)+w•l m, t (a) (9)
where M is the number of SBS and N is the total number of time slots. p and l represent the SBS power consumption and the amount of data lost, respectively after an action a has been taken. w is a weighting parameter that prioritizes between the power consumption and dropping rate.

In order to reduce the cluster's energy consumption and its data lost, we model the above problem as a discrete-state Markov Decision Process (MDP) problem, where a SBS m in state s m (t) takes an action a m (t) then moves to another state s m (t + 1). We denote by S = {R, B, I l } the state space of the SBS, where R, B and I l are the sets related to the cell throughput, buffer state, and the estimated spectral efficiency loss (bit/s/Hz) due to the interference from nearby active SBSs, per SBS, respectively.

At each time slot, the local controller decides an action a m (t) ∈ A = {Active, SM1, SM2, SM3} for a SBS after observing its state. In order to evaluate the cost associated with each action-pair, we use the cost function that takes into account the power consumption of the SBS m , and the users' QoS:

c m (s, a) = (1 -w) • p m (s, a) + w • l m (s, a) (10)
In order to solve the above MDP, we first need to derive the transition probabilities between the system states and the corresponding state-action cost. Then, a backward dynamic programming needs to be performed to evaluate the optimal value function, in order to derive the optimal policy. However, this requires a complete knowledge of the system information which is not practical for real-time applications. To alleviate the non-causal system information, we rely on Q-learning to find the optimal policy by exploring and interacting with the current state of the system. Due to the limited number of stateaction pairs, we have a fairly small lookup table that traditional Q-learning algorithm can handle. Thus, in this paper, we do not need to use advanced reinforcement learning tools such as Deep Q-learning, and we focus on the classical Q-learning algorithm.

V. ONLINE INTERFERENCE-AWARE BS SLEEPING ALGORITHM

A. Preliminaries on Reinforcement Learning

Distributed Q-learning is an online optimization technique that aims at controlling multi-agent systems, i.e., a system featuring M SBSs that take decisions (select the appropriate SM level) in an uncoordinated fashion [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF]. Each SBS has to learn independently a policy through real-time interactions with the environment. Q-learning finds the optimal policy in the sense that it maximizes the expected value of the total reward (Q-value) over all successive episodes. The agents (i.e., BSs) have a partial view of the overall system, and their actions may differ since the users are unevenly distributed over the network. In particular, the decision of a BS to choose a SM level is affected by how many users it has to serve, and on how delay-tolerant these users are.

In Q-learning, each agent takes an action a t m from an action set A, then moves to a new state s t+1 m while receiving a reward r t m (in this work, the reward function is the cost function in eq. ( 10)). This reward is then used to update the Q-value locally, Q(s t m , a t m ), indicating the level of convenience of selecting action a t m when in state s t m . The Q-value is updated following the update rule:

Q(s t m , a t m ) ← Q(s t m , a t m )+ α[r t m + γ max a ∈ A Q(s t+1 m , a t+1 m ) -Q(s t m , a t m )] ( 11
)
where α is the learning rate that represents the speed of convergence, and γ ∈ [0, 1] is the discount factor that determines the current value of the future state costs.

During the learning phase, each agent selects the corresponding action based on the -greedy policy, i.e., it selects with probability 1the action associated with the maximum Q-value, and with probability selects a random action:

a t m =        argmax a ∈A Q(s t m , a t m ), if y > rand(A), otherwise m = 1, . . . , M. ( 12
)
By implementing the -greedy policy, the BS would have explored all possible actions and avoided local minima. For more details on Reinforcement Learning (RL) and Q-learning the reader is referred to, e.g., [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF].

B. Proposed Q-Learning Algorithm

At time slot t, the local state of the small cell m is s m (t) = {r m (t), b m (t), i l, m (t)}. where r m , b m and i l, m are the states of the throughput, buffer and interference of SBS m, respectively. Their values are quantized into discrete levels. Indeed, increasing the number of levels leads to more accurate modeling of the system but it also increases the system complexity. In our work and in order to balance between performance and complexity, we quantized R, B and I l into into 4, 3 and 3 levels, respectively. Due to the limited number of stateaction pairs, the convergence of the cumulative reward towards the optimal value is not compromised. Therefore, there is no need for function approximations as that performed by Deep Q-Learning (DQL). In order to explore different combinations of states, we make the users move inside their serving cell at each time slot. We define an episode as a fixed simulation run where the users in motion request data continuously with a given minimum rate requirement. During this episode, the users move randomly in the cluster. However, they remain in the same SBS radius area to avoid offloading inside the cluster. Allowing users to freely move inside the SBS cells makes the environment more dynamic, thus an impact on the states of each cell. As a result, the algorithm explores different states to find the optimal policy that minimizes the cost function in [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF]. After the training phase is complete, a look-up table is generated mapping for every state the best corresponding action. Thus, when the system goes online to operate in realtime, the appropriate action to take for each state is fetched from that table. Since the total number of states is small, the time required to fetch this action is negligible. The Q-Learning algorithm is described in Algorithm 1.

VI. SIMULATION AND RESULTS

In this section, we demonstrate the performance of our proposed Q-learning SBS controller using MATLAB. Our simulator mimics a scenario of one macro BS serving 100 users (unless stated otherwise) covering a cluster of 4 SBSs with 30 users. The macro and cluster users are randomly distributed inside the macro cell and the cluster, respectively. We obtain the results over 200 independent runs, simulating 10 seconds of network activity. The network parameters for simulations were set according to 3GPP specifications [START_REF]3GPP TR 36.814, Evolved Universal Terrestrial Radio Access (E-UTRA); Further advancements for E-UTRA physical layer aspects (Release 9)[END_REF]. We detail in Table IV the parameters used in our simulation.

A. Convergence analysis

We analyze the convergence of the proposed Q-learning algorithm for different values of w during the offline training phase in figures 3, 4 and 5. In this phase, the system is fed with a training episode that imitates the dynamics of the network. Furthermore, we disable users offloading in the training phase. During this episode, we consider that the users require a service with R min = 1 Mbit/s and are moving inside the cluster in order gather enough experience by covering as much as visiting states as possible. Based on their partial view of the environment (cell buffer size, cell throughout and estimated spectral efficiency loss), each SBS takes an action from the Visit state s m .

7:

Select an action a m using -greedy rule in [START_REF] Salahdine | A survey on sleep mode techniques for ultra-dense networks in 5G and beyond[END_REF]. Update the Q-value q m (s, a) from [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF].

11:

end for 12: end while 13: end procedure 1: procedure Online 2: for u ∈ cluster of SBSs do

3: if r M BS,u ≥ R min then 4:
Offload u to the MBS.

Q m (s, a) = Q T m (s, a) 9:
Run Q-Learning. 10: end for 11: end procedure Fig. 3: Evolution of the Q-table with the number of epochs.

action set in A to minimize the cost function in [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF]. By repeating this episode a number of epochs, the SBSs learn how to map their observable states to actions. We present in Fig. 3 the convergence of the algorithm to a stable best action Q-Value sum that varies for different QoS trading factor w. This results from the complexity of finding the optimal trade-off for a given value of w when both energy consumption and data loss are considered. For example, when w = 0 or 1, the equation in [START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF] energy consumption or the amount of data lost, respectively. In the first case, the agent (i.e., small cell) ignores the co-channel interference between the cells, the size of the data in the buffer and the possible achieved throughput, and focuses only on reducing the energy consumption. In the second case, the SBS ignores the power consumption resulted from the actions chosen and focuses only on serving all the packets. In these special cases, the SBS is able to find the optimal policy faster compared to the other values of w where the agent requires more iterations to capture the trade-off that is influenced by different variables such as interference, buffer size, throughput and energy consumption. Regarding the different asymptotic values of the convergence, it depends on the reward function.

For example, when w = 1, the reward function converges to zero for zero data loss. However, when w = 0 the energy consumption decreases to the lowest power saving value that is greater than zero. In Fig. 4, we illustrate the evolution of the cluster's EE. The figure shows a decrease in the EE as the weighting parameter w increases. This is because the small cell tends to stay in SM for a longer duration of time when w is small resulting in lower energy consumption and higher throughput due to lower interference from neighboring cells in SM. This is opposed to the case of high values of w that not only increases the energy consumption, but also the interference between the small cells from staying active for a longer duration of time. The activity of these cells can be depicted from the heatmap in Fig. 5 showing the percentage of time the small cell is in SM. As expected, the percentage of time the SBS switches to SM decreases as w increases. For instance when w approaches zero, the SBSs are in SM for the longest duration of time (above 95%). As w approaches one, the states of the SBSs shifts to active mode to decrease the data lost. We note that in order to minimize the data lost for this setup (i.e., R min = 1 Mbit/s), the SBSs can still switch to SM almost half of the time. This will be discussed in more details in the next section.

B. Policy analysis

In this section, we discuss and evaluate the performance of the macro BS and the hotspot cluster. We compare two cases: 1) allowing users in the cluster to offload at the beginning of the first time slot, if the rate requirement condition in eq. ( 8) is met and 2) no users offloading. First, we report in Fig. 6 the different policies the SBSs take by changing the QoS trading factor (w) and for different users rate requirements without offloading. We observe that the cluster tends to save more energy when w approaches zero by switching to the deepest allowed SM level most of the time, while a gradual shift to active mode is observed when increasing w. On the other hand and by increasing the users rate requirement, the size of the data blocks requested in each time slot increases. This requires the SBS to stay in active mode for a longer period of time as can be depicted in the results in order to serve these blocks in time (i.e., within the delay constraint -TTL) and minimize the data lost. For the case when w = 1, each SBS is required to stay active for a minimum period of time to minimize the data lost. This can be observed more clearly for R = 0.5 and R = 1 Mbit/s. Once the goal of reducing data lost is satisfied (i.e., l = 0), the SBS switches randomly to any of its defined actions. This explains the almost uniform distribution state modes for SM1, SM2 and SM3 after successfully minimizing the data lost. For instance, it is enough to stay active 40% of the time to minimize the data lost for R = 1 Mbit/s, while the SBS is required to stay active 90% of the time for R = 2 Mbit/s. In figures 7 to 11, we evaluate the performance of the algorithm for different policies by varying w. We compare the performance between the macro cell and the cluster for the two cases (with and without users offload) and we set R min = 1 Mbit/s. From Fig. 7, when cluster users are allowed to offload, the service delay from the SBSs to their users drops to zero while the macro users delay increase from 5.5 ms to almost 7.5 ms. This is because all 30 cluster users are offloaded as it is illustrated in Fig. 12 where 100 macro users correspond 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 to a macro traffic load of 63%. Similarly and as depicted in Fig. 8, the normalized power consumption of the cluster drops when all users are offloaded since the small cells switch to the deepest SM level when there are no users to serve. On the other hand, the power consumption of the macro cell increases as it traffic load rises from 100 users (63% traffic load) to 130 users (81% traffic load). We note that the small and macro cells power consumption are normalized independently to the case where the former is always on (that is the MBS without SM) and the latter (SBS) is at full load. Finally, we compare our proposed multi-level SM scheme with a baseline that considers only SM3. We will note this scheme as binary sleep scheme. We illustrate the case with no offload to highlight the impact of SM on the cluster delay and energy consumption. The results show that our multi-level SM scheme outperforms the binary sleep scheme. This is due to the added flexibility with multi SM levels. Even though the average delay of the requested data in Fig. 7 falls under the delay constraint limit of 100 ms for all the discussed cases, around 80% of these requested data are dropped (lost) for small values of w when the users stay connected to the cluster as shown in Fig. 10. This is because the average delay calculated does not take into account the dropped packets. As for the case of macro users or when the cluster users are offloaded, there are no dropped packets as the macro cell, that is always active, is able to serve the packets (data blocks) to its users under the TTL constraint.

In the above discussion, it is clear that offloading the users to the MBS improves the delay and minimizes the loss of packets for the users. However, this comes at a cost. First, even though the power consumption of the small cells is reduced when all of their users are offloaded, the increase in the macro cell power consumption due to the traffic increase is more significant than the energy savings from the small cells. This is because the macro cell consumes a lot more energy than the small cell, which can be verified in Fig. 9. Fig. 9 shows the overall network power consumption during the observation time. It is clear that when the users offload to the macro cell, the network power consumption increases by 9.3% compared to the no offload scenario. We also notice that when the users offload, the network power consumption is almost constant regardless of w. This is because our parameters setup (100 macro users and R min = 1 Mbit/s) allows all cluster users to offload while guaranteeing enough radio resources from the MBS. Hence the SBSs are always in SM3 since they are not serving any users. On the other hand and for the other scenario (without offload), the network power consumption is slightly increased with w due to the increase in the cluster power consumption. This is because the power consumption of the macro cell dominates the network power consumption. Second, the idea of having a hotspot is to boost the capacity in a designated area, hence when the users are offloaded from the cluster their average rate decreases down to 10 times as shown in Fig. 11 because the macro cell that is initially serving 100 users spreads over an area of radius 500 meters providing less throughput than the cluster serving 30 users spread around an area of radius 40 meters per small cell. The reason behind having a high average throughput when w is small is due to the interference limitation from having more cells in SM which improves the SINR. This decrease in the user throughput along with the increase in the overall network energy consumption results in a decrease in the network EE as will be discussed in the next section.

C. Network performance assessment

In Fig. 12, we illustrate the number of offloaded users as a function of the macro cell traffic load and for different user rate requirement. Since, a user does not offload unless the macro cell guarantees his/her minimum rate requirement, less users are expected to offloaded as the macro traffic load or the rate requirement increases. In particular, when R = 2 Mbit/s, no users are offloaded since the macro cell cannot guarantee such rate to the offloaded users.

In Fig. 13, we present the network EE and compare the two scenarios (with and without offload). As previously explained, the users in the cluster enjoy higher throughput. Moreover since the SBSs, alternating to different SM levels, consume less energy than the active MBS, the network EE for the no offload scenario is higher, up to 5 times when w = 0, compared with the offload scenario. As the macro traffic cell increases less cluster users are prone to offload until the macro cell is fully loaded and no users are offloaded. This explains why for high macro traffic load, the network EE for the two scenarios meet. In figures 14 and 15, we illustrate the network data lost and delay as a function of the macro traffic load, respectively by averaging over all network's users. As noted in the previous section, offloading users from the small cells to the macro cell enhances their experience in terms of reduced data lost and service delay for R min = 1 Mbit/s. This can be clearly observed since the macro cell does not switch off as in the case of the small cells. Furthermore, a user does not offload to the macro cell unless enough radio resources are provided to him/her. Hence, the QoS is not affected.

VII. CONCLUSION

In this paper, we study the problem of energy saving optimization in 5G BSs by implementing multi SM levels. We consider a HetNet architecture where a MBS covers a cluster of SBSs. Given the distinct power and time consumption characteristics of these modes, we propose a distributed Qlearning algorithm that is able to learn the best policy to control the activity of the SBSs in order to minimize the energy consumption and data dropping. Thus, we investigate the trade-off between energy saving and QoS. Results show that the cluster energy consumption can be significantly reduced if the operator prioritizes energy savings at the cost of higher dropping rate. Moreover, we analyze the impact of offloading users from the small cells to the macro cell on the network performance. We show that under low traffic load, offloading users to the MBS reduces significantly their service delay and dropping rate. However, even though this reduces the energy consumption of the cluster, it comes with an increase in the network energy consumption and a decrease in the average user throughput resulting in a drop in the network EE. This could open the door to investigate load management algorithms for small cells solely equipped with renewable energy sources and batteries where EE could be scarified for a short duration of time to maintain continuous users' connectivity.

  Data blocks arrival and departure to and from the cell buffer. Arriving data blocks lifetime.

Fig. 2 :

 2 Fig.2: An example of the flow of data blocks requested by the users served by a small cell and their life time.

Algorithm 1 :

 1 Online Interference-Aware BS Sleeping 1: procedure Training (Q T m (s, a)) 2: Initialize the positions of the users and q m (s, a) = 0, ∀m ∈ M, ∀s ∈ S and ∀a ∈ A. 3: Set the weight w, and the average user velocity. 4: while Learning do 5:for m ∈ M do 6:

Fig. 4 :

 4 Fig. 4: Evolution of the cluster energy-efficiency with the number of epochs.

Fig. 5 :

 5 Fig. 5: Heatmap of the small base stations activities for different energy saving-data lost weights in the reward function with the number of epochs. The intensity of the colors represents the percentage of switch-off action (i.e., SM1, SM2 or SM3).

Fig. 6 :Fig. 7 :

 67 Fig. 6: Percentage of SMs for different users' minimum rate requirement.

Fig. 8 :

 8 Fig.8: Normalized power consumption assessment as a function of w.

Fig. 9 :Fig. 10 :

 910 Fig. 9: Network overall power consumption assessment as a function of w.

Fig. 11 :

 11 Fig. 11: Average user throughput assessment as a function of w.

2 Fig. 12 :

 212 Fig. 12: Number of offloaded cluster users as a function of Macro traffic load.

1 Fig. 13 : 1 Fig. 14 :

 113114 Fig. 13: Network energy efficiency assessment.

1 Fig. 15 :

 115 Fig. 15: Network delay assessment.

•

  We analyze the performance of the proposed algorithm for different use cases and network parameters through extensive simulations. In contrast to previous works that

	Local
	Controller
	action		state
	Cluster of local hotspot	Interference	Buffer	Rate

TABLE I :

 I Overview of binary switch-off strategy. MBSs to save energy Derive expressions of average coverage probability and users' power consumption.

	Ref.	Architecture	Tool	Strategy	Contributions
	[13]	HetNet: macro and micro cells	Stochastic geometry	Switch-off

TABLE II :

 II Overview of multi-level sleep strategy.

	Ref.	Architecture	Strategy	Contributions	Limitations
	[35] Large scale homogeneous network	Adaptive sleep mode based on	Determine the optimal BS switching solution	Heuristic algorithm
			user prediction	while ensuring a stable data queue state
				Determine the best duration of time a BS can spend in a
	[30]	Single BS	Advanced SM management	particular SM level that minimizes the network energy	Interference-limited
					Only one BS
				consumption subject to QoS constraint
	[31] Large scale homogeneous network	Adaptive sleep mode	Control the sleep depth of the BSs that maximizes the	One type of BS
				tradeoff between energy savings and network latency	Ignore users offloading
					Interference-limited
	[32] Large scale homogeneous network	Location-aware sleep strategy	Determine the BS sleep depth according to neighboring	One type of BS
				approaching users
					Focus only on handovers
	[33]	Single BS	Traffic adaptive BS energy saving algorithm	Determine the BS sleep depth according to BS traffic	Interference-limited
					Only one BS
	[34]	HetNet: macro and small cells	Switch-off SBSs according to the cells' buffer,	Find the best sleep policy that maximize the energy	Ignore users offloading
			throughput and spectral efficiency loss	savings of the network subject to QoS constraint

TABLE III :

 III 2x2 MIMO SBS Sleep Modes Characteristics[START_REF] Debaillie | A flexible and future-proof power model for cellular base stations[END_REF].

	Mode	Active 100% Idle	SM1	SM2	SM3	SM4
	Power (W)	20.7	13.2	8.22	3.55	3.36	3
	(de)Activation duration	35.5 µs 0.5 ms 5 ms 0.5 s
	Minimum sleep duration	71 µs	1 ms	10 ms	1 s

Ali El Amine received his Ph.D. degree in computer and networking from IMT Atlantique (formerly, Telecom Bretagne), Rennes, France in 2019. He also received his B.E. degree in electrical engineering and M.S. degree in computer engineering from the Lebanese American University (LAU) in 2013 and 2016, respectively. He is currently a postdoctoral fellow in CNRS-LAAS, Toulouse, France. His research focuses on green wireless communication, cellular networks, energy efficiency, network slicing optimization, NFV and machine learning.