Tingting Yuan
email: tingt.yuan@hotmail.com.

Christian Esteve Rothenberg

Katia Obraczka

Chadi Barakat
email: chadi.barakat@inria.fr

Thierry Turletti
email: thierry.turletti@inria.fr.

Harnessing UAVs for Fair 5G Bandwidth Allocation in Vehicular Communication via Deep Reinforcement Learning

Keywords: Unmanned aerial vehicles (UAV), fifthgeneration (5G), fairness, deep reinforcement learning

Terrestrial infrastructure-based wireless networks do not always guarantee their resources will be shared uniformly by nodes in vehicular networks. This is due mainly to the uneven and dynamic geographical distribution of vehicles and path loss effects. In this paper, we leverage multiple fifth-generation (5G) unmanned aerial vehicles (UAVs) to enhance fairness in network resource allocation among vehicles by positioning UAVs ondemand as "flying communication infrastructure". We propose a deep reinforcement learning (DRL) approach to determine UAVs' position to improve network resource allocation fairness and efficiency while considering the UAVs' flying range, communication range, and energy constraints. We use a parametric fairness function to attain a number of resource allocation objectives ranging from maximizing the total throughput of vehicles, maximizing minimum throughput, and achieving proportional bandwidth allocation. Simulation results show that the proposed DRL approach to UAV positioning can improve network resource allocation according to the targeted fairness objective.

and material penetration effects, which cause greater signal attenuation [START_REF] Rappaport | Overview of millimeter wave communications for fifthgeneration (5G) wireless networks-with a focus on propagation models[END_REF].

Deploying UAVs to improve communication coverage to vehicles "anywhere, anytime" is quite challenging given that UAVs have limited communication range and are powerconstrained [START_REF] Fotouhi | Survey on uav cellular communications: Practical aspects, standardization advancements, regulation, and security challenges[END_REF], coupled with the fact that the number of available UAVs to cover a specific region is often limited. Additionally, UAV's location in three-dimensional (3D) space means that their communication channel characteristics vary not only with their location relative to the ground but also their altitude. Therefore, UAV placement needs to consider many aspects, including vehicle location and bandwidth requirements, the capacity of the local terrestrial communication infrastructure, and UAV limitations, e.g., their energy budget and communication capability. Some previous efforts on UAV placement try to maximize aggregate throughput [START_REF] Ghanavi | Efficient 3d aerial base station placement considering users mobility by reinforcement learning[END_REF]- [START_REF] Saxena | Optimal UAV base station trajectories using flow-level models for reinforcement learning[END_REF], while some other proposals aim to maximize the number of users being serviced [START_REF] Alzenad | 3-D placement of an unmanned aerial vehicle base station for maximum coverage of users with different qos requirements[END_REF]. However, these approaches are either specific in their optimization target or may create unfair resource allocation as they tend to provide resources to well-connected vehicles while under-serving poorly connected ones.

Recently, computational intelligence, in particular machine learning (ML), has gained significant traction as a powerful tool to address challenges posed by new, more complex problems in a variety of disciplines and domains. The widespread use of ML techniques has been fueled in part by the everincreasing availability of computing power. As networks and their applications become increasingly complex and heterogeneous, they can benefit from ML approaches that learn and adapt to network-and application dynamics automatically and autonomously. ML techniques do not require prior knowledge of the operating environment and acquire this knowledge as they operate and adjust accordingly without the need for complex mathematical models of the system. In particular, deep reinforcement learning (DRL) [START_REF] Luong | Applications of deep reinforcement learning in communications and networking: A survey[END_REF], [START_REF] Arulkumaran | A brief survey of deep reinforcement learning[END_REF] has been viewed as a promising solution for dynamic UAV placement [START_REF] Saxena | Optimal UAV base station trajectories using flow-level models for reinforcement learning[END_REF], [START_REF] Liu | Energy-efficient UAV control for effective and fair communication coverage: A deep reinforcement learning approach[END_REF]. DRL is well-suited to tackle problems that require longer-term planning using high-dimensional observations. Additionally, DRL uses powerful deep neural networks to build a higherlevel understanding of the target environment with limited or no prior knowledge.

In this work, we propose a DRL-based approach to perform optimal, real-time UAV placement to achieve fair and efficient bandwidth allocation in 5G vehicular networks. To this end, we model the problem of dynamic UAV placement as an optimization problem whose goal is to maximize a global objective fairness function calculated over the communication bandwidth allocated to vehicles. This problem accounts for (i) aggregate communication resources, i.e., bandwidth, provided by ground wireless communication infrastructure and deployed UAVs, (ii) time-varying vehicle location, and (iii) UAV energy budget. We employ parametric fairness as the objective function, which is inspired by previous work on resource allocation for end-to-end Internet flows (e.g., [START_REF] Mo | Fair end-to-end window-based congestion control[END_REF]). UAVs are dynamically placed in such a way to realize different allocation strategies, including maximizing vehicle aggregate throughput, maximizing minimum vehicle throughput, as well as achieving vehicle throughput proportional fairness. We evaluate the proposed DRL-based UAV placement framework's performance through simulations driven by real-world vehicle mobility traces.

Overall, the main contributions of this paper can be summarized as follows:

1) We present the first study on bandwidth allocation fairness in UAV-assisted vehicular networks. Unlike previous work, we consider a novel and important objective for dynamic UAV placement in vehicular networks, namely network resource allocation fairness, such that vehicles have access to a fair share of the network bandwidth. 2) We present a DRL approach to dynamically place a swarm of UAVs such that vehicles in a vehicular network are allocated a fair share of the bandwidth as the network operates while considering UAV battery and coverage limitations. 3) Through extensive simulations, we experimentally assess the performance of the proposed dynamic UAV placement approach using different DRL algorithms using real-world vehicle mobility traces. Our results demonstrate that the proposed approach delivers superior bandwidth allocation fairness when compared to existing techniques. The remainder of the paper is organized as follows. Section II provides an overview of related work. Section III describes our system model, including how we model the underlying communication channel and coverage, UAV energy consumption, and vehicle spatial and temporal distribution. In Section IV, we formulate UAV placement for fair resource allocation as a non-linear optimization problem, and in Section V, we introduce the proposed DRL-based approach. Section VI describes our experimental methodology and presents results. Finally, Section VII concludes the paper with directions for future work.

II. RELATED WORK

Optimizing UAVs' dynamic placement to improve communication networks' performance is complex and challenging and has been the focus of some research efforts in recent years. Early work focused on single UAV placement [START_REF] Chen | Optimal positioning of flying relays for wireless networks: A los map approach[END_REF], [START_REF] Alzenad | 3-D placement of an unmanned aerial vehicle base station for maximum coverage of users with different qos requirements[END_REF], [START_REF] Li | A nearoptimal uav-aided radio coverage strategy for dense urban areas[END_REF], [START_REF] Wang | Joint uav hovering altitude and power control for space-air-ground iot networks[END_REF]. Other efforts explored the use of UAV swarms. For example, [START_REF] Lyu | Placement optimization of uav-mounted mobile base stations[END_REF], [START_REF] Li | Rechargeable multi-uav aided seamless coverage for qos-guaranteed iot networks[END_REF] proposes to minimize the number of UAVs to ensure coverage of all users, and [START_REF] Mozaffari | Efficient deployment of multiple unmanned aerial vehicles for optimal wireless coverage[END_REF] suggests maximizing the total coverage area. The placement problem is even more complex when multiple UAVs are considered, with new challenges raised by UAVs' cooperation and placement algorithms' efficiency.

In high-dimensional state-space and time-varying environments, ML-based technologies have been recently utilized for solving challenging problems in UAV placement [START_REF] Bithas | A survey on machine-learning techniques for UAV-based communications[END_REF], [START_REF] Yuan | Harnessing machine learning for next-generation intelligent transportation systems: A survey[END_REF]. DRL provides a promising solution as it can efficiently handle the challenges related to the state space's high dimensionality and the environment's dynamicity. Recent works have investigated the use of DRL for the placement of multiple UAVs. In [START_REF] Ghanavi | Efficient 3d aerial base station placement considering users mobility by reinforcement learning[END_REF], a DRL-based approach is proposed for UAV placement that complements terrestrial communication to maximize total network throughput. In [START_REF] Saxena | Optimal UAV base station trajectories using flow-level models for reinforcement learning[END_REF], DRL is also used to place UAVs to maximize aggregate network throughput while simultaneously balancing traffic load across UAVs. The work reported in [START_REF] Shokry | Leveraging UAVs for coverage in cell-free vehicular networks: A deep reinforcement learning approach[END_REF] explores DRL to maintain an acceptable quality of service for each vehicle and minimize the number of UAVs. The authors of [START_REF] Liu | Energy-efficient UAV control for effective and fair communication coverage: A deep reinforcement learning approach[END_REF] study energy-efficient UAV placement with DRL considering the fairness in coverage time. This work tries to evenly distribute service time across cells using UAVs but does not address quality-of-service on an end-user basis.

Unlike previous work, in this paper, we consider a novel and important objective for dynamic UAV placement in vehicular networks, namely network resource allocation fairness, such that vehicles have access to a fair share of the network bandwidth. To achieve this goal, we use a parametric fairness function inspired by early work on Internet resource allocation [START_REF] Mo | Fair end-to-end window-based congestion control[END_REF] as an objective function and maximize it over the offered load from vehicles.

III. SYSTEM MODEL

As illustrated in Fig. 1, our target scenario is a geographic region covered by a BS equipped with ground communication infrastructure, e.g., RSUs. A swarm of UAVs, U = {u 1 , . . . , u N }, acts as airborne wireless access points serving vehicles in this region. As such, the available communication infrastructure includes a BS, a set of RSUs, and UAVs denoted as I = {S, R, U }. Since most currently deployed infrastructure uses 4G technology, we assume the BS and RSUs are 4G devices. However, the proposed approach can be directly applied to 5G communication infrastructure technology. The BS and RSUs are stationary, and their locations and coverage radius are known. UAVs are outfitted with 5G transceivers, and their coverage radius R u depends on their current flying altitude h u . A UAV controller is co-located with the BS to determine UAVs' placements based on the vehicles' locations and UAVs' states. We assume that UAVs have limited energy and need to recharge periodically at charging stations. We assume that a UAV charging station (CS) is located in the region, and its location l charge is known. We aggregate vehicles into groups based on their location, defined at the cell level, where G refers to the set of cells in the region. Table I lists and describes the parameters used in our system model.

A. Channel Model

The signal-to-noise-ratio (SNR) between an infrastructure node i (BS, UAV or RSU) and vehicle v can be expressed based on the average path loss defined in [START_REF] Chen | Caching in the sky: Proactive deployment of cache-enabled unmanned aerial vehicles for optimized quality-of-experience[END_REF] as:

η i,v = ρ i,v 10 ϕ i,v /10 σ 2 (1)
where ρ i,v is the transmission power,

ϕ i,v = E[ϕ i,v
] is the expected path loss, and σ 2 is the additive white Gaussian noise power at the receiver. We assume that the total bandwidth of an infrastructure node i is B i and that this bandwidth is divided among the associated vehicles equally. The achievable rate of vehicle v (in bps) can be denoted as

x v = i∈I π i,v B i N i log 2 (1 + η i,v ) (2)
where N i = v∈V π i,v is the number of vehicles serviced by infrastructure node i, and π i,v is a binary access indicator, where π i,v = 1 denotes that infrastructure node i offers access service to vehicle v, and π i,v = 0 otherwise. We add the following two constraints: (1) π i,v ≤ e v,i to ensure that vehicles can only choose to associate with the infrastructure node whose communication range covers the vehicle, where e v,i is a Boolean indicator to describe whether vehicle v is in the coverage area of infrastructure node i or not; and (2)

i∈I π i,v = 1
to ensure that each vehicle can only be served by one infrastructure node. If a vehicle is in the coverage range of more than one infrastructure node, it will choose the one offering the highest rate.

Depending on the altitude of the communicating nodes, different channel models need to be used to account for different propagation conditions. For UAV-assisted communication, we consider two propagation scenarios, ground-to-ground (G2G) and air-to-ground (A2G) [START_REF] Vinogradov | Tutorial on UAV: A blue sky view on wireless communication[END_REF] channels.

1) Ground-to-Ground Channels: G2G channels are typically below 10 meters and 22.5 meters for suburban and urban environments, respectively, [START_REF] Vinogradov | Tutorial on UAV: A blue sky view on wireless communication[END_REF]. G2G channels include communication between BSs, RSUs, and vehicles. Their largescale channel attenuation (in dB) include distance-dependent path loss, whose classical model is log-distance path loss [START_REF] Zeng | Accessing from the sky: A tutorial on UAV communications for 5G and beyond[END_REF], [START_REF] Goldsmith | Wireless communications[END_REF] defined as follows:

ϕ i,v = 10ζ log 10 (d i,v ) + X 0 + X σ (3)
In this formula, d i,j = l i -l j is the Euclidean distance between two nodes in the space where l i is the location of node i. ζ is the path loss exponent that usually falls in the range between 2 and 6; X 0 is the path loss at a reference distance with definition as X 0 = 20 log( 4πfcd0 c

), d 0 being the free-space reference distance, f c is the carrier frequency and c is the light speed. X σ ∼ N (0, σ2 ) is the shadowing effect, which is modeled as a normal (Gaussian) random variable with zero mean and a certain variance σ2 . Thus, the average path loss is equal to

ϕ i,v = 10ζ log 10 (d i,v ) + X 0 .
2) Air-to-ground Channels: Obstructed A2G channel is in the range of 22.5-100 meters for urban environments [START_REF] Vinogradov | Tutorial on UAV: A blue sky view on wireless communication[END_REF]. Such a channel usually experiences a higher LoS probability than ground channels; however, it is still not 100%. In our scenario, UAVs communicate with ground vehicles in their coverage simultaneously by employing orthogonal frequencydivision multiple access (OFDMA). For UAV-Vehicle (U2V) communication, the mmWave channel of 5G is used. The mmWave propagation channel of the U2V link is modeled using the standard log-normal shadowing model with LoS and non-line-of-sight (NLoS) links by choosing specific channel parameters [START_REF] Chen | Caching in the sky: Proactive deployment of cache-enabled unmanned aerial vehicles for optimized quality-of-experience[END_REF]. In this model, the general path loss (in dB) between UAV u and vehicle v is defined as:

ϕ u,v = 10ζ LoS log(d u,v ) + X 0 + X σ LoS if LoS 10ζ N LoS log(d u,v ) + X 0 + X σ N LoS if NLoS ( 4
)
where ) is the reference path loss in the band of carrier frequency f m of mmWave and at reference distance d 0 .

The probability of having LoS transmission between UAVs and vehicles is expressed in [START_REF] Al-Hourani | Optimal lap altitude for maximum coverage[END_REF] as:

P LoS u,v = 1 1 + ae -b(-θu,v-a) ∀i ∈ U, v ∈ V (5)
where a and b are constants that depend on the environment (e.g., the environment density, such as rural, urban); and θ is the elevation angle of a point on the ground with respect to a UAV (measured in degrees), which can be expressed as

θ u,v = 180 π sin -1 ( hu du,v
). Thus, the average path loss of A2G channels can be expressed as:

ϕ u,v = P LoS u,v ϕ LoS u,v + P N LoS u,v ϕ N LoS u,v (6)
where the probability of having NLoS is defined as

P N LoS u,v = 1 -P LoS u,v
. Above a certain altitude, which depends on the environment, is called high-altitude A2G. In high-altitude A2G, all channels are in LoS, so the propagation is close to the free-space with ϕ u,v = ϕ LoS u,v .

B. Communication Coverage Model

We assume that the BS and RSUs are unmovable, so their coverage radius is fixed and denoted as R b for the BS, and R r for RSU r, respectively. For each UAV, given its height h u , the path loss increases with the distance. Thus, the coverage of UAV u is defined as the maximum radius in which the path loss is below a value (ϕ 0 ). The radius is defined as

R u = d 2 u -h 2 u
, where d u is the maximum distance between UAV u and the ground defined as

d u = arg max d (ϕ u,v ≤ ϕ 0 ).
As introduced earlier, we use Booleans e v,i to describe whether vehicles are in the coverage scope of infrastructure nodes or not, and they are defined as:

∀ i ∈ I, v ∈ V : e v,i = 1, d i,v R i 0, d i,v > R i
where R i is the coverage radius of infrastructure node i.

C. UAV Energy Consumption Model

According to [START_REF] Zeng | Energy minimization for wireless communication with rotary-wing UAV[END_REF], energy consumption for rotary-wing UAVs with speed υ is modeled as the sum of three items, namely, blade profile power, induced power, and parasite power. Blade profile power is the energy used to overcome the blades' profile drag. Induced power and parasite power are used to overcome the blades' induced drag and the UAV's fuselage drag, respectively. The power sum physically represents the UAV energy consumption per second in Joule/second (J/s) with speed υ:

P (υ) = P 0 (1 + 3υ 2 υ 2 0 ) + P 1 1 + υ 4 4υ 4 1 - υ 2 2υ 2 1 1/2 + 1 2 P 2 υ 3 (7)
where the P 0 , P 1 , and P 2 are coefficients of blade profile power, induced power, and parasite power, respectively. These coefficients are related to the aircraft's weight, air density, fuselage drag ratio, rotor solidity, etc. υ 0 and υ 1 denote the rotor blade's tip speed and the mean rotor-induced velocity in hovering. We note that when the UAV is hovering with speed υ = 0, the energy consumption is P h = P 0 + P 1 .

Combining both the propulsion energy and the communication-related energy, the energy consumption of each UAV at time interval T can be expressed as:

∆E u = T 0 P (υ t ) + P c i∈Vt π u,i,t dt, ∀u ∈ U (8)
We assume that the communication-related power is a constant expressed in Watt (W) and is denoted as P c . V t is the set of vehicles at time t.

D. Spatial and Temporal Distribution of Vehicles

As previously described, we aggregate vehicles that are in close geographic proximity into groups called cells. Vehicle spatial and temporal distribution characteristics, which will be used in our simulation experiments, are described below.

1) Spatial Variation (SV): The spatial variation can be used to show the variability in the number of vehicles between the cells. It is defined as the coefficient of variation in the number of vehicles per cell. It is expressed as the ratio of the standard deviation σ to the mean µ:

SV (D) = σ(D) µ(D) , ( 9
)
where D is the set of the number of vehicles in all cells denoted as

D = {|V 1 |, |V 2 |, ...|V |G| |}, and
|V i | is the number of vehicles in the cell i.
2) Temporal Variation (TV): For each cell, we define two factors to describe the temporal variation in the number of vehicles per cell. First, to normalize the number of vehicles per cell, we define the proportion of the number of vehicles per cell at time step t as:

∀g ∈ G, t ∈ [0, T ] : ξ g,t = |V g,t | i∈G |V i,t | where |V g,t |
is the number of vehicles in the cell g at time step t, and G is the whole set of cells. The time coefficient of variation (TCV) is then proposed to describe the variation on vehicles' distribution over time from 0 to T . The TCV is the mean coefficient of variation of the time-varying number of vehicles overall cells in G:

T CV (ξ g ) = 1 |G| g∈G σ(ξ g ) µ(ξ g ) (10)
where ξ g is the vector of proportional number of vehicles per cell g over the time period, which is a defined as ξ g = {ξ g,t |t ∈ [0, T ]}. Besides, to describe the variation on each cell over adjacent time steps, the step coefficient of variation (SCV) is proposed. It is the average change in ξ g between two adjacent time steps overall cells g, which is defined as:

SCV (ξ g ) = g∈G T t=1 |ξ g,t -ξ g,t-1 | |G|T (11)

IV. PROBLEM FORMULATION

A. UAV Placement Optimization Model

The problem of UAVs' dynamic placement can be formulated as follows: given a limited region in which some terrestrial communication infrastructure has been deployed, we try to find the optimal placement of a limited number of UAVs, say N , with an objective function calculated over the bandwidth allocated to vehicles. Note that we also need to consider the UAV charging duration during which UAVs are out of service. Recall that our objective function needs to account for the fairness of allocating aggregate network resources, i.e., aerial and terrestrial. We consider that at the beginning of the control period (t = 0), all UAVs are located at charge stations. The optimization of UAV dynamic placement where UAV location is given by L = {l u,t |u ∈ U, t ∈ [1, T ]} can then be modeled as follows:

max L,π T 0 F α (X t ) |V t | dt s.t. ∀u ∈ U, t ∈ T : l x min ≤ l x u,t ≤ l x max (12) l y min ≤ l y u,t ≤ l y max (13) h min ≤ h u,t ≤ h max (14) E u,t > 0 (15) π i,v d i,v ≤ R i ( 16
) i∈I π i,v = 1 (17)
This problem has the following constraints. Firstly, UAV locations should be within the region under consideration defined by BS's coverage radius and between l min and l max as defined by Equations ( 12), ( 13) and ( 14). Secondly, the UAVs need to go back for charging before they run out of power as indicated by Equation ( 15), i.e., l u,t = l charge . Depending on whether a UAV's target position is a charging station or not, UAVs can be either "working" or "charging". Since while charging UAVs cannot provide service, vehicles previously connected to them need to find a new infrastructure node to connect. Thirdly, Equation ( 16) ensures that vehicles can only choose to associate with the infrastructure node whose communication range covers the vehicle, while Equation ( 17) stipulates that each vehicle can only be served by one infrastructure node at a time. If a vehicle is in the coverage range of more than one infrastructure node, it will choose the one offering the highest data rate.

The objective is an aggregated parametric function, which models the fairness and efficiency of network resource allocation for UAVs and terrestrial infrastructure. The objective function is normalized by the time-varying number of vehicles to not bias the optimization against periods with fewer vehicles. To account for as many bandwidth allocation scenarios as possible, we use a parametric fairness function widely known in the literature on resource allocation on the Internet and TCP congestion control. Given a positive constant α and access rate vectors of vehicles X = {x 1 , ..., x |V | }, this fairness function is given in [START_REF] Mo | Fair end-to-end window-based congestion control[END_REF] by:

F α (X) = 1 1-α |V | i=1 x 1-α i if α = 1 |V | i=1 log x i otherwise ( 18
)
This parametric function covers different bandwidth allocation strategies according to the chosen value for parameter α. We can get the case of maximizing the sum of vehicles' access rates (namely total throughput) as α → 0; the proportional fairness case when α → 1; the harmonic mean fairness case when α → 2; the max-min fairness case when α → ∞, the latter tries to maximize the minimum access rate overall vehicles by giving priority to vehicles that have achieved lowest data rate. Proportional fair and harmonic mean fair allocations are trade-offs between maximizing total access rates and max-min access rate allocation. The first scenario of α = 0 is the mere scenario where the total access rate is maximized independently of any fairness consideration.

B. Time Discretization

An intuitive way to maximize fairness is to select the optimal locations at each time interval of ∆τ . The time discretization variables are L = {L t |t ∈ [1, T ]}, and L t = {l u,t |u ∈ U }. The UAVs are assumed to have two states in each time interval, namely, 1) flying to the target locations and hovering to serve the incoming flows, and 2) flying back to the charge station and charging. During the flying period, we assume that the UAVs have a constant velocity of υ m . The time cost by UAV u for flying from the current location l u,t-1 to the target location l u,t , which may be different from each other, is denoted as:

T u,t = ||l u,t -l u,t-1 || v m , ∀u ∈ U
The UAVs' arriving order at step t is denoted as I t . ∆T refers to the time interval between two consecutive UAV arrivals as shown in Fig. 2. The time interval of the i-th arrival in time slot t can be expressed as:

∆T i,t = ∆τ -T ui,t if u i is the last to arrive T ui+1,t -T ui,t otherwise (19)
where T ui,t is the time of arrival of the i-th UAV u i to its target location. During ∆T i,t , only the first i arriving UAVs can offer services. It follows that when the last UAV arrives at its target location, all working UAVs can offer service till the end of this time slot ∆τ . In addition, a UAV with T u,t ≥ ∆τ will not have time to work as an access point in this time slot. During the hovering period, UAVs hover over their target locations and serve the incoming flows. The mean fairness value of each step is defined as:

f α t = i∈It F α (X i,t )∆T i,t |V t |∆τ (20)
where X i,t is the vector of vehicles' access rates when the i-th UAV arrives to offer service in time slot t. Note that X i,t is calculated based on the mean distribution of vehicles in time slot t, and |V t | in Equation ( 20) is the mean number of vehicles in time slot t. The objective function can thus be approximated as T t=0 f α t . To ensure each UAV does not run out of energy, we assume that when a UAV's energy E u,t falls below level Ẽ (e.g., 10%), the UAV will go back to the charging station. Boolean parameters φ u indicate whether UAV u is in service (with value 1) or out of service to charge (with value 0).

∀ u ∈ U : φ u,t = 0 if E u,t Ẽ 1 otherwise (21)
Equation ( 8) can be reformulated as:

∆E u,t = P (υ m )T u,t + φ u,t (P h + P c N u,t )(∆τ -T u,t ) (22)
where N u,t is the number of vehicles that are connected with UAV u with definition as N u,t = i∈Vt π u,i,t . The first item of Equation ( 22) is the flying power, and the second item is the hovering and service power. If the UAVs are selected to go back to charge with φ u,t = 0, there would be no power cost for hovering and offering services. The updating of the battery of UAVs is expressed as:

∀ u ∈ U : E u,t+1 = E u,t -∆E u,t if φ u,t = 1 E f ull if φ u,t = 0 (23)
where E f ull denotes the capacity of UAVs' battery.

V. DRL FOR DYNAMIC UAV PLACEMENT

In this section, we introduce the proposed DRL-based method to solve the UAVs' dynamic placement problem.

A. DRL-Based Approach

We model trial-and-error learning as a Markov decision process (MDP). At each time t, the agent observes the current state s t of the interactive environment and gives an action a t according to its policy. Then, the environment returns reward r t as feedback, and moves to the next state s t+1 according to the transition probability P (s t+1 |s t , a). The goal to find an optimal policy can thus be formulated as the mathematical problem of maximizing the expectation of cumulative discounted return R t = T k=t γ k-t r k , where γ ∈ [0, 1] is a discount factor for future rewards to dampen the effect of future rewards on the action; r k is the reward of each step, and T is the time horizon before game over.

In our scenario, we assume that the intelligent agent who can control UAVs is deployed in the BS. The vehicles and UAVs in this region send their states (e.g., geographic locations) to the agent through the network periodically, e.g., every 2 minutes in the bus tracing dataset in Rio provided by CRAWDAD [START_REF] Dias | CRAWDAD dataset coppeufrj/riobuses (v. 2018-03-19)[END_REF]. Thus, the agent can obtain the vehicular traffic and the state of UAVs. Then, the agent makes decisions on UAVs' following locations, considering their current states and vehicles' information. Next, the agent sends the decisions to UAVs. Notice here that the UAVs' locations cannot be out of the agent's communication coverage. Thus, there are some constraints on the actions to be accounted for. The UAVs follow instructions from the agent and fly to their target locations. The key definitions of DRL for UAV placement are expressed as follows. State Space: The state is defined as s t = {D t , L t , E t } for time step t. Firstly, the state includes the distribution of vehicles D t , a vector of the mean number of vehicles in all cells over the time interval of ∆τ . Secondly, the current locations of UAVs would affect the decisions on the next locations. Thus, the vector of current UAVs' locations L t = {l u,t |u ∈ U } is also used as input. Then, the last item is the vector of UAVs' residual energy E t = {E u,t |u ∈ U }, which can also affect the decision on next locations. For example, UAVs may prefer to fly nearby to save energy and offer service for a longer time when the reward is inadequate to let them fly far away. Besides, residual energy is critical in making decisions on whether to charge or not. Action Space: The action is defined as a vector of UAVs' normalized locations a t = {w u1,t , ..., w u N ,t } of each time step t, in which w u,t = {w x u,t , w y u,t , w h u,t } is a vector denoting the normalized 3D coordinates of UAV u. The next location of UAV u can be calculated with l x u,t = w x u,t (l x max -l x min )+l x min for the x coordinate, the other two coordinates y and z can be calculated in the same way. This allows the UAVs' locations to be limited to some certain range. Besides, the UAVs' next locations depend not only on the action a t but also on the charging factor φ t . The definition for the next location of UAV u is then updated as follows:

∀ u ∈ U : l u,t = l u,t if φ u,t = 1 l charge if φ u,t = 0 ( 24
)
If a vehicle is in the coverage range of more than one infrastructure node, it will greedily choose the one offering the highest data rate. Therefore, π is not in the action space.

Reward: The value of reward depends both on the current state and on the action taken. In our problem, the reward is defined as the incremental ratio of the fair value of vehicles' access rates to the original fairness value without UAVs working for communication, which is:

r t = (f α t -f α t (0)) |f α t (0)| (25)
where f α t is expressed in [START_REF] Yuan | Harnessing machine learning for next-generation intelligent transportation systems: A survey[END_REF], and f α t (0) is the original fairness value without UAVs working for communication.

B. DRL-based Training Algorithm

DRL is known to be well-suited to tackle problems that require longer-term planning using high-dimensional observations which is the case of dynamic UAV placement to achieve fair bandwidth allocation in vehicular networks. There is a variety of DRL-based algorithms. Value-based algorithms, e.g., Deep Q-Networks (DQN) [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], use a deep neural network to learn the action-value function. However, they do not support continuous action space like the one in our problem. In the case of policy-based algorithms, e.g., Policy Gradient (PG) [START_REF] Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF], they explicitly build a representation of a policy. However, evaluating a policy without action-value estimation is typically inefficient and causes high variance. Actor-critic algorithms learn the value function (critic) in addition to the policy (actor), since knowing the value function can assist policy updates, for example, by reducing variance in policy gradients. Many existing approaches are based on actor-critic, for example, Deep Deterministic Policy Gradient (DDPG) [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF], Proximal Policy Optimization (PPO) [START_REF] Schulman | Proximal policy optimization algorithms[END_REF], Asynchronous Advantage Actor-Critic (A3C) [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF], Distributed Distributional Deterministic Policy Gradients (D4PG) [32], Twin Delayed Deep Deterministic Policy Gradients (TD3) [START_REF] Fujimoto | Addressing function approximation error in actor-critic methods[END_REF], and Soft Actor-Critic (SAC) [START_REF] Haarnoja | Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]. Both A3C and D4PG, which are asynchronous algorithms, can enable multiple worker agents to train in parallel, allowing faster training. However, in the proposed problem, the model doesn't support multiple worker agents. SAC has been shown to exhibit good and stable performance in various benchmarks and robot control tasks and it enables stronger exploration capability. As such, we chose SAC as the training algorithm which is illustrated in Fig. 3. Experience Replay Buffer: For off-policy training, the experience replay buffer, denoted as M, is used to store experiences and offer samplings for training. It is in the form of tuples, denoted as (s, a, r, s ) with states, actions, rewards, and successor states of the next time. Actor: The actor is responsible for taking actions a t based on the observed states s t . SAC employs a tractable policy π(a t |s t ; θ π ) as the actor parameterized through neural networks with parameters θ π , which targets planning of UAVs placement in continuous action spaces. SAC trains the actor with entropy regularization to maximize both the expected return and entropy. The optimal policy is π * = arg max π t E[r(s t , a t ) + βH(π|s t )], where H(π|s t ) is the entropy of the policy at state s t , and β controls the trade-off between exploration and exploitation. Then, the policy parameters can be learned by minimizing the expected Kullback-Leibler (KL) divergence [START_REF] Kullback | On information and sufficiency[END_REF] as

J π (θ π ) = E st∼M,at∼π [βlogπ(a t |s t ) -Q(s t , a t )] (26)
Critic: SAC employs value function V and Q-function Q in the critic. The value function V (s t ; θ V ) and the Q-function Q(s t , a t ; θ Q ) are parameterized using neural networks with parameters θ V and θ Q , respectively. The soft value function networks are trained to minimize the squared residue error, according to:

J V = θ V V (s t )(V (s t ) -Q(s t , a t ) + log π(a t |s t )) (27)
The Q-function can be optimized with stochastic gradients:

J Q = θ Q Q(s t , a t )(Q(s t , a t )-r(s t , a t )-γV (s t+1 )) ( 28
)
where V is from the target value network with parameters θ V . Target Network: The target value network is a copy of the estimated value function, which can improve stability by simply using a separate set of periodically updated parameters. The parameters of the target value network are periodically updated as:

θ V ← εθ V + (1 -ε)θ V ( 29
)
where ε is a coefficient between 0 and 1.

C. Overhead Analysis

Although DRL is well-suited to tackle problems that need longer-term planning using high-dimensional observations, it requires more time to collect data and memory resources for storing experiences. In this section, we analyze both the computational-and storage complexity of the proposed DRL approach. For complexity analysis purposes, the deep neural network can be viewed as a matrix multiplication problem. Thus, the complexity for the actor is O(|G|HN ), where |G| is the number of cells in the region being considered, H is the number of hidden layers, and N stands for the number of UAVs. Similarly, the critic neural network's complexity is O(|G|HN ). SAC training procedure's complexity is O(N ep K s N H|G|), where N ep is the number of training episodes, and K s is the batch size. Thus, the training is affected by the number of UAVs, which means using more UAVs will increase complexity. Real-time execution, namely calculating UAV locations based on the current state, the complexity is O(|G|HN ). However, if the brute force method is used to find optimal locations, which chooses locations with the highest reward from K possible locations, the complexity is O(K N ), which has higher complexity. The proposed DRL storage footprint depends on the maximum number of transitions stored in the experience replay buffer, denoted as |M|. All data is assumed to be a 32-bit float. Thus, each state is 4(|G| + 4N ) bytes, each action is 12N bytes, and each reward is 4 bytes. Then, one transition is 8|G| + 44N + 4 bytes, and the total size of the memory for the experience reply buffer is |M|(8|G| + 44N + 4) bytes.

VI. EXPERIMENTS AND RESULTS

We use simulation experiments to evaluate the performance of the proposed DRL-based UAV dynamic placement algorithm. In this section, we first present our evaluation methodology and then discuss our experimental results. Our code and the datasets used in our simulations are publicly available. 1 0:00 3:00 6:00 9:00 12:00 15:00 18:00 21:00 24:00

A. Evaluation Methodology

Fig. 4 shows the region used as the geographical footprint for the experiments. This region consists of a 3 × 3 km2 area of Rio de Janeiro, Brazil. This region is centered around a BS, co-located with a charging station. 3 RSUs are located at the corners of the region since they are the farthest away areas from the BS with the highest vehicle traffic. The BS has a coverage radius of around 2 km, which is 4G's nominal coverage in urban areas, and each RSU is assumed to have a 300 m coverage radius. For UAVs, the coverage radius depends on their altitude, ranging from 50 m to 100 m. The region was divided as a square grid of 10×10 cells, thus, the total number of cells |G| is 100, and the size of each cell is 300 × 300 m 2 . Such division yields a suitable distribution matrix for the number of vehicles in cells. Note that increasing the number of cells also increases the algorithm's computational complexity. For vehicle mobility and to be as realistic as possible, we use an available dataset [START_REF] Dias | CRAWDAD dataset coppeufrj/riobuses (v. 2018-03-19)[END_REF] with real-time position data reported by buses in Rio. The vehicle mobility dataset is in 24h time format. For our experiments, we select one week's worth of vehicle data, i.e., = 7, and consider every 15 minutes to be one step ∆τ = 15, which means that the agent takes actions every 15 minutes. For one day, there are 96 steps in total (T = 96). Fig. 5 shows a time series of number of vehicles |V t | over time t for the 7-day period considered, as well as the 68% confidence interval. Note that the intervals ∆τ can be set smaller, e.g., 5 minutes and 3 minutes. However, we find 15 minutes a reasonable interval for this dataset because this interval can provide an apparent variation on vehicular distribution between steps. For this dataset, the mean spatial variation over 7 days is 195%. It is defined as d=1 T t=1 SV (D d,t ) T

, where SV is in [START_REF] Alzenad | 3-D placement of an unmanned aerial vehicle base station for maximum coverage of users with different qos requirements[END_REF], and D d,t is the set of the number of vehicles in all cells at time t in day d. The mean of TCV defined in [START_REF] Luong | Applications of deep reinforcement learning in communications and networking: A survey[END_REF] and SCV in [START_REF] Arulkumaran | A brief survey of deep reinforcement learning[END_REF] over 7 days are 152.4% and 43.5%, respectively. These values show that the distribution of vehicles varies significantly over time (15 minutes as a step) and between cells (10×10). It is possible to accommodate this variability by placing UAVs differently and improving resource allocation according to the predetermined fairness objective.

In our experiments, we use the Adam optimizer [START_REF] Zhang | Improved adam optimizer for deep neural networks[END_REF] with the learning rate 0.0001 for the actor and 0.001 for the critic, and the hyper-parameters ε = 0.01, γ = 0.95. The size of the replay buffer |M| is set to be 50000, whereas the size of memory is around 50 MB, and the batch size is 800. The actor and critic are designed as 3-hidden-layer with 256-128-32 units neural networks. These values for the hyperparameters are chosen empirically after extensive experiments. Other simulation parameters and their values are listed in TABLE II.

B. Results and Performance Evaluation

Fig. 6 illustrates the results of running the proposed UAV dynamic placement algorithm with 2 UAVs. The placement's objective is to maximize the sum of the vehicles' access data rates (i.e., total throughput) with α = 0. The figure also shows U 0's and U 1's battery levels where the residual battery is represented in green and the used charge in red. Notice that U 1's remaining battery charge is lower than 10% in Fig. 6(e) and consequently U 1 has to go back to the charging station. Comparative Performance of DRL algorithms with Learning. Fig. 7 shows the comparative performance of DRL algorithms, i.e., PPO2, DDPG, SAC, and TD3, with α = 1 and using 2 UAVs 2 . The horizontal axis is the learning episodes whereas the vertical axis presents the mean value of accumulated rewards 1 d=1 T t=1 r t,d , which reflect fairness' accumulated incremental ratio. The curves show the 68% confidence interval of the accumulated rewards for 7 days. From the comparison, we find that DDPG has a relatively good learning speed at the early stages and is the fastest to converge, but SAC can converge relatively fast and achieves better policy with higher rewards. Additionally, we evaluate SAC's performance with different numbers of UAVs as shown in Fig. 8 and observe that, with more UAVs, the algorithm needs more episodes to converge due to increased state and action dimensionality. We notice a diminishing returns type behavior where the difference in accumulated reward drops from 37% when the number of UAVs increases from 2 to 4 to 9% when the number of UAVs increases from 8 to 10. Fairness Performance. We show simulation results for our proposed assignment methodology (called FD for short) compared against three other approaches: (1) Fairness-based place- Fig. 9 shows the advantages of our approach in terms of fairness compared to the four baselines. The Y-axis is the mean fairness value of vehicles F α (Xt) |Vt| with α = 1 and N = 2. The curves show the 68% confidence interval for 7 days. Combined with Fig. 5, we observe that, in general, mean fairness decreases as the number of vehicles increases. For example, during rush hour 6:00-21:00, the mean fairness is around 0.3, and between 2:00-4:00 pm, it is much higher at around 1.2. However, when compared with FE, MC, TF, and NU, FD improves mean fairness. More specifically, FD's mean improvement is around 2.12%, 6.60%, 11.08%, and 23.88%, respectively. Furthermore, we show the mean fairness improvement of our approach (FD) compared to the baselines

= 0 = 1 = 2 = 5
Different Utility Functions for different α in Fig. Performance considering different fairness parameters. Fig. 11 shows the improvement in terms of the cumulative distribution of vehicles' access rates compared with NU. Fig. 11(a) is from 1:00 am to 5:00 am in 7 days, and Fig. 11(b) represents the rush hour from 9:00 am to 13:00 am in 7 days. These curves show performance with 2 UAVs and different α values. Seen from Fig. 11(a) and Fig. 11(b), compared with NU, all curves with different α values have a significant improvement. Comparing different α values, we can conclude that UAVs are more likely to serve vehicles at a lower rate with a higher α value. This is expected as when α increases, individual fairness becomes more important than the overall performance (e.g., the total sum of access rates). Fig. 12 shows the total throughput defined as the sum of vehicles' access rates with different values of α and different numbers of UAVs. The lines show the mean value with intervals for 7 days. We observe that when more UAVs are deployed in the same environment, the overall throughput increases. The total throughput grows with decreasing α, since, as said above, lower α more inclines to improve the overall performance. In particular, α = 0 has the highest total throughput as expected since this allocation policy aims to maximize the sum of vehicles' access rates. Performance with different UAV swarm sizes. To illustrate performance with an increasing number of UAVs, we show the cumulative distribution of vehicle access data rates in Fig. 13 and vehicle access data rates over time in Fig. 14 for different numbers of UAVs varying from 1 to 6. As previously discussed, the more UAVs, the higher the access rates, but a diminishing returns effect in access rate improvement is observed. For example, around 27%, 35%, 40%, 45%, 47%, and 49% of vehicles have more than 0.4 Mbps with one to six UAVs, respectively. With the other α values, similar results can be observed. According to Fig. 14 which shows the percentage of vehicles whose access rate is higher than 0.5 Mbps, the more UAVs are deployed, the higher the access rate, whether during rush hour or not. Note again the diminishing returns 0:00 3:00 6:00 9:00 12:00 15:00 18:00 21:00 24:00 Time (h) behavior previously discussed where the mean access rate increase from 2 to 4 UAVs is more pronounced than that from 4 to 6 UAVs.

VII. CONCLUSIONS AND FUTURE WORK

In this paper, we leverage multiple 5G UAVs to enhance network resource allocation among vehicles by dynamic placement on-demand. We propose a DRL approach to determine UAVs' position to improve the fairness and efficiency of network resource allocation while considering the UAVs' flying range, communication range, and limited energy resources. We use a parametric fairness function for resource allocation that can be tuned to reach different allocation objectives ranging from maximizing the total throughput of vehicles, maximizing minimum throughput, and achieving proportional bandwidth allocation. The results of our simulations show that the dynamic placement of UAVs can improve the fairness of communication. Besides, the UAVs' locations are affected by fairness criteria since they have different preferences to serve different vehicles. Furthermore, we demonstrate that by deploying more UAVs in the same environment, it is possible to improve fairness by serving more underserved vehicles, but with the cost of increasing the training time due to increasing computational complexity.

In future work, we plan to extend our solution to consider scenarios with multiple BSs as well as account for other criteria to drive UAV placement, e.g., adaptive bandwidth allocation and minimizing handover overheads. Furthermore, to reduce the delay and computation load of a centralized agent, we plan to extend our work to a multi-agent system by deploying agents in UAVs. We will also investigate predictive methods for vehicle mobility and resource demands in the UAV placement algorithm to further improve accuracy and efficiency.

Fig. 1 .

 1 Fig. 1. Scenarios for UAV-assisted vehicular network.

ζ

  LoS and ζ N LoS are the path loss exponents of LoS and NLoS links; X σ LoS and X σ N LoS are the shadowing random variables which are, respectively, represented as the Gaussian random variables (in dB) with zero means and σ LoS and σ N LoS as standard deviations. X 0 = 20 log( 4πfmd0 c

Fig. 2 .

 2 Fig. 2. UAV arrivals in time.

Fig. 3 .

 3 Fig. 3. DRL for UAV dynamic placement.

Fig. 4 .

 4 Fig. 4. UAV-assisted vehicle communication in Rio de Janeiro, Brazil.

Fig. 5 .

 5 Fig. 5. Number of vehicles over time in Rio de Janeiro's 3×3 km 2 target region.

Fig. 6 .

 6 Fig.6. UAV locations and remaining battery levels with α = 0 and N = 2 from 7:00 to 9:30hs.

Fig. 7 .Fig. 8 .Fig. 9 .

 789 Fig. 7. Accumulated reward of DRL algorithms with α = 1 and N = 2.

Fig. 10 .

 10 Fig. 10. Mean fairness improvement of FD compared with four approaches on baselines with N = 2 and different α values.

Fig. 11 .

 11 Fig. 11. Cumulative distribution of vehicles' access rate with N = 2 showing improvement over NU.

Fig. 12 .

 12 Fig. 12. Throughput with different numbers of UAVs and α values.

Fig. 13 .

 13 Fig. 13. Cumulative distribution of vehicles' access rates with α = 1.

Fig. 14 .

 14 Fig. 14. Performance with different numbers of UAVs.

  Device location defined by coordinates (l x , l y , h), where l x and l y are the horizontal coordinates, and h is the elevation.

		TABLE I
		LIST OF NOTATIONS
	Notation	Description
	I	Communication infrastructure set including BS, RSUs,
		and UAVs.
	S, R	Set of BSs and RSUs.
	U	Set of N UAVs.
	V	Set of |V | vehicles.
	G	Set of |G| cells in the region.
	D	Total number of vehicles in all cells.
	R	Communication infrastructure coverage radius.
	l
	F α (X)	Data rate fairness criteria for access rate vector X with
		parameter α.
	π v,i	Binary access indicator -denotes if infrastructure node
		i offers service to vehicle v.
	P (v)	UAV energy consumption (J/s) in flying with speed υ.
	∆τ	Time interval between two steps.
	∆Eu	UAV u's energy consumption.
	Eu,t	UAV u's residual power at time t.
	Tu,t	Cost of UAVs to fly during time step t.

L UAV locations L = {lu,t|u ∈ U, t ∈ [1, T ]}. d i,j

Distance between two nodes i and j. ϕ

Channel path loss in dB.

X

Vehicles' access data rate X = {x 1 , ..., x |V | }. xv Vehicle v's access data rate in bps.

  [START_REF] Luong | Applications of deep reinforcement learning in communications and networking: A survey[END_REF]. The improvement is not significant compared to FE, but since FE is a brute force method, it cannot efficiently support real-time execution. Compared with the other approaches, there is a noticeable improvement in fairness for all α values, where MC yields relatively better performance among the 4 other algorithms. From these two figures, we observe that FD can improve fairness compared to the baselines because 1) neither Time Fairness (TF) nor Maximum Coverage (MC) can guarantee bandwidth allocation fairness, and because 2) DRL is also able to outperform the brute force method (FE).

	Cumulative Improvement	5% 10% 15% 20% 25% 0%	0	1	2 (a) Improvement 1:00-5:00 3 4 5 Rate (Mbps)	6	7	=0 =1 =2 =5 8
	Cumulative Improvement	5% 10% 15% 20% 25% 0%	0.0		0.2 (b) Improvement 9:00-13:00 0.4 0.6 Rate (Mbps)	0.8		=0 =1 =2 =5 1.0

https://github.com/TingtingYuan/UAV fairness

We also obtained similar performance behavior with other α values. They are not shown here due to space limitations.

ACKNOWLEDGMENT

This work was partly funded by Inria, supported by the French ANR "Investments for the Future" Program reference #ANR-11-LABX-0031-01, and UNICAMP, through the FAPESP Grant number #2017/50361-0, both in the context of the DrIVE #EQA-041801 Associated Team.