Keywords: configuration-based classification [3], monitoring [4, 5]

q * (s, a) back-up. . . . . .

It is with genuine gratitude and warm regard that I dedicate this work to my loving parents BOUKOBERINE Kada and BOUKOBERINE Kheira, to all my family members, and to all my teachers ever since my primary school days.

I would like to express my sincere appreciation and gratitude to my supervisor, Professor Mohamed Benbouzid, for the valuable guidance, support, ideas, and encouragement throughout my PhD journey. He has been a true mentor and I have learned immensely from his wisdom, discipline, deep knowledge, intuition, and his big love of research. His supported me inside and outside the laboratory, in good and difficult moments. He was not just a very good supervisor, he was my dear friend, my big brother which I couldn't have before. He was my source of inspiration in many aspects of life. Beside learning from him how to do research, he was a big school for me to learn many values such as generosity, great altruism, humbleness, self-sacrifice.

Words can not describe my gratitude to him and I am very lucky to work with him.

A special thanks to Professor Teresa Donateo, for her support, guidance, valuable comments, suggestions and also for hosting me at Salento University, Italy, for one month as a guest PhD student. Overall, it has been a wonderful experience and I am sure we will make significant contributions in this area in the coming years. I would like also to thank Dr. Nassim Rizzoug and M. Laid Degaa for hosting me at ESTACA LAVAL for two weeks as a guest PhD student and for the helpful comments and discussions.

I would also like to deeply thank the PhD thesis defense committee members: the pre-examiners, Prof. Demba Diallo from Université de Paris-Saclay and Prof. S. M. Muyeen from University de Qatar, for their efforts giving valuable comments and suggestions; Prof. Josep M. Guererro from Alborg University, Prof. Fei Gao from Université de Belfort, Dr-HDR. Nassim Rizzoug from ESTACA Laval, Dr. Muhammad Fahad Zia from NUCES Lahore. I would like to take this opportunity to thank also my friends Khalil Touimi and Muhammad Fahad Zia for their help and support during the initial stages of my PhD. A big thanks goes to Khalil Touimi for his generosity, important advises and for the very fruitful discussion we made. Thank you Khalil and Fahad again for your advices and for the amazing time we spend together in weekends. My experience in Brest would not have been enjoyable without these never ending beautiful memories. I would like also to thank Dr. Yassine Amirat, Abhinandana Boodi, Chinmayi Kanthila, Maria Bitar, Patrick El Helou, Haroon Rashid, for the good time we had together during my PhD.

I would also take this opportunity to specially thank Jocelyne Savio, Magali Gouez and Michele Kerleroux for helping me in the administrative affairs of my PhD.

Abstract

The interest in electric unmanned aerial vehicles (drones) is rapidly growing in recent years. The reason is that drones have abilities to perform some difficult or dangerous tasks, with high mobility, safety, and low cost. It should be noted that drones are revolutionizing many public services including real-time monitoring, search and rescue, wildlife surveys, delivery services, wireless coverage, and precision agriculture. To increase endurance and achieve good performance, the use of a hybrid power supply system architecture is considered as a promising option. A hybrid power architecture may combine several power sources such as fuel cells, batteries, solar cells, and supercapacitors. The choice of a suitable power source hybridization architecture with an optimal energy management system is therefore crucial to enable the efficient operation of advanced drones. In this regard, this thesis focuses on proposing energy management optimization adapted to drone platform specifications by investigating several approaches namely: rule-based, optimization-based, and machine learning-based strategies. The goal is to have an optimal power allocation extending the drone endurance while preserving sources lifetimes. An extensive overview of reinforcement learning-based energy management for hybrid electric vehicle applications is proposed to exploit the achieved machine learning advancement in developing smart strategies for drones.

Chapter 1 Introduction

Topic and Context

An unmanned aerial vehicle (UAV or drone), is a flying robot, which can operate autonomously or be controlled telemetrically to carry out a special mission [START_REF] Liu | A review of rotorcraft Unmanned Aerial Vehicle (UAV) developments and applications in civil engineering[END_REF]. Drones have received great interest in the past few years thanks to advancements in microprocessors and artificial intelligence (AI) [START_REF] Adnan | Sustainable interdependent networks from smart autonomous vehicle to intelligent transportation networks[END_REF] enabling smart drones [START_REF] Kyrkou | Drones: Augmenting our quality of life[END_REF]. This has been motivated by several advantages such as low cost and high mobility. They are employed in several applications in both military and civil domains: minesweeping, monitoring, delivery, wireless coverage, and agriculture uses. Several multinationals are hugely investing in the improvement of drone performance to extend their uses as much as possible. It was expected that the drone market value reaches US$127 billion in 2020 [START_REF] Mazur | Clarity from above: PwC global report on the commercial applications of drone technology[END_REF]. Figure 1.1 shows the top 10 drone operators ranking conducted in 2018 by Drone Industry Insights [START_REF] Droneii | TOP20 Drone Service Provider Ranking[END_REF].

Drone features and configurations vary widely according to mission requirements. Thus, various types of classifications can be found in the literature focusing on different parameters [START_REF] Hassanalian | Classifications, applications, and design challenges of drones: A review[END_REF][START_REF] Shakhatreh | Unmanned Aerial Vehicles: A Survey on Civil Applications and Key Research Challenges[END_REF]. Regarding shape it was found fixed/rotary/flapping-wing drones, hybrid and balloon configuration; regarding size, there are mini, micro, and nano drones. The North Atlantic Treaty Organization (NATO) categorized drones into three classes based on the maximum take-off weight (MTOW), where each class is divided into subcategories depending on altitude and mission radius [START_REF] Valavanis | Handbook of unmanned aerial vehicles[END_REF]. Other classifications are discussed in [START_REF] Valavanis | Handbook of unmanned aerial vehicles[END_REF].

Electric drones are favored for some of their key features such as reliability [START_REF] Jaeger | Conceptual design of a high-endurance hybrid electric unmanned aerial vehicle[END_REF], reduced noise and thermal signatures [START_REF] Bongermino | Model and energy management system for a parallel hybrid electric unmanned aerial vehicle[END_REF], high efficiency [START_REF] Hassanalian | Classifications, applications, and design challenges of drones: A review[END_REF], no pollutants emission, self-starting, and developed control devices enabling high maneuverability. Internal combustion engine (ICE)-based drones have longer endurance due to ICE high power and energy densities [START_REF] Bongermino | Model and energy management system for a parallel hybrid electric unmanned aerial vehicle[END_REF]. However, they need an auxiliary starting motor, their control is more complex [START_REF] Glassock | Multimodal hybrid powerplant for unmanned aerial systems (uas) robotics[END_REF], and their acoustic and thermal signature are high. Some researchers [START_REF] Xie | Fuzzy logic based equivalent consumption optimization of a hybrid electric propulsion system for unmanned aerial vehicles[END_REF][START_REF] Donateo | Fuel economy of hybrid electric flight[END_REF] combined the electric motor with the ICE in a hybrid architecture benefiting from the advantages of both engines. However, fossil fuel depletion and the increase in greenhouse gas (GHG) emissions decreased the interest in thermal engines and motivate the use of an electric propulsion as a green technology in different sectors such as transportation [START_REF] Adnan | An overview of electric vehicle technology: a vision towards sustainable transportation[END_REF]. International policy and market-based momentum to phase out ICE vehicles have been investigated in [START_REF] Burch | Survey of global activity to phase out internal combustion engine vehicles[END_REF], and this trend may be found in drones shortly to facilitate more environmental-friendly devices. This can be one reason to abandon 1.2. Focus and Scope ICE in drones, especially for tasks without long endurance requirements. Other techniques can be used to extend battery-based drones endurance: swapping [START_REF] Galkin | UAVs as Mobile Infrastructure: Addressing Battery Lifetime[END_REF][START_REF] Bocewicz | A Declarative Modelling Framework for Routing of Multiple UAVs in a System with Mobile Battery Swapping Stations[END_REF][START_REF] Williams | Persistent mobile aerial surveillance platform using intelligent battery health management and drone swapping[END_REF], laser-beam inflight recharging [START_REF] Galkin | UAVs as Mobile Infrastructure: Addressing Battery Lifetime[END_REF][START_REF] Ouyang | Throughput Maximization for Laser-Powered UAV Wireless Communication Systems[END_REF][START_REF] Achtelik | Design of a flexible high performance quadcopter platform breaking the mav endurance record with laser power beaming[END_REF], and tethered drones [START_REF] Gu | Novel roaming and stationary tethered aerial robots for continuous mobile missions in nuclear power plants[END_REF][START_REF] Muttin | Umbilical deployment modeling for tethered UAV detecting oil pollution from ship[END_REF][START_REF] Woodworth | Tethered Aerial System for Data Gathering[END_REF]. Swapping is a technique used to recharge the drone's depleted batteries during its mission. For this purpose, ground stations are deployed in specific locations. A flying drone can receive light power through a laser-beam transmitted from a generator deployed in a ground station. The drone batteries are therefore recharged without landing. Tethered drones can have unlimited endurance because the power will be continuously provided through connection lines linking the drone to the power supply ground station.

Focus and Scope

Electric power can be mainly provided by batteries, but their low energy density and long charging time prevent the drone to have a sufficient flighttime [START_REF] Gong | Fuel cell propulsion in small fixed-wing unmanned aerial vehicles: Current status and research needs[END_REF]. In addition, the battery may not allow a drone to conduct some maneuvers needing fast power response due to its slow power dynamics. In this context, a supercapacitor is a good option to balance battery limitations [START_REF] Gong | Analysis of a Fuel-Cell/Battery/Supercapacitor Hybrid Propulsion System for a UAV using a Hardware-in-the-Loop Flight Simulator[END_REF]. It is worth mentioning that flight endurance can be improved by aerodynamical optimization in the drone design stage [START_REF] Panagiotou | Conceptual design of a blended wing body male uav[END_REF]. Current advancements in battery technologies allow slightly increasing the endurance for about 90 min using Lithium-Polymer (LiPo) batteries [START_REF] Verstraete | Design of a Fuel Cell Powered Blended Wing Body UAV[END_REF], while increasing the number of batteries is not a practical solution due to weight and space constraints. Extending drone endurance requires then the usage of additional power sources to balance battery limitations while being compliant Chapter 1. Introduction with weight and space restrictions. In this context, a fuel cell is a good candidate due to its high specific energy and quasi-instantaneous refueling. It can typically have up to five times higher energy density than LiPo batteries, which leads to a significant increase in the hybrid-drone endurance [START_REF] Kim | Design and development of a fuel cell-powered small unmanned aircraft[END_REF]. It is worth noting that most available electrical drones are using a fuel cell as the main power source. A supercapacitor can also contribute to the power supplying process since it has a very high power density and quick response to peak power needed in drone takeoff and sudden maneuvers. Fixed-wing drones can carry solar cells and then use solar energy. Fuel consumption can be reduced [START_REF] Shiau | Design of a Solar Power Management System for an Experimental UAV[END_REF], therefore extremely increasing the endurance, while using a storage system. Donateo et al. [START_REF] Donateo | A new approach to calculating endurance in electric flight and comparing fuel cells and batteries[END_REF] provided a performance comparison between fuel celland battery-based drone power supply systems. The study has shown that battery-based system fits low-energy applications while a fuel cell system provides much higher autonomy when the carried energy exceeds 4MJ. In another recent study [START_REF] Apeland | Suitability analysis of implementing a fuel cell on a multirotor drone[END_REF], fuel cell hybrid systems (FCHS) and batteries supplying multirotor drones have been analyzed. According to the proposed assessing model, FCHS performs better in terms of endurance when the power system mass is higher than a threshold of 7.3kg. Considering a commercially available multirotor with a maximum take-off mass of 25kg as a case study, it was proved that endurance increases by +76% using FCHS. In this context, Singapore-based HES Energy Systems enterprise has launched their Hycopter multicopter for large-scale industrial maintenance inspections, the autonomy was increased from 20-30min provided by lithium-ion batteries to 3.5 h using compressed gaseous hydrogen [START_REF]HES multirotor drone, designed and built in US, has 3h flight time[END_REF]. MetaVista, a South Koreanbased company, recently completed a nearly 12h7min multicopter test flight using an Intelligent Energy 800W fuel cell power module [START_REF]Intelligent Energy showcases drone fuel cell modules in japan, fuel cells bulletin[END_REF]. This is considered as the world longest flight time.

Relevance and Importance

Depending on the mission requirement, the drone may perform some rapid maneuvers which need a fast power supply system response such as take-off and climbing. Due to its slow dynamics, a fuel cell system can not provide the appropriate response to that power peaks [START_REF] Ahmadi | Improving fuel economy and performance of a fuel-cell hybrid electric vehicle (fuelcell, battery, and ultra-capacitor) using optimized energy management strategy[END_REF]. Therefore, combining the fuel cell with the battery is proposed to improve the power supply performance. The battery is used as an energy buffer and will supply or absorb power 1.4. Questions and Objectives peaks due to its relatively short response time. For example, this configuration is adopted for the Intelligent Energy 650W fuel cell power module to power the DJI M100 quadcopter [START_REF]Intelligent Energy, 650W Fuel Cell Power Module Specifications[END_REF]. This hybridization requires an energy management strategy (EMS) to be implemented to optimize the power allocation respecting the sources characteristics and preserving their lifetimes. An EMS typically includes current and voltage sensors to track the power flow, converters to control the power sources outputs, and a processing unit handling the adopted power management strategy.

One of the biggest challenges for small electric drones is their limited endurance which is their main performance parameter. Thus, many researchers and drone companies are focusing on this issue trying to increase their flight time. Several studies were carried out by studying different aspects of long endurance fuel cell-powered drones, such as energy-based conceptual design [START_REF] Oh | Conceptual design of small unmanned aerial vehicle with proton exchange membrane fuel cell system for long endurance mission[END_REF], thermal efficiency analysis [START_REF] Ji | Thermodynamic analysis of a solid oxide fuel cell jet hybrid engine for long-endurance unmanned air vehicles[END_REF], and hydrogen generation improvement [START_REF] Okumus | Development of boron-based hydrogen and fuel cell system for small unmanned aerial vehicle[END_REF]. Fuel saving is one of the solutions to extend the autonomy in all fuel cell-based mobile applications such as hybrid electric vehicles [START_REF] Zhang | Develop of a fuel consumption model for hybrid vehicles[END_REF]. Some researchers have investigated fuel economy to optimize fuel cell hybrid systems using real-time and optimized control for the fueling regulators [START_REF] Bizon | Fuel saving strategy using real-time switching of the fueling regulators in the proton exchange membrane fuel cell system[END_REF][START_REF] Bizon | Hydrogen saving through optimized control of both fueling flows of the fuel cell hybrid power system under a variable load demand and an unknown renewable power profile[END_REF][START_REF] Bizon | Sensitivity analysis of the fuel economy strategy based on load-following control of the fuel cell hybrid power system[END_REF]. Akhtar et al. [START_REF] Akhtar | Real-time optimal techniques for unmanned air vehicles fuel saving[END_REF] studied the possibility of real-time trajectory generation based on the dynamic soaring model to optimize the fuel consumption. However, there is a significant lack of studies dealing with drone EMS to extend endurance.

Questions and Objectives

The main objective of this thesis is to respond to the key question of how to optimally manage the power supply in a hybrid fuel cell-powered drone. Indeed, online optimization strategies, which are usually implemented for hybrid electric vehicles can not fit on drone online applications due to their complexity and computational burden. Complexity requires embedding a powerful calculator inducing additional weight and energy consumption. Computational burden will affect the drone maneuverability.

Rule-based EMSs are widely implemented in drones thanks to their simplicity and the possibility of real-time execution [START_REF] Zhang | Energy management strategy of hybrid pemfc-pv-battery propulsion system for low altitude uavs[END_REF][START_REF] Boukoberine | A frequency separation rule-based power management strategy for a hybrid fuel cell-powered drone[END_REF]. Zhang et al. [START_REF] Zhang | Experimental investigation on the online fuzzy energy management of hybrid fuel cell/battery power system for uavs[END_REF] have experimentally investigated an online fuzzy rule-based EMS for a fuel cell/battery-based drone. The proposed strategy presented a low hydrogen consumption compared to state machine and passive control strategies.

Chapter 1. Introduction However, the drone load power specificity was not considered in that study, as only a pulsed-power profile was implemented for the load. It should be mentioned that many researchers have assumed their own power profiles to simulate the requested power during the flight mission due to the difficulty to get real data [START_REF] Xie | Fuzzy logic based equivalent consumption optimization of a hybrid electric propulsion system for unmanned aerial vehicles[END_REF]. In our study, a commercialized quadcopter powered by the Intelligent Energy 650W fuel cell power module is considered as the case study.

Although rule-based methods seem to be well adapted to drone application, they have poor performance regarding fuel consumption. In this thesis and as a first stage, improved rule-based strategies have been implemented to preserve sources lifetimes considering different power system topologies when trying to save fuel as much as possible. In a second stage, optimizationbased EMSs have been developed while keeping the advantages of rulebased approach to fit drone applications. As a last stage, we provide an upto-date extensive overview of how reinforcement learning (RL) can enhance EMSs for HEV applications while providing an insight into the RL approach. The goal of the last study is to exploit the achieved machine learning advancement in the HEV energy management area to develop smart EMSs for drones.

The remaining of this manuscript is organized as follows: Chapter II provides a comprehensive and critical state of the art review of the available drone power supply structures and their energy management strategies, chapter III presents the implemented rule-based strategies, chapter IV is dedicated to the optimization-based EMSs implementation, chapter V provides an up-to-date extensive overview of how reinforcement learning (RL) has enhanced EMSs for HEV applications while providing an initiation to RL approach. In chapter VI, a summary of this thesis is provided with future work.

Chapter II State of the Art Critical Review on Drones Power Supply and Energy Management 1 Introduction

In the literature, it has been found some review papers addressing different aspects of drones, such as fuel storage and generation in small fixedwing drones [START_REF] Gong | Fuel cell propulsion in small fixed-wing unmanned aerial vehicles: Current status and research needs[END_REF]; classification, advances, and research trends in small-scale UAVs [START_REF] Cai | A Survey of Small-Scale Unmanned Aerial Vehicles: Recent Advances and Future Development Trends[END_REF]; drone-related technologies and applications in civil engineering [START_REF] Liu | A review of rotorcraft Unmanned Aerial Vehicle (UAV) developments and applications in civil engineering[END_REF]; drone challenges across civil applications [START_REF] Shakhatreh | Unmanned Aerial Vehicles: A Survey on Civil Applications and Key Research Challenges[END_REF]. Moreover, they considered classification and design challenges of drones [START_REF] Hassanalian | Classifications, applications, and design challenges of drones: A review[END_REF]; fixed-wing drones path planning algorithms [START_REF] Sujit | Unmanned aerial vehicle path following: A survey and analysis of algorithms for fixed-wing unmanned aerial vehicless[END_REF]; guidance, navigation, and control of rotorcraft unmanned aircraft systems [START_REF] Kendoul | Survey of advances in guidance, navigation, and control of unmanned rotorcraft systems[END_REF]; and quadrotors modeling and control [START_REF] Shraim | A survey on quadrotors: Configurations, modeling and identification, control, collision avoidance, fault diagnosis and tolerant control[END_REF]. Unlike these previous review studies, the provided one in this chapter will first present a general basic knowledge about drone systems. Then, it will focus on drones energy aspect with a comprehensive and critical evaluation of the available power supply structures and their energy management systems. Afterward, gaps will be identified, while providing useful guiding recommendations and prospects.

This review chapter is organized as follows: Section 2 presents unmanned aerial systems (UAS) basic knowledge including UAS parts, drones classifications and applications. Section 3 discusses and analyses drones power sources and supplying architectures methods, while Section 4 provides a critical review on drones energy management strategies.

2 Unmanned Aerial System (UAS) Basic Knowledge

UAS Basic Parts

An unmanned aerial system (UAS), illustrated by Fig. 1, is basically composed by three parts:

-The unmanned aerial vehicle (drone);

-The ground control station (GCS), which can be autonomous or humanoperated;

-The command and control system ensuring communication and data links between the drone and the ground station.

Unmanned Aerial System (UAS) Basic Knowledge

Unmanned aerial vehicle (drone). As depicted in Fig. 1, the drone platform includes (1) an onboard flight control system based on processing units handling essential tasks, such as guidance, navigation and control (GNC) algorithms, in-flight data gathering and analysis, communication with the ground station, and mission planning; (2) a propulsion system including power supply sources, speed controller, converters, energy management system, motor, and propeller; (3) the required sensors to maintain an autonomous flight; and (4) payload: equipment needed for the missions, such as actuators, cameras and radar [START_REF] Cai | A Survey of Small-Scale Unmanned Aerial Vehicles: Recent Advances and Future Development Trends[END_REF]. In drone platform, the propulsion system is the main onboard power consuming part. Indeed, it allows the drone motion by converting the stored electrical energy into a mechanical power generated by the motor-propeller system. It can constitute more than half of the drone weight. Figure 2 shows the schematic diagram of a typical drone propulsion system.

The onboard sources deliver power to the DC bus through unidirectional and bidirectional converters to enable battery charging and discharging. These converters allow controlling the power flow. They receive control signals from the EMS that handles power splitting. Small drones widely used motor type is the brushless DC (BLDC) one. Thanks to its key features, such as high efficiency and power density [START_REF] Glasgo | How much electricity can we save by using direct current circuits in homes? understanding the potential for electricity savings and assessing feasibility of a transition towards dc powered buildings[END_REF], high speed and good torque characteristics, reliability, ease of control, and long lifetime [START_REF] Varshney | Speed response of brushless dc motor using fuzzy pid controller under varying load condition[END_REF]. Although induction motors are advantageous in terms of low cost and robustness, they have some limitations, such as a relatively low efficiency, cooling issue, and low torque.

Drones Classifications

Several criteria can be considered to classify drones such as size, fuselage (rotary, fixed, flapping wings) [START_REF] Shraim | A survey on quadrotors: Configurations, modeling and identification, control, collision avoidance, fault diagnosis and tolerant control[END_REF], endurance, flight range, MTOW, flying mode (lighter than air, heavier than air) [START_REF] Norouzi Ghazbi | Quadrotors unmanned aerial vehicles: A review[END_REF], and mission or application. Thus, we cannot find acknowledged unique classification in the literature. Authors in [START_REF] Abaunza | Les applications de drones aériens-L'utilisation civile des UAVs[END_REF] proposed a comparative study between three drone configurations, namely: fixed-wing, rotary wing, and hybrid. Classifications based just on weight parameter are proposed in [START_REF] Homainejad | Application of multiple categories of unmanned aircraft systems (UAS) in different airspaces for bushfire monitoring and response[END_REF]. In [START_REF] Turano | Design of a medium range tactical UAV and improvement of its performance by using winglets[END_REF], mission is considered as the main criterion to classify drones as miniature, tactical, strategic, and loitering munition. Cai et al. [START_REF] Cai | A Survey of Small-Scale Unmanned Aerial Vehicles: Recent Advances and Future Development Trends[END_REF] proposed a classification based on six characteristics of drones with a MTOW less than 25 kg. In this case, 132 drones available models were investigated leading to the proposal of three categories (Table II.1): small tactical, miniature, and micro drones. This classification is summarized in Fig. 3. Hassanalian and Abdelkefi [START_REF] Hassanalian | Classifications, applications, and design challenges of drones: A review[END_REF] introduced a new classification considering

Drone Applications

Drones have entered several both civil and military application areas, thanks to academic research and industrial projects advances. By integrating sensors and camera, drones can now perform difficult and risky missions without human intervention. This is done with high efficiency, rapidity, while maintaining low cost [START_REF] Abaunza | Les applications de drones aériens-L'utilisation civile des UAVs[END_REF]. Nowadays, drones play an important role in traffic monitoring. They continuously collect data in real-time about roads and traffic conditions and transfer information to the monitoring center. Drones present many advantages compared with traditional roads monitoring methods (radar sensors, fixed surveillance video cameras), such as flexibility, large coverage range instead of fixed one, rapidity, and accuracy to detect incidents. They can also be efficient in traffic estimation [START_REF] Ke | Real-time bidirectional traffic flow parameter estimation from aerial videos[END_REF]. Guido et al. [START_REF] Guido | Evaluating the accuracy of vehicle tracking data obtained from Unmanned Aerial Vehicles[END_REF] employ drones in their methodology of vehicle tracking. The authors emphasize the usefulness of using drones in traffic management by recording and processing collected data. Leitloff et al. [START_REF] Leitloff | An operational system for estimating road traffic information from aerial images[END_REF] proposed drone based systems as a solution for traffic monitoring in case of disaster or mass events. In this context, devices for online acquisition, assessment, and sharing traffic information, are installed onboard the drone. The literature is typically rich with traffic monitoring and vehicle tracking applications and case studies [START_REF] Tang | Vehicle detection in aerial images based on region convolutional neural networks and hard negative example mining[END_REF].

Inspection and monitoring

1) Traffic monitoring:

2) Infrastructures inspection and monitoring:

Construction of large-scale infrastructures such as big buildings and highways require a huge number of employees and machines deployed in a large working area. Drones can facilitate monitoring operations for managers to be up to date with the state of advancement of the project, and to control tasks without the need to access the construction site. of more than 65,000 cell towers based on video analytics using drones [START_REF]AT&T Working to Make Drone Tower Inspections 'Intelligent[END_REF]. Efficient deep learning algorithms are implemented to conduct online detection of system faults or malfunctions. Smart grid power lines monitoring using industrial Internet of drones is investigated in [START_REF] Zhou | Energyefficient industrial internet of uavs for power line inspection in smart grid[END_REF]. In [START_REF] Larrauri | Automatic system for overhead power line inspection using an Unmanned Aerial Vehicle-RELIFO project[END_REF], authors discussed a method for inspection of power lines, substations, and transformers. In this context, the drone is equipped by infrared (IR) cameras to detect bad conductivity in power lines by processing recorded images (Fig. 6 (b)). In parallel, the approach aimed to accurately identify buildings and vegetation that are in proximity to the power lines.

AT&T (American Telephone & Telegraph) conducts an automated inspection

In a recent experimental study, Márquez and Segovia [START_REF] García Márquez | Condition monitoring system for solar power plants with radiometric and thermographic sensors embedded in unmanned aerial vehicles[END_REF] have equipped a drone with a radiometric sensor and a thermographic camera in order to estimate dust accumulation on solar photovoltaic panels (Fig. 6 (a)). This application seems to be important since the loss in energy production caused by dust can reach up to 15 % in one year [START_REF] García Márquez | Condition monitoring system for solar power plants with radiometric and thermographic sensors embedded in unmanned aerial vehicles[END_REF]. In [START_REF] Wang | Automatic detection of wind turbine blade surface cracks based on uav-taken images[END_REF], a wind turbine blade surface cracks detection framework is proposed using drone-based aerial imaging and image processing.

In [START_REF] Mohamadi | Vertical Takeoff and Landing (VTOL) Small Unmanned Aerial System for Monitoring Oil and Gas Pipelines[END_REF], authors were interested in the inspection of water, gas, and oil pipelines. Drones conduct autonomously remote sensing, visual, and thermal inspections based on onboard sensors, which enable detection of gas leaks in dangerous or hard-to-reach sites. The literature is typically rich with infrastructures inspection and monitoring illustrations and case studies [START_REF] Sankarasrinivasan | Health Monitoring of Civil Structures with Integrated UAV and Image Processing System[END_REF].

3) Environmental monitoring:

Governments, all around the world, are in continuous interest for information and updated data about environmental changes and their impacts. In context, periodic measures are conducted on top of volcanoes, mountains, rivers, seas, and even in the atmosphere [START_REF] Abaunza | Les applications de drones aériens-L'utilisation civile des UAVs[END_REF]. Drones are then used as an effective tool for collecting samples thanks to their dynamic characteristics. Civil protection institutions can accurately monitor water resources before, during, and after a flood occurs, thus, preparing a damage control plan. Furthermore, by deploying several drones above volcanoes, it would be possible to have online measurements to safely estimate a volcano state (Fig. 7 (a)). In 2013, the NASA conducted flights to take measurements on a volcanic plume near San José, at Costa Rica, using the RQ-14 Dragon Eye drones made by AeroVironment, which have an endurance of 80 min [104]. In case of disaster, communication and computing infrastructures may be damaged [START_REF] Leitloff | An operational system for estimating road traffic information from aerial images[END_REF], so drones can quickly provide details and data to enable efficient control of rescue operations. Monitoring of pollution can be performed by drones equipped by specific sensors, enabling real-time measurement of polluting gases diffusivity, such as CO, CO 2 , SO 2 , and NO 2 (Fig. 7 (b)). In [START_REF] Zang | Investigating small-scale water pollution with uav remote sensing technology[END_REF], fixedwing drones were employed to capture high-resolution images, which help in studying water pollution. Others relevant illustrations and case studies on drone-based environmental monitoring can be found in [START_REF] Villa | An overview of small unmanned aerial vehicles for air quality measurements: Present applications and future prospectives[END_REF][START_REF] Jufri | State-of-the-art review on power grid resilience to extreme weather events: Definitions, frameworks, quantitative assessment methodologies, and enhancement strategies[END_REF].

Delivery

Several companies around the world are using drones for fast delivery of goods or packages [START_REF] Hassanalian | Classifications, applications, and design challenges of drones: A review[END_REF] (8), such as DHL postal service in Germany [START_REF] Matthias | Unmanned aerial vehicles in logisitics[END_REF], Google and Amazon in USA [111], and others. Drones can serve also for delivery of medical supplies in emergency cases [START_REF] Shakhatreh | Unmanned Aerial Vehicles: A Survey on Civil Applications and Key Research Challenges[END_REF]. A typical illustration is the case of the Federal Aviation Administration (FAA) that conducted the first drone-based medical delivery in 2015 [START_REF] Howell | The First Government Sanctioned Delivery of Medical Supplies by Remotely Controlled Unmanned Aerial System (UAS)[END_REF]. The operation was based on GPS coordinates to reach the delivery location, where the drone was equipped with a checking device to confirm that the package reaches the right customer/receiver. Other illustrative examples of drone-based delivery can be found in [START_REF] Škrinjar | Application of unmanned aerial vehicles in logistic processes[END_REF][START_REF] Chiang | Impact of drone delivery on sustainability and cost: Realizing the uav potential through vehicle routing optimization[END_REF].

Agriculture

In recent years, drones become an attractive option in agriculture to gather low-altitude high-precision images above the field [START_REF] Latif | An agricultural perspective on flying sensors: State of the art, challenges, and future directions[END_REF][START_REF] Murugan | Development of an adaptive approach for precision agriculture monitoring with drone and satellite REFERENCES data[END_REF]. Then, appropriate image processing operators or software are used to extract valuable data about the state of the cultures and health information such as moisture and soil properties. Drones can also be used in several tasks such as irrigation scheduling, disease detection, soil texture mapping, weed detection, residue cover, tillage mapping, crops management, cultivations analysis and other applications in precision agriculture (Fig. 9) [START_REF] Potrino | Drones support in precision agriculture for fighting against parasites[END_REF][START_REF] Chen | Remote sensing for vegetation monitoring in carbon capture storage regions: A review[END_REF].

FIGURE 9: Drones used in precision agriculture [START_REF]India International Agro Trade and Technology Fair[END_REF][START_REF]UAVs in agriculture[END_REF].

Wireless coverage

Drones can be used for wireless information transfer as flying access points (Fig. 10) [START_REF] Mazur | Clarity from above: PwC global report on the commercial applications of drone technology[END_REF]. They can cooperate with the cellular network to provide a better coverage for isolated areas that are badly covered due to obstacles such as mountains or buildings. Drones can be deployed as relay nodes to replace ground base stations of the communication network, in case of malfunction or damage due to disaster for example [START_REF] Shakhatreh | Unmanned Aerial Vehicles: A Survey on Civil Applications and Key Research Challenges[END_REF]. Other illustrative studies dealing with drone-based wireless coverage can be found in [START_REF] Al-Hourani | Modeling cellular-to-UAV path-loss for suburban environments[END_REF].

FIGURE 10: Drone-based wireless coverage illustrations [START_REF]Mathematicians propose to improve cellular network coverage by using UAVs[END_REF][START_REF]SkyLiTE: End-to-End Design of Low-Altitude UAV Networks for Providing LTE Connectivity[END_REF].

Military applications

Originally, drones were restricted to military applications. They were afterward extended to the civil sector. Well-known drones (generally called drones) military applications are: artillery guidance, delivery of equipment and supplies, radio and data relay, borders surveillance, spy drones, communication disruptors and electronic warfare, maritime operations (anti-ship missile defense, naval fire support, over the horizon targeting), reconnaissance flights, minesweeping raking, Etc. (Fig. 11) [START_REF] Valavanis | Handbook of unmanned aerial vehicles[END_REF]. 3 Drone Power Sources

Gas turbine engines have been used in aircraft propulsion systems for their high power-to-weight ratio [START_REF] El-Sayed | Aircraft propulsion and gas turbine engines[END_REF] and long operating time [START_REF] Hassanalian | Classifications, applications, and design challenges of drones: A review[END_REF]. However, they present good performance only for high power ranges [START_REF] Austin | Unmanned aircraft systems: UAVS design, development and deployment[END_REF], above 100 hp [START_REF] El-Sayed | Aircraft propulsion and gas turbine engines[END_REF]. They are however not suited for small-scale drone applications as they exhibit low fuel economy and a very low efficiency with a high noise level [START_REF] El-Sayed | Aircraft propulsion and gas turbine engines[END_REF].

The internal combustion engine (ICE) constituted the pillar of aircrafts propulsion system [START_REF] Lee | Power managements of a hybrid electric propulsion system for UAVs[END_REF]. Comparatively to an electric motor (EM), an ICE, thanks to its higher fuel energy and power densities, allows long flight time and large payload range, which are two important challenges in airborne applications. However, the multi-step process of producing energy reduces the ICE system efficiency [START_REF] Sharaf | An overview of fuel cell technology: Fundamentals and applications[END_REF]. EMs are preferred for drones due to many key features such as their very low thermal and acoustic signatures, well developed electronic control systems, ease-adaptation to automatic control, self-starting feature [START_REF] Glassock | Multimodal hybrid powerplant for unmanned aerial systems (uas) robotics[END_REF], low cost, and a higher reliability minimizing crash possibility due to motor shutdown or failure. It is worth noting that in electric propulsion systems, electronic speed controller (ESC) failure may occur due to overheating and melting of ESC casing. Duplicating components can be a solution to this issue [START_REF] Brown | Characterization and prognosis of multirotor failures[END_REF].

A hybrid power propulsion prototype for UASs was proposed in [START_REF] Glassock | Multimodal hybrid powerplant for unmanned aerial systems (uas) robotics[END_REF]. The purpose was to combine the benefits of both thermal and electric engines into a hybrid architecture. However, even the simulation results proved an

Chapter II. State of the Art Critical Review on Drones Power Supply and Energy Management endurance improvement of 13%, the system is complex and not environmentally friend. In the same context, other authors, such as Bonbergino et al. [START_REF] Bongermino | Model and energy management system for a parallel hybrid electric unmanned aerial vehicle[END_REF][START_REF] Bongermino | Hybrid Aeronautical Propulsion: Control and Energy Management[END_REF], and Xie et al. [START_REF] Xie | Fuzzy logic based equivalent consumption optimization of a hybrid electric propulsion system for unmanned aerial vehicles[END_REF], have also discussed a hybrid parallel powertrain architecture using both EM and ICE. Incorporating an ICE in drones is unfortunately not yet a solution of choice in terms of fuel usage and endurance optimization limited degree of freedom. In this context, this section will be restricted to electric propulsion-based drones, therefore discussing and critically evaluating electric power sources supplying drones. Interesting supplying strategies will also be presented for one-source-based drones, such as swapping, laser-beam inflight recharging, and tethered drones.

Battery-based Supplying Techniques

Battery-powered drones

Most small drones, especially quadrotors are battery-powered. Indeed, batteries are considered as the main component in battery-powered drones [START_REF] Khofiyah | Goldsmith's commercialization model for feasibility study of technology lithium battery pack drone[END_REF], their usage improves the simplicity and the flexibility of the propulsion system. Moreover, battery-based platforms can satisfy various hobbyist applications in term of flight time and cost-effectiveness. However, typical small battery-powered drones have short endurance due to constraints on the battery pack weight. They can fly for a maximum of 90 min using LiPo batteries [START_REF] Verstraete | Design of a Fuel Cell Powered Blended Wing Body UAV[END_REF]. Consequently, these small-scale drones are usually devoted for commercial ends. Lithium batteries are preferred for small drones due to their low weight and relatively high specific energy. Indeed, LiPo batteries power almost 90% of micro aerial vehicles having a weight less than 2 kg and a length less than 100 cm [START_REF] Hassanalian | Classifications, applications, and design challenges of drones: A review[END_REF]. Table II.2 details the key characteristics of four types of battery technologies, namely specific energy, energy density, and specific power. This will allow the technology choice for drone with a given application and mission. In [START_REF] Donateo | A new approach to calculating endurance in electric flight and comparing fuel cells and batteries[END_REF], different battery technologies are evaluated in term of state of charge (SOC) for a given mission.

Drone Power Sources

In [START_REF] Traub | Range and Endurance Estimates for Battery-Powered Aircraft[END_REF], Traub investigated parameters that affect performance of a batterypowered drone and developed mathematical expressions to estimate the range and endurance considering the battery discharge conditions. This mathematical formulation was not evaluated by simulation nor experimentally tested. One of the biggest challenges with batteries powered electric vehicles is the reduced autonomy. Intensive research efforts were thus focused toward improvements of batteries performance to extend their operational time and enable electric vehicles to conduct long-duration missions. Nevertheless, even with advancements in batteries characteristics, the specific energy of current battery technology still limits endurance and range. Therefore, they will not reach many drone applications needs. Additionally, both stability and safety levels are affected as a result of energy density improvement [START_REF] Mike | Whitepaper: Fuel Cell-Based Energy Systems for Commercial UAVs[END_REF]. Thus, different solutions were developed in the literature to deal with battery limitations. Fuel cells seem to be a good alternative due to their performance especially the higher specific energy. Furthermore, most existing drones are powered by more than one energy source, where batteries, fuel cells, solar cells, and supercapacitors are hybridized to form the drone power supply.

Swapping

Swapping is a technique used to recharge the drone's depleted batteries during its mission. It can be conducted autonomously or human-operated. Hotswapping is a specific technique where a depleted battery is replaced by a fully charged one, keeping the drone powered on. It can then join its hotspot and operates again. By deploying more than one drone and by managing their cooperation so that a vehicle can hand over its hotspot seamlessly to another one, the multi-agent system can provide continuous service to an area [START_REF] Galkin | UAVs as Mobile Infrastructure: Addressing Battery Lifetime[END_REF]. Three conditions must be fulfilled to complete a typical swapping operation:

(1) ground recharge station where drones can land for charging/changing batteries, (2) drone swarm for persistent applications, (3) the management system to ensure drones swarm cooperation. Figure 12 and 13 illustrate the swapping and Hotswapping techniques.

Ground stations (GSs) are deployed in specific locations such as cities or along trajectories connecting cities to establish an infrastructure network [START_REF]Neva Aerospace Announces UAV Charging Networks[END_REF].

Their installation can be done on cell towers, street lights, rooftops, power poles, or standalone pylons [START_REF] Saad | Vehicle Base Station[END_REF][START_REF] Gentry | Multi-Use UAV Docking Station Systems and Methods[END_REF]. The battery swap station includes several components: ground electronics, onboard circuit, landing frame and a contact mechanism [START_REF] Leonard | Energy management in swarm of unmanned aerial vehicles[END_REF]. The battery can be charged by contact enabling paths or by inductive coupling [START_REF] Pounds | Vehicle Replenishment. Google Patents[END_REF], and the docking platforms can be fed by power lines, big batteries and solar cells for remote stations. Figure 14 shows some commercially available GSs.

In [START_REF] Williams | Persistent mobile aerial surveillance platform using intelligent battery health management and drone swapping[END_REF], Williams and Yakimenko developed a multi-rotor aerial prototype for long-duration missions using the swapping approach, where the objective was to maintain the airborne platform operation based on the battery SOC monitoring. The concept is to keep one of three quadcopters continuously on the loiter position, when the battery SOC drops under a defined threshold.

The second quadcopter takes over and allows the former to join the ground station to put the battery on charging and get a replenished one to be ready to its next use. This system cycles through vehicles until all batteries are exhausted or the mission is accomplished. Batteries necessary number to ensure durable operation depends on the discharge time and the required charging time, and do not depend on the drones number. Nevertheless, the FIGURE 14: Drone's charging stations [START_REF]Neva Aerospace Announces UAV Charging Networks[END_REF][START_REF]14 uses of drones[END_REF].

bigger it is, the more robust the system becomes. However, changing and charging batteries on the launch platform was not done automatically, so the system is still restricted to human intervention. Additionally, regarding their field test, no special mission using sensors was conducted, only static loitering was performed.

An economical comparison between battery refilling/recharging platforms has been proposed in [START_REF] Kemper | UAV Consumable Replenishment: Design Concepts for Automated Service Stations[END_REF]. Three stations were developed based on an axiomatic design before being investigated, which enable the authors to present analysis linking the cost, complexity, and coverage levels. It was shown that refilling stations are a good choice when the coverage is low, otherwise, it is preferable to use exchange stations.

Suzuki et al. [START_REF] Suzuki | Automatic Battery Replacement System for UAVs: Analysis and Design[END_REF] presented a comparison between battery charging/replacement systems using a Petri net model. The authors also presented a detailed and illustrated discussion about the design options of autonomous swapping stations. The target was to conduct a precise drone positioning for swapping regardless of landing error. However, the estimated swap time was nearly one minute, which is a considerable time comparatively to other works [START_REF] Z.-N. Liu | QUADO: An autonomous recharge system for quadcopter[END_REF][START_REF] Liu | An Autonomous Dock and Battery Swapping System for Multirotor UAV[END_REF].

In addition, the system was not fully operational. Indeed, some modules were not prototyped nor tested.

In another study [START_REF] Leonard | Energy management in swarm of unmanned aerial vehicles[END_REF], a ground recharge station for battery powered quadrotor helicopters was designed and an algorithm was implemented to reduce the battery recharge duration. The proposed autonomous charging process uses safer electrical contacts and a balancer. It was intended for swarm applications. In this case, as in the majority of the carried studies, there were no experiments nor flight tests. It should be mentioned that the battery lifetime was neglected.

A design and hardware implementation of an automated refueling station for small-scaled drones is presented in [START_REF] Ure | An Automated Battery Management System to Enable Persistent Missions With Multiple Aerial Vehicles[END_REF]. In order to extend the operational time and enable long-duration autonomous missions with multi-agent drone systems, a planning and learning algorithm was developed and tested in 3 h long persistent flight using three drones and more than 100 battery swaps. The battery recharger mechanism to exchange batteries is based on a linear sweeping motion leading to a simple and robust design. However, the system cost was not considered, while increasing the number of batteries and drones can lead to a costly and complex to manage system.

In [START_REF] Swieringa | Autonomous battery swapping system for small-scale helicopters[END_REF] Swieringa et al. proposed a swapping system based on online algorithms to perform energy management, drone health monitoring, and accurate landing. In this approach, a servo-based lift is employed to swap and place the batteries onto a hexagonal mat horizontally mounted, which holds the charging batteries. However, that swap mechanism induces drones power losses. Even individual tests were done for all the components, the entire system was not tested.

In [START_REF] Z.-N. Liu | QUADO: An autonomous recharge system for quadcopter[END_REF][START_REF] Liu | An Autonomous Dock and Battery Swapping System for Multirotor UAV[END_REF], active infrared imaging were used to design an autonomous docking platform. The system, operating in day and night time includes a camera and an infrared filter for high precision landing. The drone landing operation is guided based on image processing and the swapping process is reduced to less than 10 seconds instead of 60 in [START_REF] Saha | Battery health management system for electric UAVs[END_REF]. In this study, the focus was just on the swapping time and landing precision. Cooperation between drones was not discussed and the improvement of the operational time was not proved by any definite long-duration mission.

A hardware platform was proposed in [START_REF] Michini | Automated Battery Swap and Recharge to Enable Persistent UAV Missions[END_REF] including a dual-drum structure to perform fast and efficient hotswapping using a buffer of 8 batteries. The experimental work includes results about the needed battery swapping average time. In this study, a flight test was conducted considering a search and track mission with communication constraints. However, the developed station with 3 quadrotors performs that mission for about 70 min as a maximum operational time.

In a recent paper [START_REF] Bocewicz | A Declarative Modelling Framework for Routing of Multiple UAVs in a System with Mobile Battery Swapping Stations[END_REF], Bocewicz et al. were interested in cyclically repeated missions such as aerial delivery services. In this approach, mobile battery swapping stations (MBSs) are proposed, they move to given swapping points at defined times according to a preset timetable. Thus, drones can join the suitable station for battery replacement and also for loading/unloading of goods. In this study, where the MBSs routing problem was considered, the aim was to optimize both the used drones number and the distance to travel

Drone Power Sources

by proposing a declarative model of routing drones and MBSs. Nevertheless, this approach fit to only a few types of missions, in addition to the fact that the mobile swapping stations approach it is not always feasible. It should be mentioned that this study was focused on the routing task with no experiments, where the swapping was not discussed.

A critical analysis of the literature swapping approaches is proposed in Table II.3.

Laser-beam inflight recharging

The swapping approach seems a good solution to extend battery-based drones operating time with the ground station constraint for charging/replacing batteries. This consequently impacts drones mission flight time and operation efficiency. In this context, wireless recharging was proposed as an alternative approach [START_REF] Galkin | UAVs as Mobile Infrastructure: Addressing Battery Lifetime[END_REF][START_REF] Lu | Wireless charging techniques for uavs: A review, reconceptualization, and extension[END_REF].

The required ground station includes a prime power source to supply the laser generator, which transmits a light beam to the drone while it is airborne. An embedded optical receiver converts light to electricity powering the drone. This technology enables drones to stay inflight indefinitely without the need of landing to recharge batteries. When recharging is needed, the drone joins an aerial power link area to receive energy. Thereby, safety is improved by eliminating takeoff and landing risks. Laser transmitters are deployed on rooftops of high buildings in order to avoid laser-beam obstruction or on a mobile station (Fig. 15). A radiative link will be established between the drone and the nearest energy source to allow fast power transfer. A working prototype was developed by LaserMotive enabling a transfer of hundreds of watts [START_REF] Ouyang | Throughput Maximization for Laser-Powered UAV Wireless Communication Systems[END_REF]. In addition, experiments in [START_REF] Achtelik | Design of a flexible high performance quadcopter platform breaking the mav endurance record with laser power beaming[END_REF] demonstrated the technique feasibility illustrating more than 12 h flight time for a quadcopter. In this study, where both the mechanical design and flight control system were presented, size, payload, and drone adaptation to a specific application were considered.

The laser-beam inflight recharging approach constraints the drone to operate at reduced heights and sometimes in a limited area to keep power transferring from the laser transmitter. In this context, the Federal Aviation Administration (FAA) regulations limit the maximum altitude for small drones to 400

Ref. Main Contribution

Advantages Limitations [START_REF] Swieringa | Autonomous battery swapping system for small-scale helicopters[END_REF] Multi-rotor aerial prototype for long-duration surveillance missions based on battery health monitoring.

Hardware prototype, the system can operate continuously, battery health considered.

No autonomous swapping, system operation managed by laptop rather than an embedded controller. [START_REF] Leonard | Energy management in swarm of unmanned aerial vehicles[END_REF] Design, test, and construction of an autonomous ground recharge station using a balancer and safer electrical contacts. Drone positioning after landing with small error.

Hardware

Precise drone positioning for swapping regardless of landing error.

Relatively long swapping duration, entire system was not tested. therefore limiting the number of deployed drones in the area or considerably increasing the approach cost [START_REF] Galkin | UAVs as Mobile Infrastructure: Addressing Battery Lifetime[END_REF].

Tethered drones

Drones, when tethered to a power supply, can have unlimited autonomy. There will be no need to repeated recharging neither in the air nor by landing on the ground because electricity will be continuously provided by a ground power supply station through connection lines. This will also allow a safe and efficient data transfer between the drone and the ground station. Generally power lines are made of copper wires. However, fiber technology is taking place in the tethered-drone area. Kilowatts of power can be transferred using high-intensity light in a fiber optic cable. Optical power decreases the detectability by eliminating the electrical signature. It can also reduce both battery payload and power lines weight up to eight-times than copper wire [START_REF]Powerlight Technologies | Power Over Fiber Case Study | Aerial Applications[END_REF]. In addition, fiber technology is more beneficial in high altitudes. Indeed, with copper wires, power losses will reduce efficiency. Figure 16 provides illustrative two examples of tethered drones.

In [START_REF] Muttin | Umbilical deployment modeling for tethered UAV detecting oil pollution from ship[END_REF], a tethered drone is proposed for maritime pollution monitoring. It is deployed on the ship to detect oil spilled on the sea to avoid heavy contamination of the shore. In their patent, Woodworth et al. [START_REF] Woodworth | Tethered Aerial System for Data Gathering[END_REF] outfitted a tethered drone for data gathering applications. However, the major drawback of this approach is the limited operating area, the connecting cable prevents the drone to fly far from its ground station. Sometimes a moving vehicle is used to carry the prime power source so that the drone can cover a larger area. In [START_REF] Gu | Novel roaming and stationary tethered aerial robots for continuous mobile missions in nuclear power plants[END_REF], Gu et al. proposed a tethered drone for extremely-long-endurance missions of nuclear power plants. It is targeted that the drone flight may last for a few days or even a few months as long as the tethered cable provides continuous power. It is worth noting that prototypes were designed and successfully demonstrated in outdoor environments.

Fuel Cell Powered drones

Hydrogen-powered drones, as illustrated by Fig. 17, can fly for hours instead of few minutes when traditional batteries are used [START_REF] Pan | Recent advances in fuel cells based propulsion systems for unmanned aerial vehicles[END_REF]. For example, LiPo batteries have specific energy up to 250 Wh/kg [START_REF] Kim | Design and development of a fuel cell-powered small unmanned aircraft[END_REF], while can reach 1000 Wh/kg in case of a fuel cell system with a compressed hydrogen tank [START_REF] Verstraete | Design of a Fuel Cell Powered Blended Wing Body UAV[END_REF]. In addition, the refueling process is done almost instantly, while it (recharging) takes a long time with batteries. In [START_REF] Cai | A sizing-design methodology for hybrid fuel cell power systems and its application to an unmanned underwater vehicle[END_REF][START_REF] Donateo | A new approach to calculating endurance in electric flight and comparing fuel cells and batteries[END_REF], three types of batteries, Li-ion, and Ni-Mh, beside fuel cell are compared and investigated considering some criteria such as energy and power densities, discharging characteristics, temperature effects, efficiency, and endurance. However, no flights test were conducted to study the power sources behavior during flights and to find out their capabilities and performance versus different airborne conditions. Figure 18, illustrating specific power vs specific energy for batteries, fuel cells, and supercapacitors, shows that fuel cells clearly outperforms the others sources in term of specific energy. They should therefore be considered as the solution of choice for higher endurance drones for a given weight [START_REF] Evangelisti | Life cycle assessment of a polymer electrolyte membrane fuel cell system for passenger vehicles[END_REF]. Fuel cells may have lower energy density compared to Lithium batteries, as the hydrogen tank volume has to be considered (Fig. 19).

In [START_REF] Belmonte | Fuel cell powered octocopter for inspection of mobile cranes: Design, cost analysis and environmental impacts[END_REF], Belmonte et al. investigated the development of a drone for mobile crane inspection. They have considered proton exchange membrane fuel cells and lithium-ion batteries and analyzed them from an economical point of view and life cycle assessment. One of the main conclusions from a commercial point of view is that fuel cells, being a niche product, are more expensive compared to the Li-ion battery.

Fuel cells efficiency issue

Elements on the process of electricity generation from fuel cells can be found in [START_REF] Pan | Recent advances in fuel cells based propulsion systems for unmanned aerial vehicles[END_REF]. In term of efficiency, fuel cells can reach a level as high as 60% [START_REF] Hwang | Lifecycle performance assessment of fuel cell/battery electric vehicles[END_REF].

It is unfortunately lower than that of lithium batteries (over 90%). Indeed, a fuel cell stack operation requires auxiliary equipement, which reduces efficiency, while the onboard hydrogen generation system increases complexity (Fig. 20) [START_REF] Pan | Recent advances in fuel cells based propulsion systems for unmanned aerial vehicles[END_REF].

FIGURE 18: Specific energy/power comparison between energy sources [START_REF] Cai | A sizing-design methodology for hybrid fuel cell power systems and its application to an unmanned underwater vehicle[END_REF].

FIGURE 19: The Hycopter drone from HES [START_REF]Hydrogen power for aerial autonomy[END_REF].

Fuel cell types

Many technologies are used in the fuel cell industry. They are typically classified according to chemical criteria such as catalysts, and electrolytes, or operating characteristics such as temperature. In their study [START_REF] Gong | Fuel cell propulsion in small fixed-wing unmanned aerial vehicles: Current status and research needs[END_REF], Gong et al. propose a brief comparison between most used fuel cell types in drones, namely polymer electrolyte membrane fuel cell (PEMFC), direct methanol fuel cell (DMFC), and solid oxide fuel cell (SOFC), where Table II.5 summarizes their main key characteristics. PEMFC seems to be the most commonly used type for drones propulsion system [START_REF] Pan | Recent advances in fuel cells based propulsion systems for unmanned aerial vehicles[END_REF]. Indeed, Intelligent Energy, which is a fuel cell development company, focused on manufacturing PEMFC for drones applications [153]. Indeed, this fuel cell technology has typical characteristics such as: lightweight, high power density, low operating temperature allowing less warm-up time, long lifetime, and low response time to load variation [START_REF] Gong | Fuel cell propulsion in small fixed-wing unmanned aerial vehicles: Current status and research needs[END_REF][START_REF] Lapeña-Rey | A fuel cell powered unmanned aerial vehicle for low altitude surveillance missions[END_REF].

Table II. [START_REF]Delivery by drone[END_REF] proposes some examples of fuel cell-powered research and industrial drones.

Fuel storage

Hydrogen has a density of only 0.089 kg/m 3 at standard temperature and pressure [START_REF] Gong | Fuel cell propulsion in small fixed-wing unmanned aerial vehicles: Current status and research needs[END_REF]. Hence, to enable drone carrying sufficient fuel for a given mission, tanks must be bulky [START_REF] Sulaiman | Optimization of energy management system for fuelcell hybrid electric vehicles: Issues and recommendations[END_REF] (Fig. 21). This is an important constraint regarding drone size and weight. Furthermore, safety reasons, there is no

Chapter II. State of the Art Critical Review on Drones Power Supply and Energy Management 12 h 07 min possibility to store pure hydrogen under extremely high pressure and low temperature [START_REF] Kendall | Hydrogen and Fuel Cells in Transport[END_REF]. There are mainly three techniques that are currently used to store hydrogen in drones [START_REF] Gong | Fuel cell propulsion in small fixed-wing unmanned aerial vehicles: Current status and research needs[END_REF]: Compressed hydrogen gas, liquid hydrogen, and chemical

Drone Power Sources

hydrogen generation. Each of these storage techniques has its advantages and drawbacks, which are analyzed and discussed in Tables II. [START_REF] Garcia | Drones to optimize logistics operations in the country[END_REF] and II.8.

Sodium borohydride Ammonia borane

Liquid hydrocarbons [151], Naval Research Laboratory [START_REF] Stroman | Liquid hydrogen fuel system design and demonstration in a small long endurance air vehicle[END_REF][START_REF] Gundlach | Unmanned Aircraft Systems Innovation at the Naval Research Laboratory (Library of Flight)[END_REF], Boeing RD Europe [START_REF] Troncoso | Solar-powered hydrogen refuelling station for unmanned aerial vehicles: Design and initial AC test results[END_REF][START_REF] Troncoso | Off-grid test results of a solar-powered hydrogen refuelling station for fuel cell powered Unmanned Aerial Vehicles[END_REF][START_REF] Troncoso | Design tool for offgrid hydrogen refuelling systems for aerospace applications[END_REF], Florida Solar Energy Center [START_REF] Garceau | Performance test of a 6 L liquid hydrogen fuel tank for unmanned aerial vehicles[END_REF], it can be concluded that liquid H 2 is not a suitable solution due to constraints related to its infrastructure [START_REF] Gong | Fuel cell propulsion in small fixed-wing unmanned aerial vehicles: Current status and research needs[END_REF]. Even if in another research at the Colorado State University [START_REF] Rhoads | Design and Flight Test Results for a 24 Hour Fuel Cell Unmanned Aerial Vehicle[END_REF], a fuel cell powered drone was designed to reach more than 24 hours flight-test using compressed hydrogen. Regarding chemical hydrogen generation, it typically needs extra equipment for hydrogen extraction therefore leading to a heavy and complex power supply system. Furthermore, hydrogen extraction is time-consuming therefore impacting the drone response-time to load changes. In [START_REF] Kim | Design and development of a fuel cell-powered small unmanned aircraft[END_REF], Kim and Kwon presented performance analysis of a fuel system with a hydrogen generator powering a small drone under real flight test of 2 hours. In this context, stack and reactor temperatures, hydrogen generation rate, and fuel cell output power were evaluated under varied power demand. This study validated the fuel cell ability to power a successful flight. However, some related issues have been highlighted, such as startup time, catalyst durability, and clogging caused by the by-product.

N aBH 4 + 2H 2 O → 4H 2 + N aBO 2 N H 3 BH 3 → N H 2 BH 2 +H 2 CH 2 O 2 → H 2 + CO 2

Properties

Furthermore, the developed system in [START_REF] Swider-Lyons | Hydrogen Fule Cell Propulsion for Long Endurance Small UVAs[END_REF] based on Protonex 550 W PEM fuel cell proved by a flight test 6 times endurance increase than Li-ion battery system.

Hybrid Power Sources Fuel cell and battery

Despite their performance and technological advancements, fuel cells have some limitations when used as a unique power source for drones. Indeed, a fuel cell is characterized by a large time-constant (in the order of seconds) since it needs to be supplied by fuel and air using pumps, valves, and compressors. The slow response is mainly due to pumps mechanical characteristics, flow delay, thermodynamic characteristics, and the capacitance effect [START_REF] Ou | Feedforward fuzzy-PID control for air flow regulation of PEM fuel cell system[END_REF]. Thus, in case of high variation in current demand, fuel starvation problem can occur, which can affect lifetime, reliability, and efficiency [START_REF] Ou | Optimized power management based on adaptive-PMP algorithm for a REFERENCES stationary PEM fuel cell/battery hybrid system[END_REF].

Combining a fuel cell with battery to form a hybrid power supply system seems therefore to be a good option that enables the drone propulsion system to benefit from advantages of both sources and to balance their drawbacks [START_REF] Donateo | A new approach to calculating endurance in electric flight and comparing fuel cells and batteries[END_REF][START_REF] Belmonte | Fuel cell powered octocopter for inspection of mobile cranes: Design, cost analysis and environmental impacts[END_REF][START_REF] Cooley | Multiconverter system design for fuel cell buffering and diagnostics under uav load profiles[END_REF].

Indeed, battery since it has higher power density, faster response, and higher efficiency than a fuel cell, will be selected to supply the required peak-power, when drone conducts typical maneuvers such as take-off and climbing. The fuel cell will afterward be the main supply in cruise or descend periods. It can also charge the battery to keep the SOC higher than the prescribed threshold.

Drone Power Sources

In [START_REF] Verstraete | Hardware-in-the-loop simulation of fuel-cell-based hybrid-electrical UAV propulsion[END_REF], Verstraete et al. discussed the performance of a hybrid drone propulsion system powered by a 200 W fuel cell and a battery by mean of hardwarein-the-loop (HIL) simulations. The behavior of each source under different tests was highlighted and analyzed. This study also considered endurance and hydrogen use. In a similar study in [START_REF] Gong | Hardware-in-the-loop simulation of a fuel-cell-based UAV propulsion system using real-world flight data[END_REF], HIL-based on real flight recorded data was performed under different load fluctuation degrees. In [START_REF] Gong | Role of battery in a hybrid electrical fuel cell uav propulsion system[END_REF], the authors focused their study on the battery contribution to the hybrid system. Their experimental investigation emphasizes the performance of the battery under several requests that can be undergone during flight mission phases. Detailed characterization of this propulsion system is presented in [START_REF] Gong | Performance of a hybrid, fuel-cell-based power system during simulated small unmanned aircraft missions[END_REF], using different mission profiles and speeds. However, the energy management strategy was not developed in these studies as power splitting was only conducted by a passive method.

Solar cells as an auxiliary power source

Application of photovoltaic (PV) generation system in moving carriers, such as drones is receiving considerable attention. In this case, a drone outfitted by PV arrays on its wings can indefinitely fly providing that battery is installed for energy storage to supply at night or in case of sun unavailability [START_REF] Shiau | Design of a Solar Power Management System for an Experimental UAV[END_REF]. Solar powered-drone are typically designed and widely used for HALE applications as illustrated by Fig. 22. HALE drones are designed to perform persistent missions (more than 1 day) at a very high altitude. Morton et al. [START_REF] Morton | Solar powered uav: Design and experiments[END_REF] proposed a solar-powered drone design method to optimize the airframe efficiency. The experimental tests on the developed prototype have shown that the amount of solar energy received was sufficient for the drone to carry the additional payload of the solar system extending its endurance [START_REF] Morton | Solar powered uav: Design and experiments[END_REF].

In [START_REF] Harvey | Development of a hybrid-electric power-system model for a small surveillance aircraft[END_REF], Harvey et al. proved that using PVs might enable up to 59% of fuel savings in addition to reducing the drone weight. Then, exploiting solar energy is a relevant contribution to drone endurance improvement. As shown by Fig. 22, solar powered-drones must have large wings in order to maximize the amount of received light energy. In this context, a maximum power point tracking (MPPT) algorithm is required [START_REF] Peng | A novel tangent error maximum power point tracking algorithm for photovoltaic system under fast multi-changing solar irradiances[END_REF]. In this context, the MPPT hardware system includes a simple converter associated with a low-cost microcontroller, current, and voltage sensors. In [START_REF] Shiau | Design of a Solar Power Management System for an Experimental UAV[END_REF], Shiau et al. proposed the design and validation of a solar power management system (SPMS) for their experimental drone powered by solar cells and batteries. The SPMS includes

Chapter II. State of the Art Critical Review on Drones Power Supply and Energy Management three cascaded stages: An MPPT stage to maximize the PV power under temperature and solar radiation variability. Then, the battery management block controls energy storage and delivery. The last stage is the DC/DC converter providing +5 V and +12 V power sources and supplying all the onboard electronic circuitries. However, this study does not consider the propulsion power in both energy management process and design. In [START_REF] Peng | A novel tangent error maximum power point tracking algorithm for photovoltaic system under fast multi-changing solar irradiances[END_REF], Pen et al. have proposed an interesting perturb and observe-based MPPT algorithm to achieve high efficiency for PV systems under fast multi-changing solar irradiances, which could be the case of drones.

FIGURE 22: Solar-powered drones [START_REF] Zhang | An overview of the global solar-powered UAV market 2017-2021[END_REF][START_REF] Conner | Helios Prototype Flying Wing[END_REF].

Supercapacitor as an auxiliary power source

Supercapacitors are recently attracting attention as faster energy storage systems are needed in a number of applications to replace or complement batteries, which suffer from sluggish charge/discharge with a limited lifetime [START_REF] Aneke | Energy storage technologies and real life applications-a state of the art review[END_REF].

A supercapacitor is characterized by a much higher power and much lower energy densities when compared to a battery. In addition, it operates in a large temperature range, with overcharge tolerance, low maintenance cost, and a reasonable cost [START_REF] Ruan | An investigation of hybrid energy storage system in multi-speed electric vehicle[END_REF]. It can also extremely reduce the DC bus voltage fluctuations. Table V.2 provides a comparative analysis of batteries and supercapacitor main characteristics.

In this context, integrating a supercapacitor as an additional power in a drone hybrid power supply will offer an additional degree of freedom in term of supplying architectures, while reinforcing power density and allowing rapid power response (Fig. 23). In hybrid power supply architecture, the fuel cell is typically the main power source, while the others are auxiliary ones. The fuel cell will therefore be selected to power the drone steady state therefore extending its lifetime [START_REF] Li | A novel equivalent consumption minimization strategy for hybrid electric vehicle powered by fuel cell, battery and supercapacitor[END_REF]. In this case an EMS is mandatory to enable each power source working in its optimal conditions. In [START_REF] Gong | Analysis of a Fuel-Cell/Battery/Supercapacitor Hybrid Propulsion System for a UAV using a Hardware-in-the-Loop Flight Simulator[END_REF], Gong et al. provided a HIL-based evaluation and analysis of a hybrid drone propulsion system including fuel cell, battery, and supercapacitor. The system was compared to a fuel cell/battery system under a flight profile to find out the role of the supercapacitor. In addition, the effect of supercapacitor capacity on fuel cell and supercapacitor behaviors were studied. The achieved results have shown good performance of the supercapacitor in load smoothing and dynamic response. In an additional work on the same drone configuration [START_REF] Gong | Flight test of a fuelcell/battery/supercapacitor triple hybrid uav propulsion system[END_REF], the authors conducted two flight tests with a real drone prototype (Fig. 23). It has been therefore clearly shown the supercapacitor considerable contribution in delivering peak power and absorbing power fluctuations during a dynamic flight with rapid changes in power load. The DC bus voltage stabilization has been also shown. The EMS strategy has not been unfortunately discussed.

A critical comparative analysis on drone hybrid power supply configurations is proposed in Table II.10.

Energy sources

Architecture

Advantages Limitations and drawbacks Related papers

Thermal energy

Gas turbine engine

High power-to-weight ratio and long operating time.

Very bad fuel economy and high noise level.

[120, 121] ICE Very high power and energy densities, high endurance, large payload range.

Reduced efficiency, thermal and acoustic signatures, GHG emission, fuel high cost.

[ [START_REF] Bongermino | Model and energy management system for a parallel hybrid electric unmanned aerial vehicle[END_REF][START_REF] Glassock | Multimodal hybrid powerplant for unmanned aerial systems (uas) robotics[END_REF][START_REF] Xie | Fuzzy logic based equivalent consumption optimization of a hybrid electric propulsion system for unmanned aerial vehicles[END_REF][START_REF] Donateo | Fuel economy of hybrid electric flight[END_REF] One electrical source Battery High energy density, energy stored (not generated) ⇒ quick response to power demand.

Low power density, reduced endurance, long recharging time with ''memory effect" for some battery types, to increase autonomy ⇒ add more batteries ⇒ increase weight and cost. [START_REF] Traub | Range and Endurance Estimates for Battery-Powered Aircraft[END_REF][START_REF] Mike | Whitepaper: Fuel Cell-Based Energy Systems for Commercial UAVs[END_REF] Fuel cell High energy density, instantly refueling without ''memory effect" , to increase autonomy ⇒ use more fuel in the same stack (weight reduction).

Energy generated ⇒ slow response to power demand, auxiliary equipment are required (compressors, regulator, etc), lack of infrastructures for hydrogen distribution, issues of hydrogen storage, safety concerns, hydrogen production high cost.

[60, 61, 162]

Hybrid power supply

Fuel cell and Battery

High energy and power densities ⇒ increase in the endurance and the response time, energy generation and storage.

Increase of weight, EMS is needed (controllers and converters) ⇒ additional weight and complexity. [START_REF] Kim | Design and development of a fuel cell-powered small unmanned aircraft[END_REF][START_REF] Verstraete | Hardware-in-the-loop simulation of fuel-cell-based hybrid-electrical UAV propulsion[END_REF][START_REF] Gong | Hardware-in-the-loop simulation of a fuel-cell-based UAV propulsion system using real-world flight data[END_REF][START_REF] Gong | Role of battery in a hybrid electrical fuel cell uav propulsion system[END_REF][START_REF] Karunarathne | Model based power and energy management system for pem fuel cell/li-ion battery driven propulsion system[END_REF][START_REF] Zhang | Experimental investigation on the online fuzzy energy management of hybrid fuel cell/battery power system for UAVs[END_REF][START_REF] Karunarathne | Adaptive neuro fuzzy inference system-based intelligent power management strategies for fuel cell/battery driven unmanned electric aerial vehicle[END_REF][START_REF] Yang | The Testing Platform of Hybrid Electric Power System for a Fuel Cell Unmanned Aerial Vehicle[END_REF] Fuel cell, Battery, and solar cells Additional energy source ⇒ endurance improvement, clean, free and available energy ⇒ decrease in energy cost, hydrogen saving.

Necessity of large drone wings, cannot be used in rotary-wing drones, an energy storage device is needed, EMS and MPPT are required. [START_REF] Shiau | Design of a Solar Power Management System for an Experimental UAV[END_REF][START_REF] Lee | Power managements of a hybrid electric propulsion system for UAVs[END_REF][START_REF] Lee | Active power management system for an unmanned aerial vehicle powered by solar cells, a fuel cell, and batteries[END_REF] [ [START_REF] Gang | Design of an energy management technique for high endurance unmanned aerial vehicles powered by fuel and solar cell systems[END_REF][START_REF] Zhang | Fuzzy State Machine Energy Management Strategy for Hybrid Electric UAVs with PV/Fuel Cell/Battery Power System[END_REF][START_REF] Gao | Energy management strategy for solar-powered high-altitude long-endurance aircraft[END_REF] Fuel cell, Battery, and supercapacitor Very high power density, fast charging, reduced weight, and reduced dc bus fluctuations; very long lifetime, minimum heat loss due to the reduced internal resistance.

EMS is needed, supercapacitor voltage regulation is necessary. [START_REF] Gong | Analysis of a Fuel-Cell/Battery/Supercapacitor Hybrid Propulsion System for a UAV using a Hardware-in-the-Loop Flight Simulator[END_REF][START_REF] Gong | Flight test of a fuelcell/battery/supercapacitor triple hybrid uav propulsion system[END_REF] Hybridization is the most suitable architecture to power the propulsion system in drones. It allows combining advantages and performance of different power sources, and balancing their limitations. Thus, power has to be optimally split between sources to achieve an efficient energy usage and to enable power sources high performance operation, while extending as long as possible their lifetime. Thereby, a power management system or strategy (PMS) must be implemented for power real time splitting among the available sources, while considering constraints such as efficiency, fast response, fuel consumption, required power, and flight conditions. This approach is an active power management strategy, where the power outputs are controlled through converters by the energy management unit. Power can also be supplied using a passive method, which is widely used for small drones as in [START_REF] Liu | Modeling and Control of a Power-Split Hybrid Vehicle[END_REF] and [START_REF] Zandi | Energy Management of a Fuel Cell/Supercapacitor/Battery Power Source for Electric Vehicular Applications[END_REF]. In this case, the power sources are directly connected to a DC link and supply the propulsion according to their own characteristics. Neither additional power converters nor controllers are needed, therefore considerably reducing complexity, weight, and power losses [START_REF] Lee | Active power management system for an unmanned aerial vehicle powered by solar cells, a fuel cell, and batteries[END_REF]. Hardware architectures of both passive and active PMSs are depicted in Fig. 24. In [START_REF] Lee | Active power management system for an unmanned aerial vehicle powered by solar cells, a fuel cell, and batteries[END_REF], Lee et al. investigated benefits and drawbacks of both active and passive PMSs using flight test results and power simulation. The studied hybrid power supply system is composed of a fuel cell, a battery, and solar cells. It is intended for low-speed long-endurance drones. Simulation results have shown that the passive PMS could not maintain the battery minimum SOC, therefore affecting its lifetime and increasing the system failure possibilities. On the other hand, power losses reached 4.7% when using an active PMS. In this context, the two PMSs were not evaluated in the same conditions. Indeed, the passive PMS was just simulated, while the active one was experimentally implemented. In addition, many flight-related conditions were neglected. Table II.11 presents a comparison between active and passive PMSs. Hereafter, we will focus on the main active PMSs proposed in the literature for drones energy management.

Rule-based Strategies

Rule-based control is one of the most widespread control strategies due to its simplicity, reliability in management using predefined conditions (if-then rules). It is typically characterized by a very low computational cost, enabling online EMS implementation [START_REF] Karunarathne | Model based power and energy management system for pem fuel cell/li-ion battery driven propulsion system[END_REF]. In [START_REF] Lee | Active power management system for an unmanned aerial vehicle powered by solar cells, a fuel cell, and batteries[END_REF], Lee et al. investigated a PMS to control a drone propelled by a hybrid power system including a fuel cell, a battery, and solar cells. The proposed PMS takes into account the required power and the battery SOC to assign each power source output. In this study, solar cells are used as the primary source since it requires no onboard fuel. The PMS considers power outputs as control variables and sets terminal voltage (20-36 V) of each source by means of DC-DC converters. The fuel cell supplies the drone only in a defined power interval (50-180 W), to keep it working in nominal conditions. In parallel with the PMS, the battery management system prevents battery from overcharging when it is fully charged. A minimum SOC of 45% was prescribed for drone safe operation.

In a recent paper [START_REF] Yang | The Testing Platform of Hybrid Electric Power System for a Fuel Cell Unmanned Aerial Vehicle[END_REF], Yang et al. proposed a state machine strategy for a fuel cell/battery drone. In this case a control logic divides the decision area

Power Management Strategies

into five states based on demand power and battery SOC values. The hybrid power system architecture includes two converters, where one is bidirectional to control battery charging/discharging. Furthermore, two PI controllers are used to regulate power and voltage references. To validate the proposed PMS, authors conducted experimental tests soliciting the power system by load profiles with different SOC initial conditions. This study was unfortunately not implemented in a real drone and tested on ground.

In [START_REF] Gao | Energy management strategy for solar-powered high-altitude long-endurance aircraft[END_REF], Xian-Zhong et al. proposed an EMS for a PV/battery-powered longendurance drone. In a first phase, the available PV energy is split into three parts, where the first one powers the drone, the second one is stored to be used in a next phase, and the last part is used to charge the battery. The second phase starts when solar irradiance decreases. In this case, the drone power deficit is covered in part by the stored energy and the use of gravitational gliding. The last phase, in case of a total solar power deficit, the battery powers the drone at low altitude and enable a safe landing (end of mission).

In this simulation study, the proposed PMS considers the wind effect and has shown about 23.5% of energy conservation in comparison with another management strategy. The availability of such kind of drone is unfortunately strongly dependent on solar energy.

In a recent study [START_REF] Gang | Design of an energy management technique for high endurance unmanned aerial vehicles powered by fuel and solar cell systems[END_REF], Gang and Kwon proposed a power switching technique based on inter-changing power supply between a PV system (including a battery) and a fuel cell-based one. Solid-state relays are used to select either the most suitable power source or both, according to their states, power requirements, and flight conditions. When the PMS activates the fuel cell system, it delivers the load power, while the PV system is switched to standby mode. In this period, the generated PV power is used to charge the battery. The authors designed and constructed their hybrid drone, and they conducted a flight test over a period of 1.5 h. However, neither the switching rules nor the control algorithm were explained.

In [START_REF] Savvaris | Development of a fuel cell hybrid-powered unmanned aerial vehicle[END_REF], Savvaris et al. proposed a simple rule-based algorithm to control a battery/fuel cell hybrid system. Power is adopted as a variable of control instead of current. Relays are used to either activate or deactivate each source power flow. In this context, three operation modes have been considered: The parallel mode, when the two sources are supplying the drone; the charging mode, when the battery is charging; and the load following mode (between the previous modes). Beside simulations, HIL-based experiments were carried out. However, there were no flight test with a real drone and the endurance issue has not been discussed.

A constrained thermostat control (CTC) strategy was proposed in [START_REF] Lee | Power managements of a hybrid electric propulsion system for UAVs[END_REF]. The algorithm set solar cells as the primary source, which can charge the battery in case of extra power. Furthermore, the strategy keeps a low threshold SOC of the battery (30%) during flight to ensure safe landing when solar cells and the fuel cell cannot cover the necessary power. The battery contributes to the power supply as long as the SOC is higher than the prescribed threshold. This was a simulation-based study that needs at least HIL-based validation.

Fuzzy Logic Strategies

A fuzzy logic-based PMS can be implemented to enhance power allocation for drone hybrid power supply system increasing energy efficiency. The fuzzy control algorithm uses inputs such as battery SOC, power demand (P D ), and photovoltaic power (if PV panels are used); and then generates control commands (i.e. fuel cell power P F C ) respecting the pre-set rules. These rules determine the management strategy and set priorities and constraints. In general, PV power has the highest priority to supply the drone, while fuel cell power has the lowest one. Table II.12 shows an example of fuzzification for a battery/fuel cell control system [START_REF] Zhang | Experimental investigation on the online fuzzy energy management of hybrid fuel cell/battery power system for UAVs[END_REF]. The battery SOC is classified as: low (L), medium (M), and high (H), respectively. The power demand P D has five fuzzy states: medium (M), high (H), very high (VH), low (L), and very low (VL). Similarly, the fuzzy output P F C has five fuzzy states: M, VH, H, L, and VL.

TABLE II.12: Fuzzificationn for a battery/fuel cell power control system [START_REF] Zhang | Experimental investigation on the online fuzzy energy management of hybrid fuel cell/battery power system for UAVs[END_REF].

P F C P D VH H M L VL SOC L VH VH H M L M VH H M L L H H M L VL VL
In [START_REF] Zhang | Experimental investigation on the online fuzzy energy management of hybrid fuel cell/battery power system for UAVs[END_REF], Zhang et al. proposed an online fuzzy EMS for a drone propelled by a hybrid fuel cell/battery power system. The designed EMS was experimentally tested. In this study, the fuel cell supplies the propulsion through

Power Management Strategies

a programmable DC/DC converter, which controls its current output. The battery is directly linked to the DC bus with no converter. Its output current is in this case indirectly determined by the power balance principle. The implemented algorithm uses the battery SOC and the power demand as input variables and calculates, under a fuzzy process, the fuel cell power as the output variable. The fuzzy EMS was compared with state machine and passive control strategies, using three types of flight missions: pulsed-power mission, flight-power mission, and long endurance mission. The proposed management strategy shows good performance regarding fuel consumption and battery SOC. However, it was implemented in a test-bench using programmable electric load to emulate power demand profiles. No real flights were carried out.

In [START_REF] Karunarathne | Model based power and energy management system for pem fuel cell/li-ion battery driven propulsion system[END_REF][START_REF] Karunarathne | Adaptive neuro fuzzy inference system-based intelligent power management strategies for fuel cell/battery driven unmanned electric aerial vehicle[END_REF], Karunarathne et al. proposed an energy and power management system to optimize power splitting between a PEMFC cell and a Li-ion battery. In the first stage, the EMS aims to reduce oxygen concentration voltage losses. It makes long term decisions based on the required power, the battery SOC, and the PEMFC control parameters. In the second stage, the PMS receives those decisions and deals with a short-time implementation. The PMS uses a rule-based system to define the PEMFC output through a unidirectional converter and controls the DC/DC bidirectional converter to enable the battery charging/discharging by switching it to buck/boost modes. The EMS acts on the PEMFC compressor motor voltage to control the inlet airflow rate. To set the compressor voltage to the optimum value, an ANFIS-based (Adaptive Neuro-Fuzzy Inference System) adaptive controller is used to predict the membership function estimation parameters. However, this study is still at the simulation stage with no experimental investigations.

In [START_REF] Zhang | Fuzzy State Machine Energy Management Strategy for Hybrid Electric UAVs with PV/Fuel Cell/Battery Power System[END_REF], Zhang et al. introduced a hybrid approach combining fuzzy logic and state machine strategies. The fuzzy logic strategy is used for power allocation for the drone battery and fuel cell, when the state machine strategy is in charge of power management for the solar cells and battery. A mission scenario was implemented in a simulation platform to study the power sources behaviors. The proposed strategy was compared to the thermostat control strategy [START_REF] Lee | Power managements of a hybrid electric propulsion system for UAVs[END_REF], implemented for the similar drone. The carried out comparison considers the battery SOC, fuel utilization, and each source contribution proportion in the supplying process.

Far from the rule-based and fuzzy logic EMS/PMS approaches, an optimal EMS using dynamic programming was introduced in [START_REF] Bradley | Energy Management for Fuel Cell Powered Hybrid-Electric Aircraft[END_REF]. The proposed algorithm was tested with several flight scenarios and different hybridization degrees (batteries contribution in power supplying), and considering fuel consumption. The investigations aimed to determine the best architecture regarding endurance improvement. This study leads to the main conclusion that hybridization can be beneficial in term of endurance only in case of the fuel cell inability to ensure the drone supplying as a unique source. The computational complexity of such an optimal method may be an obstacle for online usage.

A critical comparative analysis of drone EMS/PMS main literature is proposed in Table II.13. No flight test, endurance not discussed.

[16] 2013 Solar cell/ Battery Simulations Three-stage EMS for solarpowered drone.

High dependence on solar energy.

[195] 2009 FC/ Battery Simulations Optimal EMS using dynamic programming.

Optimization high computational cost.

Conclusions and future trends

Drones have been in continuous development and they are reaching a large range of applications. In this context and as the propulsion system constitutes the mainstay of a drone platform, this chapter focused on the onboard propulsion system energy aspect by proposing a comparative and critical

Conclusions and future trends

state of the art review on drones power supplying architectures and suitable energy/power management strategies. This comparative and critical study intends providing a starting basis for the development of performing drone power propulsion systems by facilitating trade-off in the choice of power sources. One source-based drone power supply system will be greatly limited because of their poor performance under varying operating conditions. Thus, hybridization of power sources with different characteristics is becoming a standard solution in designing a drone electric power system. In this context, hybrid power sources choice and sizing are strongly depending on drone's tasks and corresponding weight/duration constraints.

This chapter also discussed the issue of limited endurance for specific missions. Indeed, it has been presented and discussed typical drones power supplying techniques, namely swapping, laser-beam inflight recharging, and drone tethering. As it has been clearly shown in the state of art review, there are few EMS/PMS specific approaches for drones in comparison to electric vehicles. This is mainly due to drones specificities and constraints, while there were only few flight test studies investigating energy management strategies. In this context, the problem of energy optimization is hardly constrained by the drone weight that limits the onboard computational capacity for real-time optimization, in addition to the fact that embedded processors need to be supplied, therefore affecting the endurance. In this context, off-line optimization could be considered as a trend, when many constraints of on-line optimization can be overcome. Indeed, recent works proposed drones energy consumption prediction based on an a priori knowledge of the scheduled mission (profile, maneuvering actions, duration, etc.) [START_REF] Abeywickrama | Comprehensive energy consumption model for unmanned aerial vehicles, based on empirical studies of battery performance[END_REF][START_REF] Prasetia | Mission-based energy consumption prediction of multirotor uav[END_REF].

Introduction

In the previous chapter II, we proposed a comprehensive and critical evaluation of the available power supply structures and their energy management strategies identifying some gaps. Rule-based energy management strategies are widely used in drones due to their simplicity and capability to be implemented in real-time [START_REF] Zhang | Energy management strategy of hybrid pemfc-pv-battery propulsion system for low altitude uavs[END_REF][START_REF] Boukoberine | A critical review on unmanned aerial vehicles power supply and energy management: Solutions, strategies, and prospects[END_REF]. The frequency separation approach is already used in the literature for hybrid electric vehicles [START_REF] Rasheed | Power sharing based algorithm for sizing components in fuel cell hybrid electric vehicles[END_REF]. Zhang et al. [START_REF] Zhang | Experimental investigation on the online fuzzy energy management of hybrid fuel cell/battery power system for uavs[END_REF] investigated a fuzzy rule-based EMS for a fuel cell/battery drone. However, the authors didn't consider the specificity of the drone request power, they have just used a pulsed-power profile for the load. Indeed, many researchers design their own power profiles to simulate the requested power during the flight mission due to the difficulty to get real data [START_REF] Xie | Fuzzy logic based equivalent consumption optimization of a hybrid electric propulsion system for unmanned aerial vehicles[END_REF].

In this chapter, hybrid power supply system modeling will be proposed and it will be used for the rest of the studies carried out in this thesis. Then, the conducted flight test will be presented. After that, a combination of rulebased and frequency separation power management strategies is proposed for a hybrid drone power supply system consisting of fuel cell, battery, and supercapacitor, while using real flight test data for the load power demand.

Hybrid Power Supply System Topology

Integrating an energy storage system to the fuel cell based power supply system can be done respecting several architectures proposed in the literature. It is necessary to choose topology that fits well to our drone platform considering the overall system efficiency, weight, and cost. Architectures using direct connection or one DC converter present the advantage of better efficiency, however, they don't enable sources power controllability. Hereinafter we will discuss only topologies that permit to implement an effective power/energy management strategy.

Since the fuel cell is the primary power source, its power output value is assigned by the energy management system. Then, its output has to be controlled using a DC/DC boost converter. In addition, its output voltage highly fluctuates with the output power, so it is better to not connect it directly to the DC bus voltage. The battery can be directly linked to the DC bus as an energy buffer reducing the power losses and the system weight by eliminating a DC/DC converter. In this case, the battery should be oversized to 3. Flight Test match the DC bus voltage, and its charging/discharging current cannot be controlled. At any time an overcharging or overdischarging might take place due to power demand peaks reducing the battery lifetime. Therefore, connecting battery to a bidirectional DC/DC converter is important to overcome these drawbacks and also to enable DC bus voltage regulation. Using two converters, the fuel cell is in charge of supplying the low frequency or the steady-state load power, or charging the battery when needed. The higher frequency components will be supplied by the battery, which is also controlling the DC bus voltage. However, the battery will undergo a high number of charge/discharge cycles affecting its lifetime.

The role of the supercapacitor, which is characterized by a very high number of charge/discharge cycles, is to take in charge this high frequency power fluctuations, thus smoothing the battery output power curve and extending its lifetime. The supercapacitor will be directly linked to the DC bus to rapidly react to the load transients. Adding a third DC/DC converter will make the system more heavy, complex, and expensive. It is worth noting that the degree of contribution of the supercapacitor in power supplying depends on the DC bus voltage variation range. The more this range is narrow, the more the contribution is limited, so there is a trade-off to do. The selected topology for the studied system is depicted in Fig. 1.

Flight Test

In our study, an experimental flight test was carried out using a small batterypowered Y6 hexacopter from 3D Robotics company in order to extract a real load power profile. As shown in Fig. 2a, the drone has three arms in Yshape with two counter-rotating propellers for each arm to provide lift, acceleration, and stability. Table III.1 presents the drone specifications and features [START_REF] Jeff | Iproduct Review: 3D Robotics RTF Y6 Multicopter[END_REF]. To interface the drone with the ground control system, the "APM Planner 2.0" software is used.

It is an open-source ground station application for management and control of the autopilot (Pixhawk) based on the MavLink (Micro Air Vehicle Link) protocol. It can be executed on Windows, Linux, and Mac OSX. The MavLink is a communication protocol for small multirotors. It enables real-time data monitoring such as battery output voltage and current. Furthermore, it is possible to get the real position through the GPS module mounted on board. In the "Flight Plan" section we can plan a flight mission that can be autonomously performed by the drone. With the help of a micro SD mounted on board, the Pixhawk is able to record all data and transmit them via radio to the ground station. Once the extension file (bin) has been obtained, it is necessary to convert it into the .log extension to make it readable with Excel. It is worth noting that data are sampled with a frequency of 50 Hz according to the acquisition card. However, 1secaveraged data are stored in the log file.

In order to gather data with a good accuracy and to ensure flight safety, it is necessary to calibrate the sensors particularly accelerometers, compass, radio control, power module, and the electronic speed controller (ESC).

The test was conducted in Salento university campus in Italy [START_REF] Donateo | Design and performance evaluation of a hybrid electric power system for multicopters[END_REF] (Fig. 2b).

The load power curve consists of the battery delivered power during the mission, including take-off maneuver and hovering operation (Fig. 5). The other power request profiles for larger multicopters can also be estimated by assuming the proportionality between load power and drone weight with a 3/2 factor [START_REF] Johnson | Rotorcraft aeromechanics[END_REF]. Table III.2 presents the obtained power profile characteristics. To get a larger mission lifetime and to solicit even more our power supply system, the load profile is repeated several times replicating the hovering phase and extending the flight test duration to 42min as depicted in Fig. 4.

Power Requirements and Components Selection

The power system should be designed according to the load characteristics.

In our study, a small battery-powered Y6 hexacopter was used to extract a real load power profile. university campus in Italy [START_REF] Donateo | Design and performance evaluation of a hybrid electric power system for multicopters[END_REF]. The load power curve consist of the battery delivered power during the test, including take-off maneuver and hovering operation (Fig. 5). Other power demand curves for larger hexacopters can be deducted considering the proportionality between load power and drone weight with a factor of 2/3 [START_REF] Johnson | Rotorcraft aeromechanics[END_REF]. Table III.2 presents the load characteristics for the obtained power profile. Once the power profile is analyzed, we can proceed to components selection. The fuel cell nominal power should be between the average power and the maximum power. The nominal power can easily be set as the average of these two values [START_REF] Garcia | Energy management system of fuel-cell-battery hybrid tramway[END_REF]. However, due to power peaks which occur in short periods, the fuel cell risks to be oversized. Besides, being close to the average value leads to oversizing the battery to enable it supporting the additional stress. Since the fuel cell has slow dynamics in terms of power response, it will supply the power profile low frequency components (Fig. 5). Its rated power is then set to be equal to the maximal power value of the low frequency profile (1000 W). The battery maximum power can then be set to be equal to the difference between the maximal power and the fuel cell rated power (623 W).

For the supercapacitor, its peak power has to match the maximum load profile peak power (912 W), and its usable energy will depend on the duration of this peak (5 sec). By integrating the high frequency profile we can get its energy variation. The battery energy capacity can be then obtained using the maximal energy variation value ∆E max = 1633 kJ.

Considering a general charge/discharge efficiency η equal to 0.85, and a depth of discharge (DOD) of 40 % for both battery and supercapacitor, we get the final requirements Fuel cell and battery modeling literature presents different approaches, namely, empirical [START_REF] Meiler | Dynamic fuel cell stack model for real-time simulation based on system identification[END_REF], electro-chemical [START_REF] Martín | Modelling of PEM fuel cell performance: Steady-state and dynamic experimental validation[END_REF], and electric-circuit based modeling [START_REF] Ferrero | Simplified model for evaluating ripple effects on commercial PEM fuel cell[END_REF]. Empirical models are based on look-up tables or curve fitting using experimental data such as charge/discharge and polarization curves. When parameters match with an already identified case, the outputs fit exactly the measured data. Otherwise, an error will occur. This approach is simple, but the model will be restricted to the studied power source and cannot be generalized. Electro-chemical models are based on electro-chemical and thermodynamics equations, which include several sub-components such as electrode and catalyst layer. They are primarily used for design purposes representing the power source dynamic behavior and thermodynamic phenomena. However, they require specific parameters real data as electrolyte volume and electrode thickness, which are not easily available. In addition, it is worth mentioning that both previous approaches do not take into account battery SOC dynamics. Electric circuit-based modeling involves use of ideal electrical elements to represent power sources characteristics such as ohmic losses, open circuit voltage, and capacity. Experimental tests, as impedance spectroscopy and current interrupt, or frequency response, should be conducted to identify these parameters [START_REF] Njoya | A generic fuel cell model for the simulation of fuel cell vehicles[END_REF]. Although these models represent electrical properties, they are operating conditions-dependent.

                                   P F C-nom = 1000
The adopted modeling approach was developed in [START_REF] Njoya | A generic fuel cell model for the simulation of fuel cell vehicles[END_REF] and the obtained generic model is available in Simulink/SimPowerSystems (SPS). The method consists in a combination between curve fitting characteristics and electric circuit-based models. This model is appropriate to electrical simulations and its parameters can easily be derived from the manufacturer datasheet.

The hybrid energy system is simulated under Matlab/Simulik using the predefined generic models available in SimPowerSystems toolbox for power and energy sources. Model parameters are extracted from the Intelligent Energy 650W fuel cell power module specifications already listed in Table IV.1. For DC/DC converters, average value models are adopted and implemented.

Fuel cell model

PEM fuel cells are widely used for drones due to their good features such as fast starting and low operating temperature reducing the warm-up time,

Hybrid Power System Modeling

thus fast response can be achieved. The selected model is developed in [START_REF] Njoya | A generic fuel cell model for the simulation of fuel cell vehicles[END_REF].

The fuel cell output voltage V f c is given by

V f c = E OC -V act -V Ohm (III.2) with V act = A.ln(i f c /i 0 ). 1 1 + s.T d /3
(III.3)

V Ohm = r Ohm .i f c (III.4)
where E OC is the open circuit voltage (V) and it is determined by the chemical thermodynamics of the overall cell reaction. V act is the activation voltage drop and it is due to the slowness of the reactions on the electrode surface.

V Ohm represents the ohmic losses due to the resistance to the flow of electrons through the cell hardware and various interconnections, and ions in the electrolyte. A is called Tafel slope (V), and i 0 is the exchange current (A). Indeed, the cell voltage cannot track current changes instantaneously. The voltage response delay to a current variation, which is caused by the charge double layer phenomenon, is represented by T d as shown in Fig. 6. A diode is used to prevent any eventual reverse current flow. Normally, the boost converter plays that role when connected to the stack.

Under nominal operating conditions, only four points from the polarization curves are needed to determine the model parameters. However, if the operating conditions such as pressure, temperature, air and fuel flow rates are variable, parameters (E OC , A, i 0 ) are updated.

Battery model

A Li-ion battery type is adopted in this study since it is characterized by high energy density and efficiency relatively to other battery types. Hence, it is a suitable battery type for drones applications. The adopted model is based on an updated Shepherd curve fitting model where a voltage polarization term is included to represent battery SOC effect. In addition, to guarantee simulation stability, a filtered battery current (i * ) is used to calculate the polarization resistance P ol res . The battery voltage can be written as [START_REF] Tremblay | Experimental validation of a battery dynamic model for EV applications[END_REF] V

bat = E 0 -R int .i -K Q Q -it .it -K Q Q -it .i * + A. exp(-B.it) (III.5)
where E 0 represents the battery constant voltage (V) and R int is the internal resistance. Q and it = i.dt refer to the battery capacity and actual charge (Ah), respectively. The polarization constant is represented by K (V/Ah), and B is the time constant for the exponential area (Ah) -1 . The polarization resistance P ol res is represented by the term K Q Q-it , while the polarization voltage is represented by the term

K Q Q-it .it.
As in the fuel cell case, the model parameters can be set using the manufacturer datasheet or by conducting a polarization test. The model equivalent circuit is shown in Fig. 7.

Supercapacitor model

The supercapacitor voltage can be written as [START_REF] Motapon | A comparative study of energy management schemes for a fuel-cell hybrid emergency REFERENCES power system of more-electric aircraft[END_REF] 5. Hybrid Power System Modeling

V SC = Q T C T -R SC i SC (III.6)
where C T is the total capacitance (F), Q T represents the total electric charge (C), i SC is the output supercapacitor current (A), and R SC (Ω) is the module internal resistance. Figure 8 shows the supercapacitor implemented model.

DC/DC converters models

Different DC/DC converters topologies are proposed in the literature, they can be either isolated or non-isolated converters. The former are mainly used in medium and low power applications and they are characterized by high voltage ratio and galvanic isolation [START_REF] Wang | A review of dc/dc converterbased electrochemical impedance spectroscopy for fuel cell electric vehicles[END_REF]. The latter have a simple architecture and they are compact, but they provide low voltage ratio. In [START_REF] Chakraborty | Dc-dc converter topologies for electric vehicles, plugin hybrid electric vehicles and fast charging stations: State of the art and future trends[END_REF], several DC/DC converters architectures are compared and evaluated considering different criteria such as reliability, effectiveness, losses, and cost.

DC/DC converters can be implemented using either average value models or switching models. The second type enables monitoring all the switching phenomena. However, it requires high sampling frequency and consequently a large simulation time, and they are dedicated to design studies. Then, average value models are adopted and implemented using controlled current/voltage sources to replace switches. The conversion dynamics are maintained and the simulation time is highly decreased. The implemented average value models and their standard switching models are depicted in Figs. 9 and 10 where V H and V 2 come from DC bus side, η is the converter efficiency, and D is the duty cycle.

EMS design

The strategy used in this study is an improved frequency separation strategy. This latter is based on the frequency decomposition of the load power profile enabling the fuel cell to supply only the low frequency components. Thus, the fuel cell lifetime will be saved by avoiding power fluctuations [START_REF] Boukoberine | Frequency separation-based power management strategy for a fuel cellpowered drone[END_REF]. However, the energy storage system lifetime is not considered. The battery can undergo high stress, its state of charge (SOC) is not supervised, and the charge/discharge current is not controlled. It is necessary to keep a minimum of battery energy during the flight for security reasons, because it becomes systematically the primary source in case of any eventual fuel cell system or DC/DC boost converter failure. The flight mission can then be terminated safely without any crash.

In regards to fuel cell, even its output power is smoothed, the strategy can not prevent the fuel cell to operate in inefficient operating points. The manufacturer indicates the minimum sustainable output power. Operating below this limit for a long period causes some problems such as cooling and clogging [START_REF] Lee | Active power management system for an unmanned aerial vehicle powered by solar cells, a fuel cell, and batteries[END_REF]. In addition, allowing the fuel cell to operate beyond its maximal power results in reducing its lifetime and then an eventual self-shutdown might take place. Therefore, combining rule-based strategy with frequency separation can be proposed as a solution to these limitations. This strategy can maintain the battery SOC in the acceptable range extending its lifetime, and also keep the fuel cell operating in nominal condition with high efficiency. It is worth noting that rule-based control is widely used for energy management due to its simplicity, ease of real-time implementation, and low computational cost.

To define the rules, we have to set some thresholds such as: minimal battery SOC (SOC min ), maximal battery SOC (SOC max ), the charging power to deliver by the fuel cell to charge the battery when needed (P char ), minimal fuel cell output power (P F C-min ), optimal fuel cell output power (P F C-opt ), and maximal fuel cell output power (P F C-max ). The variables of decision are the battery SOC and the load power (P Load ). According to these two variables, the energy management system will calculate the output power reference (P * F C ) to be supplied by the fuel cell through the DC/DC boost converter (Fig. 11a). The remaining load power (P Load -P * F C ) will be shared by the battery and the supercapacitor according to their dynamics. Figure 11b shows the DC bus voltage V Bus regulation scheme, where V DC-ref is the reference, and D is the duty cycle to control the bi-directional DC/DC converter.

Figure 12 illustrates the six control sectors defined by the EMS based on the two decision variables P Load and SOC. If the demand power P Load exceeds the fuel cell maximal power point or fall below the minimal power limit, the fuel cell output power will be bounded by P F C-max or P F C-min respectively whatever is the SOC. If the SOC is below the lower limit, the fuel cell is supposed to charge the battery besides delivering the demand power. When P Load is between P F C-opt and P F C-min , it is preferable to assign P * F C equal to P F C-opt in order to increase the operation efficiency and the SOC. However, if the battery is fully charged, the fuel cell will just follow the load.

Simulation results

Three scenarios are considered in this study depending on the initial battery state of charge SOC 0 . The simulation results are shown in Fig. 13 It is observed in the power response that the fuel cell power follows well the energy management strategy rules. It supplies according to the pre-defined operating points (P F C-min and P opt ). Outside these points, the fuel cell can either supply the needed extra power to charge the battery when its SOC is low or follow the load power when the SOC is high. the difference between load power and fuel cell power is supplied/absorbed by the energy storage system. The supercapacitor operates as a complementary power source delivering high frequency components with rapid response. The low frequency components are handled by the battery with a relatively slow response. Because of that we can observe that the battery power response curve is less harmonious than the supercapacitor's one. It can be noticed that the fuel cell operates well as the primary energy source, when the supercapacitor operates as a secondary high-frequency power source.

The DC bus voltage, which is the same as the supercapaciotr voltage, is regulated around the reference value V DC-ref = 55 V. The DC bus voltage regulation could be more strict, but it is preferred to allow to the supercapacitor a small interval of voltage variation to enable it contributing in power supplying especially delivering or absorbing power peaks.

The hydrogen consumption (Cons H 2 ) during the flight mission can be calculates as follows

Cons H 2 = N F T f 0 i f c .dt [g] (III.7)
where F is Faraday constant (A s mol -1 ), and T f is the flight duration (s). Table III.3 shows the overall hydrogen consumption (in grams) for the three studied scenarios. Indeed, starting the flight mission with high battery SOC enables to save hydrogen during the flight extending the drone endurance. (A) Sources power responses.

capacitor Power system

In this study we consider another lightweight topology containing only one DC/DC converter and involving fuel cell and supercapacitor as energy/power sources.

Topology and EMS

In this study, we have selected a topology with one DC/DC converter connected to the fuel cell, where the supercapacitor will be directly integrated to the DC bus. This topology is depicted in Fig. 16. The boost converter is used to control the fuel cell current flow according to the selected power management method. The supercapacitor will impose the DC bus voltage and supply the power needed to reach the load power demand. Using the frequency power separation method, the power demand can be separated into high and low frequency components as depicted in Fig. 17.

The fuel cell (energy source) will supply the low frequency power demand. The supercapacitor (power source) will supply the high frequency power demand (peaks). If the load power is less than the fuel cell power, the supercapacitor will be charged by the extra power.

To separate the power frequency components, a corner frequency F c has to be chosen (Fig. 18a). We can adjust it to have a smooth fuel cell power profile and to extract the high dynamic curve to be delivered by the supercapacitor. Thus, a low pass filter has to be implemented as shown in Fig. 18b (F c = 50 mHz). One of the main advantages of this method is that the supercapacitor mean supplied power is close to zero (2.7 W). Indeed, the received power will be almost all redelivered during the flight mission.

Simulation results

Model input parameters

The hybrid power system model is built in Matlab/Simulik using the Sim-PowerSystems (SPS) toolbox which contains generic models for our system components. Table III. [START_REF]Nokia's UAV Traffic Management Concept Implements LTE Network -Unmanned Aerial[END_REF] gives the fuel cell model parameters, when Fig. 19 shows the corresponding polarization curves.

The model input parameters are given in Table III

Rule-based Fuzzy Logic Strategy (RBFL-EMS) for Fuel cell/ Battery Power System

Fuzzy logic has been widely applied for hybrid sources power management, especially for electric vehicles [START_REF] Marzougui | Implementation of energy management strategy of hybrid power source for electrical vehicle[END_REF]. This strategy is known for its robustness to measurement inaccuracies and fast response to load variations. It is also easy to implement with a low computational cost. All these features make fuzzy logic-based EMS fitting to drone applications since the in-flight computational resources are limited, contrary to electric vehicles or microgrids applications. Implementing complex algorithms requires a complex hardware platform increasing energy consumption and weight, and decreasing drone autonomy.

EMS design

For this RBFL-EMS, a fuel cell/battery topology is adopted (Fig. 26). The fuel cell power reference is calculated based on SOC and P Load trapezoidal memberships functions (Fig. 27) and the control decision is extracted from state machine strategy rules given in Table III.6. Input/output variables are categorized into four states: high (H), medium (M), low (L), and very low (VL). Mamdani's fuzzy inference technique is employed with the centroid approach for defuzzification process. The corresponding fuzzy logic control surface is depicted in Fig. 28.

Simulation results

In this study, a real case study is considered as it will be described in the next chapter where RBFL EMS is used as benchmark EMS.

Conclusions

This chapter dealt with power management for a hybrid power supply system of a fuel cell-powered hexacopter. Frequency separation and rule-based strategies have been adopted to control the power flow for each power/energy source. The proposed energy management approaches have been evaluated considering a real power requirement got from a real experimental flight test conducted using a small electric hexacopter. The simulation results of this study have shown that the fuel cell behaved as the main energy source delivering almost all the needed energy during the mission. It supplied a smooth power profile containing low frequencies with high efficiency, thus extending its lifetime by reducing power fluctuations. The load power peaks were supplied/absorbed by the battery and supercapacitor due to their fast dynamics reducing the battery charge/discharge cycles number. It has been also concluded that the proposed approaches increase the drone maneuverability by enabling fast power response and improving the hybrid power system performances.

Introduction

In the previous chapter, improved rule-based EMSs have been implemented targeting source service lives preservation. However, one of the biggest challenges for small electric drones is their limited endurance which is their main performance parameter. Thus, many researchers and drone companies are focusing on this issue trying to increase their flight time. Several studies were carried addressing different aspects of long endurance fuel cell-powered drones, such as energy-based conceptual design [START_REF] Oh | Conceptual design of small unmanned aerial vehicle with proton exchange membrane fuel cell system for long endurance mission[END_REF], thermal efficiency analysis [START_REF] Ji | Thermodynamic analysis of a solid oxide fuel cell jet hybrid engine for long-endurance unmanned air vehicles[END_REF], and hydrogen generation improvement [START_REF] Okumus | Development of boron-based hydrogen and fuel cell system for small unmanned aerial vehicle[END_REF]. Fuel saving is one of the solutions to extend the autonomy in all fuel cell-based mobile applications such as hybrid electric vehicles [START_REF] Zhang | Develop of a fuel consumption model for hybrid vehicles[END_REF]. Some researchers have investigated fuel economy to optimize fuel cell hybrid systems using real-time and optimized control for the fueling regulators [START_REF] Bizon | Fuel saving strategy using real-time switching of the fueling regulators in the proton exchange membrane fuel cell system[END_REF][START_REF] Bizon | Hydrogen saving through optimized control of both fueling flows of the fuel cell hybrid power system under a variable load demand and an unknown renewable power profile[END_REF][START_REF] Bizon | Sensitivity analysis of the fuel economy strategy based on load-following control of the fuel cell hybrid power system[END_REF]. However, there is still a lack of studies targeting drone applications.

In this chapter, a fuel cell-based drone energy management strategy is investigated targeting hydrogen consumption optimization. The commercialized DJI M100 quadcopter powered with the Intelligent Energy 650W fuel cell power module is considered as the case study [START_REF]Intelligent Energy, 650W Fuel Cell Power Module Specifications[END_REF]. This power module is controlled through a simple rule-based EMS (RB-EMS). In this context, an equivalent consumption minimization strategy (ECMS) is implemented intending hydrogen economy. Then, a genetic algorithm-based EMS optimization approach is adopted. Real data are implemented for the load power demand using the carried out flight test.

Case study

Several international companies are investing on fuel cell powered drones and among them Ballard, Intelligent Energy, HES Energy Systems, H3 Dynamics, MicroMultiCopter, etc. Intelligent Energy is a fuel cell engineering company working on the development and commercialization of its lightweight Proton Exchange Membrane (PEM) fuel cell for drone applications extending the flight time and bypassing the limitations of batteries. In this study, we have chosen the DJI M100 quadcopter powered by their 650W fuel cell as depicted in Fig. 1. The fuel cell power module specifications exploited in this study are listed in Table IV.1. In this study, three energy management strategies are proposed and compared, namely, basic rule-based strategy (RB-EMS), frequency separation rulebased strategy (FSRB-EMS), and equivalent consumption minimization strategy (ECMS).

Case study

Rule-based strategy (RB-EMS)

The energy management strategy used for the Intelligent Energy fuel cell power module is based on if-then rules. As shown in Fig. 2, the fuel cell continuously supplies a rated power, which corresponds to the hovering necessary power (800W) regardless of the load power variations. The battery will either supply the extra needed power (especially peaks) or be charged when the fuel cell rated power exceeds the load power. Indeed, this simple method

Emergency flight time 2min

Features Automatic failure detection and backup battery Data storage for firmware update, performance, and diagnostics has many disadvantages. The hovering power is not always stationary because it depends on many factors such as the payload, the type of mission, and even weather conditions. The fuel cell rated power does not necessarily match the hovering power, consequently it will operate outside of its highperforming operational zone, and even the battery can be solicited to deliver high current to supply the extra needed power. Thus, both sources will undergo rapid degradation and decrease of their lifetimes. In addition, the drone is supposed to conduct different type of missions, not only hoveringbased missions, the EMS has then to be adapted. It is worth noting that a best situation has been considered for the implementation of RB-EMS, when the nominal power matches the hovering power, which is not always the case.

ECMS

EMS design

The focus of this work is to extend the drone autonomy, for which an optimal EMS has to be implemented as FSRB-EMS is unable to achieve this goal. The strategy aims to reduce the hydrogen consumption, thereby increasing the drone operating time. Global optimization algorithms are well developed in the electric vehicle field such as dynamic programming [START_REF] Wu | Demand side energy management of ev charging stations by approximate dynamic programming[END_REF], genetic algorithm [START_REF] Lü | Energy management of hybrid electric vehicles: A review of energy optimization of fuel cell hybrid power system based on genetic algorithm[END_REF], particle swarm optimization [START_REF] Wang | Demand side management of plug-in electric vehicles and coordinated unit commitment: A novel parallel competitive swarm optimization method[END_REF]. However, a global optimization usually requires a priori driving cycle to find global optimization results. In addition, it is time-consuming and complex, and it can not be implemented in real time. Unlike electric vehicles, weight and computational effort are very important constraints in electric drones. Thus, local optimization algorithms are the appropriate alternatives for our application. The optimization period is reduced from the complete driving cycle to an instantaneous sample time to calculate the optimal control variables.

In this context, ECMS is proposed. Indeed, it is investigated in the literature for mobile applications such as electric/hybrid vehicles [START_REF] Li | Online adaptive equivalent consumption minimization strategy for fuel cell hybrid electric vehicle considering power sources degradation[END_REF][START_REF] Dong | Experiment and simulation investigation on energy management of a gasoline vehicle and hybrid turbocharger optimization based on equivalent consumption minimization strategy[END_REF] and tramway [START_REF] García | Viability study of a fc-battery-sc tramway controlled by equivalent consumption minimization strategy[END_REF][START_REF] Zhang | Optimization for a fuel cell/battery/capacity tram with equivalent consumption minimization strategy[END_REF]. The concept of ECMS is to minimize the sum of hydrogen directly consumed by the fuel cell, and the amount used indirectly by the energy storage sources (equivalent consumption) at each sampling time while insuring effective operating for each source. In this way, it does not need future prediction or a priori data about the scheduled mission. When the battery provides energy, its SOC will decrease, so the fuel cell has to supply the needed energy to recharge it by consuming hydrogen. When the battery SOC is high, its energy will be used for future maneuvers such as climbing or accelerations leading to a reduction in hydrogen consumption [START_REF] Torreglosa | Hybrid fuel cell and battery tramway control based on an equivalent consumption minimization strategy[END_REF]. Thus, the battery delivered energy is converted to an equivalent hydrogen consumption to have a single-objective minimization problem.

The battery virtual hydrogen consumption is proportional to its energy multiplied by an equivalence factor λ. It depends on the battery SOC and can be expressed empirically as [START_REF] García | Viability study of a fc-battery-sc tramway controlled by equivalent consumption minimization strategy[END_REF] λ = 1 -2β SOC -0.5 (SOC max + SOC min )

SOC max + SOC min (IV.1)
where β is the SOC balance coefficient. It is used for accurately representing the charge/discharge processes (chosen equal to 0.65 in [START_REF] García | Viability study of a fc-battery-sc tramway controlled by equivalent consumption minimization strategy[END_REF] and 0.6 in [START_REF] Motapon | A comparative study of energy management schemes for a fuel-cell hybrid emergency REFERENCES power system of more-electric aircraft[END_REF]).

The equivalence factor is the most important parameter in ECMS. The larger it is, the more the battery energy is penalized and consequently the hydrogen consumption increases to recharge the battery. On the contrary, if it is too small, the fuel cell energy is penalized and the algorithm will calculate the power references to use more energy from the battery for adjusting its SOC back to its permitted range. The equivalence factor λ is considered in the cost-function in order to make the ECMS algorithm less dependent to the balance coefficient β. The problem formulation can be as follows. Calculate the optimal solution y = [P F C , λ, P Bat ], which minimizes

F = [P F C + λ.P Bat ] .∆T (IV.2) subject to            P Load = P F C + P Bat P F C-min P F C P F C-max P Bat-min P Bat P F Bat-max 0 λ 2 (IV.3)
where P Bat-max and P Bat-min are the maximum and minimum battery powers, respectively. To solve the optimization problem under Matlab/Simulink, "fmincon" function is used. This function uses a sequential quadratic programming (SQP) method to solve a subproblem at each iteration.

ECMS (empty matrix).

A eq and B eq define the linear equalities (A eq .y = B eq ). U B and LB define the upper and lower bounds of the solution (LB y U B). OP T ION S is an argument created with the OP T IM OP T ION S function, which allows to set the optimization parameters such as maximum iterations (set to 100) and maximum function evaluations (set to 1000). For the case study, we have

y 0 = [P F C-min , 0, 0] ; A eq = 0 1 0 1 0 1 B eq =     1 -2β SOC-0.5(SOCmax+SOC min ) SOCmax+SOC min P Load     U B =    P F C-max 2 P Bat-max    ; LB =    P F C-min 0 P Bat-min   

Simulation Results and Discussion

In this study, three energy management strategies are proposed and compared, namely, basic rule-based strategy (RB-EMS), which represents the case study EMS, frequency separation rule-based strategy (FSRB-EMS), and equivalent consumption minimization strategy (ECMS). The three energy management strategies are implemented in Simulink using SimPowerSystems toolbox. To achieve a feasible comparison, same conditions are set for all strategies (load power requirement, sources characteristics, initial SOC (70%), DC bus, and DC/DC converters control parameters).

Power distribution

Figure 3 shows the fuel cell and battery delivered powers for each EMS considering converter losses. In the RB-EMS (Fig. 3a) the fuel cell is delivering constant power, the battery provides the extra needed power, especially the peaks, to match the desired load power. When the load power is less than the fuel cell one, the battery starts charging for increasing its SOC.

The fuel cell and battery act then as energy and power sources, respectively, with a relatively short response time for the battery. For FSRB-EMS in Fig.

ECMS

SOC and DC bus voltage

The SOC and DC Bus voltage results are depicted in Fig. 4. Figure 4a highlights one of the biggest drawback of RB-EMS. The SOC average is continuously increasing because the battery is receiving power more than delivering. In this context, there is high possibility that the battery will be overcharged during the mission. This issue is due to the fact that fuel cell reference power is not adapted to SOC variations. On contrary, for the RBFS-EMS (Fig. 4b), the calculated fuel cell reference is considering SOC and requested power regarding the fuel cell preferred operating points. When the SOC reach its maximum (SOC max = 90%), battery will be forced to discharge and less power will be demanded from the fuel cell. For the ECMS (Fig. 4c), the SOC is continuously decreasing in average, and stabilizes after the 7th cycle respecting the condition SOC min = 60%. The method is using battery energy as much as possible.

The DC bus voltage is well regulated around the reference of 55V similarly for the three strategies with relatively smaller peaks for ECMS.

Current and voltage responses

Figure 5 shows the power sources current and voltage responses during the mission. For the strategies, battery and fuel cell voltages are fluctuating around their nominal values (24V and 21.5V respectively), and the delivered current does not exceed the permitted range. Thus, the power sources are operating within their nominal areas.

Efficiency

The overall efficiency η is given by

η = P load P F C + P Bat (IV.4)
where P F C and P Bat are the fuel cell and battery delivered powers, respectively (input to DC/DC converters).

The efficiency average values for the three strategies are close (around 85%) with a slight advantage for ECMS. The results are shown in Fig. 6.

Hydrogen consumption

The hydrogen consumption (C H 2 ) during the flight mission is calculated as follows where F is Faraday constant (A s mol -1 ) and T f is the flight duration (s). Figure 6 illustrates the hydrogen consumption variation during the mission and Table IV.7 shows the overall hydrogen consumption (in g) for each strategy. C H 2 is similar for both RB-EMS and RBFS-EMS. However, the ECMS enables a 3% improvement.

C H 2 = N F T f 0 i f c .dt [g] (IV.5) 100

ECMS

Hydrogen saving benefits

To study fuel saving and its effect on the operating cost, a real scenario has been analyzed. In fact, there is a wide range of flight missions, and it is difficult to estimate hydrogen consumption. In this context, a study is carried out in [START_REF] Sisco | Drone Refueling[END_REF] considering two types of drones: multicopter drone (10-15kg) and fixed wing drone (20-25kg) based on Aerostack and FCAir fuel cell systems (from HES and Ballard, respectively). The hydrogen consumption rate 4. Genetic Algorithm-based Optimization cell module lifetime. In a context of drone fleets, which is the general case for many companies for specific missions such as delivery or inspection, the benefit will be multiplied by the number of deployed drones.

Hydrogen saving benefits are not just limited to the economical aspect. They have also a direct impact on the drone autonomy. Indeed, the number of cycle rounds to the ground charging station can be reduced and consequently more hydrogen will be saved while improving the drone operating efficiency. The endurance increase depends on the hydrogen tank. The greater energy density is, the greater autonomy increases. While MetaVista has achieved its world record of multirotor drone flight time (12h and 7min) in 2019 using Intelligent Energy team 800W fuel cell power module [START_REF]Intelligent Energy showcases drone fuel cell modules in japan, fuel cells bulletin[END_REF], the proposed improved EMS will lead to an endurance increase of 21.81min. Table IV.3 shows the expected increase in autonomy using the ECMS for some commercialized drones powered by Intelligent Energy fuel cell power modules (FCPM).

Genetic Algorithm-based Optimization

In this section, a fuel cell-powered drone power management approach is proposed to optimize hydrogen consumption and battery/fuel cell usage in persistent missions, which are conducted continuously taking long time. Advantages of rule-based strategy and genetic algorithm are combined in an online EMS to fit on drone applications. The commercialized DJI M100 multirotor supplied by the Intelligent Energy 650W fuel cell pack is adopted as a case study [START_REF]Intelligent Energy, 650W Fuel Cell Power Module Specifications[END_REF]. A basic rule-based EMS is implemented to manage this hybrid power supply system. In this context, an EMS based on the combination of frequency splitting and rule-based strategies is suggested as a

Chapter IV. Optimization-based Energy Management Strategies first enhancement preserving sources lifespans in [START_REF] Boukoberine | Hybrid fuel cell powered drones energy management strategy improvement and hydrogen saving using real flight test data[END_REF]. Next, a fuzzy rulebased EMS is proposed in this study according to its simplicity, robustness and fast response to load variation, which may be suitable for drones operation. Although rule-based strategies have the advantages of simplicity, flexibility, and ease of implementation, they are unable to achieve optimal power allocation [START_REF] Sellali | A novel energy management strategy in electric vehicle based on H∞ self-gain scheduled for linear parameter varying systems[END_REF]. Then, the proposed approach adopts an offline optimization strategy dedicated to a pre-known mission profile. It is worth mentioning that numerous drone missions are repetitive and determined where the drone has to perform a specific task, such as PV or wind turbine farms inspection, and traffic monitoring. In this context, a multi-objective genetic algorithm (MOGA) is used to optimize the proposed EMS parameters. GA is chosen thanks to its good convergence, uniform population movement, robustness, and suitability for multi-objective optimization. The first objective is to minimize the hydrogen consumption. The originality of our solution lies in adopting battery state of charge (SOC) deviation as a second objective function considering three operating modes namely: charge sustaining (CS), charge depleting (CD), and charge increasing (CI). In persistent missions, CS mode is used in loitering phase to preserve battery lifetime by decreasing charge/discharge cycles. It also allows keeping an amount of energy for emergency cases or to return to the fueling station. CD mode is used when returning back to the refueling station or during emergency landing. CI mode is activated when leaving the refueling station to continue mission. Since the battery charging is time consuming, it is preferable to use this mode allowing fuel cell to charge battery when returning back to operation. Simulations have been done using a real power profile for the drone load power obtained after conducting a flight test with an electric multirotor. Results have shown that the proposed approach can offer up to 5% of hydrogen consumption minimization when preserving fuel cell and battery lifetimes.

EMS design

The purpose of this study is to improve the drone endurance implementing an optimized EMS based on FSRB-EMS. The strategy has the objective of minimizing fuel consumption, thus extending the autonomy. Although FSRB-EMS presents some improvements and benefits such as fuel cell efficient operation and service life preservation, it is unable to achieve hydrogen consumption minimization. In this work, it has been opted for an off-line

Genetic Algorithm-based Optimization

multi-objective optimization method based on a priori known power profile to optimize the FSRB-EMS parameters vector [P F C-min , P F C-max , P char , SOC min , SOC max ]. Indeed, excessive SOC min leads to a deep depth of discharge aging the battery, while more fuel cell power will be required to charge the battery increasing the converters losses. Insufficient SOC max induces battery overcharging reducing battery contribution to load supplying. Excessive P F C-min or P F C-max values causes fuel cell to work outside its high efficiency area affecting its lifespan [START_REF] Bankupalli | Operational adaptability of pem fuel cell for optimal voltage regulation with maximum power extraction[END_REF]. High P char value may affect battery health, when a low P char value leads to a slow charging process affecting the EMS performances.

Global optimization strategies are widely used in the electric vehicle area, for instance particle swarm optimization [START_REF] Wang | Demand side management of plug-in electric vehicles and coordinated unit commitment: A novel parallel competitive swarm optimization method[END_REF], dynamic programming [START_REF] Wu | Demand side energy management of ev charging stations by approximate dynamic programming[END_REF], and genetic algorithms [START_REF] Lü | Energy management of hybrid electric vehicles: A review of energy optimization of fuel cell hybrid power system based on genetic algorithm[END_REF]. Genetic algorithms were chosen in this study due to their advantages relatively to other population algorithms, such as good convergence, uniform population movement, robustness, and suitability for multi-objective optimization. Since parameters vector is subject to GA-based multi-objective optimization, bounds have to be set for each parameter to keep the EMS best performances ( Table . IV.4). Schematic diagram of the GA-based optimization process is depicted in Fig. 7. The drone model is implemented under Simulink, while the GA algorithm is performed under Matlab. For each individual from the population, Simulink model should be run to evaluate the total hydrogen consumption and the final SOC.

The optimization approach has two objectives: reducing the hydrogen consumption Cons H 2 and the second one is related to the battery SOC behavior considering three scenarios: charge sustaining, charge increasing, and charge depleting. • Charge sustaining: This scenario enables to keep battery energy for emergency cases while contributing in the power supply during the mission. It can especially fit to persistent missions allowing the drone to return back to the fueling station (Fig. 8). Minimized cost-function is represented by SOC deviation, which is the difference between the final SOC (SOC f in ) and the initial one (SOC 0 ). Thus, the battery service life can be saved reducing the number of charge-discharge cycles.

• Charge depleting: This mode can be used when the hydrogen reserve reaches its lower limit, then the drone will have to return to the refueling station (Fig. 8), or in emergency cases to ensure safe landing. Minimized cost-function can be formulated as the difference between SOC 0 and the acceptable minimum SOC (chosen as 40%). It is worth mentioning that the rule-based algorithm is adapted to this scenario prioritizing the battery energy than fuel cell's for the load power supply. There is another possibility to deplete the battery as much as possible (down to 20 %), and then exploit the refueling time in the ground station to charge it up to 40% to match the SOC 0 of the charge increasing phase. This amount of energy will come from cheaper electricity avoiding hydrogen-based battery charging, and then decreasing the hydrogen consumption. The problem in this case is transformed to a single objective optimization considering only C H2 as cost-function because minimizing the hydrogen consumption leads consequently to discharge more the battery decreasing its SOC. There are no other competitive goals.

• Charge increasing: The drone can switch to this mode when leaving the refueling station to continue the mission. Since the battery charging is time-consuming, it is preferable to use this mode enabling fuel cell to charge battery while returning back to operation (Fig. 8). In this case, the minimized cost-function is formulated as the difference between SOC f in and the maximum acceptable SOC (set to 70%), where SOC 0 is set to 40%. Indeed, using hydrogen to charge the battery in persistent missions is mandatory and not an option. Swapping technique can be an option to overcome this problem [START_REF] Boukoberine | Power supply architectures for drones-a review[END_REF], however, extra logistical equipment and management will be required. In swapping case, which is not the concern of this study, CI mode will be just replaced by a second CD mode starting from the recharge station, depleting a full battery to 70% to reach the hovering area, and then CS mode will be used. The EMS will then switch to another CD mode decreasing the SOC to 40% when returning back to the recharge station.

Cost functions are formulated in (IV.6) and GA parameters chosen for this study are summarized in Table IV.5. After carrying out a pre-evaluation study, it has been concluded that 50 generations is enough to reach the algorithm convergence.

min

           F 1 = C H2 F 2 =      |SOC f in -SOC 0 | CS |SOC f in -40| CD |SOC f in -70| CI (IV.6)

Simulation Results and Discussion

In this section, the achieved simulation results of the GA-based multi-objective optimization algorithm (MOGA) are first discussed. Then, best solutions are implemented in the drone model to simulate and evaluate the proposed optimized EMS. As explained in Section 2, three scenarios are studied, namely charge sustaining, charge decreasing, and charge increasing, where the Intelligent Energy RB-EMS and RBFL-EMS are considered as benchmark strategies. Battery capacity is increased in the drone model by adding two new batteries to enable this battery pack providing the needed power especially in charge depleting mode. The resultant extra weight is considered in the drone model by updating the power profile respecting proportionality between propulsion power P prop and take-off mass m as given in (IV.7), where K is a factor representing the drone aerodynamic characteristics [START_REF] Apeland | Suitability analysis of implementing a fuel cell on a multirotor drone[END_REF].

P prop = K.m 3/2 (IV.7)

GA-based Optimization Results

Figure 9 shows Pareto front for charge sustaining case. It is worth noting that Pareto front provides more degrees of freedom when selecting the interesting variable. It presents the non dominated solutions in terms of minimal SOC deviation and hydrogen consumption. The curve is almost linear due to the coherence between the two objective functions. Minimization of hydrogen consumption induces more battery energy usage, therefore decreasing its SOC and vice-versa. SOC deviation varies from 0 to 5%, which allows a certain range of choice. For the simulation, we can opt for the best solution in terms of hydrogen consumption with almost 4.8% SOC decrease. However, in case of long and persistent missions, the drone will perform many rounds to the recharge station, consequently leading to SOC decrease over time (-4.8% in each round). In this situation, we have to be strict regarding SOC deviation choosing the 0.007% corresponding individual with 10.76g of hydrogen consumption. Pareto front shows another solution with 0.0004% SOC deviation but inducing more hydrogen consumption (11.12g). This case illustrates the interest of multi-objective optimization in this study. Indeed, if only SOC deviation has been as single objective function, we would have definitely fallen on the incorrect hydrogen consumption solution.

For charge depleting mode, Pareto front is depicted in Fig. 10. In this case we don't have many choices. Indeed, four solutions give SOC deviation less than 2%. The final SOC (SOC f in ) has to be close to 40% to avoid cumulative SOC decrease. Thus, we have to choose the individual, which match, F 2 = |SOC f in -40| = 0 with 8.82g of hydrogen consumption.

The Pareto front corresponding to charge increasing (CI) mode is depicted in Fig. 11. Since SOC f in has to reach 70%, the adopted solution match the criteria F 2 = |SOC f in -70| = 0 with 13.4g of hydrogen consumption. The selected parameters are listed in Table IV.6.

Power Distribution

Figures 12a and 12b represent power supplied by both fuel cell and battery for each EMS. For the RBFL-EMS (Fig. 12a), the fuel cell shows a rapid power response tracking load variations, resulting in a relatively fluctuating power curve. Since these fluctuations affect the fuel cell long-term durability, it 4. Genetic Algorithm-based Optimization is preferable to let the battery supply them by limiting the fuel cell current change rate. In RB-EMS (Fig. 12b), fuel cell is providing a fixed power, and battery is completing the remainder quantity to match the demanded power. When the load power has a low value, fuel cell will charge the battery raising its SOC. The battery and fuel cell operate then as power and energy sources, respectively. It is noticed that fuel cell power behavior is compatible with the energy management strategy rules. For the other scenarios, fuel cell operates in respect of the pre-set operating rules with smooth power profiles while respecting the two limits (P F C-max , P F C-min ) trying to supply as much as possible near to the rated power P opt .

SOC Evolution

SOC variations during the flight mission for all scenarios are depicted in Fig. 13. As can be seen, the final SOC reaches the targeted value in each mode. RBFL-EMS induces a SOC decrease of 11% during the 1000s cycle. Then, it cannot be adopted for persistent missions because SOC will keep depleting over time, unlike the charge sustaining mode. In addition, these results, which are compatible with GA solutions, outline a drawback of RB-EMS. SOC is continuously raising because charging energy is exceeding the supplied one. In this context, possible overcharging can therefore occur during the mission. Thus, fuel cell output power should be adapted to SOC variations. For other strategies, SOC variation is considered for fuel cell power reference calculation respecting its favorite operating areas. When the SOC reaches its maximum limit (SOC max ), battery is forced to discharge decreasing stress on the fuel cell.

Hydrogen Consumption

Hydrogen consumption (Cons H 2 ) during the flight duration T f (s) is calculated as follows

Cons H 2 = N F T f 0 i f c .dt [g] (IV.8)
where F is Faraday constant (A s mol -1 ). The hydrogen consumption behaviour for each scenario is illustrated in Fig. 14 and Table IV.7 summarizes the overall hydrogen consumption (in g). Comparison of Cons H 2 for three scenarios does not make sens because they have different goals. However, CS mode can be compared with RBFL-EMS and RB-EMS. The latter presented a consumption of 11.35g when the CS mode induced 10.76g of fuel consumption resulting on a gain of 5.1%. Indeed, the extra-consumed hydrogen in RB-EMS is due to SOC increasing (from 70% to 79%). However, this SOC increase is not beneficial and increases operating cost because charging battery using hydrogen is not cost-effective. Indeed, RBFL-EMS showed less hydrogen consumption (10.31g) than CS mode, however, this is only due to extra battery usage in supplying load depleting its SOC to 59%. To investigate hydrogen economy effect on reducing operating cost, actual case study is proposed. Indeed, it exists various types of drones and flight missions, and it is not so easy to estimate fuel usage. In this context, authors in [START_REF] Sisco | Drone Refueling[END_REF] have examined two kinds of drones: multicopter type (10-15kg) and fixed wing type (20-25kg). According to the achieved results, the fuel utilization rate for the multirotor was estimated to be about 79g h -1 . We adopt a fuel cell lifespan of 3000h ( the Ballard's case). Thereby, the saved fuel quantity by one multirotor over one fuel cell lifecycle is equal to 0.051 × 79 × 3000 = 12087g. Hydrogen price when supplied by an industrial refueling station in Norway is 0.12 eg -1 [START_REF] Apeland | Suitability analysis of implementing a fuel cell on a multirotor drone[END_REF]. Therefore, the gain can go up to 1450e per multirotor. In case of drone swarm, the profit will be increased according to the employed multirotors number.

Chapter IV. Optimization-based Energy Management Strategies

Hydrogen reduction incentives are not restricted to only economic aspect. It directly affects drone endurance. Round trips to the ground refueling site can be reduced and therefore more fuel will be saved while enhancing operating effectiveness. Indeed, extended autonomy will depend on the fuel tank size.

The larger energy density is, the more endurance is extended. The world record of fuel cell-based multirotor flight record was achieved by MetaVista (12h and 7min) in 2019 by means of Intelligent Energy's 800W fuel cell power module [START_REF]Intelligent Energy showcases drone fuel cell modules in japan, fuel cells bulletin[END_REF]. Thus, the estimated autonomy improvement is 37min (+5.1%). Table IV.8 shows the expected additional flight time for some commercially available drones when considering 5.1% hydrogen economy.

Conclusions

This chapter dealt with energy management for a hybrid power supply system of a fuel cell-powered quadcopter targeting endurance improvement. A commercially available Intelligent Energy drone was considered for the case study and real power consumption data were obtained by performing an experimental flight test of an electric hexacopter. Frequency separation rulebased and equivalent consumption minimization strategies were proposed to improve the system performance and hydrogen saving.

The equivalent consumption minimization strategy was proved capable of optimally distributing load power between the two sources leading to a 3% improvement in hydrogen use. This fuel economy has a direct impact on the drone autonomy reaching up to 21.81min for the world flight test record performed in 2019 by MetaVista and intelligent Energy. Indeed, drone endurance improvement depends on the amount of carried hydrogen: the greater it is, the greater autonomy increases. The number of cycle rounds to the

Conclusions

ground charging station during a given mission can also be minimized, leading to more hydrogen savings. The proposed hydrogen saving strategy will consequently lead to a decrease in operating costs. Indeed, it can save up to 853.2e per drone during one fuel cell module lifecycle considering the Intelligent Energy 650W fuel cell power module as a case study.

In the second study, multi-objective genetic algorithm optimization was able to induce 5% of hydrogen consumption minimization in charge sustaining mode. This hydrogen saving has a direct effect on the drone's endurance, expecting to achieve an increase of about 37min of the world record (12h 7min) realized in 2019 by Intelligent Energy and MetaVista. The frequency of return trips to the ground stations during the mission can also be reduced, inducing more hydrogen economy. Therefore, fuel saving will lead also to a reduction of the operating cost achieving up to 1450e per drone over the fuel cell stack service life, assuming the case study power system. In the case of drone swarm applications, the number of drones that are being deployed will magnify the benefit.

Introduction

In the previous chapter IV, we proposed two optimization-based approaches for EMS improvement: One is online and the other is offline. However, leveraging the advances in artificial intelligence for hybrid electric vehicle (HEV) energy management seems to be a key option to answer our research questions.

Background

Drones can be considered as electric vehicles (EVs) with some specificities such as high sensitivity to the overall system weight inducing a limited capability of integrating high energy density sources and powerful calculators. However, it is still regarded as an EV especially when studying its power propulsion system. HEVs typically combine an electric motor (EM) and an internal combustion engine (ICE) and they are usually fed by two or more energy sources such as battery, fuel cell, supercapacitor, and gasoline depending on the electrification degree. With the hybridization of different energy sources, HEVs have acquired more flexibility in supplying energy demands as compared to conventional vehicles. In addition, since each source has its own unique generation/storage characteristics, source hybridization enables combining their advantages. This hybridization also balances source limitations, thus improving the overall hybrid power supply system (HPSS) performance. The scheduling of these hybrid sources required effective implementation of an energy management strategy (EMS) to control the power sharing between sources while satisfying their constraints, conserving their lifetime, and increasing the overall system efficiency.

Energy Management for HEVs

The main challenge in an HEV is the limited availability of onboard energy. Therefore, EMS goal is to optimally supply energy to HEVs, while reducing fuel consumption and emissions. The difficulty lies in satisfying constraints and power demand concurrently, and compromises must always be made to obtain an optimal solution [START_REF] Zhang | Energy management strategies of connected hevs and phevs: Recent progress and outlook[END_REF]. The challenge is even further complicated by the need to keep vehicle performance at an optimal level [START_REF] Sabri | A review on hybrid electric vehicles architecture and energy management strategies[END_REF]. Irrespective of HEV type, its fuel economy and exhaust emissions performance are heavily affected by the selected EMS, which has been of interest for extensive studies in recent years [START_REF] Boukoberine | A critical review on unmanned aerial vehicles power supply and energy management: Solutions, strategies, and prospects[END_REF][START_REF] Saiteja | Critical review on structural architecture, energy control strategies and development process towards optimal energy management in hybrid vehicles[END_REF].

Introduction

Figure 1 illustrates a general representation of EMS for HEV applications. Energy management strategies for HEVS can be generally classified into two categories [START_REF] Saiteja | Critical review on structural architecture, energy control strategies and development process towards optimal energy management in hybrid vehicles[END_REF]: rule-based and optimization-based strategies. Rule-based EMS is a simple and commonly known strategy, which involves a set of predefined threshold-based rules on the control variables. The power system will supply in different operating modes by mapping control variables to the related thresholds, like frequency separation rule-based strategy [START_REF] Boukoberine | Frequency separation-based power management strategy for a fuel cellpowered drone[END_REF] and rule-based fuzzy logic strategy [START_REF] Sabri | Improved fuel economy of through-the-road hybrid electric vehicle with fuzzy logic-based energy management strategy[END_REF]. Optimization-based approaches are usually realized by designing an objective function with a dynamic statespace model and specified constraints. These optimization-based EMSs can be solved by linear programming (LP), mixed number linear programming (MILP), quadratic programming (QP), or nonlinear programming (NLP) solvers. Various optimization-based algorithms have been investigated in the literature, including Pontryagin's minimum principle (PMP) [START_REF] Li | Adaptive energy management strategy for fuel cell/battery hybrid vehicles using pontryagin's minimal principle[END_REF], equivalent consumption minimization strategy (ECMS) [START_REF] Boukoberine | Hybrid fuel cell powered drones energy management strategy improvement and hydrogen saving using real flight test data[END_REF], model prediction control (MPC) [START_REF] Rodriguez | Fuzzy logic-model predictive control energy management strategy for a dual-mode locomotive[END_REF] and meta-heuristic methods [START_REF] Boukoberine | Optimized energy management strategy for hybrid fuel cell powered drones in persistent missions using real flight test data[END_REF][START_REF] Chen | Particle swarm optimization-based optimal power management of plug-in hybrid electric vehicles considering uncertain driving conditions[END_REF]. Authors in [START_REF] Saiteja | Critical review on structural architecture, energy control strategies and development process towards optimal energy management in hybrid vehicles[END_REF] have proposed a critical evaluation for numerous studies that investigated EMSs dedicated to electric vehicles.

Energy management strategies have been widely implemented in practice thanks to their effectiveness in real-time power allocation. Nevertheless, some of them rely mainly on expert knowledge such as rule-based methods. PMP and MPC have a heavy computational burden. They also need accurate models and they are still sensitive to uncertainties in power load profile or the HPSS model [START_REF] Gao | A deep reinforcement learning based energy management strategy for fuel-cell electric uav[END_REF]. These stochastic disruptions would lead to HEVs HPSS performance degradation. Furthermore, the random characteristics of energy usage and the flexible use of active loads lead to more uncertainty and complexity in HPSS power management. Hence, the main problem is the difficulty of developing efficient general mathematical model that can fit in most situations. Meta-heuristic methods such as genetic algorithm (GA) and particle swarm optimization (PSO) and also dynamic programming (DP) can achieve global optimality. However, they cannot be implemented online because they need a priori knowledge on the driving mission.

Motivation and Necessity for an Up-to-Date Review

To overcome the previously mentioned limitations and achieve a high adaptability power control for HEVs HPSS, the scientific research community has shown a strong interest in learning-based approaches exploiting artificial intelligence advances. As a core area of machine learning, reinforcement learning (RL) has the strength of interactive trial and error-based self-learning in a dynamic environment [START_REF] Mason | A review of reinforcement learning for autonomous building energy management[END_REF]. It has been successfully implemented to address various real-world control decision-making tasks under uncertain conditions. RL algorithms learn how to make control decisions by evaluating actions and obtaining rewards as a feedback metric derived from their own experiences. The final goal is to learn the optimal policy respecting the predefined objective function and constraints. The RL problem is usually modeled as a Markov Decision Process (MDP) [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. Moreover, the system dynamics model is not required and the related sensitivity is then omitted. Therefore, RL is considered as a powerful tool to achieve near-optimal control decisions for nonlinear and uncertain systems. It can effectively improve the fuel economy, flexibility, and reliability of HPSS in different vehicle types and for various missions. The integration of neural networks in RL approaches has also considerably boosted their performance introducing a new branch known as deep reinforcement learning (DRL) [START_REF] Wu | Continuous reinforcement learning of energy management with deep q network for a power split hybrid electric bus[END_REF][START_REF] Han | Energy management based on reinforcement learning with double deep q-learning for a hybrid electric tracked vehicle[END_REF].

Indeed, several studies have been carried out in recent years investigating RL-based EMSs for HEVs. However, only a few review studies have been published in this research area. Xiong et al. [START_REF] Xiong | Towards a smarter hybrid energy storage system based on battery and ultracapacitor-a critical review on topology and energy management[END_REF] have reported a comparative 1. Introduction evaluation of conventional EMSs and RL-based ones. However, the study focused only on battery/ supercapacitor energy storage systems. In addition, RL concept were not explained and RL algorithms applied on HEV EMS were not introduced. Authors in [START_REF] Hu | Reinforcement learning for hybrid and plug-in hybrid electric vehicle energy management: Recent advances and prospects[END_REF] documented a comprehensive review of current advances in RL-based EMSs discussing hybrid RL approaches, which include information such as mission information, predicted operating conditions and also algorithms like MPC. In [START_REF] Li | Reinforcement learning energy management for fuel cell hybrid system: A review[END_REF], authors have reviewed RLbased EMSs for fuel cell HEVs focusing on RL environments and training process. The RL concept was well presented, and the RL impact on HEV energy management was demonstrated. However, RL algorithms were not described to enable the reader a deep understanding of the RL techniques application. Table V.1 summarizes the main contributions of the existing review studies related to RL-based EMSs. This study provides an up-to-date extensive overview of how reinforcement learning (RL) has enhanced EMSs for HEV applications while providing an initiation to RL approach. Since the drone is considered as an electric vehicle, the goal of this study is to exploit the achieved machine learning advancement in HEV energy management area to develop smart EMSs for drones. The main contributions of this review study are as follows:

• Providing an overview of RL concept and recent developments in RL algorithms to establish a complete link between RL techniques and HEV energy management field. A comparative evaluation of different RL techniques is also proposed to enable choosing the appropriate algorithm for a given energy management problem.

• Proposing a comprehensive study of the existing literature related RLbased EMSs implemented in HEV field.

• Demonstrating the potential of RL on HEV HPSS energy management in respect of effectiveness and energy economy.

• Identifying potential limitations of RL for HEV energy management and emphasizing potential locks for future exploration.

The remaining of this chapter is organized as follows: Section 2 presents an overview of the reinforcement learning concept, Section 3 and 4 critically explain RL and DRL algorithms that have been implemented for HEV energy management, Section 5 discusses RL-based EMSs achievements in the topic and highlight their limitations and perspectives. Ref.

Main contribution [START_REF] Xiong | Towards a smarter hybrid energy storage system based on battery and ultracapacitor-a critical review on topology and energy management[END_REF] Comparison between conventional and RL-based EMSs for battery/ supercapacitor energy storage systems.

[241] Discussing hybrid RL techniques which involve mission information, predicted operating conditions, and also algorithms like MPC.

[242] A comprehensive review of RL-based EMSs addressing fuel cell HEVs and focusing on RL environment construction and evolution process.

Reinforcement Learning

Reinforcement learning is a research area in machine learning that involves an autonomous agent trained to make decisions in an unknown environment for maximizing a desired cumulative reward. The agent performs a large number of trial and error experiences by being either recompensed or punished to learn the best behavior. The most popular approach for RL problem modeling is the Markov decision process (MDP) [START_REF] Sutton | Reinforcement learning: An introduction[END_REF].

Markov Decision Process

MDP is a discrete-time formulation of a sequential stochastic decision-making process. It is intended to be a direct way of defining the interactive learning problem to reach an objective. The decision-maker or controller is known as the agent. The plant, or controlled system, and everything apart from the agent, is named the environment.

During the learning process, and at each time step t (t = 0, 1, 2, ...n), the agent chooses a possible action A t (control signals) respecting a particular policy π. Consequently, the environment reacts by moving from its current state S t to a new state S t+1 generating a corresponding reward R t+1 as feedback (Fig. 2). Rewards serve for actions evaluation, thereby the agent seeks to maximize them in the course of learning. MDP yields to generate a trajectory that respects Markov property and starts as follows [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]:

S 0 , A 0 , R 1 , S 1 , A 1 , R 2 , S 2 , A 2 , R 3 , ... (V.1)
A policy is a correspondence between the states and possible action probabilities. If the agent follows a policy π, then π(a|s) is the probability of selecting action a when being in state s. Markov property (V.2) stipulates that the probability of every current state is only related to its immediately previous state-action pair, and it is independent of the other historical states and actions. The agent makes future decisions only based on one-step dynamics, which make the learning process and further online control simple, reliable, and efficient.

p(S t |S t-1 , A t-1 , S t-2 , A t-2 , ..., S 0 , A 0 ) = p(S t |S t-1 , A t-1 ) (V.2)

State Space

The state variables indicate the actual system status and they are the basis to choose actions. They can be presented by a vector with a finite dimension containing system dynamic parameters such as power/torque demand, battery state of charge, remaining fuel, bus voltage, vehicle acceleration, or vehicle velocity. Variables are sampled into discrete values to build a state grid considering all feasible combinations. Increasing variables number in the state space S will improve the environment description accuracy and reliability [START_REF] Shen | Energy management of hybrid uav based on reinforcement learning[END_REF]. Increasing sampling frequency to get higher discretization may give better results in terms of control effectiveness [START_REF] Meng | Double q-learningbased energy management strategy for overall energy consumption optimization of fuel cell/battery vehicle[END_REF]. However, with a large state dimension, the computational burden will be higher and the convergence will be slowed down. This is because the number of states often increases exponentially as the number of state variables increases. This is known as the "curse of dimensionality" issue. Thus, a trade-off should be made in choosing the appropriate state parameters with a reasonable discretization degree. For example, in [START_REF] Li | Approximate costoptimal energy management of hydrogen electric multiple unit trains REFERENCES using double q-learning algorithm[END_REF], authors settled for load power P load and battery state of charge SOC for a hydrogen electric train power system.

Action Space

The agent takes actions to control the system, while maximizing the rewards.

Actions encompass any control we want to train the agent to perform it in the best way. It can be motor torque, motor speed, fuel cell power reference, or change of fuel cell delivered power. Similar to state space, actions space 123

Chapter V. Reinforcement Learning-based Energy Management Strategies A is discrete. For example, if we consider the EMS controlling the fuel cell power change, the action space is given as a n-dimensional tuple of potential variations of the fuel cell power output [START_REF] Wu | Cost-effective reinforcement learning energy management for plug-in hybrid fuel cell and battery ships[END_REF]:

A = [∆ 1 , ∆ 2 , ∆ 3 , ..., ∆ m = 0, ..., ∆ n ] (V.3)
where ∆ 1 and ∆ n represent the maximum decrease and increase in fuel cell delivered power, respectively. ∆ m is used to keep fuel cell power constant. Figure 3 illustrates an example of the interaction between the agent and state space for a hybrid electric bus application [START_REF] Wu | Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus[END_REF]. Actions are represented by the bus control signals (engine torque, rotational speed). The environment includes vehicle state variables (speed, acceleration) and also traffic information such as link speed, which is used in traffic flow modeling and refers to other vehicles speed on a given road section.

Rewards and Return

The agent objective is to maximize not only the instantaneous received reward, but also the long-term cumulative expected rewards and known as return (G t ). Indeed, the return is represented by a discounting sum of expected future rewards.

G t = R t+1 + δR t+2 + δ 2 R t+3 + ... = T 0 δ k R t+1+k (V.4)
where δ refers to the discount factor and T is the termination time. It is worth noting that returns obey a recurrence relation as shown in (V.5):

Reinforcement Learning

G t = R t+1 + δ(R t+2 + δ 2 R t+3 + δ 3 R t+4 + ...) = R t+1 + δG t+1 (V.5)
As 0 ≤ δ ≤ 1, G t will always be bounded. δ establishes the current value of the future rewards: a reward obtained after k + 1 time steps in the future is valuable only δ k times the value it would have if it were given in the current time step t [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. If δ = 0, this implies that only the present reward is considered for evaluation. The agent in this case is short-sighted (or myopic) and is typically leading the policy to fall into a local optimum [START_REF] Li | Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information[END_REF]. Furthermore, the closer the factor δ is to 1 the more the agent strives for long-term rewards.

If δ = 1, all future rewards will have the same importance as the current one. This makes policy convergence difficult. Thus, a trade-off should be made when choosing δ. Indeed, an adaptive discount factor can be adopted. Beginning with a lower δ and increasing it along the learning stage speeds up the learning process [START_REF] Kolodziejczyk | Real-time energy purchase optimization for a storage-integrated photovoltaic system by deep reinforcement learning[END_REF].

The choice of the reward function is critical because it affects convergence speed and control quality of the reinforcement learning EMS [START_REF] Li | Reinforcement learning energy management for fuel cell hybrid system: A review[END_REF]. It is generally expressed using the objective function F of the targeted optimization problem:

R t = -F = -[β 1 .f 1 + β 2 .f 2 + ... + β n .f n ] (V.6)
where f 1 -f n represent optimization goals like fuel and electricity consumption [START_REF] Xu | Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle[END_REF], battery SOC deviation [START_REF] Li | Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information[END_REF], and energy sources degradation cost [START_REF] Wu | Cost-effective reinforcement learning energy management for plug-in hybrid fuel cell and battery ships[END_REF]. β 1 -β n represent goal weights and they can be set by tuning. Indeed, reward function is usually normalized to [0, 1] using tanh() function as demonstrated in (V.7). This normalization improves the convergence speed [START_REF] Li | Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information[END_REF].

R t = -tanh [F ] (V.7)
Besides objective functions, optimization constraints have to be considered in the reward function too. Constraints are usually used to keep power sources operating within their limits such as minimum/maximum output power, SOC range, and power change rate. If any constraint is violated during training or any generated state/action is infeasible, the agent will be penalized by introducing a negative reward as a punishment (R t = -1). On the contrary, if the agent performs very well achieving perfect results (zero SOC deviation [START_REF] Li | Approximate costoptimal energy management of hydrogen electric multiple unit trains REFERENCES using double q-learning algorithm[END_REF]), an additional recompense could be given (R t = R t + 1) as presented in (V.8).

125

Chapter V. Reinforcement Learning-based Energy Management Strategies

R t =      -tanh [F ] + 1 when recompensing -1 when punishing -tanh [F ] else (V.8)

Value Functions

Reinforcement learning algorithms generally require value function estimations. The state-value function, denoted as v π (s), is the expected return when being in state s and following the policy π. Similarly, the action-value q π (s, a) is the expected return from choosing a particular action a in a given state s while following the policy π:

           v π (s) = E π T 0 δ k R t+1+k | S t = s = E π (G t | S t = s) q π (s, a) = E π T 0 δ k R t+1+k | S t = s, A t = a = E π (G t | S t = s, A t = a) (V.9)
During learning, the agent calculates the average of all returns that have followed a given state s. This average will converge to v π (s) (Fig. 4a). In the same way, returns average for every action performed in every state will converge to the action-values q π (s, a) (Fig. 4b).

Using learning experiences, reinforcement learning algorithms always seek to match policies that provide the highest returns over the long term. These optimal policies π * have expected returns that are greater than or equal to all other policies for all states [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. They have common best state-value and action-value functions ( v * (s) and q * (s, a)):

   v * (s) = v π * (s) = argmax π v π (s) q * (s, a) = q π * (s, a) = argmax π q π (s, a) (V.10)

Reinforcement Learning Algorithms

RL has been successfully applied to HEV energy management in the literature. Figure 5 shows the implementation process starting from real driving cycles gathering to online implementation. Hereafter, we will explain the concept of RL algorithms that have been widely adopted in HEV EMS applications.

Q-Learning (QL)

Q-learning algorithm is a common algorithm proposed by Watkins [START_REF] Watkins | Q-learning[END_REF]. It is a model-free RL strategy, which seeks to estimate the optimal policy based on maximizing the expected state-action value function Q (q(s, a)) for all possible state-action pairs. The approximation of Q is done regardless of the followed policy, which makes Q-learning an off-policy-based algorithm. For each time step, the interaction between the agent and the environment yields an update in the Q table, as indicated in the following expression:

Q(S t , A t ) ← (1 -α)Q(S t , A t ) + α R t+1 + δ max a Q(S t+1 , a) (V.11)
where α ∈ (0, 1] is the learning rate. It established how much is the agent learning from new experiences rather than using directly old acquired estimates. If α = 0, the agent will just exploit prior knowledge without learning anything. If α = 1, the agent will ignore all its prior learning and considers only the current estimation. A fixed learning rate is commonly used in practice, for instance, α = 0.1. The term (1 -α)Q(S t , A t ) is the current Q value weighted by the learning rate. The quantity max a Q(S t+1 , a) refers to the maximum estimate that can be obtained in state S t+1 , discounted by δ. Indeed, the learning rate serves as a weighting factor between the observed Q value and the new estimated one.

Equation (V.12) shows that action selection is usually performed using an -greedy method, which has proven high performance than a greedy strategy [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. In an -greedy policy, the agent chooses the greedy action (which has the highest Q value) with probability 1 -, that is exploitation. On the other hand, a random action is selected with a probability of seeking for any eventual better actions, that is exploration. The advantage is that all actions will be explored a sufficient number of times leading necessarily to optimal actions selection, and then converging to the optimal Q * . In practice, a low value is often chosen (for example: = 0.001 or 0.1). The Q-learning algorithm is summarized in Algorithm 1.

Reinforcement Learning Algorithms

A t = max a Q(S t , a)
with a probability of 1a random action with a probability of (V.12) It is well known that Q-learning achieves the optimal value function with certainty [START_REF] Jaakkola | Convergence of stochastic iterative dynamic programming algorithms[END_REF]. In addition, when choosing an appropriate learning rate α, the algorithm shows an exponential convergence with trials number [START_REF] Even-Dar | Learning rates for qlearning[END_REF]. However, it suffers from a poor convergence rate, especially if the discount factor δ approaches 1. Speedy Q-learning (SQL) algorithm is then proposed to deal with this limitation [START_REF] Ghavamzadeh | Speedy qlearning[END_REF]. A decreasing learning rate is adopted instead of a fixed one when updating Q function (α t = 1/(t + 1)) while involving two estimates Q(t -1) and Q(t) related to previous and current time steps. Indeed, it was proven that SQL leads to a fast learning process [START_REF] Ghavamzadeh | Speedy qlearning[END_REF], however, it increases the algorithm complexity.

Algorithm

Double Q-Learning

Any practical estimation method can lead to cumulative errors due to the fact that real values are unknown. Those errors can be caused by non-stationary behaviors, environmental noises, or approximations [START_REF] Van Hasselt | Deep reinforcement learning with double q-learning[END_REF]. Indeed, by involving the max operator to identify the next state value in Q-learning, the action value may be strongly overestimated. That induces a propagating positive bias when selecting the highest value as an estimate, consequently, algorithm performance will be penalized.

Double Q-learning was proposed in [START_REF] Hasselt | Double q-learning[END_REF] to overcome the overestimation issue using two estimators (Q A and Q B functions) instead of a biased one, and separating the estimator selection from estimate calculation. As shown in (V.13), every Q-function is updated by involving the other Q-function with equal probabilities. When updating Q A in a state s, we observe the action a * that maximizes Q A (a * = max a Q A (s, a)), then we use the Q B estimate for that selected action Q B (s, a * ), instead of adopting Q A (s, a * ) value, and vice versa. This is regarded as an unbiased estimation since Q B is actualized on the identical environment, with different experience samples than Q A ones.

For actions selection (Algorithm 1, line 9), both Q-functions will be involved by considering the new greedy action as (A * t = max a (Q A + Q B )) while following an -greedy policy. Regarding Q-functions updating (Algorithm 1, line 14), it will be done according to (V.13) as depicted in Algorithm 2. It is worth noting that the double Q-Learning approach requires twice the memory space without increasing the computational burden.

                   Q A (S t , A t ) ← (1 -α)Q A (S t , A t )+ α R t+1 + δ max a Q B (S t+1 , max a Q A (s, a)) Q B (S t , A t ) ← (1 -α)Q B (S t , A t )+ α R t+1 + δ max a Q A (S t+1 , max a Q B (s, a)) (V.13)

Sarsa

The name Sarsa is derived from the transitions happening in Markov sequences from one state-action pair to another giving rise to the quintuple (S t , A t → R t+1 , S t+1 , A t+1 ). Starting from (S t , A t ) pair, and after performing again A t+1 from S t+1 (following an -greedy method), then Q(S t , A t ) will be if rand q < 0.5 then / * with 0.5 probability * /

Q A (S t , A t ) ← (1 -α)Q A (S t , A t )+ α R t+1 + δ max a Q B (S t+1 , max a Q A (s, a)) else Q B (S t , A t ) ← (1 -α)Q B (S t , A t )+ α R t+1 + δ max a Q A (S t+1 , max a Q B (
Q(S t , A t ) ← (1 -α)Q(S t , A t ) + α [R t+1 + δQ(S t+1 , A t+1 ))] (V.14)
Unlike Q-learning, Sarsa is an on-policy RL algorithm. q π is estimated using the actually followed policy π, not the supposedly optimal policy as Qlearning does. When Q-learning approach uses max operator in updating, it is supposed that the completely greedy policy is followed ( = 0). However, it is not the case since the agent does not actually perform only greedy actions. Thus, Q-learning will converge to an optimal solution assuming that a greedy policy will be followed after training. In other words, the target policy is not the same as the behavior policy. On the other hand, Sarsa will converge to an optimal solution assuming that we continue to follow the same policy used in the learning stage. On-policy algorithms are generally simpler, and they have lower variance and faster convergence as compared to off-policy methods. It is worth noting that if action selection is greedy ( = 0), Sarsa and Q-learning would be the same algorithms.

Deep Reinforcement Learning Algorithms

RL-based strategies can perform an efficient energy management task in general. However, when dealing with continuous state/action variables or a high-dimensional state space, the discretization results in an exponential increase of the Q function matrix size. The Q table will be ridiculously large and impractical. Exploring all possible actions with an -greedy strategy would take a very long time leading to a high computational burden and low convergence performance [START_REF] Wu | Continuous reinforcement learning of energy management with deep q network for a power split hybrid electric bus[END_REF]. This refers to the aforementioned "curse of dimensionality" issue. For example, if we consider only 4 variables with only 100 grids each, the resulting state number is 100 million (100 4 ).

In addition, -greedy strategy could lead to a poor exploration task. For example, given an environment whit 10 possible actions a 1 -a 10 , and a Q-Learning agent following an -greedy technique with = 0.1. Supposing that at a given state s i , the highest Q value corresponds to a 1 where Q(s i , a 1 ) = 1.01, Q(s i , a 2 ) = 1.00 and Q(s i , a 3 ) = 0.01. The agent will select a 1 with a probability of 0.91 (90% of greedy selection and 1% of random selection), when the other actions have only a chance of 1% to be chosen, even though Q values for a 1 and a 2 are very close. The situation is even worse when looking that a 3 and a 2 are equiprobable even though Q(s i , a 2 ) is hundred times better.

To overcome classic RL algorithms limitations, a deep Q network (DQN) based on a deep neural network (DNN) is introduced to replace the Q table

Deep Reinforcement Learning Algorithms

for Q values approximation (Fig. 6). In contrast to the Q matrix, any continuous variation in the state variables leads to a change in the DNN output decision. Thus, states discretization is avoided allowing larger state space. That also allows considering more environment variables improving accuracy and control effectiveness without increasing the computational burden.

In addition, DNN performs well as an approximation function, which enables policy fast convergence.

Deep Q-Learning (DQL)

In DQL algorithm, a DNN is implemented to replace the Q matrix for value function Q estimation following policy π as follows:

Q(S t , A t , θ) ≈ Q π (S t , A t ) (V.15)
where θ refers to the DNN parameters (weights and biases), and Q(S t , A t , θ) represents the output corresponding to state-action pair (S t , A t ). Using DQN, a vector of all possible action values Q(S t , A i t , θ) can be produced for each given state S t under policy π. DQN is trained with a Q-learning algorithm using stochastic gradient descent to update its parameters θ. The objective is to minimize the loss function L which is represented by the mean-squared temporal-difference (TD) error between the target Q value y t (θ) and the predicted one Q(S t , A t , θ), like in a typical regression problem.

       y t (θ) = R t+1 + δ max a Q(S t+1 , a, θ) L(θ) = E y t (θ) -Q(S t , A t , θ) 2 (V.16)
However, the issue is that same parameters θ are used to estimate both target and Q-value. Consequently, we will have a high correlation between the changing parameters θ of the DQN noted Q θ and the target Q value. This implies that at every training time step, Q values change but also target values do. We are approaching the target, but the target is also shifting. This results in high oscillations in training. Thus, a separate fixed Q-target network is introduced using fixed parameters θ -which are obtained from some previous iterations. This fixed DQN noted Q θ -is adopted to avoid dependency between target Q values and θ improving the algorithm stability. Q θ - is periodically updated by transferring θ from Q θ for each C steps.

       y t (θ -) = R t+1 + δ max a Q(S t+1 , a, θ -) L(θ) = E y t (θ -) -Q(S t , A t , θ) 2 (V.17)
An important element of the DQL algorithm training is the experience replay. At every time step, agent's experience represented by tuple (S t , A t , R t+1 , S t+1 ) is saved in a data set named replay memory. In practice, the replay memory has usually a limited length, only N last experiences are stored. Instead of replacing S t with the new state S t+1 for the next update as we do in standard Q-learning, a new disconnected transitions batch sample is randomly selected from the replay memory to train the DQN by applying gradient descent on the selected batch. This task is called experience replay, it is repeated every time step. Since Q-learning is an output policy-based approach, it does not necessarily have to be performed along sequential paths [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. DQL algorithm pseudo-code is provided in Algorithm 3.

Experience replay technique has several advantages. The main one is obtained by breaking the correlation between successive tuples and then reducing the variance. In addition, by eliminating dependencies between consecutive samples θ, a source of instability will be removed. Figure 7 shows an illustration of a deep Q-learning EMS diagram [START_REF] Wu | Continuous reinforcement learning of energy management with deep q network for a power split hybrid electric bus[END_REF].

Policy Gradient Algorithms

The Q-Learning process seeks to identify the Q-value function for each stateaction pair. This learning scheme is built on the concept of determining the quality of all feasible actions, and then deciding based on this knowledge. Qlearning, therefore, aims to get unbiased and full knowledge of all possible decisions. But That is also its major disadvantage since it needs a large number of trials for each possible state. The policy gradient algorithm performs more robustly. Instead of attempting to estimate the value of all actions, it simply predicts the best action to take in each state. This can be very effective in continuous or large state spaces where getting value's argmax on all possible actions would be prohibitively expensive. The policy makes actions with high rewards more likely or the opposite. It keeps what works well and discards what doesn't. The model (neural network) will directly generate the selection probability for each action p(a, S t , θ), avoiding additional calculations related to Q function. Gradient policy algorithm learns a parameterized policy that can choose actions without checking the value function.

The training objective is to increase policy performance, thus, a scalar performance measure J(θ) is defined. To update θ, the algorithm seeks then to maximize the derivative of J(θ) by applying gradient ascent, the opposite of gradient descent: where ∇J(θ) is a stochastic estimate, which approximates J(θ) gradient [START_REF] Sutton | Reinforcement learning: An introduction[END_REF].

θ t+1 = θ t + α ∇J(θ) (V.
(S i , A i , R i+1 , S i+1 ) for i = 1 : N do      y i = R i+1 + δ max a Q(S i+1 , a, θ -) L(θ) = E y i -Q(S i , A i , θ)
It is worth noting that policy gradient algorithms perform all the updates retrospectively after the end of each episode, like Monte Carlo algorithms, and differently than the temporal-difference approach. The performance measure can be defined as the return G t , which represents the total accumulated rewards from time t including all future rewards until the end of the episode. That can be defined by the state value function v π θ for the policy π θ , and evaluated in the initial state of the episode S 0 [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]:

J(θ) = v π θ (S 0 ) (V.19)

Deep Reinforcement Learning Algorithms

The gradient calculation is detailed in [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] and is given by:

∇J(θ) = G t .∇ θ ln [π(A t |S t , θ t )] (V.20)
where π(A t |S t , θ t ) is the probability of selecting action A t in state S t given parameter vector θ t .

The policy gradient algorithm is an on-policy approach. Its learning process is based only on the state-action transitions performed by the actual followed policy. This means that all data sets are used only once. Thus, experience replay and replay memory are not required as in DQN. Once the model is trained, policy parameters θ are updated at the end of each episode (not after every time step) using gradient ascent, as shown in Algorithm 4.

Actor-critic Methods

In policy gradient algorithms, the agent learns directly a policy using the performance metric which is maximized in the course of learning. There is no necessity to know the actual value of each action. However, that leads to a certain policy dependency on the performance measure. We take an example to explain this issue. Assuming that the performance measure is set as the overall reward across a 100 steps episode. We suppose that all steps yield a reward of +10 except step 83, which gave -100. The accumulated return for the full trajectory will be high and the policy will average all actions taken in this episode as good. Consequently, the agent will keep selecting the same bad action in step 83 since it does not have knowledge about its actual value. Thus, our performance metric can not detect that pitfall. To overcome this drawback, the agent can also learn state and action values in parallel with learning the policy. This combination is called actor-critic method. Two subagents are involved to learn in parallel. The Actor learns directly the policy as in the policy gradient method, while critic learns the state-action value as in Q-learning (Fig. 8). The update rule is then changed substituting the return G t by Q(S t , A t ) leading to a more complex architecture:

θ t+1 = θ t + α Q(S t , A t ).∇ θ ln [π(A t |S t , θ t )] (V.21)
The update process is performed every time step as in TD methods. The agent dose not need to wait till the end of episode to do it as in policy gradient approach.

RL-based Energy Management

Reinforcement learning algorithms have been successfully implemented in hybrid power system energy management applications, including electric vehicles [START_REF] Lin | Reinforcement learning based power management for hybrid electric vehicles[END_REF], trains [START_REF] Li | Approximate costoptimal energy management of hydrogen electric multiple unit trains REFERENCES using double q-learning algorithm[END_REF], ships [START_REF] Wu | Cost-effective reinforcement learning energy management for plug-in hybrid fuel cell and battery ships[END_REF], and drones [START_REF] Shen | Energy management of hybrid uav based on reinforcement learning[END_REF]. Typically the optimization target is fuel consumption minimization. In a study dealing with a hybrid electric tracked vehicle EMS [START_REF] Liu | Reinforcement learning of adaptive energy management with transition probability for a hybrid electric tracked vehicle[END_REF], it has been reported that Qlearning led to less fuel consumption reaching 2.4% economy as compared to the stochastic dynamic programming (SDP) approach, and up to 22.7% when compared to a rule-based strategy. In a similar study [START_REF] Liu | Online markov chain-based energy management for a hybrid tracked vehicle with speedy q-learning[END_REF], speedy Q-learning algorithm has shown a fuel consumption gain of 2.9% and 5.5%

compared to SDP and Q-learning, respectively. Bai et al. [START_REF] Bai | Research on energy management of hybrid unmanned REFERENCES aerial vehicles to improve energy-saving and emission reduction performance[END_REF] applied Qlearning algorithm for a hybrid fixed-wing UAV powered by battery and internal combustion engine to reduce emission and conserve battery SOC. The reported results regarding fuel consumption showed a gain of 2.18% relative to the power tracking method in a cruise flight mission, and about 4.4% in terrain tracking flight mission. Q-learning also showed better response to the required power high fluctuations improving UAV maneuverability.

In [START_REF] Sun | Data-driven reinforcementlearning-based hierarchical energy management strategy for fuel cell/battery/ultracapacitor hybrid electric vehicles[END_REF], authors implemented an improved Q-learning-based EMS intended for HEV powered by fuel cell, battery, and ultracapacitor. Equivalent fuel consumption minimization was adopted as an objective inspired by ECMS approach. In addition, a transition probability matrix (TPM) obtained based on more than 40 thousand real driving cycles was introduced in the algorithm. The results outline an outstanding fuel economy as compared to ECMS ranging from 7.1% to 26% depending on the driving cycle. A TPM is a stationary Markov chain, which is used to model the required power using real power profiles. It provides the probabilities p i,j of moving from a state S i to another one S j in one time step:

   p i,j = M i-j M i M i = j M i-j (V.22)
where M i-j is the number of transitions from the load power P i load to P j load , when M i represents the total number of transitions performed from P i load . Figure 9 illustrates an example of TPM for an HEV application [START_REF] Xiong | Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle[END_REF]. The TPM establishment process is demonstrated in [START_REF] Li | Reinforcement learning energy management for fuel cell hybrid system: A review[END_REF].

Online RL-based strategy updating is studied in [START_REF] Xiong | Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle[END_REF]. The proposed algorithm performs online updates to the Q-learning policy according to current driving conditions. Kullback-Leibler divergence rate threshold is adopted to decide whether the new TPM is sufficiently different from the learning one. When it is the case, the strategy is updated involving the new TPM. The results showed a 16.8% decrease in energy loss relative to the rule-based EMS.

Xu et al. [START_REF] Xu | Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle[END_REF] have proposed a Q-learning parametric analysis considering HEV fuel economy. It was found that training experiences choice has an impact on learning performance. Selecting the best experiences from available data is more beneficial than learning from every experience. This result was also confirmed in another study [START_REF] Qi | Datadriven reinforcement learning-based real-time energy management system for plug-in hybrid electric vehicles[END_REF], where near optimal driving cycles control decisions are used for initialization accelerating learning convergence. Actions and states sensitivity analysis indicates that less fuel is consumed when action resolution is improved or state number is increased while increasing state resolution is detrimental to fuel economy. The authors also reported that increasing the exploration rate up to 10% reduces fuel economy. However, beyond 10%, the opposite effect appears. In another sensitive study [START_REF] Lin | Online recursive power management strategy based on the reinforcement learning algorithm with cosine similarity and a forgetting factor[END_REF], results outline the influence of learning rate α and discount factor δ. Faster policy convergence is obtained when increasing α from 0.01 to 0.1 and δ from 0.1 to 0.9.

Ensemble supervisory control approach was tested for the first time in [START_REF] Xu | Ensemble reinforcement learning-based supervisory control of hybrid electric vehicle for fuel economy improvement[END_REF] by using multi-agent RL. Authors combined multiple strategies namely: 2/4 states-based Q-learning, ECMS, and thermostatic strategy. The results showed that the four agents combination performs better than the best single agent (4 states Q-learning) with 3.2% less fuel consumption. However, not all multiagent combinations showed better results than the best single agent. Agents selection is thus fundamental. Shen et al. [START_REF] Shen | Energy management of hybrid uav based on reinforcement learning[END_REF] have proposed an improved double Q-learning approach targeting hybrid UAV flight time extension. Their idea consists on emphasizing the most valuable state-action pairs in Q tables. These particular areas are refined during the learning process to lock the optimal policy earlier accelerating the convergence. The obtained results show more the strategy than 90% close to the optimal solution given by dynamic programming in terms of fuel economy. In another study [START_REF] Li | Approximate costoptimal energy management of hydrogen electric multiple unit trains REFERENCES using double q-learning algorithm[END_REF], a DQL algorithm is proposed to control the fuel cell/battery power system for a train application. Total Cost of Ownership (TCO) is considered besides fuel economy and SOC conservation. The approach changes the action space dimension in the course of learning depending on the SOC deviation. HIL experiments showed that the proposed strategy reached 96% of optimality relatively to the dynamic programming method while keeping battery SOC.

Deep RL (DRL)

Deep Q-learning algorithms have been widely adopted for HEVs EMS. In a study dedicated to a hybrid electric bus [START_REF] Wu | Continuous reinforcement learning of energy management with deep q network for a power split hybrid electric bus[END_REF], the DQL was compared to Qlearning regarding training, SOC deviation, and state variables. It has been reported that the DQL training process was 3.6 times faster than Q-learning (2.3 h against 8.3 h). Q-learning algorithm could not learn quickly due to the lack of information about bus acceleration and speed. At the same time, it was unable to effectively manage the connection between keeping battery SOC and enhancing fuel consumption. Both algorithms were able to maintain initial SOC. However, DQL leads to a 5.6% better fuel economy achieving 89% of dynamic programming optimality. In similar studies comparing DQL with Q-learning-based EMS, DQL was implemented in [START_REF] Hu | Energy management strategy for a hybrid electric vehicle based on deep reinforcement learning[END_REF] for an HEV decreasing equivalent fuel consumption by 2.5% relative to rule-based strategy. In [START_REF] Li | Deep reinforcement learning-based energy management of hybrid battery systems in electric vehicles[END_REF], DQL-based EMS showed a 96.5% decrease in training time and 54.5% in computation time for an HEV powered by high-power and highenergy batteries.

DNN hyperparameters have to be appropriately adjusted since they directly impact DQL performance. Their tuning can be done manually, however, it will be time-consuming. Some strategies are proposed in the literature to do this task such as random search [START_REF] Bergstra | Random search for hyper-parameter optimization[END_REF]. However, such a method is computationally expensive. In this context, Kong et al. [START_REF] Kong | Energy management strategy for electric vehicles based on deep q-learning using bayesian optimization[END_REF] proposed Bayesian optimization (BO) as an efficient strategy for hyperparameters tuning. Simulation results have shown good DQL learning performance as compared to the random search method. In addition, the implemented DQL for a supercapacitor/battery-powered vehicle performed slightly better than a nearoptimal rule-based strategy in terms of energy consumption and battery life preservation. In [START_REF] Chowdhury | Deepqgho: Quantized greedy hyperparameter optimization in deep neural networks for on-the-fly learning[END_REF], hyperparameter optimization has been carried out using a new greedy method to accelerate on-the-fly learning. It was proved that the proposed approach is more than 5× faster than BO leading to about 80% less energy consumption during training. However, BO was slightly more accurate.

In a recent study [START_REF] Zou | Dql energy management: An online-updated algorithm and its application in fix-line hybrid electric vehicle[END_REF], an accelerated DQL algorithm with prioritized replay was implemented for HEV EMS. In standard DQL, experience replay allows the agent to recall and reuse previous experiences uniformly sampled from 5. RL-based Energy Management a replay memory neglecting their importance. The prioritized experience replay approach emphasizes significant transitions involving them with higher frequency and therefore learns more effectively. A specific function has been included in the EMS to measure the priority, set transitions, and update the significance weights. The reported hardware-in-the-loop simulation results have shown up to 10.6% in fuel economy and a gain of up to 86.7% in computational cost relative to a standard DQL.

A double deep Q-learning (DDQL) algorithm was introduced in [START_REF] Han | Energy management based on reinforcement learning with double deep q-learning for a hybrid electric tracked vehicle[END_REF] for a hybrid electric tracked-vehicle EMS. Like double Q-learning, DDQL has prevented the EMS to fall into overestimation. The reported results demonstrated that DDQL outperformed the standard DQL by 7.1% in fuel economy reaching 93.2% of Dynamic programming optimality. In addition, the DDQL maintained battery SOC considering different initial SOC.

In [START_REF] Li | Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information[END_REF], authors explored a deep deterministic policy gradient (DDPG) strategy, which is an actor-critic off-policy approach that does not need any discretization for state/action spaces. It uses experience replay memory and involves two copies of both actor and critic networks, one copy for training and the other one is fixed and updated periodically. The algorithm optimality in terms of fuel economy was validated achieving an average deviation of only 3.5% from deterministic dynamic programming. DDPG approach was also investigated in [START_REF] Wu | Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus[END_REF] for a series-parallel plug-in hybrid electric bus. Real traffic information gathered by using sensors are employed to enhance learning performance. The proposed DDPG outperformed the standard Q-learning algorithm in terms of fuel consumption and showed better generality to new driving cycles which were very different from training ones. An improved DDPG algorithm is proposed in [START_REF] Li | Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information[END_REF]. The conjunction of two techniques was proven to achieve effective learning: a critic network with a duel configuration is adopted and a pre-training is performed before the learning process. A duel architecture has two streams for separately estimating the state value and advantage of every action related to that state. The two values are then aggregated to provide a good estimate of the state-action value. The pre-training is carried out using optimal experience samples obtained by DP. These ideal samples are produced and copied to the replay memory at the initial stage.

Distributed DRL architecture wass investigated in [START_REF] Tang | Distributed deep reinforcement learning-based energy and emission management strategy REFERENCES for hybrid electric vehicles[END_REF] by proposing an asynchronous advantage actor-critic (A3C) algorithm. Indeed, the Q value when considered solely dose not always reflect the action importance, and that may be confusing. Thus, the agent in A3C learns the advantage A(S t , A t ) of choosing a specific action as compared to others instead of only learning its Q value.

A(S t , A t ) = Q(S t , A t ) -V (S t ) (V. [START_REF]Aerial photography/reconnaissance hexarotor[END_REF] where V (S t ) represents the state value function. In addition, multiple agents are used to learn the same policy but updated asynchronously, which increases the learning process speed and robustness. Simulation results showed a 4 times superior convergence rate and better fuel economy achieved by A3C as compared to standard DQL.

Deep Transfer Learning

DRL algorithms have shown good generality to new driving conditions that are different from training ones. When encountering new similar control tasks or using a new HEV type, the agent has to be retrained from zero to learn the optimal policy adapting to the new environment leading to a tedious training time. In addition, it is difficult to generate a good and extensive dataset due to the cost of data acquisition [START_REF] Tan | A survey on deep transfer learning[END_REF]. In this context, transfer learning (TL) may be combined with DRL as a solution to this issue decreasing energy consumption and overcoming the problem of small data availability. TL is a machine learning approach that exploits shared knowledge to perform different tasks in related fields [START_REF] Weiss | A survey of transfer learning[END_REF]. The previous knowledge will be exploited to accelerate the learning and the ancient policy will be just adapted instead of learning from zero. Most of the parameters can be stored in the new neural network and the model will be recycled to build the new problem solution (Fig. 10). The other parameters will be randomly initialized and updated with RL. Then, fine-tuning is applied to adjust all parameters using a relatively small number of driving cycles related to the new domain. Therefore, the training time will be significantly reduced.

In a recent study [START_REF] Lian | Cross-type transfer for deep reinforcement learning based hybrid electric vehicle energy management[END_REF], authors were able to transfer energy management knowledge between two HEVs with significantly different structures. Performance of TL, when applied to DDPG, were evaluated considering the five metrics defined in [START_REF] Taylor | Transfer learning for reinforcement learning domains: A survey[END_REF] to assess the potential of the transferring task. The metrics include fuel consumption, convergence efficiency, and generalization ability. The obtained results have demonstrated that the proposed method has achieved an average gap of 70% from the baseline in terms of convergence efficiency.

RL-based Energy Management

In [START_REF] Guo | Transfer deep reinforcement learning-enabled energy management strategy for hybrid tracked vehicle[END_REF], DDPG and TL were connected in an adaptive EMS for an HEV. A two-level control scheme is implemented to generate the EMS. The upper level applies the DDPG strategy for training the EMS at various velocity intervals, while the lower level uses the TL method to transform the pre-trained neural networks for a new driving cycle. Authors in [START_REF] Xu | A comparative study of deep reinforcement learning-based transferable energy management strategies for hybrid electric vehicles[END_REF] investigated exploration strategies for a DDPG-based transfer learning method. The reported experimental results have shown that parameters space noise achieved good performance in terms of adaptation and stability.

RL Challenges and Perspectives

There are some challenges in the application of RL to HEV energy management. Indeed, the vast majority of the studies presented in the literature were carried out in simulations. Although this is an appropriate and reasonable approach to deal with these problems, but RL methods strongly rely on Chapter V. Reinforcement Learning-based Energy Management Strategies an accurate design of the simulator and data representing real-world situations. There is also a problem of an outstanding discrepancy between simulation and reality that makes it difficult for the simulated EMS to perform with same the performance in the real world. Since RL is an online learning algorithm, it could be more accurate and effective if it was implemented directly in a physical vehicle to train the energy management system instead of learning in a simulated environment. However, that approach will be risky, time-consuming, and very costly. Because the learning period will be very long due to the trial and error phase in the real environment. One potential solution is to design a high-precision training simulator to generate the optimal policy and then adjust it adaptively in real-world applications [START_REF] Wu | Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus[END_REF]. In addition, all RL strategies rely on the quality and quantity of training data. Hence, more in-depth data analysis must be conducted [START_REF] Tang | Distributed deep reinforcement learning-based energy and emission management strategy REFERENCES for hybrid electric vehicles[END_REF] to efficiently exploit them in the learning process. Contemporary vehicles are typically fitted with a variety of sensors. Thus, gathering more information from the vehicle sensors will improve the environment description, accuracy, EMS performance, and fuel economy. Furthermore, sharing online information between many vehicles is a promising tool to enhance energy management performance.

Conclusions

A comprehensive conceptualization of reinforcement learning was provided and the major motivations supporting the advancement of HEV reinforcement learning-based energy management are presented. The main contribution of this study was to provide a comprehensive critical review of different reinforcement learning algorithms on implemented in HEV energy management applications. From the carried out state-of-the-art review, it can be concluded that that:

• Reinforcement learning-based EMSs achieve a considerable improvement in the HEV energy economy. The energy optimization as compared to conventional EMSs differs from one application to another according to the specific driving conditions and energy savings were reported, varying from 2.18% to 26%. The reported results have also shown that reinforcement learning generates very near-optimal policies as compared to the benchmark optimal strategy, which is typically represented by dynamic programming.

Conclusions

• Neural networks integration has shown improved effectiveness over standard reinforcement learning methods.

• Learning convergence time is significantly reduced relative to standard offline optimization methods, especially when using deep transfer learning.

• Most of the investigated reinforcement learning-based EMSs are restricted to simulation setups. Thus, accurate and powerful simulators are needed and high-quality data representing real conditions should be obtained in the pre-training stage.

• Online information sharing between related HEVs is a valuable tool for improving energy management performance.

Conclusions

Drones are in continuous development and they are reaching a large range of applications. In this context and as the propulsion system constitutes the mainstay of a drone platform, this thesis focused on the onboard propulsion system energy aspect by proposing energy management strategies dedicated to small electric fuel cell-powered drones. Indeed, fuel cell integration has led to a significant increase in drone endurance thanks to its high energy density and quasi-instantaneous refueling. However, one source-based drone power supply system is greatly limited because of its poor performance under varying operating conditions. Thus, the hybridization of power sources with different characteristics is becoming a standard solution. In this context, power supply system design and topology selection are important to build an architecture that fits well with the drone platform considering the overall system efficiency, weight, and cost. The energy management policy also has a direct impact on fuel consumption and drone autonomy.

The proposed energy management strategies are focused on economic objectives and sources lifetime preservation while considering real power requirements, extracted from a real experimental flight test conducted using a small electric hexacopter. Rule-based methods enabled fuel cell service life extension by reducing power fluctuations and smoothing its output power profile. The load power peaks were supplied/absorbed by the battery and supercapacitor due to their fast dynamics reducing the battery charge/discharge cycles number. Drone maneuverability is improved by enabling fast power response. Optimization-based approaches were proved capable of optimally distributing load power between fuel cell and battery improving hydrogen saving. ECMS can save up to 853.2e per drone during one fuel cell module service life considering the Intelligent Energy 650W fuel cell power module as a case study. This benefit is estimated to reach 1450e in GA-based EMS. Indeed, the endurance gain depends on the transported fuel quantity: the more it is, the longer the drone flies. The frequency of return trips to the ground stations during the mission can also be reduced, inducing more hydrogen economy. Fuel saving will lead also to a reduction in operating costs.

Reinforcement learning-based EMSs have achieved a considerable improvement in the HEVs field and they can be exploited for drone energy management. The energy optimization as compared to conventional EMSs differs from one application to another according to the specific driving conditions and energy savings were reported, varying from 2.18% to 26%. The

Future Work

reported results have also shown that reinforcement learning generates very near-optimal policies as compared to the benchmark optimal strategy which is typically represented by dynamic programming. Neural network integration has shown improved effectiveness over standard reinforcement learning methods. Learning convergence time is significantly reduced relative to standard offline optimization methods, especially when using deep transfer learning. Most of the investigated reinforcement learning-based EMSs are restricted to simulation setups. Thus, accurate and powerful simulators are needed and high-quality data representing real conditions should be obtained in the pre-training stage. Finally, online information sharing between related drones is a valuable tool for improving energy management performance.

Future Work

The thesis contributions show the following promising aspects that can be investigated in the future:

• Implementation of the developed EMSs using HIL simulations for validation. Then, implementation on a real fuel cell/battery drone to assess their real impact on operation level and to fill the gap of experimental studies in the drone energy management field.

• Exploring the synergy of components sizing, energy management, and aerodynamic characteristics. The investigation in this research area will be costly since it requires the experimental design of all propulsion system components to fit on drone applications and depending on the targeted mission. However, it is very important and beneficial for deeper optimization.

• Considering the total cost of ownership (TCO) over the drone lifetime in EMS objectives.

• Exploiting reinforcement learning techniques in both drone path planning and energy management. In this context, it is necessary to conduct a high number of real flights to gather extensive and high-quality data sets for different mission types to enable efficient implementation.

Titre : Contribution à l'optimisation de la gestion d'énergie dans les drones hybrides alimentés par pile à combustible et une batterie. The reason is that drones have abilities to perform some difficult or dangerous tasks, with high mobility, safety, and low cost. It should be noted that drones are revolutionizing many public services including real-time monitoring, search and rescue, wildlife surveys, delivery services, wireless coverage, and precision agriculture. To increase endurance and achieve good performance, the use of a hybrid power supply system architecture is considered as a promising option. A hybrid power architecture may combine several power sources such as fuel cells, batteries, solar cells, and supercapacitors. The choice of a suitable power source hybridization architecture with an optimal energy management system is therefore crucial to enable the efficient operation of advanced drones. In this regard, this thesis focuses on proposing energy management optimization adapted to drone platform specifications by investigating several approaches namely: rulebased, optimization-based, and machine learningbased strategies. The goal is to have an optimal power allocation extending the drone endurance while preserving sources lifetimes. An extensive overview of reinforcement learning-based energy management for hybrid electric vehicle applications is proposed to exploit the achieved machine learning advancement in developing smart strategies for drones.

FIGURE 1 . 1 :

 11 FIGURE 1.1: Top 10 drone operators ranking in 2018 [1].

FIGURE 1 :

 1 FIGURE 1: Block diagram of a typical UAS.

FIGURE 2 :

 2 FIGURE 2: Block diagram of a drone propulsion system.

FIGURE 3 :

 3 FIGURE 3: Illustration of the classification proposed in [2].

FIGURE 4 :

 4 FIGURE 4: Drones configuration-based classification [3].

FIGURE 5 :

 5 FIGURE 5: Drone-based traffic monitoring [4, 5].

  Chapter II. State of the Art Critical Review on Drones Power Supply and Energy Management (A) Drone-based solar photovoltaic panels inspection [101, 102]. (B) Drone-based power lines inspection [39, 103].

FIGURE 6 :

 6 FIGURE 6: Drone-based infrastructures inspection.

2 .

 2 Unmanned Aerial System (UAS) Basic Knowledge (A) Drone-based volcano monitoring[START_REF]Volcano Monitoring Made More Efficient By Drones[END_REF].(B) Drone-based pollution monitoring[START_REF] Margaritoff | Polish City of Katowice Uses Drone to Combat Smog[END_REF].

FIGURE 7 :

 7 FIGURE 7: Drone-based Environmental monitoring.

FIGURE 8 :

 8 FIGURE 8: Drone-based delivery [6, 7].

FIGURE 11 :

 11 FIGURE 11: Drones military applications [12-15].

FIGURE 12 :

 12 FIGURE 12: Swapping vs hotswapping techniques [16].

FIGURE 13 :

 13 FIGURE 13: Swapping and Hotswapping Algorithms.

FIGURE 15 :

 15 FIGURE 15: Laser-powered drones [19, 20].

  Chapter II. State of the Art Critical Review on Drones Power Supply and Energy Management It has been above-presented and discussed drones battery-based power supplying techniques.

FIGURE 17 :

 17 FIGURE 17: MMC's HyDrone 1550 multicopter equipped with 1800W H1-Fuel Cell [23].

FIGURE 20 :

 20 FIGURE 20: Fuel cell system auxiliaries illustration[START_REF] Kim | Fuel cell system with sodium borohydride as hydrogen source for unmanned aerial vehicles[END_REF].

FIGURE 21 :

 21 FIGURE 21: Fuel cell storage tank constraint illustration [27].

-

  Catalyst is required -Widely used for drones -React at 80-120 • C [58] -Dehydrogenation at 120-180 • C [159] -A proper catalyst must be used to eliminate carbon monoxide Advantages -Stable, and fast reaction at ambient temperature -The high H 2 weight fraction [160] -Low hydrolysis heat -Easy to handle -High hydrogen storage capacity (19.6%) [58] -Stable under ambient conditions [161] -High volumetric and gravimetric efficiency -Storage is easy to handle -High rates of hydrogen production at ambient temperatures Disadvantages -The hydrogen generator is needed ⇒ increase in weight -N aBO 2 precipitation issue -By-product tank is needed -Slow reaction -Liberation of gaseous impurities -Extensive material expansion and foaming -Carbon monoxide poisons the PEMFC The literature on hydrogen storage shows a variety of approaches. In [162], Swider-Lyons et al. provided a comparative study considering qualitative and quantitative criteria to select the best hydrogen storage method for a 24 hours flight performed by the Ion Tiger drone. However, based on practical and flight-tested prototypes developed by several research teams (EnergyOr)

FIGURE 23 :

 23 FIGURE 23: A hybrid fuel cell-battery-supercapacitor powersupply in a drone[START_REF] Gong | Analysis of a Fuel-Cell/Battery/Supercapacitor Hybrid Propulsion System for a UAV using a Hardware-in-the-Loop Flight Simulator[END_REF][START_REF] Gong | Flight test of a fuelcell/battery/supercapacitor triple hybrid uav propulsion system[END_REF].

FIGURE 24 :

 24 FIGURE 24: Active vs passive PMS Architectures.

FIGURE 1 :

 1 FIGURE 1: Hybrid power supply system topology.

FIGURE 3 :

 3 FIGURE 3: Experimental load power profile.

FIGURE 4 :

 4 FIGURE 4: Extended load power profile.

FIGURE 5 :

 5 FIGURE 5: Load profile frequency separation.

  0.85 = 1177 W P Bat-max = 623 0.85 = 733 W P SC-max = 912 W Energy SC = 912×5 0.4×3600 = 4 W h Energy Bat = 1633 0.4×0.85×3.6 = 1335 W h (III.1)

FIGURE 6 :

 6 FIGURE 6: Fuel cell stack model.

FIGURE 7 :

 7 FIGURE 7: Battery model.

FIGURE 8 :

 8 FIGURE 8: Supercapacitor model.

FIGURE 9 :

 9 FIGURE 9: DC/DC boost converter. (a) Switching model, (b) average value model.

FIGURE 10 :

 10 FIGURE 10: DC/DC bidirectional converter. (a) Switching model, (b) average value model.

  (SOC 0 = 40%), Fig 14 (SOC 0 = 70%), and Fig. 15 (SOC 0 = 95%).

6 .

 6 FIGURE 11: (a) FSRB EMS scheme and (b) DC bus voltage control scheme.

FIGURE 12 :

 12 FIGURE 12: FSRB EMS algorithm.

FIGURE 13 : 79 Chapter

 1379 FIGURE 13: Simulation results (SOC 0 = 40%).

FIGURE 14 :

 14 FIGURE 14: Simulation results (SOC 0 = 70%).

6 .

 6 Frequency Separation Rule-based (FSRB) EMS for Fuel cell/ Battery/

FIGURE 15 :

 15 FIGURE 15: Simulation results (SOC 0 = 95%).

FIGURE 16 :

 16 FIGURE 16: Topology of the fuel cell/supercapacitor power supply system.

FIGURE 17 :FIGURE 18 :

 1718 FIGURE 17: Frequency power sharing illustration.

FIGURE 20 :

 20 FIGURE 20: Supercapacitor charging curves.

FIGURE 23 :

 23 FIGURE 23: Energy profiles.

FIGURE 24 :

 24 FIGURE 24: Fuel cell voltage and current.

FIGURE 25 :

 25 FIGURE 25: Supercapacitor voltage, current, and SOC.

FIGURE 26 :

 26 FIGURE 26: Hybrid power supply system topology.

FIGURE 27 :

 27 FIGURE 27: Membership functions.

FIGURE 28 :

 28 FIGURE 28: Fuzzy logic control surface.

Figure 29

 29 Figure 29 represents power supplied by both fuel cell and battery. The fuel

FIGURE 29 :

 29 FIGURE 29: Power distribution for RBFS-EMS.

FIGURE 1 :

 1 FIGURE 1: DJI M100 quadcopter with Intelligent Energy 650W fuel cell power module [33].

FIGURE 2 :

 2 FIGURE 2: Intelligent Energy implemented EMS [33].

y

  = f mincon(F, y 0 , A, B, A eq , B eq , LB, U B, OP T ION S) y starts at y 0 and tries to find a minimizer of the objective function F . A and B are used for describing linear inequalities (A.y B), for our case A = B = [ ]

FIGURE 4 :

 4 FIGURE 4: SOC and DC bus voltage simulation results.

FIGURE 5 :

 5 FIGURE 5: Current and voltage responses simulation results.

FIGURE 7 :

 7 FIGURE 7: GA-based EMS optimization process.

FIGURE 8 :

 8 FIGURE 8: Drone operating modes illustration.

FIGURE 9 :

 9 FIGURE 9: Pareto front for charge sustaining mode.

FIGURE 10 :FIGURE 11 :

 1011 FIGURE 10: Pareto front for charge decreasing mode.

FIGURE 12 :

 12 FIGURE 12: Power distribution. (A) RBFL-EMS, and (B) RB-EMS, CS mode, CD mode, CI mode.

FIGURE 13 :

 13 FIGURE 13: SOC response. RB-EMS, RBFL-EMS, CS mode, CD mode, CI mode.

FIGURE 14 :

 14 FIGURE 14: Hydrogen consumption. RB-EMS, RBFL-EMS, CS mode, CD mode, CI mode.

FIGURE 1 :

 1 FIGURE 1: General schematic representation of EMS for HEV.

FIGURE 2 :

 2 FIGURE 2: Agent-environment interaction illustration.

FIGURE 3 :

 3 FIGURE 3: Example of an agent-environment interaction [34].

FIGURE 4 :

 4 FIGURE 4: Illustration of (a) the state-value function v * (s) backup and (b) the action-value function q * (s, a) back-up.

FIGURE 5 :

 5 FIGURE 5: RL-based EMS implementation process.

  s, a)) end / * Q tables updating * / S t ← S t+1 / * state updating * / by Q(S t+1 , A t+1 ) instead of using max a Q A (S t+1 , a) as an off-policy would do:

FIGURE 6 :

 6 FIGURE 6: DQN illustration.

FIGURE 7 :

 7 FIGURE 7: Deep Q-learning EMS training illustration [35].

2 end 20 end 21 every

 22021 Perform gradient descent to minimize L(θ)Update θ C steps: reset Q θ -= Q θ

FIGURE 8 :

 8 FIGURE 8: Actor-Critic training diagram.

FIGURE 9 :

 9 FIGURE 9: Power transition probability matrix illustration [36].

FIGURE 10 :

 10 FIGURE 10: Transfer learning concept illustration.

Mots clés :

  Drone, gestion de l'énergie, pile à combustible, batterie, économie d'hydrogène, l'apprentissage par renforcement. Résumé : L'intérêt pour les drones s'est rapidement accru ces dernières années. La raison en est que les drones sont maintenant capables d'effectuer des tâches difficiles ou périlleuses, avec une grande mobilité, une grande sécurité et un faible coût. Il convient de noter que les drones révolutionnent de nombreux services publics, notamment la surveillance en temps réel, la recherche et le sauvetage, les études sur la faune, les services de livraison, la couverture sans fil et l'agriculture de précision. Pour augmenter ainsi l'endurance d'exploitation des drones et obtenir ainsi de bonnes performances, l'utilisation d'architectures d'alimentation hybrides est considérée comme une option très prometteuse. Une architecture d'alimentation hybride peut combiner plusieurs sources d'alimentation telles que des piles à combustibles, des batteries, des panneaux PV et des supercondensateurs. Le choix d'une architecture d'hybridation de source d'énergie appropriée avec un système de gestion d'énergie optimal sont ainsi primordiaux pour permettre un fonctionnement performant des drones. A cet égard, ces travaux de thèse se focalise sur l'optimisation de la gestion de l'énergie adaptée aux spécifications du drone en investiguant plusieurs approches, à savoir : les stratégies basées sur les règles, sur l'optimisation et sur l'apprentissage automatique. L'objectif est d'obtenir une répartition optimale de l'énergie afin d'augmenter l'endurance du drone tout en préservant la durée de vie des sources d'énergie. Un panorama complet de la gestion de l'énergie basée sur l'apprentissage par renforcement pour les applications de véhicules électriques hybrides est présenté afin d'exploiter les progrès réalisés en matière d'apprentissage automatique dans le développement de stratégies intelligentes pour les drones. Title : On energy management optimization for hybrid fuel cell/battery drones Keywords: Drone, energy management, fuel cell, battery, hydrogen saving, reinforcement learning, Abstract: The interest in electric unmanned aerial vehicles (drones) is rapidly growing in recent years.

TABLE II

 II

	.2: Comparison of different batteries [128].
	Characteristic	Ni-Mh LiPo Li-S
	Specific energy (Wh/kg)	80	180 350
	Energy density (Wh/l)	300	300 350
	Specific power (W/kg)	900	2800 600

TABLE II .

 II 3: Critical evaluation of available swapping studies.

  Table II.4 proposes then a critical comparison of the available literature on these techniques.

	FIGURE 16: Tethered drones [21, 22].

TABLE II . 4 :

 II4 Comparison between battery-based supplying techniques.

	Technique	Advantages	Limitations and drawbacks	Ref.
	Swapping Unlimited operating time, good	Necessity of a ground station
		option for long-range missions,	(GS), reduced operational effi-	[16,
		uses only one energy source ⇒	ciency, increased batteries and	51,
		Weight and complexity of power	drones number ⇒ high cost;	52,
		management are extremely re-	concerns of cooperation between	131-
		duced.	drones and the GS; issues in	140]
			autonomous swapping: landing,
			battery changing operation.
	Laser-	Unlimited operating time, wire-	Necessity of GS, reduced operat-
	beam	less refueling, no need to land,	ing heights, constraints related to	[16,
	charging	one energy source, persistent mis-	laser-beam obstruction, reduced	19,
		sions, the operating range is ex-	range.	53,
		tended.		54]
	Tethered	Unlimited operating time, no	The necessity of GS, limited oper-
	drones	need to land, one energy source,	ating area, drone damage in case	[55-
		safe and effective data transfer,	of tethering loss.	57]
		persistent operation.

TABLE II .

 II

	5: Key features of most used fuel cell types in drones
	[149-152]

Fuel cell type Fuel Efficiency (%) Temp. ( o C) Stack specific power (W/kg) System specific power (W/kg)

	PEMFC	Hydrogen	40-60	30-100	>500	>150
	DMFC	Methanol	20-30	20-90	>70	>50
	SOFC	Hydrocarbon	30-50	500-1000	>800	>100

TABLE II .

 II 6: Fuel cell-powered drones.

	Organisation	drone	drone	FC	Fuel type	Endurance
		name	type	type
	EnergyOr (2011)	FAUCON	Fixed	PEM Compressed	10 h 04
		H 2	wing		H 2	min
	U.S. Naval Re-	Ion Tiger Fixed	PEM	Cyrogenic	48 h
	search Laboratory		wing	(550	liquid H 2
	(2013)			W)
	EnergyOr (2016)	H 2 Qauad	Multirotor PEM	Compressed	2 h
		1000		(900	H 2
				W)
	SKYCORP (2018)	e-Drone	Multirotor PEM	Compressed	2 h
		Zero		(800	H 2 (3 l, 300
				W)	bar)
	BATCAM (2019)	/	Multirotor PEM Compressed	70 min
					H 2 (6 l)	(with
						5	kg
						payload)
	MetaVista (2019)	/	Multirotor PEM	Liquide H 2
				(800	(6 l)
				W)

TABLE II .

 II 7: Comparison between compressed and liquid hydrogen storage.

		Compressed H 2	Liquid H 2
		-Most used		-Density: 71 kg/m	3 at 1 bar
		-Storage pressures:	and -252.87 • C
	Properties	35-70 MPa [157] -Density: 42 kg/m	3 at	-A thermal insulation is needed to keep a very low temperature
		70 MPa [158]		-Fit on large scale applications
		-Simple		-Increased safety
	Advantages	-Low storage mass penalty	-High density
		-Rapid refueling capability	-Reduction of tank weight
				-High liquefaction energy ⇒ Costly
	Disadvantages	-Very low storage efficiency -Safety risks -System larger volume	process -Impractical small-scale production -Liquefaction requires very low temperature.
				-Difficult handling

TABLE II

 II

.8: Chemical hydrogen generation techniques comparison.

TABLE II .

 II 9: Comparison between batteries and supercapacitor[START_REF] Wang | A review of developments in energy storage systems for hybrid excavators[END_REF].

	Type	Energy	Power	Cycle	Efficiency	Merits		Drawbacks
		density	density	life	(%)
		(Wh/kg)	(W/kg)	(Times)
	Lead-	30-40	200-300 300-400 75	Low cost, high dis-	Poor performance at
	acid					charging rate, and	low temperature.
	battery					high recycling rate.
	Ni-MH	60-80	800-	1000	75	High energy density,	High	self-
	battery		1500			high charging and	discharging	rate,
						discharging speed,	need for a cooling
						and long lifetime.	system, and higher
									manufacturing cost.
	Li-ion	100-120 600-	1000	90	High voltage, high	Lifetime decrease at
	battery		2000			energy	density,	high	temperature,
						lightweight,	long	non-overcharge,
						cycle life, low self-	non-over discharge,
						discharging rate, no	and high security
						memory effect, and	requirement.
						no pollution.
	Super-	4-15	1000 -	100,000	85-98	Fast charging and	Low energy density.
	capacitor		10,0000			dicharging	speed,
						pollution-free and
						extremely long life.

TABLE II

 II

.10: Comparison of power supply configurations.

TABLE II .

 II 11: Comparison between active and passive PMS.

	PMS	Advantages	Disadvantages
	Active	-Optimized power usage -Safety of power system -Efficiency -Sources prior sizing is not needed	-Weight -Complexity -Power losses in converters
	Passive	-Simple -Light -Power losses are reduced	-Power distribution Low efficiency -Sources Reduced lifetime -Energy sources strict sizing required for reliable DC bus sharing

TABLE II .

 II 13: Power management strategies proposed for drones.

	Ref.	Power supply	Experiments/	Contribution	Limitations
		system	Simulations
	[188]	FC/ Battery/ Solar	Experiments	Flight test for 3.8 h of an imple-	Night operation not tested.
	2014	cells		mented rule-based PMS in 200
				W class drone.
	[187]	FC/ Battery	Both	State machine strategy tested in	Only ground test.
	2018			different SOC conditions.
	[189]	FC/ Battery/ Solar	Experiments	Switching technique between	Strategy algorithm missed.
	2018	cells		solar and fuel cell systems.
	[62] 2009 Battery/ Solar cells Experiments	Battery management in a solar	Power for propulsion system
				system to power the electronic	neglected.
				circuits in a designed drone.
	[185]	FC/ Battery	Experiments	Online fuzzy rule-based EMS	No flight test.
	2018			using	one	programmable
				DC/DC converter.
	[184, 186]	FC/ Battery	Simulations	Intelligent EMS and PMS using	Strategy not tested with
	2010,2009			ANFIS-based controller.	drone profile mission.
	[122]	FC/ Battery/ Solar	Simulations	Constrained thermostat control	The proposed architecture
	2012	cells		(CTC) strategy.	may make the system heavy.
	[190]	FC/ Battery/ Solar	Simulations	Combination of fuzzy logic and	No experimental validations.
	2018	cells		state machine strategies.
	[194]	FC/ Battery	Both	HIL simulation of simple rule-
	2016			based algorithm.

TABLE III

 III

		.1: 3D Robotics Y6 hexacopter specifications.
		Weight (with battery)	1905g
		Motors	850 KV brushless motors
	Airframe	Propellers	10 × 4.7 slow-fly APC
		Electronic Speed Controller (ESC)	SimonK firmware (20A)
		Payload capacity	600g
		Average Autonomy	12 -15min
		Type	Lipo 14.8V
		Weight	602g
	Battery	Capacity	6000 mA h
		Maximum power	1135W

Features Pixhawk autopilot system with GPS navigation

Remote control (Spektrum dx7s) Autonomous flight modes, waypoint navigation, loiter, circle, and return to launch

(A) (B)

FIGURE 2: (A) 3D Robotics Y6 hexacopter

[START_REF] Jeff | Iproduct Review: 3D Robotics RTF Y6 Multicopter[END_REF]

and (B) the Y6 drone in take-off phase.

TABLE III .

 III 2: Power profile characteristics.

				Profile characteristics	Value
				Duration			260s
				Average power		544W
				Maximum power		1319W
				Transient peak power	+741W (2s) -696W (3s)
		1.4
		1.2
	Load Profile (kW)	0.4 0.6 0.8 1
		0.2
		0	0	500	1000	Time (s)	1500	2000	2500

TABLE III

 III

	6. Frequency Separation Rule-based (FSRB) EMS for Fuel cell/ Battery/
	Supercapacitor Power system
	1500					Load Power FC power
						Bat power
						SC power
	1000
	500 Power (W)
	-500
	0	50	100	Time (s)	150	200	250
			.3: Total hydrogen consumption.
		SOC (%)	40	70	95
		Hydrogen consumption (g) 3.18 3.12 2.12

TABLE III .

 III

	P F C		VL	L	P Load	M	H
		L	L	M	H	H
	SOC	M	VL	L		M	H
		H	VL	L		M	H

6: Fuzzy Decisions Rules. VL: Very Low, L: Low, M: Medium, H: High.

TABLE IV

 IV

	.1: Intelligent Energy 650W fuel cell power module
		specifications [33].
		Maximum continuous power	650W
		Maximum peak power	1000W
	Fuel cell	Output voltage	29.6 -25.2V
		Weight	810g
		Sizes	196 × 88 × 140 mm
		Weight	250g
	Hydrogen	Maximum pressure	300bar
	regulator	Output pressure	0.5bar ± 0.25bar
		Maximum tank weight	10kg
		Capacity	1300mA h
		Weight	230g
	Battery	Sizes	140 × 30 × 20 mm

TABLE IV

 IV

	.2: Total hydrogen consumption.
	EMS	RB	RBFS ECMS
	Hydrogen consumption (g) 22.047 22.004 21.385

TABLE IV .

 IV

		3: Expected autonomy improvement in Intelligent
		Energy's fuel cell powered drones.
	FCPM	Tank	Hydrogen	Flight time	Endurance improvement
	650W	276W h kg -1 2L,	Compressed 85min	+2.55min
		6L	Liquid	10h 50min	+19.5min
	2.4kW	13L,	Compressed 111min (1	+3.33min
		4435W h kg -1		kg payload)
	800 W	6L	Liquid	12h 7min	+21.81min

TABLE IV

 IV

		.4: GA Input Parameters Constraints.
	Parameter	Lower bound Upper bound Initial value
	P F C-min (W)	100	300	150
	P F C-max (W)	650	1000	950
	SOC min	20	65	60
	SOC max	70	90	90
	P char (W)	30	60	30

TABLE IV

 IV

	.5: GA Setting Parameters.
	Parameter	Value
	Generation number	50
	Population size	50
	Crossover fraction	0.2
	Mutation fraction	0.8
	Pareto front fraction	0.35
	Function tolerance	1e-3

TABLE IV .

 IV 6: GA-based Optimization Solutions.

	Mode	Parameters SOC F 2 Objective functions
	Charge sustaining	41.18	81.69	476.65	677.3	43.31	10.76	0.007
	Charge depleting	33.75	80.40	349.82	918.28	49.53	8.82	0.001
	Charge increasing	42.01	80.02	137.82	923.15	47.61	13.43	0.0001

min SOC max P F C-min (W) P F C-max (W) P char (W) F 1 (g)

TABLE IV

 IV

	.7: Total Hydrogen Consumption.
	EMS	RB	RBFL CS	CD	CI
	H 2 consumption (g)	11.35 10.31 10.75 8.82 13.43

TABLE IV

 IV

		.8: Expected Extended Endurance for Intelligent En-
		ergy's Fuel Cell-based Drones.
	FCPM	Tank	Hydrogen	Flight time	Endurance improvement
	650W	276W h kg -1 2L,	Compressed 85min	+4.3min
		6L	Liquid	10h 50min	+33.1min
	2.4kW	13L,	Compressed 111min (1	+5.6min
		4435W h kg -1		kg payload)
	800 W	6L	Liquid	12h 7min	+37min

TABLE V .

 V 1: Existing reviews related to RL-based EMSs.

16 end 17 end 18 end

		1: Q-learning algorithm pseudo-code
		Inputs: δ ∈ [0, 1] (discount factor), α ∈ (0, 1] (learning rate), ∈ [0, 1)
		(exploration rate), rand , (a random number ∈ [0, 1] generated
		every time step)
	1 . for every state-action pair: do
	2	initialize Q(s, a) table
			/ * usually by a zeros matrix * /
	3 end
	4 for every learning episode: do
	5	while state = final state do
	6	for each time step: do
		observe the current state S t
		if rand < then
		A t = max a	Q(S t , a)
			/ * exploitation * /
		else
		A t = a random action
			/ * exploration * /
		end
		perform action A t , observe S t+1 and R t+1
		Q(S t , A t ) ← (1 -α)Q(S t , A t ) + α R t+1 + δ max a	Q(S t+1 , a)
			/ * Q table updating * /
		S t ← S t+1
			/ * state updating * /

  A (S t , a) + Q B (S t , a)]

		3. Reinforcement Learning Algorithms
		Algorithm 2: Double Q-learning algorithm pseudo-code
		Inputs: δ ∈ [0, 1] (discount factor), α ∈ (0, 1] (learning rate), ∈ [0, 1)
		(exploration rate), rand , rand q (random numbers ∈ [0, 1]
		generated every time step)
	1 . for every state-action pair: do
	2	initialize Q A (s, a) and Q B (s, a) tables
		/ * usually by zeros matrices * /
	3 end
	4 for every learning episode: do
	5	while state = final state do
	6	for each time step: do
		observe the current state S t
		if rand < then
		A t = max
		/ * exploitation * /
		else
		A t = a random action
		/ * exploration * /
		end
		perform action A

a [Q t , observe S t+1 and R t+1

18 )

 18 Algorithm 3: DQL algorithm pseudo-codeInputs: δ ∈ [0, 1], ∈ [0, 1), rand ∈ [0, 1], Q θ action value DQN, θ, θ -(Q θand Q θ -parameters), D (replay memory), N (batch size). 1 Initialize Q θ and Q θ -networks 2 for every learning episode: do

	3	while state = final state do
	4	for each time step: do
		observe the current state S t
		if rand < then
		A t = max a	Q(S t , a, θ)
			/ * exploitation * /
		else
		A t = a random action
			/ * exploration * /

end perform action A t , observe S t+1 and R t+1 Store experience in D Sample a random batch from D: N i=1

Table V

 V

	5. RL-based Energy Management
	.2 presents a comparison between the discussed RL
	algorithms.
	138

TABLE V .

 V 2: Comparison between RL Algorithms.

	Algorithm	Policy type	Advantages	Limitations
	Q-learning	Off-policy	Temporal difference method: does	Slow convergence.	Overestima-
		Value-based	not need to wait till the end of	tion. Curse of dimensionality. Not
			the episode to update. Flexible	adapted to continuous tasks. Deter-
			and adapts to multi-decisions tasks.	ministic: Can't learn stochastic poli-
			Directly learns the optimal policy.	cies
			Preferred in offline learning than
			Sarsa.
	Double QL	Off-policy	Unbiased estimation.	Increased memory requirement.
		Value-based
	Sarsa	On-policy	Better than QL in online learning.	Learns a near optimal policy: De-
		Value-based	Conservative: Avoid dangerous op-	caying in -greedy strategy may
			timal paths. Adapts to costly real	be a solution. However, this leads
			systems training.	to hyperparameter tuning. Instabil-
				ity due to correlation between con-
				secutive experiences.
	Deep QL	Off-policy	Good stability thanks to experience	Higher per-sample variance than
		Value-based	replay. Powerful computing perfor-	Sarsa.	Complicated computa-
			mances. Adapts to continuous and	tions.	Hyperparameters tuning
			complex tasks. No curse of dimen-	issue.	Less-adapted to high-
			sionality: More state variables can	dimensional and continuous action
			be handled improving accuracy.	spaces.
	Policy Gra-	On-policy	Adapts to stochastic environments	High variance and slow conver-
	dient	Policy-based	and continuous control. Robust,	gence. Unrealistic rewards due
			with good convergence. Effective	to global averaging in the end of
			in high-dimensional or continuous	episodes. Typically converge to a
			action spaces.	local optimum than a global one.
				Difficult to stabilize model param-
				eters. Dependency on performance
				measure.
	Actor-Critic	On-policy	Adapts to stochastic policies. In-	Require extensive hyperparameter
		Policy-based	dependent on performance metric.	search. Low stability and general-
			Combine strengths of both value-	ization. Time-consuming learning.
			and policy-based methods.

Unmanned Aerial System (UAS) Basic Knowledge

(A) (B)

Acknowledgements

List of Publications

The research work carried out during this Ph.D. has resulted in the following publications:

Chapter III. Rule-based Energy Management Strategies

Input parameters Values

Voltage at 0 A and 1 A [V 0 (V), V 1 (V)] [START_REF] Williams | Persistent mobile aerial surveillance platform using intelligent battery health management and drone swapping[END_REF][START_REF] Bongermino | Model and energy management system for a parallel hybrid electric unmanned aerial vehicle[END_REF] Nominal operating point [START_REF]HES multirotor drone, designed and built in US, has 3h flight time[END_REF][START_REF] Szczys | Laser power system keeps UAVs flying indefinitely[END_REF] Cells number 50

Nominal stack efficiency (%) 40

Operating temperature (

Results discussion

The power response for the two sources are shown in Fig. 21. It is observed that the fuel cell power profile is smooth in comparison to the supercapaccitor's one. All the power demand positive and negative peaks are handled by the supercapacitor. Its power mean value is almost equal to zero. As depicted in Fig. 22, the fuel cell has supplied the low frequency components, when the supercapacitor has supplied the high frequency components behaving as complementary source. Figure 23 presents the delivered energy for each source. As observed, almost all the energy required to load is delivered by the fuel cell. For this hybridization topology, it can be noticed that the fuel cell behaves well as the unique energy source, when the supercapacitor operates as a power source. Figures 24 and25 show voltage and current variation during the mission for fuel cell and suepercapacitor respectively. The supercapacitor voltage has been augmented for most of mission time. That means that it has been discharged for only few whiles during the mission. Practically, all the received energy has been redelivered to the load. 3b, it is observed that the fuel cell power follows well the energy management strategy rules. It supplies power according to the pre-defined operating points (P F C-min and P opt ). Outside these points, the fuel cell either supplies the needed extra power to charge the battery when its SOC is low or follow the load power when the SOC is high. For the ECMS in Fig. 3c, the power profile is slightly more fluctuating due to the optimization algorithm, which tries, for each sample time, to find out the good reference that minimizes the hydrogen consumption.

Chapter IV. Optimization-based Energy Management Strategies is estimated 79g h -1 for the multirotor drone. The Ballard fuel cell stack lifetime is 3000h. Thus, the amount of saved hydrogen during one lifecycle by one drone is equal to 0.03 × 79 × 3000 = 7110g. The hydrogen cost when provided by an industrial gas supplier in Norway is 0.12 eg -1 [START_REF] Apeland | Suitability analysis of implementing a fuel cell on a multirotor drone[END_REF]. Thus, the economical benefit is estimated to be 853.