Keywords: group pursuit, multiagents, hunting, drones, flocking group pursuit, multiagents, hunting, drones, flocking. Publications

In the last decade, we have witnessed significant advances in multi-robot systems. Much attention has been paid to the modeling of coordinated movement, called flocking; however, some current applications require more than the ability to navigate cohesively and without collision, for example, the use of drones to intercept a faster intruder. This application, to which this thesis is dedicated, although it is a subset of collective motion, has characteristics contrary to flocking, such as dispersion, in place of aggregation, and capture, instead of keeping desired distance.

, such as local sense, limited interaction, and decentralization decision-make. Nevertheless, we differ from most of them for considering the environment's perception of polar and relative coordinates, which is a consistent assumption considering embedded sensors (LIDAR and camera). Besides, our work extends known techniques of navigation guidance laws to the multiagent problem. In other words, pursuer behavior is given by modified guidance laws, where the parameters are adapted according to the pursuit's engagement. As a result, we have emergent group ambush with non-crossing trajectories between pursuers.

Introduction

Motivation In the last decades, we have seen an abrupt popularization of mobile robots around the world. Consecutively, civil authorities have been facing new challenges, such as the mitigation of criminal robots. In fact, unauthorized drone flights over protected areas, such as over airports and prisons, have been common in newspaper headlines.

Although there are preventive solutions, the current technologies have serious drawbacks, such as high-cost structure required, side effects or inefficiency. Against this backdrop, an interesting solution has arisen lately: the use of antidrone drones, which has proved itself an efficient and less costly solution. A more detailed discussion about the drone threat and the anti-drone solutions will be discussed in Section 1.1.

However, the solution discussed in this document aims at a scenario in the near future in the anti-drone fight, where the intruders can act autonomously, reactively, and can have superior capabilities than the single pursuer. This new scenario would require a group effort to accomplish the capture. The group of robots must perform more than a flight in formation or maintain the group's cohesion. They should be able to navigate cooperatively to ambush the most agile and reactive invaders, such as a group of predators behaves in real life to chase their prey.

Although countless examples could be listed here, we left to the reader three special examples to introduce our case: first, a flock of hundreds of starlings in a complex and fluid movement, splitting in the presence of predators and smoothly joining again, producing an amazing aerial display. Second, the perfect synchronization and symmetry of a jets fleet in an airshow, with a breathtaking maneuver in very high velocities.

Moreover, finally, a wolf-pack's wit and coordination to take down a stronger or faster prey, like a bison or a hare. In this last case, the struggle for survival from both parts provides these amazing spectacles. The preys adopt unpredictable escape trajectories, and the pursuers execute well-coordinated teamwork strategies to ambush and capture their prey. In turn, the wolf-pack, although it has similarities to both, represents a completely new branch, where the individuals do not seem to care much about the formation or even cohesion of the group. Instead, they trace their own trajectory towards the prey and adapt it in the neighbors' presence, which results in a surprising emergent ambush behavior.

Therefore, let us states the distinction between the two definitions: on the one hand, the term "flocking" refers to agent-based and decentralized approaches, where the individuals are homogeneous, having no explicitly hierarchy and able to sense and act locally. It is widespread in natural phenomena. Interacting units end up moving with about the same absolute velocity. This approach in multi-robots applications can also be referenced as "behavioral-based," where each agent acts based on a set of rules, or laws, such in [START_REF] Virágh | Flocking algorithm for autonomous flying robots[END_REF][START_REF] Olfati-Saber | Flocking for multi-agent dynamic systems: Algorithms and theory[END_REF].

On the other hand, by fleet formation, we refer to all the multi-agent strategies that keep a specific pattern (constellation). It can be referred to as virtual structures strategies in multi robots, which normalize centralized and leaderfollowing strategies. Each agent tries to keep a specific relative positioning towards the neighbors and the leader (physical or virtual). Examples of this kind of implementation can be seen in [START_REF] Turpin | Trajectory design and control for aggressive formation flight with quadrotors[END_REF][START_REF] Saif | Real-time flocking of multiple-quadrotor system of systems[END_REF][START_REF] Oh | A survey of multi-agent formation control[END_REF].

Both techniques, fleet formation and flocking, have been applied to the group chase evasion problem [START_REF] Lin | Bat swarming as an inspiration for multi-agent systems : predation success , active sensing , and collision avoidance[END_REF][START_REF] Gong | A GA based combinatorial auction algorithm for multi-robot cooperative hunting[END_REF][START_REF] Wang | A rapid hunting algorithm for multi mobile robots system[END_REF][START_REF] Saito | Group chase and escape model with chasers' interaction[END_REF][START_REF] Kim | Three-dimensional discrete-time controller to intercept a targeted UAV using a capture net towed by multiple aerial robots[END_REF]. Nevertheless, there are several differences between these distinct problems; and the obtained results fall short to reproduce the collective hunting phenomena. Formation based strategies, such as in [START_REF] Gong | A GA based combinatorial auction algorithm for multi-robot cooperative hunting[END_REF][START_REF] Wang | A rapid hunting algorithm for multi mobile robots system[END_REF][START_REF] Kim | Three-dimensional discrete-time controller to intercept a targeted UAV using a capture net towed by multiple aerial robots[END_REF], does not differ much from the case of a single pursuer. The constellation of robots behaves as a single agent, once their movements are constrained by the formation. Similarly, flocking-based strategies, such as in [START_REF] Lin | Bat swarming as an inspiration for multi-agent systems : predation success , active sensing , and collision avoidance[END_REF][START_REF] Saito | Group chase and escape model with chasers' interaction[END_REF], although does not require fixed formation, are still much worried about group's cohesion, which goes contrary to spreading out behavior of hunters. The alignment (or viscous term), frequently used in flocking, actually limits the action of the individual pursuer and, once more, preventing the necessary detachment necessary for the ambushing. These strategies mentioned above do not perform an efficient and realistic pursuit because they were not designed for this, but rather adapted from flocking or formation algorithms.

Work in progress as of 20 th June, 2021

Group pursuit-evasion One last topic, which could not be left out of this introductory chapter, is the recent field of group pursuit-evasion. It aims to describe prey-predators' complex behavior considering agent-based models, such as in [START_REF] Angelani | Collective predation and escape strategies[END_REF][START_REF] Saito | Group chase and escape model with chasers' interaction[END_REF][START_REF] Janosov | Group chasing tactics: how to catch a faster prey[END_REF].

Moreover, as pointed by [START_REF] Kamimura | Group Chase and Escape[END_REF], this recent and opened field inherits from two consolidated literature: pursuit-evasion and collective motion. Firstly, it can be seen as an extension of the single pursuit-evasion problem, taking advantage of vast literature involving: analytical solutions of pursuit, differential games, and missiles guidance studies. Furthermore, it can be considered as extension of the collective motion, taking advantage of the extensive literature in self-propelled particles, such as [START_REF] Vicsek | Novel type of phase transition in a system of selfdriven particles[END_REF], which has been used to model numerous complex systems, like bacteria, insect, physical particles, etc.

Recent work in escape from group chase [START_REF] Janosov | Group chasing tactics: how to catch a faster prey[END_REF] proposed modified versions of Vicsek's self-propelled particles to adapt the conventional flocking algorithm into a group pursuit. It was done by adding an extra chase term and modifying the alignment one. Although interesting behaviors can be exhibited in the simulations and efficient ambush behavior appears during the chase, their model has many variables, which made more difficult tunning and required optimization.

Furthermore, its applicability in the real world is limited due to its assumptions in observation; it relies on the knowledge of position and velocities of pursuers and targets expressed in the global frame. Although these assumptions can be partially afforded by considering communications between pursuers, the evader is not cooperative in real-world applications. Its global position and velocity are normally not available for the pursuer.

Furthermore, the application of these group pursuit strategies in robots in real-time will be studied in this thesis. For that, a multilayer motion controller architecture is proposed, allowing the application of high-level navigation laws in a quadcopter.

Our strategy share concepts with classical multi-agents methods [START_REF] Craig | Flocks, herds and schools: A distributed behavioral model[END_REF][START_REF] Olfati-Saber | Flocking for multi-agent dynamic systems: Algorithms and theory[END_REF][START_REF] Vicsek | Novel type of phase transition in a system of selfdriven particles[END_REF], such as local sense, limited interaction, and decentralization decision-make. Nevertheless, we differ from most of them for considering the environment's perception of polar and relative coordinates, which is a consistent assumption considering embedded sensors (LIDAR and camera).

Besides, our work extends known techniques of navigation guidance laws to the multi-agent problem. In other words, pursuer behavior is given by modified guidance laws, where the parameters are adapted according to the pursuit's engagement. As a result, we have emergent group ambush with non-crossing trajectories between pursuers.

Outline of the thesis:

In Chapter 1, we review essential literature related to group-pursuit applied to drones. Firstly, we go deeper into the counter-drone problem and the current solutions. Then, we present an overview of the collective motion literature, highlighting behavioral-based approaches and collective robots. After that, we introduce an overview of the pursuit-evasion problem, emphasizing navigation guidance laws and reinforcement learning. Finally, we introduce the emerging field of group pursuit-evasion, and we highlight the recent works.

In Chapter 2, we study the scenario where a group of pursuer and a single evader are traveling in a bounded 2-dimensional workspace. To solve the capture problem, we propose three distinct behavioral-based strategies for the pursuer team. The first one is based on the Deviated Pure Pursuit guidance, DPP, and the different values of the offset angle for each pursuer gives them different behavior in the stalking. The second strategy is an improvement of the previous one, where a Parallel Navigation term is added to the DPP. This allows the pursuer to take more efficient paths toward the target and still deviate from each other, avoiding collisions.

Work in progress as of 20 th June, 2021

Finally, we propose a Deep Reinforcement Learning framework to obtain through simulations close-to-optimal policies for the pursuers.

In Chapter 3, slightly different scenarios or assumptions are studied. Firstly, we will assume that the target's global positioning and velocity are available for the pursuers. With that, a predictor scheme is chosen to foresee the target's future position. Secondly, we consider a scenario where just one pursuer realizes the target's stalking; nevertheless, other non-cooperative agents, or mobile obstacles, are traveling in the same workspace. In a third and last scenario, we are interested in larger amounts of agents, where not all of them are informed about the target. In this, we investigate the flocking formation considering only the relative and polar information about the neighbors.

In Chapter 4, we investigate the application of the proposed strategies into a group of quadcopters. First, we describe the experimental setup, detailing the material used for the real-time flights. Then, we describe our work on state estimation, in which we implemented a Kalman filter algorithm for filtering and estimating position and velocity. Finally, we present the motion control architecture for a drone pursuer.

Finally, we propose a careful discussion about the thesis's proposed strategies, where criteria such as performance and viability of implementation will be taken into account. Besides, we will give a general conclusion and the research prospect of this work.

Work in progress as of 20 th June, 2021

The best way to defeat a tank is with another tank.

Popular saying

Chapter1

State of the art

In this state-of-the-art, we review the essential literature related to group-pursuit applied to drones. First, we present an overview of the counter drones problem and its current solutions in the civil environment. We evidence the promising solution of anti-drone drone (ADD), and we argue the need to use a group of ADD's to contain the coming threats. Then, we delve into the problem of multi-agents, giving an overview of some key works in the literature on collective motion and a particular focus on multi-robots in the pursuit. After that, we present an overview of the classical pursuit-evasion problem, emphasizing the emerging field of group chase-evasion. Moreover, we summarize the well-established literature in guidance laws, giving theoretical concepts for its development in this work. 1

.1 The counter-drone fight

The recent boom in the production and marketing of drones has profoundly affected our society. As reported in [START_REF] Gordon | Drones for Good: How to Bring Sociotechnical Thinking into the Classroom[END_REF], the commercial use of drones has a vast potential, and they have shown themselves as a versatile tool already being used for mapping, public security, humanitarian aid, among others.

However, their illness is also a reality that has dramatically concerned the population as a whole [START_REF] James | The dark side of our drone future[END_REF]. For example, in [START_REF] Yoo | Drone delivery: Factors affecting the public's attitude and intention to adopt[END_REF], the authors point out that drone delivery's acceptance by the public opinion is divided. From one side are those who see the drone as a faster and more environmentally friendly alternative, and on the other side who are reluctant and fearful of the risks. Furthermore, drones' illegal activities have been the most diverse, such as terrorism, espionage, drug trafficking, violation of privacy, and airport disruption [START_REF]Commercial UAV Market Analysis by Product (Fixed Wing, Rotary Blade, Nano, Hybrid)[END_REF]. For example, since 2016, at least ten serious worldwide incidents involving drones in international airports resulted in serious losses [129].

Although legislation has improved in most countries [START_REF]Master List of Drone Laws (Organized by State & Country)[END_REF], the authorities have faced enormous difficulties in reinforcing the law, i.e., to mitigate the illegal use of these machines [START_REF] Loffi | Examining Unmanned Aerial System Threats & Defenses: A Conceptual Analysis[END_REF].

Anti-drone solutions

In the military world, the combat against drones has been used for several decades [START_REF] John | Anti-UAV defense for ground forces and hypervelocity rocket lethality models[END_REF], and consolidated techniques have been deployed since then. Although the Air Forces' anti-AUV systems may not be directly applied to the mini-drone neutralization, as exposed in [START_REF] Peri | Expanding Anti-UAVs Market to Counter Drone Technology[END_REF], its extensive literature and know-how have been the starting point for much of the recent civil drone countermeasures. According to [START_REF] Ding | An Amateur Drone Surveillance System Based on the Cognitive Internet of Things[END_REF], most military techniques can be divided into the following classes: Jamming, Spoofing, and Mitigation.

While spoofing consists of inducing the drone a landing using false command signals, jamming, for its turn, disturbs the control with substantial signal interference, which can carry the drone to an emergency landing [START_REF] Ding | An Amateur Drone Surveillance System Based on the Cognitive Internet of Things[END_REF][START_REF] Peri | Expanding Anti-UAVs Market to Counter Drone Technology[END_REF]. However, these techniques can resent a relatively good efficiency [START_REF] Lefebvre | Conceptual design study of an Anti-Drone Drone[END_REF]; they present a drawback of the short-range of actuation since their deployment is done conventionally by fixed ground stations. Furthermore, still after [START_REF] Lefebvre | Conceptual design study of an Anti-Drone Drone[END_REF], interference techniques also present a relatively high cost, resulting in the demanded structure.

Work in progress as of 20 th June, 2021

The techniques of mitigation consist of destroying or intercept the invader physically [START_REF] Ding | An Amateur Drone Surveillance System Based on the Cognitive Internet of Things[END_REF][START_REF] Peri | Expanding Anti-UAVs Market to Counter Drone Technology[END_REF]. One emblematic example of mitigation was the "Anti-UAV Falconry" deployed by the Dutch National Police [START_REF] Slavimir | An innovative response to commercial UAV menace: Anti-UAV falconry[END_REF]. They studied the use of eagles and other prey birds to capture invasive drones. Evidently, this was a temporary exit, which carries numerous disadvantages, such as dependence on a bird's master, the animal's physical integrity, and its short-range actuation. Other examples of physical neutralization can be net launchers, water cannons, and projectile shooting [START_REF] Lefebvre | Conceptual design study of an Anti-Drone Drone[END_REF]129], which are generally deployed from the ground. Those techniques usually present the low cost of implementation but also have limited efficiency. Nevertheless, one particular case of mitigation, the laser weapons, such as the Boeing's Compact Laser Weapon System (CLWS), can offer excellent efficiency and a good range of actuation. But of course, all this in exchange for a very high installation cost.

The techniques presented so far do not have a viable cost or a good efficiency or operating range at once. Furthermore, all of these attempts are deficient in their potential collateral effects, i.e., the possibility of physical damage when the drone is neutralized. A drone landing forcibly or falling after an intercepting can cause harmful consequences in a civilian environment.

Anti-drone drone (ADD)

The anti-drone drone (ADD) is a recent technology that has stood out for its low cost, good efficiency, and potential for capturing the intruder and phase it out in a safe place. Furthermore, the ADD has an intrinsic advantage since it has the same "nature" of the threat, having essentially similar features. In this way, the countermeasure can evolve with similar speed to that of threat.

In recent literature, several works have been dedicated to ADD [3,[START_REF] Lefebvre | Conceptual design study of an Anti-Drone Drone[END_REF][START_REF] Hert | Autonomous Predictive Interception of a Flying Target by an Unmanned Aerial Vehicle[END_REF][START_REF] Thomas | Guidance Laws for Partially-Observable UAV Interception Based on Linear Covariance Analysis[END_REF][START_REF] Rothkrantz | SURVEILLANCE AND PROTECTION BY DRONES[END_REF]. In [START_REF] Lefebvre | Conceptual design study of an Anti-Drone Drone[END_REF], the author proposes a conceptual design of an ADD using a net launching system to neutralize the target. They considered the intruders' performance, the capturing system, and the missions' requirements in its design. In [3], [START_REF] Hert | Autonomous Predictive Interception of a Flying Target by an Unmanned Aerial Vehicle[END_REF], and [START_REF] Thomas | Guidance Laws for Partially-Observable UAV Interception Based on Linear Covariance Analysis[END_REF], the authors dedicated to the trajectory planning for an ADD. In [START_REF] Hert | Autonomous Predictive Interception of a Flying Target by an Unmanned Aerial Vehicle[END_REF], the author proposed an improvement in the model predictive control scheme for allowing aggressive trajectories.

Work in progress as of 20 th June, 2021

Moreover, in [3], the author chooses to compare missile guidance-based strategies with an optimal control planner. In [START_REF] Thomas | Guidance Laws for Partially-Observable UAV Interception Based on Linear Covariance Analysis[END_REF], the author proposed an optimal controller based on a pursuit-evasion game formulation. They concentrated on a case where the target is partially observable, and they investigate the state-estimation as well.

While in the last examples, the authors considered the quadcopter as ADD aircraft, in [START_REF] Rothkrantz | SURVEILLANCE AND PROTECTION BY DRONES[END_REF], the author considered fixed-wing models. In this work, ADD behavior was modeled after a human-based model on aircraft combat pilots' experience. Besides, several ADD solutions are already commercially available. Companies, such as Theiss [130] and Hertz [131], for example, provide similar ADD solutions based on the net launch system. Both solutions are no-destructive and offer a low risk of damage in the civil application. Although an essential advantage of the ADD is the possibility of capturing the intruder, other kinds of interception have also been considered. For example, several works [START_REF] Hert | Autonomous Predictive Interception of a Flying Target by an Unmanned Aerial Vehicle[END_REF][START_REF] Rothkrantz | SURVEILLANCE AND PROTECTION BY DRONES[END_REF] consider the kamikaze mode (or kinetic attack), where the pursuer attempts to crash into the target, disabling it. This attack can also be a backup mode in case of launch net failure. Another example is the American startup Anduril, which proposed an anti-drone system in which the kinetic attack is the means interception. Another possibility of neutralization is the use of interference guns on the drone. The Boreades project [134], proposes a counter UAV system where the ADD uses jamming guns to neutralize the intruder.

Work in progress as of 20 th June, 2021

Until now, it has been evident the benefits of using the ADD in the counterdrone fight in civilian environments. However, since the ADD shares the same "nature" as its prey, i.e., they have virtually similar capabilities, the capture task could not be easy for the pursuer. The same technology that can allow autonomous pursuit can also favor autonomous evasion, making the one-vs-one pursuit a challenging task for the ADD. Nevertheless, to overcome the problem of a faster or stronger opponent, one recurrent solution observed in nature is cooperation, the use of a group instead of the single. In fact, group hunting increases the success rate of predators, as exposed in [START_REF] Philip | Cooperative hunting in lions: the role of the individual[END_REF], and [START_REF] Muro | Wolf-pack (Canis lupus) hunting strategies emerge from simple rules in computational simulations[END_REF]. In a multiagent system, the perception and the actuation area is significantly augmented, which can allow even the capture faster or more agile prey [START_REF] Janosov | Group chasing tactics: how to catch a faster prey[END_REF].

Collective motion

Collective behavior has been observed as an evolutionary advantage in many natural systems. Throughout the animal kingdom is evident how the group phenomena -as flocking, swarm, and pack -increases the rate of success of an individual living being significantly [START_REF] Zafeiris | Why We Live in Hierarchies?: A Quantitative Treatise[END_REF]. Even for the modern and corporate human being, teamwork is an appreciated characteristic of an individual. In short, humanity always new about the power of the collective, as the Aristotelian maxim said: "the whole is greater than the sum of its parts."

Bio-inspiration

Understanding and modeling the complex behaviors observed in nature has been the primary motivation for many pioneers in multi-agent systems. Reynolds' seminal work [START_REF] Craig | Flocks, herds and schools: A distributed behavioral model[END_REF] aimed to graphically reproduce the behavior observed by flocks of birds and schools of fish. His studies observed that specific "laws" of behavior at the individual level, considering only local observation, could reproduce a verisimilar group behavior observed in natural systems. Namely, the three rules are: separation -a short-range repulsion to avoid crowded neighbors. Alignment -the tendency to align itself with its neighbors' average heading. And cohesion -long-range attraction towards the center of the flock.

Work in progress as of 20 th June, 2021 Universality of flocking Later, with the advancement of data acquisition techniques, observation studies found the flocking property in the most diverse groups of individuals, from bacteria, through insects to human groups on vessels, as evidenced in the Vicsek and Zafeiris' survey [START_REF] Vicsek | Collective motion[END_REF]. Individuals' tendency to collectively align their directions seems to indicate a certain universality of the collective motion. From Vicsek's seminal work [START_REF] Vicsek | Novel type of phase transition in a system of selfdriven particles[END_REF], the physical community would explain this universal behavior as a result of the interaction of self-propelled particles. This collective movement, where individuals tend to align themselves fluidly, even in the absence of a leader or centralized coordination, is also called as flocking.

Models for flocking behavior are numerous in the literature, and they range from the simplest ones, such as primary Vicsek particle [START_REF] Vicsek | Novel type of phase transition in a system of selfdriven particles[END_REF], passing through deterministic models, such as Cucker-Smale [START_REF] Cucker | Emergent behavior in flocks[END_REF] and its numerous variation, and control-theoretical frame-work, such as in [START_REF] Olfati-Saber | Flocking for multi-agent dynamic systems: Algorithms and theory[END_REF].

Flocking and hunting Although flocking behavior is present in practically all living beings moving in group [START_REF] Vicsek | Collective motion[END_REF], flocking models are not sufficient to describe all collective behavior types. Group hunting, for example, is a complex phenomenon that is not fundamentally described by flocking models. The description of lionesses' collectively hunting, carried out by Stander [START_REF] Philip | Cooperative hunting in lions: the role of the individual[END_REF], indicates the lionesses' previous tendency to disperse and then ambush the prey. Disper-Work in progress as of 20 th June, 2021 sion behavior is contrary to the aggregation trend described in most flocking models. This difference is evident in the work of Janosov [START_REF] Janosov | Group chasing tactics: how to catch a faster prey[END_REF]; the authors proposed the addition of a dispersion term into the conventional flocking model [START_REF] Virágh | Flocking algorithm for autonomous flying robots[END_REF][START_REF] Vásárhelyi | Outdoor flocking and formation flight with autonomous aerial robots[END_REF] to obtain a more realistic hunting modelization.

Furthermore, Reynold's alignment rule, with its correspondent in the field of statistical physics, the viscosity term, acts as a force against a reactive hunting movement. The alignment makes the direction of a stalker tend towards its neighbors' average direction, which may be contradictory with the tendency of a hunter who orientates above all with respect to his prey. Some recent multi-agents approaches have tried to model group hunting using principles of collective motion. Later in this chapter, we will make a brief review of the emerging field of group chase-evasion.

Collective robots

It is possible to highlight several similarities between group animals and collective robots, such as limited sensing and communication and decentralized decision-make. These similarities explain why flocking models have been widely integrated into robots, as proposed in [START_REF] Virágh | Flocking algorithm for autonomous flying robots[END_REF][START_REF] Hauert | Reynolds flocking in reality with fixed-wing robots: communication range vs. maximum turning rate[END_REF][START_REF] Ferrante | Flocking in stationary and non-stationary environments: a novel communication strategy for heading alignment[END_REF] and in many other works [START_REF] Brambilla | Swarm robotics: a review from the swarm engineering perspective[END_REF].

However, as expressed in Viragh [START_REF] Virágh | Flocking algorithm for autonomous flying robots[END_REF], real-life conditions, such as delays, communication failures, and uncertainty, can cause instability among agents. Consecutively, the design of multi-robots must go beyond the theoretical flocking model, and the physical characteristics of the robot and its sensors must be taken into account. Therefore, multi-robots can be seen as an extension of the classic models of group movement models, as described in the previous section, but taking into account the robots' dynamics and observation.

A vast literature collective robot (or swarm of robots) has been formed in the last two decades, and several efforts have been applied to organize and classify them, such as in [START_REF] Brambilla | Swarm robotics: a review from the swarm engineering perspective[END_REF][START_REF] Oh | A survey of multi-agent formation control[END_REF]. For the sake of clarity, we consider henceforth a classification similar to one proposed in [START_REF] Brambilla | Swarm robotics: a review from the swarm engineering perspective[END_REF], which classifies the systems of multi robots based on their emergent behavior. For example, as a collective behavior, Work in progress as of 20 th June, 2021 we can highlight the following group tasks: reaching consensus, coordinated motion (flocking), or pattern formation. Next, we will review two of the most popular behaviors applied to aerial robots and their application to group chase.

Pattern Formation

In the field of aerial robotics, impressive results have been achieved in fixed and dynamic pattern formation, such as described in Figure 1.5. Autonomous formation control has been extensively addressed in the control and robotics literature. This approach aims to create geometrical patterns with the agents by controlling their relative positioning.

The main control structures used to achieve pattern formation are namely: Leader-follower [START_REF] Jose | Mini rotorcraft flight formation control using bounded inputs[END_REF][START_REF] Jossué | Cooperative control for load transportation using two PVTOL vehicles with a passivity approach[END_REF][START_REF] Hou | Distributed leader-follower formation control for multiple quadrotors with weighted topology To cite this version : Distributed Leader-Follower Formation Control for Multiple Quadrotors with Weighted Topology[END_REF], Virtual Structure [START_REF] Balázs | Adaptive leadership overcomes persistence-responsivity trade-off in flocking[END_REF][START_REF] Kushleyev | Towards a swarm of agile micro quadrotors[END_REF] and Behavioral-Based [START_REF] Olfati-Saber | Flocking for multi-agent dynamic systems: Algorithms and theory[END_REF][START_REF] Saif | Real-time flocking of multiple-quadrotor system of systems[END_REF]. Although there are huge differences between the architectures, in this work, we will stick only to their final results, which are the formation of patterns themselves. Here, we are interested in whether the formation of patterns is interesting or not for group persecution. Figure 1.5 -Examples of real-time applications of pattern formation with quadcopters. In the left picture, one of the earliest works with dynamic pattern formation [START_REF] Kushleyev | Towards a swarm of agile micro quadrotors[END_REF]. In the right picture, picture of the Intel Drone Light show at the Winter Olympics of 2018 [START_REF]Intel Drone Light Show at The Olympics[END_REF].

Several attempts have been made to apply pattern-formation techniques to the group chase problem. In one of the earliest works [START_REF] Yamaguchi | A cooperative hunting behavior by multiple nonholonomic mobile robots[END_REF], the author proposed a feed-back control to regulate a non-holonomic pursuer's position in a "stalking" formation around the target. Each agent's behavior was defined by a "formation vector," and the resulting pattern was a half-circle around the target. In [START_REF] Weitzenfeld | A biologicallyinspired wolf pack multiple robot hunting model[END_REF], the Work in progress as of 20 th June, 2021 author proposed a bio-inspired wolf-pack's model for a multi-robot application. This proposition involves a hierarchical system, and a finite state machine determines the robot's motion; nevertheless, the system relies on a fixed formation and fixed roles.

Moreover, other commons approaches, such as in [START_REF] Gong | A GA based combinatorial auction algorithm for multi-robot cooperative hunting[END_REF][START_REF] Wang | A rapid hunting algorithm for multi mobile robots system[END_REF], the encirclement is achieved by each pursuer tracking a fixed point around the "virtual target." In this case, the problem becomes a collision free trajectory generation problem, which has been solved by genetic algorithms [START_REF] Gong | A GA based combinatorial auction algorithm for multi-robot cooperative hunting[END_REF] or virtual ranges [START_REF] Wang | A rapid hunting algorithm for multi mobile robots system[END_REF].

Although the previous approaches could provide an encircling formation around the target, they did not consider the target dynamics. In [START_REF] Wu | A multi-robot cooperative hunting approach based on dynamic prediction of target motion[END_REF][START_REF] Duan | Multi-robot dynamic virtual potential point hunting strategy based on FIS[END_REF], the target movement is considered, and the pursuit is made toward a predicted position target. Nevertheless, both techniques still rely on virtual points around the predicted position of the target. Once more, the problem becomes the generation of trajectories toward a virtual position. In a most recent work [START_REF] Kim | Three-dimensional discrete-time controller to intercept a targeted UAV using a capture net towed by multiple aerial robots[END_REF], the author proposed a motion controller for a group of drones towing a capture net to intercept an intruder UAV. Here, since the net physically links the agents, the maintenance of the fixed pattern is needed.

In short, the use of pattern formation in the pursuit, although it can bring some advantages to groups, such as increased perception, robust and acted area, is still a very rudimentary technique of pursuit, which reduces the flexibility of the group in a more challenging pursuit. The formation can be seen as a single body, and the most advanced techniques of ambush and stalking require more than the maintenance of the formation. Furthermore, most of the above-cited techniques require knowledge of the target and neighbors position in the inertial frame, which requires communication between all the pursuit actors.

Work in progress as of 20 th June, 2021

Flocking of robots

Flocking is another prevalent collective behavior for aerial robots. Here, we consider multi-robots techniques which are based or similar to the "flocking" models seen previously, such as Reynolds [START_REF] Craig | Flocks, herds and schools: A distributed behavioral model[END_REF] and Vicsek [START_REF] Vicsek | Novel type of phase transition in a system of selfdriven particles[END_REF] models. We suppose that the agents do not follow a fixed path or relative positioning in a pattern formation; instead, they are independent members of a cohesive group, sensing and acting locally and based on simple rules. As pointed out by [START_REF] Brambilla | Swarm robotics: a review from the swarm engineering perspective[END_REF], flocking behaviors with multi-robots are generally designed based on virtual physics, where virtual forces of attraction and repulsion ensure the distance between agents and the alignment ensures the coherence of the movement.

One of the first successful attempts to reproduce Reynolds rules with aerial robots is given in [START_REF] Hauert | Reynolds flocking in reality with fixed-wing robots: communication range vs. maximum turning rate[END_REF]. In this work, ten fixed wings UAVs were deployed, and they focused on analyzing the effects of communication range and maximum turning rate in the emergent formation. Other impressive examples of UAV's flock are given in [START_REF] Virágh | Flocking algorithm for autonomous flying robots[END_REF][START_REF] Vásárhelyi | Outdoor flocking and formation flight with autonomous aerial robots[END_REF][START_REF] Balázs | Adaptive leadership overcomes persistence-responsivity trade-off in flocking[END_REF][START_REF] Vásárhelyi | Optimized flocking of autonomous drones in confined environments[END_REF]. The authors deployed a fleet of over large numbers of quadcopter flights in an entirely autonomous and decentralized manner in those works. In those works, the flocking algorithm is based on the in the Vicsek model [START_REF] Vicsek | Novel type of phase transition in a system of selfdriven particles[END_REF]. Pictures of the real-time experiment can be seen in Figure 1.6.

For the best of our knowledge, no flocking based strategies applied to group Work in progress as of 20 th June, 2021 pursuit have been implemented in real-time robots. Nevertheless, several approaches have been made in simulations, such as [START_REF] Muro | Wolf-pack (Canis lupus) hunting strategies emerge from simple rules in computational simulations[END_REF][START_REF] Lin | Bat swarming as an inspiration for multi-agent systems : predation success , active sensing , and collision avoidance[END_REF][START_REF] Saito | Group chase and escape model with chasers' interaction[END_REF][START_REF] Janosov | Group chasing tactics: how to catch a faster prey[END_REF]. Such approaches will be better described in subsection Pursuit-evasion.

Pursuit evasion

Similar to flocking, pursuit-evasion is also a very popular phenomenon in nature, which can be easily observed in several biological activities, such as foraging, searching for partners, and disputing territories [START_REF] Nahin | Chases and Escapes The Mathematic[END_REF]. As we will see below, persecution has a long mathematical tradition; however, the recent computational advances have allowed a new approach to the problem, allowing more complex ones, with large numbers of persecutors and evaders.

Single Pursuit-evasion

Pursuit's curbs Although the problem stated dates from farther away, the first successful attempt to describe the trajectories of pursuit is attributed to Pierre Bourger, in Les courbes de Poursuite (1732) [START_REF] Shneydor | Missile Guidance and Pursuit: Kinematics, Dynamics and Control[END_REF][START_REF] Nahin | Chases and Escapes The Mathematic[END_REF][START_REF] Kamimura | Group Chase and Escape: Fusion of Pursuits-Escapes and Collective Motions[END_REF]. Although the problem can be stated simply, closed-form analytical solutions are hard to obtain, even using simplistic assumptions such as constant velocities and linear trajectories. For example, a generalized extension of the closed-form solution of Bourger was only obtained late in 1991, as exposed in [START_REF] Kamimura | Group Chase and Escape: Fusion of Pursuits-Escapes and Collective Motions[END_REF].

Differential game This approach formulates the pursuit-evasion as a two players game. The players (decision-makers) must maximize or minimize an objective function, which often is the duration of the episodes. Such formulations have been extensively studied in modern robotics problems. In [START_REF] Chung | Search and pursuit-evasion in mobile robotics[END_REF], the authors made an interesting survey highlighting game-based approaches applied to robotics; consequently, the main classes of techniques were Pursuit-Evasion Games and Probabilistic Search. The authors remarked a tendency to use the second one in adversarial games, where the optimization expects to maximize the evader detection probability. In [START_REF] Vidal | Probabilistic Pursuit -Evasion Games : Theory , Implementation , and Experimental Evaluation[END_REF], the authors implemented this technique in a heterogeneous team with ground and aerial vehicles. They demonstrated Work in progress as of 20 th June, 2021 [START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF]. Right: Example of the generalized closed-form solution for a 2D pursuit, exposed in [START_REF] Kamimura | Group Chase and Escape: Fusion of Pursuits-Escapes and Collective Motions[END_REF] empirically that their method could locate a target but with no guaranteed capture.

Others work using pursuit-evasion games in complex environments are available in [START_REF] Pan | Pursuit , Evasion and Defense in the Plane[END_REF], [START_REF] Wilson | Contributions to Pursuit-Evasion Game Theory[END_REF], and [START_REF] Pierson | Cooperative multi-quadrotor pursuit of an evader in an environment with no-fly zones[END_REF]. In those works, we can find the following configuration consecutively: convex polygons arenas, with obstacles and no-fling zones.

The multi-pursuer single-evader game is also a scenario extensively studied in the literature. In [START_REF] Huang | Guaranteed decentralized pursuit-evasion in the plane with multiple pursuers[END_REF], the authors used a differential game formulation for multi pursuers chasing a single evader. In their proposition, each agent minimizes the evader's action area, characterized by the Voronoi cells, similarly, in [START_REF] Li | Coordinated multi-robot target hunting based on extended cooperative game[END_REF], a cost-function is proposed and computed based on the relative angle and distance between agents and targets, seeking pursuit and avoid a collision.

Although game-based methodologies guarantee the optimality input for pursuit, the large amount of constraint and precise knowledge about the whole system makes it, for instance, impracticable for real robots application. Moreover, it becomes more challenging to define an appropriate objective function with increasing problem complexity, such as a larger number of agents or motion constraints. Finally, as exposed in [START_REF] Janosov | Group chasing tactics: how to catch a faster prey[END_REF], the apparatus of the differential Work in progress as of 20 th June, 2021 game formulation requires a lot o "unrealistic and biologically questionable assumptions."

Group pursuit

The emerging "group chase escape" analysis has raised a recent early interest in the physics and biology communities. This field aims to describe prey-predators' complex behavior considering agent-based models, such as in [START_REF] Angelani | Collective predation and escape strategies[END_REF][START_REF] Saito | Group chase and escape model with chasers' interaction[END_REF][START_REF] Janosov | Group chasing tactics: how to catch a faster prey[END_REF]. Moreover, as pointed by [START_REF] Kamimura | Group Chase and Escape[END_REF], this recent and opened field inherits from two consolidated literature: pursuit-evasion and collective motion.

Firstly, it can be seen as an extension of the single pursuit-evasion problem, taking advantage of an extensive literature involving: analytical solutions of pursuit, differential games, and missiles guidance studies. Secondly, it can be considered an extension of the collective motion, taking advantage of the extensive literature in self-propelled particles, such as [START_REF] Vicsek | Novel type of phase transition in a system of selfdriven particles[END_REF], which has been a powerful tool model complex systems, such as bacteria, insect, and animals [START_REF] Vicsek | Collective motion[END_REF].

The first proposal to bring together chase-evasion and multi-particle systems is possibly from Kamikura et al. [START_REF] Kamimura | Effective construction of linear statevariable models from input/output functions[END_REF]. In this work, the authors established a multi-agent pursuit-evasion problem running in a grid world with periodic boundary conditions. The pursuers, which have no interactions between themselves, had to capture the nearest evaders, while the evader must escape from the closest pursuer. Despite the simplicity of the model, interesting behaviors emerged, raising the scientific community's interest.

Although those earlier works, such as [START_REF] Saito | Group chase and escape model with chasers' interaction[END_REF][START_REF] Vicsek | Closing in on evaders[END_REF][START_REF] Kamimura | Effective construction of linear statevariable models from input/output functions[END_REF], allowed the first analysis of emergent behavior, such as segregation and encircling, they had elementary assumptions, which kept it far from real-time applications. Nevertheless, since then, approaches with more realistic simulations, such as continuous and boundary world representation, were proposed [START_REF] Muro | Wolf-pack (Canis lupus) hunting strategies emerge from simple rules in computational simulations[END_REF][START_REF] Angelani | Collective predation and escape strategies[END_REF][START_REF] Janosov | Group chasing tactics: how to catch a faster prey[END_REF], and the group-chase started to become closer to robotic applications.

In [START_REF] Muro | Wolf-pack (Canis lupus) hunting strategies emerge from simple rules in computational simulations[END_REF], the author implemented an agent-based approach, based on simple rules, to mimic a wolf-pack hunting strategy. The authors pointed out that there was no need for communication to mimic encircling behavior. They showed Work in progress as of 20 th June, 2021 Figure 1.8 -Group-chase algorithms and its bio-inspiration. In A a screenshot from [START_REF] Vicsek | Closing in on evaders[END_REF], it is illustrating the initial positions of pursuers (wolves) and evaders (deers) using the algorithm proposed by [START_REF] Kamimura | Effective construction of linear statevariable models from input/output functions[END_REF]. B illustrates the motivation of the previous: modeling the group chase evasion natural phenomenon. In C a screenshot from [START_REF] Muro | Wolf-pack (Canis lupus) hunting strategies emerge from simple rules in computational simulations[END_REF], where the authors proposed to mimic a wolf-pack's behavior, as illustrated in D.

that simple rules of attraction toward the target and regulating distance between pursuer were enough to reproduce this performance. However, their algorithm does not describe the complexity of a cooperated pursuit of faster prey, which requires more sophisticated techniques to stalk and ambush the prey. In another agent-based approach, [START_REF] Angelani | Collective predation and escape strategies[END_REF] constructed the model based on the Vicsek particle, and [START_REF] Lin | Bat swarming as an inspiration for multi-agent systems : predation success , active sensing , and collision avoidance[END_REF] modeled a similar multi-agent hunting system but in a three-dimensional environment.

Recent work in escape from group chase [START_REF] Janosov | Group chasing tactics: how to catch a faster prey[END_REF] extended the previous particlebased model [START_REF] Angelani | Collective predation and escape strategies[END_REF] to more realistic simulations, where delays, inertial effects, and three-dimensional work-space were considered. In this work, the pursuer behavior is calculated by the sum of the virtual interactions: chase (attraction to the prey), inter-agent repulsion, alignment, and short-range collision-avoidance.

Work in progress as of 20 th June, 2021

Furthermore, they considered the prediction of prey's future position to improve the verisimilarly of the hunting.

Although impressive behaviors can be exhibited in the simulations, their model considered many tunable parameters, which increased the difficulty of tunning and required optimization. Furthermore, its applicability in the real world is limited due to its assumptions in the agents' observation; it relies on the knowledge of position and velocities of pursuers and targets expressed in the global frame. Although these assumptions can be partially afforded by considering communications between pursuers, the evader is not cooperative in real-world applications; consecutively, its global position and velocity usually are not available for the pursuer.

Guidance Laws (GL)

Guidance is a crucial part of any autonomous vehicle system. As once defined by Shneydor [START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF], guidance can be said: "the process for guiding the path of an object towards a given point, which in general may be moving". Considering a complex robotic complex system, which involves several layers in the motion control, the guidance layer can be considered high-level control, which indicates the direction to be taken.

Here, "guidance laws" refer to feedback algorithms in which a particular geometrical rule is implemented. The geometrical rule is the essence of GL and expresses how the pursuer must behave in geometrical terms. For example: "head the vehicle towards the target's current position," which is the rule behind the Pure Pursuit (PP) law. Since GL is an essential concept in this thesis's propositions, we will give more detail in the next subsections.

GL in robotics

As pointed out by [START_REF] Breivik | Guidance laws for planar motion control[END_REF], possibly the richest literature in guidance can be attributed to the missiles' community. This fact is not strange considering that Work in progress as of 20 th June, 2021 Figure 1.9 -Guidance laws have been widely applied in guided missiles. Left: the German TV guided bomb Hs-293D, considered the first PP-bomb. According to [START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF], an operator transmits commands to the bomb according to its location on the screen. Right: Guided Missiles R-27T1, an example of modern air-to-air missiles, which implements PN based guidance law [START_REF]Guided Missiles R-27T1 / R-27ET1[END_REF] missiles were one of the first autonomous vehicles and, in a way, even the predecessor of the current UAV [START_REF] John | A brief history of early unmanned aircraft[END_REF].

GL for ground robots Considering ground robot applications, GL's have been extensively explored, see [START_REF] Belkhouche | A method for robot navigation toward a moving goal with unknown maneuvers[END_REF][START_REF] Belkhouche | Parallel navigation for reaching a moving goal by a mobile robot[END_REF][START_REF] Teimoori | A biologically inspired method for robot navigation in a cluttered environment[END_REF]. [START_REF] Belkhouche | A method for robot navigation toward a moving goal with unknown maneuvers[END_REF] investigated GLs to guide a single robot towards an evader in an environment with obstacles. They applied two classical guidance approaches: Velocity Pursuit Guidance (VPG) and the Deviated Pure Pursuit (DPP). In [START_REF] Belkhouche | Parallel navigation for reaching a moving goal by a mobile robot[END_REF], the authors implemented a feedback controller based on Proportional Navigation Guidance (PNG). Similarly, in [START_REF] Teimoori | A biologically inspired method for robot navigation in a cluttered environment[END_REF], the authors applied a sliding-mode based control to keep a constant line-of-sight, which is the geometrical rule for the PNG.

GL for UAVs Classic guidance laws application stills pique interests on the scientific community, and it has been widely used in aerial robotics, such as in [START_REF] Jung | Perception , Guidance , and Navigation for Indoor[END_REF]3,[START_REF] Thomas | Guidance Laws for Partially-Observable UAV Interception Based on Linear Covariance Analysis[END_REF][START_REF] Tan | Proportional navigation (PN) based tracking of ground targets by quadrotor UAVs[END_REF]. In [3], the author applied PP and PNG guidance algorithms into quadcopters for target interception. They compared GLs against optimal control trajectory planners, analyzing the time and energy required to achieve the task. Nevertheless, results are only presented in simulations. In [START_REF] Thomas | Guidance Laws for Partially-Observable UAV Interception Based on Linear Covariance Analysis[END_REF], the author analyses three missiles guidance laws applied to interception trajectory in the case of a partially-observable target. They formulate a pursuit-evasion game to determine the guarantees of capture under evasive maneuvers, determining an optimal controller.

Work in progress as of 20 th June, 2021

In [START_REF] Tan | Proportional navigation (PN) based tracking of ground targets by quadrotor UAVs[END_REF], the authors used the PNG scheme in a drone landing on a moving platform. Its final solution mixed PNG with PD -proportional derivativealgorithms, and it showed effectively and carried out less oscillation and small errors compared to the traditional tracking PD. Moreover, the PNG has also been used as guidance law in an autonomous drone race in [START_REF] Jung | Perception , Guidance , and Navigation for Indoor[END_REF]. In that case, the authors adapted the PNG algorithm to handle the quadrotor's decoupled dynamics between roll and yaw to follow visually desired trajectories in a drone race.

Relative engagement

The analysis of the GL is based on the relative kinematic between pursuer and target. This kinematics equation can be easily deduced from the geometrical engagement of pursuer-target, as represented in Figure 1.10. From Figure 1.10, the straight line, also called Line-of-Sight (LOS), connecting them is denoted by r iT . The bearing angle, formed by the LOS and the inertial axis-x, is denoted here by λ iT . The vector v i is the straightforward velocity in the body frame, and ψ i represents the heading (yaw) of the pursuer i.

As explicated in [START_REF] Belkhouche | Parallel navigation for reaching a moving goal by a mobile robot[END_REF][START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF], from this engagement a relative kinematic model can be obtained. For that, lets consider the relative velocity between, ṙiT = Ṗ T -Ṗi . The relative velocity can be decomposed along (v ) and perpendicular (v ⊥ ) to the LOS :

v = ṙiT = v T * cos(α T ) -v i * cos(α i ), v ⊥ = r iT * λiT = v T * sin(α T ) -v i * sin(α i ). (1.1)
Work in progress as of 20 th June, 2021

where α T = ψ iλ iT and α i = ψ Tλ iT . In other words, with this equation we have represented the velocities of target seem by the pursuer along the the LOS.

These equations will be useful to determine conditions of convergence, or capture, for the proposed pursuit strategies.

Pure Pursuit (PP)

In the most intuitive guidance law, the Pure Pursuit law, the pursuer aims to align its heading (or velocity vector) towards the present location of the target (ψ i → λ iT ). Therefore, assuming this perfect alignment (ψ i = λ iT ) the relative kinematics equations 1.1 become:

ṙiT = v T * cos α T -v i , r iT * λiT = v T * sin α T . (1.2)
Note that, a sufficient condition for the capture is that v i > v p , once this condition assures ṙiT < 0, for any values of α T . This statement evidences the limitation of the use of a single pursuer against a faster target. However, this problem will be better discussed in Chapter 5.

Finally, considering the kinematic model of an agent (3.16), an intuitive law to apply the PP (ψ i → λ iT ) is:

f ψ i = -K pp * (ψ i -λ iT ) (1.3)
where K p defines a positive constant gain.

Work in progress as of 20 th June, 2021

1.4.4 Pure Deviated Pursuit -DPP:

As stated by Shneydor1998, the DPP can be seen as a variation of the classical PP, it can be a result of manufacturing default or can be intentionally designed in order to point the missile in position ahead of the target, causing a "lead pursuit." However, if the offset drives the motile towards a position behind the target, it causes a "lag pursuit."

Its basic principle relies in driving the heading error not to zero by to a constant (ψ i -λ iT → α 0 ). Once considering this principle, the relative kinematics become:

ṙiT = v T * cos(α T ) -v i * cos(α 0 ), r iT * λiT = v T * sin(α T ) -v i * sin(α 0 ).
(1.4)

From the above equation, we can established one necessary condition about the offset angle: |α 0 | < π/2. Nevertheless, under this strategy the capture cannot be assured for all v i > v p .

The further interest in applying this technique is not to choose a α 0 that optimizes the pursuer displacement. Instead, the idea is that each agent assumes different offsets, consecutively, having different degrees of lead, and lag pursuit, spreading the pursuers around the target. In Chapter 2, one group pursuit strategy will be established based on this technique.

Finally, as exposed in [START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF], an intuitive law to apply the DPP (ψ i → λ iT ) is:

f ψ i = -K dpp * (ψ i -λ iT -α 0 ) (1.5)
where K dpp defines a positive constant gain.

Proportional Navigation Guidance (PNG)

The Proportional Navigation is maybe the most popular guidance law for missiles. Its basic principle is called Parallel Navigation or constant bearing rule. It consists of keeping a constant bearing angle (λ iT ) with respect to the target, and since the LOS rate ( ṙit ) is decreasing, both elements are in the collision route.

Work in progress as of 20 th June, 2021

The same principle was observed in animals hunt, as the reputed faster predator, Falcon peregrine, cited in [START_REF] Caroline H Brighton | Terminal attack trajectories of peregrine falcons are described by the proportional navigation guidance law of missiles[END_REF], and also in insects, as described in [START_REF] Samuel T Fabian | Interception by two predatory fly species is explained by a proportional navigation feedback controller[END_REF], where miniature brains predators execute fast maneuvers to catch its prey.

As exposed by Shneydor in [START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF], for the planar case, the parallel navigation rule can be stated as: λiT (t) = 0, ṙiT (t) < 0.

(1.6)

The above equations are also known as "straight-line collision course conditions". Applying this principle (1.6), to the relative kinetic model (1.1), the following relation can be obtained:

v T * sin(α T ) = v i * sin(α i ), v T * cos(α T ) < v i * cos(α i ).
(1.7)

If the above conditions are satisfied and the velocities remain constant, the straight-line collision course conditions prevail, and both bodies will converge.

We can see from (1.7) that, contrary to the Pure Pursuit (1.3), solutions can exist even for v T > v i . In other words, depending on the initial conditions of the engaged, a faster target can still be in a collision course with a slower pursuer. This property will be explored in Chapter 2, where the faster evader case will be studied.

Finally, a logical choice for a controller to implement the PNG can be defined as:

f ψ i = N * λiT (1.8)
where N is a positive constant called the navigation gain.

Conclusion

Nowadays, there is a consistent threat by the unlawful use of civilian drones. The classic counter-UAV solutions have proven to be insufficient, and there is a growing trend in the use of anti-drone drones (ADD). However, in a futuristic

Work in progress as of 20 th June, 2021 but next scenario, with autonomous intruder drones, the use of a single ADD may be insufficient, as the intruder may have similar capabilities to the ADD. In this case, it would be relevant to use a group ADD to cooperate and ambush the intruder, similarly as predators in nature do. However, coordinating robots for a chase is not an easy task. Although group robots are becoming popular and extensive literature exists, the existing multi-robots techniques are not appropriate for group pursuit problem. On the contrary, most of the existing techniques are in charge of pattern formations or flocking navigation, which is very different from a cooperative hunt.

On the other hand, the study of pursuit-evasion has been a topic widely addressed and implemented in the military community. Although the existing single-pursuit analysis can serve as insights for group-pursuit, the large degree of freedom of this second makes the problem practically unfeasible to be solved analytically with realistic conditions. In this context, group chase-evasion arises, which uses multi-particle simulation techniques to analyze complex collective motions.

Finally, our work comes in the context of group chase-evasion strategies and aims to contribute to this field with an approach more adapted to aerial robots' real applications.

Work in progress as of 20 th June, 2021 "But ask the animals, and they will teach you, or the birds in the sky, and they will tell you; Job Chapter 2

Pursuit in the horizontal plan

In Chapter 2, we study the scenario where a group of pursuer and a single evader are traveling in a bounded 2-dimensional workspace. To solve the capture problem, we propose three distinct behavioral-based strategies for the pursuer team. The first one is based on the Deviated Pure Pursuit guidance, the DPP, and the different offset values give, for each pursuer, different behavior in the stalking. The second strategy is an improvement of the previous one, where an Parallel Navigation term is added to the DPP. This allows the pursuer to take more efficient paths toward the target and still deviate from each other, avoiding collisions. Finally, we propose a Deep Reinforcement Learning framework to obtain close-to-optimal policies for the pursuers.

Problem statement

Let us consider multi-agents pursuit-evasion problem involving N pursuers and a single evader moving in a horizontal plane. The pursuit takes place inside a bounded work-space W in IR 2 . The final goal for the pursuer team is r iT < R c , i.e. that at least one of the pursuers has a relative distance (r iT ) towards the target inferior than a threshold R c . The evader paths are described by P T (t) = [x T (t), y T (t)], where (x T , y T ) is its cartesian coordinates in W . Similarly, the pursuers path are described as P i (t) = [x i (t), y i (t)], where i 1, 2, ...N .

Figure 2.1 -Group pursuit in a bounded work-place. The pursuer (blue circles) attemps to capture the evader (red circle). Furthermore, it is in evidence the relative engagement between the pursuer i and one neighbors j and the target T .

Pursuer and target are moving in the work-space with a linear velocity (v i ) and heading (ψ i ). The motion model can be given by:

ẋi = v i cos ψ i , ẏi = v i sin ψ i , v i = f v i , ψi = f ψi (2.1) which f i = [f v i , f ψi ]
T the control vector of the particle. The turn-rate and velocity control are saturated, being f v i < v max and f v i > 0 and |f ψi | < ω max .

Work in progress as of 20 th June, 2021

Note that the non-holonomic equations (2.1) describe only the agents used in simulation; the quad-copter dynamics under the pursuit will be discussed further chapters. Nevertheless, it is worthy to lies some advantages of choosing this kinematic model for pursuer agents:

• It is a model compatible for a large number of vehicles, such as car-like, airplanes, etc.

• Most of the works dealing with the pursuit problem rely on this hypothesis, see Chapter 1.

• Even for holonomic robots, which is the case of the quadcopter, these constraints can be useful to conciliate the perceptions constraints (senseand-go), see in Chapter 4.

The assumptions of v max and ω max will vary along with this thesis, and it will be evidenced at the beginning of each experiment. In a similar way, the number of pursuers N , the initial positions, and the shape of W can vary along with this document.

Perception

Limited sensing is a common feature between multi-agents solutions [START_REF] Angelani | Collective predation and escape strategies[END_REF][START_REF] Craig | Flocks, herds and schools: A distributed behavioral model[END_REF][START_REF] Vicsek | Collective motion[END_REF]. Nevertheless, in this work, we decided to consider the perception of the environment (neighbors, target, obstacle, and boarders) given in relative and polar polar coordinates, as illustrated in Figure 2.1. This assumption is justified mainly for two reasons:

• Considering the robotics application, relative and polar coordinates are an expected output from commons embedded sensors, such as lidar, sonar, and camera.

• Most of the pursuit-evasion analysis are done based on this coordinate system; thereby, we could take advantage of its extensive literature.

Many works on drone's pursuit, such as [3,[START_REF] Hert | Autonomous Predictive Interception of a Flying Target by an Unmanned Aerial Vehicle[END_REF], or on navigation in a complex environment, such as [START_REF] Jung | Perception , Guidance , and Navigation for Indoor[END_REF], rely mainly on the frontal camera for the environment sensing.

Work in progress as of 20 th June, 2021

Target behavior

The ultimate advantage of using multi-agents in pursuit is the possibility of a cooperative ambushing for intercepting a more agile target. The term "agile" can mean having superior velocity or having more degrees of freedom. Along with this thesis, both of these hypotheses will be assumed.

In the following, the intruder will be up to twice as fast as the pursuers. Furthermore, the intruder will not be constrained to non-holonomic equations 2.1, but by a simple particle model, which gives him locomotion advantages. This configuration is a very known problem in differential games, and it is called "the homicidal chauffeur," [START_REF] Isaacs | Differential Games[END_REF].

Two strategies are assumed in most of the paper regarding the target behavior: predefined trajectories and reactive evader. First, the trajectories can be off-line defined; for example, by collecting real-data flights, or functions or algorithms can give it as circular or square trajectories. In the second, the reactive behavior is defined by a repulsion model. Each pursuer and the arena border exerts a repulsive force towards the evader. The forces decrease proportionally to the distance squared. The instant target's velocity is calculated as follow :

v i = j P j -P T r 2 jT + r arena × (sin(γ), cos(γ)) -pos d 2 wall (2.2)
where γ is the direction of the agent to the closest point on the wall.

Work in progress as of 20 th June, 2021

Group Deviated Pursuit

This section presents our first attempts to model the collective hunting phenomenon using Navigation Guidance Laws (NGL). This simplistic strategy bases on the Deviated Pure Pursuit law (DPP), and it is inspired by the behavior of a group of lionesses hunting.

The content of this section is currently under the second revision for a journal publication. The paper is entitled "Local interaction and navigation guidance for hunters drones." C. de Souza Jr, P. Castillo, B. Vidolov.

Opening remarks

Imitation of nature has always been an excellent starting point for many robotics technologies. With that in mind, we decided to seek collective behaviors in nature that could be interesting for our problem, multi-agent pursuit. Therefore, in lions hunting, we find an interesting analogy with robotics' application; since lionesses are usually individual hunters but can cooperate in the presence of same pride individuals. This ability can be a desirable behavior for a fleet of stalking robots, in the sense that they are not dependent on each other, neither in the formation itself; however, they can cooperate for the endeavor in the presence of neighbors.

Work in progress as of 20 th June, 2021 [START_REF] Philip | Cooperative hunting in lions: the role of the individual[END_REF] analyzed the cooperative hunting of groups of lionesses (Panthera Leo) in the semi-arid plains in Namibia. The author observed that some individuals tend to orient themselves directly towards the target, while others tend to take "indirect routes" or even encircling the prey. He classified the hunting roles according to its positioning in the formation: wings (right and left) and centers, alluding to football players' roles. The "wings" ambush (taking indirect routes) and drive the prey towards the center. The resulting formation is described in Figure 2.2. Moreover, in the lioness' case, the roles are most likely assigned to the same individual.

In our work, the term "role" will be employed similarly to [START_REF] Philip | Cooperative hunting in lions: the role of the individual[END_REF], which refers to the "trajectory" adopted by the pursuer towards the target. Later on, we will see that, by using the Deviated Pure Pursuit (DPP), the trajectory's shape can be modified by changing an offset value. However, other than the observed by [START_REF] Philip | Cooperative hunting in lions: the role of the individual[END_REF], in our approach, the roles are not assigned to one individual, rather they are given based on the current engagement with the prey and its neighbors, as illustrated in Figure 2 Furthermore, in this approach, unlike most other multi-agents techniques, the agents have no constraints about fleet formation or cohesion. Instead, each agent goes toward the target and adapt its route in the presence of neighbors.

The absence of repulsive/attractive forces in the formation avoids a common

Work in progress as of 20 th June, 2021 oscillation problem (spring effect), accentuated in real-world applications, where delays, loss of data, or long sample times are present [START_REF] Virágh | Flocking algorithm for autonomous flying robots[END_REF].

Finally, our strategy relies, a priori, on bearing-only information about the target and neighbors, which can be easily obtained from ordinary onboard sensors, such as a camera or lidar. Furthermore, the assumption of bearingangle information solves the recurrent problem of observing the neighbors' position/velocity, which often requires explicit communication premises.

Group Deviated Pursuit (GDP) algorithm

In order to introduce our strategy, let us consider a conventional form of the DPP, exposed previously in (3.9). By considering different values offset angle (for α ∈ R| -π 2 < α < π 2 ), different trajectories can be observed, such as in Figure 2.4. Notice also that the Pure Pursuit (PP) can be described as an especial case of DPP with α = 0.

Work in progress as of 20 th June, 2021

As stated previously, the "roles" in this strategy are related to the pursuer's trajectory shape. Therefore, the idea is that each agent can assume different references of α, causing consecutively different trajectories and increasing the possibility of capturing the intruder. Thus, let us consider a variable offset angle of the agent i (α i ) given by:

α i = α 0 * N j!=i δ i,j (ξ i , ξ j )
where N is the number of neighbors of the i-agent, and α 0 is a constant offset step. The δ ij is a function indicating the positioning of agent j in the Pursuit frame of the agent i (P i ). Thus, δ ij can be defined as:

δ i,j = y P i j |y P i j | (2.3)
where δ i,j is calculated for each neighbor, and it basically indicates in which side the neighbors j is in relation to the λ iT : left side δ i,j = 1 and right side δ i,j = -1. In equation (2.3), y P i j represents the lateral position of the agent j in P i . The origin of P i coincides with body frame of i, nevertheless it is rotated in (λ iTψ i ), in such way that its x-axis is always pointing to the target. To obtain the coordinates, ξ P i j = [x P i j y P i j z P i j ] T , of the agents neighbors j in the P i frame, the following relation is used

ξ P i j = T P i I * ξ j (2.4)
where, T P i I is a matrix transformation representing inertial position ξ j in the pursuit plane P i . Finally by substituting (2.3) in (3.9), the final guidance law for a multi-agent pursuit can be obtained:

f ψ i = -K dpp *         ψ i -λ iT -α 0 * N j!=i δ i,j (ξ i , ξ j )         (2.5)
Work in progress as of 20 th June, 2021

Numerical Validation

In order to analyze the performance of the algorithm (2.5), we considered the framework proposed in the section 2.1, where the agents modeled as a particle with constant velocity and moving in a plane. The guidance law (2.5) was applied, and the following three scenarios were considered: Fixed target: The first simulation is done for a simple scenario where the intruder drone stays in a fixed position. It is proposed to illustrate the formation pattern in a simple case. Notice from Figure 3.7 the patterns of "wings" and "center" can be determined with three pursuers Escape trajectories: Several simulations were carried out with a different number of pursuers (1, 2, 3 and 4), see Figure 2.6. For these simulations, the intruder velocity was set twice as fast as the pursuers. Besides, in this scenario, we propose reactive escape behavior for the evader. We implemented the repulsive behavior, as described in subsection 2.1.2.

Work in progress as of 20 th June, 2021 4). In all cases, the target does an escape trajectory-based on repulsion forces, with a velocity of 2 times bigger than the pursuers. The red circles signalize the end of the target trajectories.

In Figure 2.6, observe that in the two firsts cases (A and B), the target was not caught. Nevertheless, in the two last cases (C and D), the target was successfully intercepted before the limit of time, evidencing the importance of the increasing number of pursuers in the chase's success.

Work in progress as of 20 th June, 2021

Large number of pursuers: Although not being designed for a large number of agents, the proposed algorithm can be easily scalable due to its decentralized architecture and local observation. We illustrate that in a pursuit with up to 15 pursuers, see Figure 2.7. Remark from this figure that an emergent "flocking" pattern appeared, where each agent computes independent trajectories following the target with no information of neighbors distance. Effect of the offset angle: This qualitative evaluation aims to assess the effect of the offset (α 0 ) on the total number of catches. To this end, we subject the GDP strategy to the benchmark with seven different offset values, varying incrementally from π/32, as indicated in the legend of the Figure 2.8. More details about the benchmark setup will be given in Section 2.5

Work in progress as of 20 th June, 2021

Besides, note that the first value (α 0 = 0) is equivalent to the classic Pure Pursuit (PP), where all agents point to the target's current position. However, applying this configuration, the cooperative behavior does not emerges, and the total catches remain practically the same during all the numbers of agents.

Therefore, with the smallest increment of offset (α 0 = π/32), the size of the group starts to affect on the total amount of catches. The number of captures tends to increase, culminating in a cumulative total of 4233 catches for an offset of π/4. Please note also that for the offset value α 0 = 3π/32, our strategy achieve full benchmark from 5 pursuers on, showing itself as an efficient strategy for chasing a faster evader.

For offset values bigger than π/4 the capture's performance tends to decay. This behavior was already observed in the experiments, and its justification is due to the fact that large offset values result in very open trajectories towards the target, allowing the evader to escape between two pursues.

Due to time constraints, we have not carried out additional work to find optimal parameters for this pursuit. Most likely, the use of smaller increments between the two best results (π/4 and 3 * π/32) could culminate in even better results. However, we restrict ourselves to the current results, which seems to be sufficient to reaffirm the cooperative behavior emerging from the GDP strategy.

Finally, note that for the two bigger offset values (5 * π/32 and 6 * π/32), although showing average yield in the total number of catches, they perform well for smaller groups of pursuers (2 and 3). This shows us that the current way of calculating the offset, which is directly proportional to the number of neighbors, tends to harm individuals at the extremities, who receive high offset values. A natural solution to this dilemma could be the use of a nonlinear function for calculating the offset, which would tend to "saturate" the offset values from a certain number of pursuers.

Work in progress as of 20 th June, 2021

Effect on the collision: The graph on Figure 2.9 clearly illustrates the effect of the offset on collision avoidance. For offset = 0, collisions tend to grow exponentially, reaching values up to 52 times higher than using α 0 = 3 * π/32.

The collision between pursuers is clearly undesirable; nevertheless, it is difficult to practice since all agents tend to have a common point, the evader. However, when dealing with robotic implementations, we have established a low-level safety layer that ensured collision avoidance during the experimental tests (See Experiments Section 4.2). Work in progress as of 20 th June, 2021

Group Mixed Pursuit

In this section, we present our second proposal for collective hunting using guiding laws. This new algorithm seeks to go beyond ambush behavior and reproduces a more efficient pursuit, especially in the presence of faster or more agile prey. The name "mixed persecution" comes from the fact that this new GL incorporates two customary laws: the Deviated Pure Pursuit (DPP) and the Proportional Navigation Guidance (PNG). This section's content is part of an article, still in the process of preparation, which will be entitled: Flocking, pursuit, and avoidance: a constant bearing guidance law for multi-agents C. de Souza, P. Castillo, B. Vidolov. Figure 2.10 -Predators par excellence in nature use "Parallel Navigation" rule in their final approach. Left: the Falcon Peregrine, which its hunting trajectory has been described in [START_REF] Caroline H Brighton | Terminal attack trajectories of peregrine falcons are described by the proportional navigation guidance law of missiles[END_REF]. Right: the Dragonfly, excellent insect predators that use a "camouflage strategy" to deceive their prey [START_REF] Pal | A Review of Target Pursuit Strategies in Aerial Species[END_REF].

Brief preface and literature review

In the previous section, we could see the interest in using guidance laws in collective hunting algorithms. The GL was able to produce verisimilar hunting behaviors, and the use of different deviation angles was responsible for the ambushing behavior and avoiding collision at once. Although interesting behaviors were obtained, we observed that this algorithm does not perform well in the presence of a faster target.

Work in progress as of 20 th June, 2021

In fact, the algorithm used is based on Pure Pursuit (PP), where the predator always addresses the prey's current position. This strategy constantly led to a "tail chase," pursuit, which ultimately gives no chance for the slowest chaser.

The disadvantages of PP towards moving targets are well known, and that is why it has been constantly replaced by Parallel Navigation (or constant bearing pursuit). This geometrical rule, apparently known since antiquity, has been extensively used by mariners to provoke an interception or even avoid a collision with other ships [START_REF] Matthew | Collision Avoidance and Navigation of UAS Using Vision-Based Proportional Navigation[END_REF][START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF]. Moreover, its principle is straightforward: if the bearing angle of two bodies is constant, and the distance is decreasing, these vehicles are on a collision course. Furthermore, this principle guarantees the optimum trajectory of the pursuer in the event of a non-maneuvering target. This principle is so simple yet so powerful that it has been the most popular in handling missiles [START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF]. The family of guidance laws that implements this rule is called Proportional Navigation Guidance (PNG). The first flying bombs that implemented it were fired in the sixties, and today it still arouses interest in the scientific community in guided munitions [START_REF] Weiss | Adaptive proportional navigation guidance[END_REF][START_REF] Chen | A modified cooperative proportional navigation guidance law[END_REF]. Recently, in addition to missiles, several robotic systems have used this principle for various applications, such as autonomous landing [START_REF] Tan | Proportional navigation (PN) based tracking of ground targets by quadrotor UAVs[END_REF], navigation in a clustered environment [START_REF] Jung | Perception , Guidance , and Navigation for Indoor[END_REF], and intruders' interception [START_REF] Thomas | Guidance Laws for Partially-Observable UAV Interception Based on Linear Covariance Analysis[END_REF].

Furthermore, with the recent image acquisition technologies, scientists have identified this principle as the attack behavior of several predators, such as the peregrine falcon [START_REF] Caroline H Brighton | Terminal attack trajectories of peregrine falcons are described by the proportional navigation guidance law of missiles[END_REF] and some insects [START_REF] Samuel T Fabian | Interception by two predatory fly species is explained by a proportional navigation feedback controller[END_REF]. In this context of natural predation, the proportional navigation law (PNG) is confused with the concept of camouflage laws, where the predator moves towards the prey so that it sees it "static" to the background [START_REF] Wei | Pursuit and an evolutionary game[END_REF].

Its importance is evidenced by the applications and observations above; however, Parallel Navigation presents certain disadvantages mainly in the final stage and through maneuvering targets (changing directions abruptly). Besides, it requires high values of lead angle when the speed ratio is low (which can be quite common in the presence of a more agile prey) and can provide loss of sight of the target, in the case of a field of view sensor limited [START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF].

Work in progress as of 20 th June, 2021

To minimize these disadvantages, Shneydor, in [START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF], devised a "mixed" guidance law of between "Pure Pursuit" and "Parallel Navigation", since this first one, although less efficient, does not suffer from the low-speed ratio.

With similar motivation, we propose in this work a mixed guidance law, composed of two terms, one from PN and the other from DPP. The first one is desired to be predominant in the pursuit, since it has a more efficient trajectory. In turn, the second term has dual functionality: first, ambushing (as explained in the previous work), and second limiting the lead angle of the pursuer to target, preventing the loss of sight of the evader.

Group Mixed Pursuit (GMP) algorithm

Considering the evolution model exposed in (3.16), let us consider the following guidance law, which composed of two terms, one for the Proportional Navigation (f png ) and the other for the Deviated Pursuit (f dpp ). Thus the mixed GL will have the following format:

f ψ i = σ a (f png ) + σ b (f dpp ) (2.6)
where σ x (.) is a saturation function that can be defined by

σ a (x) =                a, if x > a x, if a ≥ x ≥ -a -a, if x < -a (2.7)
Therefore, the threshold of σ b must be bigger than σ a , to allowed the second term to overcome the first. This is necessary since f dpp is responsible for shotrange collision avoidance between two pursuer. In the next, we present the composition for both terms.

Work in progress as of 20 th June, 2021

Proportional Navigation term A simplistic GL, which implements the Parallel Navigation geometrical, can be defined as follow:

f png = K pn * λiT (2.8)
where K pn is a positive gain. This GL is also called Proportional Navigation Guidance (PNG), and it is described in Chapter 1.

As stated in the introduction, the PNG carries the drawback of provoking a large leading angle (ψ i -λ iT ) from the pursuer to target when the ration (v i /v T ) is low, which can cause loss of sight of the target, considering limited field-of-view of the pursuer.

Deviated Pursuit term

In order to avoid the target's loss of sight, which can be a non-desired effect of the PNG, we proposing the following modified DPP law:

f dpp = K dpp (d min ) * tan ψ i -λ iT -α i β * π -1 (2.9)
where β is the opening angle of the pursuer (or perception sensor). The tan(.) function was chosen for being odd-symmetric and not defined in ψ iλ iT = β.

Thus this term provides asymptotic values when the heading error is close to the limits of the field of vision β.

Furthermore, we introduce the gain K dpp (.) that is inversely proportional to the smallest distance between pursuer d min and is null when the distance is bigger than a threshold R 0 . It can be computed as follow:

K dpp (d min ) =          K 0 * d min -R 0 R 0 2 , if d min < R 0 0, else (2.10)
As a result, the DPP term has a short-range performance, being canceled when two pursuers are not on collision imminence; and evidenced by the proximity of two pursuers.

Work in progress as of 20 th June, 2021

The mixed GL The first term of (2.6), corresponding to PNG law, is desired to be the predominant during most of the pursuit since it provides a more efficient trajectory. Nevertheless, there are two situations where the DPP term should overcome the PNG: first in the eminence of collision with the closest neighbor (d min < r 0 ), and second, when the target is getting close to the limit of the field of vision, i. e., |ψ iλ iT | → β.

Numerical validation PNG vs PP

In the scenario illustrated in Figure 2.11, we intended to give the reader unfamiliar with guidance laws an expository comparison between the two methodologies covered in this section. Therefore, to illustrate the advantage of the Proportional Navigation Guidance (PNG) over the Pure Pursuit (PP), the following scenario is proposed: two identical pursuers are located initially in the bottom coordinates (x, y) = (2, 0)m; while a target (red) starts from the position (x, y) = (-8, 10)m. are not aware of each other, i.e, there is no cooperation. In the left picture, the target performs a linear trajectory, while in the right one, the target executes a maneuvering trajectory.

Work in progress as of 20 th June, 2021

The first pursuer (pink circles) implements a PP algorithm while the second (blue circles), the PNG algorithm. In Figure 2.11-left, observe the pursuer's performances when tracking a target with v i = 2 * v T ; note that PNG algorithm provides a shorter path and prevails at the end of each course.

In Figure 2.11-right, a similar scenario is illustrated; nevertheless, the target does not have linear movement. Notice that even if the algorithm is not designed for non-maneuvering targets (red crosses), the PNG (blue circles) succeeds and provides a shorter path than the pure pursuit (pink circles).

Mixed Pursuit vs Deviated Group Pursuit

In this second example, we will make a qualitative comparison between the mixed pursuit (2.9) and the methodology presented in the section 2.2.2, the Deviated Group Pursuit (DPG) (2.5).

Figure 2.12 -Comparison between Deviated Group Pursuit (DGP, left picture) and Mixed Group Pursuit (MGP, right picture). In both cases the target is navigating in the same trajectory.

Work in progress as of 20 th June, 2021

For this, consider the scenarios shown in Figure 2.12: Three pursuers (red stars) are initially located at the bottom of the arena, positions (0, -5) (0.5, -5) (1, -5)m, moving with fixed speed of 0.45 meters/frame. The target performs a maneuvering trajectory with variable speed, but that is up to 3 times higher than the pursuer speed. The target behavior was obtained from recording experimental trajectories with a quadcopter. Still, in Figure 2.12, the left image represents a group chase where the agents implemented the DPG. Note that even though the ambush formation is created, the trajectory is not efficient, and the target is not captured in the course of this screenshot. As previously discussed, this is since the pursuit is directed towards the target's instantaneous position, and no consideration is given of its dynamics.

In Figure 2.12-right, we can see that the target is captured, and the agents still kept an ambush formation. Note that the "straight line condition course," the geometric rule behind PNG, promotes an efficient path even when it comes to maneuvering trajectories.

Effect of the offset: Similar to the previous section, we submitted the GMP to the pursuit benchmark, considering the same offset intervals. The results regarding the number of captures and average time can be seen in the respective Figure 2.13.

Note that the same performance patterns presented in GDP were repeated for this strategy. Firstly, the best results in general terms are for the offset values between 3 * π/32 and 4 * π/32. Secondly, the highest offset values, 5 * π/32 and 6 * π/32, have relatively high capture values for 2 and 3 pursuers, but their yield drops as the number of pursuers increases.

These results reinforce the conclusions of the previous section. Besides, note that, unlike the last section, the results for offset = 0 do not correspond to the classic PNG navigation law. This is due to the tangent term (Eq. 2.9), which restricts the pursuer's heading movement, differentiating it slightly from the classic implementation of PNG.

Work in progress as of 20 th June, 2021 Work in progress as of 20 th June, 2021

Reinforcement Learning Pursuit

In this section, we propose an approach for multi-agent pursuit using Deep Reinforcement Learning (Deep RL). Henceforth, instead of using guidance law established to model the pursuit, the pursuer behavior is controlled by a neuronal network policy. To determine the policy, we considered a multi-agent environment, bounded and using the kinematics equations (2.1). In this, the purser will "learn" its behavior by trial-and-error interaction in a simulated environment.

Brief preface and literature review

Until now, all pursuit strategies have been designed in a bottom-up manner; the microscopic relationships have been previously established, and then the emergent behavior was analyzed. Nevertheless, in this section, we will use an automatic behavior design technique by applying Deep Reinforcement Learning.

Work in progress as of 20 th June, 2021

In this, the purser will "learn" its behavior by trial-and-error interaction in a simulated environment.

In the last decade, with the advances and popularization of Deep Learning, another recent technology has stolen the spotlight from the robotic community, Deep Reinforcement Learning. These advances allowed learning of complex tasks in continuous environments, which has increasingly enabled Reinforcement Learning in robotic applications, such as driving [START_REF] Isele | Navigating occluded intersections with autonomous vehicles using deep reinforcement learning[END_REF], grasping [START_REF] Levine | End-to-end training of deep visuomotor policies[END_REF] and manipulation [159].

Deep RL has also proved to be a powerful tool for the complex task of multiagents pursuits, such as in [START_REF] Hüttenrauch | Deep Reinforcement Learning for Swarm Systems[END_REF][START_REF] Xu | Multi-agent Deep Reinforcement Learning for Pursuit-Evasion Game Scalability[END_REF][START_REF] Lowe | Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments[END_REF]105]. However, most of these works consider models and hypotheses very distant from real-world applications, such as local measurement and locomotion restrictions.

In this work, we propose a multi-agent pursuit approach against a faster evader. We consider it a decentralized system, where the pursuers decide their own actions based on local observations of the environment. The perception of the target and the neighbors are given in local and polar coordinates, which are consistent with commons embedded sensors for autonomous navigation, such as camera and lidars.

We use Twin Delayed Deep Deterministic Policy Gradient (TD3) [START_REF] Fujimoto | Addressing Function Approximation Error in Actor-Critic Methods[END_REF], a stateof-the-art deep reinforcement learning algorithm that was successfully applied to other domains [START_REF] Matas | Sim-to-Real Reinforcement Learning for Deformable Object Manipulation[END_REF][START_REF] Chen | Model-free Deep Reinforcement Learning for Urban Autonomous Driving[END_REF], with a state representation that encapsulates relative positional information of neighboring agents as well as the target, and use a group reward structure that encourages good formations. During training, curriculum learning is applied to start with an easier version of the problem, and gradually learn the task with increasing difficulty.

Related work

The pursuit-evasion game is a highly studied task in multi-agent RL [START_REF] Mostafa | The residual gradient FACL algorithm for differential games[END_REF][START_REF] Sameh | Q(λ)-learning adaptive fuzzy logic controllers for pursuit-evasion differential games[END_REF][START_REF] Jouffe | Fuzzy inference system learning by reinforcement methods[END_REF]. However, most approaches apply only to omni-directional agents, which cannot be easily transferred to real robotic applications without a loss in performance.

Work in progress as of 20 th June, 2021

Lowe [START_REF] Lowe | Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments[END_REF] presented an approach for multi-agent reinforcement learning using an adapted version of an actor-critic algorithm which they extended to multi-agents. They use their approach on the pursuit-evasion game with omnidirectional agents and showed that it outperformed Deep Deterministic Policy Gradient (DDPG) [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] on the same scenario. Xu [START_REF] Xu | Multi-agent Deep Reinforcement Learning for Pursuit-Evasion Game Scalability[END_REF] considered pursuit-evader games with non-holonomic agents, where new agents can join the game. They adapted Bi-directional Recurrent Neural Networks [START_REF] Schuster | Bidirectional recurrent neural networks[END_REF] and DDPG, however, they only consider a situation with 3 and 5 agents. Furthermore, the observation also assumes global information about other agents which limits the applicability to real world situations. Hüttenrauch [START_REF] Hüttenrauch | Deep Reinforcement Learning for Swarm Systems[END_REF] studied multi-agent pursuit evasion systems by considering the agents as a swarm. They consider agents in the swarm as interchangeable and the exact number irrelevant. They create a new state representation based on mean embedding of distributions. Their work focuses on scalability and shows that their system can operate with up to fifty agents.

Curriculum Learning Curriculum learning [START_REF] Bengio | Curriculum learning[END_REF] is a learning paradigm to improve the speed of convergence and reduce local minima by gradually increasing the complexity of training data. This learning paradigm has been been widely used for RL [START_REF] Portelas | Automatic Curriculum Learning For Deep RL: A Short Survey[END_REF][START_REF] Tidd | Guided Curriculum Learning for Walking Over Complex Terrain[END_REF] and deep learning [START_REF] Graves | Automated Curriculum Learning for Neural Networks[END_REF]. Gupta [105] used curriculum learning in multi-agent RL to solve problems which were previously considered intractable. They show that Trust Region Policy Optimization (TRPO) [START_REF] Schulman | Trust region policy optimization[END_REF] has a better performance compared to Deep Q Networks (DQN) [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF] in the discrete domain and DDPG in the continuous domain.

Our work, while borrowing ideas from both classical methods and learningbased methods, focuses on using DRL to improve the pursuit performance and consider operational metrics such as capture success rate and the average time to capture. Furthermore, we propose a method that is suitable for sim-to-real policy transfer with realistic observation models and non-holonomic constraints. To our knowledge, we are the first to demonstrate a real-world multi-agent pursuit implementation using a DRL policy.

Work in progress as of 20 th June, 2021

Methodology

Multi-Agent Deep RL: We consider our task to be a Markov Decision Process (MDP) defined by tuple {S, A, R, P , γ} where s t ∈ S, a t ∈ A, r t ∈ R are state, action and reward observed at time t, P represents an unknown transition probability from s t to s t+1 taking a t , and γ stands for the discount factor. The reinforcement learning goal is to maximize the sum of future rewards R = T t=0 γ t r t , where r t is provided by the environment at time t. For continuous control, actions are sampled from a deep neural network policy a t ∼ π θ (s t ), where a t is the angular velocity of an individual pursuer.

Our approach is based on distributed, in-series learning with parameter sharing. For learning, we use Twin Delayed Deep Deterministic policy gradient approach (TD3) [START_REF] Fujimoto | Addressing Function Approximation Error in Actor-Critic Methods[END_REF] using an experience replay buffer. The agents are considered homogeneous, allowing sharing the same policy and parameters, although the execution remains decentralized.

State-representation:

The observed states of a pursuer i for a total of n pursuers s i = [ψ i , ψi , s iT , s i1 , s i2 , .., s in-1 ] where ψ represents the heading with respect to a fixed world frame, s iT denotes the state of the target relative to pursuer i and s ij defines the state of pursuer j relative to pursuer i. The relative state of the target with respect to pursuer i is s iT = [r iT , ṙiT , α iT , α iT ] and the relative state of the pursuer j with respect to pursuer i (i j) denotes s ij = [r ij , α ij ], where r ij is the Euclidean distance between pursuers i and j and α ij represents the heading error defined in local an polar coordinate frame attached to pursuer i.

In order to generalize the learning, the neighbors states s ij are sorted with respect to their proximity to the center of the heading of the agent i, i.e., the first agent j = 1 is the one with smaller heading error α ij = ψ iλ ij . Another generalization is limiting the number of observed neighbors (N O ) and filling up this sub-group with the N O closest pursuers. This strategy allows the scalability of the system.

Work in progress as of 20 th June, 2021

Reward structure: At each time step, each agent individually receives a reward designed to incentivize the evader's capture and encourage an adequate formation of pursuers. The reward function is:

Re i =                Re captor , if r iT ≤ R cap Re helper , if r iT ≤ d cap , ∃j i -w q q -w d r iT , otherwise
At each step that the target is not captured, every agent receives a negative reward that is a linear combination of an individual reward (its distance to the target r iT ) and a group reward (q score proposed by [START_REF] Kamimura | Group Chase and Escape: Fusion of Pursuits-Escapes and Collective Motions[END_REF] which encourages the agents to make a good formation). The q score is a scalar number in the range [0, 2], which provides a metric for evaluating the fitness of the formation of the pursuers (lower is better). Also, we penalize r iT , which helps encourage the agents to get close to the evader while being in a good formation. The weights w q and w d can be chosen such that when the agents are close to the target, the reward is dominated by the q score, encouraging good formation. Therefore, the formation score (q) is defined as:

q = 1 n n i=1 (r 0T • r iT + 1) (2.11)
where r iT represents the normalised vector between agent i and the target, and n is the number of agents. In this equation, the closest agent to the target is defined as an agent 0. The formation score encourages agents to spread around the target (i.e., approach the target from different directions). Early experiments with the formation score as part of the reward function showed that when the formation is the only component of this reward, pursuers would only form a good formation but would not attempt to capture the evader. To avoid this situation, we penalize the distance to the target, which helps encourage the agents to get close to the evader while being in a good formation. The weights w q and w d were chosen such that when the agents are close to the target, the reward is dominated by the formation score, encouraging good formation.

Work in progress as of 20 th June, 2021

However, when the agents are far from the target, the reward will be dominated by the target's distance, encouraging the agents to move closer to the evader. We analyze the effect of the formation score on pursuit performance in Sec. 2.4.3.

If the target is captured at a time step, then the pursuer who captures the evader receives the reward r captor , while the rest of the agents receive r helper , such that r captor > r helper . This encourages the pursuers to act selfishly and go for the final capture while encouraging collaboration.

Curriculum Learning We apply a curriculum for learning by starting from a more accessible version of the task and gradually increasing the difficulty until the actual difficulty is achieved. There are two main factors in determining the difficulty of a pursuit-evasion game: the target's relative speed with respect to the pursuers and the capture radius d cap . We vary the capture radius by starting from a large radius (so it is easier to capture the target), then gradually making it smaller. This encourages agents to not adopt a straightforward chasing tactic at the beginning of learning but to form more sophisticated behaviors, which could be transferred to smaller capture radii. We also experimented with reducing the pursuer speed to reduce the task's difficulty; however, we found that pursuers mostly learned to follow the evader directly, and it was harder to explore more sophisticated behaviors afterward.

Curriculum learning helps exploration, especially during the early stages of learning because early on, it helps the pursuers to capture the target, which would take a longer time in the actual and more difficult scenario. This helps alleviate the sparse reward problem, which is a well-known challenge in reinforcement learning [START_REF] Nair | Overcoming exploration in reinforcement learning with demonstrations[END_REF]. We analyze the effect of curriculum learning on the pursuit performance in the subsection 2.4.3.

Work in progress as of 20 th June, 2021

Numerical results

We use the following simulation parameters: Arena radius R arena = 430 pixels, T timeout = 500 iterations, rewards r captor , r helper and the weights w q and w d were set to 10, 100, 0.1, 0.002 respectively. The capture radius R cap was set to 30 pixels for testing, although this value was varied as part of curriculum learning. For fixed linear velocity cases, the pursuers' speed v p was 10 pixels per timestep, while the target's speed v T varied from 0 to 20 pixels per timestep. The maximum angular rate ω max was fixed at π/10 per timestep. The numbers of pursuers n varied between 1 and 8, and they were initialized at random positions within a circular area with radius of 100 pixels around the center, while the evader was initialized at a random position outside the circular area with a radius of 300 pixels. The rest of this section covers the evader behaviors and baseline methods. We provide a partial reward at every timestep in order to encourage good formations. We analyze the effect of supplying this dense reward component to each agent. Fig. 2.16 compares the evolution of the capture performance with and without the formation score, with respect to the number of training steps. These experiments were conducted with n = 3 agents. As can be seen in the figure, benchmark scores were slightly higher when the formation score is used as part of the reward. Furthermore, when the formation score is used, the average capture time for successful episodes is decreased.

Effect of Curriculum Learning

Work in progress as of 20 th June, 2021

Qualitative analysis of emergent behavior

We observe two interesting learned emergent behaviors that often leads to successful captures: ambushing and splitting up. Fig. 2.17 shows the splitting up behavior with n = 3 agents. This behavior was more common with a smaller number of agents. The agents tend to split up in to two groups, trying to push the evader in to a wall before attempting to block the two opposite directions.

Work in progress as of 20 th June, 2021 Fig. 2.18 shows the ambushing behavior with n = 8 agents. This behavior was more common with a larger number of agents. The agents tend to form a circle, attempting to move such that they can completely surround the evader and then approach from all directions. This is similar to pack behaviors observed in Muro's work [START_REF] Muro | Wolf-pack (Canis lupus) hunting strategies emerge from simple rules in computational simulations[END_REF]. When the agents execute this strategy to trap the evader, it is often very difficult for the evader to escape.

Work in progress as of 20 th June, 2021

Variable linear velocity

Previous sections considered constant linear speed and variable angular speed for pursuers, primarily because this is an assumption for classical algorithms. We now consider the more general case, where agents can also vary their linear velocity between 0 and v p . Therefore, we train the network with two outputs: linear and angular velocity. We consider the 3 agent scenario, in which the agents achieve 100% capture rate with and without velocity control, in both the fixed, Work in progress as of 20 th June, 2021 and reactive benchmarks.

The agents often displayed a "Stalking" strategy as illustrated in Figure 2. [START_REF] Samuel T Fabian | Interception by two predatory fly species is explained by a proportional navigation feedback controller[END_REF]. With this strategy, the agents move towards the target, before slowing down and waiting, until the opportunity presents itself to attempt to capture the evader. This behavior may have analogues in nature, where pursuers will stalk their prey and position themselves such to maximise the likelihood of attack [START_REF] Philip | Cooperative hunting in lions: the role of the individual[END_REF].

Work in progress as of 20 th June, 2021

Qualitative comparison

In this section, we propose to evaluate the performance of our propositions quantitatively, not only among themselves but also with three baseline methods of the state-of-the-art. Two of them are based on classic multi-agent strategies, Angelani [START_REF] Angelani | Collective predation and escape strategies[END_REF] and Janosov [START_REF] Janosov | Group chasing tactics: how to catch a faster prey[END_REF], where traditional flocking self-propelled agents are adapted to carry out a pursuit. The third one, Hüttenrauch [START_REF] Hüttenrauch | Deep Reinforcement Learning for Swarm Systems[END_REF], is part of an interesting strategy for applying deep reinforcement learning in swarming problems. Therefore, we conduct the following performance analysis: effect of the number of pursuers, arena size, relative evader speed.

Benchmark

To assess our propositions' performance qualitatively, we have established a benchmark for pursuit, which will be reported below. A group pursuit environment was implemented in python, where pursuers and evaders evolve in the 2D plane. The navigable area is limited by a circular border with a variable radius. However, the standard size of the arena is R arena = 430pixels.

Work in progress as of 20 th June, 2021

The initial positions of the pursuers and the target are initiated randomly. However, to avoid initial conditions with the target already at a disadvantage, two different initialization areas are determined. The pursuers are started within a circle of radius equal to 100 pixels, concentric with the arena. The evader is initialized outside of another concentric circle, with a radius of 300 pixels.

The standard pursuit evaluation consists of 700 episodes, where the relative speed of the target to the pursuers varies incrementally every 100 episodes, i. e., v T /v p = 0.8, 1.0, . . . , 2. An episode lasts until captured by one of the pursers or until the limit of interactions 500 is reached.

For most scenarios, pursuers and target move at constant speeds. However, while the target moves in an omnidirectional manner (and with constant absolute speed), the pursuer moves according to the Equation (2.1), in its discrete form.

Through this benchmark, the following metrics are being calculated:

• The number of captures: which is equivalent to the number of successful episodes (Max 700). In this work, the capture radius was set to R cap = 30 pixels.

• The average time of capture: which corresponds to the average of interactions needed to capture the target.

• Collisions: the number of times that two pursuers had their relative distance below a collision threshold, R col = 30 pixels.

Besides, to test the benchmark repeatability, we propose the following experiment: the same pursuit scenario, with 3 pursuers implementing the GDP, were repeated 30 times. The total value of the catches is shown in Figure 2.20. As expected, the variability occurs more at high relative speeds, reaching its apex at Vt/Vp = 2. However, the total catch's standard deviation is 6.36, which corresponds to a variation of 0.90% in the captures.

Work in progress as of 20 th June, 2021

Effect of the number of pursuer

The number of captures and the average time of six pursuit strategies are shown in Figure 2.21. Our RL pursuit strategy surpasses all others presented, reaching a complete benchmark for all agent numbers from 3 onwards. The other best results are respectively GMP, Janosov, Hüttenrauch, GDP and Angelani.

Although the GDP has a slightly lower result than Huttenrauch and Janosov, it is still an interesting option, especially when considering its minimalistic implementation. Besides the GDP computational simplicity (Equation 2.5), it is the single one possible to implemented by using only bearing information for the whole pursuit. Which gives you a good advantage in practical terms since it can be implemented using only an RGB camera.

GMP fulfills its role of improving GDP, and its results are clearly more performant. Although more information is required for carrying the pursuit in GMP, it is still compatible with the implementation in real systems. As shown in equation (2.9), this strategy requires, besides the bearing, the relative distances, and the bearing angle rate towards the target. Although more hardware is needed for implementation, these perception hypotheses are still acceptable in an actual robot application. They can be provided ideally either by an RGBD camera or by a fusion of camera + Lidar, thus adding the relative distance to the bearing information.

However, it should also be noted that the good results of Janosov and Hüttenrauch are based on hypotheses that are difficult to put into practice in real case pursuit, where the target is not cooperative. Although both strategies use relative distance as input (the first in polar coordinates and the second in Cartesian), both also consider observing the target's velocity in an inertial frame, which is an improbable assumption considering the current embedded sensors.

Work in progress as of 20 th June, 2021 Work in progress as of 20 th June, 2021

Effects of the relative velocity

In the following two graphs, we will see the relative velocity (V T /V p ) effect on the capture quantities. The results still maintain the same pattern presented in the previous experiments-our RL pursuit with the best yields, followed by GMP, Janosov, Huttenrauch, GDP, and Angelani. Besides, as expected, increasing the relative speed reduces the capture rate of all analyzed algorithms. Besides, it is evident the Angelani's low performance for faster target (V T /V p > 1). It is predictable considering that they do not employ any "prediction" strategy for the target's pursuit. This implementation basically consists of a Pure Pursuit towards the target and the repulsion force between pursuers to avoid collisions.

The need for "prediction" for capturing faster targets had already been indicated in Janosov's work. However, their solutions rely on knowing the target's speed in global coordinates, which is a hard hypothesis considering a noncooperative target. However, the GMP shows that good performance can be achieved when considering parallel navigation. As argued in state of the art, this phenomenon has optimal properties under certain pursuit conditions.

Work in progress as of 20 th June, 2021 Work in progress as of 20 th June, 2021

Effects of the arena size

As we showed previously, with an increasing number of agents, the task decreases in difficulty for a fixed arena size. Therefore, with a larger number of agents, most approaches can perform the task successfully. The pursuit-evasion game is played in an obstacle-free arena, but the agents can use the arena boundaries to constrain the evader movements. In this section, we investigate the effect of larger arena sizes using n = 3 agents.

In the results exposed in Figure 2.23, the only parameter changed between the evaluations and the arena's radius size, which is multiplied by a factor (Radius factor). In the graphs below, we have the performance metrics depending on the arena's size, which varies between its standard size (Radius factor = 1) up to twice this (Radius factor = 2). Note that, although the loss of performance is common to all strategies, our strategy RL pursuit and GMP are less likely to suffer from the increase in the arena's size.

Work in progress as of 20 th June, 2021 Work in progress as of 20 th June, 2021

Conclusion

In this chapter, we propose three different techniques to model predators' behavior in a planar pursuit. We also consider the case where the prey is faster than the pursuer, which requires an "intelligent" strategy from the pursuers. We also consider hypotheses consistent with real-world applications, such as non-holonomic motion and instantaneous observations in relative and polar coordinates. In none of the three cases of communication are necessary.

In the first approach, Group Deviated Pursuit, we propose a simple model using a "guidance law" to describe hunting behavior. We propose a modified version of the classic Deviated Pure Pursuit (DPP), where the offset angle is responsible for the different trajectories during an ambush. The offset is calculated instantly based on the relative engagement between pursuers and targets.

In the second proposition, Group Mixed Pursuit, we explore the use of another "guidance law," which is still very popular in missiles, the Proportional Navigation Guidance (PNG). Furthermore, its geometric rule, the "Parallel Navigation," is found in several natural predators' behavior. The PNG term provides a more efficient pursuit, while a DPP term is still maintained to avoid collisions and generate different trajectories.

Finally, in the third section, Group Pursuit using RL, we studied Deep Reinforcement Learning for the multi-agents Pursuit. In this approach, the hunting behavior is given Multi-Layer Neuronal Networks, in which the weighs were obtained by try-and-error in a simulated pursuit-evasion environment.

A qualitative analysis of this and other techniques will be present in the last chapter, Discussion and Conclusion, all the propositions will be submitted to a Benchmark, and different metrics will be highlighted.

Work in progress as of 20 th June, 2021

Chapter 3

Improving algorithms: prediction, avoidance and flocking

In this chapter, we propose extensions for the previous group pursuit on the plane. We do not propose new models of pursuit behavior but complementary scenarios with different problem statements. In short, three new scenarios will be studied. In the first scenario, we will assume that the target's global position and velocity are available for the pursuers. With that, a predictor scheme is chosen to foresee the target's future position. In the second scenario, just one pursuer realizes the target's stalking; nevertheless, other non-cooperative agents, or mobile obstacles, are traveling in the same workspace. In a third and last scenario, we are interested in more massive amounts of agents, where not all of them are informed about the target. In this, we investigate the flocking formation considering only the relative and polar information about the neighbors.

Pursuit with target prediction

In this chapter, we assume that all pursuers have target information within a global framework. Based on this information and based on a predictive scheme, the target's future trajectory is estimated. Henceforth, the whole of pursuit is based on this collective knowledge of prediction and its uncertainty intervals.

This work is a cooperation between the Laboratory Heudiasyc -UTC and the Universidad Politecnica de Valencia. The results are in preparation for a close submission. The paper will be entitled: "Autonomous drone pursuit using a fleet of drones with target prediction" by C. de Souza Jr, A. Castillo, P. Castillo, B. Vidolov. Work in progress as of 20 th June, 2021

Brief preface and literature review

As seen briefly in Chapter 1, some of the anti-drone drones, such as [START_REF] Kim | Three-dimensional discrete-time controller to intercept a targeted UAV using a capture net towed by multiple aerial robots[END_REF]134,[START_REF] Lima | Target localization and pursuit by sensor-equipped UAVs using distance information[END_REF], depend not only on the perception of the ADD itself but also on external location systems. These external locators can accurately deliver the target's location to pursuers and possibly improve the capture task's performance.

With the recent increase in illicit use of mini-drones, numerous studies have been dedicated to the problem of location. In [START_REF] Ding | An Amateur Drone Surveillance System Based on the Cognitive Internet of Things[END_REF] and in [START_REF] Shi | Anti-drone system with multiple surveillance technologies: Architecture, implementation, and challenges[END_REF], the authors point out numerous approaches relevant for their mini-drone detection and tracking. Some of the most common solutions are audio detection, thermal detection, radar detection, and RF (radio frequency). Since these attacks require a quick and precise counter-response, state-of-the-art solutions are usually based on multi-sensors, such as ADS-ZJU and DroneShield [START_REF] Shi | Anti-drone system with multiple surveillance technologies: Architecture, implementation, and challenges[END_REF], or smart-sensors, such as the anti-UAV Defense System (AUDS) [START_REF] Ding | An Amateur Drone Surveillance System Based on the Cognitive Internet of Things[END_REF].

In [START_REF] Lima | Target localization and pursuit by sensor-equipped UAVs using distance information[END_REF], the authors already considered the problem of an intruder's pursuit depending on an external target location system. They considered a system of beacons installed in an area, and the target positioning was obtained through triangulation techniques. Despite the similarities, this work focused more on location and less on the pursuit itself. Besides, this work considers a singlepursuer configuration, and its disadvantages have already been seen in State-ofthe-art.

In [START_REF] Kim | Three-dimensional discrete-time controller to intercept a targeted UAV using a capture net towed by multiple aerial robots[END_REF], the authors consider the target's location made by a 3D air defense radar, and the information is transmitted to the pursuers. Although this work considers a group of drones, they are physically connected by a towed net. Consequently, as seen in Chapter 1, in the case of fixed formation, the multiagent system behaves as a single body, losing the advantages of smart group strategies, such as ambushing and stalking.

In this work, we assume the deployment of a location system, in such way that the target position and velocity are delivered to the pursuers without compromising on delays. The pursuit rule is similar to the Deviated Group Pursuit, proposed in 2, where the offset angle provides the desired formation and collision avoidance for the group.

Work in progress as of 20 th June, 2021

However, to enhance the capturability, a predictor scheme for estimating the target's future position is chosen, and the ambushing formation is based on the prediction error.

Problem formulations

Similar to Chapter 2, we also have a multi-agents pursuit-evasion problem involving N pursuers and a single evader moving in a horizontal plane. The conditions of capture and the kinematic model of pursuer (see equation 2.1) and target still are the same. Based on evader capabilities, the prediction error can be estimated h i . Nevertheless, while in the previous chapter, the target perception was done only by relative and polar coordinates, in this section, we suppose the target's position and velocity available in the global frame for all pursuers. In a practical scenario, the global information could be acquired either by beacons localization systems, such as in [START_REF] Lima | Target localization and pursuit by sensor-equipped UAVs using distance information[END_REF]; or by 3D radar systems, such as in [START_REF] Kim | Three-dimensional discrete-time controller to intercept a targeted UAV using a capture net towed by multiple aerial robots[END_REF]; or even by any technology of localization described in [START_REF] Ding | An Amateur Drone Surveillance System Based on the Cognitive Internet of Things[END_REF] and [START_REF] Shi | Anti-drone system with multiple surveillance technologies: Architecture, implementation, and challenges[END_REF].

Work in progress as of 20 th June, 2021

Furthermore, in this section, the pursuit will not be towards the instantaneous position of the target, but for a predicted future position ( xT , ŷT ). Details about the predictor scheme and estimation of the future position are given next.

Target prediction

The simple pursuit, towards the instantaneous target's position, frequently leads to a tail pursuit. Nevertheless, one natural behavior for a chaser should be to take advantage of the target's known states and by predictions to point towards its future position, reducing efforts as depicted in Figure 3.1.

In this work, a predictor scheme, initially proposed in [START_REF] Castillo | Predicting the future state of disturbed LTI systems: A solution based on high-order observers[END_REF], is used for estimating future target positions defined in a prediction horizon. For this, we consider that all pursuers have information about the current position of the target and other agents in the inertial frame.

Target position ahead

Denote by x T (m) the target position, by v T (m/s) the target velocity and by m T (kg) the target mass. Then, using Newton-Euler, its movement is defined as

ẋT (t) = v T (t), vT (t) = 1 m T F(t) a T 0 (t). (3.1)
where F(t) defines the external force producing its movement. From (3.1), the future target-position, x T (t + ∆ T ), is

x T (t + ∆ T ) = x T (t) + ∆ T v T (t) + t+∆ T T (t + ∆ T -s)a T 0 (s)ds. (3.2)
Observe that equation (3.2) provides the desired value x T (t + ∆ T ), nevertheless it is not possible computed it directly because a T 0 (s) is not known for s ∈ [t, t + ∆ T ]. However, consider that a T 0 (t) and its first i derivatives can be estimated. Therefore, by Taylor, the function a T 0 (s) could be reconstructed, for Work in progress as of 20 th June, 2021 all s ∈ [t, t + ∆ T ], as a T 0 (s) ≈ âT 0 (s) a T 0 (t) + (st) ȧT 0 (t) + ... + (st) r r! a

(i) T 0 (t), (3.3)
with an error, h i (s) a T 0 (s) -âT 0 (s), bounded by

h i ≤ (∆ T ) i+1 (i + 1)! µ (3.4) being µ max η∈[t,s] {a (i+1)
T 0 (η)}.

From (3.2)-(3.4), the future target-position, x T (t + ∆ T ), can be estimated by

xT (t + ∆ T ) = x T (t) + ∆ T v T (t) + t+∆ T T (t + ∆ T -s) âT 0 (s)ds. (3.5)
with an error bounded by

x T (t + ∆ T ) -xT (t + ∆ T ) ≤ (∆ T ) i+2 (i + 1)! µ (3.6)
Therefore, for estimating the future target position, the drone needs to compute equations (3.3), (3.5) in real-time. The variables a T 0 , ȧT 0 , ..., a

ȧT i = l 1i (x T -xT ) + l 2i (v T -vT ), (3.7)
being xT , vT , estimate values of x T , v T , respectively; and âT 0 , âT 1 , âT 2 , ..., âT i estimates of a T 0 , ȧT 0 , äT 0 , ..., a (i)

T 0 , respectively.

Work in progress as of 20 th June, 2021

Interval of prediction (∆ T )

The ∆ T represents how far in time will be the target's prediction. For that we rely in the idea of time-to-interception, considering the current distance r iT and assuming a constant closing rate ṙiT (measured by the pursuer):

∆ T = r iT ṙiT , T max min (3.8)
The up-bound (T max ) is determined experimentally in order to avoid outliers errors in the target prediction.

Once obtained ∆ T , the predicted position xT x T (t + ∆ T ) can be determined and used in the pursuit. Consecutively, in the following sections the pursuit engagement will be described in relation to the predicted target xT , as illustrated in Figure 3.1.

Pursuit with prediction

Similar to Chapter 2, let us considering the guidance law for implementing a deviated pursuit (DPP):

f ψ i = K p * (ψ i -λ iT -α i ) (3.9)
where f ψ i is the control input for the heading velocity (ψ i ), and α 0 is the offset angle. Note that, considering (1.1), with the pursuer under (3.9), one necessary conditions for convergence of r iT are α i < π/2.

The selection of different offsets α can results in different non-crossing trajectories, as can be seen in Figure 2.4. These curvilinear trajectories can be used as pursuer's behavior to ambush the prey from different directions. Furthermore, the paths described in Figure 2.4 can be easily compared to the description of lionesses chasing trajectories given by [START_REF] Philip | Cooperative hunting in lions: the role of the individual[END_REF].

In order to create this ambushing behavior, let consider the following offset angle's function:

Work in progress as of 20 th June, 2021

α i =          αiT 0 * N j!=i δ ij (ξ i ,ξ j ) N -1 N > 1 0, N = 1
where N is the number of neighbors of the i-agent and δ ij is a summation function previously described in (2.3). In short, the outcome is similar to repulsive forces applied in the perpendicular direction of the LOS. Thus, δ ij can be written as:

δ ij        -1, if j is in the left side 1, if j is in right the side
The offset reference ( αiT 0 ), is the maximum offset angle that one agent can assume in a pursuit, and it is based in the prediction error (h i ), previously explained (eq. 3.4) and in the distance to target |r iT |.

αiT 0 = α 0 , tan -1 h i r iT max (3.10)
where α 0 is the minimum offset kept to avoid collision between two neighbors. The principle behind this is that the pursuers in the extremities will travel towards the extremes case in the target prediction, i.e., they will assume the higher error in the prediction and intercept the target there.

Simulations

One pursuer vs one intruder Figure 3.3 illustrates the performance when a pursuer tracks one intruder. The red circle is the predicted target position intersection, and the pink circle is the estimated error in the prediction. In this scenario, the target starts in the position (x, y) = (-7, 3)m and moves with constant velocity in its positive longitudinal axis. Observe that the estimation stabilizes and the pursuer is in the intersection target route after the third frame.

Work in progress as of 20 th June, 2021

Comparison with Group Deviated Pursuit

In Figure 3.4, two scenarios with three pursuers' pursuit are illustrated; one using the predictor scheme, Figure 3.4-top, and the other using the simple GDP, Figure 3.4-bottom. Similar to the previous case, the target assumes a constant positive velocity on its x-axis. Observe that when using the prediction scheme, the performance is improved, and the total trajectory traveled for each pursuer is smaller than for the first case.

Work in progress as of 20 th June, 2021 Work in progress as of 20 th June, 2021

Pursuit with non-cooperative agents

This section will address a different problem in multi-agents pursuit, where a single pursuer must track a target in an environment with non-cooperative agents. From the pursuer point of view, these non-cooperative agents can be seen as mobile obstacles, and then the problem becomes similar to the traditional obstacle evasion. In this strategy, instead of using conventional repulsive force to avoid the obstacles or find a route through an optimization problem, we propose a simple and reactive solution based on the "straight-line condition course." This principle is the same for the traditional family of navigation guidance PNG (Proportional Navigation Guidance).

This work was published in the 23rd IEEE International Conference on Intelligent Transportation System, under the title of Reactive drone pursuit and obstacle avoidance based on parallel navigation, in September 2020.

Brief preface and literature review

Tracking a mobile object in real time is a demanding task in the UAV field; it can be used in several applications, such as surveillance, cinema recordings, and sports broadcasting. Although several commercial drones already feature tracking the desired object autonomously using its on-board sensors, collision avoidance with mobile obstacles is still a drawback for its applications in more Work in progress as of 20 th June, 2021 complex scenarios, such as navigation in a big city. Besides, drones' futures applications will require multiple drones sharing the same aerial space while attending different tasks. Those non-cooperative agents must make fast decisions to avoid each other, as vehicles and pedestrians cross each other in a big city.

As reviewed in the Chapter 1, numerous examples of robotic applications have used GL for tracking. Moreover, its uses are the most varied, from autonomous navigation [START_REF] Belkhouche | Parallel navigation for reaching a moving goal by a mobile robot[END_REF][START_REF] Jung | Perception , Guidance , and Navigation for Indoor[END_REF], passing through autonomous landing [START_REF] Tan | Proportional navigation (PN) based tracking of ground targets by quadrotor UAVs[END_REF], until interception of intruder drones by ADDs [3,[START_REF] Thomas | Guidance Laws for Partially-Observable UAV Interception Based on Linear Covariance Analysis[END_REF]. However, unlike its original application, missile guidance, mobile robots' working area constantly also has the presence of obstacles, which can be mobile or static. However, for an effective GL in mobile robotics applications, obstacle avoidance techniques must also be incorporated into the system.

Many efforts have been made to conciliate the guidance laws to the opposite case, collision avoidance. One recurrent solution adds the use of changing modes switching between pursuit and avoidance task. In [START_REF] Belkhouche | A method for robot navigation toward a moving goal with unknown maneuvers[END_REF] and [START_REF] Belkhouche | Parallel navigation for reaching a moving goal by a mobile robot[END_REF], the authors considered an intermediate goal, a collision-free point, for the pursuer to go until the overcoming of the obstacle. One alternative is the use of the parallel navigation principle in the collision avoidance [START_REF] Pa Wilson | A line of sight counteraction navigation algorithm for ship encounter collision avoidance[END_REF], [START_REF] Matthew | Collision Avoidance and Navigation of UAS Using Vision-Based Proportional Navigation[END_REF], and [START_REF] Belkhouche | Reactive path planning in a dynamic environment[END_REF]. In [START_REF] Pa Wilson | A line of sight counteraction navigation algorithm for ship encounter collision avoidance[END_REF] the authors proposed a PNG based method Line of Sight Counteraction Navigation (LOSCAN) to avoid collision between ships; the algorithm proposes command to be executed by the sailor in order two avoid a collision.

Similarly, in [START_REF] Belkhouche | Reactive path planning in a dynamic environment[END_REF], the author introduces the concept of a virtual plane to identify the collision based on the rendezvous-course condition. In this plane, the dynamic moving obstacles are transformed into stationary objects, and through it, the navigable paths are obtained. In [START_REF] Matthew | Collision Avoidance and Navigation of UAS Using Vision-Based Proportional Navigation[END_REF] is proposed collision avoidance guidance laws based on the PNG principles to sense and avoid capabilities in a UAV, their solution for multiples target is based on objective-based cost functions to find the optimal path.

Work in progress as of 20 th June, 2021

In this work, we propose a navigation guidance-based solution for tracking and avoiding non-cooperative agents or obstacles. We extended the traditional Proportional Navigation Guidance (PNG) for collision avoidance of multiples obstacles, including mobile objects with random movements. This work differs from the previous in the following aspects: no need of changing mode between pursuit and avoidance, as in [START_REF] Belkhouche | A method for robot navigation toward a moving goal with unknown maneuvers[END_REF], [START_REF] Belkhouche | Parallel navigation for reaching a moving goal by a mobile robot[END_REF], in our strategy, the pursuit and evasion are components of the same guidance law;

does not rely on the optimization calculation as in [START_REF] Matthew | Collision Avoidance and Navigation of UAS Using Vision-Based Proportional Navigation[END_REF]; being a computationally fast solution and easy to implement.

A comparative of the proposed algorithm with respect to the pure pursuit is presented with numerical simulations. Different scenarios are provided for verifying the good performance of the proposed algorithm.

Problem statement

The challenge is to propose a solution for a mobile target's drone tracking problem in a dynamic environment where non-cooperative agents are evolving. This signifies that the pursuer drone must reduce its distance to the target (r iT ) and avoid the collision with other agents (r ij ).

The scenario is illustrated in Figure 3.6, the pursuer (blue circle) must track the target (red circle) while avoiding collision with the non-cooperative agents (grey circles) flying with unknown movements. The green zone in the picture illustrates the perception zone where obstacles outside will be ignored. All agents in this work have equal capacities and the same configuration. Also, the pursuer will have a velocity equal to or higher than the target and obstacles.

Pursuit and avoidance guidance law

In order to determine the steering law for f ψ i , let us first consider the PNG (Proportional Navigation Guidance), previously exposed in Chapter 2. This basics principle is to keep a constant bearing angle towards the target. Moreover, Work in progress as of 20 th June, 2021 a logical choice for a controller to nullify λiT can be defined as:

f ψ i = N * λiT (3.11)
where N represents the gain navigation constant and λiT is the angular rate of the bearing angle between pursuer and target. Remark that a constant bearing is a necessary condition for the collision as stated in (3.1), hence, for assuring the inequality of this condition, it must be enough to assure the non-convergence between two bodies, i.e., in a navigation scenario, the avoidance.

Therefore, the function f (x) = x -1 is chosen for assuring the inequality λij 0, and the following avoidance component for a guidance law can be proposed:

f ψ i = -k λij (3.12)
where λ ij is the angular rate of bearing angle and and k denotes a positive gain.

The negative sign assures that the pursuer heading in the opposite direction of

λ ij .
Work in progress as of 20 th June, 2021

Although the above equation can assure the non-collision with a single obstacle, it can be an inconvenient due the fact the distance is not taken into account, and in a multi-obstacles scenario a closer obstacle will not have priority over a further one. For overcome this issue a second term is proposed, where the control output will be weighted based in the inverse of its relative distance (d ij ). Thus

f ψ i = -k λij * σ R col -r ij R per -R col (3.13)
For this new term, R col and R per stand for the perception and collision radius, respectively. σ () function can be described as:

σ (x) =                0, if x > R per x, if R per ≥ x ≥ R col 1, if x < R col (3.14)
Observe from (3.13) the second term in the equation expresses a kind of collision coefficient which has its maximum value for distances smaller than R col and null value if the distance is higher than R per . Besides, this function increases the repulsion force for closer objects or to ignore others that are not in the eminence of collision.

Finally, the complete equation for target tracking and obstacle avoidance is proposed as

f ψ i = N * λiT + -k λij * j∈O * σ R col -r ij R per -R col (3.15)
where O defines the set of all non-cooperative agents (or obstacles) inside the perception area, see Figure 3.6. This equation combines the PNG (3.11) and the avoidance (3.13) generalized for the multi-obstacle case.

Simulations

To validate numerically our proposition, we implemented the proposed solution in Matlab, considering the agents modeled as a particle moving in a two-dimensional space and under the kinematics equations (3.16).

Work in progress as of 20 th June, 2021

The sample period, T e in simulations is 0 : 01s. For simulation purposes, the crosses represent the evader's path, while the circles illustrate the pursuers' path. The black arrows denote the heading orientation of each pursuer. The collision condition in simulation is the target be in a distance less than the capture radius R min = 0.3 m of one agent.

Fixed obstacles, fixed target In the following scenario, the pursuer must navigate between numerous static targets using the pursuit-avoidance strategy (3.15). The trajectories can be seen in Figure 3.7. In this figure, several episodes were taken considering different initial positions for the pursuers. Notice that the pursuer does not know the global location of all obstacles; therefore, the optimal path can not be assured. However, remark that this approach provides a smooth collision-free path towards the target using a simple and fast calculation algorithm.

Fixed obstacles, moving target A slightly different scenario is prosed here. Random obstacles were also placed, but now the target is executing a linear trajectory; see Figure 3 Moving obstacles: In this last scenario, the pursuer must track the target into an environment where non-cooperative agents are evolving. These noncooperative agents are moving with different velocities along the way of the pursuer towards the target, as can be seen in Figure 3.9. For this simulation, the agents only move in the x-axis; nevertheless, the result can be easily extended for non-cooperative agents moving in 3D.

In this scenario, the pursuer starts at position (x 0 , y 0 ) = (0, -6)m and displaces successfully between the non-cooperative agents toward the target positioning at (x T , y T ) = (0, 5)m. Observe, still from Figure 3.9, the good performance of the algorithm facing a complex scenario with multiples accelerating non-cooperative agents.

Observe that the principle is not a pair-wise repulsion force as a function of the distance from the pursuer to the obstacle. Instead, to avoid the nullification of bearing-angle rate pursuer-obstacle ( λiT ), consecutive, even if an obstacle is close to the pursuer, it will not change its course if they are not in collision route.

Work in progress as of 20 th June, 2021

For example, from the screenshots 4 in Figure 3.9, we can remark easily that even with the last obstacle passing close to the pursuer, i.e., inside the perception radio (R per ), it does not affect the route of the pursuer, once they were not in collision route.

Note also, the pursuer observes only relative and polar based information ( λiT ,d iT ) from the obstacles, which can be easier obtained from embedded sensors, such as camera and lidar.

The simulation results can be observed in the following link: https://youtu.be/ctrN42mLIUQ. Work in progress as of 20 th June, 2021

Flocking

This last section will tackle a slightly different problem from the previous ones; we will not study group-pursuit strategies but the collective herd movement. However, we will see that the flock's principle is very similar to the used in grouppursuit, which will lead us to propose a generalized guidance law, addressing at once the pursuit, obstacle avoidance, and flocking. Our flocking model is based on the principle of "parallel navigation," where the flock's individual tends to maintain a constant bearing angle about the estimated flock's centroid. Moreover, we show that a cohesive motion can be obtained with only relative and polar neighbors' information. This section's content is part of an article, still in the process of preparation, which will be entitled: Flocking, pursuit, and avoidance: a constant bearing guidance law for multi-agents. C. de Souza, P. Castillo, B. Vidolov.

Brief preface and literature review

Flocking is not an exclusive behavior of herd's and school's animals, such as sheep, birds, and fishes. Far from that, Vicsek and Zafeiris' survey [START_REF] Vicsek | Collective motion[END_REF] points out that collective, coordinated, and cohesive movements are present in practically all moving beings, including physical particles, bacteria, insects, or even humans.

Work in progress as of 20 th June, 2021

Considering this, we propose a flocking model that could be compatible with the previously proposed group-pursuit models. In this way, the pursuers, even not being in a pursuit action, could maintain a cohesive collective movement, and the transition between flocking to the pursuit, and vice-versa, could be done smoothly.

Before going any further, it is important to remind the distinction assumed in this thesis about "Flocking" and other collective movements. As pointed out in the introductory of this thesis, we refer by flocking "the fluid and cohesive motion promoted by agent-based and decentralized approaches, with homogeneous individuals, having no explicitly hierarchy and able to sense and act only locally." Differentiating so from all centralized or pattern formation approaches.

As exposed in the Chapter 1, there are numerous models for flocking in the literature. They range from the simplest ones, such as primary Vicsek particle [START_REF] Vicsek | Novel type of phase transition in a system of selfdriven particles[END_REF], passing through deterministic models, such as Cucker-Smale [START_REF] Cucker | Emergent behavior in flocks[END_REF], and its numerous variation and control-theoretical frame-work, such as in [START_REF] Olfati-Saber | Flocking for multi-agent dynamic systems: Algorithms and theory[END_REF]. Furthermore, several successful attempts have implemented into drones, such as [START_REF] Virágh | Flocking algorithm for autonomous flying robots[END_REF][START_REF] Vásárhelyi | Outdoor flocking and formation flight with autonomous aerial robots[END_REF][START_REF] Balázs | Adaptive leadership overcomes persistence-responsivity trade-off in flocking[END_REF][START_REF] Vásárhelyi | Optimized flocking of autonomous drones in confined environments[END_REF], where several dozen quadcopters can display amazing formations in an opened sky, see Figure 3.10. Nevertheless, the above implementation relies on communication between neighbors to share their position and velocities in the global frame. This assumption seems to be not very verisimilar compared to the observed real-life flock. This phenomenon is observed even in the most simplistic life, where no communication is identified. One realistic flocking should be obtained, relying only on relative information that could be easily obtained from embedded sensors, such as cameras or lidars.

Furthermore, the problem is those classic algorithms, in addition to the components of attraction and repulsion (attractive/repulsive pairwise potential forces), also require either term of heading's alignment, such as [START_REF] Craig | Flocks, herds and schools: A distributed behavioral model[END_REF][START_REF] Iain D Couzin | Self-organization and collective behavior in vertebrates[END_REF], or velocity consensus (viscosity term), such as in [START_REF] Olfati-Saber | Flocking for multi-agent dynamic systems: Algorithms and theory[END_REF][START_REF] Vicsek | Novel type of phase transition in a system of selfdriven particles[END_REF][START_REF] Cucker | Emergent behavior in flocks[END_REF]. These terms guarantee cohesion in the movement since the individual's velocity or orientation is decided based on the average of those measures of his neighbors.

Work in progress as of 20 th June, 2021

However, measuring the orientation, or the absolute speed of another agent, is not an easy task. What makes a large part of flocking work is to assume a communication between agents, sharing this information among themselves.

Therefore, we propose a solution where neighbors' velocity, or orientation, does not need to be measured. We rely on the centroid estimation of the perceived neighbors, and we apply "parallel navigation" towards it. The Flock Centroid FC is a feature that can be obtained by image processing since it can be stated as a simple optimization problem, such as exposed in [111]. In short, our approach proposes that velocity alignment is not a requirement for flock formation. Instead, alignment is the outcome when applying simple geometrical rules that will be described next. Let us consider the flocking problem involving N agents moving in a horizontal plane, such as illustrated in Figure 3.11. The agent's path is described by P i (t) = [x i (t), y i (t)], where (x i , y i ) is its Cartesian coordinates in the bounded work-space W ∈ IR 2 . The initial goal for each agent is to keep the group's cohesion while do not collide with each other. The agents are evolving under the non-holonomic equations (3.16) and their behavior is controlled by the dual input [f ψ i , f v i ], which are the angular rate and the linear velocity, respectively.

Problem statement

An agent perceives its environment in polar and relative coordinates, i.e., the distance r ix and a bearing angle λ ix , between the agent i and the object x. Polar and relative coordinates seem a coherent assumption since that information can be obtained from a common robot's embedded sensors, such as a camera and lidar. Besides, we suppose that agents can infer their heading (ψ i ) in the global frame, which is also a consistent assumption for robotic applications, considering electronic compass sensors.

To adopt more realistic conditions we also suppose a limited fiel-of-vision FoV , characterized by an opening angle β and a radius R F . The agent i is able to perceive only the neighbors inside FoV i . Finally, an individual must also be able to estimate its perceived flocking centroid FC i , which is the center of mass of all agents inside the field-of-vision FoV i , i.e., FC i = 1 N F j∈FoV i P j , as illustrated in Figure 3.11.

Finally, to realize more complex collective behaviors, we borrow the concepts of "informed individuals" from [START_REF] Iain D Couzin | Effective leadership and decision-making in animal groups on the move[END_REF], which consist of specific individuals who have the flock's mission. In a certain sense, they can be considered "leaders," but we kept its original homogeneity concepts, where one agent can not distinguish from whom is informed or not.

Work in progress as of 20 th June, 2021

Flocking model

Simplistic model

In this flocking strategy, the agent i regulates its orientation to keep a constant bearing angle to respect to FC i , i.e., λFC i = 0. For its turn, the linear velocity of i is regulated to keep a safety distance toward the closest agent. The simplest form for this flocking is stated below:

f ψ i = K f * λFC i (3.16a
)

f v i = u max * σ a r min -R saf R per -R saf (3.16b)
where K f defines a positive gain and λFC i is the bearing angle rate between pursuer and its observed centroid (FC i ). Also, R saf denotes the closest safety distance between two agents, and R per is the perception radius, which is the limit to how far the neighbor an agent will be perceived by another. This function makes the pursuer start to break from R per linearly until to achieve R saf . The

σ a (x) is a function defined in x ∈ IR | x ≥ 0: σ (x) =          0, if x > R per x, if R per ≥ x ≥ R saf (3.17)
Note that at this point, each agent tends toward the perceived center of mass, since under the PNG, if v i > v FC i then ṙFC i < 0.

Alignment In order to illustrate how the alignment of heading can be achieved using the law (3.16), let us consider two agents, a target T , and a follower i. The target, which can be seen as the informed agent, is executing a non-maneuvering trajectory with v T positive and constant. The follower i, is operating under the flocking law (3.16), and since T is the only member of the flocking, λ FC i = λ iT . Now, considering the relative kinematic equations described in (1.1), and considering that under (3.16), λiT → 0 and ṙiT → 0, thus, the system (1.1) becomes:

Work in progress as of 20 th June, 2021

v i * cos(α i ) = v T * cos(α T ), v i * sin(α i ) = v T * sin(α T ). (3.18)
where the clear solution are v i = v T and α i = α T , which indicates the alignment, i.e., same linear velocity and same heading for i and T .

Generalization: pursuit, avoidance and flocking

We proposed a unified guidance law responsible for the behavior of tracking, flocking and collision avoidance. The steering control input is given by the sum of the components:

f ψ i = f track ψ i + f avoid ψ i + f f lock ψ i (3.19)
The track component f track ψ i can be given by (2.9) or by other guidance law, such as in Sections 3 or 4. The avoidance component f avoid ψ i is stated in (3.12), and finally, the flocking component f f lock ψ i can be given by (3.16).

Leadership strategies can be established to conciliate the conflict between "flocking" and "tracking" tendencies. For example, just the closest agents can see the target, while others attempt to maintain cohesion with the group. Alternatively, heterogeneous weights can be given for the agents. Making in some agents a stronger track impulse(f track ψ i )over the cohesion impulse (f

f lock ψ i
), thus making predetermined leaders.

Simulations

Next, three scenarios are proposed to demonstrate the capabilities of our proposition. The simulations were done through Matlab, and the agents are implemented by using the kinematic equation (3.16).

Consensus achievement

In this scenario, 50 agents are placed randomly inside a square arena of 100x100. The agents have their velocity and turning rate controlled by the flocking law (3.16). In this first example, there are no informed Work in progress as of 20 th June, 2021 agents, and all agents are equally tuned. In order to show the effect of the field-of-view, we proposed three episodes for three values of opening angle β = π/4, π/2, π, 2 * π, while the FoV radius keeps the same, R F = 30 .

Work in progress as of 20 th June, 2021

In Figure 3.12, note that for smaller values of the field-of-vision (A and B), the transitory phase is longer, and we can see that agents travel longer trajectories until settling in the aggregation (more dense paths). However, for bigger values of opening angle (C and D), a consensus can be achieved faster since the agents traveled less (more sparse paths ). Nevertheless, we can see that more segregated groups are formed for bigger values, as pointed out by the blue circles in Figure 3.12-D. This segregation can be explained by the fact that increasing the opening angle also increases the influence of the closest neighbors. With that, even a neighbor even located in the "back" of the agent would interfere in its behavior. With smaller opening angles, the agent tends to "ignore" the neighbors more.

Work in progress as of 20 th June, 2021

Informed agents In this scenario, we illustrate the influence of a single agent over the whole. For that, 15 agents are also placed randomly initially; nevertheless, one of the agents is informed about the target position and follow it. The informed agent is the same along with all this episode.

In the first screenshot from Figure 3.13, the flocking agents are all aggregated, while the informed agent is following the target (red cross). In the second screenshot, we can see that some of the agents started to be influenced by the informed one, and gradually all the agents start to migrate. In the last screenshots, they all stop in a new aggregation. This pattern was observed several times in this simulation.

Note that, although the group movement is not uniform, one agent alone is enough to influence the positioning of the whole flock. We will see in the next simulation that the presence of more informed agents are sufficient to obtain a more uniform flocking movement.

Work in progress as of 20 th June, 2021

Cruising trajectory and obstacle avoidance In this last scenario, we propose to validate the generalized guidance law (3.19), where all agents have the same tendency of flocking, avoidance and pursuit. For that, 25 agents have been initialized in the bottom left part of the arena, with the following configuration of field-of-vision: β = 1.2π and R F = 25. The target is traveling in a linear trajectory with a constant velocity. Figure 3.14 shows the target (red cross) starting at the center of the arena and moving towards the up-right corner.

Henceforth, there are no chosen informed agents, but all agents tend to pursue the target if it is inside their field-of-view. Furthermore, target and obstacle (grey square) are also perceived inside only when inside the FoV .

Figure 3.14 we can see in screenshots 1 and 2 that the flock is cohesively following the target. Once the target overtakes the obstacles, screenshot 3, the agents progressively split into two groups. The two groups keep segregated until the obstacle does not interfere anymore in the formation, and gradually the two groups start to join again (screenshot 11 and 12).

Work in progress as of 20 th June, 2021

Conclusion

In this chapter, we went beyond the classical pursuit, and we investigated three complementary scenarios. We have tried different assumptions on the target's observation and different missions as well. Following a summary of the propositions of this chapter.

In the section Pursuit Target Prediction: we suppose that all agents knew the target's position and velocity during the pursuit. With that, we proposed implementing a predictive scheme to foresee the future location of the target. Furthermore, all the pursuit was then based on the prediction, and the positioning of the agents was based on the error estimated prediction error. We saw a consistent improvement in the performance compared to the simple Group Deviated Pursuit (Chapter 2). Although the need for target's global information is hard to implement in real-life, this methodology can be used as "groundtruth" to evaluate the previous pursuit approach, from Chapter 2. Furthermore, qualitative analysis in the pursuit's approaches will be provided in Chapter 5.

In the section, Pursuit with non-cooperative agents: a reactive algorithm for pursuit and collision avoidance for non-cooperative agents was proposed. We considered the scenario where a single pursuer had to track a target in an environment shared with non-cooperative agents. The navigation algorithm was obtained using the geometrical rule of parallel navigation and the pursuer target tracking and obstacle avoidance properties. The algorithm was validated in simulations for three different scenarios, including when the pursuer navigates into an environment where non-cooperative agents are evolving.

Finally, in the section, Flocking: we investigated the cohesive collective movement, and we conciliate it with the pursuit and avoidance previously seem. This new flocking approach is based on parallel navigation and dispenses the use of "alignment" or velocity consensus terms, which is one advantage for real-time implementation since the neighbors' velocity is not easy to be measured by ordinary onboard sensors. We validate our proposition through simulations, where the scenarios of aggregation, independent traveler, and cruising with obstacles were analyzed.

Work in progress as of 20 th June, 2021

Drone model

For the experimental validation, we choose the conventional quadcopter configuration, as illustrated in Figure 4.1. Due to its simplistic mechanics and capability to hover and perform indoor flying, the quadcopter has been a frequent prototyping choice in UAV research. Although the high-level guidance law's output could be applied into a vast range of vehicles, we have specially dedicated to quadcopter once it has been the source of criminal usage, as reported in Chapter 1 Furthermore, this configuration is also a recurrent choice for an anti-drone drone (ADD), as we pointed out in section 1.1.2. As exposed in [171], the quadcopter dynamical model can be obtained by representing it as a solid body evolving in a three-dimensional space under the action of the main thrust and three torques, see Figure 4.1. Therefore, two simplified drone models will be used along with this chapter, one in the inertial frame and the other in the body frame. The first one will be used in the task positioning estimation, where we are interested in the drone location in an inertial frame. The second one will be used in the drone's motion control since the inputs from the GLs are given in the body frame.

Work in progress as of 20 th June, 2021

Inertial Frame

Let us consider the following dynamical model:

ẍ = -sin (θ) 1 m u ÿ = cos (θ) sin (φ) 1 m u z = cos (θ) cos (φ) 1 m u -g φ = θ ψ I y -I x I x - I r I x θΩ + l I x τ φ θ = φ ψ I z -I x I y - I r I y φΩ + l I y τ θ ψ = θ φ I x -I y I z + l I z τ ψ
where the position coordinates are x, y, z and the orientation is given by the Euler angles; roll (φ), pitch (θ) and yaw (ψ). The distance between each motor to the gravity center of the vehicle is denoted by . The inertia of the vehicle in each axis is defined by I x , I y and I z while the inertia of the motor is represented by I r , the speed of the rotor is defined by Ω and u is the main thrust, τ i defines the torque control input.

Chain of integrators Therefore, the main dynamics in its linear form to be studied are the altitude, the longitudinal, and the lateral movements. Without loss of generality, it is well known that the altitude dynamics can be represented by two integrators in cascade and the longitudinal/lateral dynamics by four. This is a very common simplification, and it is justified since considering smalls values for the orientation; where the trigonometric function can be simplified as follow: sin(x) ≈ 1 and cos(x) ≈ x. Thus, the longitudinal drone's dynamic can be expressed as

ẍ = -θ, (4.1a) θ =u 2 (4.1b)
Work in progress as of 20 th June, 2021

In this cascade representation the control input u 2 is equivalent to τ θ . Similarly, the lateral dynamics of a quadcopter can be described as a chain of four integrators:

ÿ =φ, (4.2a) φ =u 3 (4.2b)
In this cascade representation the control input u 3 is equivalent to τ φ . Finally, the vertical dynamic can be described by a chain of two integrators, z = u, where u is equivalent to the main thrust.

Body frame

Considering the conventional quadcopter configuration showed in Figure 4.1. We use the Newton-Euler approach, where, the mathematical equations for the quadcopter in the body frame can be written as:

m v = F + R T F g η = B(η)Ω J Ω = τ -[Ω] × JΩ (4.3)
The bold letters represent vectors. F denotes the thrusts generated by the motors, F g is the vector force of gravity. η = [φ i θ i ψ i ] is the vector of Euler angles, Ω = [ω x ω y ω z ] means the angular velocity in the body frame. m indicates the mass of the drone and v = [v u w] T defines the velocity vector of the aerial robot in the body system, R describes the rotation matrix generated in the order yaw-pitch-roll. B(η) represents the matrix that relates the angular velocity and the derivative of the Euler angles. J is the inertia matrix of the drone.

[Ω] × means the skew symmetric matrix of angular velocity and τ defines the torques applied to the vehicle.

Work in progress as of 20 th June, 2021

Motion control hierarchy 4.2.1 Introduction

In Chapters 2 and 3 we proposed several multi-agent strategies in a planar environment. in this initial setup, for simplicity, we considered very simple the kinematics model, also known as the uni-cycle model. Simplified setups are very common in multi-agents, navigation guidance, and differential game analysis. Nevertheless, they do not provide a straightforward implementation into real-systems once they do not consider the robots' dynamic and robots' dimensions.

In this section, we will discuss the extension of the agent behavior into an aerial vehicle, considering the application of a quadcopter. In a few words, the robot behavior is given by a multi-layer controller, as illustrated in

Work in progress as of 20 th June, 2021

This multi-layer control has a similar composition with the motion control hierarchy used by marine surface vessels, as the one described in [START_REF] Breivik | Guidance laws for planar motion control[END_REF]. In the higher level is located the Guidance Law, which gives the velocities and turn rates inputs f vi , f ui , f wi and f ψi . Following, we have the Safety constraints level, which are responsively to add virtual constraints such as delimited workspace, maximal velocities and close-range collision avoidance. The command from this level are expressed by f saf vi , f saf ui , f saf wi , and f saf ψi . Next, the Motion constraints level is a crucial component which will provide drone's compatible commands, f θi , f φi , f ψi and f żi , which in this case are the expressed in angular position. Finally, the Attitude control is the lower level control, which gives the thrust and torques inputs to the physical system u 1 , u 2 , u 3 and u 4 . in the following, more detailed description about each layer is given.

Higher layer -GL

The pursuit algorithms (or the guidance laws) are responsible for the high-level, or navigational control. It is basically composed by four command laws: f vi , f ui , f wi and f ψi , which are, respectively, the references for forward, and lateral velocities, vertical acceleration and angular rate in yaw. Nevertheless, the planar guidance proposed in Chapters 2 and 3, proposes only guidance law for the turn rate f ψi , since they consider linear velocity constant f vi = constant, and non-holonomic constraints in the 2D plan, resulting in f ui = 0 and f ẇi = 0.

One exception is for the pursuit strategy based on Reinforcement Learning, where besides the turn rate, law for forwarding velocity f vi is also provided. Furthermore, as will see further in the section 4.2.4, an extension for the vertical plane pursuit will be proposed, and a law for the vertical displacement f ẇi will given.

Safety layer

This layer is responsible for the material's integrity during the experimental validation and under laboratory conditions. Therefore, additional assumptions are considered, such as knowledge of all agents' position and velocity in the Work in progress as of 20 th June, 2021 inertial frame. With this complete knowledge of the system, we can assure collision avoidance by adding several virtual constraints, such as delimited workspace, maximal velocities, and close-range collision repulsion. Although the ambushing trajectory, generated by the pursuit algorithms (higher layer), has collision avoidance term, they can not guarantee full collision-free once they do not consider all degrees of freedom, dynamic, and dimensions of the robot application. Before going into details, let us consider the illustration from Figure 4.3. In the right picture is illustrated three drones inside a circular arena of radius R are , and the origin of the inertial frame is the center of the arena. Also, r i and r j are the position vector of the the agents i and j. Furthermore, in the left picture, a drone with its body velocities (v i and u i ) and two concentric areas are illustrated. The inner circle is the "collision imminence area," characterized by the radius R col . If a neighbor is inside this area, the repulsion will be maximized. The outer circle is the "safety area," characterized by the radius R saf . The safety layer will ignore all neighbors outside this zone once they propose no collision risk.

The Safety layer is composed by the following laws: saf ui ] T we borrow the concepts of [START_REF] Virágh | Flocking algorithm for autonomous flying robots[END_REF] and [START_REF] Vásárhelyi | Outdoor flocking and formation flight with autonomous aerial robots[END_REF], where their "flocking law" was given by the sum of several virtual forces: repulsion, friction, and virtual barriers. Nevertheless, in our case, the virtual forces are active only when objects are inside the safety area, such as indicated in Figure 4.3.

f saf vi , f saf ui , f
Without further ado, the safety law for the velocities is computed as following:

f saf vi = f pur i + f wall i + j i f rep ij + f f ric ij θ R saf -|r ij | (4.4)
where

f pur vi = [f vi f ui ]
is the velocity reference from the higher level (guidance law). It is important to notice that as in the guidance law consider non-holonomic constraints, there is no lateral velocity, consequently f ui = 0. Besides, f rep ij and f f ric ij denote the repulsion and friction components for each neighbors. The therm f wall vi is the component of repulsion from the virtual limits, the "virtual walls", as illustrated in Figure 4.3-Right. Also, r ij is the vector connecting agents i and j. And finally θ(x) is a function that the output is 1 if x ≥ 0 and 0 if x < 0.

The repulsion component from an agent j towards another agent i is computed as follow

f rep ij = D(r ij -R saf ) |r ij | θ(R saf -|r ij |), (4.5)
where D is a coefficient for strength of the repulsion.

For its turn, the friction component from two agents is calculated as follow

f f ric ij = C (v j -v i ) (max{R col , |r ij |}) 2 , ( 4.6)
where C is the coefficient of alignment, which tends to align the agents' velocity as explicated in Chapter 1. In [START_REF] Virágh | Flocking algorithm for autonomous flying robots[END_REF], the authors pointed out the importance of the alignment term to "damp" the agents' response for the repulsion forces in interaction. In contrast with the upper layer (GL), this layer requires knowledge of neighbor velocity in the inertial frame, which is justifiable considering experimental validation laboratory conditions.

Work in progress as of 20 th June, 2021

Finally, the "virtual wall" component are given by:

f wall i = sig |r ij |, R are , R wall r i |r i | , ( 4.7)
where R are is the radius of the arena, and R wall is the thickness of the virtual barrier, as illustrated in Figure 4.3-Right. Besides, sig(.) is a Sigmoid function, which can be defined as

sig (x, R, d) =                0, if x ∈ [0, R] sin π 2 (x-R)-π 2 +1 2 if x ∈ [R, R + d] 1, if x > R + d (4.8)
Please notice that the safety velocity, such as calculated in (4.4) is only a requirement for experimental validation in laboratory conditions, where workspace is limited, and the drones travel very close to each other. In a realworld application, this safety velocity based on flocking will be much less needed since the pursuer's runnable area is much greater than its "safety area."

Motion constraints

The third level is responsible for transforming the safety velocity input from the upper levels into commands compatible with the robotic platform, which in this case are commands for the angular position: f θi , f φi , f ψi and acceleration in zaxis f żi . However, the robot may have more degree of freedom than the GL's output, which is clearly the case of the quadcopter equations (4.3), and the the kinematics model (3.16) considered in Chapters 2 and 3. Therefore, to adapt the quadcopter to the GL commands, two motion constraints model are proposed: non-holonomic and heading-to-target constraints.

Work in progress as of 20 th June, 2021

Non-holonomic constraints

Possibly the most intuitive way to apply the GL commands into quadcopter configuration would be to impose the non-holonomic constraints into it. It can be done by nullifying the drone's lateral velocity and forcing it to act only in turn around the z-axis (ψ) and in the forward velocity. With that, the quadcopter states, exposed in (4.3) will have the following tendency:

v i → f saf v i , u i → f saf u i , w i → 0, ω zi → f ψ i (4.9)
Considering the quadcopter's simplified model (4.2), and applying the same simplification into the the model 4.3, we can see that vi ≈ -θ i and ui ≈ -φ i . Then, a simple law to assure the condition 4.9 can be:

f θi = K θ (f saf vi -v i ), f φi = K φ (f saf ui -u i ), f ψi = t 0 f ψi dt, (4.10)
Note that, for instance, we are not yet considering the vertical displacement.

Work in progress as of 20 th June, 2021

A section dedicated to the vertical pursuit plan will be given further, see section 4.2.4. For the moment, we consider that the quadcopter is already stabilized in altitude and its vertical velocity is null (v z = 0). This motion constraint has an interesting property while considering the frontal camera as a navigation sensor: the drone will displace only in the direction of the field of vision, the sensed area, tightly coupling the perception to the movement. For the same reason, this kind of drone motion is commonly observed in FPV piloting, where the only information for the pilot is the frontal camera image, Figure 4.5. In this case, the remote-pilot tends to cancel the drone's lateral drift by combining a roll movement while turning the drones heading.

Heading-to-target

In this second motion type, the drone will regulate its heading always toward the target (ψ i → λ iT ) and will keep a positive velocity (v i ) in that direction. The NGL command will be given as lateral acceleration f a , perpendicular to the LOS. In the quadcopter, the lateral acceleration in the body frame is represented by the state u.

v 1 → f saf v i , ui → f a , w i → 0, ψ i → λ iT (4.11)

Work in progress as of 20 th June, 2021

This technique's advantage is to improve the target's perception once it is always in the center of the image plane. Furthermore, it allows a straightforward implementation for GL that provides the command in terms of normal acceleration of f a , rather than the turn rate of f ψ .

Considering the application of the PNG in terms of f a (3.11), once the lateral acceleration of the drone is already perpendicular with the line-of-sight (LOS), this configuration allowed a direct implementation of the law, which is also called True Proportional Navigation (TPN) [START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF].

Altitude controller

In order to establish a guidance controller for the vertical plane, let us at first consider the three-dimensional engagement between a pursuer and an evader as proposed in Figure 4.6. The straight line connecting them, the Line-of-sight (LOS), is denoted by r iT . The angle formed by the LOS and the inertial axes x and y are denoted here by λ v iT and λ h iT , respectively. The vectors v i and u i are the straightforward velocity and the upward velocity in the body frame. ψ i represents the yaw of the drone i, and θi is a virtual pitch determined by the composition of the v i and w i .

Therefore, let us consider a Deviated Pure Pursuit (DPP) guidance law for the vertical plan:

f θi = K p * θi -λ v iT -β , ( 4.12)
where θi defines a pitch angle and K p a positive gain. The LOS angle now is defined in the vertical plane (λ iT ) and the offset angle is called β.

Since for a quadcopter, the displacement in the vertical plan is function of the thrust F (see Eq. 4.3) it is necessary a function to relate F to the turning rate output f θi from (4.12). For that, we consider a virtual pitch that is equivalent to the angle between the vector v and the horizontal plane. It can be given by θi = tan -1 v i w i (4.13)

Work in progress as of 20 th June, 2021 For safety navigation, we consider that the straightforward velocity of the pursuer v i is changing so slowly that can be considered constant by some time instants; therefore, the derivative of the expression the above becomes

θi = 1 1 + ( w i v i ) 2 * ẇi v i (4.14)
where ẇi is the vertical acceleration of the pursuer i. Therefore, from (4.14), and considering f θi as the virtual pitch rate, it is clear to write

f ẇi = v i + w 2 i v i * f θi , ( 4.15)
where, f ẇi is the control input for the vertical acceleration.

For avoiding singularity, v i is imposed to have a minimum value in the interception part. Note from the above that the thrust control is not based on regulation with the target altitude, but with its relative polar coordinates, which can be easier obtained from an on-board camera.

Work in progress as of 20 th June, 2021

Material

In this section, we described the hardware used for the experimental validation. We describe firstly the indoor flight arena, the localization system, and the used drones. Finally, we briefly explain the adopted framework, and we expose the software architecture.

Hardware

Flight arena For real time validation, we had at disposal two flight arenas, one indoor and another outdoor. However, except for a few experiments with data-fusion, most of the results exposed here were done in the indoor flight arena. Figure 4.7 shows a picture of four drones inside the indoor arena, also called "volière." It was chosen due to several advantages, such as low-price, robustness, and easy maintenance. The prototype has already several sensors embedded from the factory. It supports an average of 400g of payload, which can be used by a GPS receiver for some applications. In this work, the drones had the original firmware erased, and they have been "flashed" with FL-Air compatible firmware, developed at Heudiasyc. This operation allowed low-level operations once we could have access, for example, to raw data from the sensors and write PWM signal directly to the motors. It has an interesting configuration: an onboard computer (a dual-core processor with CPU quad-core), ultrasound and pressure sensors, 3-axis gyroscope, accelerometers, and magnetometer GNSS chip-set. Similar to the previous one, it is also compatible with FL-Air. However, along with this thesis, we have used the Bebop under two configurations, using FL-Air and using its original set-up. For this last one, original firmware and communicating with a remote computer using the bebop autonomy SDK [START_REF]Bebop Autonomy ROS Driver for Parrot Bebop Drone[END_REF], which support ROS (Robotic Operating System) [START_REF]ROS (Robot Operating System[END_REF].

FL-Air

The Framework libre Air, available in [START_REF] Ros | Fl-AIR -Framework libre AIR[END_REF], is a framework written in C++ for the development of UAV applications. This tool is composed of different applications running in a Linux system and communicating via socket.

Work in progress as of 20 th June, 2021 One important application is the World Simulator, which allow a reliable evaluation before using the robotic platform. Furthermore, it has a customizable ground-station, which allows on-line parameters tuning and real time flight data checking. More details about the platform can be found in the web-page [START_REF] Ros | Fl-AIR -Framework libre AIR[END_REF] or in [START_REF] Roberto | CUSCUS: An integrated simulation architecture for distributed networked control systems[END_REF].

Although this frame-work is sufficient for most applications, some high-level projects (involving Neuronal Networks, for example) were not supported by FL-Air. Therefore, since we had different design needs, we chose to use alternatives hardware architectures.

Architectures

The following two software architectures were used along with this thesis. The first one, onboard computation, has all the control architecture levels implemented inside the drone. The second one, remote computation, has its high-level guidance is calculated by a remote computer.

On-board computation

In this configuration, all layers of the motion control, described in Figure 4.2, are implemented in the embedded processor of the drone. This configuration is a fast solution for simple algorithms, which does not require too much computational resource. Furthermore, it is the standard configuration by using the framework Work in progress as of 20 th June, 2021 CHAPTER 4. Implementation and Experiments Fl-Air, and all the code was implemented in C++.

Most of the validation using guidance laws based in geometrical rules (such as in sections 2.2 and 2.3 were implemented in this configuration. Nevertheless, the embedded processor was not sufficient to implement the neuronal network policies for section 2.4, which required the next architecture development.

Remote computation

In this configuration, the high-level motion control is calculated in a remote computer and transmitted to the drone by wireless, using UDP protocol. A ROS node is responsible for managing the Optitrack server and the NN policies. A scheme of this implementation can seem in

State Estimation

To deal with the state measurement's recurrent problems, such as loss and degradation of data, we implemented a Kalman Filter (KF) scheme to stateestimation. The objective is to have a reliable position estimation considering degraded data from a GPS or motion capture system. In this section, we describe the filter design, discussing the models and the composition of the matrices. We propose a dynamic observation matrix C, which would vary according to the data's availability.

These results were published in our work entitled: Enhanced UAV pose estimation using a KF: experimental validation. C. de Souza Jr, P. Castillo, B. Vidolov, R. Lozano. Presented in the International Conference on Unmanned Aircraft Systems, Dallas -US, 2018.

Related work and problem statement

One of the main problems for outdoor navigation is to locate with reasonable precision the aerial vehicle. In diverse conditions, such as a vehicle is hovering or close to the buildings, the GPS may not offer the desired precision [START_REF] Orton | Relative Navigation of Micro Air Vehicles in GPS-Degraded Environments[END_REF]. Furthermore, the GPS signal can be degraded or lost (non-available) due to a series of reasons as boarded in [START_REF] Hornbostel | Propagation problems in satellite navigation[END_REF]. Currently, drone applications demand a high precision level of sensory because it is an unstable natural system that demands a high frequency of closed-loop control.

A very popular approach for improving vehicle pose estimation is the Kalman Filter algorithm (KF). Its intensive uses in robotics navigation go from land vehicles, marine robots to spacecraft. The use of KF for attitude estimation in aerospace applications has been extensively explored since the Apollo project (1960) to nowadays. In [START_REF] Mohinder | Applications of Kalman filtering in aerospace 1960 to the present [historical perspectives][END_REF] an interesting review that boarded the evolution and maturation of this application in the aerospace scenario is presented. For UAV's application in an urban environment, the Kalman Filter has been implemented mainly to improve localization estimation. In [START_REF] Yun | Enhancement of GPS signals for automatic control of a UAV helicopter system[END_REF], the authors implemented a KF together with a complementary filter to lead with the discontinuity in the GPS output while navigating in a big city.

They got interesting results in the pose estimation; however, they only applied it in an open-loop experiment. In [START_REF] Vago | Outdoor waypoint navigation with the AR.Drone quadrotor[END_REF] and [117], the authors implemented an outdoor navigation controller using an Out-of-sequence KF to deal with the problem of GPS latency. They validated their experiments with outdoors closedloop flights, although the control architecture is computed in a ground station and not into the drone (on-board).

In this work, a Kalman filter is used for improving UAV pose estimation. Two real cases were studied: loss and degradation of data. The solution is based on fusing data from an INS and a localization system with a linear and low computational cost KF applied to the quadcopter kinematics equations. Several flight tests were realized using a camera motion system emulating some commons problem of a GPS.

The localization problem is pragmatically boarded in this work. We consider low power signals arriving at the drone, consecutively being attenuated and shadowed by buildings or vegetation. For more details about this problem, see [START_REF] Nuredini | Bio-inspired Obstacle Avoidance: From Animals to Intelligent Agents[END_REF], where the authors described a detailed overview of propagation problems in GPS.

Hereafter, the degradation is the nomination for the diminution of precision in the data. It is still available meanwhile, in this case, having normal noise. The loss is the non-availability of the data. In our study, it is considered momentarily loss, typically an issue of communication. Both problems are firstly added in simulation using Matlab, and then they are implemented in real time. In this last case, a data problem routine was implemented into the drone firmware, and the ground-station provides its activation with the joystick.

Evolution model

As exposed in section 4.1, the dynamic model of a quadcopter (4.1.1) can be simplified by decoupled chains of integrators. Let us consider the simplified drone's longitudinal dynamic (4.2), which can be represented by chain four integrators. Therefore, (4.2) can be expressed in a state-space representation in a discrete domain:

Work in progress as of 20 th June, 2021 where X = [x, ẋ, θ, θ]. The discretizated state-space model was done using Ho-Kalman methods as described in [START_REF] Ho | Effective construction of linear statevariable models from input/output functions[END_REF]. The transition matrix A can be checked bellows:

A =                  1 T s -T 2 s /2 -T 3 s /6 0 1 -T s -T 2 s /2 0 0 1 T s 0 0 0 1                 
.

Similarly the control matrix can be seen below:

B = -T 4 s /24 -T 3 s /6 -T 2 s /2 T s T .
In both matrices, the terms concerned with derivatives with order superior to 1 (as T 2 s /2 and T 3 s /6) are neglected in real time implementation. Since the sample time is short (T s = 0.05), those terms become insignificant in the final results. We consider u 2 k = 0; nevertheless, the input for the translational movement considered here will be the angular acceleration, which is not measured.

Kalman filter equations

The KF filter is a recursive predictor/estimator of optimal gain primarily used in many engineering sectors. The algorithm is divided into two steps, estimation and prediction. The first stage computes the estimation of the states, X k|k , and the covariance error, P k|k . The mains equations of this step are

X k|k =X k|k-1 + K(Y k -C * X k|k-1 ), (4.21a
)

P k|k =(I -K * C)P k|k-1 (I -K * C) T + K * R * K T (4.21b)
Using (4.21) the estimation vector X k|k can be estimated taking account the last prediction X k|k-1 and the current measurement Y k . It takes the weighted average based on the Kalman gain K.

Work in progress as of 20 th June, 2021

For the error covariance estimation P k|k it was used the Joseph's form (4.21)b. For calculating it, the equation takes into account the last value P k|k-1 , the matrix R (sensor noise covariance), the matrix C (observation), and the Kalman gain.

The second step is characterized by the calculation of the predictions (X k|k+1 and P k|k+1 ) and the new updated Kalman gain. Basically, it takes into account the updated estimation (X k|k and P k|k ) and predicts the next sample step, as the equations below :

X k+1|k =A * X k|k1 + B * U k , ( 4.22a
)

P k+1|k =A * P k|k1 + Q, (4.22b) K =P k|k-1 * C T * (C * P k|k-1 * C T + R) -1 (4.22c)
In (4.22), the next step prediction for the states X k|k+1 and error P k+1|k are calculated. Besides, the Kalman gain K is updated.

KF matrices and tuning

The complete states vector applied to this filter are X = [x, ẋ, θ, θ, y, ẏ, φ, φ, z, ż], where (x, y, z) denotes the three dimensional position, ( ẋ, ẏ, ż) defines its velocities and (θ, φ, θ, φ) corresponds to the Newton's angles of pitch, roll and its consecutive rates. The yaw angle is considered previously controlled and aligned to the inertial frame.

Evolution model matrix: A describes the drone dynamics in a discrete statespace representation. Details about its development can be checked in the section 4.1. For our dynamics, this matrix has the following form

The states measurements matrix Y is the input of the KF, composed by the state's measurements. For knowledge, in the following experimental validation, the localization system's positions and velocities are given by the OptiTrack; besides, the angles and angular rates are provided by the Inertial Measurement Unit (IMU).

Work in progress as of 20 th June, 2021

A =                                                 
1 T s 0 0 0 0 0 0 0 0 0 1 -T s 0 0 0 0 0 0 0 0 0 1 T s 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 T s 0 0 0 0 0 0 0 0 0 1 -T s 0 0 0 0 0 0 0 0 0 1 T s 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 1 T s 0 0 0 0 0 0 0 0 0 1

                                                 .
The observation matrix C, describes the observability of the system. For instance, all the states are considered observable, which results in an identity matrix C 10×10 . Dynamic changes in this matrix are going to be exposed in the case of data loss.

The measurements covariance matrix R is a diagonal matrix (since the sensor errors must be not correlated) composed by measurement variance of each sensor. For example, when using an OptiTrack for positioning, the variance was set as 0.1 m, the orientation angles (using an IMU) to 0.1 rad, and the altitude (using an ultrasound) to 1 m due to its noisy behavior.

The noisy covariance matrix Q is harder to determine, and normally its definition is subject to trial and error. Therefore, the covariance for the positions (x, y, z) and the angles (θ, φ) were set to a small values (such as 0.1), because in the model, they are results of a simple derivative. The covariance for the velocity ( ẋ and ẏ) were set higher values like 10, dues their linearization that gives less reliability in the model. The angular rates were set to a high value as the system's input is unknown by the predictor.

The initial values of X (states estimation) were set all to 0 and the initial values of P (error covariance matrix) were set to 100, making P an identity matrix (10x10) multiplied by 100.

Work in progress as of 20 th June, 2021

Altitude estimation

The following section, which is not present in the original work, presents an alternative for the altitude estimation when barometer and ultrasound sensors are available. It comes from the need to have precise estimation when flying over irregular surfaces and conciliate the measure of two very distinguished sensors.

Measurements:

The barometer gives a pressure measurement that can be easily converted to an absolute altitude measure (z bar ). The output of altimeter sensors (z son ), commonly based in sonar, provides a measurement of the relative altitude z k . Furthermore, if available, we can also use vertical acceleration measurements from the common accelerometers as the system's input u z k .

Model: Now, let us consider the following evolution model for the vertical dynamics of a drone: From that, the matrix C for the Kalman scheme can be easily determined.

z k+1 = z k + żk ∆t + ∆t 2 2 u z k , z gnd k+1 = z gnd k , żk+1 = żk + u z k ∆t
Note that, the relative altitude z k is being measured explicit by zson and implicit by zbar . Allowing the system to have redundancy of an information by considering an intermediate state z gnd k , which is not relevant for the control of system.

Experimental results

The proposed methodology was applied in real time in our prototype. Several flight tests were carried out with different scenarios. All those experiments were realized in an indoor environment assisted by a motion capture system (OptiT rack). It is well known that this kind of sensor measures an object's pose; nevertheless, our idea was to emulate a GPS sensor. Thus, the measures coming from the OptiTrack system were degraded with noise and data lost. Also, the data frequency information was modified to approximate the frequency measurement of a GPS.

The algorithms were implemented in our framework, FLAir, and all the code was written in C + +. In this architecture, all the algorithms are computed on-board, keeping the ground station just in charge of analyzing the system and acting in emergency cases. The closed-loop system's sample time is on average 5.2ms, and it is transmitted to the ground station by a local WiFi network also in the same frequency. Three scenarios were boarded: data lost, data degradation, and low-frequency information.

Work in progress as of 20 th June, 2021

In all cases, four way-points were given as desired coordinates, the sequence is given by (-3, -3)m, (-3, 3)m, (3, 3)m, (3, -3)m forming a square in the horizontal plane, as one can check in Figure 4.12. In this figure, the first test is showed; here, the quadcopter tracks the desired coordinates in ideal conditions (a good measurement of the position).

Data lost

In this first scenario, the drone losses for some periods its position information.

As explained in the previous section, the loss is detected, and the observation matrix was adapted for a while. For our experiment, the indoor system is "advertised" by a flag (U AV lost) in the complete loss of the camera tracking, then it was used as criteria to modify the system.

As results of this scenario, the two following graphs expose the filtering estimation in Figures 4.13 and 4.14. The first one, Figure 4.13, successive loss of 1 second on average were applied to the drone while it was flying. The data loss was applied manually through the ground station. During that, the filter tends to estimate those not observable states more based on the model. We can observe its linear behavior of velocity and position estimated. For brevity, only graphs in the x coordinate are introduced; nevertheless, similar performance is obtained for the other coordinates.

Work in progress as of 20 th June, 2021 The following graphs in Figure 4.14 exposes results of the same case but with a real complete fail from Optitrack. A tunnel was improvised, seeking to deteriorate and cause a complete loss of the cameras' target. Observe in graphs that the positioning system keeps the last possible position till the object recovers it. It is also clear to verify the overshoot caused by the velocity provided by the system. It happens since the positioning system does not measure the velocity but provides it by a discrete derivative of the position. Notice that even if the quadcopter lost its position, the KF can estimate it and ensure a stable behavior in the closed-loop.

Position degradation

This second scenario considers the degradation only in the position data. Although the Optitrack system also provides an estimation of velocity, in this case, it was not degraded. Gaussian noise with a mean of 0 and a standard deviation of 5m was added to the OptiTrack position measurement, see Figure 4.15. The noise was implemented using a standard class template distribution for C + +, and the ground station activates it. As explained previously, the noise was applied while the drone was following the desired coordinates.

Work in progress as of 20 th June, 2021 Observe from Figure 4.15 that when adding the noise, the data coming from the positioning system is severally degraded (solid gray line), and when applying the filter, the position is recovered.

The strategy for dealing with data degradation was identifying the signal's quality and adapting the covariance matrix. In real applications, this quality information can be provided by the quantity of camera (in the OptiTrack case) or, in the GPS case, by another quality indicator. Figure 4.16 illustrates three quadcopter performances: The first one, the vehicle response in ideal conditions and non-KF (solid red line). The second one, the black dotted line indicates the quadcopter response when tracking the desired coordinates with data degradation and without KF. Finally, the blue dashed line presents aerial vehicle behavior with data degradation and using the KF. Notice here that the performance is recovered. Finally, the blue dashed line presents aerial vehicle behavior with data degradation and using the KF. Notice here that the performance is recovered.

Work in progress as of 20 th June, 2021

Low frequency information

In this scenario, a non-synchronized measurement Kalman Filter is proposed. The idea is to emulate a GPS performance in real applications since the arrival time, and the frequency of the sensors (GPS and IMU) are not synchronized.

Being the GP S frequency from 5 to 10Hz and the IMU close to 200Hz.

Work in progress as of 20 th June, 2021

Our strategy was to set pose as non-observable in matrix C, when there is no GPS available. With that, the system keeps the estimation basing on the IMU input and the evolution model. The position information given by the OptiTrack system is upgraded with a frequency of 3Hz. This frequency value is normally even lower than a standard one provided by a low-cost GPS for drones.

Note in Figure 4.17 how the KF recovers the position in a case of low frequency and no synchronized GPS data. Some experiments of this work are available in a video in the following link: https://youtu.be/k0FobmF7Fv0 Figure 4.17 -Quadcopter response in a case of low frequency data information. The position provided with 50Hz by the OptiTrack was reduced 3Hz.

Altitude estimation

In this last scenario, we expose the results obtained by implementing the KF scheme exposed in Figure 4.18. The experiments were conducted in the outdoor arena, and the quadcopter had to take over, staying in a position hold for a few seconds and then landing again. The plot with the measurements and the estimation are available in Figure 4.18.

Work in progress as of 20 th June, 2021 In Figure 4.18 we can see that the altimeter provides a very noisy measurement when the drone is flying above 2 meters approximately. In contrast, the barometer, although less precise, does not suffer from this altitude effect. Finally, we can see by the fusion of these two sensors that a smooth estimation can be obtained by applying the KF scheme section 4.2.4.

Work in progress as of 20 th June, 2021

Experiments with Pursuit

In this section, we present an experimental validation -proof of concept -for several group-pursuit strategies. The validated strategies namely are Group Deviated Pursuit from Chapter 2, Three dimensional Pursuit from section 4.2.4

and Reinforcement Learning Pursuit from Chapter 2. However, all of those highlevel guidance law were applied in the the real time quadcopter by using the multi-layer control scheme, proposed in section4.2.

Group Deviated Pursuit -GDP

Three scenarios are proposed to validate experimentally the DPG (2.5). In the first one, the intruder drone remains in a fixed position, and three pursuers drones track it. In the second scenario, a single evader travels toward the target, passing by two static obstacles. Finally, in the last scenario, the target, moving with a user's random movements, is tracked by three pursuers aerial drones.

Three pursuers and one static target

In this first experiment, a target is placed in the ground with a distance of 6m from the pursuers. The pursuers are placed in an initial triangular formation with a distance of approximately 2m from their closest neighbors. This experiment aims to reproduce the simulation pattern (see Figure 3

One pursuer, two obstacles, and one static target

In this scenario, we consider just one drone in the bottom-left side of the flight arena and a fixed target located in the upper-right position, as can be seen in Figure 4.20. Two static obstacles were placed in the trajectory from the drone to the target. The goal of this experiment is to highlight the obstacle avoidance properties included in the proposed algorithm. The parameters used in this experiment are done in Table 4.1. The agents "detect" the obstacles when they are located inside an interaction radius, R int < 2m and inside of its field-of-vision of 120 o . These parameters are different from those used in the cooperative pursuit, where the range-of-interaction is bigger, and no limitation of the field of vision (360 o ) is imposed.

The difference in scenario 2 concerning the others is due for two reasons; firstly, the idea is to use onboard cameras to detect the obstacles with a limited vision and range field. Second, impose a small deviation angle, α, is enough to Work in progress as of 20 th June, 2021 reproduce a collision-free trajectory in a pure obstacle avoidance case, while in the cooperated pursuit, bigger deviation angles are necessary to increase the covered area by the fleet. In Figure 4.20, the algorithm performance, when tracking a static target with obstacle avoidance, is illustrated. Notice here the good performance of the control scheme.

Three pursuers and one faster intruder

In this last scenario, an operator manually controls the target, trying to get away from the three pursuers. In contrast to the scenario presented in simulations, here, the goal for the pursuers is not to intercept the target but to keep a safe distance from it. Besides, to ensure our drones' security and for limitations of the workspace, the target will fly in a different plan than the pursuers, choosing z d = 0.8m for the intruder and z d i = 1.7m to the pursuers. The parameters for this flight are also exposed in Table 4.1.

In Figure 4.21, some snapshots are introduced to illustrate aerial drones' performance pursuing the faster intruder. This figure was done using real data coming from the experiment; the figure is composed of 9 screen-shots taken with an interval of 2.75s from one to the other.

Work in progress as of 20 th June, 2021 The collision avoidance property was implemented only in the pursuers. Since the evader was piloted manually, there were no guarantees of free-collision, and for this reason, the target flew in a different plane from the pursuer. Nevertheless, the virtual wall was the only property implemented into the evader to avoid crashes and plotting purposes. These flight tests can be appreciated in : https://youtu.be/Kr3UFtZ-dd8

Work in progress as of 20 th June, 2021

We can appreciate the evident cooperative hunting pattern that emerged during these flight tests. Therefore, even facing a complex pursuit, the pursuer's drones could track and intercept the evader's movements with a faster and reactive target. Note that none formation is intended to be kept; instead, the drones re-arrange between themselves to corral the intruder. Observe that the roles of positioning (center, left, and right-wings; as exposed in Chapter 2 are constantly changing in the function of the fleet's relative position to the target.

Moreover, taking into account the size of the virtual arena (imposed to 6×6m) and the presence of 4 drones flying inside, this approach has demonstrated a robust, practical performance in the disturbances produced by the airflow of the aerial vehicles.

3D Pursuit

The 3D pursuit is implemented using two decoupled GLs: the GDP, from section 2.2 for the horizontal navigation, and the Vertical DPP, from section 4.2.4, for the navigation in the vertical plane.

Three experiments were proposed to validate these strategies. In the first two ones, we aimed to validate the vertical guidance law (4.15) alone, and in the last one, we proposed a three-dimensional pursuit with two pursuers and an evader.

Altitude control

This first real time experiment was carried-out to verify the performance of the vertical guidance law (4.15) with only one pursuer and one intruder in two different scenarios, one for fixed and other for variable offset angles. For both cases, the initial position of the pursuer and target is given by: (x i , y i , z i ) = (-2, 0, 0.8)m and (x t , y t , z t ) = (5, 0, z t d )m, where z t d will change for each case.

In the first test, see Figure 4.22, the target (red crosses) was placed with three different altitudes, z t d 1 = 0.2, z t d 2 = 1 and z t d 3 = 1.7 all in meters. The black crosses indicate the pursuer's traveled trajectory, and the green circle with an arrow is its final position and virtual pitch θi . We can see that in all cases, the pursuer traveled towards the target interception. For security reasons, the Work in progress as of 20 th June, 2021 Notice in these screen-shots that the pattern of ambushing forming in the space is again presented. A video of this experiment is available at: https://youtu.be/8CUZZIrBBqA

Reinforcement Learning

An extract from the real time experiment can be seen in Figure 4.25. In these four sequential pictures, we can see clearly the emergence of the ambushing behavior: the pursuers once close to each other (frame 1), spread toward the target (frames 2 and 3), and finally, they regroup by cornering the target (frame 4).

The results are satisfactory, and they seem even more promising once considering the simplistic conditions of training. Remember that the considered model (3.16) does not consider the non-linear dynamic of the quadcopter during the Work in progress as of 20 th June, 2021 This result also arouses the interest in proposing more complex scenarios. The capacity of having a large number of inputs into a Neuronal Network leaves the possibility of including a larger observation vector, which could consider, for example, multi-forms obstacles and no-go zones.

Finally, although pursuit strategies have also been identified, training in a more realistic environment is required for a real application of this technique. For example, the fact that there is a security layer and that the capture does not occur generates the pursuers situations not experienced during the training, which results in inappropriate actions.

Work in progress as of 20 th June, 2021

Conclusion

This chapter exposes a complete framework for implementing guidance laws in a group of real time drones. Besides, real time flight trajectory recordings were shown, enabling a qualitative analysis of the pursuit strategies. In the following, we leave to the reader a summary of each section's achievements.

In this first section, we went deeper into the drone model in question, the quadcopter. Consequently, we expose two mathematical representations of a drone's dynamics, one in the inertial frame and the other in the body frame. These models are useful to understand how to apply the guidance laws on this specific robot platform and estimate states.

In section Motion Control Hierarchy, we propose a multi-layered control system that aims to take the GL inputs and transforms them into control inputs for the robot platform. To do this, we went first through the safety layer, where collision-avoidance criteria was added to safe the experiment. Next, we propose a layer of "motions constraints" aiming to adapt GL's inputs with the quadcopter model. Finally, we chose an attitude control to generate the thrust and touch commands for the quadcopter.

Work in progress as of 20 th June, 2021

In section Hardware, we expose which materials were used throughout this validation process. Starting from the flight arena and localization system, then exposing the used drones, and finally, giving details of the software architecture.

In section state-estimation, we aimed to face a practical problem, where the positioning of drones in outdoor navigation was inaccurate. We studied KF filtering schemes to filter and estimate quadcopter states through inaccuracies and data loss.

Finally, in Experiments with Pursuit section, we show results from three implementations of pursuit strategies. In Group Deviated Pursuit, one can see the algorithm's efficiency and the formation of ambush trajectories. Similarly, these results were extended to the three-dimensional space, and similar patterns were also observed in the three-dimensional trajectories. Finally, we have the preliminary results of the implementation of the Reinforcement Learning algorithms.

Work in progress as of 20 th June, 2021

Conclusions

The intriguing phenomenon of collective motion has been progressively unveiled by the scientific community. Fascinating examples of flocking with autonomous drones and impressive aerial choreographies are already realities, and it has become a common activity in entertainment shows. However, the numerous previous efforts to reproduce the collective motions fail to reproduce the equally fascinating collective hunting phenomenon. Although the last one can be seen as a subgroup of the first, striking differences distinguish between flocking behavior and collective hunt behavior.

As discussed in the introduction, the characteristics of cohesion and alignment, so common in flocking, give rise to dispersal and ambush in collective hunting activity. In view of this theoretical gap, this thesis proposes to model the collective hunting behavior differently, not only seeking to describe the given behavior but also concerning applying such models in real robots.

The interest in artificially reproducing collective hunting is an actual demand from different areas. For example, in biology, ethologists seek to understand and model animal behavior to propose strategies for handling and preserving species. In robotics, the understanding of collective hunting can help perform tracking tasks that would be very complex for a single robot, such as the pursuit and interception of a faster and more agile intruder.

As discussed in Chapter 2, the anti-drone drone has already proved to be a handy tool in combat against civilian drones' unlawful use. However, the pursuit made by a single stalker is already disadvantageous due to a faster or more agile target.

145

Our approach: The mainline of this thesis tries to reproduce the collective behavior through geometric rules and laws of orientation. We propose multiagent solutions to the conventional virtual physical laws, common in much of the literature in the area. We show that the ambush behavior can emerge from simple geometric rules such as Deviated Pursuit and Parallel Navigation with offset. Besides, we also investigate multi-agent persecution with deep reinforcement learning techniques. Furthermore, following this thesis's main methodological line, we investigated problems complementary to the collective pursuit, such as the evasion of dynamic obstacles and different navigation approaches in flocks. Following, a summary of the work carried out in this thesis.

Summary of the chapters

In chapter 1, we carried out a brief but careful review of the main work related to the drone's group chase. First, we present an overview of the current civilian problem with drones, the difficulties of enforcing the law, and the current containment solutions. Among those solutions, we point out the drone antidrone as a promising one, but holding in its great advantage also its weakness: having the same nature and the threat.

Then, we review the current collective motion literature. We emphasize the importance of bio-inspiration, which was the motivation for the main models of collective movements, and we also reviewed the main multi-agent techniques applied to robots. Then, we introduced the vast pursuit-evasion field, highlighting remarkable works in the area involving the pursuit curves. Besides, we expose the recent group-pursuit field, which tries to reconcile the flocking with classic pursuit analyses.

Finally, we study the classic guidance laws, which are extensive literature deeply related to bellicose navigation but recently extensively applied to robotic navigation. Further, we expose the kinematic models and the geometric engagement used in their analysis, as well as some of the main classic models used in this text.

Work in progress as of 20 th June, 2021

In Chapter 2, we studied the problem of the group-pursuit of a single-evader in 2D. We propose three different approaches to solve this problem. The first GDP, the simplest of all, is based on the classic deviated pursuit (DPP), where pursuers choose different offsets, depending on their current engagement toward the target and its neighbors. These different angles provide different routes that reproduce ambush trajectories.

The second approach is based on the Parallel Navigation principle, where each of the pursuers aims to maintain a constant bearing angle to the target, which allows for more efficient trajectories. Similar to the previous approach, the offset angle concept was also maintained here, which avoids collision between two pursuers and allows different ambush routes.

Our third methodology aims to build collective hunting behavior based on an automatic mechanism, reinforcement learning, where Neuronal Networks are trained to describe an individual's behavior. In this approach, the pursuers interacting initially at random will discover by trial and error the optimal behavior in this pursuit in a simulated environment.

In Chapter 3, we went beyond the planar group pursuit, and we explored three different scenarios. First, we start with a different pursuit hypothesis, where all agents can know the target's position in the global frame. From this global information, it was possible to use a predictor scheme to estimate the target's future trajectory. Then, we adapted our first deviation chase algorithm (GDP), making each individual chase the target in a future position, and finally, the chase angles are based on estimated prediction errors.

In the second part of this chapter, still a different hypothesis, we study a scenario where a single-pursuer tends to chase a dynamic target with an unknown trajectory in an environment with non-cooperative agents (or mobile obstacles). In this case, we also use the Parallel Navigation principle; the geometric rule, once used in the interception, was now used for collision avoidance.

Finally, we went beyond the pursuit itself, and we presented an approach to reproducing generalized collective motions, such as focusing or flocking, but also through geometric laws.

We propose that from estimating the centroid of the perceived neighborhood and the geometric rule of constant angle, a collective movement's verisimilar behavior can be obtained.

Chapter 4 presents the implementation of the previous algorithms on a robotic platform, the quadcopter. To do this, we first studied the model of the quadcopter dynamic; both defined in the inertial frame and the body frame. Then, we introduced our hierarchical control of movement, composed of four layers.

The highest level is responsible for the orientation law and gives commands at linear and angular velocity. Right after, we have the safety layer, responsible for maintaining the physical integrity of the vehicle. In indoor experimental tests, where collision is not desired, additional safety measures, such as repulsion forces and virtual barriers, have been implemented.

In the sequence, motion control is then implemented to adapt the high-level (and non-holonomic) commands to the drone model (holonomic). Finally, we present some experimental results of the proposed guidance laws implemented in real-time drones. We present screenshots of the pursuers and evaders' trajectories and provide a qualitative analysis of the chase.

Discussion

Pursuit: large vs. small groups This work aims to add a small contribution to the emerging field of group-pursuit, which according to [START_REF] Kamimura | Group Chase and Escape: Fusion of Pursuits-Escapes and Collective Motions[END_REF], is a fusion between the fields of collective movements (flocking) with classic pursuit techniques. Although the concept of group pursuit has been in place for more than a decade, most work considers predators' behavior to be very minimalist, which differs little from flocking behavior. These models' simplicity aims to apply and analyze the pursuit on a large scale, as in [START_REF] Angelani | Collective predation and escape strategies[END_REF] and [START_REF] Kamimura | Effective construction of linear statevariable models from input/output functions[END_REF], where teams of hunters of the order hundreds are used. Indeed, these models have great value when analyzing and describing individual organisms' behavior, which in short have a similar order of magnitude.

Work in progress as of 20 th June, 2021 However, as explained in Chapter 3, this minimalism in the predatory model does not correspond, for example, with the collective hunting behavior of superior mammals, such as the groups of lions or packs of wolves. The differences occur both in terms of the strategy itself and in the number of agents. For example, in several initial works, such as in [START_REF] Kamimura | Effective construction of linear statevariable models from input/output functions[END_REF] and [START_REF] Angelani | Collective predation and escape strategies[END_REF], the pursuit behavior is purely in aligning its velocity (or orientation) directly to the prey's instant location (Pure Pursuit). In contrast, our proposal focuses on the agent and its interaction with its close neighbors. Furthermore, the pursuit's efficiency is taken into account, along with the formation of an ambush pattern.

Efficiency in pursuit: Similar to our approach, Janosov's work [START_REF] Janosov | Group chasing tactics: how to catch a faster prey[END_REF] is also concerned with the efficiency of the pursuit. In this case, the model parameters were subjected to optimization processes to improve catch rates. However, the strategy requires knowledge of the prey in global coordinates, which is not ordinarily applicable to the pursuit of non-cooperative targets. Furthermore, this approach is based on virtual forces (attraction and repulsion) and velocities alignment (similar to Vicsek's flocking models [START_REF] Vicsek | Novel type of phase transition in a system of selfdriven particles[END_REF]) and additional components for the pursuit behavior. Moreover, this large number of parameters also makes tuning work difficult.

As pointed out in Chapter 3, our work has an efficiency very similar to Janosov's work, yet dispensing with hypotheses of knowledge of the target in global coordinates: in the three approaches proposed in this chapter, the target is perceived only in relative and polar coordinates. Besides being realistic and compatible with natural predators, such hypotheses also allow for a more reliable implementation in robotic systems since polar and relative coordinates are consistent with sensors commonly embedded in robots, such as a camera or a Lidar.

Flocking and collision avoidance: Besides the group persecution, this thesis also proposes mild additions to the broader term of collective behavior. The results presented in Chapter 4, although preliminary, point to a possible alternative for reproducing the flocking phenomenon, not based on the sum of artificial physical forces but based on guidance laws and geometric relations.

Work in progress as of 20 th June, 2021

Once our approach relies only upon local and polar information, our work attempts to contribute to the field of vision-based or communication-less flocking.

One of the main challenges for applying flocking in real-robots is measuring neighbors' orientation or measuring their velocity, which is required in most of the flocking algorithms, such as [START_REF] Craig | Flocks, herds and schools: A distributed behavioral model[END_REF][START_REF] Olfati-Saber | Flocking for multi-agent dynamic systems: Algorithms and theory[END_REF][START_REF] Vicsek | Novel type of phase transition in a system of selfdriven particles[END_REF]. Usually, works ignore these two states' measurements, consider robotic systems with very slow dynamics, such as in [START_REF] Ferrante | Flocking in stationary and non-stationary environments: a novel communication strategy for heading alignment[END_REF]. In the case of applications with UAVs, as pointed out by [START_REF] Virágh | Flocking algorithm for autonomous flying robots[END_REF], the quadcopter's reactivity and dynamics lead very easily to system instability, and knowledge of speeds and orientation is necessary to dampen the system.

In our approach, the alignment is not calculated by the average of the neighbor's velocity, but it naturally occurs when applying the constant bearing and constant range rule, which can be obtained through the range and bearing measurements.

In the field of navigation with uncooperative agents (or avoiding dynamic obstacles), most strategies are also based on the sum of virtual forces, of attraction and repulsion, such as [START_REF] Henrique | Control and Obstacle Avoidance for an UAV Carrying a Load in Forestal Environments[END_REF], which can work even well under the condition of action of non-dynamic obstacles. However, as they do not consider the dynamics (and kinematics) of the obstacles, and consecutively these techniques do not perform well with mobile obstacles. Another line, such as [START_REF] Matthew | Collision Avoidance and Navigation of UAS Using Vision-Based Proportional Navigation[END_REF], considers the agents' dynamics/kinematics and predicts the trajectories to determine the imminence or not of collisions. However, these consist of computationally costly approaches. Our strategy aims at a reactive and straightforward way to avoid collisions with dynamic objects, not requiring global knowledge, the position requiring optimization techniques. Again, our hypotheses are based on the relative and polar perception of obstacles, implementing real robots more feasible.

Applicability In addition to this thesis's main problem, the use of drones in defense of civil air space against intruder drones, the models proposed here can be applied in several other tasks. Still, in the field of surveillance, any other task that requires the tracking of a more agile and fast target can be solved with techniques similar to this one.

Work in progress as of 20 th June, 2021

Another possible area could be the ethological studies in Biology, where models describe animal behavior and better understand a specific animal population's dynamic.

Future work

Perception: For more practical applications, it is necessary to work more deeply with questions of perception-for example, the acquisition and processing of data obtained by camera how to deal. In addition to the challenge of identifying a quadcopter in a moving image, the difficulty is also linked to the distinction between cooperative and non-cooperative agents.

A clue to solve the perception problem would be the fusion of data between position information to be communicated between individuals and the information obtained by the individual's own perception.

Low Level Control: As this is not the main focus of this study, the low-level control part was conducted simplistically. However, to obtain more reactive results, it would be interesting to analyze the low-level reference segment and study more effective and robust control laws for tracking velocities references.

Reinforcement Learning In the RL experiments, the considered agent's model in simulation was pretty straightforward. Certain restrictions on practical applications, such as repulsion due to avoidance of obstacles, were not considered in training and may cause effects not previously known by behavioral policies. A future direction would be to take into account more realistic dynamic models with robotic platforms.

Multi-agent:

The works with obstacle avoidance and flocking are still in a very preliminary phase. Deepening in the theoretical field, using metrics and comparisons with previous works are necessary. Besides, more real-time implementations must be made to analyze the capabilities and deficiencies of these approaches.

Work in progress as of 20 th June, 2021

Figure 1 -

 1 Figure 1 -Examples of collective motions: a) Flocking of starlings b) An aerial demonstration squadron c) Wolves chasing a hare

Contents 1 . 1 8 1. 2 12 1. 3 18 1. 4 22 1. 5

 1182123184225 The counter-drone fight . . . . . . . . . . . . . . . . . . . . Collective motion . . . . . . . . . . . . . . . . . . . . . . . . Pursuit evasion . . . . . . . . . . . . . . . . . . . . . . . . . Guidance Laws (GL) . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

Figure 1 . 1 -

 11 Figure 1.1 -Miscellaneous of criminal usage of drones: terrorism, airplanes accidents, traffic of drugs and flight over nuclear plants.

Figure 1 . 2 -

 12 Figure 1.2 -Examples of anti-drone solutions. A) A policeman carrying a dronejammer [136]. B) Equipment used for protocol manipulation (spoofing) [137]. C) Illustration of a net-launcher deployed by a policeman. D) Example of anti-UAV falconry.

Figure 1 . 3 -

 13 Figure 1.3 -Examples of anti-drone drone (ADD). A) An ADD equipped with a net-launcher. Similar solution has been commercialized by [130] and [131]. B) ADD with a fixed net deployed by the Tokyo police [154]. C) Kamikaze ADD developed the American startup Anduril [133].

Figure 1 . 4 -

 14 Figure 1.4 -One of the most classic flocking models [49] was created to graphically reproduce the behavior of collective animals, such as fish in a school. In the left figure, three-dimensional "boids" are represented [146]. On the right, a picture of a school of fish, from [147].

Figure 1 . 6 -

 16 Figure 1.6 -Examples of real-time applications of flocking with quadcopters done by Vásárhelyi et al [148]. Thirty drones are fly in an virtually confined space.

Figure 1 . 7 -

 17 Figure 1.7 -Curbs of pursuit. Left: the classic Bourger's solution for different velocity's ratio between target and pursuer, extracted from[START_REF] Neryahu | Missile guidance and pursuit: kinematics, dynamics and control[END_REF]. Right: Example of the generalized closed-form solution for a 2D pursuit, exposed in[START_REF] Kamimura | Group Chase and Escape: Fusion of Pursuits-Escapes and Collective Motions[END_REF]

Figure 1 . 10 -

 110 Figure 1.10 -Geometrical engagement between pursuer and target.

Contents

Figure 2 . 2 -

 22 Figure 2.2 -Lionesses hunting formation. The left figure, extracted from [45], illustrates the distribution of a group of lionesses towards the prey. The right figure shows the final moment of a lionesses group attack.

  .

  3.

Figure 2 . 3 -

 23 Figure 2.3 -Engagement between three pursuer and a target.

Figure 2 . 4 -

 24 Figure 2.4 -Paths of deviated pursuit according to different offsets of the angle α. The agents begin the trajectory in the bottom part and keep a constant velocity until they get the target.

Figure 2 . 5 -

 25 Figure 2.5 -Trajectories generated by three agents and a fixed target. The pursuers start at the bottom of the square, and they pursued the target with constant velocity. The deviation offset (α 0 ) for this pursuit is 30 degrees.

Figure 2 . 6 -

 26 Figure 2.6 -Snapshots for pursuits with different numbers of agents (1, 2, 3 and 4). In all cases, the target does an escape trajectory-based on repulsion forces, with a velocity of 2 times bigger than the pursuers. The red circles signalize the end of the target trajectories.

Figure 2 . 7 -

 27 Figure 2.7 -Case of 15 pursuers and an intruder. Notice the well performance of the pursuers when tracking the target.

Figure 2 . 5 -

 25 Figure 2.5-top shows the number of captures as function of the pursuers' group size. For its turn, the graph on Figure 2.5-bottom, represents the average time taken to capture.

Figure 2 . 8 -

 28 Figure 2.8 -Effects of the offset value in the pursuit.

Figure 2 . 9 -

 29 Figure 2.9 -Effect of the offset value (α 0 ) on the collision between pursuers.

Figure 2 . 11 -

 211 Figure 2.11 -Comparison between Pure Pursuit (PP) and Proportional Navigation (PNG). The PNG pursuer (blue circles) and the PP pursuer (pink circles)are not aware of each other, i.e, there is no cooperation. In the left picture, the target performs a linear trajectory, while in the right one, the target executes a maneuvering trajectory.

Figure 2 . 13 -

 213 Figure 2.13 -Effects of the offset value in the pursuit.

  This work was done in collaboration with the Robotic Centre of Monash University, Melbourne, Australia; during an institutional exchange, funded by the UTC doctoral school and the Heudiasyc laboratory. This section's content has been recently organized in a paper entitled: Distributed Multi-Agent Pursuit using Deep Reinforcement Learning. C. de Souza Jr, R. Newburry, A. Cosgun, P. Castillo, B. Vidolov, D. Kulić. Accepted to ICRA/RA-L 2021.

Figure 2 .

 2 Figure 2.14 -Scheme for the Deep Reinforcement Learning.

Figure 2 . 15 -

 215 Figure 2.15 -Comparison of capture success rate with and without curriculum learning, with respect to the number of training steps. With curriculum learning, the benchmark scores are much higher and more consistent.

Figure2. 15

 15 Figure2.15 compares the effect of using our curriculum learning strategy described in subsection 2.4.2, for n = 3 agents. The network was trained three times, and during training, we analyze the evolution success rate trained policy. At regular intervals, we stop training and evaluate the policy on the repulsive evader benchmark. The results show that curriculum learning is very helpful for capture performance: it converges to about 100% success rate after 1.5 million training steps, whereas without curriculum learning, the average success rate was below 80% even at 4 million training steps. Furthermore, curriculum learning performance was much more consistent, as evidenced by the low variance among the three runs.

Figure 2 . 16 -

 216 Figure 2.16 -Using a formation score as a dense reward results in more captures, in less number of timesteps on average.

Figure 2 .

 2 Figure 2.17 -"Split Up" strategy learned by 3 pursuers. Timestep (T) and formation scores (Q) are shown at four snapshots. The target is shown as the black circle. The agents start in a random direction (Top Left), push the agent towards the wall (Top Right), split into two groups (Bottom Left) before going for the capture (Bottom Right).

Figure 2 .

 2 Figure 2.18 -"Ambush" strategy learned by 8 pursuers. Timestep (T) and formation scores (Q) are shown at four snapshots. The target is shown as the black circle. The agents start in random directions (Top Left), move as a circle (Top Right), ambush the target (Bottom Left), and capture it (Bottom Right).

Figure 2 .

 2 Figure 2.19 -"Stalking" strategy learned by 3 pursuers with velocity. Timestep (T) and formation scores (Q) are shown at four snapshots. The target is shown as the black circle. The agents start in random directions (Top Left), move towards the target (Top Right), slow down and angle themselves such that they can surround the target (Bottom Left), and capture it (Bottom Right).

Figure 2 . 20 -

 220 Figure 2.20 -Detailed benchmark results for 3 pursuer over 100 repetitions.

Figure 2 . 21 -

 221 Figure 2.21 -Comparison between our proposed approaches and the baseline strategies.

Figure 2 . 22 -

 222 Figure 2.22 -Comparison between our proposed approaches and the baseline strategies.

Figure 2 . 23 -

 223 Figure 2.23 -Comparison between our proposed approaches and the baseline strategies.

Contents 3 . 1 74 3. 2 83 3. 3 91 3. 4

 31742833914 Pursuit with target prediction . . . . . . . . . . . . . . . . Pursuit with non-cooperative agents . . . . . . . . . . . . . Flocking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 73

Figure 3 . 1 -

 31 Figure 3.1 -Illustration of a pursuit with prediction. The pursuer (darker quadcopter) is pointing to the future position of the target ( xT , ŷT ) instead its current position (x T , y T ).

Figure 3 . 2 -

 32 Figure 3.2 -Problem formulation of the predicted pursuit. All the pursuer knows the target position and velocity in the global frame and its foresee trajectory.Based on evader capabilities, the prediction error can be estimated h i .

  are estimated by using ẋt = vT + l 11 (x T -xT ) + l 21 (v T -vT ), vt = âT 0 + l 12 (x T -xT ) + l 22 (v T -vT ), ȧT 0 = âT 1 + l 13 (x T -xT ) + l 23 (v T -vT ), . . .

Figure 3 . 3 -

 33 Figure 3.3 -Single pursuer against an evader. The pursuer (blue circle) heads towards the future position of the target (red circle) instead of its instantaneous position (red cross). The violet circle denotes the "error in the prediction".

Figure 3 . 4 -

 34 Figure 3.4 -Pursuers behaviors when hunting an intruder drone using the DPP based methodology. Top: without prediction algorithm and bottomwith the predictor scheme.

Figure 3 . 5 -

 35 Figure 3.5 -Illustration of a futuristic city, where the autonomous drone would tracking a target in a crowded and shared air-space.

Figure 3 . 6 -

 36 Figure 3.6 -Engagement between the pursuer (blue circle), the target (red circle) and one non-cooperative agent j (grey circles). The green area represents the obstacle's perception area.

Figure 3 . 7 -

 37 Figure 3.7 -A pursuer autonomously navigating into an environment with multiples fixed obstacles for tracking a static target.

. 8 .

 8 Observe in this figure the successful target capture and the well-defined obstacle avoidance.Work in progress as of 20 th June, 2021

Figure 3 .

 3 Figure 3.8 -A pursuer autonomous navigating into an environment with multiples fixed obstacles for tracking a moving target.

Figure 3 . 9 -

 39 Figure 3.9 -Pursuer performance when navigating into an environment with non-cooperative agents are evolving.

Figure 3 . 10 -

 310 Figure 3.10 -Examples of flocking: on the left, a flock of autonomous drones from [148]. On the Right, Extract from [111], an example of centroid's identification from a bird's swarm image.

Figure 3 . 11 -

 311 Figure 3.11 -Geometry between one agent i and the rest of the flock. Its closest neighbors are identified as j, and the flock's centroid FC is denoted by a green cross. Note that the FC is estimated based on the neighbors inside the field of vision (green area). The yellow circles represent the "informed agents."

Figure 3 . 12 -

 312 Figure 3.12 -Paths for aggregation under different opening angle (β): β = π/4 in A, β = π/2 in B, β = π in C and β = 2 * π in D.

Figure 3 . 13 -

 313 Figure 3.13 -Screenshots for a simulation with 15 flocking agents and one informed.

Figure 3 . 14 -

 314 Figure 3.14 -Screenshots for generalized scenario: target tracking (red cross), obstacle avoidance (grey square) and flocking.

Figure 4 . 1 -

 41 Figure 4.1 -Body frame of the quadcopter.

  Figure 4.2.

Figure 4 . 2 -

 42 Figure 4.2 -Motion control hierarchy of a quadcopter using guidance laws.

Figure 4 . 3 -

 43 Figure 4.3 -Scheme of the safety level. On the left side, a representation of a drone, with its velocities and its safety radii R saf and R c ol. On the right side, three drones and their position vectors are represented inside a virtual arena of radius R are with the thickness of R w all.

  saf wi , and Work in progress as of 20 th June, 2021 f saf ψi . For computing the law for the velocities, f saf vi = [f saf vi f

Figure 4 . 4 -

 44 Figure 4.4 -Representation of the two proposed quadcopter's motion constraints: a) non-holonomic b) heading-to-target

Figure 4 . 5 -

 45 Figure 4.5 -Illustration of an FPV drone race: once the pilots (left picture) have only the front camera information as perception, they tend to drive the quadcopters in a non-holonomic way.

Figure 4 . 6 -

 46 Figure 4.6 -Three dimensional geometry between two quadcopters.

Figure 4 . 7 -

 47 Figure 4.7 -Picture of a group-pursuit experiment in the indoor flight arena.

Figure 4 . 8 -Bebop 2

 482 Figure 4.8 -Parrot AR Drone 2 (left) and Parrot Bebop 2 (right). The two commercial quadcopters used for the experimental validations.

Figure 4 . 9 -

 49 Figure 4.9 -Screenshot of the FL-Air simulator in an outdoor environment.

  Figure 4.10.

Figure 4 . 10 -

 410 Figure 4.10 -Scheme for the remote computation architecture.

Figure 4 . 11 -

 411 Figure 4.11 -Schema for the altitude estimation, evidencing the states of relative altitude (z) and the ground altitude z g .

( 4 . 23 )

 423 Work in progress as of 20 th June, 2021 where z gnd k denotes the ground altitude and z k denotes the relative altitude. From this model, the matrices A and B can be obtained, similarly to section 4.4.4.For its turns, the sensor model can be expressed as: ỹson = z k , ỹbar = z k + z

Figure 4 . 12 -

 412 Figure 4.12 -Quadcopter response when tracking the desired points in ideal conditions (no disturbances).

8 Figure 4 . 13 -

 8413 Figure 4.13 -Quadcopter performance with a KF when applying simulated data lost.

Figure 4 . 14 -

 414 Figure 4.14 -Quadcopter performance with a KF crossing a tunnel. The localization system is totally obstructed for 1.5s in average (equivalent to 300 sample time).

Figure 4 . 15 -

 415 Figure 4.15 -Quadcopter performance with a KF applied to degraded position data.

Figure 4 . 16 -

 416 Figure 4.16 -Quadcopter response using KF for position data degradation. The solid red line is the standard performance (no disturbance), black dotted, and blue dashed lines respond in case of degradation. The first one without KF, and the second one is applying the algorithm.

Figure 4 . 18 -

 418 Figure 4.18 -Quadcopter taking over and landing after few seconds of hover. The altimeter measurements zson are very imprecise when the quadcopter is flying above two meters.

  and 4.19 is reached. Nevertheless, the irregular trajectories observed in the pursuer's agents are caused by the airflow produced by the drones' proximity and the absence of control position of each one.

Figure 4 .

 4 Figure 4.19 -real time performance when three aerial drones pursuit a fixed target. The first image (left) shows the displacement of the drones in the flight arena. The second one is a picture of the experiment.

Figure 4 .

 4 Figure 4.20 -real time behavior when an aerial drone pursuits an aerial intruder drone in presence of static obstacles.

Figure 4 .

 4 Figure 4.21 -real time performance when three aerial drones pursuit an aerial intruder drone with random movements.

Figure 4 .

 4 Figure 4.23 -Ambushing behavior in the vertical plan: the black crosses indicates the path followed by the pursuer (quadcopter) towards a fixed target under different values of offset α iT .

Figure 4 .

 4 Figure 4.24 -real time three-dimensional pursuit with quadcopters: The path of two pursuers (denoted black star and green circle) and the target(red crosses) are given in these screenshots. The columns denotes three sequential timestamps, while in the each line is exposed the traveled path along xy and xz planes

Figure 4 . 25 -

 425 Figure 4.25 -Screenshot for the real time experiment.

Introduction 1 State of the art 1 . 1

 111 The counter-drone fight . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Anti-drone solutions . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Anti-drone drone (ADD) . . . . . . . . . . . . . . . . . . . 1.2 Collective motion . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Bio-inspiration . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Collective robots . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Pursuit evasion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 Single Pursuit-evasion . . . . . . . . . . . . . . . . . . . . . 1.3.2 Group pursuit . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Guidance Laws (GL) . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 GL in robotics . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.2 Relative engagement . . . . . . . . . . . . . . . . . . . . . . 1.4.3 Pure Pursuit (PP) . . . . . . . . . . . . . . . . . . . . . . . 1.4.4 Pure Deviated Pursuit -DPP: . . . . . . . . . . . . . . . . 1.4.5 Proportional Navigation Guidance (PNG) . . . . . . . . . 1.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Pursuit in the horizontal plan 2.1 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Perception . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Target behavior . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Group Deviated Pursuit . . . . . . . . . . . . . . . . . . . . . . .

Table 4 .

 4 .7). The parameters used in this test are presented in Table4.1. 1 -Control parameters used in the real time scenarios.Work in progress as of 20 th June, 2021From Figure4.19, the obtained result for this scenario can be observed. It is clear that the hunting pattern described inFigures 2.4

	Parameters	Scenarios 1 and 3 Scenario 2
	α	20 o	10 o
	u max	1.3	1.3
	R cap Field -vision	1.6 360 o	1.6 120 o
	R int	5	2
	R col	1.5	0
	a 0	1	0

Work in progress as of 20 th June, 2021

Work in progress as of 20 th June, 2021

Bibliography

Work in progress as of 20 th June, 2021

Chapter 4

Implementation and Experiments

This chapter describes the experiments and the implementation of the pursuit strategies, from the previous chapters, into real time robots. Firstly, we present the quadcopter model, which will be used along with the chapter. Then, we propose a motion control hierarchy for a drone pursuer. This multi-layer controller is responsible for transforming the high-level command from the guidance law (GLs) into the low-level quadcopter inputs. Following, we discuss the hardware platform, exposing the used robots, the software architecture, and the installations. Next, we describe our work on state estimation, in which we implemented a Kalman filter scheme for improving position and velocity estimation. Finally, we expose proof-of-concept results for group-pursuit with quadcopters, implementing several proposed strategies in real time.

Attitude Controller

The lower-level control is responsible to generate control inputs for thrust (u 1 ) and torques (u 2 , u 3 and u 4 ) for the quadcopter, following the reference given by the upper level control. This level requires high-frequency processing and is the one most dependent on the vehicle's dynamics.

Therefore, we choose a non-linear controller based on saturation function, proposed in [START_REF] Sanahuja | Stabilization of n integrators in cascade with bounded input with experimental application to a VTOL laboratory system[END_REF], to stabilize the lateral dynamic of the quadcopter 4.2 and follow the angular references. The control law for longitudinal dynamic is given by:

where, u 2 i is the control input for τ θ , and k 1 and k 2 are positive gains. Besides, σ (.) is a saturation function, previously defined in (2.7).

In a similar manner a control law for lateral dynamic can be given by:

where k 3 and k 4 are positive gains.

The control input for the thrust can be directly obtained from from (4.15),

However, for a pursuit in a fixed plan, we can given an altitude reference (z ref i ) and apply a similar non-linear controller such as:

where k a and k b are positive gains.

Finally, control law for the turn around the z-axis is given by:

where k 5 and k b are positive gains.

The proof of convergence of the above control laws and more details about its implementation can be checked in [START_REF] Sanahuja | Stabilization of n integrators in cascade with bounded input with experimental application to a VTOL laboratory system[END_REF].

Work in progress as of 20 th June, 2021

Varying offset

In the second experiment, see Figure 4.23, the target altitude was fixed at z t d = 1.8m, and the control goal is to test the displacement in the vertical plan for different values of α iT angles. Observer that, similar to Figure 2.4, the ambushing behavior also emerged in the vertical plane.

3D pursuit

Once validated the pursuit algorithm's performance for different altitudes, the proposed algorithm is evaluated in real time in a 3D environment. In this scenario, two drones were used as pursuers acting autonomously when applying the altitude control (4.15) and the pursuit algorithm (3.9) with a fixed offset, α iT = π/6. The target is controlled manually by the user.

In