Pierre De Saqui-Sannes

Rob A Vingerhoeds
email: rob.vingerhoeds@isae-supaero.fr

Nasrine Damouche
email: nasrine.damouche@isae-supaero.fr

Eric Razafimahazo
email: eric.razafimahazo@isae-supaero.fr

Ombeline Aïello
email: ombeline.aiello@isae-supaero.fr

Maisa Cietto
email: maisa.cietto@usp.br

Mind Maps Upstream SysML v2 Diagrams

Keywords: Mind Maps, SysML v2, Real-Time Systems

Over the past two decades, the promoters of Model Based Systems Engineering (MBSE) have encouraged systems engineers to transition from document-centric approaches to model-based ones. Literacy of systems engineers in reading, sharing and elaborating models has therefore become an issue. Whatever the modeling language, elaboration of models is a highly complex intellectual process and SysML is no exception. Feedback from industry practitioners and MBSE lecturers suggests that developers of SysML models often stumble on the same problem: thinking about the system before modeling it in SysML. The authors of this paper propose to ease that elaboration process by using mind maps. With their graphic form and rather flexible way of organizing ideas, mind maps turn out to be a good candidate to help thinking about the system. Unlike approaches that directly switch from mind maps to SysML diagrams dedicated to one specific system, this paper introduces an intermediate step: mind maps first enable elaboration of diagrams patterns. The latter may in turn be instantiated on one or several systems. Without loss of generality, the proposed approach is step-wise illustrated on real-time systems monitored by software controllers. Patterns are proposed to cover need expression, requirement capture, use case driven analysis and design.

I. INTRODUCTION

Over the past two decades, the promoters of Model Based Systems Engineering (MBSE) have encouraged systems engineers to transition from document-centric approaches to model-based ones. Literacy of systems engineers in reading, sharing and elaborating models has therefore become an issue. In this context, the standardization and tool support of graphic modeling languages -particularly, the Unified Modeling Language (UML [START_REF]OMG Unified Modeling Language, Object Management Group[END_REF]) and the Systems Modeling Language (SysML [START_REF]OMG Systems Modeling Language, Object Management Group[END_REF]) -have been presented as enablers of MBSE adoption in industry [START_REF] Wolny | Thirteen years of SysML: A systematic mapping study[END_REF].

Nevertheless, recent position papers, internships offers and posts on social networks have indicated that many industry practitioners are still in the process of evaluating the pros and cons of MBSE. Several reasons may been found. From their experience as MBSE lecturers, the authors of this paper wish to highlight one of them: the difficulty to address a system by immediately "diving" into the activity of modeling the system in SysML.

Despite of its graphic syntax and its categorization among semi-formal languages, SysML indeed constrains the way of thinking systems in terms of diagrams. This is fully appropriate when the concern is to step-wise implement a system engineering process with its successive address of requirement capture, analysis and design steps. By contrast, elaborating SysML models first requires forms of brainstorming and sharing of knowledge (with clients, regulators). This step would benefit from a graphic support to express ideas, but in a way that is less formal and coercive than diagramming in SysML. The graphic support which is needed must be close to human way of addressing problems and thinking about systems.

This paper therefore addresses the issue of 'Thinking about the system before modeling it in SysML', and proposes to use mind maps for the thinking part of the objective whilst keeping SysML as systems modeling language.

The paper is organized as follows. Section II presents the rationale behind the approach developed in the paper. Section III surveys related work. Section IV explains how the remainder of the paper sets mind maps upstream SysML v2 diagrams. Section V focuses discussion on real-time systems modeling. Sections VI, VII, VIII and IX respectively achieve needs expression, requirements analysis, use-case driven analysis, and design in a process associated with SysML. Section X concludes the paper and outlines future work.

II. RATIONALE

Developing a system, be it a product, a service, or an organisation, passes several life cycle phases. The first stage in a system's life cycle, the concept stage, focuses on understanding the implications of a system's mission and core functionality, a business case together with requirements, their interconnections and dependencies specified in e.g., key performance indicators (KPIs) and trade-off indicators such as Figures of Merit (FoM) [START_REF] Brazier | Complex Systems: Design, engineering, governance, ser. Projects and People: Mastering Success[END_REF]. A threefold objective can be seen for the concept stage:

1) To interpret/understand a mission statement, supported by a positive (potential) business case, 2) To produce an initial definition of stakeholder requirements and KPIs with respect to the mission, and 3) To produce an initial logical/conceptual description of a design.

The starting point for this phase concerns the needs and desires expressed by the stakeholders, which needs and desires are over the course of the concept phase transformed into requirements, potentially separated in Functional and Non-Functional Requirements, the latter potentially split in different types of non-functional requirements. This first phase, in which there may not be a very precise idea on what the stakeholders really want, is crucial as this is where the main decisions are taken. Once the requirements are considered clear and a first logical/conceptual description of the design is available, the main orientations are known. The developers then have a logical architecture of the system's design and its subsystems (the upper-level architecture) that meets system requirements: a preliminary design of the product, service or organisation to be developed. The outcome of this first stage is orientating the next stages. It is important to provide developers with a good support to structure the information coming from the stakeholders, and to allow them to more clearly see the requirements space and the solution space. Starting from expressed needs and desires, from discussions, developers may find it complicated to formulate the requirements. This paper advocates the use of mind maps for this first phase and to have a transformation from mind maps to the first definition of requirements in a model-based systems engineering environment. Often used for structuring ideas or taking notes amongst others, a mind map is a diagram that allows for visually organise diverse types of information, as well as the relations between the different parts of the information [5] [6].

Mind maps can be seen as being built on top of semantic networks. Semantic networks are one of two closely related families of reasoning systems, the other family being description logics. Semantic networks date back to the early 20th century and offer graphical possibilities for visualisation of a knowledge base and efficient algorithms for inferring properties of an object on the basis of its category membership [START_REF] Russell | Artificial Intelligence: a modern approach[END_REF]. Closely related, description logics provide a formal language for constructing and combining category definitions and efficient algorithms for deciding subset and superset relationships between categories.

The intuitive use of mind maps to take notes and to structure knowledge, makes for a powerful tool. This paper investigates the possibilities of using mind maps as "frontend" to requirements engineering, so as to allow good, well-structured and well-supported collection of information of the concept stage. On the other hand, it also investigates a structured way to transform this information into requirements for further use.

III. RELATED WORK

Mind maps assist persons in generating, classifying, categorizing and visualizing ideas. Many papers and books have addressed mind maps informally. In [START_REF] Siochos | Developing a formal model for mind maps[END_REF] Siochos and Papatheodorou formalize mind-maps as bi-partite graphs.

In [START_REF] Vilchez | Mind maps in requirements engineering: A systematic mapping[END_REF] Quispe Vilchez and Pow-Sang Portillo survey literature on using mind maps in requirement engineering processes. Mind maps are identified as enablers of improved communication between the persons involved in the process of creating requirements and validating the latter.

Mind maps indeed need to be formalized to be used in conjunction with other modeling languages. For instance, to implement model transformation techniques. Such techniques are used in [START_REF] Wanderley | From goals to systems modeling: a user-centered modeling approach[END_REF] where Wanderley, Belloir, Bruel, Hameurlain, and Araújo state that people in charge of eliciting requirements are not necessarily comfortable with SysML requirement diagrams. These authors propose to specify requirements using cognitive models such as mind maps. They use model transformation techniques to transform mind maps into KAOS (Knowledge Acquisition in Automated Specification) models that are subsequently transformed into SysML requirement diagrams. Thus, the paper by Wanderley et al. exclusively associates mind maps with requirement diagrams elicitation. Conversely, next sections of this paper consider that mind maps may influence not only requirement diagrams, but also use case, sequence and block diagrams. This paper also addresses needs, an important issue in systems engineering and an issue not addressed by any SysML diagram.

IV. GLOBAL APPROACH

For over a decade, the authors of this paper have been lecturing on SysML and supervising students projects. Students of various background have been trained, ranging from aerospace engineers to mechanical engineers, computer engineers, and building engineers. Despite of the broad variety of student profiles, the following issues may be addressed:

• Experience in Systems Engineering is often limited. Part of the students have been lectured on systems engineering, including MBSE, but the art of modeling systems is scarcely addressed in terms of what to do before developing a model. The idea of thinking about the system before modeling is not so commonly addressed during SysML classes. • Making abstractions is a stumbling block in the art of modeling and this is verified in the context of SysML. Newcomers to SysML need assistance. This paper proposes to use mind maps as a stepping stone to SysML diagrams creation. • Examples imitation is a way of working that students use to feel comfortable. Instead of multiplying examples of closely similar systems, the concept of pattern enables modeling a reference for a class of systems. For instance, following sections of this paper are focused on the class of real-time systems. • Models are not just documentation artifacts and need to be associated with tools and methods that enable addressing key issues, such as requirement traceability and checking of models against design errors. Therefore, next sections of this paper use SysML v2 and develop models with free software TTool [11]. Previous concerns lead the authors of this paper to develop the 3-step approach depicted by Fig. 1 in the form of a SysML activity diagram. Three main activities are identified:

1) Brainstorming where the mind maps are created.

2) Categorizing where SysML diagram patterns are created. 3) Instantiating where the patterns are instantiated on one or several systems.

V. SOFTWARE CONTROLLERS OF REAL-TIME SYSTEMS

The term "Real-Time Systems" was coined to identify a class of systems that interact with their environment, run under the stimuli of the latter, and need to meet temporal constraints. There exists a large variety of real-time systems, from the simplest to the most sophisticated ones, ranging from microwave ovens to alarm controllers embedded onboard aircraft.

As far as modeling of real-time systems is concerned, the perimeter of the system needs to be clarified first. The problem may be phrased by a question: are we going to model the complete real-time system or a subset of it? That subset may be a (presumably software) controller that monitors the realtime system. In this paper, a real-time system controller was selected to be modeled, not the complete real-time system.

It is common practice to address a real-time system controller that is connected to a set of sensors and a set of actuators. Sensors provide the controller with input values and signals. Actuators are triggered by the outputs of the controller. Fig. 2 depicts a mind map that accordingly contains three branches for the sensors, the control function itself, and the actuators. A branch is added for maintenance, which is a key issue in systems design. The sensors (resp. the actuators) are categorized depending on whether they interact, or not, with human beings, particularly customers and supervisors. Control functions include the set up and shutdown procedures to be used when switching the real-time system on and off, respectively. Distinction is further made between nominal and degraded behaviors.

VI. NEEDS EXPRESSION

Needs definition is a crucial step for successful system conceptual design. The main objective of this step is to converge to a common top-level description of the system, compatible with the perspectives of all the stakeholders. The term 'stakeholders' refer to people, organizations and other systems having any interest into the system. The stakeholders include users, providers, maintenance team, authorities, and assurance companies [START_REF] Hirshorn | NASA Systems Engineering Handbook[END_REF]. The system description should be understandable by all people who are involved in its development, and will serve as a basis for planning the tasks and actions to be carried out.

SysML [START_REF]OMG Systems Modeling Language, Object Management Group[END_REF] does not offer any need definition diagram. Nor does SysML v2 [START_REF] Omg | SysML v[END_REF]. In this paper, it is proposed to formulate needs in the form of sentences. The latter are labeled by identifiers.

Table I identifies needs associated with nominal behavior of the drink machine controller. Degraded behaviors -for instance "What to do when the drink machine fails connecting to the payment device?" -should also be taken into account. For space reasons, degraded behaviors are not addressed in this paper. Wait for payment before starting the preparation. H

Table I contains the needs pattern which is instantiated on a drink machine controller. The pattern categorizes the needs in two groups: 1) Functional needs: they correspond to the functions that are the services offered by the system. For instance, the need 1.1 corresponding to mixing the ingredients to provide hot drinks is a function provided by the drink machine. This function instantiates the item Non Human Devices from the branch Actuators presented in the mind map in Fig. 2. 2) Non-functional needs: they define system attributes such as security, reliability, performance, maintainability, scalability, and usability, and serve as constraints or restrictions on a system design. For example, the need identified with the identifier 2.4 (wait for payment before starting the preparation) is derived from the item Access Granting of the branch Control as shown on Fig. 2.

Note that the same process associating needs (Tab. I) and mind map (Fig. 2) has been applied in this paper for writing the needs and the requirements.

Once needs have been written and specified in cooperation with the involved stakeholders, they should be prioritized. This paper reuses the three-level scale presented in [START_REF] Wiegers | First things first: prioritizing requirements[END_REF]:

• High (H): what is critical to complete the mission.

• M edium (M): what is eventually required to complete the mission. • Low (L): what would be nice to have someday if resources permit. These needs are meant to be transformed into requirements. Defining and classifying them is very helpful for the next steps of the system development.

VII. REQUIREMENTS CAPTURE

Requirements have to be well-defined and specified to ensure successful development of new systems. To do so, collaborative work between the different stakeholders is necessary to translate their needs into requirements.

In this paper, the mind map depicted by Fig. 2 is used to guide the systems engineer when defining the requirements, The system shall process scheduled maintenance operations. F M.2 UnscheduledMaintenance

The system shall process unscheduled maintenance operations. F especially for real-time systems. The requirements are derived from the needs defined in section VI:

• Sensors requirements provide the system with the necessary inputs to perform its mission. • Control requirements are related to the controller: they process the inputs received from the Sensors part to provide the controller's outputs to the Actuators part. • Actuators requirements are related to the actions to be performed according to the controller's outputs. • Maintenance requirements are related to maintaining the well functioning of the system. Once requirements have been defined and understood, they can be classified into Functional and Non-Functional requirements. This last category can be split into three sub-categories such as Behavioral, Structural and Experiential as presented Fig. 4. Instantiating the requirement table pattern on the drink machine using SysML v2 textual notation. by Brazier, Van Langen, Lukosch and Vingerhoeds in [START_REF] Brazier | Complex Systems: Design, engineering, governance, ser. Projects and People: Mastering Success[END_REF]:

• Functional (F) requirements state the functions provided by the system. • Behavioral (B) requirements specify the desired behavior of the system together with key performance indicators. • Structural (S) requirements specify the components of the system and their relationships. • Experiential (E) requirements define the desired impact of the system in the real world with real people. This classification of requirements into these four categories helps systems engineers to know which requirements to consider at any point of the development stage [START_REF] Jiménez | A system engineering approach to predictive maintenance systems: from needs and desires to logical architecture[END_REF]. Functional requirements are considered in particular during the use-case driven analysis step (see section VIII); whereas behavioral and structural requirements are used in the design step (see section IX). Experiential requirements are specific since they can be translated into functional, behavioral and structural ones.

Table II depicts a requirement table pattern for real-time systems monitored by software controllers. This table contains an excerpt of high-level requirements, which can be further specified into lower-level ones to better describe the system. For instance, the requirement with the identifier A.2: 'The system shall use devices to inform Humans' can be refined into A.2.1: 'The system shall use visual devices to inform Humans' (S) and A.2.2:

'The system shall use sound devices to inform Humans' (S).

The requirement pattern in Table II is instantiated on the controller of the drink machine which is depicted by Fig. 4, following SysML v2 [START_REF] Omg | SysML v[END_REF] syntax. Compared to the previous version of SysML, Version 2 offers textual notation to model concepts including requirements which is intended to improve the precision, expressiveness, and usability [START_REF] Omg | SysML v[END_REF].

VIII. USE CASE DIAGRAM

A use case diagram identifies the main functions or services to be offered by the system. The pattern depicted by Fig. 5 can now be instantiated. In this paper, the pattern is instantiated on a drink machine. It may be applied to other real time systems, such as microwave oven controllers and smartphones. The result of the instantiation is shown in Fig. 6.

Referring to Fig. 5, for instance, the use case processBeverageSelection is realized by the BeverageSelector actor that corresponds to the actor labelled by number 2 on Fig. 5 and Fig. 6. The same process is applied to the rest of the instantiation.

IX. DESIGN STEP

The design step defines the architecture of the system in terms of interconnected blocks that SysML v2 names "parts" [START_REF] Omg | SysML v[END_REF]. The latter distinguishes between part definition and part instantiation. Further, parts may be pairwise connected by ports.

Conforming to the approach depicted by Fig. 1, this section first proposes a diagram pattern and subsequently instantiates it on one concrete system, a drink machine controller. Fig. 7 depicts a pattern for an architecture of real-time systems using SysML v2 textual notation, and referring to the mind map in Fig. 2. The maintenance part is not developed in this paper. ] As shown in Fig. 7, each part is first defined in the section 'Definition of parts' using the keyword part def. The part definition distinguishes be- The section called 'Usage of parts' instantiates the part definition RealTimeSystem which shows that Sensors, Controller, and Actuators belong to the part RealTimeSystem.

One of the advantages of the textual notation of SysML v2 is that the links between two parts are visible in the last section called "Binding between signals". Indeed, the keyword bind describes the link between an input signal from the transmitter part and an output signal to the receiver part. For instance, in Fig. 9 the input signal Ctrl_NHS_signal is received by the part NonHumanInteraction_Sensors and it is associated with the output signal Ctrl_NHS_signal sent by the part RTS_Controller. The RTS pattern is instantiated on a drink machine (Fig. 8). The drink machine's sensors part includes a Drink_Selector to select drinks, a Mug_Detector for sustainability concerns, and a Coin_Slot to proceed payments. These parts allow one to satisfy the needs 1.2 (Choose drink), 1.4 (Detect customer's personal mug) and 2.4 (Wait for payment before starting the preparation), respectively. The drink machine's actuators part includes a Dispenser which mix the ingredients to prepare hot drink, satisfying the need 1.1 (Mix ingredients to provide hot drinks).

The RTS pattern presented in this paper could be easily instantiated on different real time systems, such as electrically assisted bicycles, drones exploring buildings [START_REF] Razafimahazo | Mastering complexity for indoor inspection drone development[END_REF], and drones exploring high-voltage lines [START_REF] Aïello | Populating MBSE models from MDAO analysis[END_REF].

X. CONCLUSIONS Whatever the modeling language, elaboration of models is a highly complex intellectual process and SysML is no exception. The authors of this paper propose to ease that elaboration process by using mind maps.

From their experience in teaching SysML modeling of systems, the authors of this paper may witness the interest of mind maps to help students thinking about the system to be modeled before developing SysML diagrams. Having one or several mind maps as a common reference to the requirement capture, analysis and design steps enables some kind of harmonization in the way requirement, modeling assumptions, use case, sequence and block diagrams are designed.

Unlike approaches that directly switch from mind maps to SysML diagrams dedicated to one specific system, this paper introduces an intermediate step: mind maps first enable elaboration of diagrams patterns. The latter may in turn be instantiated on one or several systems. The proposed approach is step-wise illustrated on real-time systems monitored by software controllers.

Further investigations are needed to partly automate the proposed approach inside a methodological assistant. The latter will advantageously combine Cased Based Reasoning Techniques with Mind Maps.

Fig. 1 .

 1 Fig. 1. The approach developed in the paper.

Fig. 2 .

 2 Fig. 2. Mind Map for Real-Time Systems.

Fig. 3 .

 3 Fig. 3. Instantiating the Mind Map for Real-Time Systems : Drink machine.

  Fig. 5 depicts a use case diagram pattern for software controllers of realtime systems. The use case diagram names the system

Fig. 5 .

 5 Fig. 5. Use Case Diagram: Pattern for Real-Time Systems Software Controller.

Fig. 6 .

 6 Fig. 6. Use Case Diagram for a Drink Machine.

Fig. 7 .Fig. 8 .

 78 Fig. 7. Architecture: Pattern for Real-Time Systems using SysML V2 textual notation.

Fig. 9 .

 9 Fig. 9. Excerpt of Figure 7.

TABLE I PATTERN

 I AND INSTANTIATION OF NEEDS DEFINITION FOR REAL-TIME SYSTEMS CONTROLLER

		Pattern item		Drink Machine
	ID Need	ID	Need	Priority
			1.1	Mix ingredients to provide hot drinks.	H
			1.2	Choose drink (coffee or tea?).	H
	1	Functional needs: perform the functional needs for the system.	1.3 1.4 1.5	Set the amount of sugar. Detect customer's personal mug. Display the preparation progress.	M M L
			1.6	Switch to standby mode when not in service.	M
			2.1	Manage the resources.	H
	2	Non-functional needs: satisfy the non-functional needs for the system.	2.2 2.3 2.4	Use electric power. Be able to clean the machine.	H H

TABLE II PATTERN

 II FOR REQUIREMENTS DEFINITION FOR REAL-TIME SYSTEMS CONTROLLER

	Pattern item	ID	Name	Text	Requirement attributes	Type (Sec. VII)
		S	Input	The system shall receive inputs.	F
	Sensors	S.1	NoInteractionWithHumans	The system shall use sensors not interacting with Humans.	S
		S.2	InteractionWithHumans	The system shall use sensors interacting with Humans.	S
		C	Controller	The system shall process the inputs from the sensors.	F
	Control	C.1 C.2	MainFunctions Customization	The system shall perform its main functions. The system shall allow customization of the mission.	F F
		C.3	AccessGranting	The system shall grant access to the user.	F
		A	Actuator	The system shall drive actuators.	F
	Actuators	A.1	NonHumanDevices	The system shall use non-human devices.	S
		A.2	InformHumans	The system shall use devices to inform Humans.	S
		M	Maintenance	The system shall be maintainable.	F
	Maintenance	M.1 ScheduledMaintenance

ACKNOWLEDGEMENTS

SysML diagrams and mind maps have been edited with free sofware TTool and Xmind, respectively. Thanks are due to Prof. Ludovic Apvrille for supporting SysMLv2 textual notation in TTool.


Adrien Gauffriau

David Bertoin

Jayant Sen Gupta

Hijacking an autonomous delivery drone equipped with the ACAS-Xu system

Keywords: Autonomous and connected systems, Resilience, Artificial Intelligence, Reinforcement Learning, Security, ACAS-Xu

In this paper, we want to show that automated anticollision systems in aeronautical industry such as ACAS-Xu are vulnerable to hijacking threats in a urban environment which is less controlled than conventional airspace. Using reinforcement learning methods, we demonstrate the possibility to hijack the mission of a delivery drone equipped with the ACAS-Xu system in a simulated environment. Our objectives are first, to illustrate the security (interception) vulnerabilities of autonomous system and secondly, to enrich reinforcement learning benchmarks with a new one that comes from an industrial aeronautical application.

I. INTRODUCTION

Autonomy is one of the hot topics where the potential of artificial intelligence, both for perception and decision making tasks, opens new possibilities. Nevertheless, autonomous systems also raise public acceptance and certification challenges. If autonomous cars are one of the best known examples, we see the emergence of autonomous systems in aeronautics, including delivery drones, autonomous air cabs, or airplanes [START_REF] Iii | Airbus concludes attol project that featured world-first automated takeoffs and landings[END_REF]. This paper will focus on these particular systems.

Whereas most important concerns of the aeronautic certification are dependability and safety, the ability of a system to resist malevolent attack is also a major issue. Our work mainly focus on this topic with the development of an attack (interception) of an avionics system. Aviation security is mostly focused on ensuring that airports are secured by controlling people and goods passing through. Once the aircraft is flying, security is obtained by monitoring specific parts of the airspace all aircraft should respect, like airways. Airways are designed to ensure separation between pairs of aircraft, and any aircraft not respecting these rules is identified, tracked, and eventually neutralized. Nevertheless, what is valid to ensure security from malevolent attackers for standard aviation will no longer be applicable in the urban air mobility (UAM) context. Even if the concept of airway is extended in UAM, the distance to the ground, the distance between airways, and the potential number of threats will make it extremely difficult for humans to supervise all the traffic. Systems like autonomous avoidance systems are thus needed to ensure, among others, the safety, and security of UAM. Nevertheless, new methods for attacks inspired by video game testing (for instance [START_REF] Ariyurek | Automated video game testing using synthetic and human-like agents[END_REF] and [START_REF] Joakim Bergdahl | Augmenting automated game testing with deep reinforcement learning[END_REF]) may change the paradigm that was traditionally used. In this work, we illustrate this statement by developing an attack on the ACAS-Xu avoidance system (see [START_REF]Minimum Operational Performance Standards For Airborne Collision Avoidance System Xu (ACAS Xu)[END_REF]) that will possibly be used by future autonomous vehicles.

We consider the following setting: an autonomous delivery drone (target), equipped with the ACAS-Xu system for collision avoidance, whose mission is to reach a predetermined delivery area, is attacked by another drone (attacker) that tries to hijack it and lead it to a different delivery area. When no risk of collision is detected, it follows the heading to the delivery area. When an aircraft (drone) enters the risk zone around the target, its heading is updated according to ACAS-Xu recommendation. The attacker policy (its strategy) is coded in a neural network which is trained to bring the target to the alternate delivery area using reinforcement learning (RL). In this paper, we want to show how vulnerable is a drone that uses a systematic way to avoid collisions. We limited our study to the horizontal recommendations of ACAS-Xu as it is preferred to deal with non-cooperative traffic (see [START_REF] Guido Manfredi | An introduction to acas xu and the challenges ahead[END_REF]).

This paper is organized as follows. Section II presents the ACAS-Xu avoidance system. Section III presents the related works. Section IV introduces the principles of reinforcement learning (RL). Section V presents an experimental setup demonstrating the effectiveness of such attacks. Finally, section VII summarizes and concludes this paper on future perspectives.

II. OVERVIEW OF THE ACAS SYSTEM

Among the family of ACAS X, the ACAS-Xu is dedicated to drones and urban air mobility. It provides a horizontal resolution for conflicts using in real-time a set of lookup tables (LUT) that were computed offline. Using geometric parameters, the ownship consults the LUT on collision's probability for five different advisories : COC (Clear of Conflict), WL (Weak Left), WR (Weak Right), L (Left), R (Right). The system does not rely on the fact that the intruder (the attacker in our setting) also applies the same avoidance system. Therefore, this system can also be used to avoid any static object (tower, crane) or birds as

The selected advisory is the one that minimizes the probability of conflicts. The geometry of a conflict is given in figure 1, the parameters definition stands as: Fig. 1: ACAS-Xu geometry [START_REF] Katz | Reluplex: An efficient SMT solver for verifying deep neural networks[END_REF] • τ (s): Time until loss of vertical separation • R n-1 : Last recommendation provided by the ACAS-Xu The 23 LUT provides the transitions costs between the previous advisory and the next advisory. When the ownship is not in the COC state, it has to initiate the turn given by the advisory. Otherwise it can continue its mission. More information on the ACAS-Xu system can be found in [START_REF] Manfredi | An introduction to acas xu and the challenges ahead[END_REF].

III. RELATED WORKS

The use of RL methods to search for attack and interception scenarios is relatively classical. The originality comes from the use of these methods to find security weaknesses of an avionic systems. This will be illustrated in the following state of the art targeting the security in avionics, the safety of the ACAS-Xu system and the use of RL for unmanned aerial vehicle.

A. Security in avionics

The interest in the security of aeronautic systems is not new. In [START_REF] Series | Aircraft hacking[END_REF], the authors exploit vulnerabilities of the ACARS network to upload new flight plans in the Flight Management System (FMS) of aircraft. Nevertheless, the pilots still keep control of the aircraft, the attack mainly leads to increases in their workloads.

It is also possible to attack ground infrastructure like Instrument Landing System (ILS) [START_REF] Sathaye | Wireless attacks on aircraft instrument landing systems[END_REF]. The authors developed two different attacks of the ILS using radio signals. They demonstrate a systematic success rate with offset touchdowns of 18 meters to over 50 meters in lateral and longitudinal. Therefore, an attacked aircraft that performs a fully automatic landing miss the runway.

The literature is not limited to hacking the global system. [START_REF] Akram | Challenges of security and trust in avionics wireless networks[END_REF] [34] focus on attacking aircraft networks. Nevertheless, most of these results remain theoretical or academic due to the complexity and cost of deploying such attacks. Moreover, none of these attacks enable complete control of the aircraft.

Even if there is an active research on safety in avionics, the results obtained are often mitigated by the presence of humans in the loop. Moreover, the popularization of drones and their accessibility has led to the emergence of new safety issues. In the future, the introduction of low complex autonomous systems may change this paradigm. Our contribution aims to make the avionic community aware of this future challenge.

B. ACAS-Xu

Detect and avoid is a task that guarantees the safety of flying vehicles, completed by the intervention of the pilot. For large aircraft, it is still the responsibility of the pilot that may be helped by systems that give advisory and recommendations for avoidance. The most famous one is the TCAS [START_REF]of transportation Federal Aviation Administration[END_REF] that requires both planes to be equipped. The ACAS system [START_REF]Minimum Operational Performance Standards For Airborne Collision Avoidance System Xu (ACAS Xu)[END_REF] was developed for autonomous vehicles and mainly based on [START_REF] Kochenderfer | Next-generation airborne collision avoidance system[END_REF]. Several methods were explored to guarantee that this system is safe among Petri model [START_REF] Netjasov | Development, validation and application of stochastically and dynamically coloured petri net model of acas operations for safety assessment purposes[END_REF] or formal methods [START_REF] Jeannin | Formal verification of acas x, an industrial airborne collision avoidance system[END_REF]. The memory size (more than 4 GBytes) required by the look-up table of the ACAS-Xu system may be incompatible with the electronics of small drones. Thus, [START_REF] Julian | Deep neural network compression for aircraft collision avoidance systems[END_REF] developed training of neural networks that replace look-up tables with a small memory footprint. This raises the question of the safety of the neural network, which is still an open problem. [20] [7] propose formal methods to analyze neural networks and prove avoidance properties. Another approach [START_REF] Damour | Towards certification of a reduced footprint acas-xu system: A hybrid ml-based solution[END_REF] also based on formal methods, demonstrates that trained neural network behaves like the look-up table defined by the ACAS-Xu standard.

Our contribution aims to raise the security question of the ACAS-Xu system that differs from previous work that mainly focus on its safety. Up to our knowledge, this has never been done.

C. Reinforcement learning for Unmanned Aerial Vehicle

Recent successes in reinforcement learning have demonstrated the ability of reinforcement learning agents to outperform humans in many tasks [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], [START_REF] Schrittwieser | Mastering atari, go, chess and shogi by planning with a learned model[END_REF], [START_REF] Silver | Mastering the game of go with deep neural networks and tree search[END_REF], [START_REF] Silver | A general reinforcement learning algorithm that masters chess, shogi, and go through self-play[END_REF], [START_REF] Vinyals | Grandmaster level in starcraft ii using multi-agent reinforcement learning[END_REF]. Several recent works have sought to capitalize on the progress made in RL and Deep RL for the control and navigation of UAV. There are mainly two classical use cases of reinforcement learning for UAVs. The first one is flying control where the RL agents aim at providing stability and control and navigation. The second deals with mission planning where the agent is responsible for the high-level policy while control systems are implemented using classical methods such as Proportional-Integral-Derivative (PID) control systems [START_REF] Dorf | Modern control systems[END_REF].

While PID control has demonstrated excellent results in stable environments, it is less effective in unpredictable and harsh environments. Recently, several research projects have explored the possibility of using reinforcement learning to address its limitations. [START_REF] Waslander | Multi-agent quadrotor testbed control design: Integral sliding mode vs. reinforcement learning[END_REF] compared the efficiency of a model based reinforcement learning controller with Integral Sliding Mode (ISM) control [START_REF] Young | A control engineer's guide to sliding mode control[END_REF].

The authors of [START_REF] Hwangbo | Control of a quadrotor with reinforcement learning[END_REF] trained neural-network policy for quadrotor controllers using an original policy optimization algorithm with Monte-Carlo estimates. The learned policy manages to stabilize the quadrotor in the air even under very harsh initialization, both in simulation and with a real quadrotor.

[21] train autonomous controllers flight control systems with state-of-the-art model free deep reinforcement learning algorithms (Deep Deterministic Policy Gradient [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF], Trust Agent Environment a t s t+1 r t Fig. 2: Standard RL framework Region Policy Optimization [START_REF] Schulman | Trust region policy optimization[END_REF], Proximal Policy Optimization [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]) and compare their performance with PID controllers.

In [START_REF] Becker-Ehmck | Learning to fly via deep model-based reinforcement learning[END_REF], a sequential latent variable model is learned from flying sequences of an actual drone controlled with PID. This latent dynamic model is used as a generative model to learn a deep model-based reinforcement learning agent directly on real drones with a limited number of steps.

In [START_REF] Pham | Autonomous uav navigation using reinforcement learning[END_REF], the authors combine a Q-learning [START_REF] Watkins | Q-learning[END_REF] algorithm focusing on navigation policy with PID controllers. In [START_REF] Tong | Uav navigation in high dynamic environments: A deep reinforcement learning approach[END_REF], the navigation problem is decomposed into two simpler subtasks (collision-avoidance and approaching the target), each of them solved by a separate neural network in a distributed deep RL framework. An active field of research focuses on interception and defense against malicious drones. In a 1 vs 1 close combat situation, [START_REF] Vlahov | On developing a uav pursuit-evasion policy using reinforcement learning[END_REF] demonstrates the effectiveness of an A3C [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF] RL agent versus an opponent with Greedy Shooter policy [START_REF] Shaw | Fighter combat. Tactics and Maneuvering[END_REF]. In a multi-agent context, [START_REF] Xuan | Uav swarm attack-defense confrontation based on multi-agent reinforcement learning[END_REF] uses a Multi-Agent Deep Deterministic Policy Gradient algorithm (MADDPG) [START_REF] Lowe | Multi-agent actor-critic for mixed cooperative-competitive environments[END_REF] in an attack-defense confrontation markov game. [START_REF] Min | Learning-based defense against malicious unmanned aerial vehicles[END_REF] proposes a ground defense system trained with Qlearning to choose between high-level defense strategies (GPS spoofing, jamming, hacking, and laser shooting). While [START_REF] Gnanasekera | Range measurements based uav navigation for intercepting ground targets[END_REF] and [START_REF] Barrado | Counter a drone in a complex neighborhood area by deep reinforcement learning[END_REF] use RL to train a drone attacker to intercept a target drone, [START_REF] Cheng | Autonomous decision-making generation of uav based on soft actor-critic algorithm[END_REF] place the agent in the defender's position and train it with a Soft Actor-Critic algorithm [START_REF] Haarnoja | Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor[END_REF] to avoid capture.

Our contribution also aims at intercepting a target UAV using an RL agent. However, it differs on two significant points. First, it highlights the security flaws of a deterministic policy dictated by the ACAS-Xu system for collision avoidance. Second, our attacker does not seek to capture the target directly but to guide it to a specific area where it can potentially be captured. This strategy does not require any attack equipment directly implemented on the attacking UAV and can easily be applied to any UAV.

IV. REINFORCEMENT LEARNING

Reinforcement Learning is a specific field of machine learning considering sequential decision-making problems. In RL, an agent interacts with its environment during a sequence of discrete time steps, t = 0, 1, 2, 3, .... At each time step t, the agent is provided a representation of the environment state s t ∈ S, where S defines a state space. According to s t , the agent takes an action a t ∈ A, where A is the set of all possible actions. Performing this action in the environment causes the environment to transition from s t to s t+1 , and as a consequence of this transition, the agent receives a numerical reward r t ∈ R. Figure 2 illustrates the agent-environment interaction. The mapping of a state s to a probability of taking each possible action in A is called the agent's policy and denoted π(a|s) = P[A t |S t = s]. Considering a discount factor γ ∈ [0, 1], the return is defined as the discounted sum of rewards R t = T k=0 γ (k) r t+k . Deep Reinforcement learning algorithms aim at finding the policy π φ , represented by a neural network with parameters φ, that maximizes the expected return J (π φ ) = E τ ∼π φ [R(τ )] with τ = (s 0 , a 0 , ..., s T +1 ) the trajectory obtained by following the policy π φ starting from state s 0 . For continuous control problems (such as motor speed control), policy gradients methods aim at learning a parametrized policy π φ through gradient ascent on J(π φ ). These methods rely on the policy gradient theorem [START_REF] Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF]:

∇ φ J(φ) = E s∼ρ π ,a∼π φ [∇ φ log π φ (a | s)Q π (s, a)] ,
where

Q π (s, a) = E s∼ρ π ,a∼π φ [R t |s, a] is the action-value function. Q π (s, a)
represents the expected return of performing action a in state s and following π afterwards.

Policy gradients methods typically require an estimate of Q π (s, a). An approach used in actor-critic methods consists in using a parametrized estimator called critic to estimate Q π (s, a) (π φ thus represents the actor part of the agent). By relying on this principle, [START_REF] Silver | Deterministic policy gradient algorithms[END_REF] propose the deterministic policy gradient algorithm to compute ∇ φ J(φ):

∇ φ J(φ) = E s∼pπ ∇ a Q π (s, a)| a=π(s) ∇ φ π φ (s) .
The DDPG algorithm [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] adapts the ideas underlying the success of Deep Q-Learning [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF] [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] to estimate Q π with a neural network with parameters θ. In DDPG the learned Qfunction tends to overestimate Q π (s, a), thus leading to the policy exploiting the Q-function estimation errors. Inspired by the Double Q-learning [START_REF] Van Hasselt | Deep reinforcement learning with double q-learning[END_REF], the Twin Delayed DDPG (TD3) [START_REF] Fujimoto | Addressing function approximation error in actor-critic methods[END_REF] addresses this overestimation by taking the minimum estimation between a pair of critics and adding a noise to the actions used to form the Q-learning target. These tricks, combined with a less frequent policy update (one update every d critic updates) result in substantially improved performance over DDPG in a number of challenging tasks in the continuous control setting. Algorithm 1 describes TD3's complete training procedure.

V. DEVELOPMENT OF AN ATTACK ON ACAS-XU A. Notations

The following notations are used in the next sections:

) from B ã ← π φ (s ) + , ∼ clip(N (0, σ), -c, c) y ← r + γmin t=1,2 Q θ i (s , ã) Update critics θ i ← argmin θi 1 N (y -Q θi (s, a)) 2 if t mod d then
Update φ by the deterministic policy gradient:

∇ φ J(φ) = 1 N ∇ a Q θ1 (s, a) a=π φ (s) ∇ φ π φ (s) Update target networks: θ i ← τ θ i + (1 -τ )θ i φ ← τ φ + (1 -τ )φ end end
• The attacker corresponds to the intruder of the ACAS-Xu.

The attacker is not equipped with the ACAS-Xu.

B. Set up and objective

We consider an environment composed of two agents and two areas. The first agent is the delivery drone that has the mission to reach the delivery area D 1 . The target (T) is equipped with the ACAS-Xu system to trigger autonomous avoidance actions by updating its heading. The second is the attacker (A) drone. The attacker aims at hijacking the target towards an alternate delivery area D 2 , located in a different position than D 1 by exploiting the target's utilization of the avoidance ACAS-Xu system. The attacker's policy is trained for this purpose using Deep Reinforcement Learning. In our setting, for the sake of simplicity, both agents can only move in the same horizontal plan. Ascending and falling are not allowed. The figure 3 provides a graphical representation of the set-up.

The Xu version of the ACAS system is dedicated to drones, and only provides horizontal avoidance recommendations. Among the other version of the ACAS, it exists the Xa version dedicated to large aircraft that provides vertical avoidance like the TCAS. Thus, we restrict the interception to an horizontal plan since ACAS-Xu does not provide vertical avoidance recommendation. Even if, the target may have a vertical flight plan, it is very easy to configure, without reinforcement learning, the attacker to be always on the same horizontal plan of the target. In a future work, we may extend the study by considering a target equipped with the Xu and the Xa version and train an agent for a 3D interception. c) Transitions: The target agent's velocity is constant during the whole episode. Its heading is updated according to the ACAS-Xu system advisory. If the advisory provided is different from COC, the following heading update δα n will be used:

• WL : + 0.15 rad • WR : -0.15 rad • L : + 0.3 rad • R : -0.3 rad If the advisory is COC, the λα update enables the target to reach the heading to the delivery area with a maximum variation of 0.3 rad. We limit this variation to be more representative of a real drone maneuverability and avoid instability due to big turns. For both agents, the update of the speed vector is given by

V n+1 = V n + δV n α n+1 = α n + δα
R n =      0 if n = 0 R n-1 if D n-1 ≥ D n R n-1 + (D n -D n-1 ) if D n-1 < D n with D n = P T n -P E .
The reward is increased for each step that globally reduces the distance between the target agent and the interception destination. We used the state-of-the-art Gym [START_REF] Brockman | Openai gym[END_REF] framework to implement our training environment.

VI. EXPERIMENTS AND RESULTS

This section presents the different scenarios of experiments conducted. In all experiments, the target has a fixed speed of 400 ft/s and is following ACAS-Xu avoidance recommendations in case of presence of another flying object in its vicinity. When there is no obstacle, the drone follows the direction leading to its delivery area.

A. Training setups

We train three different agents depending on the maximum speed we allow the attacker in order to study the influence of the speed ratio between the attacker and the target. Three configurations are tested:

• 300 ft/s • 600 ft/s • 1000 ft/s We fix the size of the playground as a square of 100000 feet. We developed a gym environment for training purposes to simulate the target behavior, following the ACAS-Xu function, and train the attacker policy using our RL algorithm. For each training scenario, we randomly draw the positions of the delivery zone, the alternative delivery zone where the attacker has to bring the target, the initial position of the target, and the initial position of the attacker. At the end of this stage, we get three trained agents designated by A 300 , A 600 and A 1000 .

In every setups, we trained an RL agent using Stable-baselines3 [START_REF] Raffin | Stable baselines3[END_REF] implementations of TD3. In each experiment, the agent is constituted by an actor and two critics. We use a two-layer feedforward neural network of 400 and 300 hidden nodes respectively, with rectified linear units (ReLU) between each layer for both the actor and critic, and a final two neurons output layer with tanh for the actor.

TD3 is an off-policy algorithm, during training transitions (s t , a t , s t+1 , r t+1 ) are stored in a replay-buffer [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF] and drawn randomly in the form of mini-batches during the weight update phase. We conducted all of our experiments using a replaybuffer of size 50000 and a mini-batch size of 512. TD3 trains a deterministic policy in an off-policy way, which is not favorable for exploration during training. We encouraged exploration of the TD3 agent by adding an action noise drawn from the Ornstein-Uhlenbeck process, as suggested in [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF]. For every scenario, we trained our agents on 6 million steps.

B. Evaluation setups

Once the different models are learned, we evaluate their performance by randomly sampling new testing scenarios where we fix the positions of the delivery zone, the alternative delivery zone where the attacker has to bring the target, and the initial position of the target. For each of these scenarios, we randomly draw a significant number of initial positions of the attacker in the playground. For each of these random initializations, we run the scenario using the three trained policies and to assert whether the attacker succeeds in hijacking the target. For each test scenarios, we plot a map of the successful initial attacker positions and unsuccessful initial positions and estimate a percentage of successful attempts (success rate). We run 1000 different scenarios with the three trained policies.

Depending on the ratio between the speed of the target and the maximum speed of the attacker, we may theoretically know if the interception is possible depending on the initial geometry. We define the disc D centers on the delivery zone with a radius of R that is computed taken into account the distance d T D between the initial position of the target (I t ) and the delivery zone (D 1 )and the ratio v max a vt between the speeds of the attacker and the target.

R = v max a v t d ItD1 (1)
When the attacker is in D, the interception is theoretically achievable since the attacker has the possibility to reach the delivery zone before the target and can thus interact with it. For a given scenario where the delivery area, the alternative delivery area and the initial position of the target are fixed, D is also fixed for all episodes.

To evaluate the performance of the hijacking policy, we can empirically estimate the reconstructed probability density function of the success rate on all test scenario. We also define following metrics: M1: Number of successes over the number of episodes for initial target position in playground. M2: Number of successes over the number of episodes for initial target position in D. M3: Number of successes over the number of episodes for initial target position not in D. n step : Mean of the number of steps when the episode is a success r dist : Mean of the distance covered by the target over the initial distance from the target to the alternate delivery area when the episode is a success. M1, M2 and M3 are success rates depending on the initial position of the target. n step , and r dist are complementary metrics that evaluate the difficulty for the attacker to hijack the target.

C. Results

1)

Step 1 Exploration of scenarios: Firstly, we generate 1000 random scenarios by fixing I t and D 2 as described in the previous section. In order to ease analysis, D 1 is always in the center of the playground. This choice does not change initial geometry exploration, although it may mask some corner cases. For each scenario, we run for the three trained agent A 300 , A 600 and, A 1000 1000 episodes with random position for I t . As expected during training, the agent A 300 was not able to perform interceptions. Thus, we only focus on A 600 and A 1000 in the rest of this section.

We present on figure 4 the distribution of the success rate for the 1000 scenarios for agent A 600 and A 1000 with histograms. We see that the maximum speed of the attacker has a huge impact on the capacity of the attacker to intercept the target. This is confirmed by an average success rate of 39.2% for A 600 and 91.4% for A 1000 . Then, in figure 5, we present D 2 and I t for the scenarios that have a good success rate (> 90%) and a bad success rate (< 10%) for trained agents A 600 and A 1000 .

These figures highlight the impact of the initial geometry (relative position of the different elements) on the success rate. We recall that the D 1 area is in the center of the playground for all scenarios. When the ratio v max a v t is high (top 2 of Figure 5), the distance I t D 1 has a huge impact on the success rate of a scenario. The closer the target is to the first delivery area D 1 , the more the interception is difficult. Interpretation is not that straightforward when the ratio is low (bottom 2 of Figure 5). It seems that the learned policy is more efficient when the alternate delivery area D 2 is between or almost between the initial position of the target I t and the initial delivery area D 1 .

Fig. 5: Alternate delivery areas and initial positions of the target

2)

Step 2 Exploration of selected scenarios : this section focuses on some selected scenarios to provide a deeper look into how behave the different policies. In Figure 6, we plot the success rate of A 1000 vs the success rate of A 600 . From this graph, we select 3 scenarios S 1 , S 2 and S 3 .

• S 1 is a scenario where both trained agent have a bad success rate; • S 2 is a scenario with a good success rate for A 1000 and a relatively poor for A 600 ; • and finally S 3 where both agents have a good success rate. Each scenario correspond to a specific value of D 1 , D 2 and I t . For all scenarios, A 1000 performs better than A 600 . This confirms the global success rate metric and highlights the necessity for the attacker to have a higher velocity than the target. The average ratio also shows that the task hijacking is more accessible when the maximum speed of the attacker is higher.

The success rate inside R (m 3 ) is consistently higher the than global one (m 1 ), (even if not always equals to 100% as we could expect).

From Figure 7, one characteristic which seems discriminant is the distance between the initial position of the target I t and the delivery zone D 1 : the smallest it is, the hardest it is to perform the hijacking. Even inside the R, the success rate is poor, especially in S 1 . This can be explained by the fact that the circle has been defined considering the attacker goes directly to D 1 . In practice, the initial heading the attacker is random so it has to change direction to head towards D 1 . When there is not a lot of time, the change of heading is too long and the attacker cannot hijack. When time is longer, the effect is less visible, especially for A 1000 .

Failures inside R for S 3 seem to form a pattern that would require additional investigations. It may show that our policy is not perfect or that our training scenarios did not explore enough certain configurations. This is the object of next step.

3) Step 3 Exploration of trajectories : In this section, we present some representative observed trajectories to understand the red area of (S 3 , v max 1000 ) and the red area that is inside R in (S 2 , v max 1000 ). Figure 8 presents the trajectories for two close initial positions of the attacker for scenario (S 3 , v max 1000 ). This corresponds to the red area upper right of D 1 on figure 7. With the first position, the interception is not achieved. We can observe that the attacker has a trajectory that intercepts the target after D 1 . On the contrary, for the second episode, the attacker intercepts the target before D 1 . Failures of the (S 3 , v max 1000 ) scenario have similar behaviors. The attacker tries to intercept the target after the D 1 area. This is mainly due to the reward function used during the training of the attacker agent. We used an heuristic that rewards the attacker when the target reduces the distance with D 2 in order to ease the learning of the policy. In order to push the attacker to hijack before the target reaches D 1 , we would need to modify the reward to alleviate this undesired behavior. The proportion of episodes leading to such cases is low but definitely the reward function should be updated to avoid such phenomena.

Figure 9 shows the trajectories for two close initial positions of the attacker for the scenario (S 2 , v max 1000 ). This corresponds to the red area inside R. In the first episode, the initial velocity of the attacker is small and its acceleration is not strong enough to be able to interact with the target before it reaches D 1 . When the initial velocity of the attacker is higher, in the second episode, the attacker is able to interact with the target almost from the beginning, and it is able to hijack it. These situations can occur when the attacker is opposite to the target compared to D 1 . It could be solved by increasing the minimal initial speed of the attacker, fixing the initial heading of the attacker towards D 1 or increasing the attacker maximum acceleration.

VII. CONCLUSION

This work highlights the possibility of using reinforcement learning to train a malicious agent to hijack a delivery drone equipped with the ACAS-Xu avoidance system. Up to our knowledge, we are the first to demonstrate the possibility of fuzzing the algorithm of an autonomous avoidance system. With the current ambition in the aerospace industry for the development of autonomous systems (specifically air taxis and autonomous delivery drones), we are highlighting a security breach. The acceptance of these autonomous vehicles requires the resolution of these security issues.

As the ACAS-Xa system uses the same principle of tables, we believe that our method also applies.

Although we have made simplifying assumptions within the simulation environment, we believe that RL is efficient for attacking the ACAS-Xu system. An extension could be focused on adding complexity to the environment by adding static and dynamic obstacles and considering the vertical dimension. This should imply an extension of the action state of the attacker (new dimension) and an increase of the complexity of the reward function to handle obstacles.

In a future work, we consider training both agents with reinforcement learning in a zero-sum two-player games (as in [START_REF] Silver | Mastering the game of go with deep neural networks and tree search[END_REF], [START_REF] Silver | A general reinforcement learning algorithm that masters chess, shogi, and go through self-play[END_REF]) to produce collision avoidance system policies robust to malicious attacks.

We believe that, in order to improve the security, a reference use case should be provided with the ACAS-Xu. Access to Look Up Tables is unfortunately not possible for organizations that are not part of the RTCA/EUROCAE, the authority responsible for the normalization of the ACAS-Xu. Nevertheless, thanks to [START_REF] Katz | Reluplex: An efficient SMT solver for verifying deep neural networks[END_REF], neural networks approximating ACAS-Xu lookup tables are available. Therefore, we would like to replace the ACAS-Xu look-up table with these neural networks inside the gym environment and free it through our git repository (will be available for the conference). This would enable Reinforcement Learning and Security communities to work on this topic with the final objective of improving security in aeronautics.

ACKNOWLEDGEMENT

This project received funding from the French "Investing for the Future -PIA3" program within the Artificial and Natural

• D 1 :

 1 Delivery area. The target's destination objective • D 2 : Alternate delivery area. The attacker destination objective: the interception zone • I t : Initial position of the target. • I a : Initial position of the attacker. • v t : Speed of the target that is constant. • v a : Speed of the attacker. v max a is the maximum possible speed of the attacker • The target corresponds to the ownship of the ACAS-Xu Algorithm 1: TD3 Initialize critic networks Q θ1 , Q θ2 , and actor network π φ with random parameters θ 1 , θ 2 , φ Initialize target networks θ 1 ← θ 1 , θ 2 ← θ 2 , φ ← φ Initialize replay buffer B for t = 1 to T do Select action with exploration noise a ∼ π φ (s) + , ∼ N (0, σ) and observe reward r and new state s Store transition tuple (s, a, r, s ) in B Sample mini-batch of N transitions (s, a, r, s

Fig. 3 :, π 2 ]

 32 Fig. 3: Set up

  n and position update by P n+1 = P n + V n+1 . d) Reward model: The choice of the reward function is not trivial. The training capacity and future policy of the RL agents are deeply impacted by the reward model used during training. The reward model design, often called reward shaping, may lead to strange and unexpected behaviors.

Fig. 4 :

 4 Fig. 4: Distribution of the scenario's success rate for A 600 and A 1000

Fig. 6 :

 6 Fig. 6: Scenarios selected for exploration

Fig. 7 :Fig. 8 :

 78 Fig. 7: Success/Failures for different initial position of the attacker among selected scenarios

  Table I provides a complete description of the training parameters used during training.

	Parameter	Value
	Training steps	6,000,000
	Learning rate	0.001
	γ	0.99
	Policy delay	2
	τ	0.005
	target policy noise	0.2
	Ornstein-Uhlenbeck Noise	0.01
	Replay buffer size	50,000
	Mini-batch size	512

TABLE I :

 I Hyper-parameters used in TD3 agents training

TABLE II :

 II Metrics for (S x , A y ) x∈(1,2,3),y∈[START_REF] Akram | Challenges of security and trust in avionics wireless networks[END_REF][START_REF] Ariyurek | Automated video game testing using synthetic and human-like agents[END_REF]