domain: spi phys math info

text:
Xavier Olive

Jeremy Grignard

Thomas Dubot

Detecting Controllers' Actions in Past Mode S Data by Autoencoder-Based Anomaly Detection

Keywords: trajectory analysis, anomaly detection, ATC order detection, autoencoder

The preparation and execution of training simulations for Air Traffic Control (ATC) and pilots requires a significant commitment of operational experts. Such a mobilisation could be alleviated by a decision support tool trained to generate a realistic environment based on historical data. Prior to studying methods able to learn from a dataset of traffic patterns and ATC orders observed in the past, we focus here on the constitution of such a database from a history of trajectories: the difficulty lies in the fact that past flown trajectories are properly regulated, that observed situations may depend on a wide range of potentially unknown factors and that ownership rules apply on parts of the data. We present here a method to analyse flight trajectories, detect unusual flight behaviours and infer ATC actions. When an anomaly is detected, we place the trajectory in context, then assess whether such anomaly could correspond to an ATC action. The trajectory outlier detection method is based on autoencoder Machine Learning models. It determines trajectory outliers and quantifies a level of abnormality, therefore giving hints about the nature of the detected situations.

Results obtained on three different scenarios, with Mode S flight data collected over one year, show that this method is well suited to efficiently detect anomalous situations, ranging from classic air traffic controllers orders to more significant deviations. Detecting such situations is not only a necessary milestone for the generation of ATC orders in a realistic environment; this methodology could also be useful in safety studies for anomaly detection and estimation of probabilities of rare events; and in complexity and performance analyses for detecting actions in neighbouring sectors or estimating ATC workload.

I. INTRODUCTION

The preparation and execution of training simulations for controllers and pilots relies on realistic environments which are based on common trajectory patterns, involving ATC orders typical of a context and traffic situations in a given area of interest. In the context of pilot simulations, the generation of relevant ATC communications consistent with traffic surrounding a simulated flight is a costly task requiring the involvement of an operational expert, if not an air traffic controller certified on this area. A decision support tool, able to generate realistic traffic situations and associated ATC actions and to adapt to the behaviour and actions of the pilot student could ease the process. Similarly, a tool assisting air traffic controllers, highlighting and suggesting required actions based on how peers would react in a similar situation could be a significant asset in ATC training.

Current state of the art regarding optimisation methods solving common ATC problems mainly focus on how to efficiently deconflict traffic, preferring optimality at the expense of realism of the inherent actions in a given context. Optimisation techniques have proven to be efficient at solving a wide range of conflicts at a tactical or strategic level: they are often illustrated with toy problem scenarios like the round-about or the wall problems, proven to be scalable to a large number of aircraft. Ensuring such methods are still efficient in corner-case situations is a necessary validation step. Nevertheless, it turns out to be difficult to relate to operational situations. In a context where realistic simulations are key for a safe and efficient training for pilot and controller students, such techniques require to be fine-tuned with operational feedbacks to guarantee that the generated orders resemble those an expert would have given in a similar context.

On the other hand, Machine Learning (ML) methods properly trained to learn from a dataset of historical situations seem appealing to generate realistic ATC orders. However, they require building a catalog of past ATC actions. Such a dataset is not easy to build from the observation of real traffic: flight plans are optimised at a strategic level; regulations are emitted upstream by ATC centers to avoid overload situations; and above all, no loss of separation is supposed to be observed as ATC process traffic based on surrounding aircraft, meteorological events and usual practices.

A first semi-supervised learning framework [START_REF] Srinivasamurthy | Iterative learning of speech recognition models for air traffic control[END_REF] combining speech and radar data to predict controller commands has been implemented using Automatic Speech Recognition. The method considers communication with ATC and its impact on relevant trajectories; research is still ongoing to improve the accuracy of speech recognition models applied to ATC communications [START_REF] Subramanian | Custom IBM Watson speech-to-text model for anomaly detection using ATC-pilot voice communication[END_REF]. We consider here a complementary approach, based only on flight tracks, without a systematic transcription of ATC communications.

ML methods are an excellent tool for detection and prediction on datasets displaying the more variance in the more limited scope. In order to get the most of such methods, we focus here on routes between city pairs, i.e. trajectories flying from city A to city B. We assume that most flights are executed according to their flight plans, where interactions with ATC would most likely be limited to standard clearances. By comparing how aircraft fly this route all year long, we assume that most ATC actions result in a deviation on the trajectory: we therefore focus on deviations from the route drawn by most flights to infer ATC actions and try to understand their causes.

Filed flight plans (DDR files), ADS-B (possibly augmented with multilateration) trajectories, weather conditions (METAR) and even radio recordings (www.liveatc.net) are sources of information that can be used to assess the intent behind each ATC order, whether it resembles more a classic clearance, a deconfliction order placed to avoid a loss of separation, or a reaction to more unusual events. Different types of detected orders could support different types of simulations, requiring classical ATC actions or more specific actions related to more specific situations like thunderstorms, rerouting, runway change at destination airfield.

We present in the following an autoencoder-based anomaly detection method able to detect unusual trajectory patterns and to quantify the degree of abnormality of each outlier. Section II presents a literature review of the state of the art of ML techniques applied to aircraft trajectories. Section III focuses on the mathematics behind mentioned techniques. Section IV describes the studied dataset, including one year of traffic between three city pairs, mostly based on the OpenSky Network [START_REF] Schäfer | Bringing up OpenSky: A large-scale ADS-B sensor network for research[END_REF] platform; then explains how to prepare and format the data to train the autoencoder. In Section V, we attempt to put some trajectories detected as unusual into context and suggest plausible interpretations explaining why controllers could have processed the traffic that way. Section VI puts the results in perspective, showing how the method could be used both for our primary goal and for estimating probabilities of rare events in safety analyses.

II. RELATED WORK AND LITERATURE REVIEW

Research addressing ML techniques to tackle ATM problems is rather recent compared to their equivalent modelling and optimization problems, often solved by mixed-integer linear programming [START_REF] Cafieri | Aircraft deconfliction with speed regulation: new models from mixed-integer optimization[END_REF], constraint programming [START_REF] Allignol | Constraint programming for air traffic management: a survey[END_REF] or metaheuristics [START_REF] Durand | Automatic aircraft conflict resolution using genetic algorithms[END_REF]. ML techniques have then been used to learn ATC workload from past operations [START_REF] Gianazza | Forecasting workload and airspace configuration with neural networks and tree search methods[END_REF], to predict traffic flows above California [START_REF] Lü | Traffic flow prediction with big data: A deep learning approach[END_REF] or to predict diversions in freight transportation [START_REF] Di Ciccio | Detecting flight trajectory anomalies and predicting diversions in freight transportation[END_REF].

ML brings a new perspective to ATM related problems: unsupervised machine learning focuses on detecting and describing how aircraft behave [START_REF] Olive | Quantitative assessments of runway excursion precursors using Mode S data[END_REF], [START_REF] Nicol | Functional principal component analysis of aircraft trajectories[END_REF]; supervised machine learning applies to labeled dataset and focuses on predicting aircraft trajectories [START_REF] Lü | Traffic flow prediction with big data: A deep learning approach[END_REF], [START_REF] Di Ciccio | Detecting flight trajectory anomalies and predicting diversions in freight transportation[END_REF], [START_REF] Liu | Predicting aircraft trajectory choice -a nominal route approach[END_REF]. Reinforcement learning addresses more traditional optimisation problems [START_REF] Brittain | Autonomous aircraft sequencing and separation with hierarchical deep reinforcement learning[END_REF] where the search space may be explored through interactions with a simulator or based on a history of situations.

Unsupervised machine learning techniques have produced convincing results in terms of trajectory clustering [START_REF] Gariel | Trajectory clustering and an application to airspace monitoring[END_REF], [START_REF] Rocha Murca | Trajectory clustering and classification for characterization of air traffic flows[END_REF], with applications to flow prediction, anomaly detection and safety analyses. In particular, Principal Component Analysis has proved useful in its functional form to detect variational patterns with applications to final approaches [START_REF] Olive | Quantitative assessments of runway excursion precursors using Mode S data[END_REF] and climb profiles [START_REF] Nicol | Functional principal component analysis of aircraft trajectories[END_REF]. Multi-kernel anomaly detection (MKAD) [START_REF] Das | Multiple kernel learning for heterogeneous anomaly detection: algorithm and aviation safety case study[END_REF] is arguably the current state of the art in the field of flight track anomaly detection, with convincing use cases on US terminal maneuvering areas [START_REF] Matthews | Automated discovery of flight track anomalies[END_REF]. The method combines kernel functions (a measure of similarity) on discrete and continuous features in order to isolate a predefined percentage of anomalous trajectories.

We present in this paper a similar approach based on autoencoders, a particular kind of neural networks which recently made a name in anomaly detection. They have already been used to find breakpoints in time series [START_REF] Lee | Time series segmentation through automatic feature learning[END_REF] and to predict realistic transitions in sector configurations [START_REF] Dubot | Predicting sector configuration transitions with autoencoderbased anomaly detection[END_REF]. Autoencoders are comparable to compression methods. They are trained to reconstruct, i.e. compress then decompress, data (trajectories) passed in input. In practice, they learn to reconstruct most trajectories in a training dataset and fail to reconstruct the more atypical ones. Anomaly detection is based on the distance, the reconstruction error, between real trajectories and their reconstructed copies. The most notable difference with MKAD lies in the fact that no assumption needs to be made on the proportion of anomalies present in the dataset. Detected anomalies may be related to peculiar ATC situations and investigating them may also help refining safety models designed to assess risks and estimate probabilities of rare events.

III. AUTOENCODER-BASED ANOMALY DETECTION A. Machine Learning Techniques for Anomaly Detection

Machine Learning techniques presented in the literature review address the unsupervised learning problem. This task infers the hidden structure behind unlabeled data. Instead of working on data with information known a priori about the context, including hazardous situations, we focus on a set of unlabeled trajectories, try to grasp their structure and eventually find outlying elements.

Principal component analysis (PCA) explains the variance in data by projecting trajectories on axes holding the more variance, determined by the eigenvalue of the covariance matrix of the samples. Variance ratio diagrams plot how much each component holds of the total variance. The user must then choose an appropriate number of components that describes the data best. Projecting the data on a limited number of components may reveal some structure in the data, sometimes call for some clustering analysis analysis in order to understand and later detect particular behaviours and atypical elements falling outside emerging structures. t-distributed Stochastic Neighbour Embedding (t-SNE) is a similar dimension reduction and visualisation technique addressing non-linear transformations and which can improve the performance of classification methods.

One-class support vector machines (SVM) are extensively used for anomaly detection. SVM excel at learning decision boundaries. One-class SVM are designed to learn to separate nominal and anomalous data points based on a pairwise kernel (similarity) matrix and identify statistically significant anomalous examples. The kernel function defining the similarity between samples is crucial to the performance of the algorithm. A commonly chosen kernel is the radial basis function (rbf) but other kernels like the cosine similarity may yield better results depending on the case study. The rbf calls for a hyperparameter σ that determines the width of the Gaussian distribution over the vector space. In addition, a hyperparameter ν must be provided by the user which corresponds to the maximum fraction of data assumed to be anomalous. Eventually anomalous samples are ordered by their distance to the separating hyperplane: a higher distance may statistically correspond to a higher degree of anomaly. It may happen that some outliers found by a one-class SVM are consistent with those found by clustering techniques applied on projections resulting from a PCA.

Multiple kernel methods are linear, non-linear or datadependent combinations of several kernels [START_REF] Gonen | Multiple kernel learning algorithms[END_REF]. The simplest and most common method is a weighted combination of kernels. In unsupervised ML problems such as anomaly detection, the resulting kernel is a convex combination of all kernels which are computed over multiple features. Each kernel is weighted based on the a priori knowledge of the expert on the study case. Matthews [START_REF] Matthews | Automated discovery of flight track anomalies[END_REF] and Das [START_REF] Das | Multiple kernel learning for heterogeneous anomaly detection: algorithm and aviation safety case study[END_REF] use multiple kernel methods to detect potential safety anomalies in large databases of flight tracks consisting of discrete and continuous data.

B. Autoencoders

Autoencoders enter the artificial neural networks category. As the name suggests, they consist of two stages: encoding and decoding. A single-layer autoencoder is a kind of neural network consisting of only one hidden layer. Autoencoders aim at finding a common feature basis from the input data. They reduce dimensionality by setting the number of extracted features less than the input. Autoencoder models are usually trained by backpropagation in an unsupervised manner. The underlying optimization problem aims at minimizing the error of the reconstructed results from the original inputs.

s 0 s 1 s 2 s 3 . . . s d y 0 y 1 . . . y h " s 0 " s 1 " s 2 " s 3 . . . " s d
where (u, v) is a loss function decided according to the input range, typically the square loss:

(u, v) = ||u -v|| 2 (2)
or the cross-entropy loss:

(u, v) = ∑ i u i • log(v i ) + (1 -u i ) • log(1 -v i ) (3)

IV. APPLICATION TO TRAJECTORY ANOMALY DETECTION A. The Trajectory Dataset

Mode S has become one of the most important technologies in air traffic management as it supports the operation of secondary surveillance radar (SSR), traffic alert and collision avoidance systems (TCAS), and Automatic Dependent Surveillance-Broadcast (ADS-B). In practice, transponders in aircraft are selectively interrogated by sensors (radars) to provide situational awareness through the exchange of binary encoded information.

To be able to selectively interrogate aircraft, transponders aboard aircraft have been assigned a unique 24-bit identifier. The assignment of addresses is done by the national authority where the aircraft is registered. These identifiers are included in all Mode S messages and identify each aircraft.

Aircraft reply to ground sensor requests with messages of different types, called downlink formats (DF). We focus here on DFs 17 and 18 which are not transmitted upon interrogation. They contain all information needed to determine the aircraft's identity, location, and velocity. These squittered information are called Automatic Dependent Surveillance-Broadcast (ADS-B). Extended surveillance data from avionics such as intent-and status information is provided via Comm-B messages (DF 20, 21): such data could be of great help in future work to detect ATC actions on the indicated airspeed since ADS-B only contains ground speed.

The OpenSky Network [START_REF] Schäfer | Bringing up OpenSky: A large-scale ADS-B sensor network for research[END_REF] is a crowd-sourced sensor network collecting air traffic data. The collected data used for this study contains only ADS-B data of specific callsigns from January to December 2017. We consider in the following three study cases, associated to three different city-pairs.The data we collected consists of one year worth of data on:

1) 28 different callsigns associated to flights from Paris Orly (LFPO) to Toulouse Blagnac (LFBO) airports (3536 trajectories); 2) 8 different callsigns associated to flights from Paris Charles-de-Gaulle (LFPG) to Frankfurt (EDDF) airports (2105 trajectories); 3) and 4 different callsigns associated to flights from Amsterdam Schiphol (EHAM) to Madrid Barajas (LEMD) airports (977 trajectories). (44

• 17 N, 0 • 57 E) (45 • 32 N, 2 • 45 E) LFPG EDDF (49 • 9 N, 5 • 40 E) (51 • 1 N, 7 • 40 E) EHAM LEMD (41 • 18 N, 2 • 48 W ) (44 • N, 0 • 37 W )

TABLE I DEFINITION OF BOUNDING BOXES FOR THE THREE CASE STUDIES

In the following, we will refer to these scenarios respectively as the Toulouse, the Frankfurt and the Madrid scenarios.

B. Data preparation

Trajectories are mathematical objects used to describe the evolution of a moving object. They are described by a state vector with parameters (x(t), y(t), • • • ) that evolve in time. In practice, this state vector is only known at some sampled times. For clarity concerns, we name trajectory a sequence of recordings associated to an aircraft. We describe in this section the full chain of data pretreatments applied to each trajectory in our dataset.

Trajectories from the OpenSky Network are unequally sampled and subject to errors happening in different steps of the acquisition chain: errors before the emission of data (imprecision in the positioning, quantification artifacts) and errors in the receiving and decoding of the data by feeders. A cascade of basic median filters is first applied to filter out irrelevant values from each individual trajectory.

For each scenario, the second step consists in defining a bounding box containing the set of trajectories. We chose to consider roughly drawn bounding boxes including most trajectories (definitions in Tab. I) before they enter the terminal maneuvering area (TMA, the greyed area in Fig. 2). We also chose to include in the bounding boxes the first navigational aid (i.e. a set of geographical coordinates) of the standard arrival (STAR) procedures (NARAK in Fig. 2).

The next step in the data preparation consists in resampling the trajectories in the bounding box we selected. Since autoencoders have a fixed number of inputs (d in the example of Fig. 1), we resample each subset of trajectory cropped to our learning box so as to get only d equally distributed samples. Fig. 2 plots a resampling with d = 15 for illustrations' sake but we chose a larger number of samples d = 150 for our experiments. This choice is arbitrary and we found that other values of d in the same order of magnitude have no significant impact on the results.

Features are chosen among all data provided in the ADS-B specifications: latitude, longitude, GPS and barometric altitude, track angle, ground speed, vertical speed. Different airspeeds (CAS, IAS, TAS, etc.) are sent by aircraft upon request on DF 20 and 21 but for the sake of clarity, we chose to keep these features out of our dataset for future work beyond the scope of this paper. Controller's actions are most often expressed in terms of altitude ("climb to flight level 310"), track angle ("turn left heading 210", "route direct NARAK"), and speed ("reduce speed to 160 kts"); since speeds are expressed in IAS and not in ground speed, we focused on track angles and altitude profiles.

Data normalization, also known as feature scaling, is widely used in machine learning in order to deal with variations in a wide range of shorter or larger intervals by standardizing the features. Lee [START_REF] Lee | Time series segmentation through automatic feature learning[END_REF] shows that feature scaling allow gradient descent algorithms to converge much faster than without it. Various methods can be used to rescale the range of features:

• min-max normalization rescales features in [0, 1]:

x =
xmin(x) max(x)min(x) [START_REF] Cafieri | Aircraft deconfliction with speed regulation: new models from mixed-integer optimization[END_REF] has been used for the results presented in the following. • standardization rescales features so that each feature has a zero mean and unit variance:

x = x -µ(x) σ (x) (5)

C. Learning protocol

For the sake of clarity, we focus here on an analysis on normalized track angles. The input dimension, i.e. the number of neurons on both input and output layers of our autoencoder, has been set to d = 150. The embedding dimension, i.e. the number of neurons on the hidden layer has been set to a lower value of 64. All neurons are defined with a sigmoid activation function. The loss function used is the mean squared error as defined in [START_REF] Subramanian | Custom IBM Watson speech-to-text model for anomaly detection using ATC-pilot voice communication[END_REF], which compares vectors with their reconstructions, i.e. their images through the autoencoder. The learning rate is set to 10 -2 and the training stops when the loss stops improving by more than 10 -4 between two epochs. Fig. 3 plots the loss evolution on the Toulouse scenario.

As a result of our training process, we get a reconstruction error, i.e. a measure of the difference between a given trajectory and its autoencoded representation. Fig. 4 plots the distribution of these reconstruction errors. The model is trained to minimize the sum of all reconstruction errors, so the distribution is centered around zero.

The same distribution is plotted on a logarithmic y-axis to emphasize the few specific trajectories with higher reconstruction errors: we study in the following section the contextual situations associated to such specific trajectories pointed on the distribution. We focused first on trajectories with the highest reconstruction errors, then on a few situations with lower reconstruction errors, closer to the tail of the bell-shape distribution.

V. ANALYSIS OF ANOMALOUS

The training phases on the three different scenarios yields reconstruction errors distributed as a half bell-shaped distribution centered on zero; few trajectories are ranked with higher values. (Fig. 4, 5 and 6) We focus our attention on trajectories The analysis of the context was made after analysing traffic around anomalous detected trajectories. ADS-B may be an incomplete source of data for a thorough analysis of the situation since some aircraft are still not properly equipped. In that case, flight plans and trajectories inferred by multilateration give a hint but situations analysed in this paper have been selected so as to be representative of the wide panel of detected situations and to be explainable with ADS-B tracks openly accessible. A perfect analysis would have also involved confirmation of In general, we found that very high reconstruction errors are associated to less common situations; explanations are more to be found in the METAR history or regulation history. Conversely, reconstruction errors closer to the tail of the bell-shaped distribution, are more prone to yielding nominal situations which may be explained by ATC orders issued for a deconfliction or sequencing purpose.

A. Highest values of reconstruction errors relate to less common situations 1) Traffic interruption: Flight AFR51ZU on June 28th (Fig. 7) yields a high (although not the topmost) reconstruction score and its peculiar route on final approach called the authors' attention. This day was marked with a lot of delays because of weather. METAR on that day is particularly explicit: thunderstorm (TS), presence of cumulonimbus (CB) and a gusting wind forecast (20G35KT). Local news (Fig. 8) reported that traffic was interrupted on that day when AFR51ZU was in approach. This interruption may explain the two loops during final approach.

2) Weather impact on STAR procedures: Traffic around Toulouse airport on July 19th was also impacted by cumulonimbus located on the STAR procedure, around the NARAK beacon. All aircraft (including AFR47FW, see Fig. 9) coming from the North-East were deviated to the West prior to entering the TMA, so as to be sequenced on the usual STAR procedures applicable to aircraft coming from the North-West.

The background on Fig. 9 has been produced with data from the Spinning Enhanced Visible and Infrared Imager (SEVIRI) collected by the Meteosat Second Generation series of satellites. Thermal IR data has been used to estimate the location of cumulonimbus, with red areas indicating the most severe convection.

3) QFU change: QFU refers to the magnetic heading of the runway in use. ATC may change the QFU at any time depending on the weather conditions. On August 27th, EZY24EH took a very peculiar route (Fig. 7) as other aircraft were landing from the south on QFU32. After the last flight has landed (16:51), a first aircraft landed at 17:08 on QFU14 and EZY24EH was on hold before landing 3 minutes later.

4) Avoiding regulations: Many regulations were in place between Paris and Frankfurt (Reims and Langen ACC) in early evening on June 2nd, 2017, because of cumulonimbus in the area. In particular, DLH2F was impacted by a 15 minute delay before departure because of a regulation filed by Frankfurt arrivals. Figure 11 In contrast, we found more classical and conventional ATC situations close to the right-hand side of the bell-shape of the reconstruction error distributions.

1) Arrival sequencing: On Figure 12, AFR27GH has a higher reconstruction error, probably because its trajectory is pushed quite to the East of the NARAK beacon, a less usual pattern for trajectories on the LFPO-LFBO route. As we investigate closer into the situation, it appears that AFR27GH was flying behind EZY81GE (from Lille, to the North) before being instructed to turn left. As EZY743L arrived from Lyon (to the North-East), AFR27GH was sequenced behind with an appropriate ATC order. RYR3YM arrives next and is sequenced behind AFR27GH in a similar manner.

2) Deconfliction: On Figure 12, DLH4J flies into Frankfurt area. The density of traffic converging on this IAF at this time of the day would probably not explain this shift in trajectory. The explanation could come from TAP571 which took off a bit earlier from runway 18. Without a turn left order to DLH4J, her climb path could have crossed DLH4J's trajectory, probably causing a loss of separation. Similarly, Figure 14 plots a deconfliction between AFR61UJ landing in Toulouse and ISS3932 bound for New York. The ATC order enabled AFR61UJ to descend through the level of ISS3932 without conflict.

3) Delaying: On Figure 15 both KLM13T and IBK3ME come to Madrid, resp. from Amsterdam and Oslo. In this situation, KLM13T comes before and flies two levels below IBK3ME. About 15 minutes before entering the TMA, KLM13T is delayed by a maneuver to the left so that IBK3ME can overtake her.

VI. CONCLUSIONS

This paper presented an anomaly detection technique well suited for identifying uncommon situations and ATC orders in Mode S data. We believe that this method is well suited to build a dataset of real situations with suggestions of plausible resolution actions. Studying uncommon or late ATC actions could also be a way to estimate ATC workload or to refine safety models and probability estimations associated with incident-prone situations. Future works could consider more evolved structures of networks, then include flow identification and outlier detection techniques inside a control sector so as to prescind from the limitation of the city pair paradigm.

Figure 1 .

 1 Figure 1. Autoencoder neural network architecture with one layer

Figure 2 .

 2 Figure 2. Data preparation: each trajectory is filtered, cropped, resampled; each feature is normalized before feeding the neural network. bottom-left coordinates top-right coordinates

Figure 3 .

 3 Figure 3. Evolution of the loss during the training process; example of the Toulouse scenario.

Figure 4 .

 4 Figure 4. Distribution of reconstruction errors on the Toulouse scenario. The bottom distribution replicates the top distribution on a semi-log y-axis.

Figure 5 .

 5 Figure 5. Distribution of reconstruction errors on the Frankfurt scenario. The bottom distribution replicates the top distribution on a semi-log y-axis.

Figure 6 .

 6 Figure 6. Distribution of reconstruction errors on the Madrid scenario. The bottom distribution replicates the top distribution on a semi-log y-axis.

Figure 7 .

 7 Figure 7. Situations with high reconstruction errors on the Toulouse scenario

Figure 8 .

 8 Figure 8. Archives of local news: traffic was interrupted in Toulouse airport around 18:30 local time. No aircraft were diverted during the interruption.

Figure 9 .

 9 Figure 9. Weather impact on STAR procedures on the Toulouse scenario. The orange background shows the location of cumulonimbus at 21:09 UTC.

  plots the last filed flight plan (Filed Tactical Flight Model, FTFM) and the Current Tactical Flight Model (CTFM), refined version of the FTFM based on live positions for DLH2F. Such a pattern suggests that DLH2F adapted its route to avoid further regulations in the area.

Figure 10 .Figure 11 .

 1011 Figure 10. Solutions with high reconstruction errors on the Frankfurt scenario

Figure 12 .

 12 Figure 12. Aircraft are sequenced for landing before entering Toulouse TMA; AFR27GH is vectored behind EZY81GE and EZY743L.

Figure 13 .

 13 Figure 13. Deconfliction between DLH4J scheduled for landing and TAP571 taking off from Frankfurt airport, runway 18.

Figure 14 .

 14 Figure 14. Deconflicting turn right actions on AFR61UJ, landing in Toulouse, and ISS3932, en route on FL300.

Figure 15 .

 15 Figure 15. In the Madrid scenario, IBK3ME flies above and behind KLM13T but enters Madrid TMA before her.

ACKNOWLEDGMENTS

The authors would like to thank Simon Proud from Oxford University for his help with the satellite image on Fig. 9 and Enrico Spinielli from Eurocontrol for his help in analysing recorded flight plans and regulations related to Fig. 11.


text:
B Le Saux

A Beaupère

A Boulch

J Brossard

A Manier

G Villemin

RAILWAY DETECTION: FROM FILTERING TO SEGMENTATION NETWORKS

Keywords: Rail detection, Railway detection, Filtering, Neural Networks

This paper deals with classification of remote sensing data to extract objects for industrial mapping. While land-cover or urban mapping have been extensively studied, industrial cartography remains a field yet to explore, in spite of tremendous needs. We present and compare here four approaches for railway detection in very high resolution images. They use various kind of filtering approaches, including the trained filters of fully convolutional networks. Moreover, they benefit from different a-priori and post-processing techniques to make them more robust. We evaluate all approaches on a challenging dataset captured on an operating station site with complex objects.

INTRODUCTION

With the development of new, practical means such as Unmanned Aerial Vehicles (UAVs), monitoring of industrial sites by remote sensing is now a reality. Remote sensing has been adopted in various industries such as oil and gas, mining or transports. In this latter field, it may answer to a crucial need of railway operators: ensuring the optimal safety of the passengers. Indeed, the knowledge of the current operational state of the infrastructure is essential to network administrators [START_REF] Vassilios | Railway management and engineering[END_REF].

The recent use of drones in this sector allows a significant increase in information gathering efficiency, notably by providing in good time extremely high resolution aerial images of railway furnitures. Currently, these images are used as a visual support for manually marking relevant elements of the infrastructure (rails, speed beacons, overhead line poles and catenaries, etc.), aggregated as a normalized blueprint of the railroad studied division. Automating image recognition procedures for cartography stands as a crucial optimization in this sphere.

Most railway detection works benefit from the newly available sensing capacities. The measurement precision is crucial, and first works relied on airborne LiDAR data to extract railroad objects [START_REF] Neubert | Extraction of railroad objects from very high resolution helicopter-borne lidar and ortho-image data[END_REF]. At that time, it was considered that the enormous amount of image data and radiometric variations imped the use of imagery. More recently, approaches to process very high resolution imagery were proposed: in [START_REF] Espino | Turnout detection and classification using a modified hog and template matching[END_REF], detection is performed through histogram of oriented gradients and support-vector machine learning, mostly applied to bolique images.

This paper is organized as follows. In Section 2 we present how data are acquired and pre-processed. We describe four approaches for rail detection in Section 3. We assess and compare these approaches on a challenging dataset which corresponds to an operational use case in Section 4. We present our concluding remarks in Section 5.

DATA ACQUISITION

In the following, we present the procedures for data acquisition and geometric processing which allow to build orthorectified images of a railway station. They were established by SNCF, the French railway operator, and are deployed in railway cartography projects, such as the one which lead to the Rennes dataset used in the experiments of Section 4.

Data acquisition. The raw images are collected with a Sony Alpha NEX-7 camera, embedded on a light multi-rotor drone performing at an average altitude of 50 m above ground level. The internal pixel size of the camera is 3.9 µm and renders 6000 × 4000 RGB, 8-bit-depth-per-band images, with a 4:2:0 chrominance compression format. Each frame is tagged with Global Positioning System (GPS) and inertial measurements of the drone at the instant of shooting. Ground Control Points (GCPs) are physically set on the studied scene and their geolocation is manually obtained by means of a high precision GPS pole. GCPs are used to improve the precision and accuracy of the produced data.

Data processing. A 3-dimensional scene reconstruction based on the GCPs and points automatically matched by a SIFT algorithm is computed using the inertial and GPS values with the Pix4D software 1 . Then, the ground surface is modeled using multi-scale geometric and color features [START_REF] Becker | Classification of aerial photogrammetric 3d point clouds[END_REF]. The final ortho-rectified image is obtained by projecting retroactively segmented ground pixels from the original frames onto the terrain model, allowing a vertical rendering at any point. This process drastically reduces perspective errors and rail obstruction hazards. The parameters used during acquisition and processing ensure a 0.011 m/pixel final resolution.

Cartography and ground truth. The ortho-rectified image is used by image analysts to manually draw maps of the railway station, which include platforms, signals, buildings and above all, rails. These vector maps follow the SNCFdeveloped EPURE norm, which implies that rails are represented as segmented lines anchored to points located on the center of the rail width every 10 m. This standard does not take rail curvature into account to avoid interpolation in railway track studies and analysis. Consequently, the real ground-truth is defined as the geo-located anchor points, an not as the whole map.

RAIL DETECTION ALGORITHMS

In the following we present the four approaches for rail detection which we will compare in Section 4.

User aided correlation matching. The first approach is a looped cross correlation matching process. The process is the following. A user provides an oriented rectangular sample of a rail in the image. The direction parameter serves in establishing an oriented triangular search area with a top apex angle of 2 × 11 • , consistent with rail curvature. Several matches are found in the area but only the nearest to the original is retained if it gets a correlation score over 0.4. The new found oriented rail segment becomes the next matching pattern and the process is repeated until no further match is found. Using the previous user input, the algorithm restarts from the beginning using the opposite direction. This approach allows to drastically reduce the time for manual extraction of rails in aerial images, but nevertheless requires user inputs for each railway.

Line detection with conditional morphological and orientation filters. In the second approach we use various morphological and orientation filters to extract rails (cf. Fig. 1). To reduce false detections, we first eliminate shadows from the images using Normalized Difference Index as described by Singh et al. [START_REF] Singh | Shadow detection and removal from remote sensing images using ndi and morphological operators[END_REF]. Knowledge of pixel size allows us to apply a differential Top-Hat transformation [START_REF] Allan | Morphological operators on the unit circle[END_REF] using a disc as a structuring element with a diameter corresponding to the rail width, increasing the relevance of the filter. Lines are detected on the resulting image using the curved-line extraction algorithm described in [START_REF] Steger | Extraction of curved lines from images[END_REF].

Two kinds of constraints are introduced to improve the results through post-processing. First we use the French railway Geographic Referential of Infrastructure (GRI) to get a global orientation of the tracks. We compare it to the resulting lines and thus eliminate mis-oriented results. A set of morphological operations (i.e. dilation, opening, union, closing) is then applied on the candidate regions to concatenate the results. Second, using the knowledge that each rail is located 1.5 m away from its sibling, a discriminant parallelism and distance analysis is finally applied. Line detection with Touzi filters. Touzi filters [START_REF] Touzi | A statistical and geometrical edge detector for sar images[END_REF] are commonly used for edge extraction in extremely noisy environments, originally introduced for speckle filtering in radar images. Given an orientation a priori (available from the GRI for example), the filter computes a radiometry ratio between rectangular areas on both sides of the chosen direction. By thresholding the resulting map, pixels which belong to strong, long-enough edges are extracted.

In a second time, a vectorization process is performed by applying a step-by-step search along the direction of the detected edges. It allows to get rid of mid-detections due to wires or pylons.

Semantic Segmentation Networks. In this last approach, we use a Convolutional Neural Network (CNN) designed for supervised, semantic segmentation. CNNs are feed-forward neural networks comprised of learnable convolution kernels. Those filters perform feature extraction in order to build an internal, abstract representation of the input, optimized for later classification. We use a standard fully convolutional network (FCN) with an encoder-decoder structure: SegNet [START_REF] Badrinarayanan | SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation[END_REF]. FCNs usually get RGB images as inputs and infer structured dense predictions by assigning a semantic class to every pixel of the image. We follow the approach of Audebert et al. [START_REF] Audebert | Semantic Segmentation of Earth Observation Data Using Multimodal and Multi-scale Deep Networks[END_REF][START_REF] Audebert | Segment-before-Detect: Vehicle Detection and Classification through Semantic Segmentation of Aerial Images[END_REF] defined for aerial image segmentation.

CNNs are supervised algorithms which require training data. Manually designed maps described in Sec. 2 are used to generate training rasters at the image resolution. They comprise a range of objects which can be extracted from the vector maps such as rails, buildings, electrical equipment, etc. However they are often inconsistent: more than 90% of the image points remain unlabeled, some classes (e.g. buildings) are partially annotated, and some classes only have a few samples while others are better represented.

The encoder part of the SegNet is initialized with pretrained VGG-16 weights, while the decoder part is randomly set. It is trained on 128 × 128 patches with stochastic gradient descent. Moreover, to deal with the class imbalance problem, partial loss of class c in the cross-entropy function is weighted by

RAILWAY DETECTION RESULTS

The Rennes dataset used in this study was acquired over the railway station of Rennes, France, in 2017, following the procedure described in Section 2. For processing purposes, it was segmented in 5000×5000 ortho-rectified images at resolution of 2cm per pixel. It comprises 34 images for training (i.e setting hyper-parameters of the algorithms or actually training the segmentation network) and 21 images for testing which were not seen before the evaluation (see Fig. 3-a for some examples).

The test area implies 1172 rail control points (RCPs) (cf. 2) which establish a consistent common testing set. The ground-truth for evaluation comprises those RCPs and a small neighborhood around them (12 pixels) for taking into account the width of rails, which lead to more than 14k pixels. This allows to measure the localization precision of the detection with respect to the rail: centered or on the edge. The various rail detection algorithms were evaluated according to the producer accuracy Acc. = T |P| where |P| is the number of rail pixels in the ground-truth, and T is the number of true positives i.e the number of points that received the "rail" label. These accuracy results are summarized in Table 1. Besides the overall results on the whole test set (denoted by all test) which also contains neighboring urban areas with buildings and roads, we also focus on a smaller set of 3 images with mostly railways (hereafter the rail subset) which corresponds to roughly 200 RCPs and 2.4k pixels. This allows to compare the rail extraction methods when no other salient, distracting features are present in the images, as would be the case for example if a mask obtained from open-source maps was applied as a preprocessing step.

In Fig. 3 we show rail detection results obtained by all methods along the corresponding tile. First, it appears that all approaches are quite successful at detecting rails: all retrieve the railways in the upper part of the images while having more-or-less false positives in the neighbor station or city buildings. At second sight, differences appear. As predictable, the user-initialized correlation matching (Fig. 3-b) yields no false alarms, and most rails are found by the user. Mainly, errors occur when the step-by-step tracking of the rail fails. Conditional morphological and orientation filters (Fig. 3-c) obtain good railway outlines (i.e. pair of rails) thanks to the strong a priori about orientation or parallelism that is introduced in the approach. A few false alarms occur in city environments, and the false negatives are mostly due to the parallelism a priori: some rail may be discarded if no companion rail appears due to image default or vegetation.

Touzi filters (Fig. 3-d) use only orientation a priori and follow a step-by-step process for vectorization of rail detection.

The get extremely good results, and do not miss much rails. However, this also the approach which leads to the most numerous false alarms, typically on buildings and road edges which have the same orientation as the a priori. Finally, the fully-convolutional segmentation network (Fig. 3-e) is the most brute-force approach. Indeed, no a priori is given apart from the training data, and no post-processing is performed to obtain continuous rail lines (vectorization process). However it performs remarkably well: most railways are retrieved as seen in the detection maps, and false alarms are structurally different from those of conditional or Touzi filters (no line effect) so they are easier to spot. The main problem is the dashed line effect which occurs if looking closer and appears when a rail is rusty (e.g. for secondary tracks) or occluded by wires or vegetations. Overall, it appears that adaptability (to adaptation or local appearance) is a feature to care about when designing a railway detection algorithm. Table 1 summarizes the accuracy measures. Considering all 21 test images with city surroundings, the filtering approaches obtain good results with accuracies around 75% or 80%. However conditional or Touzi filters do not require user supervision at test which is an advantage for large scale deployment. The segmentation network gets mixed results since only half of the control points are detected due to the lack of constrained continuity in the detection. If we focus on the rail subset however, Touzi filtering and the segmentation network reach around 90% accuracy. This hints that a priori constraints as simple as an infrastructure ground coverage mask are a promising way to improve the performances.

CONCLUSION

In this paper we have presented 4 filter-based approaches for rail detection. They differ by the nature of the filters, which can even be learned from a dataset in the case of a fully- convolutional neural network, and by the amount of a priori which is introduced, including user supervision, geographical and thematic constraints, or no a priori. All approaches lead to more-or-less satisfying results depending on the use case and the available resources. If aiming at the most automated approach, user supervision is not required since 3 other methods got equivalent or better results, and orientation and parallelism are not necessary, since they can be learned from the data. However the infrastructure coverage and vectorization by means of a step-by-step line building procedure help predict better detection maps. Thus, by taking the best of each approach, this benchmark shows it is possible to design an automated, even more efficient process for rail detection. A promising path consists in mixing simple a priori constraints into the fully-convolutional segmentation networks.

Fig. 1 .

 1 Fig. 1. Workflow of the line detector based on conditional morphological and orientation filters.

  med 1≤i≤c (freq i freq cfor all c.

Fig. 2 .

 2 Fig. 2. Fully-convolutional network for railway detection and training data. At test phase the encode-decoder network gets unknown images and predict a segmentation map.

Fig. 3 .

 3 Fig. 3. Railway detection results (in blue) with methods of Sec. 3: (a) Original image; (b) User correlation matching; (c) conditional filter; (d) Touzi filter; (e) Segmentation network.

Table 1 .

 1 Railway detection accuracies for methods of Sec. 3, computed on the whole test set (21 images) and on a subset with mostly railways (3 images).

	Method	All test	Rail subset
		(14k samples) (2.4k samples)
	User corr. matching	79.67	76.87%
	Conditional filter	75.40%	71.62%
	Touzi filter	76.66%	92.74%
	Segmentation network	50.38%	89.00%