Vinicius Lucas Leandro Nesi

Garcia Pinto

Marcelo Cogo Miletto

Lucas Mello Schnorr

Garcia Vinicius

Pinto

Lucas Mello

Schnorr Ufrgs

StarVZ: Performance Analysis of Task-Based Parallel Applications

Keywords: performance analysis, high-performance computing, trace visualization, task-based applications, runtime systems, heterogeneous CPU/GPU computing, tidyverse, ggplot2, R

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

High-performance computing (HPC) is becoming more heterogeneous with a combination of traditional multi-core processors with multiple general-purpose graphical processing units (GPGPU) in computing nodes. Such combinations represent 70% in the first ten systems listed in the TOP500 list [START_REF] Strohmaier | Development over Time | TOP500 Supercomputer Sites[END_REF]. Parallel applications running on these computer systems are increasingly complex, with an irregular workload on top of different architectures with varying performance capabilities.

The performance analysis of parallel applications frequently uses timestamped events in execution traces to identify bottlenecks, load imbalance, and communication problems. While some tools like Paje (de [START_REF] De Kergommeaux | Pajé, an Interactive Visualization Tool for Tuning Multi-Threaded Parallel Applications[END_REF], Paraver [START_REF] Pillet | Paraver: A Tool to Visualize and Analyze Parallel Code[END_REF] and Vite [START_REF] Coulomb | An Open-Source Tool-Chain for Performance Analysis[END_REF] have extensibility features, others, like Vampir [START_REF] Knüpfer | The Vampir Performance Analysis Tool-Set[END_REF] and Projections [START_REF] Wilson | Gantt Charts: A Centenary Appreciation[END_REF], target specific programming environments. Most of them fail to be adaptable because they rely on compiled languages, demanding users to consider scripting languages to test performance analysis hypothesis.

Task-based parallel applications consist of a directed acyclic graph (DAG) of tasks with edges representing task dependencies. A runtime system is responsible for scheduling the DAG to the computing resources, respecting such dependencies. Consequently, the application performance depends on the parallelism of the DAG and the scheduling heuristic of the runtime. Traditional tools for such parallel applications are generally unfit because they lack support for a DAG-oriented analysis. The existing solutions, such as DAGViz [START_REF] Huynh | DAGViz: A DAG Visualization Tool for Analyzing Task-Parallel Program Traces[END_REF], Grain Graphs [START_REF] Muddukrishna | Grain Graphs: OpenMP Performance Analysis Made Easy[END_REF], Temanejo [START_REF] Brinkmann | Task Debugging with Temanejo[END_REF], TaskInsight [START_REF] Ceballos | Analyzing Performance Variation of Task Schedulers with TaskInsight[END_REF], remains monolithic, built on top of compiled languages.

Combining factors above -solely the availability of monolithic tools and the lack of scriptbased performance analysis for parallel applications -motivate the necessity of a modern workflow based on established data science tools. The requirements for such workflow include the possibility of analyzing large execution traces gathered in different levels of the computing system (application, runtime, operating system), and the freedom to extend the tools according to each parallel application's specific necessities. This paper presents the StarVZ package, a framework, and toolkit, implemented in the R programming language, tailored for the performance analysis of task-based parallel applications. Since the package relies on established tools from the tidyverse meta-package, all internal data and views can be exported with the patchwork package [START_REF] Pedersen | patchwork: The Composer of Plots. R package version 1.0[END_REF] to fit the users' needs. As far as we know, StarVZ is the first R package dedicated to the performance analysis of task-based parallel applications using execution traces. More than that, it is probably the first framework that employs the flexibility of scripting languages with the performance delivered by the dplyr package to deliver synchronized panels built with the ggplot2 package. Our experience with multiple case studies from parallel task-based linear algebra solver and other types of applications serves as validation of our strategy.

Section 2 presents the background concepts about HPC and task-based programming. Section 3 presents the methodology and philosophy of the StarVZ package, while Section 4 details the available visualization panels. Section 5 presents case studies to illustrate how the package can be used to conduct performance analysis.

Background concepts

This section presents background concepts on high-performance computing and task-based parallel programming. These concepts are useful to understand the applicability of the StarVZ package, its requirements, and design choices.

High-performance computing landscape

Parallelism is a long-established strategy to increase the processing power of computing systems. It has been part of the hardware-design of general-purpose CPU architectures through programmer-transparent techniques as the instruction level parallelism (ILP) [START_REF] Hennessy | Computer Architecture: A Quantitative Approach[END_REF]. Higher-levels of parallelism is present in the development of scientific applications as weather forecasting and computational fluid dynamics (CFD). Their characteristics favored the use of what is called high-performance computing. The HPC community has developed programming models as the message passing interface (MPI) [START_REF] Gropp | Using MPI: Portable Parallel Programming with the Message-Passing Interface[END_REF] to execute computations on physically distributed computers and the OpenMP specification [START_REF] Chapman | Using OpenMP: Portable Shared Memory Parallel Programming[END_REF] to explore different processors in the same computing node.

The HPC landscape has changed in the last decade. While CPUs with an increasing number of multiple processing cores have become ubiquitous, power constraints have motivated the use of hardware accelerators as Graphical Processing Units (GPU). A GPU is a specialized processing device with its computing cores and memory acting as a hardware accelerator. Current GPUs can process much more than graphical workloads and can carry out generalpurpose parallel processing as linear algebra operations. An analysis of the TOP500 list [START_REF] Strohmaier | Development over Time | TOP500 Supercomputer Sites[END_REF], which ranks the powerful supercomputers globally, shows the growing presence of hybrid platforms combining accelerators with multicore processors.

Widely-spread parallel programming tools target only one parallelism layer at once. Despite many research efforts, the HPC software landscape remains an assortment of independent tools. The MPI specification is a de facto standard when handling parallelism in distributed platforms, i.e., a cluster of independent computers cooperating to solve a problem by communicating through a local network. MPI includes operations to exchange point-to-point or collective messages among processes. The specification supports C and Fortran, but there are binds for other programming environments as R [START_REF] Yu | Rmpi: Parallel Statistical Computing in R[END_REF]. OpenMP is the reference model when exploring parallelism among processor cores sharing a single memory space. As MPI, OpenMP focuses on C/C++ and Fortran. CUDA (NVIDIA 2020) is the most used tool to program general-purpose applications that explore GPUs while OpenCL (OpenCL Working Group 2020) offers a vendor-independent approach. Both CUDA and OpenCL focus on C, but only OpenCL offers a bind to R (Urbanek and Puchert 2020). HPC software stack frequently comprises an ad hoc combination of independent programming tools with hand-made optimizations and weak performance portability. Since the hardware platforms continue to become more heterogeneous, there is room for better software abstractions. The task-based model, discussed in the next section, is an example of programming abstraction for heterogeneous platforms.

Task-based programming

When implementing an algorithm in the task-based programming model, the programmer breaks down the program in small portions of work called tasks, potentially executed in parallel while respecting dependencies among them. A runtime system handles load balancing and scheduling at execution time, effectively decoupling application code from the target hardware. Cilk [START_REF] Blumofe | Cilk: An Efficient Multithreaded Runtime System[END_REF] was one of the first tools to implement this concept by adding two keywords (spawn to and sync) to a standard C code. A spawn allows a function to be executed in parallel, while sync acts as a local barrier to wait for the end of local spawned functions. The OpenMP standard offers the syntactically similar equivalents task/taskwait. Despite efficient use for recursive algorithms, this approach implies artificial synchronizations in other widely-used algorithm structures as loop-based ones. As a result, the task-based programming model's current implementations offer data-flow dependencies where each task includes the access mode of its parameters [START_REF] Augonnet | StarPU: a Unified Platform for Task Scheduling on Heterogeneous Multicore Architectures[END_REF][START_REF] Duran | OmpSs: a Proposal for Programming Heterogeneous Multi-Core Architectures[END_REF]OpenMP Architecture Review Board 2018). At runtime, based on sequential consistency, the runtime system detects actual dependencies, thus enabling fine-grained synchronizations.

Hybrid resources, composed of CPU and GPU cores, are captured by the task-model by providing multiple implementations per task. This feature is available on task-based runtime systems such as StarPU [START_REF] Augonnet | StarPU: a Unified Platform for Task Scheduling on Heterogeneous Multicore Architectures[END_REF], XKaapi [START_REF] Gautier | XKaapi: A Runtime System for Data-Flow Task Programming on Heterogeneous Architectures[END_REF] and OmpSs [START_REF] Duran | OmpSs: a Proposal for Programming Heterogeneous Multi-Core Architectures[END_REF]. The programmer provides additional implementations to the same task to target each hardware device (e.g., CPU core, GPU card, vectorial units, manycore accelerators). When scheduling the tasks on computing resources, the runtime system dynamically decides which implementation will run based on the implementations available and the platform characteristics. Accelerator devices as GPUs have their own memory address space, which is inaccessible from the host system processor. Since task-based applications are decoupled from the hardware, modern runtime systems as StarPU automatically handle these separated address spaces using the MSI cache-coherence protocol [START_REF] Hennessy | Computer Architecture: A Quantitative Approach[END_REF]. Some runtimes also implement several optimizations to reduce computing resources idleness, overlapping data transfers with computations and considering data locality. Figure 1 shows an example of the tiled-LU factorization algorithm in the task-based model. Each call to sgetrf_nopiv(), strsm(), and sgemm() functions represents the creation of a new task. Dependencies among them are inferred from the runtime system using each parameter's provided access mode (e.g., R, RW). The DAG on the right shows the dependency chain when N = 4. The number of each circle identifies the outer-loop iteration of the task creation. A task may have many implementations that are dynamically chosen by the runtime according to data locality, resources availability, scheduling policies, and load balancing.

The task-based programming model remains an excellent alternative to tackle heterogeneous computing nodes since it simplifies the programming effort by relying on established scheduling heuristics. Nevertheless, the performance analysis of such applications may provide hints to runtime and application developers on possible optimizations. The next section presents the StarVZ package, which provides a scriptable framework to create visualizations to guide the analyst intuition on performance problems.

StarVZ methodology and philosophy

StarVZ aims to provide comprehensive performance analysis for task-based applications supported by runtime systems on hybrid CPU-GPU platforms. Unlike other analysis tools, our tool remains open to alternative methodologies besides graphical representations of application traces gathered by the runtime. To the best of our knowledge, StarVZ is the first HPC performance analysis tool conceived on top of modern data science tools. It relies on the expressiveness of the R language (R Core Team 2017) and its rich set of statistical and data manipulation packages [START_REF] Wickham | Welcome to the tidyverse[END_REF][START_REF] Wickham | ggplot2: Elegant Graphics for Data Analysis[END_REF][START_REF] Wickham | dplyr: A Grammar of Data Manipulation[END_REF][START_REF] François | arrow: Integration to 'Apache[END_REF][START_REF] Berkelaar | lpSolve: Interface to 'lp_solve' v. 5.5 to Solve Linear/Integer Programs[END_REF][START_REF] Glur | data.tree: General Purpose Hierarchical Data Structure[END_REF]. Thanks to that, we can process raw input data, perform cleanups, filtering, merges, statistic computations, and correlations, producing much richer visualizations than those that could be made by directly plotting the raw traces. Earlier

for (k = 0; k < N; k++) { sgetrf_nopiv(RW, A(k, k)); for (m = k+1; m < N; m++) strsm(R, A(k, k), RW, A(m, k)); for (n = k+1; n < N; n++) { strsm(R, A(k, k), RW, A(k, n)); for (m = k+1; m < N; m++) sgemm(R, A(m, k), R, A(k, n), RW, A(m, n)); } }
Figure 1: Pseudo-code of the LU algorithm and its corresponding DAG (N = 4); each node is a task of a certain type (color) with its outler loop iteration identification (the k value). publications [START_REF] Pinto | Analyzing Dynamic Task-Based Applications on Hybrid Platforms: An Agile Scripting Approach[END_REF][START_REF] Pinto | A Visual Performance Analysis Framework for Task-Based Parallel Applications Running on Hybrid Clusters[END_REF][START_REF] Nesi | Visual Performance Analysis of Memory Behavior in a Task-Based Runtime on Hybrid Platforms[END_REF] present the initial steps and the first contributions related to the creation of the StarVZ R-package.

Workflow overview

Figure 2 presents a detailed representation of the workflow. StarVZ input consists of a set of text files in the comma-separated values (CSV) format. These files can be directly obtained from the runtime system raw traces using standard and open-source tools such as paje, sort, sed, and recutils. These pre-processing steps appear in the left part of the Figure 2 and are outside of the scope of the StarVZ R package (represented in the right side). In the first run with a given dataset, a call to starvz_phase1() will process these CSV files performing several filtering, joins, and statistical operations to build the package's internal tables. Such tables are dumped to compressed parquet binary files to save time at further runs. Once this initial processing is the more costly step of the package workflow, additional runs on the same dataset are faster, allowing quick exploration of complete visualizations. Since HPC experiments usually run on powerful dedicated platforms (e.g., clusters), this initial phase can be optionally executed as a post-processing step of the application's execution.

Using these internal tables' data, detailed in Section 3.3, StarVZ can generate several different panels visually representing the application's performance. Section 4 presents the most relevant panels. A user can customize the final visualization by setting properties on a configuration file at the YAML format. A default setup takes place when a configuration file is absent. Online fine-tuning is also possible by changing settings before a call to starvz_plot() or passing alternative arguments to the functions calls creating the visualization panels.

Basic usage with code snippets

The StarVZ package is available from CRAN at https://CRAN.R-project.org/package= starvz. The installation procedure is as simple as:

R> install.packages("starvz")

In R, a basic usage is as follows:

R> library("starvz") R> starvz_phase1(directory = "data/basic-usage", whichApplication = "lu", app_states_fun = lu_colors) R> jss_sample_lu <-starvz_read(directory = "data/basic-usage", selective = FALSE)

Once the data is at the jss_sample_lu variable, one can produce a basic plot (see Figure 3 for the output) with:

R> starvz_plot(jss_sample_lu)

Data

The origin of all input data is the runtime specific traces. In the case of StarPU, this is a FxT file. As described in Section 3.1, the first phase of StarVZ applies runtime-specific tools and functions to these traces translating them to CSV files. The scripts ultimately transform these text files into a list of tables that constitutes the primary StarVZ data. The structure of these CSV files reassembles the final data structure. However, many transformations and filtering are applied to reduce and prepare them for the analysis/visualization. These transformations are specific for each case and are unimportant for the end-user. We provide only a brief overview of the principal operations that we do. The common transformations that the first phase executes are popular data-science operations. For example, many character columns are converted to R factors to save space. Furthermore, the CSV might be split in more than one table because they refer to very distinct data categories with different columns. Also, we select and derive detailed information. For example, StarPU may inform one entity as NAME_NODE, a redundant as we may already have the node, and all components of that table have that NAME. The first phase computes relevant structures like the application DAG. We also shift the timestamp of all operations setting as zero the beginning of the first appropriate application task. Events that occur before this will have a negative timestamp. This time shift is relevant because we are usually interested in the application execution. We keep all negative timestamped information as we do not want to lose track of prior initialization and overhead. Also, this phase may apply application-specific functions during the CSV reading. For example, the QRMumps application uses a generic StarPU field to store application-related data properly converted by StarVZ. Furthermore, some aesthetics decisions remain fixed in these transformations, like colors of tasks, names of entities, and vertical ordering of elements. Moreover, most of first phase's operations, when possible, are isolated from each other. They load the CSV file, do the procedures, save the parquet files, and free all their data, keeping the memory footprint as low as possible.

The StarVZ primary data is an R 'S3' Class with a list of 'Tibbles'. Not all tables need to be present, and their structure may differ depending on the runtime and version. Each panel function requires different tables/columns for their views. That means that the structure is flexible and lazy and could contain a subset of the data to execute only desirable functions.

Only the views required data must be present or be loaded. Panel functions enforce the data requirements by the internal function starvz_check_data(). We provide a sample StarVZ data in the package stored in the variable starvz_sample_lu.

The tables' data can be accessed directly by accessing a named list element in R. For example, to access the table Application we could use:

R> starvz_data$Application

Although the amount of data registered in the data is vast, StarVZ also computes additional information dynamically. We decided to keep them absent from the registered parquet data because they are relatively inexpensive to compute or require user configuration. For example, StarVZ can classify outlier tasks using different performance models. It may also be necessary to apply aggregation functions per time step because of the excessive amount of data. Some panels may present metrics such as critical path bound (CPB), area bound estimation (ABE), and Pmtool's performance bounds (externally computed by the pmtool (Eyraud-Dubois 2019) application). The parameters that tailor this additional information might be passed by the user or defined in the configuration file. One example of such a parameter is the time-step interval of data aggregation.

Views

This Section exposes the main functionalities of the StarVZ package, documenting the functions that create performance visualization plots. All plotting functions produce ggplot objects and have their names prefixed with panel_. These functions always receive as first argument the data value (see Section 3.3 for details), which represents the contents of the execution trace after post-processing during the first phase.

Worker view

The panel_st_raw() function represents application and runtime states in a space/time organization. The panel has the workers listed on the Y-axis and their states in the X-axis, associating colors to state type. We use this function as follows:

R> panel_st_raw(data = jss_sample_lu, labels = "FIRST_LAST", ST.Outliers = TRUE, runtime = FALSE, idleness = TRUE, makespan = TRUE, abe = TRUE, taskdeps = TRUE, tasklist = c("1_6849"), levels = 7)

Figure 4 presents a view with many additional features to the classical space/time plot. As an example, we list some parameters that activate them. Option labels controls workers labels in the Y-axis. It accepts, for example, "ALL" to display all resources names, and "NODES_only" to show just the node ID. Parameter ST.Outliers highlights (with darker colors) those tasks classified as anomalous by the performance models. The option idleness plots per-resource idleness quantification in percentage. The option abe plots a vertical gray bar showing a makespan estimation computed with a linear program. Such estimation is also helpful to check load balancing among nodes. The options taskdeps, tasklist, and levels enable plotting of last-solved task dependencies (as red edges).

The performance of HPC task-based applications is heavily dependent on runtime system decisions which makes relevant to visualize its states when analyzing the overall performance.

The function panel_st_raw() with parameter runtime = TRUE generates a plot (see Figure 5) with the runtime-oriented space/time view:

R> panel_st_raw(jss_sample_lu, runtime = TRUE, labels = "FIRST_LAST")

Application view

Elimination tree

Function panel_atree() plots the structure of the elimination tree along time, defining the start and end of each tree node, connecting them with arrows to represent the child and parent relationship. The function parameters can control the representation of computation, communication, initialization, and anomalies in the tree. The Figure 6 shows the resulting panel when we call the function as follows:

R> sample_qr_trace <-starvz_read(directory = "data/qr-trace") R> panel_atree(data = sample_qr_trace, step = 100, computation = TRUE, pruned = TRUE, initialization = TRUE, anomalies = TRUE, communication = TRUE) The function parameters described here are all booleans that enable/disable the tree structure's visual appearance. The computation and pruned parameters control the resource utilization representation by factorization tasks for tree nodes and pruned tree nodes. The blue/red color gradient, positioned in the Y-axis according to the tree nodes' submission order, depicts each tree node's aggregated computational intensity. Rectangles with smaller height represent the pruned nodes, which are grouped and aggregated by a common parent.

The initialization controls the representation of node memory allocation tasks (green rectangles in Figure 6). The communication parameter is responsible for controlling communication tasks, presented as black rectangles in a node line. Lastly, the anomalies parameter indicates whether to plot the location of anomalous tasks in the tree structure, using a point with the same color of the tasks in panel_st_raw().

The functions panel_activenodes() and panel_nodememuse() represent the application memory behavior related to the tree structure. The first one shows the number of allocated nodes in memory at a given time, and the other presents the total memory usage in MegaBytes of these active nodes. Their parameters allow plotting the raw data values over time when aggregation is FALSE, or plot the aggregated values in time by a given step size. Usage and output (see Figure 7) of these functions are as follows:

R> library("patchwork") R> library("ggplot2") R> active <-panel_activenodes(data = sample_qr_trace, step = 100, aggregation = TRUE) + theme(axis.text.x = element_blank(), axis.title.x = element_blank()) R> memuse <-panel_nodememuse(data = sample_qr_trace, step = 100, aggregation = TRUE) R> active / memuse To represent the computational resource utilization related to the tree structure we can use the functions panel_utiltreenode() and panel_utiltreedepth(). They depict how many computational resources (Y-axis) the compute tasks are using inside a given time step (Xaxis). The difference between them is that the former paints the area representing the resource utilization by individual tree nodes, while the latter paints by the tree depth. Hence, these views allow us to observe how much tree and node parallelism the algorithm is exploring. Usage and output (see Figure 8) of these functions are as follows:

K-iteration

Many parallel applications use loop-based algorithms. The panel_kiteration() function enables one too check iterations overlap along time. This overlap is possible in a task-based model since it uses fine-grained synchronizations, i.e., there is no global barrier among loopiterations. The Y-axis depicts the iteration index while the X-axis is the time. Black curves represent iteration start (left) and iteration end (right). Each running task is plotted with an alpha value between these curves once many tasks can run simultaneously on different resources. Blank spaces mean no task of this iteration is running. The shape of this plot changes according to the DAG traversal implemented by the scheduling policy. A wider shape represents more parallelism, giving flexibility to the runtime system. Figure 9 depicts an example obtained with the following call:

R> panel_kiteration(jss_sample_lu)

Runtime memory management

Memory management is a crucial aspect of HPC applications' performance. Some runtimes, including StarPU, automatically controls the application memory using the DAG structure. In the case of StarPU, the minimal memory unit (used by tasks) is the data handle, and each resource memory (RAM, GPUs) has an entity called memory manager to control all the related operations (transfers, allocations, and frees). StarPU uses the MSI protocol to guarantee the coherence of data handles across all managers. Each data unit has one of the possible three states (own/share/invalid) on each manager. StarVZ provides two timeoriented visualizations for enabling inspection of the automatic memory control of StarPU. These views are a result of an earlier publication [START_REF] Nesi | Visual Performance Analysis of Memory Behavior in a Task-Based Runtime on Hybrid Platforms[END_REF].

The first view, panel_memory_state(), presents a StarPU's memory manager oriented view using Gantt Charts. Figure 10 depicts the resulting panel, where the Y-axis lists the memory managers, and the X-axis is time. The states are issued operations like allocation of data, transfers, and free. The combined option enables the visualization of transfers between entities using arrows where the horizontal arrow length corresponds to its duration. The following command gives an example of such a view:

R> panel_memory_state(data = jss_sample_lu, show_state_total = FALSE, show_transfer_total = FALSE, combined = TRUE, x_end = 200)
The second view, obtained with the panel_handles() function, provides a StarPU's data handle-oriented visualization. It may be valuable to visualize the progress of a memory unit across all the execution. The visual structure of this panel is similar to the previous one (see Figure 10). The Y-axis has the list of memory managers, and the X-axis is time. However, space/time states represent the data handle's MSI situation on that entity. Additionally, this panel presents all the handle events, including allocation/transfer requests and events such as fetches or pre-fetches. The view also shows an inner state inside the MSI to show current tasks accessing the data handle. The colors of the task states are equal across all StarVZ plots. These states can present two borders; solid lines indicate that the task had write permissions, while dashed ones the task only had read permissions. Again, this view uses arrows to show communications between memory managers. Because many blocks exist, a simultaneous visualization in the same image using this panel is not scalable and not advisable. Instead, the panel function provides the lHandle and JobId arguments to select a list of handles or handles of a specific job, respectively. The handles are identified per node using its address and across multiple nodes using a Tag. This crude information seems not human friendly, so the user may replace it using custom functions (using the argument name_func) to name them while StarVZ matches the handles across all nodes. If the application provides the coordinates of data handles to StarPU, StarVZ will use them, as default, to create names of the form: "Memory Block X Y" for 2D-cases. A code example of such panel utilization (see Figure 11 for the output) selecting the data handles of task 1_300 is given by: R> panel_handles(data = jss_sample_lu, JobId = "1_300", x_start = 15000, x_end = 16500)

Variables

All the Variable panels share the same principle: to filter the StarVZ data for timestamped numerical metrics, apply temporal aggregation to smooth and reduce the number of data points, and depict them as a line chart where the Y-axis is the variable, the X-axis is the time, and each line color distinguishes the correlated entity (for instance, the scheduler of different compute nodes). Besides aesthetics arguments, variable panels require the step argument that indicates the temporal aggregation's granularity. An example of a variable panel is the panel_ready() that shows the number of ready tasks per node scheduler, as shown in Figure 12. We can obtain it using a step of 50ms with the following code:

R> panel_ready(data = jss_sample_lu, step = 50)

Aggregated views

Sometimes the number of events recorded in the traces is too high to be graphically represented in the plot, leading to known rendering issues [START_REF] Schnorr | Visualizing More Performance Data Than What Fits on Your Screen[END_REF]. Too much detail can also hide relevant behavior, reducing the plot readability and usefulness. StarVZ handles that with explicit aggregation controlled by the user. The function panel_st_agg_static() slices the time in fixed intervals and computes how much each task type appears within the interval. Optionally, outliers can be print on top of aggregated areas. The function panel_st_agg_dynamic() offers a more dynamic approach, aggregating tasks until it reaches an exclusion criterion (e.g., a specific task type, outlier, or minimum task duration) which prevents relevant states from being eclipsed by aggregation. Figures 13 and 14 illustrate the static and dynamic temporal aggregation as created by the following code snippets:

R> panel_st_agg_static(jss_sample_lu, outliers = FALSE, step = 500)

Figure 13: The space/time plot with static temporal aggregation.

R> panel_st_agg_dynamic(jss_sample_lu)

Figure 14: The space/time plot with dynamic temporal aggregation and outliers identification.

The function panel_st_agg_node() offers a summarized per-node view that aggregate tasks running on resources of the same type. Such a panel enables a much more scalable visualization of large platforms with high CPU/GPU-count. The code to obtain Figure 15 is as follows:

R> panel_st_agg_node(jss_sample_lu) The function panel_lackready() presents another use of the temporal/spatial aggregation feature, as depicted in Figure 16. It indicates moments when the amount of ready tasks is not enough to fill all compute resources, i.e., so we expect idleness to appear.

R> panel_lackready(jss_sample_lu, x_start = 0)

Figure 16: The plot to pinpoint when the number of ready tasks is smaller than the number of computing resources, indicating insufficient parallelism from the application.

Non-temporal views and metrics

This Section presents some auxiliary views and metrics' functions with an non time-oriented output. For many applications, the data has a spatial meaning. For example, linear algebra solvers represent a matrix into 2D organized tiles. Each tile has a coordinate correlated to the application algorithm. Sometimes, it is useful to view the memory using this application information correlating it with its algorithm. StarVZ has three visualizations that follow these characteristics for matrix-oriented applications. These views present the data in a 2D grid using the blocks coordinates. (i) panel_memory_snap() presents the current state of the memories on a specific moment defined by the argument selected_time, each major tile is the MSI state of that block, and the inner minor tile is tasks that are using it at that moment. The shape of the inner tile is the type of access (write or read). This view also has a step argument to show all tasks that started after selected_time -step and finished before selected_time to give a progression perspective. The ABE provides an estimation to the makespan (see Section 4.1) computed with a linear program. While the estimation helps assess whether there is room for improvements, the linear program's solution can provide hints to achieve this goal. The panel_abe_solution() function plots bullets with the ideal partition of each task per resource type and per node; and bars representing the observed partition in the real execution. Figure 18 depicts such a view obtained with the code:

R> panel_abe_solution(data = jss_sample_lu)

As we use a regression model to classify tasks with anomalous duration in the cases where they have irregular costs, like in sparse matrix factorization, it is essential to have a diagnostic plot to observe if the model fits well in the data. The panel_model_gflops() represents the relationship between the theoretical GFlops and the duration of computing tasks. Each panel depicts such a relationship for each task and resource type. Figure 19 depicts the view obtained with the code:

R> qr_model_data <-starvz_read(directory = "data/qr-model")

R> panel_model_gflops(data = qr_model_data, freeScales = FALSE)

Extensibility

Although StarVZ provides the function starvz_plot() that generates and assembles all active plots in the configuration, it may be of user interest to reassemble plots using a different layout or even combine multiple StarVZ plots. StarVZ provides two functions (used internally in starvz_plot()) for manual assembly purposes. The function starvz_plot_list() returns a list of ggplots of all enabled panels. This function is useful if the user wants to customize the final plot layout (using, for example, patchwork with the individual ggplot2 elements), or individual ggplot characteristics (like colors, spacing, and guides). For example, if the user wants to show the state view and summary_nodes view horizontally, as depicted in Figure 20: R> library("ggplot2") R> jss_sample_lu_2 <-starvz_read(directory = "data/compare-usage", config_file = "data/compare-usage/config.yaml", selective = FALSE) R> jss_sample_lu_2$config$limits$end <-18900 R> jss_sample_lu$config <-jss_sample_lu_2$config R> jss_sample_lu$config$title$text <-"LWS" R> trace1_plots <-starvz_plot_list(jss_sample_lu) R> trace2_plots <-starvz_plot_list(jss_sample_lu_2) R> starvz_assemble(trace1_plots, trace2_plots, config = jss_sample_lu_2$config)

Case studies

We present here case studies that illustrate the use of StarVZ in the performance analysis process. These case studies come from common linear algebra operations: the dense tiled LU factorization of Chameleon [START_REF] Agullo | Faster, Cheaper, Better -a Hybridization Methodology to Develop Linear Algebra Software for GPUs[END_REF], and the sparse QR factorization available in QRMumps [START_REF] Agullo | Multifrontal QR Factorization for Multicore Architectures over Runtime Systems[END_REF]. We also present a proof-of-concept to show task-based OpenMP applications analyzed with our framework.

LU

Most of the prior visualizations used an execution of the LU factorization provided by Chameleon [START_REF] Agullo | Faster, Cheaper, Better -a Hybridization Methodology to Develop Linear Algebra Software for GPUs[END_REF]. Figure 21 The main difference is the total makespan from 18822 to 9046. The plots have the same horizontal time limit (manually configured), according to the worst execution. The explanation of these time differences is on the StarVZ plots. The idle time values per worker show that the GPUs with the dmdas scheduler have lower idle times than the lws. Because the GPUs are more computationally powerful, such utilization difference reflects on the final makespan a lot. Moreover, checking the runtime states, the number of idle states on the first half of the execution using the dmdas is lower than the lws, that has idle states on GPUs during all the execution. The final explanation relies on scheduler algorithms. The dmdas scheduler derives from the heft algorithm [START_REF] Topcuoglu | Performance-effective and low-complexity task scheduling for heterogeneous computing[END_REF], it will greedily place a task on the worker with the lower time ending estimation considering data transfers. This behavior enables the dmdas algorithm to know when a worker will need some data and make more prefetches. On the other hand, the lws algorithm makes workers steal tasks from others when they do not have tasks to execute. Because they can steal a task that they do not have all the data, it will have to do a fetch, leading to lack of the positive effects of communication and computation overlap. Using the StarVZ workflow, we can also verify the raw values for fetches and prefetches for both cases using the following code. The jss_sample_lu is the lws case while the jss_sample_lu_2 is the dmdas case. The dmdas case makes much more prefetches (4122) than fetches (677) while lws does fetches (4982) most of the time.

R> library("tidyverse") R> jss_sample_lu$Link %>% filter(Type == "Intra-node data Fetch" | Type == "Intra-node data TaskPreFetch" | Type == "Intra-node data PreFetch ") %>% group_by(Type) %>% summarize(N = n(), .groups = "drop") %>% data.frame() R> jss_sample_lu_2$Link %>% filter(Type == "Intra-node data Fetch" | Type == "Intra-node data TaskPreFetch" | Type == "Intra-node data PreFetch ") %>% group_by(Type) %>% summarize(N = n(), .groups ="drop") %>% data.frame()

Type N 1
Intra-node data Fetch 4982 2 Intra-node data TaskPreFetch 10

Type N 1

Intra-node data Fetch 677 2 Intra-node data TaskPreFetch 4122

QRMumps

QRMumps [START_REF] Agullo | Multifrontal QR Factorization for Multicore Architectures over Runtime Systems[END_REF]) is a task-based parallel sparse matrix factorization solver, built on top of StarPU. This solver implements the Multifrontal method [START_REF] Duff | The Multifrontal Solution of Indefinite Sparse Symmetric Linear[END_REF], a common approach to sparse solver parallelization. It breaks the factorization steps in smaller and denser sub-problems called frontal matrices, organized with an Elimination Tree structure. QRMumps dilutes the tree nodes in a fine-grained DAG of tasks, but some structure characteristics still influence the execution. For example, the communication between nodes and the available tree-level parallelism. We present in Figure 22 two executions where we only change the StarPU scheduler from lws (left) to prio (right). The scheduler affects the tree traversal and the moment where communication happens. For instance, when we analyze the prio scheduler, where the computational tasks have higher priority than the communication and initialization tasks, we can observe that the communication tasks are often executed just in the nodes' end. This effect limits the tree parallelism because the parent nodes have to wait for these communications to occur. In contrast, in the lws scheduler execution, we observe that the communication tasks are performed along with the computational tasks, enabling more tree parallelism, as we can observe in the Figure . Hence, such visualizations are interesting to verify the effects of the application and runtime configuration for a given workload, helping to understand and improve the application. They can be created as follows:

R> qr_lws <-starvz_read(directory = "data/qr-sched/lws/") R> qr_prio <-starvz_read(directory = "data/qr-sched/prio/") R> endTime <-max(c(qr_lws$Application$End, qr_prio$Application$End)) R> qr_lws$config$limits$end <-endTime R> qr_lws$config$title$text <-"LWS" R> qr_prio$config$title$text <-"PRIO" R> starvz_assemble(starvz_plot_list(qr_lws), starvz_plot_list(qr_prio), config = qr_prio$config)

OpenMP SparseLU via KStar

StarPU can execute OpenMP programs with the support of the KStar [START_REF] Agullo | Bridging the Gap between OpenMP and Task-Based Runtime Systems for the Fast Multipole Method[END_REF] source-to-source compiler. In this use-case, we select the SparseLU benchmark, which computes the LU factorization over a sparse matrix. This benchmark is part of the KASTORS suite [START_REF] Virouleau | Evaluation of OpenMP Dependent Tasks with the KASTORS Benchmark Suite[END_REF], which provides benchmarks with the recent OpenMP task depend construction. This primitive enables data-flow dependencies in OpenMP codes translating to fine-grained task-synchronizations as those obtained with the native StarPU API. The SparseLU benchmark stresses the runtime system scheduling capabilities since it presents intrinsic unbalanced loads due to the matrix's sparse structure. One can see in Figure 23 that all workers receive a similar workload. The ready tasks curve, although unstable, shows that the amount of ready tasks remains high enough to keep computing resources busy, resulting in low resource idleness.

R> starvz_phase1(directory = "data/kstar-sparselu") R> jss_sample_kstar <-starvz_read(directory = "data/kstar-sparselu", selective = FALSE) R> starvz_plot(jss_sample_kstar)

Final remarks

The performance analysis is an essential step in the development and optimization of parallel applications. However, the complexity of applications and computing platforms makes the analysis process laborious and difficult. The StarVZ package helps to tackle such complexity in the context of task-based parallel applications. The framework relies on modern data science tools to provide a configurable analysis workflow, combining proposed visualizations panels and scripting capabilities. In this paper, we present the StarVZ R-package available on CRAN, its design and philosophy, including methodological aspects, and most of the functions that are part of the package. We demonstrate the usefulness of the StarVZ package by giving examples of its functionalities in three real task-based applications comparing runtimes settings, application configurations, and appropriateness for generic OpenMP parallel applications. An analyst can use the views of StarVZ and quickly adapt it for their needs, as it uses standard and modern R-packages, providing the data and panels in familiar structures, such as tibble and ggplot objects. Moreover, StarVZ provides means of data aggregation for the analysis scalability.

Although StarVZ already provides handy functionalities, it is possible to expand further to improve the methodology and workflow. Future work includes additional support for other runtimes, including native support for the OpenMP task dependencies without relying on the KStar for execution thought StarPU. We also intend to include support for nested tasks with preemption. While the visualization may be enough for experienced analysts, automatic reports may be useful for beginners in the field. As a consequence, we intend to include textual suggestions to cover known performance problems, such as wrong worker pinning, resource performance verification (NUMA, contention, variability), possible causes for idle times (late execution of tasks with satisfied dependencies), and lack of ready tasks by problems in the application DAG, granularity or communication.

Acknowledgments

We would like to thank our colleagues Arnaud Legrand, Luka Stanisic, Guilherme Alles and Rémy Drouilhet for all the discussion and help during the development of StarVZ. We also would like to thank the StarPU team, especially Samuel Thibault for his insights. This study was financed by the "Coordenação de Aperfeiçoamento de Pessoal de Nível Superior" (CAPES) -Finance Code 001, the CNPq -Brazil scholarship for the 1º (141971/2020-7) and 3º (131347/2019-5) author, and the projects: FAPERGS MultiGPU (16/354-8) and Green-Cloud (16/488-9), the CNPq project 447311/2014-0, the CAPES/Brafitec EcoSud 182/15, the CAPES/Cofecub 899/18, Petrobras (2016/00133-9, 2018/00263-5). Some experiments are carried out in the PCAD infrastructure, http://gppd-hpc.inf.ufrgs.br, at INF/UFRGS.

Figure 2 :

 2 Figure 2: Detailed view of the workflow with its the pre-processing phase (A, left part) and data visualization and analysis (B, right), transforming execution traces into visualization.

Figure 3 :

 3 Figure 3: The space/time panel depicting the behavior of the LU factorization when executed with two computing nodes, each one with two GPUs and four CPUs.

Figure 4 :

 4 Figure 4: The space/time panel depicting application states.

Figure 5 :

 5 Figure 5: The space/time panel depicting runtime states.

Figure 6 :

 6 Figure 6: The elimination tree behavior along time.

Figure 7 :

 7 Figure 7: Number of active elimination tree nodes along time (top), and the total memory utilization along time (bottom).

Figure 8 :

 8 Figure 8: Resource utilization along time by elimination tree node (top), and by elimination tree depth (bottom).

Figure 9 :

 9 Figure 9: The k iteration panel showing iteration timespan and application states along time.

Figure 10 :

 10 Figure 10: The memory states panel depicting memory-related actions along time.

Figure 11 :

 11 Figure 11: The data handles history plot illustrating events related to a specific data block.

Figure 12 :

 12 Figure 12: The ready plot, as an example of variable panel, to depict values along time.

Figure 15 :

 15 Figure 15: The application space/time plot with node aggregation.

  Figure 17: The memory snap (left), memory heatmap (center), and distribution (right) plots.

Figure 18 :

 18 Figure 18: Illustrating the solution for the area bound estimation.

Figure 19 :

 19 Figure 19: GFlop and task duration diagnostic plot.

  Figure 20: A custom layout with the space/time view (left) and summary nodes (right). Moreover, if the goal is to present multiple StarVZ plots (of different executions) side-by-side for comparing purposes, the function starvz_assemble() can be used. The argument of starvz_assemble() is a list of views (for one plot only) or a list of views' lists (for multiple trace visualization). For coherence, only one argument config configuration should be use. The function automatically removes Y-axis titles (argument remove_Y_info) of the second and following plots and avoid legend duplication (argument remove_legends). An example of assembling two different trace visualizations is shown in Figure 21. The starvz_plot_list() function generates each plot, we combined them with a call to the starvz_assemble() function, as follows:

Figure 21 :

 21 Figure 21: Comparing two traces using the assemble function.

  is a great example of a use-case utilization of StarVZ. The figure presents identical executions except for one parameter, the StarPU scheduler. The left plot utilizes the lws scheduler while the right one utilize the dmdas scheduler.

Figure 22 :

 22 Figure 22: Execution comparison between lws (left) and prio (right) schedulers.

Figure 23 :

 23 Figure 23: The visualization for the KStar/OpenMP SparseLU application.

Table 1 :

 1 Table 1 lists the supported tables as available in the StarVZ version 0.4.0. Summary of data tables and their description.

	Table	Description
	Version	Single value describing the StarVZ data version
	Colors	Table containing tasks color information
	Y	The vertical structure of workers used in the Y-axis of different plots
	Application	The application tasks that include Start, End, and name of the Task
	Starpu	The internal tasks of the StarPU runtime
	Gaps	The computed temporal intervals between tasks
	Link	The data transfers or entities among workers/nodes
	Dag	The application directed-acyclic graph
	Pmtool	The performance bounds computed by pmtool (Eyraud-Dubois 2019)
	Pmtool_states	The tasks scheduling computed by pmtool (Eyraud-Dubois 2019)
	Atree	Information about the elimination tree present in some algorithms
	Tasks	Additional information about tasks
	Data_handles	Information about application data registered by the runtime
	Task_handles	Information about the data used by tasks
	Memory_state	States of the memory controller entities of the runtime
	Events_memory Events related to memory operations
	Events	Generic table for events
	Variable	Statistics and metrics computed during execution, like ready tasks
	Comm_state	States of the MPI communication entity
	Other_state	Generic table for other states of other entities
	Papi	Information about the tasks' hardware counters
	Origin	The original directory of the trace
	Zero	Timestamp of the beginning of the first application task

  Table 2 lists the panel functions organized by category. The arguments shared by all functions include data, legend, x_start, x_end, and step. The legend boolean parameter indicates whether to enable/disable legends.The x_start numeric parameter indicates the X-axis start value and defaults to the first application task's beginning timestamp. The x_end numeric parameter indicates the Xaxis end value and defaults to the last application task's end timestamp. All time-aggregated panels share a step parameter that contains a numeric value to define the size in milliseconds of the time aggregation interval. The default value for this parameter is 0.1% of the application makespan. The next Sections provide details about each of the panels, including an explanation about their parameters. These Sections use Chameleon[START_REF] Agullo | Faster, Cheaper, Better -a Hybridization Methodology to Develop Linear Algebra Software for GPUs[END_REF] and QRMumps[START_REF] Agullo | Multifrontal QR Factorization for Multicore Architectures over Runtime Systems[END_REF] executions traces as examples.

Table 2 :

 2 Summary of functions by category.

	Category	Function
	Worker	panel_st_raw()
		panel_st_agg_node()
		panel_st_agg_dynamic()
		panel_st_agg_static()
	Application panel_atree()
		panel_activenodes()
		panel_nodememuse()
		panel_utiltreedepth()
		panel_utiltreenode()
		panel_kiteration()
	Variable	panel_usedmemory()
		panel_gflops()
		panel_gpubandwidth()
		panel_mpibandwidth()
		panel_mpiconcurrentout()
		panel_mpiconcurrent()
		panel_ready()
		panel_submitted()
	Memory	panel_handles()
		panel_memory_snap()
		panel_memory_state()
	MPI	panel_mpistate()
	Overview	panel_lackready()
		panel_node_summary()
		panel_node_events()
	Pmtool	panel_pmtool_kiteration()
		panel_pmtool_st()


published or not. The documents may come

Understand and optimize natural and synthetic metabolic networks of microorganisms


Ikhlas Selmi

Paul Thevenon

Christophe Macabiau

Olivier Julien

Mikaël Mabilleau

Jaron Samson

Switzeland A G -Blox

Mikael Mabilleau

Test of the filter design effect on Evil Waveform monitoring in SBAS for Galileo E1 and E5a signals

INTRODUCTION

The so-called Evil Waveform (EWF) [START_REF] Enge | Detecting Anomalous Signals from GPS Satellites[END_REF] are signal distortions that occur at the satellite level due to payload failure. The first important GNSS EWF was observed in 1993 and other cases [START_REF] Shallberg | Catalog and Description of GPS and WAAS L1 C/A Signal Deformation Events[END_REF] occurred recently confirming the potential integrity and continuity risk of this type of signal anomaly. To characterize those distortions and protect civil aviation users from their effect, three threat characterizations, also known as Threat Models (TM), were adopted in 2001 by ICAO for GPS L1 C/A [START_REF] Phelts | Multicorrelator techniques for robust mitigation of threats to GPS signal quality[END_REF]. They are classified in three categories: TM-A, TM-B and TM-C. TM-A is associated with a failure in the Navigation Data Unit. It appears on the C/A code as a leading or lagging falling edge on all the positive chips with respect to their expected end-time. TM-B models degradation in the analog section of the satellites by an amplitude modulation applied to the correct signal. It can be also seen as the output of a second order system taking the nominal C/A code as an input. TM-C is a combination of both digital and analog failures. The ICAO Threat Space (TS) standardized for GPS L1 C/A is given in Table 1. The proposed TM for GPS L1 C/A signal is here assumed to be also valid for new GNSS signals as the Galileo E1 and E5a ones. The ICAO TM for Galileo E1 and E5a is given in the last update of Galileo SARPS (June 2020). To assess the effect of the filter design on the EWF cases located at the borders of the standardized TS and for the sake of more conservative assumptions, a larger TS (that goes beyond the standardized one) is used to test the EWF biases for Galileo signals as given in Table 2. The second step after defining the three TMs is to design a Signal Quality Monitoring (SQM) algorithm [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF] [6] able to detect EWF threat. The presented work here is conducted in the frame of Dual Frequency Multi-Constellation (DFMC) SBAS systems. So, the designed SQM has to cover all Hazardous EWF for SBAS users while respecting the SBAS integrity and continuity requirement. Hazardous TM-A, TM-B and TM-C correspond to the interval of TM parameter giving distortions with differential bias larger than the Maximum Error Range (MERR) that can be tolerated by the SBAS user. The SQM compliance test is conducted within both satellite scenarios risen and rising:

• The so-called risen scenario appears when the signal is undergoing an EWF when being visible to several SBAS monitor stations. So, for each SBAS user, the EWF bias is the maximum transient differential bias since the considered reference stations observe the evolution of the bias measurement before and after the EWF occurs. • In the rising scenario, the EWF event occurs when the affected satellite is not visible from the SBAS network.

The affected satellite only becomes visible from the SBAS network later. In this case only the steady state differential bias is considered to characterize each EWF.

The SQM detection process is mainly performed by SQM metrics which are computed using multi-correlator outputs [START_REF] Phelts | Multicorrelator techniques for robust mitigation of threats to GPS signal quality[END_REF]. An additional monitor called the Code-Carrier-Incoherence (CCI) [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF] is also applied on the reference bias measurement and can be used to detect jumps on the pseudorange measurement. However, this CCI monitor is available only during the risen satellite phase since it requires that the time variation of the code to carrier measurements is observed when the EWF occurs. In this paper, the impact of the design of the airborne receiver filter and reference station filters on the EWF differential bias and SBAS monitoring performance for Dual-Frequency Multi-Constellation (DFMC) system is to assess. The presented user space and Signal Quality Monitoring (SQM) performance in [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF] are referenced in this study as the baseline for the assessment of the impact of the tested aspects of the filter design. To evaluate the impact of the considered filter design parameters, the following elements are assessed and the observed changes when comparing them with baseline results of [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF] are analyzed:

• The differential tracking biases • The area delimited by the Maximum tolerable Error Range (MERR) set to 1 m for Galileo E1 signal and 2 m for Galileo E5a. The used approach to compute the MERR is presented in [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF]. • The compliance of the baseline SQM proposed in [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF] on the tested TM-A, B and C TS The four baseline filters used in [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF] are two Butterworth and two resonator filters, which are implemented to cover the worst case of filters with highest differential bias that can be experienced by SBAS user. The detailed description of these four filters is given in first section. Based on the analysis of the existing filter designs carried out by some manufacturers, two aspects of the filter design are assessed:

• the shift of the filter's center frequency due to temperature variation or filter aging

• the gain roll-off out of the filter bandwidth Filter analysis conducted by receiver manufacturer confirms the center frequency instability phenomena that shifts the filter center frequency with a certain percentage of its bandwidth. In this study, several values of frequency shift are tested with the four filters to assess its effect on the EWF differential bias with TM-A and TM-B. An increase of the observed TM-A and TM-B biases means an extension of the area to be covered by the SQM and so a potential increase of its complexity. The tolerable center frequency shift is defined as the limit value that does not induce larger TM-A and TM-B biases with respect to the baseline one.

The effect of different values of gain roll-off suggested by the receiver manufacturer are also tested. In the baseline filters, two values of filter gain roll-off are considered: 36 dB per octave for Butterworth filters, and 30 dB for resonators. Those filter roll-off values were considered too demanding by receiver manufacturers. In order to test the effect of using less selective filters, resonator filter roll-offs are replaced with values lower than 30 dB/octave, such as 24 dB/octave for airborne receiver filter. Based on the analysis of the induced EWF bias variation for TM-A and TM-B, the impact of the tested filter design on the area to be covered by the SQM is assessed. When the results show an increase of the EWF bias (due to the change in the filters design), an additional test of the SQM compliance is added to verify the impact on the SQM complexity. Based on this analysis, the conclusion on the tolerable gain roll-off is derived.

Since the TM-C area to be covered by the SQM is a combination of TM-A and TM-B areas, the conclusion on the effect on the TM-C differential bias is derived by the TM-A and TM-B results. However, when testing the SQM compliance, all TM-A, B and C are considered to check its performance on all hazardous cases.

In the next paragraphs, the details of this methodology are described and the obtained results are shown. Then, the used filter and filtering procedure to evaluate the baseline of the EWF bias and SQM are given. First, the used procedure to test the frequency shift and compare the result to the EWF baseline is presented. Then, the used procedure to test the filter roll-off and compare the result to the EWF baseline is presented.

EWF BASELINE SIMULATION

In this section, the EWF simulation algorithm including the signal generation and acquisition steps are described. It is mainly focused on the signal generation step at the payload and signal or correlation function filtering at the receiver.

Baseline EWF generation steps

The first step is the generation of the nominal signal and the nominal correlation function. Then, the TM-A, TM-B and TM-C distortions are applied. The second step consists in filtering the signal to simulate the effect of the antenna and RF/IF filter. In this study, a single filtering stage is used to model both the antenna and RF/IF filters. The effect of this filtering step on the distorted signal is here simulated following one of these two different procedures:

1. When only digital signals at baseband (around a carrier frequency of 0 Hz) are considered, the filtering step is applied to the correlation function of the distorted signal. This filtered correlation function is then used to evaluate the absolute tracking delay associated to the considered EWF case. 2. When the tested filters are implemented as pass-band around an intermediate frequency (IF), the baseband digital signal is translated to the IF before going through the filtering step. The filtered signal is translated again to baseband and correlated to the local replica to generate the distorted correlation function. In this study an IF of 150 MHz is chosen when testing filters at IF. This procedure is only applied when testing the effect of a non-zero frequency shift. For the baseline of the EWF characterization and SQM presented in [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF], signal generation and filtering step are applied following the first methodology using only baseband signals. In the next paragraph, the design of the baseline filters is detailed.

Design of the baseline filters

To simulate the precorrelation filtering impact (including the RF/IF filter and the antenna filter) at the airborne and the reference station receiver, four filters are considered:

• Filter 1: 6 th -order Butterworth.

• Filter 2: resonator filter type with a constant group delay equal to zero. The design of this filter is based on the full definition of the transfer function: o The amplitude response is 1 within the desired filter bandwidth. The filter roll-off outside of the filter bandwidth is set to 30 dB/octave. o The phase response is such that the group delay is constant and equal to 0.

• Filter 3: resonator filter type with a concave group delay and a 150 ns maximum differential group delay.

The design of this filter is based on the full definition of the transfer function: o The amplitude response is 1 within the desired filter bandwidth. The filter roll-off outside of the filter bandwidth is set to 30 dB/octave. o The phase response is such that the differential group delay reaches 150 ns at the edge of the filter bandwidth following a 3 rd order function and then goes back down following a 9 th order function. • Filter 4: the amplitude response is that of a 6 th -order Butterworth filter. The phase response is that of smallest order Butterworth filter leading to a differential group delay greater than 150 ns.

Figure 1 shows the transfer functions of the four filters in term of amplitude (in dB), phase (in degree) and group delay (ns). In Figure 1, the filters are generated in the baseband with a double-sided bandwidth of 24 MHz. These filters are used to characterize the baseline EWF bias. The parameters used to define the user and reference station receiver are given in Table 3 Table 3 Differential tracking error is estimated as the maximum on all combinations of user and reference configurations.

-5 -

Figure 1. Gain (top), phase (middle) and group delay (bottom) variation of the four filters generated at baseband

EFFECT OF THE CENTER FREQUENCY SHIFT

The center frequency of the tested baseline filters is set to the signal carrier frequency (0 Hz when only baseband signals are considered). Receiver manufacturer indicated that on real filter, the center frequency is never aligned with the signal carrier (at IF/RF) because of the frequency instability due to temperature variation or filter parts aging [START_REF]EUR 38-20 / WG62-107, Minutes of the 50th Meeting of EUROCAE WG-62 'Galieo[END_REF].

A frequency offset that can go up to 25% of the filter bandwidth is usually observed on the filter center frequency. In this section, the frequency instability effect on the TM-A and TM-B differential bias is evaluated.

Tested frequency shifts

To test the center frequency shift effect, the baseband signals and filters are translated around an Intermediate Frequency (IF) of 150 MHz. In this study, only the results with a positive shift of the filter center frequency are given and analyzed. The effect of a negative frequency offset on the signal spectrum is symmetrical to the positive one and so induces the same bias variation. The following frequency offsets are added on the filter center frequency:

• for the reference station (filter 1 with a bandwidth of 24 MHz), the frequency offset is set to 1 or 1.5 MHz (~4% and 6% of the filter bandwidth) and the case without frequency shift is also tested. Intermediate values of frequency offset as 200 and 500 kHz have also been tested on the reference station giving similar results, but will not be presented in this article. • for the aircraft filter, two values of positive frequency shift are tested with the four filters and all the tested bandwidths between 12 and 24 MHz: o 15 and 20% of the filter bandwidth for Galileo E5a. o 10 and 15% of the filter bandwidth for Galileo E1.

Different values of frequency offset are tested on the reference station and the user receiver since the temperature variation that can be experienced on these two filters is not of the same level. The aircraft receiver has to be tested for a higher level of temperature variation in the range of [-10.7; +88]°C. Here, it is assumed that the user filter is experiencing a temperature variation of 100°C (≅ 88 + 10.7). On the reference station, the temperature variation is assumed in the range of [-2; 30]°C. In this study, the reference station filter is tested for a temperature variation of 32°C. The theoretical frequency offset due to temperature variation can be computed as [START_REF] Blais | Feasibility of a Direct Sampling Dual-Frequency SDR Galileo Receiver for Civil Aviation[END_REF]: With the tested 𝛼 of -10 and -30 𝑝𝑝𝑚/°𝐶, the frequency shift on the aircraft receiver ranges between -1.6 and -4.7 MHz on for Galileo E1 and between -1.1 and -3.5 MHz for Galileo E5a. The tested ranges of frequency shift (15 and 20% of the bandwidth for Galileo E5a and 10 and 15% for Galileo E1) are inspired from these values obtained with the theoretical expression.

Δ𝑓 = 𝛼 × Δ𝑇 × 𝑓 𝑐𝑒𝑛𝑡𝑒𝑟 • 𝛼

Results on the EWF bias

In this section, the obtained TM-A and TM-B differential bias when shifting the filter center frequency with the previously defined offset on the reference station and user receiver are compared to the baseline differential bias obtained with the baseline filters (perfectly centered around 𝑓 𝑐𝑒𝑛𝑡𝑒𝑟 ). In this study, the results obtained for Galileo E5a and E1 signals are presented.

The EWF differential tracking bias considered here to characterize the EWF effect is the steady state differential bias computed as:

𝐸 𝑖 = max 𝑘,𝑚 (|(𝑈𝑠𝑒𝑟_𝑏𝑖𝑎𝑠 𝑚 𝑖 -𝑈𝑠𝑒𝑟_𝑏𝑖𝑎𝑠 𝑚 𝑛𝑜𝑚𝑖𝑛𝑎𝑙 ) -(𝑅𝑒𝑓_𝑏𝑖𝑎𝑠 𝑘 𝑖 -𝑅𝑒𝑓_𝑏𝑖𝑎𝑠 𝑘 𝑛𝑜𝑚𝑖𝑛𝑎𝑙 )| )
where

• 𝑈𝑠𝑒𝑟_𝑏𝑖𝑎𝑠 and 𝑅𝑒𝑓_𝑏𝑖𝑎𝑠 are respectively the user and Reference station pseudorange measurements at the steady state of the smoothing filters • 𝑖 denotes the i th distortion of the threat space • 𝑚 represents the m th type of user of the user design space • k represents the kth type of reference receiver of the reference receiver design space Results for Galileo E5a

Results for TM-A

The obtained differential bias when adding the tested frequency shift of 15 and 20% of the filter bandwidth on the user filters, and 0 and 1 MHz on the reference station filter are given in the left of Figure 2. The black curve of Figure 2 corresponds the baseline differential bias where the baseline filters are used (without any frequency shift). The curves on the right of Figure 2 are the result of the difference between the differential bias obtained with the tested shifts and the baseline one. It appears that for all Δ lower than 0.14 µs, the baseline differential bias is larger than the one obtained when adding the tested values of frequency offset. It means that when shifting the filter center frequency, the effect of these TM-A on the differential bias is decreased. It may be due to the fact that part of the distortion is removed when the filter bandwidth is shifted (as shown in the example of Figure 4). With a frequency shift of 15% of the filter bandwidth on the user filter (and both tested values on the reference station), the curves of the differential bias remain below the baseline one.

For Δ larger than 0.14 µs, the differential bias is higher than the baseline one only when 20% of the filter bandwidth is added to the user filter center frequency. However, the shown differential bias in these cases is caused by the chosen DLL design which is not enough efficient to track the heavily distorted signal as explained in the Appendix. In this study, the high differential biases obtained in these cases are due to the ENAC DLL design limitation and not the EWF effect. Based on the presented results for the frequency shift of 15 or 20%, the baseline differential bias used to characterize the TM-A for Galileo E5a is not exceeded (except cases where the DLL design reaches its limitation). It is then possible to tolerate a frequency shift of 15 or 20% of the filter bandwidth without modifying the baseline TM-A differential bias used to test the SQM compliance.

Results for TM-B

Figure 3 shows the TM-B differential bias iso-contour at the MERR (of 2 m) obtained with a frequency shift equal to 15% and 20% of the filter bandwidth applied to the used filter and with 0 and 1 MHz of frequency offset on the reference station filter. With a frequency shift of 15% of the filter bandwidth applied to the user filter (curve blue and green), the area determined by the iso-contour at 2 m is included in the one defined with the baseline results (black curve) except a small area for 𝑓 𝑑 between 5 and 6.5 MHz. The area defined by this iso-contour results in differential biases that are below (lower than) the ones obtained in the baseline case with respect to the TM-B parameters. It means that in most parts of the TM-B TS the observed differential bias is decreasing compared to the baseline one. As it is mentioned for TM-A, this lower differential bias can be due to the fact that with a shifted bandwidth, part of the effect of the distortion on the signal is filtered as shown in the example of Figure 4. However, when testing a frequency shift equal to 20% of the filter bandwidth (curve red and purple), the iso-contour of 2 m is moved to higher 𝑓 𝑑 values. It means that more TM-B cases are exceeding the MERR of 2 m and so need to be covered by the SQM. Note that with a frequency shift lower than 15% such as 5% or 10% of the filter bandwidth have also been tested on user filters. The obtained differential bias in this case is in between the baseline one and the result with a frequency shift of 15%. Hence, it can be concluded that the highest frequency offset that can be tolerated without modifying the area of TM-B distortion to be covered by the SQM can be set to 15% of the user filter bandwidth.

Results for Galileo E1

Results for TM-A

The obtained differential bias with the baseline filters and when adding the tested frequency shift of 10 and 15% of the filter bandwidth on the user filters and 0 and 1.5 MHz on the reference station filter are given in the left of Figure 5. The right figure shows the difference between the differential bias affected by the frequency shift applied to the center frequency and the baseline one. It appears that with all the tested values of frequency shift, the differential bias remains lower than the baseline one.

Figure 5. Differential bias obtained with the tested frequency shift on the user filter of 10 and 15% and on the reference station of 0 and 1 MHz (left figure) and difference on these differential biases with the baseline one (right figure) for TM-A GalileoE1

Results for TM-B

Figure 6 represents the TM-B differential bias iso-contour at the MERR of 1 m with baseline filters and with a frequency shift of 10% and 15% of the filter bandwidth on the user filter, and 0 and 1.5 MHz on the reference station filter.

It appears that, when allowing center frequency shifts up to 15% of the user filter bandwidth (red and purple curves), the iso-contour at the MERR of 1 m slightly exceeds the 𝑓 𝑑 limit (around 20 MHz) defined in the baseline case (black curve). However, with a frequency shift of 10% of the filter bandwidth applied to the user filter (blue and green curves), the area defined by the iso-contour at the MERR of 1 m is included in the same area shown with the baseline results (except a very tiny area for 𝑓 𝑑 around 18 MHz and 𝜎 lower than 0.3 Mnepers/s) These results suggest that 10% of the filter bandwidth is the highest frequency shift that can be accepted on the user filter center frequency without modifying the TM-B TS area to be monitored by the SQM for Galileo E1 signal.

Figure 6. the MERR iso-contours (on the left and a zoom on low 𝝈 on the right) with baseline filters and the tested frequency shift of 10 and 15% of the filter bandwidth on the user filter and 0 and 1.5 MHz on the reference filter for TM-B applied to Galileo E1 signal

Conclusion on tolerable frequency shift

Different values of center frequency offsets are tested on the user and reference station filters. Those values are inspired from the results of the theoretical expression of the frequency instability due to temperature variation applied to the user and reference station receiver. For Galileo E5a, two values of frequency offset are tested on the reference station filter: 0 and 1 MHz. On the user filter, the results with a frequency shift equal to 15 and 20% of the filter bandwidth are presented for Galileo E5a signal. For Galileo E1, the tested values of frequency offset on the reference station filter are 0 and 1.5 MHz. On the user filter, the results with a frequency shift equal to 10 and 15% of the filter bandwidth are presented.

All the tested frequency offsets are only simulated as a positive shift of the filter center frequency since the negative shift gives the exact same result as with the positive one. The TM-A and TM-B differential bias variation caused by the center frequency shift applied to Galileo E5a signal suggests that the highest frequency offset that can be tolerated on the user filter is 15% of the filter bandwidth. The obtained results for Galileo E1 signal prove that the highest tolerable frequency shift can be set to 10% of the user filter bandwidth to avoid including TM-B distortions with larger 𝑓 𝑑 in the set of EWF to be monitored.

Based on the analysis of the TM-A and TM-B differential bias results, a 10% of the receiver filter bandwidth shift on the aircraft's filter center frequency is acceptable for Galileo E5a and E1 signal processing.

EFFECT OF THE FILTER GAIN ROLL-OFF

For GPS signals, the minimum acceptable gain roll-off to be used with double delta discriminator is set to 30 dB/octave in DO-229E [START_REF]DO 229E -Minimum Operational Performance Standards (MOPS) for GPS/WAAS Airborne Equipment[END_REF]. However, there is no requirement on the filter gain roll off when using the Early-Late discriminator that is also allowed in [START_REF]DO 229E -Minimum Operational Performance Standards (MOPS) for GPS/WAAS Airborne Equipment[END_REF]. These remarks raised a question about the possible need for a similar requirement in the DFMC SBAS MOPS [START_REF] Eurocae | ED-259A v0.5 -MOPS for Airborne Galileo GPS SBAS Satellite Receiving Equipment[END_REF] when using the Early-Late discriminator which is the only tracking technique allowed in [START_REF] Eurocae | ED-259A v0.5 -MOPS for Airborne Galileo GPS SBAS Satellite Receiving Equipment[END_REF]. In the present study, the need for such a requirement in the case of Galileo signals is assessed based on the analysis of its effect on the EWF induced tracking bias and SQM compliance.

In this section, a lower value filters gain roll-off is tested to evaluate the effect of the use of a less selective filter by the aircraft receiver on the EWF bias and SQM compliance. The obtained results are compared to those of the EWF baseline with the baseline filters which respect a gain roll-off requirement of 30 dB/octave.

.

Tested gain roll-off

Among the baseline filters, two of them have a gain roll-off of 36 dB/octave (filter 1 and 4) as per 6 th order Butterworth filter definition. For these filters, the gain roll-off will not be modified to avoid changing the filter type. The transfer function filter 2 and 3 is defined based on two inputs: the gain roll-off and group delay variation. In the baseline filters, the value of 30 dB/octave is chosen for filter 2 and 3. In order to test the effect of a less steep gain roll-off, this value is replaced by 24 dB/octave. Figure 7 shows the curve of the filter 2/3 attenuation with respect to the frequency with a gain roll-off of 30 and 24 dB/ octave for a bandwidth between 12 and 24 MHz. The black dotted line represents the requirement on the antenna frequency selectivity given in DO-229E [START_REF]DO 229E -Minimum Operational Performance Standards (MOPS) for GPS/WAAS Airborne Equipment[END_REF] for L1 signals. It appears that with a gain roll-off of 24 dB/octave, filter 2 with the largest tested bandwidth of 24 MHz is slightly exceeding the requirement on the frequency selectivity. 24 dB/octave seems to be the lowest gain roll that can be accepted to respect L1 frequency selectivity requirement for all tested bandwidth. The dotted magenta line corresponds to the suggested frequency selectivity for L5 filters given in DO-292 [START_REF] Rtca | DO 292 -Assessment of Radio Frequency Interference Relevant to the GNSS L5/E5A Frequency Band[END_REF]. This L5 frequency selectivity is not considered here since it will be updated in the future version of DO-292.

The new tested set of filters includes:

• Filter 1 and 4 as 6 th order Butterworth filters with a gain roll-off of 36 dB/octave.

• Filter 2 and 3 with a gain roll-off of 24 dB/octave (instead of 30 dB/octave). In this study, the EWF differential bias obtained when applying the new set of filters is compared to the one given by the baseline filters.

Gain roll-off results for Galileo E5a and E1 Galileo E5a signal

Results on the TM-A

Figure 8 shows the differential bias obtained with the baseline filters and the ones modified with a gain filter roll-off of 24 dB/octave on the figure on left. The figure on the right shows the difference between the two biases that goes up to 22 cm.

Figure 8. the Steady state differential biases with baseline filters and modified filters with 24 dB/octave filter roll-off (left) and the difference between them (right) for TM-A on Galileo E5a

Results on the TM-B

The differential bias obtained for Galileo E5a signal affected by TM-B is given in Figure 9. The iso-contours on the left figure represent the result obtained with the baseline filters (gain roll-off set to 30 dB/octave on filter 2 and 3) and those on the right figure are obtained with the second set of filters using 24 dB/filter roll-off. An additional part of the iso-contour at 2 m, which corresponds to the MERR for Galileo E5a, appears around an 𝑓 𝑑 of 14 MHz on the contour figure on the right. It means that using a gain roll of 24 dB/octave instead of 30 dB/octave adds more TM-B distortions exceeding the MERR level. And so more EWF cases need to be covered by the SQM to protect the SBAS user. In the next paragraph, the compliance of the baseline SQM proposed in [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF] is tested with the set of distortions induced when using the modified set of filters (with 24 dB/octave gain roll-off requirement).

Figure 9. Steady state differential bias with baseline filters (left) and modified filters with 24 dB/octave (right) for TM-B on Galileo E5a

SQM compliance test applied to TM-A, B and C

The result of the SQM compliance test is evaluated in this section to check if using 24 dB/octave requirement on the filter gain roll-off changes the baseline SQM performance. The SQM compliance test is assessed for both risen and rising satellite scenarios defined in [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF]. The tested TM-A, B and C TS are given in Table 2. It corresponds to the same large TS tested in [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF] to validate the compliance of the baseline SQM which uses two types of metrics: • The CCI (Code-Carrier Incoherence) monitor which detects a jump on the pseudorange measurement of the reference station. The compliance of each metric 𝑀 is checked by computing the metric test as:

•
𝑇𝑒𝑠𝑡 𝑀 =
𝑀 -𝑀 𝑛𝑜𝑚𝑖𝑛𝑎𝑙 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 𝑀 Where 𝑀 𝑛𝑜𝑚𝑖𝑛𝑎𝑙 is the nominal value of the metric and 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 𝑀 is the metric threshold defined by the standard deviation of the metric and the probability of false alarm and missed detection that need to be satisfied for SBAS user for both satellite scenarios: risen and rising. More details on the evaluation of the metric threshold is given in [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF]. For each distortion of the tested TS, the SQM is considered compliant with the SBAS requirements if one of the tested metrics satisfy: 𝑇𝑒𝑠𝑡 𝑀 ≥ 1. Otherwise the considered distortion is not detected by the SQM. For the tested TS, the compliance of the SQM is validated when all EWF with a differential bias exceeding the MERR have at least one compliant metric (among the tested SQM metric and CCI monitor) for each satellite scenario: risen and rising.

Figure 10 shows the result of the maximum differential bias (y axis) as a function of the highest metric test (x axis) in both risen and rising satellite scenarios. In the left figure, the results are shown for high differential bias (until 200 m) and the figure on the right corresponds to a zoom on the low values of bias. All the distortions located above the MERR level (represented by the red dotted line at 2 m) correspond to a positioning failure situation for SBAS user and need to be covered by the SQM. The distortion located at the right side of the black vertical line (at 𝑇𝑒𝑠𝑡 𝑀 = 1) are detected by the tested SQM while satisfying the false alarm and missed detection SBAS requirement. EWF cases with a differential bias above the MERR line and non-compliant SQM performance (located at the left side of the vertical black vertical line) are sources of integrity risk events for SBAS user. Figure 10 shows that almost all hazardous cases are detected by the baseline SQM except few cases with a high differential bias around 180 m. These undetected EWF are further investigated by looking at the power loss due to these distortions. Where 𝐼 0,𝐸𝑊𝐹 and 𝐼 0,𝑛𝑜𝑚𝑖𝑛𝑎𝑙 are the prompt correlator outputs on the distorted and nominal correlation function respectively. The red circles in Figure 11 that represent the undetected cases (located at the left side of the black line in Figure 10) induce a high level of power loss that exceeds 35 dB, represented by the green vertical line. It means that when these undetected distortions occur, the acquisition or tracking loop of the aircraft receiver will lose the satellite signal because of its low 𝐶/𝑁 0 . Hence, these distortions will not have an impact on the integrity of the SBAS user and so can be ignored by the SQM1 .

Figure 11. EWF differential bias as a function of the correlation loss

Based on the presented results, the proposed SQM in [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF] protect the SBAS users against the EWF TS resulting from the set of modified filters with the use of a 24 dB/octave gain roll-off requirement.

Galileo E1 signal

Results on the TM-A

Figure 12 shows the steady state differential bias observed when applying the requirement of 30 and 24 dB/octave on the user filters on the left, and the difference between them on the right with respect to the TM-A parameter Δ. It appears that only a difference of few centimeters (lower than 8 cm) is computed on the differential bias when going from a requirement of 30 to 24 dB/octave on the filter gain roll-off.

Figure 12. the Steady state differential biases with baseline filters and modified filters with 24 dB/octave filter rolloff (left) and the difference between them (right) for TM-A on Galileo E1

Results on the TM-B

The observed differential biases when applying a gain roll-off requirement of 30 and 24 dB/octave on the Galileo E1 TM-B TS are given in Figure 13. The difference between the two contour figures is given in Figure 14. It appears that a difference of less than 10 cm is observed on the most part of the TM-B TS. A larger difference that goes to 40 cm is observed on a small area around 𝑓 𝑑 of 19 MHz and 𝜎 lower than 2 Mnepers/s. But this difference does not induce a significant change in the shape of the MERR iso-contour at 1 m. Hence, relaxing the gain roll-off requirement from 30 to 24 dB/octave can be accepted for Galileo E1 without any change in the area to be monitored by SQM for TM-B TS.

Conclusion on tolerable gain roll off

Even if a gain roll-off requirement of 30 dB/octave will be more suitable from a SBAS SQM perspective, the presented results confirm that a 24 dB/octave requirement on the filter gain roll-off can be accepted based on the EWF results for Galileo E5a. The proposed SBAS SQM in [START_REF] Selmi | Signal Quality Monitoring Algorithm Applied to Galileo Signals for Large Evil Waveform Threat Space[END_REF] is able to cover the Galileo E5a TS induced by the modified set of user filter with the gain roll-off set to 24 dB/octave. Similar results are also obtained with Galileo E1 with a very low change in the differential bias curve. The observed changes do not have an impact on the area of hazardous distortion exceeding the MERR. So, no additional SQM analysis is needed to confirm that the gain roll-off requirement could be relaxed to 24 dB/octave without a significant impact on the EWF characterization. The final definition for gain roll-off requirements (possibly more stringent) will then be driven by spectrum management and resistance to interference constrains at user level.

CONCLUSIONS

In this paper, filter design tests are carried out to evaluate its effect on the EWF characterization in term of differential bias variation and SQM compliance test. Two aspects of the filter design are considered:

o Filter gain roll-off: Two values of gain roll-off requirement are tested here: 30 dB/octave which is already used in the baseline filters and 24 dB/octave. The results of differential bias for Galileo E1 show a very low difference caused when relaxing the gain roll-off requirement to 24 dB/octave. However, with Galileo E5a signal, a more significant change in the differential bias curves is observed. To check if such a change will have an impact on SBAS user protection against EWF events, the SQM compliance test is rerun with the TM-A, B and C results obtained when applying a gain roll-off of 24 dB/octave. The SQM test confirms that all hazardous distortions remain covered when going from 30 to 24 dB/octave gain roll-off requirement. In order to keep a safe margin with respect to the tested values, it is preferable to keep the requirement on the filter gain roll-off set to 30 dB/octave. o Filter center frequency shift due to frequency instability caused by temperature variation. Different values of center frequency offset are tested on the user and reference station filter. Those values are inspired from the results of the theoretical expression of the frequency instability due to temperature variation applied to the user and reference station. For Galileo E5a, two values of frequency offset are tested on the reference station filter: from 0 to 1.5 MHz. On the user filter, the results with a frequency shift from 10 to 20% of the filter bandwidth are presented. All the tested frequency offsets are only simulated as a positive shift of the filter center frequency since the negative shift gives the exact same result as with the positive one. The TM-A and TM-B differential bias variation caused by the center frequency shift applied to Galileo E1 and E5a signal suggests that the highest frequency offset that can be tolerated on the user filter is ±10% of the filer bandwidth.

FUTURE WORK

During the RTCA/EUROCAE meeting on January 2020, more aspects of the filtering step simulation were pointed:

o The antenna filtering effect and the RF/IF filtering step are simulated by a single filter stage using the four filters presented in this paper. To assess the effect of gain roll-off requirement on the RF/IF filter, the antenna filter needs to be simulated separately. To do so, an additional filter stage will be added based on the antenna frequency selectivity given in [START_REF]DO 373 -Minimum Operational Performance Standards (MOPS) for GNSS Airborne Active Antenna Equipment for the L1/E1 and L5/E5a Frequency Bands[END_REF] o When defining the transfer function of filter 2 and 3, the considered gain roll-off requirement is used to define the amplitude slope out of the filter bandwidth. However, the minimum tolerable value for this slope is not defined. Hence, the filter amplitude is assumed to continue decreasing for higher frequency. In order to make the behavior of the simulated filters closer to real ones, a minimum amplitude level needs to be considered. A minimum filter amplitude of -50 and -70 dB is suggested for Galileo E1 and E5a respectively. These remarks will be taken into account to update the simulation and the presented results in this study.

In this paper, each of the two filter design aspects: gain roll off and center frequency offset are studied separately, and their effect is assessed independently from the other one. In order to conclude on the tolerable margin for each aspect, an additional test will be carried out, while taking into account the acceptable values for each filter design aspect (i.e. 24 dB/octave on the gain roll off and a center frequency shift of ±10% of the filer bandwidth).

As it is explained in the Appendix, the implemented DLL design at the user and the reference station receiver has some limitation that have an impact on the presented results. This DLL design (used in ENAC EWF tool) will be reviewed to improve its robustness to EWF cases with multiple zero crossing. Then the conclusion of this paper will be updated with the new results.

Figure 2 .

 2 Figure 2. TM-A Differential bias obtained with the tested frequency shift on the user filter of 15 and 20% and on the reference station of 0 and 1 MHz (left figure) and difference on these differential biases with the baseline one (right figure) for Galileo E5a.

Figure 3 .Figure 4 .

 34 Figure 3. the MERR iso-contours with baseline filters and the tested frequency shift of 15 and 20% of the filter bandwidth on the user filter and 0 and 1 MHz on the reference filter for TM-B applied to Galileo E5a signal

Figure 7 .

 7 Figure 7. Filter 2 and 3 amplitude with a gain roll-off of 30 (left) and 24 (right) dB/octave

oo

  The SQM metrics based on a combination of the multi-correlator outputs. The expression of the used metrics in the baseline SQM are given as a function of 𝐼 0 the prompt correlator output and 𝐼 𝑥 the correlator output located 𝑥 chip away from 𝐼 0 : o The simple ratio metric 𝑀 𝑠𝑟 = 𝐼 𝑥 𝐼 0 The difference ratio metric 𝑀 𝑠𝑑𝑟 = 𝐼 𝑥 -𝐼 -𝑥 𝐼 0 The double difference ratio metric 𝑀 𝑑𝑑𝑟 = (𝐼 𝑥 -𝐼 -𝑥 )-(𝐼 𝑦 -𝐼 -𝑦 ) 𝐼 0

Figure 10 .

 10 Figure 10. EWF differential bias as a function of the highest metric test (left) and a zoom on low bias (right) Figure 11 represents the EWF differential bias as a function of the power loss (in dB) induced by the presence of the EWF on the correlation function. This power loss can be expressed as: 𝑃 𝑙𝑜𝑠𝑠 (𝐸𝑊𝐹) = 10 log ( 𝐼 0,𝐸𝑊𝐹 𝐼 0,𝑛𝑜𝑚𝑖𝑛𝑎𝑙 )

Figure 13 .

 13 Figure 13. Steady state differential bias with baseline filters (left) and modified filters with 24 dB/octave (right) for TM-B on Galileo E1

Table 1 .

 1 Standardized Threat Space for GPS L1 C/A

		∆ (µ𝑠)	𝜎 (Mnepers/s)	𝑓 𝑑 (MHz)
	TM-A	[-0.12; 0.12] chip or [-117.3; 117.3] ns	-	-
	TM-B	-	[0.8; 8.8]	[4; 17]
	TM-C	[-0.12; 0.12] chip or [-117.3; 117.3] ns	[0.8; 8.8]	[7.3; 13]

Table 2 .

 2 Tested Threat Space for Galileo E1 and E5a

	ICAO parameters	∆ (µ𝑠)	𝜎 (Mnepers/s)	𝑓 𝑑 (MHz)
	TM-A	Galileo E1 Galileo E5a	[-0.16; 0.16] [-0.16; 0.16]	-	-
	TM-B	Galileo E1 Galileo E5a	--	[0.1; 700] [0.1; 370]	[0.1; 55] [0.1; 30]
	TM-C	Galileo E1 Galileo E5a	[-0.16; 0.16] [-0.16; 0.16]	[0.1; 700] [0.1; 370]	[0.1; 55] [0.1; 30]

  is the filter technology temperature coefficient in 𝑝𝑝𝑚/°𝐶 o The standard value of 𝛼 is -30 𝑝𝑝𝑚/°𝐶 o Lower 𝛼 associated to SAW technology is -10 𝑝𝑝𝑚/°𝐶 • Δ𝑇 is the temperature variation • 𝑓 𝑐𝑒𝑛𝑡𝑒𝑟 is the signal carrier frequency of 1575.42 MHz on L1 and 1176.45 MHz on L5.

The red circle in the bottom left corner is at (0, 0) so below the MERR.

ACKNOWLEDGMENTS

Dr. Olivier JULIEN's work contributing to this paper was exclusively performed when he was ENAC employee.

Mikael Mabilleau is a standardisation engineer on SBAS in the EGNOS exploitation team of the European GNSS Agency (GSA). Since his graduation as engineer from the French civil aviation school (ENAC) in 2006, he has been involved in GNSS standardization activity carried by the main civil aviation standardisation bodies such as the EUROCAE WG 62 on Galileo, RTCA Special Committee 159 on GPS and the International Civil Aviation Organisation (ICAO) Navigation System Panel in charge of ICAO standard. Mikael is involved in the evolution of GNSS concepts (ARAIM and SBAS L1L5) supporting the development of the associated standards (SARPs and MOPS) for use in aviation. Jaron Samson has been working as a System Engineer at ESA's EGNOS Project Office in Toulouse (France) since 2012. Prior to 2012, he worked as a Navigation Engineer at ESA's Technical Centre (ESTEC), supporting Galileo, EGNOS and the GNSS Evolutions Program. Jaron holds an M.Sc-degree in Physical and Mathematical Geodesy from

APPENDIX

For Δ larger than 0.14 µs, the shown TM-A differential bias, for Galileo E5a, is higher than the baseline one only when 20% of the filter bandwidth is added to the user filter center frequency. However, the shown differential bias in these cases corresponds only to the result of user filter 4 with a bandwidth of 12MHz (giving the maximum bias) where the DLL in ENAC EWF tool loses the tracking of the correlation peak. Figure 15 shows two examples of the zero crossing points (green crosses) obtained with the user filter 4 when its center frequency is shifted by 20% of its 3dB bandwidth. In these figures, the correlation functions and the discriminators obtained with filter 4 (red and blue lines) at a bandwidth of 12 (right) and 20 MHz (left) are compared to the ones obtained with baseline reference filter (in black and dotted purple lines). It appears clearly that in the right figure (filter 4 with 12MHz bandwidth), the implemented DLL is not tracking the right zero crossing (represented by the red circle). Then, the obtained bias is not due to the tested EWF, but to the limitation of the DLL design. This DLL anomaly is only observed with one tested user filter (filter 4 at 12MHz bandwidth). It also appears that if the right zero crossing is chosen, the obtained differential bias will be in the same order of magnitude than the ones obtained with Δ below 0.14 µs. Therefore, this is not changing the conclusion of the study.