Sébastien Schwartz
email: sebastien.schwartz@isae-supaero.fr

Juan José Montero Jiménez

Rob Vingerhoeds

Michel Salaün

An unsupervised approach for health index building and for similaritybased remaining useful life estimation

Keywords:

Predictive maintenance techniques attempt to trigger a maintenance intervention at the right moment by estimating the life expectation. Predictive maintenance is increasingly implemented by automated approaches able to perform diagnostics and prognostics. The main part of recent research in these approaches is focused in machine learning structures whose reasoning is implicit and cannot be easily explained. This poses a problem for their implementation in highly constrained area such as aeronautics. To overcome this constraint, explicit reasoning approaches such as the Similarity-Based Model (SBM) can be implemented. The SBM has been widely used for fault diagnostics and the remaining useful life (RUL) estimation, but the development of SBM includes tasks that often rely on high skilled experts. For instance, data reduction techniques required for SBM are often performed by experts judgment whose outcomes are not always consistent. The produced features from these techniques are used to build the Health Index that can be used to create the degradation trends that serve as a reference for the SBM. To overcome these difficulties, an automatic and unsupervised approach based on the Kernel Principal Component Analysis is proposed to enhance the Health Index creation. It preserves as much of the sensor information as possible improving the similarity-based RUL estimation. Additionally, when estimating the RUL of a system, the most similar degradation trends stored in the SBM library are used to compute individual RULs, the final RUL is obtained by a fusion rule technique that combines all these individual RULs into a consolidated value. For the fusion rule techniques, a self-adaptive method that does not rely on human expertize is proposed. This fusion rule can benefit of the accumulated knowledge over the SBM operation. This unsupervised approach to develop a SBM is validated with promising results against an equivalent and supervised algorithm that came out best in the 2008 prognostic health management challenge.

Introduction

Predictive maintenance refers to a specific maintenance strategy, aiming at identifying incipient faults, forecasting future failures and at triggering the maintenance actions accurately when needed [START_REF] Montero Jimenez | Towards multi-model approaches to predictive maintenance: a systematic literature survey on diagnostics and prognostics[END_REF]. Maintenance strategies are often classified into three categories: (1) corrective maintenance where actions are performed to restore a system after a breakdown or a deteriorated functional behavior, (2) preventive maintenance where maintenance actions are performed at a fixed operational interval, and (3) predictive maintenance where specific maintenance actions are based on measurements on the concerned system. Remaining Useful Life (RUL) estimation plays an important role in predictive maintenance; it provides insight on the system deterioration due to faults appearance or wear, and tries to show when the system would no longer perform its intended function. As such, accurate RUL estimation improves safety, reliability and availability of the system. It attempts to avoid sudden breakdowns minimizing unnecessary maintenance time / cost [START_REF] Gu | A framework of multi-index modeling for similarity-based remaining useful life estimation[END_REF]. Therefore, it is important to focus on techniques that can improve RUL predictions, especially on high-risk engineering systems (Li et al., 2018;[START_REF] Yu | Remaining useful life estimation using a bidirectional recurrent neural network based autoencoder scheme[END_REF][START_REF] Zhao | Remaining useful life prediction of aircraft engine based on degradation pattern learning[END_REF].

According to [START_REF] Ramasso | Performance benchmarking and analysis of prognostic methods for CMAPSS datasets[END_REF], prognostics for maintenance can be divided into three broad categories as: functional mappings between set of inputs and RUL, such as [START_REF] Li | A directed acyclic graph network combined with CNN and LSTM for remaining useful life prediction[END_REF], [START_REF] Jiang | A novel ensemble fuzzy model for degradation prognostics of rolling element bearings1[END_REF], functional mapping between Health Index (HI) and RUL, such as Hassani et al. (2019), [START_REF] Climente-Alarcon | Combined model for simulating the effect of transients on a damaged rotor cage[END_REF], and similarity-based matching, such as [START_REF] Bleakie | Analytical approach to similarity-based prediction of manufacturing system performance[END_REF], [START_REF] Liang | A degradation degree considered method for remaining useful life prediction based on similarity[END_REF], [START_REF] Zhang | A fault early warning method for auxiliary equipment based on multivariate state estimation technique and sliding window similarity[END_REF]. The main focus of this work is on the use in aeronautics, where it is important that the https://doi.org/10. 1016/j.compind.2022.103716 approaches can be perfectly understood and explained fully. The last category, similarity-based matching, is therefore interesting as it contains approaches that are more explainable than machine learning approaches. The current study focuses on similarity matching for RUL computation. A Similarity-Based Model (SBM) uses operational records to represent historic degradation trends for a given system (e.g. an aircraft, a vehicle, etc.), that are later used to assess the degradation of systems of the same type [START_REF] Wang | A similarity-based prognostics approach for Remaining Useful Life estimation of engineered systems[END_REF]. These degradation trends are built upon a representative set of run-to-failure system instances (e.g. engines, bearings, battery cells, etc.). A physical degradation trend should be generated for each system instance. The RUL for these degradation trends is known and they are stored in a library. Prognostic assessments are performed by comparing the degradation evolution of a system against those stored in the SBM library. The degradation trends can be represented by direct measures such as sensor records, or indirect measures such as health indexes computed from sensor records. The use of HI is often privileged over raw sensor data as it can consolidate several data sources in a single value that facilitates the understanding and visualization of a system degradation. Another important aspect in the SBM for RUL estimation is the uncertainty management in the prognostics. A SBM can retrieve multiple reference degradation trends from its library that are similar to the current degradation of a system under test, but with different RUL estimations. A fusion rule is often used to obtain a consolidated value of RUL, considering the contribution of the different reference degradation trends that have been retrieved by the SBM.

The HI development and the selection of suitable reference degradation trends to estimate a RUL are complex tasks that rely on experts' knowledge and judgment. This study attempts to simplify these processes by the implementation of data engineering techniques such as the Kernel Principal Component Analysis (KPCA) for the HI development and a self-adaptative fusion rule for RUL estimation. The KPCA can help to automate and simplify the feature extraction and data processing tasks that are exclusively performed by experts. Performing manual data reduction requires human knowledge and expertize in order to produce a proper feature subset. Manual tuning on feature techniques is done if the case study needs it. Automatic approaches, that are not specific to a specific case study, are promising, as they make the approach more consistent, as opposed to humans that may or may not always perform work in the same way. In addition, automatic approaches allow to estimate the loss of the data and try to come up with something as close as possible to reality. The proposed fusion rule also attempts to automate the selection of the reference degradation trends retrieved by the SBM to compute the final RUL of a system unit under test. RUL can be presented as a discrete value or as an interval. Both options are studied in the proposed SBM. The goal is to facilitate the implementation of SBM for RUL computation even for those professionals who are not specialists in HI development, health/degradation modeling, and fusion rules for RUL estimation using SBM.

The paper is organized as follows: Section 2 presents the state of the art in SBM for RUL computation and justifies the current work objectives. Section 3 presents the methodology leading to the RUL estimation using the KPCA technique and a SBM. Section 4 presents the case study selected to assess the proposed approach, the 2008 Prognostic Health Management (PHM08) challenge [START_REF] Wang | A similarity-based prognostics approach for Remaining Useful Life estimation of engineered systems[END_REF]. Section 5 presents the results and discussion of the proposed approach implementation on the case study. Section 6 concludes the paper and proposes future work perspectives.

Related work

Prognostics in maintenance can be carried out by different types of models [START_REF] Montero Jimenez | Towards multi-model approaches to predictive maintenance: a systematic literature survey on diagnostics and prognostics[END_REF][START_REF] Berri | Computational framework for real-time diagnostics and prognostics of aircraft actuation systems[END_REF]. These models can be classified in three main families: data-driven models, knowledge-based models, and physics-based models. Models from these families can be combined and these multi-model combinations are often called hybrid models. Within the data-driven models the Similarity-Based Models (SBM) are found. SBM have been widely used for prognostics purposes as they are relatively simple to implement and provide results that are explainable.

One of the main challenges in prognostics models is related to the Health Index (HI) generation that allows the study of degradation trends of a system of interest. For a SBM, these degradation trends are stored in the SBM library and are used as reference to assess the degradation of other systems of the same type. There have been some attempts to automate the creation of HI from raw data for SBM. For example, in [START_REF] Wang | A similarity-based prognostics approach for Remaining Useful Life estimation of engineered systems[END_REF], sensor reduction is performed, followed by a linear regression on the remaining sensor set to compute the HI. Then, an exponential nonlinear regression model is used to generate the degradation trends. In [START_REF] Huang | Improved trajectory similaritybased approach for turbofan engine prognostics[END_REF], HIs are produced by multi-linear regressions and trajectory similaritybased prediction is used for real-time estimation of remaining useful life. In order to improve RUL estimation, several variables can be considered to build the HI, increasing the complexity in the creation of the HI. For example, [START_REF] Cai | A similarity based methodology for machine prognostics by using kernel two sample test[END_REF] uses a similarity matching procedure to query similar run-to-failure profiles from historical data library. The selection is performed by the technique of the kernel two sample test [START_REF] Gretton | A kernel two-sample test[END_REF]. The probability distribution of the RUL is obtained by a Weibull analysis. Other approaches use artificial neural networks with similarity-based approaches such as in [START_REF] Yu | An improved similarity-based prognostic algorithm for RUL estimation using an RNN autoencoder scheme[END_REF], or [START_REF] Bektas | A neural network framework for similarity-based prognostics[END_REF].

The level of complexity increases with the data size for training the SBM. Large databases can demand high computational resources to compute the HI and obtain the degradation trends. To overcome these obstacles, data reduction is crucial. However, reducing the data and keeping the relevant information poses a challenge to predictive maintenance systems developers [START_REF] Kumar | Maintenance in the era of industry 4.0: issues and challenges[END_REF]. Two main techniques are often implemented for the data reduction: feature selection and feature extraction. Feature selection techniques reduce a data set by discarding inappropriate or redundant variables (features), but may result in a loss of information from discarded features [START_REF] Guyon | An introduction to variable and feature selection[END_REF]. Feature extraction techniques attempt to extract new features from an initial set of data measurements. Feature extraction tends to preserve more information than feature selection after the data reduction. It involves a transformation that is often not reversible due to information loss during the process, an example of feature extraction in the principal component analysis Pearson [START_REF] Pearson | LIII. On lines and planes of closest fit to systems of points in space, The London, Edinburgh, and Dublin[END_REF]. An example of this can be found in [START_REF] Chen | Remaining useful life estimation of aircraft engines using a modified similarity and supporting vector machine (SVM) approach[END_REF], where sensors showing evident trends are manually selected and a Principal Component Analysis (PCA) is later performed to create a new basis generated by the two principal components. In this new basis, a failure center is determined. It corresponds to a point reached by system instances when a failure occurs. The distance between a projected instance and the failure center is used to represent the system health state. An extension of the PCA is the Kernel Principal Component Analysis (KPCA) which can be used to generate a degradation indicator by combining KPCA outputs (i.e. principal components) with Arrhenius and Eyring models [START_REF] Feng | A kernel principal component analysis-based degradation model and remaining useful life estimation for the turbofan engine[END_REF]. Model parameters are estimated with the maximum likelihood estimation. As part of the current study contributions, a KPCA is implemented to build a HI, reducing the amount of data that allows a reduction of computation power to be processed, but also avoiding the loss of important information during the data reduction.

Another challenge of SBM for prognostics in maintenance is related to the RUL computation. When a new system unit is tested with the SBM, several reference degradation trends can be retrieved from the SBM library with a close similarity in terms of the degradation evolution but with different RUL estimations. To overcome this uncertainty problem, fusion rules are often implemented in SBM to estimate a consolidated value from a contribution of the individual RUL i obtained from comparing a testing unit against the reference degradation trends in the SBM library. [START_REF] Khelif | RUL prediction based on a new similarity-instance based approach[END_REF] and [START_REF] Bektas | A neural network framework for similarity-based prognostics[END_REF] propose a fixed number of individual RUL i to be fused which is fixed by a human expert while [START_REF] Wang | A similarity-based prognostics approach for Remaining Useful Life estimation of engineered systems[END_REF] considers all available RUL i and proposes some expert rules to discard non-representative ones. However, the selection of the individual RUL i that will be used to estimate the final RUL is not a trivial task and has a high dependence on human expert judgments that will select the number of RUL i to be considered or will discard manually non-representative reference degradation trends from the available options. As these fusion rules highly rely on human experts, there is no possibility to incorporate new knowledge acquired during the SBM operation. The current study proposes a self-adaptive method that autonomously proposes a set of individual RUL i to compute the final RUL. The goal is not only to avoid the dependence on the human experts to discard non-representative degradation trends, but also to incorporate the accumulated knowledge during the SBM operation that can be used to refine the fusion rule.

This study attempts to improve RUL estimation using a SBM. The contributions of the current study can be seen as:

1. An unsupervised building of a Health Index with limited information loss by the implementation of a KPCA in the data preprocessing to train the SBM. 2. A self-adaptive fusing rule to estimate the final RUL without the dependence of human experts manipulation that is also capable to incorporate accumulated knowledge acquired from the tested units over the SBM operation.

Proposed methodology

Overview

A Similarity-Based Model (SBM) development includes two main parts: the creation of the library of known references which in the case of predictive maintenance are the known degradation references, and the similarity measures that will compare a target data record against the known references in the SBM library. In the current study, this comparison will allow to determine the RUL of a system. Fig. 1 summarizes the proposed SBM model for RUL estimation. It has been divided in three main steps: data processing, performance assessment and RUL estimation.

On the left side of Fig. 1 the training of the SBM is explained. The raw training data is assessed with the Kernel Principal Component Analysis (KPCA) to obtain the principal components to build the health index with a reduced data but keeping as much degradation information as possible. The health index for each operational cycle of each run-to-failure system in the training data can be used to build the degradation reference trends.

On the right side of Fig. 1 the RUL estimation process is presented. The data from the target system to be assessed is processed (the same way as for the training of the SBM) in order to obtain the actual health index. Then, by accumulating a health index from each operational cycle of the target system a degradation trend can be obtained. This degradation trend is compared against those in the SBM Library to obtain individual RULs from each comparison. At this point, several comparisons with a very high similarity can be identified, but often these multiple comparisons provide very different RUL estimations. To overcome this uncertainty problem a fusion rule is proposed in order to obtain the actual RUL of the evaluated system. This chapter addresses the theoretical background of the proposed SBM for RUL estimation and extends the explanations of the main steps presented on the right side of Fig. 1.

Similarity-based model library

A vital component in a Similarity-Based Model (SBM) for prognostics in predictive maintenance is the library that stores known degradation trends, that are later compared against the current degradation of a system to compute the RUL. These degradation trends can be built from raw sensor data in form of time series. A data processing step is often performed on the sensor data to select features that can describe the system behavior and to eliminate unusable and redundant data [START_REF] Guyon | An introduction to variable and feature selection[END_REF]. Variance offers a good mathematical tool to automatically discard features with a low evolution. Performing variance sensors with a custom threshold produces a subset of selected features.

In the proposed methodology, the Kernel Principal Component Analysis (KPCA) [START_REF] Schölkopf | Kernel principal component analysis[END_REF] is used. The approach is able to represent the high order statistics of data thanks to its nonlinear nature [START_REF] Datta | PCA, kernel PCA and dimensionality reduction in hyperspectral images[END_REF], compared to the principal component analysis, which is a linear transformation. The KPCA is an unsupervised technique that reduces the data set and leads to a lower feature set, with a normalization on the interval [-1;1]. A min-max normalization method is adopted favouring the KPCA to find a better subset representation with all variables in the same scale. No knowledge about system specificities nor system lifetime is used. KPCA might be confused with an auto-encoder approach. Despite the similarities, it is important to recall that auto-encoders rely on a machine learning structure that requires training and testing phases that are not required in a KPCA approach [START_REF] Fournier | Empirical comparison between autoencoders and traditional dimensionality reduction methods[END_REF]. In addition, because of its structure, an auto-encoder approach may lead to reasoning that can not be explained. Most of the time, feature selection and feature extraction techniques are used to reduce the amount of information so to perform prognostic estimation efficiently. The proposed methodology aims at automating these feature selection and extraction processes using KPCA in such a way the approach becomes consistent, supported by more formal approaches that allow to estimate the loss of the data and keeping as much information as possible in the reduced subset of data.

Let us now describe briefly how KCPA works. KPCA is an extension of Principal Component Analysis (PCA) that generalizes it to nonlinear dimensionality reduction by using kernel methods.

Considering n as the number of samples and m as the number of features, the input space of data x (k, l) ∈ R (m×n) is mapped to the feature space F: Φ(x k ) with k ∈ [1, m] by the non-linear function Φ as:

F : n m (1)
It is assumed that feature data are centered, i.e.

x ( ) 0

k m k 1 = = . The covariance matrix C is: C m x x 1 ( ) ( ) j m j j T 1 = = (2)
The Principal Component (PC) of the feature space F are determined through the eigenvalues λ≥0 and the unitary eigenvectors V from the relation:

V C V . = (3)
This formula is equivalent to:

x V x CV k m ( ), ( ), for all [1, ] k k = (4)
where 〈x, y〉 is the dot product operation of x and y. Eigenvectors V can be written as:

V x ( ) i m i i 1 = = (5)
where α i is the corresponding eigenvector coefficient. Substituting Fig. 2 and Fig. 5 into Fig. 4 and defining the kernel m × m matrix K as:

K x x i m j m ( ), ( ) , [1, ], [1, ] ij i j = (6)
the following expression is exhibited:

m K K 2 = (7)
where [ , ..

., ]

m T 1

=

. The resolution of this relation is performed through the sub-system:

m K = (8)
In general, the ϕ(x i ) are not centered. In this case, matrix K is replaced by the kernel-centered matrix K [START_REF] Schölkopf | Kernel principal component analysis[END_REF], which is then computed by:

K K I K K I I K I . m m m m = + (9)
where I m is the diagonal m × m squared matrix with 1∕m as value on the diagonal. There exist several kernel non-linear functions to compute the matrix K. A Gaussian Radial Basis Function (RBF) kernel is used here. For a given couple x x ( , ) i j , the RBF kernel, named G, is defined as:

G x x exp K G x x i m j m ( , ) ( ( , )) , [1, ], [1, ] i j x x ij i j ij 2 i j 2 2 = = ( 10
)
where σ is the dispersion coefficient of the RBF. Thus, the k th KPCA feature is expressed as:

PC x V x x x ( ) , ( ) ( ), ( ) k k i m i k i 1 , = = = (11)
where V k is the k th eigenvector, Φ(x) is a test point and PC k is the k th principal component (PC). The number of PCs has an influence on the computed health index (HI). First, PCs are ordered in a descending order on the corresponding eigenvalue and each PC contribution is evaluated using:

Contr 100% k k i m i 1 = × = (12)
where Contr k is the contribution of the k th PC. Then, the number of Principal Components (nPC) is evaluated using the Cumulative Contribution Percent (CCP) with a custom threshold (CCP th ) as:

CCP Contr CCP i nPC k th 1 = = (13)
The Health Index (HI) is then computed by linear combination weighted with the percentage of contribution as:

HI x Contr PC x ( ) ( ) i nPC k k 1 = = (14)
The generated HI is rescaled in the interval [0;1] and represents the system health state. A high value equal or close to 1 represents a healthy system whereas a low value close to 0 corresponds to a faulty system. In addition to the physical degradation trends, confidence bounds are estimated with bootstrap techniques (Carpenter and Bithell, 2000). Confidence bounds are the lower and upper intervals that indicate the likelihood to predict correctly the remaining useful life within a given certainty.

The training of the similarity-based model is performed using all available run-to-failure instances of a system. The generated physical degradation trends expand the similarity database in order to be used by the similarity method itself for the RUL estimation.

Similarity assessment for RUL estimation

A degradation trend M i from each training unit (i.e. a system instance used to train the similarity-based model) is created and stored in the model library. These trends can be described as functions y in terms of a time t:

M y f t t T : ( ), [0, ] i i = (15)
where T i is the final time associated to the trend M i . The physical degradation trend shape depends on the application. Different physical models could be used to create the function that describe the system health state. The similarity degree d i between a testing unit (i.e. a system instance under investigation) and training units is determined with a distance function d(Y j , M i ) where Y j is the output of the testing unit j of T j instances as Y j = y y T [ (0), ..., ( )] j T . In this paper, Y j represents the health index of a testing unit j. A small distance means high similarity whereas the opposite represents no relation. This function can be defined in various ways such as the Manhattan distance. The Euclidean distance is the simplest one [START_REF] Wang | A similarity-based prognostics approach for Remaining Useful Life estimation of engineered systems[END_REF]:

d d Y M Y M ( , ) i j i j i 2 = = (16)
The two vectors Y j and M i are respectively of length T j and T i . The distance is then computed on the minimum of the length as T T min( , ) j i . A first attempt to estimate the RUL could be performed by assessing the most similar degradation reference trend to the current degradation of the system under study. It is important to note that the age at which data is recorded, as well as the initial degradation of time series, has no influence in proposed approach. In contrast to other approaches such as a Cox model, in which the initial risk of failure is required to estimate a RUL, the proposed similarity-based degradation assessment and RUL estimation can be performed on systems with unknown initial degradation. Incidentally, the case study in the next section is composed of degradation records with an unknown initial point that helped confirm the proposed approach advantages. Different reference trends from the SBM library might show the highest similarity with a different RUL estimation. This phenomena is often observed when the system is in the incipient phase of degradation and the process until failure could last a few or several cycles. overcome this uncertainty problem, SBM for predictive maintenance often use fusion rules to estimate the RUL based on several retrieved references from the SBM library. For the proposed model, a distance measure d i is estimated for each degradation reference trend M i , and using a fusion rule the individual RUL i are consolidated in a final RUL value:

RUL w RUL w w w w ¯. , ¯1, ī i i i i i i k k = = = ( 17
)
where w i and w are respectively a weight that is estimated by the inverse of the degree of similarity d i and d k , using Fig. 16. The weights w ¯i represent the percentage of similarity between a testing unit j with a unit i in the SBM library. The weight sum of all considered units i are then equal to 1 (i.e. 100%). It means that a unit with a high degree of similarity will have a weight value near 1. The number of unit i used to compute the final RUL is addressed by a fusion rule (see Fig. 3.4). This number is the same for the sum of index k. At the minimum, only 1 engine in the SBM library is considered and at the maximum, all of them. Fig. 17 provides a guideline for estimating the RUL as single value in terms of operational cycles.

In real life applications, the RUL estimation is often provided in intervals. This practice is intended to deal with uncertainties and facilitate the decision-making process based on the RUL estimation. Confidence bounds are a tool that is often implemented to obtain a confidence interval from a RUL estimation. It is a custom threshold that estimates the lower and upper bounds of health index/degradation functions. The built-in Matlab function 'predint' allows the user to compute the confidence bounds of a fitted function with a custom confidence threshold. Fig. 2 represents the confidence bounds of a degradation function example of a jet-engine that shows a exponential degradation behavior. This example is part of the casestudy in section 4 and shows a confidence interval of 95% for the degradation of the jet-engine. In order to provide an alternative option to present the RUL estimation of the proposed SBM, each degradation reference trend in the SBM library is assigned with a confidence interval of 95% leading to lower and upper RUL estimation (respectively RUL lower i and RUL upper i ). The fusion rule is applied on each bound estimation from each trend in M i of a training unit i in order to estimate the RUL as follows:

RUL w RUL w w w w ¯. , ¯1, lower i i lower i i i i k k i = = = (18) RUL w RUL w w w w ¯. , ¯1, ūpper i i upper i i i i k k i = = = (19)
where w i are the same weight than from Fig. 17, RUL lower i and RUL upper i are respectively the lower and the upper final RUL estimations.

Fusion rule for RUL estimation

The final RUL is estimated using a fusion rule that takes into consideration the contribution of individual RUL i that are obtained from the comparison of the current degradation of a system under study and the retrieved degradation reference trends of the SBM library.

The proposed fusion rule is based on the principle that data acquired from the tested units with the SBM can be used to improve the fusion rule. The number of the most similar Retrieved Reference Degradation Trends (RRDT) to be considered to compute the individual RUL i changes depending on the accuracy of the final RUL estimation on previous tested units.

Given a SBM library with n reference trends and a testing unit j at a specific lifespan point with known true RUL:

1. The final RUL j of the testing unit j is computed using as number of RRDT all possible options from 1 until n. 2. Each estimated final RUL j is assessed against the true RUL using a performance indicator (see Fig. 3.5). 3. Best number of RRDT is obtained from best performance of RUL j against the true RUL, and it is selected for the next testing unit. 4. The testing unit with known true RUL is stored in the SBM library to improve the reference trends records. 5. Any time a testing unit reaches its end of life and its true RUL information becomes available it can be used to update the number of RRDT following the previous steps.

It is important to notice that the number of RRDT to be considered for the final RUL estimation can benefit of the acquired knowledge from tested units only when they have been run-tofailure and the true RUL becomes available. The number of RRDT will be updated once all this knowledge is assessed and incorporated into the SBM library. If the tested units are replaced before failure, the knowledge is incomplete and cannot be used to update the number of RRDT. For the validation of the proposed fusion rule, two scenarios are being considered:

1. With the assumption that the tested units do not reach the endof-life, then the number of RRDT remains static from the first proposal. 2. With the assumption that the tested units reach the end-of-life and the true RUL is available. The accumulated knowledge is used to reevaluate and refine the number of RRDT.

The main advantage of this fusion rule is a reduction of human interaction in the RUL estimation. It is an automatic approach that self-adapts to any application and can benefit from accumulated knowledge acquired over the SBM operation.

Prognostics evaluation

In order to assess the impact of the KPCA for the degradation reference trends generation, and the self-adaptive fusion rule, the Remaining Useful Life (RUL) should be assessed. A first step to perform the prognostics evaluation is by comparing the estimated RUL against the true RUL. For a given testing unit j, this comparison can be estimated as follows:

RUL RUL Estimated True j j j = (20)
where Δ j is the difference between the estimated and the true RUL of a testing unit j (i.e. the error in number of cycles, see Fig. 3). This comparison can also be performed if the RUL estimation is performed using a confidence interval instead of a single RUL value. The upper and lower bounds RUL create a confidence interval. If the true RUL belongs to this interval, the estimation is considered to be true. Hence, Fig. 20 This comparison between the estimated RUL and the true rule can be used in different indicators calculation so that the performance of the SBM can be assessed. One example of these performance indicators can be found in Saxena et al. (2008a), in which a custom score that provides an absolute linear penalty for RUL estimations (see Fig. 3). It is computed as follows:

score S S e e , 1, 0 1, 0 k j j j j b j 2 j b 1 = = > (22)
where S i is the computed score for the tested unit i, b 1 and b 2 are both determined constants according to the required system criteria.

In Saxena et al. (2008a) these constants are set as b 1 = 13 and b 2 = 10, for the case study used in this paper. In the current paper, the same values were used. This performance indicator will be referred to as PHM08 score as it was original used to assess the results in the Prognostics and Health Management (PHM) prognostics challenge in 2008. The PHM08 score has been adopted as it is the performance indicator that has been mostly used in the case study used in the current research (see Fig. 4). The Root Mean Square Error (RMSE) is also a performance indicator that is often used to assess prediction models for RUL estimation. It is less restrictive than the PHM08 score for the estimations that are to the true RUL but as the RMSE uses an exponential penalty, the error is higher than the one obtained from the PHM08 score for estimations that are far from the true RUL (see Fig. 3). The RMSE is calculated as follows:

RMSE N 1 j N j 1 2 = = (23)
where N is the number of evaluated samples.

The mean error indicator is also used to have an overview of the RUL estimation on the database. This traditional error provides a global overview of the testing phase. As the mean error relies on the True RUL for its computation is not simple to plot the penalty behavior as for the RMSE and the PHM08 score.

Mean error

N RUL

1 True j N j j 1 = = (24)
A prognostic approach could provide good results with a specific performance indicator, and bad results with another one. Therefore, having good results using multiple performance indicators simultaneously is a challenge. It improves the approach validation. For this research effects, the three performance indicators are then used in the validation of the proposed approach as they provide a wider overview of the its performance. Using only one of these indicators may not be representative to evaluate the results.

Case study

Overview

To illustrate the proposed approach, a case study using the Commercial Modular Aero-Propulsion System Simulation (CMAPSS) is proposed (Saxena et al., 2008a;Saxena et al., 2008b). This case study is a NASA benchmark and is composed of simulated run-tofailure data of commercial jet engines and has been widely used in academic research to test predictive maintenance models. CMAPSS is a tool for the simulation of a realistic large commercial turbofan engine (Fig. 4) for the 90,000 lb thrust class. Thanks to editable input parameters, it is possible to specify operational profile, closed-loop controllers, environmental conditions such as altitude, etc. Furthermore, various degradation profiles can be managed in different components of the engine.

The CMAPSS case study available in the NASA repository NASA (NASA, 2007) is composed of five data sets (see Table 1), one of which for the 2008 Prognostic Health Management (PHM08) challenge. In these data sets, the simulated engines have one or six operational conditions driven by engine control settings (Altitude, Mach number and Throttle Resolver Angle) and one or two fault modes. Each engine has different initial wear, degradation rate, the exact moment of first use is not known and in addition, all data have process and measurement noises. In the current study, there are three datasets with one fault and two with two faults (see Table 1).

Each dataset consists of multi-variable time series, divided into a training and a testing file. Each data set contains three input variables representing the control/environment settings and 21 outputs sensors (see Table 2).

Case study adaptation

For this research study, only the three data sets with one failure mode out of the five available ones have been selected: FD001, FD002 and FD005. Each CMAPSS data set is composed of two main files, the first one is intended for training of prognostic models and the second one for testing. The training file includes several engines that have been run-to-failure, meaning that the true RUL for each engine cycle is known. For the testing file the engines have not been run-to-failure and their true RUL is not provided. This testing file cannot be directly used to test prognostic models. To overcome this limitation and obtain faster results the training file has been split in two parts: 80% will remain as training data and 20% will be used for testing with a known RUL value.

The extracted testing subset (i.e. the 20% of the original training file) has been adapted in order to have multiple engine data records with known RUL that can be used for the SBM validation. For each engine in the testing subset, partial sections of the lifetime records have been extracted. These sections cover the 40%, 50%, 60%, 70%, 80%, and 90% of the total life time records available for each engine in the extracted testing subset. For example, considering an engine with a run-to-failure life of 100 cycles: a life time records percentage of 40% will contain the 40 first cycles of the engine and the true RUL for this engine would be 60 cycles. This adaptation of the testing subset will also allow to evaluate how the proposed SBM behaves with different amounts of data records.

Results and discussion

Physical degradation reference trends generation

For the implementation of the proposed approach on the CMAPSS case study, the feature selection is performed by using variance analysis with a threshold of 0.001; this aims at removing sensors with constant or binary values that do not provide much information for RUL estimation (Li et al., 2018). All sensors with a variance below or equal to this threshold are discarded. From the list of 21 sensors (see Table 2), the number 1, 5, 6, 10, 16, 18, and 19 are removed. The remaining 14 sensors are used as input for the KPCA feature reduction after a re-scaling in the interval [0,1] for the minimal and maximal values of each variable. The 14 input sensors result in 6 principal components after performing the KPCA. Fig. 5 shows the individual and the cumulative contribution of these six main principal components for datasets FD001 and FD002.

For the training phase of the SBM, the KPCA is used on each engine and on each operational mode to obtain the degradation reference trends. These degradation reference trends are stored in the SBM library. As a custom threshold in this research, it has been decided to keep 85% of the original sensor data to compute the health indexes of the engine. This threshold is motivated by observations in the literature such as [START_REF] Feng | A kernel principal component analysis-based degradation model and remaining useful life estimation for the turbofan engine[END_REF] on the same case study and [START_REF] Jiang | Weighted kernel principal component analysis based on probability density estimation and moving window and its application in nonlinear chemical process monitoring[END_REF] on a nonlinear chemical process monitoring. It means that based on the information provided in Fig. 5 and using Fig. 13 with CCP th = 85%, the four first principal components are selected. Then, the selected principal components are fused using Fig. 14 to obtain the degradation reference trends (i.e. the Health Indexes (HIs)). These reference trends are first of all obtained as series of points. To facilitate their analysis, these series of points go through a curve fitting process. It allows to manage noise and to have smoothed degradation trends. For the CMAPSS case study, the degradation trends are fitted using a generic exponential regression model, inspired by the fitting model used by Saxena et al. (2008a):

f t a e ( ) b t c = + (25)
where a, b and c are constants. These constants are obtained by a regression process on the reference trend of a unit. The resulting curve is the HI curve of the unit.

For each testing unit, the data processing must be done following the same way as for the training instances of the SBM. A degradation trend of the testing unit is obtained from the same four principal components and using Fig. 14 and Fig. 25. Once the degradation trend of the testing unit has been obtained, the degree of similarity between the testing engine and each reference degradation trend (from SBM library) is estimated with Fig. 16. The RUL is estimated using equation Fig. 17 for a single value RUL, or using Fig. 18 and Fig. 19 for a confidence interval of RUL. For the validation of the Proposed Approach (PA) of a SBM for RUL estimation, a series of tests are performed and the results are compared against those of a Reference Approach (RA) from [START_REF] Wang | A similarity-based prognostics approach for Remaining Useful Life estimation of engineered systems[END_REF]. It is important to note that the reference approach was the winner at the PHM08 contest (Saxena et al., 2008a), and provides accurate results on RUL computation with an explainable similarity-based approach on the CMAPSS case study. Other approaches have been developed to address the same case study that outperform the Reference Approach but are inspired in Machine Learning techniques whose reasoning cannot be explained. As the proposed approach is oriented to explainable solutions, Machine Learning techniques are not been considered. The main goal of the comparison between the PA and the RA is to demonstrate that the PA provides benefits at computing RUL of technical systems with an explainable reasoning compared to other approaches in the same conditions. Explainable reasoning is still a crutial point for the implementation of predictive maintenance in complex systems such as aircraft. The RA implements a SBM for RUL estimation of aircraft jet engines (see Fig. 4). The main differences between the PA and RA are in the way the HI is generated and the fusion rule for the RUL estimation. The RA performs a manual process for feature selection. A linear regression model is used to merge the selected features into a consolidated degradation reference trend for each training unit. For the fusion rule, the RA uses the contribution of almost all possible individual RUL i that can be obtained from comparing a testing unit with the reference degradation trends in the SBM library. Some non-relevant individual RUL i are discarded manually using an expert rule. However, as the authors of the RA point out, the expert rules they implement for the final RUL calculation are highly application dependent. It means that the rule to fuse the individual RUL i should not be directly used for a different application, some expert knowledge is needed for its adaptation. The RA came out best in the 2008 Prognostic and Health Management challenge (Saxena et al., 2008a). A little adaptation of RA is done in the current study as it turns out difficult to replicate the expert rule the RA used for deleting non-representative reference degradation trends for the fusion rule. In the context of this research, the RA uses all possible individual RUL i without deleting the non-representative reference degradation trends.

Remaining Useful Life (RUL) estimation using the PA and the RA is assessed using the performance indicators of Section 3.5. A performance gain is used to study the benefits of the proposed approach implementation. A performance gain is a percentage that shows a relative improvement compared to the reference approach. A performance gain of 100% means an ideal improvement in which the error has been reduced to 0 and the estimated RUL is equal to the true RUL in all the tested engines. The performance gain is computed as follows:

PG PA RA 1 100% = × (26)
If PG > 0, it means the PA performed better than the RA. If PG < 0, it means the PA does not improve results compared to the RA.

KPCA contribution assessment

the assessment of the KPCA contribution for HI generation, a comparison of the RUL estimations using the PA and the RA is performed. The fusion rule of RA is used for both PA and RA. This is to measure the actual contribution of the KPCA without any influence of the fusion rule. Table 3 summarizes the PHM08 score for RUL estimation using the proposed approach and the reference approach on the testing subset introduced in Section 4.2. The proposed approach improves significantly the performance in RUL estimation as can be seen on the performance gain percentage in Table 3. The results of the proposed approach always outperformed those of the reference approach. The performance was also evaluated using the RMSE and the mean error with similar results as for the PHM08 score.

Complete approach contribution assessment

The complete proposed approach of a SBM for RUL estimation includes the use of the KPCA for HI generation and a fusion rule for the RUL estimation that can benefit from the acquired knowledge during the SBM operation to improve the RUL estimation. For the proposed fusion rule assessment, two different scenarios are being considered. The first scenario assumes that the testing units do not reach the end of life before they are taken out from operation, as it can happen because of a maintenance policy. In this first scenario the accumulated knowledge over the SBM operation can not be used for to improve the fusion rule and the number of Retrieved Reference Degradation Trends (RRDT) to compute the individual RUL i remains the same as it was initialized (see Fig. 3.4). The second scenario assumes that the testing units do reach their end of life and the true RUL of the testing units become available. The information from the tested units is used to update the number of RRDT for the fusion rule. For the complete assessment of the proposed approach, three datasets of CMAPSS have been selected: FD001, FD002 and FD005. These datasets have been selected because they present one single failure mode at different operational modes. Tables 4, 5 and 6 shows the assessment of the fusion rule of the proposed approach considering the first scenario, and comparing the results against the reference approach using three performance indicators: PHM08 score, RMSE, and mean error correspondingly. The results of the proposed approach outperformed those of the reference approach independently of the performance indicators for the three selected datasets.

It is important to point out that the proposed approach needs fewer data records to have higher performance in the RUL estimation. For example with the FD001 dataset, it can be noticed for example that the PHM08 score of the PA for a lifetime at 40% is better than the RA with a lifetime at 60%. A normal behavior in SBM for RUL estimation is that they have better results when more records are available to perform the similarity analysis between the testing units and the reference degradation trends in the SBM library. The need of fewer records to achieve better results is an evidence of the benefits of the proposed approach over the reference approach; accurate results of the RUL will be known earlier and this will lead to a better decision making process in a maintenance department. Overall, there is better results with the proposed approach for the three databases (i.e. FD001, FD002, and FD005). The "RMSE" and "Mean error" show as well homogeneous improvements of the PA over the RA on all studied datasets (see Table 5 andTable 6). The minimum performance gain is around 13%, for the FD002 database.

In interesting behavoir can be observed for FD005 in Table 4, when the lifetime records percentage increases, the PHM08 score sometimes also increases which means a RUL estimation error increment. For example, the performance of the proposed approach is lower at 50% of the lifetime records than at 40% using the PHM08 score. This is not an expected behavior as prognostics tend to be more precise with more data to be assessed. Having a closer look on the RUL estimation for the engines in the FD005 dataset, the estimated value was often higher than the true RUL which is highly penalized by the PHM08 score. This can provide an explanation of lower performances with more percentage of lifetime records for RUL estimation in Table 4. This behavior is not visible using the RMSE and the mean error as performance indicators (see Table 5 andTable 6) as they are based on an average value. The performance indicator choice has a high impact on the approach evaluation.

Table 7 summarizes the results of the proposed approach using the second scenario for the fusion RUL and the PHM08 score as performance indicator. The overall improvement on the RUL estimation is at least 38% on the three datasets used. This is not only a improvement compared to the reference approach but also an improvement compared to the first scenario of the proposed approach. Similar results where also observed using the RMSE and the mean error as performance indicator. Comparing the results in Tables 4 and7, it can be seen the second scenario outperforms the benefits of the of the first scenario of the proposed approach. It means that the more available data to estimate the number of RRDT for the fusion rule, the better results are obtained.

Proposed approach assessment using a confidence interval

Table 8 summarizes the PHM08 score at a confidence of 95% for the three databases. As can be expected, the performance indicator improves (PHM08 score decreases) since providing a RUL interval instead of a single value increases the chances to estimate closer values to the true RUL. For the three databases, a confidence threshold of 95% seems to improve the RUL estimation compared to an approach without confidence bounds, but the results have to be analyzed carefully. By decreasing the confidence threshold, the score decreases as well. Fig. 6 shows the score versus the evolution of the confidence threshold for the RUL estimation on FD001 sliced at 40% can be observed. The blue line represents the score for the proposed approach without the use of confidence interval. It results in a constant curve. The red line represents the score of the proposed approach with the use of various values of confidence thresholds. With a confidence interval of 100%, the proposed approach with confidence bounds provides the same result than without confidence bounds (single value RUL estimation represented by a blue 6). With a confidence threshold of 0.95, it leads to a confidence interval of 95% on the RUL estimation. The score is then evaluated using Fig. 21 and Fig. 22, which resulting into a single value (i.e. no confidence interval on results that are from a confidence interval). Even if the performance indicator seems to be improving while decreasing the confidence threshold, it does not mean it is advisable. Fig. 6 shows the perverted effect of the confidence threshold on the RUL estimation and the score. The lower the confidence threshold is, the bigger the RUL estimation interval is and the bigger the uncertainty is when computing the RUL. A different methodology is needed in order to evaluate the performance of the RUL estimation using confidence interval. For this aim, a Percentage of Disparity (PoD) that can evaluate the confidence interval for a given confidence threshold is proposed. The PoD is computed through the quotient between the relative variation and the mean estimation (PoD = RelativeRUL/MeanRUL) and it is an indicator of how dispersed the values are in the confidence interval. Table 9 presents the average range of the relative RUL variation (i.e. difference between the maximum and the minimum RUL estimation, respectively Fig. 18 and Fig. 19), the mean RUL estimation and a Percentage of Disparity (PoD). It can be observed that the PoD and the relative RUL range is different for each database. The mean PoD for FD001, FD002 and FD005 are respectively 5%, 21% and 20%. To analyzed these results a maintenance policy that allows no more than 10% of error on the RUL estimation is assumed. For FD005, a PoD of 20% means the confidence threshold of 95% is not a good choice as it does not meet the maintenance policy. Some modifications would be needed in the confidence threshold to reach the desired error value. The confidence interval can be seen as an allowed error rate for the RUL estimation. A big interval leads to big uncertainty on the RUL. The PoD highlighted this problem with the database FD005. This interval width is directly linked to the confidence threshold. Decreasing the confidence threshold leads to the increase of the RUL estimation interval.

Conclusion

This study proposed two different improvements for similaritybased models for RUL estimation. As a first contribution a health index building method using the Kernel Principal Component Analysis (KPCA) was proposed. It allowed the creation of the degradation reference trends that are stored in the SBM library in a generic way, by using automatic and unsupervised techniques. As a second contribution a novel fusion rule for RUL estimation was proposed. It is a self-adaptive method that can benefit from the accumulated knowledge over the SBM operation.

For the assessment of both contributions different tests where performed using the CMAPSS case study and comparing the results against a reference approach that also implements a SBM for RUL estimation but in contrast, it uses several experts rules that are not easily adaptable on different applications. The validation of the proposed SBM aimed at proving that it can outperform other approaches with explainable reasoning. This is an important point for the implementation of predictive maintenance in critical systems such as aircraft.

In a first row of test, the contribution of the KPCA for the HI generation was successfully validated. The results were better than those of the reference approach. In a second row of test the complete approach was validated: the implementation of the KPCA and the proposed fusion rule. Different performance indicators where used for the approach validation: the PHM08 score, the RSME and the mean error. It provides a better idea of how the SBM performs with the proposed approach. For the fusion rule two different scenarios where explored. The first scenario does not consider the accumulated knowledge over the SBM operation. The second scenario benefits from the accumulated knowledge to update the fusion rule. Both scenarios shown better results compared to the reference approach that uses experts rules for the RUL estimation.

As a complementary analysis, the proposed approach was also studied if the RUL estimation should be in confidence interval. The performance indicators used for single value RUL estimation do not provide a good overview of the RUL estimation using confidence bounds. Another performance evaluation was proposed using the percentage of disparity in the confidence interval. Performance of the RUL estimation would rely on maintenance policies and the confidence threshold that should be modified to meet the expected results.

Further work focuses on enhancing the proposed approach so to consider the presence of several failure modes with the same case study, which is currently a limitation of the presented approach. In addition, the proposed SBM for RUL estimation can also be implemented in other case studies. Spontaneous important health index improvements, such as may happen in batteries [START_REF] Zhou | A novel health indicator for on-line lithiumion batteries remaining useful life prediction[END_REF], are not managed by the current approach, and will be addressed in the future.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Fig. 1 .

 1 Fig. 1. Proposed similarity-based model for RUL estimation.

Fig. 2 .

 2 Fig. 2. Confidence interval for the Health Index/degradation function of a jet-engine example.

Fig. 3 .

 3 Fig. 3. Prognostic evaluation metrics.

Fig. 4 .

 4 Fig. 4. Simplified diagram of the 90K engine Saxena et al. (2008a).

Fig. 5 .

 5 Fig. 5. KPCA: individual and cumulative contribution of Principal Components (PCs).

Fig. 6 .

 6 Fig. 6. Confidence threshold influence for RUL estimation on FD001 sliced at 40%.

  Conceptualization, Methodology, Software, Writing -Original Draft, Juan Jose Montero Jimenez: Writing -Review & Editing, Rob Vingerhoeds: Supervision, Writing -Review & Editing, Michel Salaün: Supervision, Writing -Review & Editing

  is modified as follows:

			0	if RUL	lower	RUL	true j	RUL	upper
	j	=	RUL	upper	RUL	lower	if RUL	upper	RUL	true j
			RUL	lower	RUL	upper	if RUL	true j	RUL	lower	(21)

Table 1 C

 1 -MAPSS datasets characteristics.

	Id	Name	Operational	Fault	Failed	Number of
			conditions	Modes	System	Engines
					Part
	#1	FD001	1	1	HPC	100
	#2	FD002	6	1	HPC	260
	#3	FD003	1	2	HPC, Fan	100
	#4	FD004	6	2	HPC, Fan	549
	#5	FD005	6	1	HPC	218
		(PHM08)

Table 2

 2 List of 21 output variables from CMAPSS tool.

	Sensor id	Symbol	Description	Units
	1	T2	Total temperature at fan inlet	∘ R
	2	T24	Total temperature at LPC outlet	∘ R
	3	T30	Total temperature at HPC outlet	∘ R
	4	T50	Total temperature at LPT outlet	∘ R
	5	P2	Pressure at fan inlet	psia
	6	P15	Total pressure in bypass-duct	psia
	7	P30	Total pressure at HPC outlet	psia
	8	Nf	Physical fan speed	rpm
	9	Nc	Physical core speed	rpm
	10	epr	Engine pressure ratio (P50/P2)	-
	11	Ps30	Static pressure at HPC outlet	psia
	12	Phi	Ratio of fuel flow to Ps30	pps/psi
	13	NRf	Corrected fan speed	rpm
	14	NRc	Corrected core speed	rpm
	15	BPR	Bypass Ratio	-
	16	farB	Burner fuel-air ratio	-
	17	htBleed	Bleed Enthalpy	-
	18	Nf_dmd	Demanded fan speed	rpm
	19	PCNfR_dmd	Demanded corrected fan speed	rpm
	20	W31	HPT coolant bleed	lbm/s
	21	W32	LPT coolant bleed	lbm/s

Table 3

 3 Performance gain of implementing the KPCA for Health Index generation using testing units at different lifetime percentage.

	Lifetime	Proposed Approach	Reference	Performance
	records	performance with the	approach	Gain (%)
	(%)	RA fusion rule	performance
		(PHM08 score)	(PHM08 score)
	40	17299	117648	85.30
	50	7080	78723	91.01
	60	2117	10184	79.21
	70	642	1136	43.48
	80	503	653	22.97
	90	1639	6370	74.27

Table 4

 4 RUL estimation assessment using the first scenario of the proposed fusion rule and the PHM08 score as a performance indicator.

	LRP (%)	FD001			FD002			FD005
		PA (PHM08 score)	RA (PHM08 score)	PG	PA (PHM08 score)	RA (PHM08 score)	PG	PA (PHM08 score)	RA (PHM08 score)	PG
	40	2 259	39 041	94.21%	13 521	15 512	12.83%	6 561	12 361	46.92%
	50	489	21 710	97.75%	7 874	11 349	30.62%	10 180	16 084	36.70%
	60	235	5 303	95.57%	5 522	9 996	44.76%	3 551	20 883	83.00%
	70	137	1 385	90.13%	5 049	5 955	15.22%	4 018	19 537	79.43%
	80	103	1 435	92.85%	3 060	3 760	18.62%	2 495	18 712	86.67%
	90	91	3 898	97.66%	1 364	2 770	50.78%	1 776	12 352	85.62%
	LRP: Lifetime Records Percentage, PA: Proposed Approach, RA: Reference Approach, PG: Performance Gain

LRP: Lifetime Records Percentage, RMSE: Root Mean Square Error, PA: Proposed Approach, RA: Reference Approach, PG: Performance Gain line in Fig.

Table 6

 6 RUL estimation assessment using the first scenario of the proposed fusion rule and the mean error percentage as a performance indicator.

	LRP (%)	FD001			FD002			FD005
		PA (mean error %)	RA (mean error %)	PG	PA (mean error %)	RA (mean error %)	PG	PA (mean error %)	RA (mean error %)	PG
	40	28.47	32.96	13.60%	27.71	31.95	13.26%	26.83	31.57	15.02%
	50	27.72	38.07	27.19%	31.09	37.28	16.60%	30.71	37.84	18.84%
	60	25.10	45.00	44.22%	36.64	44.53	17.71%	32.28	47.75	32.39%
	70	28.05	52.70	46.79%	40.09	52.99	24.34%	42.64	62.78	32.08%
	80	33.96	64.03	46.97%	52.33	64.06	18.32%	53.88	91.30	40.99%
	90	55.35	183.29	69.80%	94.16	128.31	26.62%	97.84	201.75	51.50%
	LRP: Lifetime Records Percentage, PA: Proposed Approach, RA: Reference Approach, PG: Performance Gain

Table 7

 7 RUL estimation assessment using the second scenario of the proposed fusion rule and the PHM08 score as a performance indicator.

	LRP (%)	FD001			FD002			FD005
		PA (PHM08 score)	RA (PHM08 score)	PG	PA (PHM08 score)	RA (PHM08 score)	PG	PA (PHM08 score)	RA (PHM08 score)	PG
	40	1 140	39 041	97.08%	7 135	15 512	54.00%	5 428	12 361	56.09%
	50	300	21 710	98.62%	6 123	11 349	46.05%	3 688	16 084	77.07%
	60	129	5 303	97.56%	4 124	9 996	58.74%	2 267	20 883	89.14%
	70	31	1 385	97.78%	3 614	5 955	39.31%	1 259	19 537	93.55%
	80	55	1 435	96.18%	2 332	3 760	37.97%	309	18 712	98.35%
	90	61	3 898	98.43%	784	2 770	71.72%	956	12 352	92.26%
	LRP: Lifetime Records Percentage, PA: Proposed Approach, RA: Reference Approach, PG: Performance Gain

Table 8

 8 RUL estimation comparison between the PA and PA-CI for FD001, FD002, and FD005 databases using a PHM08 score as a performance indicator.

	40	1 932	2 259	14.48% 6 132	13 521	54.65% 3 188	6 561	51.41%
	50	412	489	15.73% 3 956	7 874	49.76% 4 455	10 180	56.24%
	60	197	235	16.27% 2 769	5 522	49.86% 1 762	3 551	50.37%
	70	112	137	17.81% 2 573	5 049	49.03% 1 780	4 018	55.69%
	80	84	103	17.97% 1 494	3 060	51.18%	1 265	2 495	49.29%
	90	76	91	16.62% 785	1 364	42.43% 903	1 776	49.14%

LRP (%) FD001 FD002 FD005 PA-CI (PHM08 score) PA (PHM08 score) PG PA-CI (PHM08 score) PA (PHM08 score) PG PA-CI (PHM08 score) PA (PHM08 score) PG LRP: Lifetime Records Percentage, PA-CI: Proposed Approach with Confidence Interval, PA: Proposed Approach, PG: Performance Gain

Table 9

 9 RUL estimation assessment using a Relative RUL estimation method and a Mean RUL estimation method for FD001, FD002, and FD005 databases and using a percentage of disparity as a performance indicator. Lifetime Records Percentage, M-RUL: Mean RUL estimation score, R-RUL: Relative RUL score, PoD: Percentage of Disparity

	LRP (%) FD001			FD002			FD005
		M-RUL (PHM08	R-RUL (PHM08	PoD	M-RUL (PHM08	R-RUL (PHM08	PoD	M-RUL (PHM08	R-RUL (PHM08	PoD
		score)	score)		score)	score)		score)	score)
	40	129	4	2.84% 127	15	11.57% 135	16	11.77%
	50	110	4	3.40% 112	15	13.66% 116	15	12.86%
	60	88	4	4.23% 95	17	18.00% 97	16	16.26%
	70	67	4	5.64% 75	17	22.13% 79	18	22.59%
	80	48	4	7.29% 57	16	28.89% 57	15	25.95%
	90	31	3	8.13% 39	13	32.73% 39	11	29.82%
	LRP:


Maxime Prieur

Cédric Du Mouza
email: cedric.dumouza@lecnam.net

Guillaume Gadek
email: guillaume.gadek@airbus.com

Bruno Grilheres

Evaluating and Improving End-to-End Systems for Knowledge Base Population

Keywords: Knowledge Base Population, Entity Linking, Supervised Learning, Data Mining, Method and Evaluation

Knowledge Bases (KB) are used in many fields, such as business intelligence or user assistance. They aggregate knowledge that can be exploited by computers to help decision making by providing better visualization or predicting new relations. However, their building remains complex for an expert who has to extract and link each new information. In this paper, we describe an entity-centric method for evaluating an end-to-end Knowledge Base Population system. This evaluation is applied to ELROND, a complete system designed as a workflow composed of 4 modules (Named Entity Recognition, Coreference Resolution, Relation Extraction and Entity Linking) and MERIT, a dynamic entity linking model made of a textual encoder to retrieve similar entities and a classifier.

INTRODUCTION

Knowledge bases (KB) are data structures, generally relying on a predefined ontology, which are very useful for aggregating information in order to simplify its visualization and analysis. Knowledge bases are used in many fields such as business intelligence [START_REF] Shue | The development of an ontology-based expert system for corporate financial rating[END_REF] to improve decision making or to extract elements linking scientific publications [START_REF] Ammar | Construction of the literature graph in semantic scholar[END_REF]. However, manual construction and updating of Knowledge Bases are extremely costly since the domains in which they are deployed usually exploit constantly evolving information. An alternative solution would be an automatic population that would extract and add the desired elements from sources of interest into the knowledge base.

Nevertheless, the existing solutions are not yet sufficient and this subject is still the focus of many workshops [START_REF] Ghosal | Report on the 1st workshop on argumentation knowledge graphs (argkg 2021) at akbc 2021[END_REF], which aim at proposing new approaches either on the end-to-end process workflow or on one of its subtasks. Most proposals on the literature focus on the optimization of one or two particular modules but not of an end-to-end system. However, we have no guarantee to get the best performances for an end-to-end system when trying to optimize each module separately. Unfortunately, to optimize an end-to-end Knowledge Base Population (KBP) system as a whole, there exists, to the best of our knowledge, no evaluation protocol that are both automatic and exhaustive [START_REF] Min | When ace met kbp: End-to-end evaluation of knowledge base population with component-level annotation[END_REF][START_REF] Mesquita | Knowledgenet: A benchmark dataset for knowledge base population[END_REF].

In this paper, we attempt to address this issue with the following contributions:

• we formalize a method for evaluating end-to-end Knowledge Base Population systems from texts as a whole;

• we present ELROND, an end-to-end system implemented as a 4-step processing workflow whick could be considered as a baseline for comparison with future solutions;

• as an improvement of ELROND baseline, we introduce MERIT, a textual encoder-based entity linking solution for entity resolution when building a dynamic knowledge base;

• we measure and compare the performances of the proposed models by following the presented evaluation method and applying it to the DWIE dataset. [START_REF] Zaporojets | Dwie: An entity-centric dataset for multi-task document-level information extraction[END_REF].

The remaining of this article is structured as follows: after a presentation of recent work on knowledge base population (KBP) evaluation approaches, on end-to-end systems as well as on models for the entity linking task (Section 2), we formalize and detail our evaluation protocol (Section 3). Then we describe our end-to-end implementation proposal, EL-ROND, and our linking solution, MERIT (Section 4).

We present the experiments conducted and the results obtained in Section 5 before discussing the possible perspectives to explore in future work.

RELATED WORK

Knowledge Base Population

Knowledge Base Population from texts consists in extracting the elements and their relations of interest in order to add them to the already known and structured information. This task usually involves several steps: named entity recognition (NER), coreference resolution, relation extraction, and entity linking. Tinker-Bell (Al-Badrashiny et al., 2017), one of the first endto-end systems, consists of a NER module, combining two Bi-LSTMs taking respectively the text and the linguistic features to tag the words in the document. Meanwhile, entity linking is solved through the sum of popularity, similarity and consistency scores. KnowledgeNet [START_REF] Mesquita | Knowledgenet: A benchmark dataset for knowledge base population[END_REF] also addresses relation extraction with a Bi-LSTM (Long Short-Term Memory) [START_REF] Huang | Bidirectional lstm-crf models for sequence tagging[END_REF] model which receives linguistic features and embeddings generated by a BERT (Bidirectional Encoder Representations from Transformers) model (Kenton and Toutanova, 2019). This model is however applied at the sentence level and discards supra-phrastic relations [START_REF] Yao | Docred: A large-scale document-level relation extraction dataset[END_REF]. Moreover these approaches are based on Wikipedia and are not adequate when the sources have a large proportion of entities not listed in the encyclopedia. KBPearl [START_REF] Lin | Kbpearl: a knowledge base population system supported by joint entity and relation linking[END_REF] suggests to use open information extraction (OIE) frameworks. The system extracts and links knowledge from the text using a graph densification method applied to the semantic graph built from the text. In addition to a profusion of potentially uninteresting information for the user caused by the lack of predicates in OIE frameworks, the selection of candidates is performed by alias matching, which is not very suitable when new mentions appear.

Entity Linking

Entity linking is the process of determining whether the mentions in a text refer to entities in the database. This process generally establishes a list of similar entities before selecting, or not, one of the candidates. In addition to dictionary-based approaches [START_REF] Al-Badrashiny | Tinkerbell: Cross-lingual cold-start knowledge base construction[END_REF], encoder-based methods have been proposed. For instance BLINK [START_REF] Wu | Scalable zero-shot entity linking with dense entity retrieval[END_REF], uses two BERT models to compare entity mentions and Wikipedia descriptions projected into a sin-gle representation space. This bi-encoder returns descriptions similar to the input mention before being re-ranked by a more fine-grained encoder. On the other hand, the list of candidates is not entirely rejected if the entry does not correspond to any entity in the database (NIL prediction). To overcome this issue, [START_REF] Zhang | Entqa: Entity linking as question answering[END_REF] applies a Q&A approach, returning the entities likely to be in the document before classifying whether a textual segment mentions a candidate. The NIL classification is only partially solved since they cannot add the description of a new entity to predict it later. In addition, Blink and En-tQA use three BERT encoders with separate weights which makes the system quite cumbersome.

KBP Evaluation

While there are numerous benchmarks and metrics for the evaluation of subtasks ("F1-score" for REN and relation extraction, "Hit@k" for linking, etc), few solutions exist for end-to-end systems that build knowledge bases from text. The TAC KBP workshops evaluate a system by computing the accuracy on 1-hop queries, "What is Frodo carrying?", and 2hop queries, "Who created what Frodo is carrying?", the number of hops representing the number of relations which separate the subject entity from the object entity. The cost of manually evaluating systems that return a large number of responses [START_REF] Ellis | Overview of linguistic resources for the tac kbp 2015 evaluations: Methodologies and results[END_REF] compels evaluators to focus on a small number of queries and thus they do not evaluate the entire database. [START_REF] Min | When ace met kbp: End-to-end evaluation of knowledge base population with component-level annotation[END_REF] makes automatic evaluation feasible by measuring the alignment of triplets (subject, relation, object) between the reference and the output. An output entity is linked to a reference one if the produced entity shares more than 50% of the mentions with the reference entity. This alignment raises the questions of the situation where the system would only extract a small number of mentions and also the arbitrary choice of the threshold at 50%. KnowledgeNet [START_REF] Mesquita | Knowledgenet: A benchmark dataset for knowledge base population[END_REF] measures the F1 score on the extraction of annotated triples in sentences and the linking of the subject and object entity pair to their Wikidata page. Since each sentence in the dataset annotates only one pair of entity and one relation, it is not possible to properly evaluate the accuracy since results could be mistakenly considered as false positives. Like [START_REF] Min | When ace met kbp: End-to-end evaluation of knowledge base population with component-level annotation[END_REF], the evaluation is done at the textual level and discards the construction of a base. These incomplete evaluation methods highlight the need for a protocol that evaluates the performance of an entire KBP system.

MODELING AND EVALUATION

Modeling a Knowledge Base

A KB is composed of elements (entities, attributes and relations between them) relying on a defined ontology. It can thus be modeled by a graph in which the nodes are the various elements and the edges express the existence of a relation between these elements. We define a KB as follows: Definition 3.1 (Knowledge Base). A Knowledge Base is a data structure that can be modeled by a graph G = (V, E, Φ, Ψ) where V is the set of vertices in the graph, E the set of edges between two nodes of V , Φ : V → A is a function which for any vertex v i of V assigns a set of attributes A i ∈ A denoted by tuples (type, value) and Ψ : E → E a function that associates to each edge e i ∈ E an edge type E i ∈ E, with A and E referring to the set of attributes and the set of edge types respectively.

Populating a KB with textual content consists therefore in adding elements extracted from texts according to an ontology. To link information related to the same entity found in several texts, the entities must have a unique identifier (URI). This allows to obtain for a set of k texts, a reference KB,

G k = (V k , E k , Φ k , Ψ k )
and to measure the proportion of information correctly extracted by a system building a base

G ′ k = (V ′ k , E ′ k , Φ ′ k , Ψ ′ k ).
Example of workflow for the KBP task. A KBP system is built around components or modules that form the processing workflow to solve the KBP task. The first component is in charge of recognizing named entities (NER) and other elements of interest in the text (attributes, unnamed entities) while assigning them a type using the document. For instance with the sentence "Joe Biden, the U.S. President, went viral on Trump", the step yields [(Joe Biden, Per), (the U.S. President, Per), (President, Role), (U.S., Nationality), (Trump, Per)]. The second processing block groups the textual elements that co-reference. The mentions from the NER belonging to a cluster composed of mentions of the same type are kept in coreference and the remaining mentions are considered as different entities. Using the previous example, we get two clusters [(1, PER, Joe Biden, the U.S President), (2, PER, Trump)]. It is then possible to identify, by a third module, the relations linking the elements [(1, Is against, 2), (1, Role, President), (1, Nationality, U.S.)]. These relations, in addition to being part of the information to be extracted, constitute a support for the last step, the entity resolution. Each entity in the text, when it is possible, is associated with an entity in the database. "Trump" must be linked to the entity "Donald Trump" and not to "Fred Trump". Finally, all the information extracted from the text enriches the information in the database by completing those of the entities already known or by adding new entities.

Evaluating KBP Systems

In order to evaluate the systems designed for the KBP task, we present below a process to measure the performance of the methods which extract and aggregate information either to an existing base (warm start) or to an initially empty base (cold start).

Entities are defined by attributes and relations which link them to other entities in the database. When comparing a reference entity to a built entity, it is necessary to verify that both the attributes and the relation match.

Definition 3.2 (Similarity of attributes and relations).

We use the following similarity definitions for attributes and relations:

• Attribute similarity: we consider that 2 attributes match if they have the same type, value and inference text (in which the attribute appears). Although including the inference text creates a multiplication of information, it verifies that the system correctly extracts the information each time it is mentioned. • Relation similarity: we consider that 2 relations are similar if they involve the same predicate (type of relation), the same inference text and that all the mentions of the object entity of the constructed relation are included in the mentions of the object entity of the reference base.

To check if an entity has been correctly extracted, we compare the extracted attributes and relations with those possessed by the reference entity. In order to measure the proximity, we adapt the precision, recall and F1-score based on Definition 3.2 as follows: Entity alignment. We align each entity of the reference KB with an entity of the output base using the F1 score defined above and the Hungarian algorithm [START_REF] Kuhn | The hungarian method for the assignment problem[END_REF]. The alignment is possible only for pairs with a non-zero similarity score. G k and G ′ k entities without match are respectively considered as false negatives and false positives. In the warm-start scenario, entities that are initially present remain aligned between G k and G ′ k , their F1-score only takes into account the new information. This matching phase leads to the construction of Ω k a set of pairs

P v i ,v j ,k = α|T P rel | + β|T P att | α(|T P rel | + |FP rel |) + β(|T P att | + |FP att |) R v i ,v j ,k = α|T P rel | + β|T P att | α(|T P rel | + |FN rel |) + β(|T P att | + |FN att |) F1 v i ,v j ,k = 2 P v i ,v j ,k × R v i ,v j ,k P v i ,v j ,k + R v i ,v j ,k (
(v i,G k , v j,G ′ k ).
Global quality scores. The comparison between the constructed and the reference bases after proceeding k texts is done by aggregating the similarity scores of the previously formed pairs of entities. Two F1-scores, one F1 micro and one F1 macro measuring the proportion of correctly extracted information, are computed:

P micro,k = ∑ (v i ,v j )∈Ω k α|e v j = e v i | + β|a v j = a v i | α|E G ′ k | + β|A G ′ k | R micro,k = ∑ (v i ,v j )∈Ω k α|e v j = e v i | + β|a v j = a v i | α|E G k | + β * |A G k | F1 micro,k = 2 P micro,k × R micro,k P micro,k + R micro,k F1 macro,k = ∑ (v i ,v j )∈Ω k F1 v i ,v j ,k |Ω k | + |FN| + |FP| (2)
With e v i , e v j edges of node i and j and E G k , A G k , respectively the set of edges and attributes in the reference base. The F1 macro is an average of the similarity scores of the aligned entities and does not take into account the difference of distribution (number and type of relations or attributes) that could exist between the entities, unlike the F1 micro . The latter is a weighted F1 calculated according to the identical elements between the aligned entities.

Benefits of the evaluation process. An issue when considering an end-to-end system, which is composed of several modules, is that an error caused by a module can be re-used by another and added to the database. For example, a person mentioned in a text can be linked to the wrong person in the database and thus can lead to an erroneous assignment of a relation or attribute. The proposed protocol measures at different text intervals the distance to the baseline, showing the resilience of a system to errors that can be made. The evaluation can be conducted both in a warm-start and a cold-start scenario. The impact of a single module on the whole processing chain is measured using the ground truth results on the rest of the workflow. The choice to make the F1-score measurement more flexible, by replacing the exact matching of entities by the proportion of identical information in a pair, brings a better representativeness of the systems' performances.

MODELS FOR KBP

This section presents ELROND, a baseline system for the KBP task and MERIT, an entity linking task module which improves the baseline.

ELROND, an End-To-End System for KBP

We introduce ELROND (Entity Linking and Relation extraction On New textual Documents). ELROND is an implementation that follows the KB enrichment process explained in Section 3.1. The main components, with their interactions, are illustrated in Figure 1. This system is used as a baseline and shows the interest of the evaluation method detailed in Section 3.2. For each module we implement a recent proposal found in literature which exhibits good results.

Named Entity Recognition. The NER block consists of a pre-trained and fine-tuned RoBERTa model [START_REF] Liu | Roberta: A robustly optimized BERT pretraining approach[END_REF]. The choice of RoBERTa was motivated by its current performance on various benchmarks (SWAG1 , GLUE2 ) for named entity recognition.

Resolving co-references. For the co-reference task, we use in parallel of the NER model, the pre-trained model Word-level Coreference Resolution (Dobrovolskii, 2021). This model creates, with the help of RoBERTa, groups of words in co-reference (named and unnamed entities, pronouns, etc). The representation of a token is given by weighting the vectors of its sub-tokens produced by RoBERTa. The weights are obtained by applying a softmax function to the projection of the vectors through an attention matrix. Finally, the model predicts a co-reference when the

NER

Coreference Resolution

Relation Extraction

Knowledge Base sum of a bilinear projection between two tokens and the output of a neural network taking the two tokens as input is positive. We choose to integrate Wordlevel Coreference Resolution to ELROND because of its performance on this task.

Relation Extraction. Relations are extracted using the ATLOP model [START_REF] Zhou | Document-level relation extraction with adaptive thresholding and localized context pooling[END_REF] which represents each entity by applying a pooling function on the mention vectors obtained by a PLM (Pre-trained Language Model). For each pair of entities, an attention coefficient is obtained before using it in a bilinear function to compute the plausibility of a relation type. If the score is greater than the Null type, the relation is considered as existing. Relationships that are impossible due to the type of entities are filtered in post-processing.

Entity Linking. The last step applies a search by mention and a selection by popularity. For each entity in the text, the solution returns the entities in the database, of the same type and sharing at least one mention with the entity in the text. In case the mentions do not return any results, an extended search is performed with the acronyms of these mentions. If no element is returned, a the textual entity is added to the database. If several entities of the database correspond to mentions of the textual cluster, a selection by popularity, similar to Al-Badrashiny et al. ( 2017) is applied. The entity with the most occurrences, considering all mentions, is selected.

MERIT: Model for Entity Resolution In Text

Candidate Retriever. The popularity-based linking approach described previously that serves as base-line is not usable for dealing with new mentions and is prone to errors during ambiguity resolution, since it favors popular entities. The proposed method, MERIT, addresses these shortcomings by relying on the context of the documents, drawing inspiration from previous approaches such as Blink [START_REF] Wu | Scalable zero-shot entity linking with dense entity retrieval[END_REF] or EntQA [START_REF] Zhang | Entqa: Entity linking as question answering[END_REF]. Like these models and as illustrated in figure 2, we propose as a first component an encoder that projects a portion of a text targeting an entity to a representation space allowing similarity comparisons. The retriever takes as input a query text in which the target entity mentions are enclosed in tags. This retriver differs from EntQA and BLINK on several points. First, the text samples are expanded to a size of 256 tokens to include a broader context for a better discrimination. Secondly and as illustrated by [START_REF] Soares | Matching the blanks: Distributional similarity for relation learning[END_REF] all mentions of the entity of interest in the text sample are wrapped with special tags ([Ent] and [/Ent]) to improve the quality of representations. The distance between the vector representations of the text samples is measured using cosine similarity instead of a scalar product. In addition to obtaining better performance in our case, [START_REF] Luo | Cosine normalization: Using cosine similarity instead of dot product in neural networks[END_REF] show that this provides a more stable learning. Lastly, a simple encoder replaces the dual-encoder since the search base is no longer composed of descriptions, but with textual portions of the same style as those given in the query. The BERT encoder is replaced by the ALBERT architecture, which is lighter and more efficient for semantic similarity tasks (according to the STS benchmark3 ).

Classifier. The most similar text samples retrieved are then concatenated with the query one. The set of 512 tokens is given to an ALBERT model with a lin-ear layer for classification at the end. If none of the samples are classified as similar, a new entity is created in the base with the query sample as the first support text. In the case where several samples are classified, the one with the higher classification score is selected. The model then returns as output the database entity that corresponds to the request entity (if a match has been established).

EXPERIMENTS

Implementation details

All the proposed approaches are implemented in Python and use the Pytorch library4 . The NER model is trained using the Flair [START_REF] Akbik | Flair: An easy-to-use framework for state-of-the-art nlp[END_REF] framework , the encoders used by MERIT are initialized from the library Hugging Face5 . The MERIT retriever module training uses the contrastive loss [START_REF] Khosla | Supervised contrastive learning[END_REF] which leads to a better clustering of identical elements in a representational space. We use the hard-negative technique [START_REF] Gillick | Learning dense representations for entity retrieval[END_REF] which consists in submitting negative samples considered as ambiguous by the model during training. This process brings a better selection of parameters and makes the model more robust in its predictions. The classification module is trained by the binary crossentropy function. We use the index structure Annoy6 to return elements according to their cosine similarity.

Datasets

For a thorough measurement of systems, the completeness of dataset annotation on all dimensions of the information to be extracted is necessary. We decide therefore to use DWIE [START_REF] Zaporojets | Dwie: An entity-centric dataset for multi-task document-level information extraction[END_REF], the only free dataset complying with this constraint. The dataset is composed of 800 press articles in English. In the 700 training texts and 100 test texts that constitute DWIE, entities are annotated according to a multi-level ontology for a total of about 170 classes and relations. For the KBP task, types and aliases are used as attributes. The 700 training texts constitute the KB to be completed in Warm-start. We also used AIDA [START_REF] Hoffart | Robust disambiguation of named entities in text[END_REF] to compare the linking approch using mentions popularity to MERIT. AIDA consists of 1393 news articles for which entities and their Wikipedia page names are listed when available. For DWIE, entities that are not linked to Wikipedia only appear in a single text, so we can consider them as unique entities. This is not the case for AIDA. Entities in AIDA that do not have an identified link are thus discarded from the linking task.

To estimate the complexity of the linking task on a dataset, we introduce an ambiguity score, χ, computed as the average for each textual mention, of the inverse of the number of entities named by it:

χ(dataset) = ∑ mentions |E(mention)| -1 |mentions| (3)
With E(mention) the set of entities sharing this mention. We obtain a score of χ(DW IE) = 0.889 and χ(AIDA) = 0.794.The entity linking task is therefore more complex for the dataset AIDA.

Since the existence of certain items in a database, and thus the order in which the texts allowing the extraction of these items are submitted, may or may not benefit the operation of the systems, the performance is measured and averaged over 10 different orderings of the test set. The data and scripts to compare these results are available on a git7 repository.

Results and Discussion

ELROND's performance. The graphs in Figure 3 show the score of ELROND for the KBP task on the 100 test texts (average taken over 10 runs). For the warm-start scenario, the original KB is the information contained in the 700 training texts. We observe that the initial distance between the KBs is greater in warm-start due to the difficulty to link the information of the texts with those already possessed. In both cases (although more contrasted in warm-start), the performance declines over the texts, which attests of an accumulation of errors during the process. The micro F1-score is higher than the macro F1-score in both cases and seems more stable over the end of the test set. This is explained by the fact that popular entities (countries and cities for example) which are mentioned more often in the texts have on the one hand more weight in the final database, and on the other hand are easier to recognize. This characteristic will therefore tend to increase the F1 micro compared to the F1 macro which smoothes the difference in distribution between entities.

Comparison of linking approaches. For entity resolution, we use two types of metrics. The Hit@k, that computes the frequency with which the queried entity is found among the first k entities. This metric is only measured on query entities that are in the database. The second metric is accuracy: a result is valid if the entity is in the KB and is returned by the model or if this entity is NIL and the model does not return a result. We add for comparison a linking solution that randomly select an entity in cases of ambiguity. For DWIE, we measure the performance with and without filtering on the type of entities as well as taking into account the relations that exist in the KB. Due to the lack of annotation on AIDA, the performances of the approaches do not those elements. To reduce the prediction of false positives, MERIT retains only the first 10 candidates obtained by similarity search.

All the results are recorded in tables 1 and 2. For similarity retrieval and for both datasets, MERIT gives better results than the mentions popularity approach and shows a benefit when used to propose results during a semi-automatic process. The trend is more nuanced on DWIE when dealing with NIL entities. This is explained by the fact that MERIT applies a classification on more candidates and is therefore more prone to predict false positives, while the mention popularity approach filters out identical mentions and is limited to 1 or 2 candidates. AIDA contains more ambiguity in its texts but with distinct writing styles and contexts between entities sharing mentions. This may explain why MERIT has better results than the mentions popularity module. We observe that re-ranking the returned entities according to relationship consis-tency slightly improves the results. We also studied a combination of the two approaches. This fusion use MERIT with a classification on the most similar candidate and selects the search results by mention popularity in case of negative classification. The results show in this case study a slight interest for this fusion when applied to DWIE, but are finally less interesting on AIDA. The drop on AIDA is explained by the influence of the search by popularity of mentions.

Finally, we compared for the KBP task ELROND, ELROND with the Fusion entity linking solution and the model proposed in the DWIE paper. Since no linking method is given for the latter, we used the mention popularity method. The results presented in table 3 show the interest of the proposed approaches which improve the results by up to 2.2% in the Warm-start scenario, and the complementarity of MERIT with the mention approach.

CONCLUSIONS

In this paper, we have formalized and presented an automatic, complete and scalable evaluation method for the KBP task from texts. It allows to compare and select methods in warm-start and cold-start scenarios. This protocol has been used to measure the performance of ELROND, a system that serves as a first basis for future improvements. We were able to improve

DWIE Model

Hit@1 Hit@10 Hit@50 Accuracy the entity linking module of the ELROND baseline by proposing MERIT, an entity resolution model based on textual encoders and capable of integrating NIL entities during inference. Future work may study the contribution of a larger dataset for training a supervised linking model, extend the elements of interest to unnamed entities, to study linking approaches that would only have a structured KB and the role that ontology can play in KBP systems. We are also working on the production of a French dataset to train and evaluate KBP systems. We plan to extend the presented models to the French language.

  1)With TP, FP and FN for true positives, false positives and false negatives respectively, rel for relation and att for attribute v i and v j vertices belonging to the reference base G k and the built base G ′ k , and 0 ≤ α, β ≤ 1 weights such as α + β = 1, which allow to give a different importance to attributes and relations. Other weights could be added to differentiate along attribute or relation types.

Figure 1 :

 1 Figure 1: Diagram of the proposed processing chain for ELROND.

Figure 2 :

 2 Figure 2: Diagram of MERIT.

Table 1 :

 1 Performance of the linking approaches on the DWIE dataset.

	Unfiltered
	Random Linking	84.7	92.5	92.5	91.9
	Mentions	89.8	93.6	93.6	92.2
	MERIT	91.8	95.4	95.8	93.0
	Fusion	92.1	95.2	95.8	92.6
	Filtered
	Random Linking	88.5	93.1	93.1	93.7
	Mentions	91.5	93.6	93.6	94.6
	MERIT	94.2	96.5	96.7	93.9
	MERIT+Mentions	94.3	96.6	96.7	94.6
	Filter + relations
	MERIT	95.0	96.5	96.7	94.1
	MERIT+Mentions	94.4	96.5	96.7	94.8
				AIDA
	Model	Hit@1 Hit@10 Hit@50 Accuracy
	Random Linking	77.8	92.0	92.5	82.8
	Mentions	77.5	92.6	92.6	82.8
	MERIT	94.6	98.2	98.6	92.1
	Fusion	94.2	97.6	97.9	90.1

Table 2 :

 2 Performance of the linking approaches on the AIDA dataset.

https://paperswithcode.com/sota/ common-sense-reasoning-on-swag

https://gluebenchmark.com/

https://paperswithcode.com/sota/ semantic-textual-similarity-on-sts-benchmark

https://pytorch.org/

https://huggingface.co/

https://github.com/spotify/annoy

[https://github.com/Todaime/KBP]