Zhiguo Zeng

Francesco Di Maio

Enrico Zio

Rui Kang

A hierarchical decision making framework for the assessment of the prediction capability of prognostic methods

Keywords: Prognostic and Health Management (PHM), Remaining Useful Life (RUL), prediction capability, Analytical Hierarchical Process (AHP), fuzzy similarity, feed-forward neural network, hidden semi-Markov model

In Prognostics and Health Management (PHM), the prediction capability of a prognostic method refers to its ability to provide trustable predictions of the Remaining Useful Life (RUL), with the quality characteristics required by the related maintenance decision making. The prediction capability heavily influences the decision maker's attitude towards taking the risk of using the predicted RUL to inform the maintenance decisions. In this paper, a four-layer, top-down, hierarchical decision making framework is proposed to assess the prediction capability of prognostic methods. In the framework, prediction capability is broken down into two criteria (Layer 2), six sub-criteria (Layer 3) and 19 basic sub-criteria (Layer 4). Based on the hierarchical framework, a bottom-up, quantitative approach is developed for the assessment of the prediction capability, using the information and data collected at the Layer-4 basic sub-criteria level. Analytical Hierarchical Process (AHP) is applied for the evaluation and aggregation of the sub-criteria and Support Vector Machine (SVM) is applied to develop a classification-based approach for prediction capability assessment. The framework and quantitative approach are applied on a simulated case study to assess the prediction capabilities of three prognostic methods of literature: fuzzy similarity, feed-forward neural network and hidden semi-Markov model. The results show the feasibility of the practical application of the framework and its quantitative assessment approach, and that the assessed prediction capability can be used to support the selection of the suitable prognostic method for a given application.

Introduction

Prediction capability of a prognostic method refers to its ability to provide trustable predictions of the Remaining Useful Life (RUL), with the quality characteristics required by the related maintenance decision making. In Prognostics and Health Management (PHM), the predicted RUL (either by model-based prognostic methods [START_REF] Pecht | A Prognostics and Health Management Roadmap for Information and Electronics-Rich Systems[END_REF][START_REF] Vichare | Prognostics and Health Management of Electronics. Components and Packaging Technologies[END_REF][START_REF] Wang | Remaining Useful Life Estimation Under Degradation and Shock Damage[END_REF][START_REF] Dalal | Lithium-Ion Battery Life Prognostic Health Management System Using Particle Filtering Framework[END_REF] or data-driven prognostic methods [START_REF] Zhang | Degradation Modeling-Based Remaining Useful Life Estimation: A Review On Approaches for Systems with Heterogeneity[END_REF][START_REF] Tsui | Prognostics and Health Management: A Review on Data Driven Approaches[END_REF][START_REF] Si | Remaining Useful Life Estimation -a Review On the Statistical Data Driven Approaches[END_REF]) is typically used by decision makers to schedule proper and timely maintenance. Usually, the decision makers choose between Condition-Based Maintenance (CBM) policy [START_REF] Peng | Current Status of Machine Prognostics in Condition-Based Maintenance: A Review[END_REF][START_REF] Lipi | A Condition-And Age-Based Replacement Model Using Delay Time Modelling[END_REF] or Preventive Maintenance (PM) policy [START_REF] Yun | A Preventive Replacement Policy Based On System Critical Condition[END_REF]. The choice of maintenance policies, then, depends on the decision makers' attitude to the risk of relying on a predicted RUL to plan maintenance services. Undoubtedly, such attitude is heavily influenced by the prediction capability of the prognostic method used. For instance, a prognostic method with high prediction capability might make the decision maker risk-prone, because he/she feels that he/she can trust the RUL predictions provided by the method. As a result, he/she is willing to take the risk of using them to plan PM. On the other hand, if the prediction capability of the prognostic method is not sufficient, the decision maker might be risk-averse towards using the RUL predictions to support any maintenance decision. Assessment of the prediction capability for a prognostic method is, then, an important task in PHM.

Conventionally, the prediction capability is assessed by calculating some purposely defined Prognostic Performance Indicators (PPIs), based on test or benchmark data [START_REF] Saxena | Performance Evaluation for Fleet-Based and Unit-Based Prognostic Methods[END_REF]. Most commonly used PPIs are related to the accuracy and precision of a prognostic method [START_REF] Saxena | Metrics for Evaluating Performance of Prognostic Techniques[END_REF]. Accuracy PPIs quantify the closeness between the model prediction and the true measured values [START_REF] Saxena | Performance Evaluation for Fleet-Based and Unit-Based Prognostic Methods[END_REF][START_REF] Walther | The Concepts of Bias, Precision and Accuracy, and their Use in Testing the Performance of Species Richness Estimators, with a Literature Review of Estimator Performance[END_REF]. Precision PPIs measure how confident the model prediction is and the degree to which the prognostic method will yield the same results if repeatedly applied [START_REF] Saxena | Performance Evaluation for Fleet-Based and Unit-Based Prognostic Methods[END_REF][START_REF] Walther | The Concepts of Bias, Precision and Accuracy, and their Use in Testing the Performance of Species Richness Estimators, with a Literature Review of Estimator Performance[END_REF][START_REF] Saxena | On Applying the Prognostics Performance Metrics[END_REF]. In general, good values of the PPIs give confidence to the decision makers about the predicted RUL, and make them prone to use the prediction results for supporting maintenance decisions. For example, by calculating some accuracy PPIs, Tobon-Micea et al. [START_REF] Tobon-Mejia | A Data-Driven Failure Prognostics Method Based On Mixture of Gaussians Hidden Markov Models[END_REF] compare the prognostic performance of a proposed wavelet-based prognostic method to that of a traditional time-domain method, and conclude that the new method can be applied to support CBM. Using accuracy PPIs, Micea et al. [START_REF] Micea | Online State-Of-Health Assessment for Battery Management Systems. Instrumentation and Measurement[END_REF] compare the prognostic performances of two prognostic methods for application to Ni-MH-batteries. Hu et al. [START_REF] Hu | Online Performance Assessment Method for a Model-Based Prognostic Approach[END_REF] develop an online assessment method for the PPIs of model-based prognostic methods. In [START_REF] Fan | Prognostics of Lumen Maintenance for High Power White Light Emitting Diodes Using a Nonlinear Filter-Based Approach[END_REF], both accuracy and precision PPIs are used to compare the performances of two prognostic methods applied to high-power white Light Emitting Diodes (LEDs). Other similar examples can be found for Lithium-Ion batteries [START_REF] Hu | A Particle Filtering and Kernel Smoothing-Based Approach for New Design Component Prognostics[END_REF], rotating machinery [START_REF] Miao | A Comparison Study of Support Vector Machines and Hidden Markov Models in Machinery Condition Monitoring[END_REF], composite laminates [START_REF] Peng | In-Situ Fatigue Life Prognosis for Composite Laminates Based On Stiffness Degradation[END_REF], etc., where accuracy and precision PPIs are used to compare the prognostic performances of different prognostic methods.

Although fundamental in practice, the existing PPIs reflect only one dimension of the prediction capability, i.e., the degree to which a prognostic method is able to explain the available data (referred to as the prediction performance in this paper) [START_REF] Saxena | Metrics for Offline Evaluation of Prognostic Performance[END_REF][START_REF] Goebel | A Comparison of Three Data-Driven Techniques for Prognostics. in 62nd meeting of the society for machinery failure prevention technology (mfpt)[END_REF]. Indeed, the prediction capability of a prognostic method is also influenced by the trustworthiness of the method, which is defined in this paper as the confidence that the prognostic method can provide an accurate and precise RUL, with correct and fair quantification of its related uncertainty. Such confidence comes from our knowledge on the prognostic method, such as its proven records of successful applications on similar problems or our knowledge on its inherent methodological characteristics in relation to RUL predictions. Suppose that two prognostic methods, denoted by method A and method B respectively, perform equally well in terms of prediction performances (measured by the PPIs computed on the same available data); while method A has been applied successfully in various scenarios of setting similar to the one of interest, method B is newly developed and has rarely been applied before: it seems reasonable that in this situation, a decision maker would prefer to implement and use method A to support maintenance decisions.

In this view, when evaluating the prediction capability of a prognostic method, both the prognostic performance (in terms of PPIs) and the trustworthiness of the prognostic method should be considered.

Whereas the assessment of the prognostic performance is relatively mature through the quantification of PPIs [START_REF] Saxena | Metrics for Evaluating Performance of Prognostic Techniques[END_REF][START_REF] Saxena | On Applying the Prognostics Performance Metrics[END_REF][START_REF] Saxena | Metrics for Offline Evaluation of Prognostic Performance[END_REF], the assessment of the trustworthiness of the prognostic method deserves further consideration. In literature, the trustworthiness of a method or a process is often measured in terms of its maturity [START_REF] Oberkampf | Predictive Capability Maturity Model for Computational Modeling and Simulation[END_REF][START_REF] Paulk | Capability Maturity Model, Version 1.1. Software[END_REF]. The concept of maturity originated in the 1970s, when a model was developed to assess the maturity of an information system's function [START_REF] Gibson | Managing the Four Stages of EDP Growth[END_REF]. Later, the Software Engineering Institute (SEI) developed the Capability Maturity Model (CMM) to assess the maturity of a process for developing software with desirable quality/reliability/trustfulness characteristics [START_REF] Herbsleb | Software Quality and the Capability Maturity Model[END_REF]. Based on the CMM, a Prediction Capability Maturity Model (PCMM) has been recently developed to assess the maturity of modeling and simulation efforts [START_REF] Oberkampf | Predictive Capability Maturity Model for Computational Modeling and Simulation[END_REF].

Other approaches of maturity assessment are being developed and applied in different areas, e.g., master data maturity assessment [START_REF] Spruit | MD3M: The Master Data Management Maturity Model[END_REF], enterprise risk management [START_REF] Farrell | The Valuation Implications of Enterprise Risk Management Maturity[END_REF], hospital information system [START_REF] De Carvalho | Towards an Encompassing Maturity Model for the Management of Hospital Information Systems[END_REF], etc. However, there is no existing maturity assessment methods in the specific context and for the specific aim of prognostics and maintenance decision making.

To this aim, in this paper, we consider both the prediction performance and the method trustworthiness to assess the prediction capability of a prognostic method. It should be noted that an initial effort on prediction capability assessment was published by the authors in [START_REF] Di Maio | Prediction Capability Assessment of Data-Driven Prognostic Methods for Railway Applications[END_REF], however without considering the contribution of method trustworthiness and using only a simple weighted average of the PPIs to quantify prediction quality.

The rest of the paper is organized as follows. The hierarchical framework is presented in Section 2 and, then, used in Section 3 to assess the prediction capabilities of three data-driven prognostic methods of literature. In Section 4, we draw some conclusions and give some ideas of future research.

Hierarchical framework and assessment approach

We present the hierarchical framework developed to assess the prediction capability of prognostic methods in Subsection 2.1, considering two main attributes, RUL prediction quality and method trustworthiness. Prediction quality is assessed in Subsection 2.2 and AHP method is applied in Subsection 2.3 to assess the method trustworthiness. In Subsection 2.4, a classification-based method is developed to determine the prediction capability based on the prediction the quality and method trustworthiness.

Framework of prediction capability assessment

We present a four-layer hierarchical model to support the assessment of prediction capability, as shown in Figure 1. The prediction capability represented by C (Layer 1 in Figure 1) is characterized in terms of RUL prediction quality and prognostic method trustworthiness (Layer 2 in Figure 1). The former, represented by Y , measures the performance of the prognostic method with respect to the specific application and data, while the latter, represented by X , measures the confidence based on knowledge related to the fact that the prognostic method provides trustworthy predictions, in terms of point estimates and uncertainty quantifications. The inhibit (conditional) gate indicates the logical relationship between X and Y in determining the prediction capability: to have a good prediction capability, the prognostic method should at least satisfy a minimum requirement of prediction quality; once this minimum requirement is satisfied, the prediction capability is determined jointly by the prediction quality and the method trustworthiness. Measures the spread and variability of the RUL predictions, i.e., the precision of the prognostic method. A precise prognostic method is preferred.

X Reliability

Measures the capability of the method to yield the same RUL prediction quality, when different analysts apply it on similar sets of data related to similar problems: the larger the reliability, the more trustworthy the method is.

X Validity

Measures the capability of the method to achieve the same RUL prediction quality, when applied to solve different problems with similar characteristics: the larger the validity, the more trustworthy the method is for use in different problems of similar characteristics.

3

X

Mathematical modeling adequacy

Measures the capability of the method to deal with problems of given complexity: a less advanced method may handle well linear and simplified problems, whereas a more complex and advanced method is needed to deal with more realistic problems, e.g., nonlinear and non-stationary problems. In these situations, such methods would be more adequate and trustworthy compared to less mathematically complex and advanced methods.

4

X

Resources requirements

Measures the required resources by the prognostic methods, e.g, the data requirements, the computational costs, the number of hyper-parameters, etc. A prognostic method with lower resource requirements is more controllable and verifiable during the training phase under the realistic available data, and therefore, is more trustworthy for such settings.

The six criteria in Layer 3 are further decomposed into a layer of 19 basic sub-criteria (Layer 4 in Figure 1), where data and information can be used to support the assessment of prediction capability. Detailed descriptions of all the 19 basic sub-criteria can be found in the Appendix. Depending on the nature of the basic sub-criteria, they might take either numerical or linguistic values. The basic sub-criteria used to evaluate the RUL prediction quality are, in fact, quantitative PPIs related to accuracy and precision of a prognostic method. All of them take numerical values, e.g., the Timeless Weighted Error Bias (TWEB, 11 Y ) in Figure 1.

The basic sub-criteria used to evaluate the method trustworthiness, on the other hand, represent evidence on various aspects of the trustworthiness of the prognostic method. Some of them are objective in nature, and, therefore, can be measured by numerical indicators, e.g., the number of academic evidence ( 11 X ) in Figure 1.

Others are qualitative in nature and can only be represented by linguistic or non-numerical values: evaluation of these basic sub-criteria requires the involvement of subjective judgements.

To assess the prediction capability, data and information are collected to support the evaluation of the Layer 4 basic sub-criteria ( ij X and ij Y ). Then, the basic sub-criteria are aggregated to assess the criteria in Level-3, and further aggregated to assess the Level-2 attributes of prediction quality and method trustworthiness. Finally, the prediction capability of the prognostic method is determined based on the joint contributions of the two Level-2 attributes, as shown in Figure 2. The obtained prediction capability incorporates the influences from both prediction quality and method trustworthiness, and, therefore, can be used to support the selection of appropriate prognostic methods for given maintenance planning requirements.

Method trustworthiness assessment

Evaluation of reliability

Evaluation of mathematical modeling adequacy

Evaluation of validity

Evaluation of resources requirements

Aggregation for prediction quality Y

Aggregation for method trustworthiness X

Figure 2 Procedures for prediction capability assessment

Prediction quality assessment method

As shown in Figure 2, the assessment of prediction quality starts from collecting data and information for the Level-4 basic sub-criteria related to prediction quality. Various numerical indicators, referred to as Prognostic Performance Indicators (PPIs), have been defined in literature to assess the performance of a RUL prediction method with respect to both point estimates and uncertainty quantifications. In this work, the PPIs listed in Table A.1 are adopted as the Layer-4 basic sub-criteria; their values, denoted by

ij Y , 1, 2, 1, 2, ,5 j i  
, are calculated based on the formula listed in Table A.1.

Next, the PPIs are aggregated to evaluate the two Layer-3 criteria related to prediction quality, i.e., the RUL point estimate quality and uncertainty quantification quality. As shown in Table A.1, the values of all the ij Y s are bounded in the range ( ,1]  . A weighted-average method is used to aggregate the Layer-4 basic sub-criteria: . The weights represent the relative contribution of a basic sub-criteria to the corresponding Layer-3 criterion. In practice, the weights can be obtained by expert assessments or through some structured analysis method, e.g., the Analytical Hierarchical Processes (AHP) method [START_REF] Saaty | A Scaling Method for Priorities in Hierarchical Structures[END_REF]. It is easy to verify from (1) that both 1 Y and 2 Y take values in ( ,1]  , where a value close to 1 indicates good performance.

1 1, 2, , , , 1, 2, i n i ii i j j j Y Y i jn        (1)
The two Layer-3 criteria are again aggregated to yield the prediction quality Y by the weighted average:

2 1 exp 1 , i i i YY          (2) where i  is the weight for i Y and 2 1 1. i i    
As for the weights in the Layer-3 calculations, the i  can also be determined by experts assessments based on structured analysis methods such as the AHP method [START_REF] Saaty | A Scaling Method for Priorities in Hierarchical Structures[END_REF].

The exponential function in (2) is used for normalization: since

( ,1] i Y   , it is easy to verify that (0,1] Y 
and a value close to 1 indicates good prediction quality.

Method trustworthiness assessment method

Since the assessment of method trustworthiness involves multiple quantitative (i.e., the [START_REF] Saxena | Performance Evaluation for Fleet-Based and Unit-Based Prognostic Methods[END_REF]

32

XX 

in Figure 1) and qualitative sub-criteria (i.e., the 41

43

XX  in Figure 1), it is formulated as a Multi-Criteria Decision Analysis (MCDA) problem [START_REF] Streimikiene | Multi-Criteria Analysis of Electricity Generation Technologies in Lithuania[END_REF]. As a widely applied MCDA method [START_REF] Vaidya | Analytic Hierarchy Process: An Overview of Applications[END_REF], AHP is selected for the assessment. AHP, first introduced in 1977 [START_REF] Saaty | A Scaling Method for Priorities in Hierarchical Structures[END_REF], is a hierarchical framework to support multi-criteria decision analysis, where the decision problem considered (the first, top, layer in the hierarchy) is decomposed into several layers of criteria and, eventually, the last, bottom layer containing the alternatives available for the solution of the decision problem.

Through pairwise comparisons among elements in the same layer, the alternative solutions in the bottom layer can be ranked with respect to the decision problem in the top layer [START_REF] Saaty | A Scaling Method for Priorities in Hierarchical Structures[END_REF]. For a detailed discussion on the implementation procedures of AHP, readers might refer to [START_REF] Huguenin | Data Envelopment Analysis and Non-Discretionary Inputs: How to Select the Most Suitable Model Using Multi-Criteria Decision Analysis[END_REF][START_REF] Dožić | An AHP Approach to Aircraft Selection Process[END_REF]. The AHP model for method trustworthiness assessment is illustrated in Figure 3.

X p 2 X p 3 X p 4 X p 1 1 X p 1 2 X p 2 1 X p 2 2 X p 3 1 X p 3 2 X p 4 1 X p 4 2 X p 4 3 X p , k ij M X p

Figure 3 The AHP model for trustworthiness assessment

Based on the AHP model of Figure 3, the assessment of method trustworthiness involves three steps:

Step 1: Determine the inter-level priorities for the criteria ( ). The inter-level priorities quantify the relative importance of the lower-level elements with respect to the corresponding high-level element. For the qualitative sub-criteria, experts compare their relative importance using the 1-9 scaling system defined in [START_REF] Saaty | Decision Making with the Analytic Hierarchy Process[END_REF], where scale 9 represents " i is extremely more important than j ", scale 1 represents "equally important" and scale 1 / 9 represents " j is extremely more important than i ". Pairwise , Once the comparison matrix for a given level of the hierarchy has been constructed, the eigenvalue method is used to calculate the inter-level priorities [START_REF] Saaty | A Scaling Method for Priorities in Hierarchical Structures[END_REF]. Suppose the priorities associated with a comparison matrix

k ij k ij k ij MX MX n MX k X p X    (3) where
A are denoted by 1 , , ,2 ,,
    , , 1, 2, , .
,

Mi i T MM M MM i A n      p p pp pp (4)
Finally, the consistency of the comparison matrix is checked to see if the calculated priority vector makes

sense. A comparison matrix

A is consistent if it satisfies both the reciprocity rule [33]:

1 ij ji a a  (5)
and the transitivity rule [START_REF] Streimikiene | Multi-Criteria Analysis of Electricity Generation Technologies in Lithuania[END_REF]:

, ij ik kj aaa   (6)
where ij a is the element in the i th row and j th column of A and ,, i j k are indexes for the criteria or alternative solutions in A . The consistency can be checked following the procedure in Figure 4, where RI is the CI (Confidence Index) of a randomly generated n n  matrix whose values can be found in [32, 37]. The three-step procedures are repeated for each criteria, sub-criteria and alternative solutions, until all the , Step 2: Calculate the global priority for each alternative solution.

A bottom-up synthesis process is used to calculate the global priority for each alternative solution, with respect to the top goal of the hierarchy:

4 , 11 , X i ki i j k ij n M X M X X ij p p p p      (7)
where The method trustworthiness for each prognostic method, denoted by , 1, 2, ,

i M Xi n 
, is then determined based on the global priorities:

  max 1 , max i i i M M n iM pX X p    (8)
where max X is the method trustworthiness of the prognostic method with the largest global priority, which is evaluated based on expert judgements. The value of max X ranges in   0,1 , where a value closer to 1 indicates that the prognostic method is more trustworthy.

Prediction capability assessment and prognostic method selection

Prediction capability

C is an integrated metric that supports the selection of appropriate prognostic methods for a given application scenario. Depending on the role that the predicted RUL plays in maintenance planning, three typical application scenarios are usually distinguished: fully supportive, where the predicted RUL is used to support Predictive Maintenance (PM) planning; partially supportive, where the predicted RUL is used to support Condition-Based Maintenance (CBM) planning; and non-supportive, where the predicted RUL is not directly applicable in maintenance planning. Therefore, the prediction capability is assumed to take three discrete values,

  0 1 2 ,, CC C C  , where 0 C , 1 C , 2
C correspond to the required prediction capability for the non-supportive, partially supportive and fully supportive application scenarios, respectively.

The issue of prediction capability assessment, is, then, formulated within a classification framework: given a prognostic method, which is characterized in terms of prediction quality and method trustworthiness, select among the above three candidates a proper value for its prediction capability.

In this paper, we assume that training data are available to construct a classifier for prediction capability assessment using supervised learning algorithms. The training data comprise of prognostic methods with known prediction quality, method trustworthiness and prediction capability. In Figure 5 predicted RUL to plan maintenance services. It can be seen from Figure 6 that to be qualified to support PM, the decision maker thinks that a prognostic method needs to have both high prediction quality and high trustworthiness (fully supportive region). Also, when 1 Y e   (roughly speaking, it means that the average prediction error between the predicted and true RUL is higher than the total life, see Table A.1 and Eq. ( 2)), the decision maker is not willing to apply the prognostic method to support any kind of maintenance decisions (non-supportive region), regardless of how well the method trustworthiness is. This fact is also reflected by the conditional gate in Figure 1. If the minimum requirement of Y is satisfied ( 1 Ye   ), the prediction capability further depends on the value of method trustworthiness: if the method trustworthiness is medium or high (roughly speaking, 0.3 X 

), the decision maker would apply the method to support CBM (partially supportive region); otherwise, only with higher prediction quality (roughly speaking, 0.8 Y 

), the prognostic method can be qualified to support CBM.

Figure 6 A classifier constructed for prediction capability assessment

A major strength of the developed prediction capability assessment framework is that it integrates both prediction quality and method trustworthiness of the prognostic method, while existing frameworks, such as those in [START_REF] Saxena | Metrics for Evaluating Performance of Prognostic Techniques[END_REF] or [START_REF] Saxena | On Applying the Prognostics Performance Metrics[END_REF], often neglect method trustworthiness. To demonstrate the strength of the developed framework, we also apply the prediction quality based framework on the training data in Figure 5. Since only the prediction quality is considered, we only use Y to construct the classifier. We again use SVM to construct the classifier and the result is given ). The comparison shows that by considering the method trustworthiness, the developed assessment framework provides a more comprehensive description of the prediction capability.

Application

In this section, the framework developed in Section 2 is implemented to assess the prediction capabilities of three prognostic methods of literature, i.e., Fuzzy Similarity (FS) [START_REF] Di Maio | Failure Prognostics by a Data-Driven Similarity-Based Approach[END_REF][START_REF] Zio | A Data-Driven Fuzzy Approach for Predicting the Remaining Useful Life in Dynamic Failure Scenarios of a Nuclear System[END_REF], Feed-forward Artificial Neural Networks (FANN) [START_REF] Bai | A Self-Cognizant Dynamic System Approach for Prognostics and Health Management[END_REF] and Hidden Semi-Markov Model (HSMM) [START_REF] Dong | A Segmental Hidden semi-Markov Model (HSMM)-based Diagnostics and Prognostics Framework and Methodology[END_REF][START_REF] Moghaddass | An Integrated Framework for Online Diagnostic and Prognostic Health Monitoring Using a Multistate Deterioration Process[END_REF][START_REF] Miao | Condition Monitoring and Classification of Rotating Machinery Using Wavelets and Hidden Markov Models[END_REF]. A simulation case study of nine run-to-failure trajectories is considered as data, as shown in Figure 8 [START_REF] Di Maio | Prediction Capability Assessment of Data-Driven Prognostic Methods for Railway Applications[END_REF]. These data represent the failure trajectories that can be extracted based on simulated mono-dimensional signal (e.g., temperature, pressure, or vibration signal) of a generic component. The three methods are applied to predict the RUL, and their prediction quality and method trustworthiness are assessed in Subsections 3.1 and 3.2, respectively. The prediction capabilities of the three methods are determined in Subsection 3.3 by combining the prediction quality and method trustworthiness.

Prediction quality assessment

The three prognostic methods are applied to predict the RUL of the case study in Figure 8. Leave-one-out cross validations are used to compare the prediction quality of the three methods, where for each validation, one of the nine samples is left out while the rest eight are used as training samples. The RUL of the left-out sample is regarded as the true RUL so that the PPIs in Table A.1 can be calculated. Empirical Mode Decomposition (see [START_REF] Huang | The Empirical Mode Decomposition and the Hilbert Spectrum for Nonlinear and Non-Stationary Time Series Analysis[END_REF] for details) is used for the three methods to pre-process the raw signal and construct health indicators (HIs). The RUL prediction from each method is given in Figure 9. The accuracy and precision PPIs calculated based on Table A.1 are listed in Table 2 and Table 3. Y , the weights of each Layer-4 basic sub-criteria should be determined first.

In this case study, experts are invited to rank all the PPIs in terms of their relative importance in affecting the corresponding prognostic performance. Then, the weight of each PPI can be calculated by:

      1 1 1 21 , 1 1 1 i n i i ni n i nn n              ( 9
)
where i is the ranking of the PPI (in descending order of importance) and n is the total number of the PPIs in the same category. Suppose that [START_REF] Saxena | Performance Evaluation for Fleet-Based and Unit-Based Prognostic Methods[END_REF] respectively. According to [START_REF] Lipi | A Condition-And Age-Based Replacement Model Using Delay Time Modelling[END_REF], their weights are calculated and listed in Table 4. Y and 2 Y are calculated based on (1) and the results are given in Table 5. The prediction quality Y is, then, calculated based on (2) and Table 5, where the RUL point estimate quality and uncertainty quantification quality are assumed to have equal weights, 12 0.5.

 

The results are tabulated in Table 6. The results in Table 6 show that considering both the point estimate and uncertainty quantification quality, Fuzzy Similarity performs the best among the three prognostic methods in terms of prediction quality, whereas the prediction quality of Hidden-Semi Markov Model is the worst among the three methods.

Method trustworthiness assessment

Step 1: Determine the inter-level priorities

For the quantitative basic sub-criteria 11 32 XX 

, the numerical values for the criteria are collected in Table 7, where 1 M , 2 M and 3 M correspond to FS, FANN and HSMM, respectively. Based on (3), the local priorities are calculated and given in Table 8. It should be noted that the numerical values in Table 7 are simulated for illustrative purposes. In practice, these values should be collected based on actual data extracted from literature and engineering applications. , the local priorities are obtained by constructing pairwise comparison matrices. Altogether, there are eight pairwise comparison matrices that need to be constructed: one for the criteria in Layer 2, four for the sub-criteria in Layer 3 and three for the alternative solutions in Layer 4. For simplicity and illustrative purposes, we assume that all the criteria and sub-criteria in Layer 2 and Layer 3 are indifferent, so that all the elements in these pairwise comparison matrices are 1 and

4 11 4 1 3 0.5 X X X X p p pp      
. For the methods in Layer 4, experts are invited to make pairwise comparisons among the three methods in terms of computational costs, numbers of hyper-parameters and historical data requirements, respectively. The pairwise comparison matrices are constructed following the 1-9

scaling system introduced in Section 2.3. The resulted pairwise comparison matrices are 41 42 43

1 4 2 1 6 4 1 1 / 3 1 / 2 1 / 4 1 1 / 3 , 1 / 6 1 1 / 3 , 3 1 2 . 1 / 2 3 1 1 / 4 3 1 2 1 / 2 1 X X X A A A                                 
The inter-level priorities are calculated using (4) and listed in Table 9. The value of CR for each comparison matrix is also calculated following the procedures in Figure 4 to check the consistency. It can be seen from Table 9 that all the three CR are less than the threshold value 0.1: therefore, all the three comparison matrices are consistent. Equation ( 7) is, then, used to determine the global priority for each alternative solution, where the local priorities involved have been determined in Section 3.2.1 (see Table 8 and Table 9). The obtained global priorities are given in Table 10. ; then, the trustworthiness of the other two methods can be determined using [START_REF] Peng | Current Status of Machine Prognostics in Condition-Based Maintenance: A Review[END_REF], as shown in Table 11.

Prediction capability assessment and method selection

The prediction capabilities of the three prognostic methods are assessed using the classifier in Figure 6,

where the values of X and Y are given in Table 11 and Table 6, respectively. The result is given in Table 12. Based on the assessment results, FS can be used to support CBM decision making for this specific application, while FANN and HSMM should not be used to support maintenance decisions due to their relative poor prediction capabilities in this case study.

M ) i M C 1 C 0 C 0 C

Conclusions

In this paper, a hierarchical framework is developed to assess the prediction capability of prognostic methods. The framework considers the joint contributions from prediction quality and method trustworthiness (Layer 2). The prediction quality and method trustworthiness are further decomposed into six sub criteria (Layer 3) and 19 basic sub-criteria (Layer 4), where information and data can be collected to support the prediction capability assessment. A bottom-up method is developed to determine the prediction capability based on the information and data collected in the Layer-4 basic sub-criteria, in which the AHP method is applied for the aggregation of qualitative sub-criteria. A classification-based method is developed for the assessment of prediction capability. Based on the assessed prediction capability, the appropriateness of the prognostic method for supporting maintenance decisions can be determined, i.e., labelling it as qualified to support predictive maintenance, qualified to support condition-based maintenance or not qualified to support any maintenance decision.

The framework proposed in this paper does not pretend to be exhaustive in the criteria and factors considered, nor rigidly prescriptive in the methods used for their evaluation. In the end, the prediction capability assessment is framed as a process of classification: given all the available information and knowledge, classify the prognostic methods based on their prediction capabilities. Therefore, in the future research, classification algorithms, e.g., Naï ve Bayes classifier, majority rule sorting, etc., will also be investigated to develop efficient prediction capability assessment methods. Furthermore, various uncertainties exist in the process of prediction capability assessment. For example, the number of evidence in Figure 1 is often estimated based on sampling approaches. Hence, uncertainty arises from sampling errors. Also, the qualitative basic sub-criteria are evaluated based on pairwise comparisons and, therefore, subjected to uncertainty resulting from incomplete knowledge. How to address the effect of uncertainty in prediction capability assessment deserves further investigations too. Y is 1 , indicating that the average absolute percentage error for all units throughout their lifetime i T is small. A low value tells the user that a discrepancy between the estimated RUL and the true one occurs.

(

  performance               21
i T N it i it i Y b t NT RUL t b RUL t EO t Pt                  



Measures the average fraction of points, during the lifetime i T over all N units, for which the prediction of the RUL estimated at a specific time t before failure is, with  confidence, the true RUL at  i : index for the identification of the unit under test (e.g., the equipment).  N : total number of units under test.  t : index for the time instant.

 T : failure time of the unit. Note that each unit has a different i T value.  EOP: End-Of-Prediction, time at which the unit is expected to fail, as predicted by the prognostic model.

3 sub

 3 of the Layer-4 basic sub-criteria associated with the i th Layer-of the Layer-4 basic sub-criteria. In this paper,



  comparison matrices, indicated with the symbol A in this paper, are constructed by filling out each element ij a with the numerical value of relative importance and considering the reciprocity property, which indicates that 1 .For the quantitative basic sub-criteria 1132XX  , their inter-level priorities can be determined by calculating priority weights as:

  -level priority of the k th prognostic method with respect to the basic sub-criteria value that the k th prognostic method takes with respect to the basic sub-criteria ij X .

Figure 4

 4 Figure 4 Procedures for consistency tests[START_REF] Saaty | A Scaling Method for Priorities in Hierarchical Structures[END_REF][START_REF] Saaty | Decision Making with the Analytic Hierarchy Process[END_REF]

Step 3 :

 3 priority of the k th prognostic method and i X n is the number of sub-criteria under the criterion i X . Note that the global priorities should sum up to 1 , i.e., Determine the method trustworthiness.

CC  , 1 CC  and 2 CC

 12 , we present 200 training data, which are constructed by randomly generating 200 samples of X and Y , and then, inviting decision makers to assess the prediction capability for each combination of X and Y . Support Vector Machine (SVM) is used to construct a classifier for prediction capability assessment. We directly apply the SVM algorithm in MATLAB® R2015b and the result is shown in Figure 6. A 10-fold cross validation is conducted to validate the classifier. The average misclassification rate of the classifier is 1 0.04  , which indicates good classification performance. The XY  plane is partitioned in non-supportive, partially supportive and fully supportive regions, corresponding to 0  , respectively. The prediction capability of a prognostic method can, then, be determined based on its position in the XY  plane of Figure 6.

Figure 5 Figure 6

 56 Figure 5 Training data for prediction capability assessment Figure 6 reflects, based on the training data, the decision makers attitude to the risk of relying on a

Figure 7 .

 7 A 10-fold cross validation is conducted. The average misclassification rate for this classifier is 2 0.22  , which is much larger than that of the developed framework ( 1 0.04 

Figure 7

 7 Figure 7 Training data and classifier when only Y is considered

Figure 8

 8 Figure 8 The nine simulated run-to-failure degradation trajectories

Figure 9

 9 Figure 9The predicted RUL of FS, FANN and HSMM[START_REF] Di Maio | Prediction Capability Assessment of Data-Driven Prognostic Methods for Railway Applications[END_REF]

. 23 Y

 23 The optimum value for 21 Y is1 , indicating that all estimated RULs have still an accuracy at least of  at a relative distance  from the current prediction time t . Low values indicate that the prediction made at time t is not reliable in the future time window defined by  . The parameter  is the confidence modifier and  is the time window modifier.Considers the standard deviation of the weighted prediction error during the entire lifetime i T for all N units. The optimum value for[START_REF] Saxena | Metrics for Offline Evaluation of Prognostic Performance[END_REF] Y is1 , indicating that all units either share a similar average weighted prediction error or that it is small. A low value of 22 Y indicates a high dispersion, and thus, a low precision.Considers the standard deviation of the average error over the lifetime i T for all N units. The optimum value is1 , indicating that all errors for all units are closely similar. A low value of[START_REF] Goebel | A Comparison of Three Data-Driven Techniques for Prognostics. in 62nd meeting of the society for machinery failure prevention technology (mfpt)[END_REF] Y indicates that the dispersion of the errors within the N units is high.

( 25 Y

 25 of the Root Mean Squared Error of the N units during the entire lifetime i T . The optimum value of 24 Y is 1, indicating that the error between the estimated RUL and the true RUL is consistent in the model. A low value indicates that the discrepancy between the estimated and the true RUL is inherently stochastic. deviation of the Indicator M for all 𝑁 units. The 25 Y measures how the indicator M varies through all N units. The optimum value of is 1, indicating that the standard deviation of the indicator is 0: thus, the indicator is concentrated on one value, reducing the variability of the performance throughout the units. A low value indicates that the indicators behavior varies between units.

  

Prognostic method prediction capability RUL point estimate quality 1 Y Uncertainty quantification quality 2 Y Method trustworthiness X Validity 2 X Resources requirements 4 X Mathematical modeling adequacy 3 X Reliability 1 X Academic evidence 11 X Industrial evidence 12 X Academic evidence 21 X Industrial evidence 22 X Successful non- linear applications 31 X Successful non- stationary applications 32 X TWEB 11 Y SME 12 Y MAPE 13 Y MSE 14 Y

	SMeE	P  
	15 Y	21 Y
		22 Y

WPS 23 Y SSD 24 Y RMSE 25 Y PSTWEB RUL prediction quality Y Layer 1 Layer 2 Layer 3 Layer 4

	INHIBIT gate
	(conditional)

Quantitative information measured by the PPIs based on data Qualitative information measured based on prior knowledge on the prognostic method Historical data requirements 43 X Computational costs 41 X Hyper- parameters in the model 42 X C TWEB: SME: MAPE: MSE: SMeE: : WPS SSD: RMSE: PS TWEB : Timeliness Weighted Error Bias Sample Mean Error Mean Absolute Percentage Error Mean Square Error Sample Median Error α-λ Performance Weighted Prediction Spread Sample Standard Deviation Root Mean Square Error Prediction Spread P   Figure 1 Hierarchical framework for prediction capability assessment The two attributes in Layer 2 are further broken down into factors that influence them, leading to the six

	criteria in Layer 3: RUL point estimate quality ( 1 Y ) and uncertainty quantification quality ( 2 Y ), which
	contribute to the RUL prediction quality and reliability ( 1 X ), validity ( 2 X ), mathematical modeling adequacy
	( 3 X ) and resources requirements ( 4 X ), which influence the method trustworthiness. Detailed descriptions of
	the criteria are given in Table 1.

Table 1 Descriptions of the Layer-3 characteristics

 1

	Notation	Meaning	Description
	1 Y	RUL point estimate quality	Measures the distance of the RUL point estimates from the true RUL values, i.e., the accuracy of the prognostic method. An accurate prognostic method is obviously preferred.
		Uncertainty
	2 Y	quantification
		quality

Reliability X 1 Academic evidence X 11 Industrial evidence X 12 Academic evidence X 21 Industrial evidence X 22 Successful non- linear applications X31 Successful non- stationary applications X32 Computational costs X 41 Hyper- parameters in the model X 42 Historical data requirements X 43 Layer 2 Criteria Layer 3 Sub-criteria Layer 4 Alternative solutions 1

	Layer 1
	Goal
			Prognostic
		Validity X 2	method 1
	Select the
	prognostic method with		Prognostic method 2
	maximum trustworthi -ness	Mathematical modeling adequacy X 3	...
			Prognostic
			method n
		Resources
		requirements
		X 4

  ,

	p	L  LL n p p p  	T	. The eigenvalue method first calculates the eigenvector of	A that
	corresponds to the largest eigenvalue, denoted by

M p and M  , respectively. The priority vector p is, then, calculated by normalizing the vector M p , as in (4) below, where () i p and   , Mi p represent the i th component in p and M p , respectively:

Table 2 Accuracy PPIs for FS, FANN and HSMM

 2

	PPIs	FS	FANN	HSMM
	11 Y	0.98	0.94	0.11
	12 Y	0.37	0.56	-9.47
	13 Y	0.85	0.63	-1.44
	14 Y	-5.14	-7.56	-143.79
	15 Y	0.98	0.57	-10.65

Table 3 Precision PPIs for FS, FANN and HSMM

 3

	PPIs	FS	FANN	HSMM
	21 Y	0.61	0.34	0.02
	22 Y	0.97	0.94	0.67
	23 Y	-0.71	-1.11	-4.16
	24 Y	-0.28	-1.42	-10.20
	25 Y	0.98	0.97	0.14
	To assess the values of 1 Y and 2

Table 4 Weights of the PPIs

 4

			Accuracy PPIs			Precision PPIs
	PPIs	11 Y	12 Y	13 Y	14 Y	15 Y	21 Y	22 Y	23 Y	24 Y	25 Y
	Weights 0.333 0.267 0.200 0.133 0.067 0.333 0.267 0.200 0.133 0.067
	Then, the values of 1

Table 5 Evaluation results of 1

 5

	Y and 2 Y

Table 6 Prediction quality of the three prognostic methods

 6

	Y	Y	Y
	FS	FANN	HSMM
	0.433	0.307	1.63 10  	6

Table 7 Numerical values for the basic sub-criteria

 7

			X	11	X	12	X	21	X	22	X	31	X	32
	M	1	24	18	22	16	22	18
	M	2	39	31	41	38	38	31
	M	3	38	32	35	24	28	22

Table 8 Inter-level priorities of the alternative solutions with respect to 11

 8

	XX 	32

Table 9 Inter-level priorities of the alternative solutions with respect to

 9

	41 XX 	43

Table 10 Global priorities for the three prognostic methods

 10

	p		p				p
	M	1	M	2			M	3
	0.312	0.366			0.322
	3.2.3. Step 3: Determine the method trustworthiness
	It can be seen from Table 10 that FANN ( 2 M ) is the most trustworthy one among the three prognostic
	methods. Its method trustworthiness is, then, evaluated by expert judgements and serves as benchmark for the other
	two methods. Suppose the experts judge that the trustworthiness of	2 M is	2	0.85

M X 

Table 11 Method trustworthiness for the three prognostic methods

 11

	X		X		X
	M	1	M	2	M	3
	0.72	0.85	0.75

Table 12 Prediction capabilities for the three prognostic methods

 12

	Prognostic methods FS ( 1 M ) FANN ( 2 M ) HSMM ( 3

Appendix Detailed definitions of the basic sub-criteria 1Table A .

 A 1 Descriptions of the Layer-4 basic sub-criteria related to prediction quality 2 , which puts more weights on errors made at the end of lifetime. The optimal value for the TWEB is 1, indicating that the average penalized weighted prediction value is centered on the true RUL. Values smaller than 1 indicate that the predictions dispersion is above, or under, the true RUL.

	Notation	Name									Formula			Description	Range Category
														Calculate the penalized weighted prediction error over
			  Yt 1 11 1 1 1 i N i t T i NT    * ii i RUL t RUL t                	,	the entire lifetime penalize late predictions ( T , with a penalty function () z  i 0 z  ) against early predictions ( 0 z  ). The weighting function   i t 	to is a
	11 Y	Timeliness Weighted Error Bias		() z	       	exp ex p	1 2 || || z a z a       	1 for 1 for  z z 	0, 0,		Gaussian Kernel Function with a mean value standard deviation 0.5	i T and a	( ,1] 	Accuracy PPI
			12 aa 		0.
														Measure the average sum of errors over all sample
														points up to	i T . The optimal value of 12 Y is 1 ,
	12 Y	Sample Mean Error		12 Y	1  	  ii * RUL t RUL t      11 11 t T N it i NT     		indicating that the sum of prediction errors of all the sample points is 0 . Therefore, the predicted RUL is equally distributed to both sides of the true RUL. Low	( ,1] 	Accuracy PPI
														values of 12 Y indicate greater discrepancy between the
														predicted and true RUL.
														Exploits the average absolute percentage error of all
														N units throughout their lifetime	T . The optimum
	13 Y	Mean Absolute Percentage Error		13 Y		1 	  ii   * ii 11 11 i T N it RUL t RUL t   N T RUL t       		value for 13	i

i T

  RULs are equal to the real ones for all units i . A low value indicates that, during the lifetime of the N components, the errors in the RUL estimates are high.

	Exploits the absolute value of the median of all mean indicating that the estimated Y Sample Median Error       , * 15 1,2, 1 1 1 Median i T i N i i t i Y RUL t RUL t T           errors, for all N units, over their lifetime i T . An optimum value for 15 Y is 1 , indicating that the Y modulus of the median error is zero. A low 15	( ,1] 	Accuracy PPI
						indicates that most RUL estimates are wrong.
	Y
								1]	Accuracy PPI
	14 Y	Mean Square Error	14 Y	1  	  ii * RUL t RUL t      2 11 11 i T N it i NT     	Takes into account the average for all N units of the average quadratic error of the RULs estimated during the lifetime i T . The optimum value for the 14 Y is 1 ,	( ,1] 	Accuracy PPI

  i RUL t : Estimated Remaining Useful Life (RUL) for the unit i , at time index t . : Real RUL value for the unit i , at time index t . PPI calculated for the unit i , at time t . TableA.2 Descriptions of the Layer-4 basic sub-criteria related to method trustworthiness 1

	Notation	Meaning	Sub-criterion	Type
	X X	Number of academic evidence that supports the method's reliability Number of industrial evidence that supports the method's reliability	Reliability	Quantitative Quantitative
	X X	Number of academic evidence that supports the method's validity Number of industrial evidence that supports the method's validity	Validity	Quantitative Quantitative
	X	Number of successful applications dealing with non-linear problems	Mathematical	Quantitative
	X	Number of successful applications dealing with non-stationary problems	modeling adequacy	Quantitative
	X X X	Requirements on computational costs Number of hyper-parameters that needs to be tuned Requirements on historical data	Resources requirements	Qualitative Qualitative Qualitative
	2
	3
		 
		 

n : number of total measurements.    * i RUL t i Mt:

Acknowledgement

This work has been performed within the initiative of the Center for Resilience and Safety of Critical