Tasneem Bani-Mustafa

Zhiguo Zeng

Enrico Zio

Dominique Vasseur

A new framework for multi-hazards risk aggregation

Keywords: Quantitative Risk Assessment (QRA), Risk-Informed Decision Making, Trustworthiness in Risk Assessment, Multi-Hazards Risk Aggregation (MHRA), Strength of Knowledge (SoK), Nuclear Power Plants (NPP) Analytical Hierarchy Process SoK: Strength of Knowledge DST: Dempster Shafer Theory

In this paper, we develop a new method for Multi-Hazards Risk Aggregation (MHRA). A hierarchical framework is first developed for evaluating the trustworthiness of the risk assessment. The evaluation is based on two main attributes (criteria), i.e., the strength of knowledge supporting the assessment and the fidelity of the risk assessment model. These two attributes are further broken down into sub-attributes and, finally, leaf attributes. The trustworthiness is calculated using a weighted average of the leaf attributes, in which the weights are calculated using the Dempster Shafer Theory-Analytical Hierarchy Process (DST-AHP). Risk aggregation is, then, performed by a "weighted posterior" method, considering the level of trustworthiness. An application to the risk aggregation of two hazard groups in Nuclear Power Plants (NPP) is illustrated.

Introduction

In Risk-Informed Decision-Making (RIDM), risk metrics are first calculated through Multi-Hazards Risk Aggregation (MHRA) by combining all the relevant information on risk from different contributors (hazard groups) and, then, used to support Decision-Making (DM) [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF]. A fundamental criticism of the current practice is that the aggregation is conducted by a simple arithmetic summation of the risk metrics from different hazard groups, without considering the heterogeneity in the degrees of maturity and realism of the risk analysis for each hazard group [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF]. For example, in Nuclear Power Plants (NPP), the Probabilistic Risk Assessment (PRA) for internal events has been developed for many years and considered relatively mature compared to external events [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF] or to fire [START_REF] Siu | FIRE PRA MATURITY AND REALISM: A DISCUSSION AND SUGGESTIONS FOR IMPROVEMENT[END_REF]. Simply adding up the risk indexes can be misleading because it does not consider any information on the trust in the risk indexes calculated for each hazard group. This is a real problem as the results of the PRAs to be aggregated often involve different hazard groups with different levels of realism and trustworthiness.

Various factors contributing to the trustworthiness of risk analysis have been discussed in the literature, including the strength of background knowledge, conservatism, plausibility and realism of assumptions, uncertainty, level of sophistication and details in the analysis, value-ladenness of the assessors, experience, number of approximations and assumptions made in the analysis, etc. [START_REF] Epri | Practical Guidance on the Use of Probabilistic Risk Assessment in Risk-Informed Applications with a Focus on the treatment of Uncertanity[END_REF], [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF]. Communicating these factors to the decision maker can better inform decision making [START_REF] Flage | Expressing and communicating uncertainty in relation to quantitative risk analysis[END_REF], [START_REF] Epri | Practical Guidance on the Use of Probabilistic Risk Assessment in Risk-Informed Applications with a Focus on the treatment of Uncertanity[END_REF], (Aven, 2013b), [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF], [START_REF] Veland | Improving the risk assessments of critical operations to better reflect uncertainties and the unforeseen[END_REF]. For this, some experts propose a broad representation of risk that highlights uncertainties rather than probability [START_REF] Flage | Expressing and communicating uncertainty in relation to quantitative risk analysis[END_REF], (Aven, 2013b), (Aven and Krohn, 2014). In Aven (2013a), the risk is described in terms of events, consequences, uncertainty (𝐴, 𝐶, 𝑈) and a structure is presented for linking the elements of a Data-Information-Knowledge-Wisdom hierarchy to this perspective. In [START_REF] Flage | Expressing and communicating uncertainty in relation to quantitative risk analysis[END_REF], the authors apply the concept of uncertainty as the main component of risk, whereas the probability is regarded as an epistemic-based expression of uncertainty. Their argument is that for decision making purposes, a broad and comprehensive representation of risk is required to cover the events, consequences, predictions, uncertainty, probability, sensitivity, and knowledge. In addition, they propose a simple and practical method to classify uncertainty factors and evaluate the background knowledge given the following criteria: the inter-alia assumptions and presuppositions (solidity of assumptions), historical field data (availability of reliable data), understanding of phenomena, and agreement among experts. Some attempts are also found in the literature that focus on treating the uncertain assumptions as an implication of new risk perspectives. Aven (2013b) proposed a method for assessing the assumption deviation risk by three elements: (i) the degree of the expected deviation of the assumption from reality and its consequences (ii) a measure of uncertainty of the deviation and consequences; (iii) the knowledge on which the assumptions are based. [START_REF] Berner | Strengthening quantitative risk assessments by systematic treatment of uncertain assumptions[END_REF] summarize four approaches for treating uncertain assumptions: (i) law of total expectation; (ii) interval probability; (iii) crude strength of knowledge and sensitivity categorization; (iv) assumption deviation risk.

In this work, they extend the method in Berner and Flage (2015) that evaluates the assumption deviation risk based on three criteria: belief in the deviation from the assumption, sensitivity of the risk index and its dependency on the assumption, and SoK on which the assumptions are made. Six settings are identified for the corresponding scenarios resulting given the three criteria. Guidance for treating the uncertainty related to the deviation of assumptions is given for each setting. Finally, an application of Numeral Unit Spread Assessment Pedigree (NUSAP) is proposed for analyzing the strength, importance, and potential value-ladenness of assumptions through a pedigree diagram [START_REF] Van Der Sluijs | Combining Quantitative and Qualitative Measures of Uncertainty in Model-Based Environmental Assessment: The NUSAP System[END_REF], [START_REF] Boone | NUSAP: a method to evaluate the quality of assumptions in quantitative microbial risk assessment[END_REF], [START_REF] Kloprogge | A method for the analysis of assumptions in modelbased environmental assessments[END_REF], [START_REF] De | Assumptions in quantitative analyses of health risks of overhead power lines[END_REF]. The pedigree diagram uses seven criteria for evaluating the quality of assumptions: (i) plausibility; (ii) inter-subjectivity peers; (iii) intersubjectivity stakeholders; (iv) choice space; (v) influence of situational limitations; (vi) sensitivity to view and interests of the analyst (vii) and influence on results [START_REF] Van Der Sluijs | Combining Quantitative and Qualitative Measures of Uncertainty in Model-Based Environmental Assessment: The NUSAP System[END_REF], [START_REF] Boone | NUSAP: a method to evaluate the quality of assumptions in quantitative microbial risk assessment[END_REF], [START_REF] Kloprogge | A method for the analysis of assumptions in modelbased environmental assessments[END_REF], [START_REF] De | Assumptions in quantitative analyses of health risks of overhead power lines[END_REF].

In addition, some attempts are found in the literature for directly evaluating the trustworthiness and other relevant quantities. In Bani-Mustafa et al. (2017), the trustworthiness of risk assessment models is evaluated through a hierarchical tree that covers the different factors including modeling fidelity, SoK, number of approximations, amount and quality of data, quality of assumptions, number of model parameters, etc. Trustworthiness is also measured in the literature in terms of maturity and credibility. For example, in Model and Simulation (M&S) and information system, a capability maturity model is used to assess the maturity of a software development process in the light of its quality, reliability, and trustworthiness [START_REF] Paulk | Capability Maturity Model for Software, Version 1.1[END_REF]. A predictive capability maturity model has been developed to assess the maturity of M&S efforts through evaluating the representation and geometric fidelity, physics and material model fidelity, code and solution verification, model validation and uncertainty quantification, and sensitivity analysis [START_REF] Oberkampf | Predictive capability maturity model for computational modeling and simulation[END_REF]. In [START_REF] Zeng | A hierarchical decision-making framework for the assessment of the prediction capability of prognostic methods[END_REF], a hierarchical framework has been developed to assess the maturity and prediction capability of a prognostic method for maintenance decision making purposes. The hierarchical tree covers different attributes that are believed to affect the prediction capability of prognostic methods and the trustworthiness of the results. In (Nasa, 2013), a framework is proposed for assessing the credibility of M&S through eight criteria: (i) verification; (ii) validation; (iii) input pedigree; (iv) results uncertainty (v) results robustness;

(vi) use history; (vii) M&S management; (viii) people qualification. In (Bani-Mustafa et al., 2017), the trust of the model is evaluated based on the level of maturity of the risk assessment model through four main criteria: (i) uncertainty; (ii) knowledge; (iii) conservatism; (iv) sensitivity. Also, the quality of M&S is assured by the American Society of Mechanical Engineers (ASME) through verification and validation [START_REF] Schwer | Guide for Verification and Validation in Computational Solid Mechanics[END_REF]. Verification is concerned with evaluating the accuracy of the computational model in representing the conceptual and mathematical model, and validation is concerned with evaluating the accuracy of the model in representing reality [START_REF] Schwer | Guide for Verification and Validation in Computational Solid Mechanics[END_REF].

As seen from the discussions above, there are a number of works concerned with the realism and trustworthiness of risk assessment. These works, however, discuss the contributors to trustworthiness separately: different frameworks cover different aspects of the trustworthiness based on different terminologies. A unified and complete framework that covers all the factors contributing to trustworthiness is lacking. Besides, the current state of the art only focuses on the evaluation of trustworthiness but does not consider how to integrate the trustworthiness into the results of risk assessment, neither does it show how to aggregate the risk of different contributors with different levels of trustworthiness.

In this work, we define the trustworthiness of risk assessment as a metric that reflects the degree of confidence in the background knowledge that supports the PRA, as well as in the suitability, comprehensiveness and completeness of the PRA model formulation and implementation in a way that reflects, to the best possible, reality.

With this, the objective is, then, to provide a new approach for MHRA considering trustworthiness. Compared to the existing works, the contributions of the current work include:

(i) a unified framework is developed for the evaluation of trustworthiness in risk assessment;

(ii) a method is developed to integrate the trustworthiness in the result of the risk assessment of a single hazard group;

(iii) an approach is developed for MHRA considering the trustworthiness of risk assessment.

The rest of this paper is organized as follows. In Section 2, we present a hierarchical framework for assessing the trustworthiness of PRA models and in Section 3 we show how to apply it in practice. In Section 4, we show how to aggregate the risks considering trustworthiness. Section 5 applies the developed methods to a case study from the nuclear industry. Finally, in Section 6, we conclude this paper and discuss the potential future work.

A hierarchical framework for assessing the trustworthiness of a risk model

As illustrated previously, various factors have been discussed in the literature in relation to the trustworthiness of risk assessment. In this paper, we only focus on some of the most relevant factors. For example conservatism, uncertainty, level of sophistication and details in the analysis, experience, number of approximations and assumptions made in the analysis are identified in [START_REF] Epri | Practical Guidance on the Use of Probabilistic Risk Assessment in Risk-Informed Applications with a Focus on the treatment of Uncertanity[END_REF] and [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF] as fundamental factors that influence the realism and trustworthiness of a risk analysis. Background knowledge that supports the risk assessment is also widely accepted as an essential contributor to the trustworthiness [START_REF] Flage | Expressing and communicating uncertainty in relation to quantitative risk analysis[END_REF], (Aven, 2013a), (Aven, 2013b), [START_REF] Epri | Practical Guidance on the Use of Probabilistic Risk Assessment in Risk-Informed Applications with a Focus on the treatment of Uncertanity[END_REF], [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF], [START_REF] Bani-Mustafa | Strength of Knowledge Assessment for Risk Informed Decision Making[END_REF]. The assumptions that are inevitably made because of incomplete knowledge or for simplifying the analysis [START_REF] Kloprogge | A method for the analysis of assumptions in modelbased environmental assessments[END_REF] are considered crucial for the suitability of risk representation and hence, the trustworthiness of its analysis [START_REF] Boone | NUSAP: a method to evaluate the quality of assumptions in quantitative microbial risk assessment[END_REF], [START_REF] Kloprogge | A method for the analysis of assumptions in modelbased environmental assessments[END_REF], [START_REF] De | Assumptions in quantitative analyses of health risks of overhead power lines[END_REF], [START_REF] Berner | Strengthening quantitative risk assessments by systematic treatment of uncertain assumptions[END_REF]. The conservatism is also identified as a pivotal contributor to the realism, maturity, and trustworthiness of risk assessment [START_REF] Aven | On the use of conservatism in risk assessments[END_REF], (Bani-Mustafa et al., 2017). Sensitivity analysis is also needed for a comprehensive description of risk [START_REF] Flage | Expressing and communicating uncertainty in relation to quantitative risk analysis[END_REF], (Bani-Mustafa et al., 2017).

Other factors for evaluating the credibility of M&S include verification, validation, input pedigree, result uncertainty, result robustness, use history, M&S management and people qualification (Nasa, 2013).

The factors mentioned above are included in the trustworthiness assessment framework proposed in this paper.

Other relevant factors are also considered, for a complete representation of trustworthiness. The trustworthiness of risk assessment is defined in this paper as the degree of confidence that the background knowledge is strong enough to support the PRA and that the PRA model is suitable, correctly and robustly made to make the best use of the available knowledge in order to reflect to the best, reality. Obviously, the background knowledge that supports a risk assessment affects significantly the trustworthiness of its results [START_REF] Flage | Expressing and communicating uncertainty in relation to quantitative risk analysis[END_REF], (Aven, 2013a), (Aven, 2013b), [START_REF] Bani-Mustafa | Strength of Knowledge Assessment for Risk Informed Decision Making[END_REF]. However, having a strong background knowledge is not sufficient to ensure the trustworthiness in the results: the fidelity of the modeling should be also verified. This gives rise to the need of a technically adequate and mature model that is known for its high quality and representativeness of reality [START_REF] Oberkampf | Predictive capability maturity model for computational modeling and simulation[END_REF], (Nasa, 2013), [START_REF] Zeng | A hierarchical decision-making framework for the assessment of the prediction capability of prognostic methods[END_REF]. In addition, the modeling process should follow a high quality and thorough application procedure, in order to have trustworthy risk analysis results (IAEA, 2006), [START_REF] Oberkampf | Predictive capability maturity model for computational modeling and simulation[END_REF], [START_REF] Schwer | Guide for Verification and Validation in Computational Solid Mechanics[END_REF]), (Nasa, 2013), [START_REF] Zeng | A hierarchical decision-making framework for the assessment of the prediction capability of prognostic methods[END_REF]. Hence, the suitability of the selected model and the quality of its application are considered as relevant attributes in the proposed framework. In fact, since the risk metrics are calculated as a result of modeling and simulation, it is intuitive to understand that the trustworthiness of the risk assessment results can be affected by: the suitability of the selected model, the comprehensiveness and correctness of the application of the model, as well as the background knowledge that supports the modeling and analysis. Besides, having results that are highly sensitive to changes in the input is an indication that the assessment is less trustworthy, as the results might be dramatically affected by even a small change in the input parameters and assumptions [START_REF] Flage | Expressing and communicating uncertainty in relation to quantitative risk analysis[END_REF], (Bani-Mustafa et al., 2017). Accordingly, the robustness of the results is regarded as another factor that affects the trustworthiness of risk analysis. In this framework, we use the acronym SoK to represent the strength of the background knowledge that supports the risk assessment and the term "modeling fidelity" to represent the suitability of the selected model, the quality of its application and the robustness of the results, as shown in Figure 1.

These two top-level attributes are further decomposed into more tangible sub-attributes.

It should be noted that in general, knowledge includes explicit knowledge, which can be documented and transferred directly, and implicit knowledge, which is possessed by individuals and cannot be documented or transferred directly. The SoK defined in Figure 1 only concerns the explicit knowledge, whereas implicit knowledge is mostly related to the construction and application of the model. Hence, implicit knowledge is viewed as related to the modeling fidelity. The background knowledge is evaluated in [START_REF] Flage | Expressing and communicating uncertainty in relation to quantitative risk analysis[END_REF] considering: (i) availability of reliable data; (ii) phenomenological understanding; (iii) quality and plausibility of assumptions; (iv) agreement among peers. In [START_REF] Bani-Mustafa | Strength of Knowledge Assessment for Risk Informed Decision Making[END_REF], the background knowledge is evaluated by (i) the solidity of assumptions;

(ii) the availability of reliable data; (iii) the understanding of phenomena. Each attribute is further broken down into more tangible sub-attributes that define it. For example, the reliability of data is evaluated by its completeness, consistency, validity, accuracy, and timeliness [START_REF] Bani-Mustafa | Strength of Knowledge Assessment for Risk Informed Decision Making[END_REF].

The quality of assumption is evaluated in the literature by different factors. For example, in an application of Numeral Unit Spread Assessment Pedigree (NUSAP), the quality of assumptions is evaluated by (i) plausibility; (ii) inter-subjectivity peers; (iii) inter-subjectivity stakeholders; (iv) choice space; (v) influence situational limitations;

(vi) sensitivity to view and interests of the analyst (vii) and influence on results [START_REF] Van Der Sluijs | Combining Quantitative and Qualitative Measures of Uncertainty in Model-Based Environmental Assessment: The NUSAP System[END_REF], [START_REF] Boone | NUSAP: a method to evaluate the quality of assumptions in quantitative microbial risk assessment[END_REF], [START_REF] Kloprogge | A method for the analysis of assumptions in modelbased environmental assessments[END_REF]. In this paper, we group these factors into three main categories [START_REF] Bani-Mustafa | Strength of Knowledge Assessment for Risk Informed Decision Making[END_REF]: (i) quality of assumptions; (ii) value-ladenness; (iii) sensitivity. Value ladenness is, in turn, considered as an independent variable that affects the quality of the assumptions and is evaluated using seven main criteria (i) the personal knowledge; (ii) the sources of information; (iii) the non-biasedness; (iv) the relative independence; (v) the past experience; (vi) the performance measure; (vii) the agreement among peers [START_REF] Zio | On the use of the analytic hierarchy process in the aggregation of expert judgments[END_REF], [START_REF] Bani-Mustafa | Strength of Knowledge Assessment for Risk Informed Decision Making[END_REF].

Nevertheless, some of the SoK attributes are more related to the implicit knowledge and affect the construction and formulation of the modeling process and, hence, they are considered under modeling fidelity and not under SoK.

For example, the quality and solidity of assumptions are more related to modeling fidelity, since they affect the formulation of the model. Also, since assumptions are made by experts and inevitably affected by their subjectivity, agreement among peers is considered as a sub-attribute under solidity of assumptions.

In this paper, only the availability of reliable data and phenomenological understanding from [START_REF] Flage | Expressing and communicating uncertainty in relation to quantitative risk analysis[END_REF] are considered for evaluating the SoK. As said earlier, the quality and solidity of assumptions are treated under modeling fidelity. Finally, we add another attribute to cover the data and information related directly to the known hazards. The known potential hazards attributes are next broken down into three sub-attributes that cover: the number of documented known hazards, the accident analysis report and the expert's knowledge about the hazards. The data and phenomenological understanding attributes are further broken into sub-attributes and leaf attributes (illustrated in Figure 1) according to the framework proposed in [START_REF] Bani-Mustafa | Strength of Knowledge Assessment for Risk Informed Decision Making[END_REF].

Other factors related to the suitability of the model and quality of application are also found in the literature.

Examples of these factors are: conservatism, level of sophistication and details in the analysis, experience, number of approximations and assumptions made in the analysis, sensitivity, results robustness, use history, level of details and verification [START_REF] Paté-Cornell | Uncertainties in risk analysis: Six levels of treatment[END_REF], [START_REF] Flage | Expressing and communicating uncertainty in relation to quantitative risk analysis[END_REF], [START_REF] Epri | Practical Guidance on the Use of Probabilistic Risk Assessment in Risk-Informed Applications with a Focus on the treatment of Uncertanity[END_REF], (Nasa, 2013), [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF], [START_REF] Aven | On the use of conservatism in risk assessments[END_REF], (Bani-Mustafa et al., 2017). These attributes are allocated in the hierarchy according to their relevance to the modeling fidelity and categorized in three groups, i.e., suitability of selected model, quality of the application and robustness of the results, whereas other attributes have been added to complement the overall framework for the trustworthiness of the risk assessment. The overall hierarchical framework is presented in Figure 1, and detailed definitions of the attributes, sub-attributes and "leaf" attributes are given in Table 1-

Attribute Definition

Modeling fidelity (𝑀𝐹 = 𝑇 1 )

The degree of confidence that the selected PRA model is technically adequate for describing the problem of interest and that the model is implemented in a trustable way so that the results can reasonably represent reality, relative to the decision making involved

The strength of knowledge

(𝑆𝑜𝐾 = 𝑇 2 )
The amount of high-quality explicit knowledge that is available to support the PRA The technical adequacy of the tool, maturity and ability to model the problem of interest Quality of application (𝑄𝐴𝑝 = 𝑇 1,3 ) The degree to which the analysis is implemented with the minimum required levels of details and modeling adequacy that have the degree of quality, suitable for supporting the application of interest Knowledge of potential hazards and accident evolution processes

(𝑃𝑜𝐻 = 𝑇 2,1 )
The availability of documentation and knowledge of abnormal events, accidents and their evolutions, from similar systems Phenomenological understanding

(𝑃ℎ = 𝑇 2,2 )
The knowledge that supports the comprehension of the system functionality and the related phenomena Data (𝐷 = 𝑇 2,3 )

The amount and quality of data needed for estimating the model parameters 2

Attribute Definition

Model sensitivity (𝑀𝑆 = 𝑇 1,1,1 )

The degree to which the model output varies when one or several parameters change Impact of assumptions (𝐼𝑜𝐴 = 𝑇 1,1,2 ) The degree to which the model output varies when one or several assumptions change Robustness of the model

(𝑅𝑜𝑀 = 𝑇 1,2,1 )
The capability of the model to keep its performance when applied to a different problem settings Suitability of the model for the problem (𝑆 = 𝑇 1,2,2 )

The ability to capture all the important details and characterizations of the problem of interest Historical use (𝐻𝑈 = 𝑇 1,2,3 )

The degree of confidence gained in this method by the long historical usage

Conservatism (𝐶𝑣 = 𝑇 1,3,1 )
The intentional acts for overestimating the risk by making conservative assumptions out of cautiousness

The accuracy of calculations

(𝐴𝑐𝐶 = 𝑇 1,3,2 )
The degree of the voluntarily accepted error in the calculation, e.g., significant figures, simulation errors, and cutoff errors Quality of assumptions (𝑄𝑜𝐴 = 𝑇 1,3,3 )

The degree to which the assumption is valid, representing reality and supporting the model Verification (𝑉𝑟 = 𝑇 1,3,4 )

The degree of assurance that the analysis maintains the requirements of quality control The documented experience on known hazards that might affect the system of interest Availability of accident analysis reports (𝑁𝐻 = 𝑇 2,1,2 )

The availability of technical reports that cover thoroughly the different sequences of any abnormal activity, incident or accident in the time frame and the progressions of each phase Experts knowledge about the hazard

(𝑁𝐻 = 𝑇 2,1,3 )
The undocumented experience possessed by experts on known hazards Years of experience (𝑌𝐸 = 𝑇 2,2,1 )

The amount of experience (measured in years) regarding a specific phenomenon

Number of experts involved (𝑁𝐸 = 𝑇 2,2,2 )
The number of experts who are explicitly or implicitly involved in understanding the phenomena and the risk analysis Academic studies on the phenomena

(𝐴𝐸 = 𝑇 2,2,3 )
The number of academic resources, i.e., articles, books, etc., available about the phenomena of interest Industrial evidence and applications on the phenomena (𝐼𝐸 = 𝑇 2,2,4 )

The number of industrial applications and reports related to the specific phenomena or events of interest Amount of available data

(𝐴𝐷 = 𝑇 2,3,1 )
The amount of data that are needed to evaluate the model parameters Reliability of data (𝑅𝐷 = 𝑇 2,3,2 )

The degree to which the properties of data satisfy the requirements of risk analysis 1 2

Attribute Definition

The plausibility of assumptions

(𝑃𝑙 = 𝑇 1,3,3,1 )
The degree of realism of the statements made in the analysis, in cases of lack of knowledge or to facilitate the problem solution

Value ladenness of assessors (𝑉𝐿 = 𝑇 1,3,3,2 ) The experts' degree of objectivity, professionalism, skills and competencies, past fulfillment of assigned missions and level of achievement Agreement among peers (𝐴𝑔 = 𝑇 1,3,4,1 )

The degree of resemblance between the peers on the analysis and assumptions made, if they were asked to perform the analysis separately Quality assurance (𝑄𝐴 = 𝑇 1,3,4,2 )

The degree of following the standards in the process of implementing the analysis Level of granularity (𝐿𝑜𝐺 = 𝑇 1,3,5,1 )

The depth of analysis and subdivision of the problem constituting elements Number of approximations (𝑁𝑜𝐴 = 𝑇 1,3,5,2 )

The intentional simplifications made to facilitate the modeling Level of details (𝐿𝑜𝐷 = 𝑇 1,3,5,3 )

The degree with which the important contributing factors are captured in the modeling compared to the requirement of the analysis (e.g., the dependency among components)

Completeness (𝐿𝑜𝐷 = 𝑇 2,3,2,1 )
The degree to which the collected data contain the needed information for the risk modeling and assessment

Consistency (𝐿𝑜𝐷 = 𝑇 2,3,2,2 )
The degree of homogeneity of data from different data sources

Validity (𝐿𝑜𝐷 = 𝑇 2,3,2,3 )

The degree to which the data are collected from a standard collection process and satisfy the syntax of its definition (documentation related) Timeliness (𝐿𝑜𝐷 = 𝑇 2,3,2,4 )

The degree to which data correctly reflect the reality of an object or event Accuracy (𝐿𝑜𝐷 = 𝑇 2,3,2,5 )

The degree to which data are up-to-date and represent reality for the required point in time

Evaluation of the level of trustworthiness

In this section, a bottom-up method for evaluating the level of trustworthiness is developed in Section 3.1. Then, a combination of Dempster Shafer Theory (DST) and Analytical Hierarchy Process (AHP) are used in Section 3.2 to determine the weights of the attributes/sub-attributes in the method proposed in Section 3.1.

Evaluation of the trustworthiness

In this framework, five levels of trustworthiness are defined with their corresponding settings:

1. Strongly untrustworthy (𝑇 = 1) : represents the minimum level of trustworthiness and, therefore, the decision maker has the lowest confidence in the result of the PRA. The analysis is made based on weak knowledge and/or nonrealistic analysis, leading to an estimated value that might be far from the real one.

Further analysis and justifications need to be implemented on the risk analysis to enhance its trustworthiness. Otherwise, the risk assessment is not considered representative and one should not rely on its results to support any kind of decision making.

2. Untrustworthy (𝑇 = 2): represents a low level of trustworthiness and, therefore, the decision maker has low confidence in the results of the PRA. At this level, the analysis is made based on relatively weak knowledge and/or nonrealistic analysis, leading to unrealistically estimated risk values. Further analysis and justifications need to be implemented on the risk analysis to enhance its trustworthiness. The decision maker can use the results with caution and only as a support for decision making.

3. Moderately trustworthy (𝑇 = 3): represents a moderate level of trustworthiness and, therefore, the decision maker has an acceptable level of confidence in the results of the PRA. The analysis is made based on relatively moderate knowledge and/or relatively realistic analysis. The decision maker can rely cautiously on the model output to make the decision.

4. Trustworthy (𝑇 = 4): represents a high level of trustworthiness and, therefore, the decision maker has quite high confidence in the results of the PRA. The analysis is made on a relatively high level of knowledge and realistic analysis. The decision maker can rely confidently on the models output to make decisions.

5. Highly trustworthy (𝑇 = 5): represents the maximum level of trustworthiness. At this level, the PRA model outputs accurately predict the risk index with a proper characterization of parametric uncertainty. The decision maker can rely on the models output to support decision making involving severe consequences, e.g., loss of human lives.

In practice, the trustworthiness of risk assessment might be between two of the five levels defined above: for example, 𝑇 = 2.6 means that the level of trustworthiness is between untrustworthy and moderately trustworthy.

In this paper, the level of trustworthiness of risk assessment is evaluated using a weighted average of the "leaf" attributes in Figure 1.

𝑇 = ∑ 𝑊 𝑖 • 𝐴 𝑖 𝑛 𝑖 (1)
where 𝑊 𝑖 is the weight of the leaf attribute that measures its relative contribution to the trustworthiness of risk assessment; 𝐴 𝑖 is the trustworthiness score for the i-th leaf attribute, evaluated based on the scoring guidelines presented in the Appendixes; 𝑛 is the number of the leaf attributes (in Figure 1, we have 𝑛 = 27). The weights 𝑊 𝑖 are determined based on Dempster Shafer-Analytical Hierarchy Process (DST-AHP) [START_REF] Dezert | Multi-criteria decision making based on DSmT-AHP[END_REF], as discussed in Section. 3.2.

Dempster Shafer Theory -Analytical Hierarchy Process (DST-AHP) for trustworthiness attributes weight evaluation

The weights of the different attributes in Figure1 can be determined using the AHP method to compare their relative importance with respect to the trustworthiness of risk assessment [START_REF] Saaty | Decision making with the analytic hierarchy process[END_REF]. AHP is used because it can decrease the complexity of the comparison process, as it allows comparing only two criteria at a time, rather than comparing all the criteria simultaneously, which could be very difficult in complex problems. It should be noted that since there are no alternatives to be compared in this framework, pairwise comparison matrixes of AHP are only used for deriving the attributes (criteria) weights.

To consider the fact that experts are subjective, not fully reliable and might have conflicting viewpoints, as well as considering the incomplete knowledge of the experts, Dempster-Shafer-Analytical Hierarchy Process (DST-AHP) is used. This allows combining multiple sources of uncertain, fuzzy and highly conflicting pieces of evidence with different levels of reliability [START_REF] Dezert | Multi-criteria decision making based on DSmT-AHP[END_REF], [START_REF] Jiao | Combining sources of evidence with reliability and importance for decision making[END_REF]. In this method, the assessors are asked to identify the focal sets that comprise of a single or group of criteria. The experts determine the criteria contained in the focal sets in such a way that they are able to compare them (the focal sets), given their knowledge. Then, pairwise comparison matrices are constructed for the focal sets. Using focal sets instead of single criteria allows taking into account the partial uncertainty between possible criteria. The basic belief assignments (BBA) of the corresponding focal sets are derived from the pairwise comparison matrices. The BBAs from different experts are combined using the Dempster fusion rule. The weights for each criterion are assumed to be BBA of the corresponding focal element (single criterion), and are derived based on the maximum belief-plausibility principle in Dempster-Shafer theory, or on the maximum subjective probability obtained by probabilistic transformations using the transferable belief model [START_REF] Dezert | Multi-criteria decision making based on DSmT-AHP[END_REF], [START_REF] Dezert | Evidential reasoning for multi-criteria analysis based on DSmT-AHP[END_REF], [START_REF] Jiao | Combining sources of evidence with reliability and importance for decision making[END_REF]. Again, note that in this work, this method is applied only to derive the relative weights of the criteria, rather than using it to rank alternatives. Similar ideas have been used in [START_REF] Tayyebi | Combining multi criteria decision making and Dempster Shafer theory for landfill site selection[END_REF], [START_REF] Ennaceur | Handling partial preferences in the belief AHP method: Application to life cycle assessment[END_REF]. The procedure for calculating the weights of the leaf attributes based on DST-AHP is presented below.

I. Constructing pairwise comparison matrices

First, the experts are asked to construct pairwise comparison matrices (also known as knowledge matrices) to compare the relative importance of the attributes and sub-attributes in the same level of the hierarchy with respect to their parent attribute. For example, the pairwise comparison matrix for the attribute modeling fidelity (𝑇 1 ) is a 3 × 3 matrix that compares the relative importance of the modeling' fidelity daughter attributes:

𝑇 1,1 𝑇 1,2 𝑇 1,3 𝑇 1,1 𝑇 1,2 𝑇 1,3 = [ 1 𝑀𝐹 12 𝑀𝐹 13 𝑀𝐹 21 1 𝑀𝐹 23 𝑀𝐹 31 𝑀𝐹 32 1 ]
where the columns correspond to the pairwise comparisons of the daughter attributes: robustness of the results (𝑇 1,1 ), suitability of the selected model (𝑇 1,2 ), and quality of the application (𝑇 1,3 ), respectively. The element 𝑀𝐹 𝑖𝑗 is assigned by assessing the relative importance of attribute 𝑖 to attribute 𝑗 following the scoring protocols in [START_REF] Saaty | Decision making with the analytic hierarchy process[END_REF]. For example, the element 𝑀𝐹 12 is assigned by comparing the relative importance of 𝑇 1,1 𝑡𝑜 𝑇 1,2 .

Compared to conventional AHP comparison matrices, the expert is free to choose, based on his/her belief, the elements of the pairwise comparison matrix. These elements can be focal elements that represent a single criteria, e.g., {𝐴} or a distinct group of criteria, e.g., {𝐴, 𝐵} that are comparable favorably (to the best of expert's knowledge)

to the universal set that contains all the criteria, which allows accounting for the uncertainty in the judgment [START_REF] Beynon | An expert system for multi-criteria decision making using Dempster Shafer theory[END_REF], [START_REF] Ennaceur | Handling partial preferences in the belief AHP method: Application to life cycle assessment[END_REF], [START_REF] Jiao | Combining sources of evidence with reliability and importance for decision making[END_REF]. For example, the expert can choose a focal set of {𝑆𝑜𝑀, 𝑄𝐴𝑝} if he/she believes that it can be compared favorably to the universal set {𝑆𝑜𝑀, 𝑄𝐴𝑝, 𝑅𝑜𝑅}; i.e., the set of {𝑆𝑜𝑀, 𝑄𝐴𝑝} can be compared to {𝑆𝑜𝑀, 𝑄𝐴𝑝, 𝑅𝑜𝑅} (the sub-attributes SoM, QAp, RoR were defined in Table 1-4). Then, the expert is asked to fill the pairwise comparison matrices to represent his/her belief in the relative importance of a given set (of one or multiple attributes) compared to the others. Favoring the universal set {𝑆𝑜𝑀, 𝑄𝐴𝑝, 𝑅𝑜𝑅} over {𝑆𝑜𝑀, 𝑄𝐴𝑝}, means that the universal set contains an element that is not contained in the other set, and at the same time it is more important than the elements of the other set, i.e., 𝑅𝑜𝑅 is more important than 𝑆𝑜𝑀 and 𝑄𝐴𝑝. Finally, as in the conventional AHP method, the consistencies of the matrixes need to be tested and the assessors are asked to update their results if the consistency is lower than the required value [START_REF] Saaty | Models, methods, concepts & applications of the analytic hierarchy process[END_REF].

II. Computing the weights

In this step, the weights are derived using the conventional AHP technique, according to which the normalized principal eigenvector of the matrix represents the weights. A good approximation for solving the eigenvector problem in case of high consistency is to normalize the columns of the matrix and, then, average the rows for obtaining the weights. For more details on AHP and deriving the weights from pairwise comparison matrices, the reader might refer to [START_REF] Saaty | Analytic hierarchy process[END_REF]. Please note that, as mentioned earlier, the weights derived from the pairwise comparison matrices are assumed to be the BBA of the associated focal sets.

III. Reliability discounting

Usually, multiple experts are involved in evaluating the weights. Each expert is regarded as an evidence source.

Reliability of an evidence source represents its ability to provide correct measures of the considered problem [START_REF] Jiao | Combining sources of evidence with reliability and importance for decision making[END_REF]). Shafer's reliability discounting is often used to consider the reliability of the source information in DST-AHP [START_REF] Shafer | A mathematical theory of evidence[END_REF]:

𝑚 𝛿 (𝐴) = { 𝛿 • 𝑚(𝐴) ∀𝐴 ⊆ Θ, A ≠ Θ (1 -𝛿) + (𝛿) • 𝑚(Θ), A = Θ , 𝛿 ∈ [0,1] (2)
where Θ represents the complete set of criteria, 𝐴 is the focal element in the power set 2 Θ , 𝑚(𝐴) is the BBA for 𝐴, 𝑚 𝛿 (𝐴) is the discounted BBA, 𝛿 is the reliability factor. A value of 𝛿 = 1 means that the source is fully reliable and a value of 𝛿 = 0 means that the source is fully unreliable. The reliability factor of the experts is determined by the decision maker, based on their previous knowledge and experience.

IV. Combination of experts opinions

Next, Dempster's rule of combination [START_REF] Shafer | A mathematical theory of evidence[END_REF]) is used to combine two independent pieces of evidence assigned by different experts. The discounted BBAs from different experts are combined by [START_REF] Jiao | Combining sources of evidence with reliability and importance for decision making[END_REF]:

𝑚 1,2 𝛿 (𝐶) = (𝑚 1 𝛿 ⊕ 𝑚 2 𝛿 )(𝐶) = { 0 𝐶 = 𝜙, 1 1-𝐾 • ∑ 𝑚 1 𝛿 (𝐴) • 𝑚 2 𝛿 (𝐵) 𝐴∩𝐵=𝐶≠𝜙 𝐶 ≠ 𝜙, (3)
where 𝑚 1,2 𝛿 (𝐶) is the new BBA resulting from the combination of the two discounted BBA 𝑚 1 𝛿 (𝐴) and 𝑚 2 𝛿 (𝐵) of the two experts. 𝐾 is the conflict factor in the opinions of experts and given by:

𝐾 = ∑ 𝑚 1 𝛿 (𝐴) • 𝑚 2 𝛿 (𝐵) 𝐴∩𝐵=𝜙 (4)
V.

Pignistic probability transformation

The belief functions resulted from the discounting and combination are defined for focal sets (might contain one or multiple leaf attributes). To obtain the weights of each leaf attribute, the masses (𝑚 1,2 𝛿 (𝐶)) assigned to the focal sets need to be transformed into masses for the basic elements. In this paper, the transferable belief model proposed by [START_REF] Smets | The transferable belief model[END_REF] is used for the transformation. In this method, the masses 𝑚 1,2 𝛿 (𝐶) on the credal level are converted to the pignistic level using the insufficient reason principle [START_REF] Smets | The transferable belief model[END_REF], [START_REF] Aregui | Constructing consonant belief functions from sample data using confidence sets of pignistic probabilities[END_REF]:

𝑤(𝑥) = ∑ 𝑚(𝐶) 1-𝑚(𝜙) 1 𝐶(𝑥) |𝐶| , ∀𝑥 ∈ 𝛩 𝐶⊆𝛩,𝐶≠𝜙 (5)
where 𝑤(𝑥) denotes the belief assignment of a single element (𝑥) on the pignistic level, 1 𝐶 is the indicator function of 𝐶: 1 𝐶 = 1, 𝑖𝑓 𝑥 ∈ 𝐶 𝑎𝑛𝑑 0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒. |𝐴| is the length of A (the number of elements in the focal set).

The mass functions obtained from the pignistic probability transformation represent the relative "believed weights" of the attributes.

After obtaining the local weights of the leaf attributes with respect to their parent attribute, the global weights with respect to the top-level attribute, i.e., the trustworthiness, need to be determined. This can be done by multiplying the weight of the daughter attribute by the weights of the upper parent attributes in each level. For example, the "global weight" of the historical use with respect to the trustworthiness, denoted by 𝑊 𝑔𝑙𝑜𝑏𝑎𝑙 (𝐻𝑈), is calculated by:

𝑊 𝑔𝑙𝑜𝑏𝑎𝑙 (𝐻𝑈) = 𝑤(𝐻𝑈) × 𝑤(𝑆𝑜𝑀) × 𝑤(𝑀𝐹)
where 𝑤(𝐻𝑈), 𝑤(𝑆𝑜𝑀) 𝑎𝑛𝑑 𝑤(𝑀𝐹) are the local weights of the historical use, the suitability of the model, and the modeling fidelity. For simplicity reasons, hereafter the global weights for the leaf attributes are denoted by 𝑊 𝑖 and in the framework of Figure 1, we have 𝑖 = 1,2, ⋯ ,27.

Evaluation of the risk considering trustworthiness levels

In this section, the "weighted posterior" method [START_REF] Groen | Behavior of weighted likelihood and weighted posterior methods for treatment of uncertain data[END_REF] is used for integrating the risk index with the trustworthiness of the PRA for a single hazard group (Section 4.1). In Section 4.2, a structured methodology is developed for determining the weights in the Bayesian "weighted posterior" model. Finally, MHRA considering the level of trustworthiness is discussed in Section 4.3.

Evaluation of the risk of a single hazard group

After evaluating the level of trustworthiness for the PRA of a given hazard group, the next question is how to integrate the estimated risk from the PRA with the level of trustworthiness. In this paper, we develop a Bayesian averaging model for integrating the trustworthiness based on the "weighted posterior" method [START_REF] Groen | Behavior of weighted likelihood and weighted posterior methods for treatment of uncertain data[END_REF]. Let us consider two scenarios: the risk assessment is trustable, denoted by 𝐸 𝑇 , and its complement, i.e., the risk assessment is not trustable (𝐸 𝑁𝑇 ). The risk after the integration can, then, be calculated as:

𝑅𝑖𝑠𝑘|𝑇 = 𝑃(𝐸 𝑇 ) • Risk|𝐸 𝑇 + (1 -𝑃(𝐸 𝑇 )) • Risk|𝐸 𝑁𝑇 ( 6
)
where 𝑅𝑖𝑠𝑘|𝑇 is the estimation of risk after considering the trustworthiness of the PRA; 𝑃(𝐸 𝑇 ) is the subjective probability that 𝐸 𝑇 will occur and is dependent on the trustworthiness of the risk assessment; Risk|𝐸 𝑇 is the estimated risk from the PRA. Due to the presence of epistemic (parametric) uncertainty in the analysis, Risk|𝐸 𝑇 is often expressed as a subjective probability distribution of the risk index. Risk|𝐸 𝑁𝑇 is an alternate distribution of the risk when the decision maker thinks the PRA is not trustable. In this paper, we assume Risk|𝐸 𝑁𝑇 is a uniform distribution in [0,1], indicating no preference on the value of the risk index. Similar models have been used in literature to consider unexpected events in risk analysis [START_REF] Kaplan | On the quantitative definition of risk', Risk analysis[END_REF]. For example, [START_REF] Kazemi | Improving default risk prediction using Bayesian model uncertainty techniques[END_REF] developed a similar model to calculate the default risk in similar scenarios considering the unexpected events.

The following steps summarize how to use Eq. ( 6) to evaluate the risk given the trustworthiness of the risk assessment:

i.

The risk distribution Risk|𝐸 𝑇 is evaluated for each hazard group using conventional PRA considering the parametric uncertainty propagation.

ii.

The level of trustworthiness of PRA of the corresponding hazard group is assessed, using the procedures in Section 3.

iii.

The subjective probability of trusting the PRA is determined by the detailed procedures described in Section 4.2.

iv.

The level of trustworthiness is integrated in the risk using Eq. ( 6).

Determining the probability of trusting the PRA

The probability 𝑃(𝐸 𝑇 ) in Eq. ( 6), which represents the decision maker's belief that the risk assessment results are correct and accurate, needs to be elicited from the decision makers. The elicitation process needs to be organized and structured to ensure the quality of the elicitation.

Different methods can be found in the literature for the assessment of a single probability using experts elicitation, such as probability wheels, lotteries betting, etc. [START_REF] Jenkinson | The elicitation of probabilities: A review of the statistical literature[END_REF]. In this work, we choose the "certainty equivalent gambles" for the elicitation. Before presenting the procedure for this method, some general recommendations need to be followed to ensure the quality of the elicitation process [START_REF] Jenkinson | The elicitation of probabilities: A review of the statistical literature[END_REF]:

i.

Background and preparation: uncertain events need to be defined clearly.

ii.

Identification and recruitment of experts: The experts who are conducting the elicitation are chosen carefully with low-value ladenness, and a preference of being both substantively and normatively skilled.

iii.

Motivating experts: the purpose and use of the work need to be explained to the experts, to motivate them for the elicitation.

iv.

Structuring and decomposition: the dependencies and functional relationships need to be first identified by the client and agreed on and modified by the experts if necessary.

v.

Probability and assessment training: the experts need to be trained to elicit probabilities.

vi.

Probability elicitation and verification: the expert needs to elicit the probabilities paying caution to zero values, cognitive biases, etc. After making the elicitation, the expert needs to make a summary of the elicitation and verify its adequacy.

Then, a "certainty equivalent gamble" is designed to elicit the probability of trust:

i.

The elicitor informs the decision maker about the definition of the different levels of trustworthiness and their physical meaning, based on the definitions in Section 3.1.

ii.

The decision maker is asked to compare two scenarios: (1) he/she participates in a gamble (given the information from the PRA model) where he/she wins $1,000 if an accident occurs and $0 if the accident does not occur;

(2) he/she wins $𝑥 for sure.

iii.

The experts exchange information between them and discuss.

iv. Suppose that a PRA was conducted and predicted that the consequences occur for sure, and the trustworthiness of the PRA is one of the five levels defined in Section 3.1. Then, for each level of trustworthiness, the elicitor varies the value of 𝑥 until the decision maker feels indifferent between the two scenarios.

v.

The probability of trust at the current level of trustworthiness is, then, calculated by:

𝑝 = 𝑥 1000 (7)
where 1000 here represents the $1000 that the expert gains if the accident occurs (the model prediction is correct).

vi.

The elicitor fits a suitable function to the five data points, in order to determine the probability of trust for trustworthiness levels between the defined levels. The shape of the fitted function should be determined based on the assessors' behavior towards taking risk in trusting a low fidelity PRA:

• A convex function should be chosen if the assessor is risk-averse, meaning that the decision maker trusts only the PRA with high levels of trustworthiness.

• A linear function is chosen if the assessor is risk neutral.

• A concave function is chosen if the assessor is risk-prone, meaning that although a PRA might not have a very high level of trustworthiness, the decision maker is willing to assign a high probability of trust to it.

The risk assessor can eventually use this function to estimate the probabilities of trust for each hazard group.

MHRA considering trustworthiness levels

The main steps for MHRA considering trustworthiness are presented in Figure 2. Trustworthiness in the PRA of each single group is evaluated and integrated into the risk estimate for the corresponding hazard group first. After the integration, the risk is expressed as a subjective distribution on the probability that a given consequence will occur.

Then, the estimated risk from different hazard groups is aggregated. This step can be done by simply adding the risk distributions from different hazard groups, as shown in Eq. ( 8), where 𝑅𝑖𝑠𝑘 𝑡𝑜𝑡𝑎𝑙 is the total risk considering the level of trustworthiness; (𝑅𝑖𝑠𝑘 𝑖 |T) is the risk from the hazard group 𝑖 given the level of trustworthiness; 𝑛 is the number of hazard groups. Monte-Carlo simulations can be used to approximate the distribution of 𝑅𝑖𝑠𝑘 𝑡𝑜𝑡𝑎𝑙 .

𝑅𝑖𝑠𝑘 𝑡𝑜𝑡𝑎𝑙 = ∑ (𝑅𝑖𝑠𝑘 𝑖 |T)

𝑛 𝑖=1

(8)

Case study

In this section, we apply the developed framework to a case study for two hazard groups in the nuclear industry:

the external flooding and internal events hazard groups. The PRA models of the two hazard groups were developed and provided by Electricité De France (EDF) [START_REF] Bani-Mustafa | Strength of Knowledge Assessment for Risk Informed Decision Making[END_REF]. The level of trustworthiness is, then, assessed for each hazard group (Section 5.2). The risk distributions from each hazard group are, then, recalculated considering the level of trustworthiness. Finally, the risk is aggregated from the two hazard groups (Section 5.3).

Description of the PRA model

The two hazard groups considered in this framework are external flooding and internal events. The external flooding refers to the overflow of water that is caused by naturally induced hazards such as river overflows, tsunamis, dam failures and snow melts (IAEA, 2003), (IAEA, 2011). The internal events refer to any undesired event that originates within the NPP and can cause initiating events that might lead to abnormal states and eventually, a core meltdown [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF]. Examples of internal events include structural failures, safety systems operation and maintenance errors, etc. (IAEA, 2009).

In risk analysis of NPP in general, risk analysis of different hazard groups are performed on the basis of PRA models for the internal events, which considered as relatively mature and realistic compared to other hazards groups.

In external flooding hazard group, it is usually difficult to assess the probability of the flood hazard, especially that no reliable method is available for such a kind of analysis. For example, statistical models might be used to extrapolate external flooding frequencies from historical data. However, Only limited data are usually available on flooding and corresponds to few hundreds of years (usually 100-150 years). These data are used to extrapolate the flooding frequencies on different time interval, which, especially in extreme cases (where no data are available), would result

Single hazard group risk assessment considering trustworthiness

Evaluate the trustworthiness of the PRA of the corresponding hazard group following the procedures in Sect. 3.

Assess the degree of belief (weight) in the model following the procedure in Sect.

4.

Risk assesmnet considering the trustworthiness based on Eq. ( 7)

Aggergate the risk from the different hazard groups in large uncertainty [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF].

One of the most challenging point in external flooding risk analysis that the frequency and severity of each flood is site-specific, which would reduce the applicability of data from other sites. Also, the response and of NPP staff, in cases of floods, cannot be assessed easily as there are many factors that could affect their actions. However, it should be noted that it is highly recommended by regulatory bodies to perform some deterministic approaches for analyzing the floods hazards [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF].

In this case study, the risk analysis is provided by EDF [START_REF] Bani-Mustafa | Strength of Knowledge Assessment for Risk Informed Decision Making[END_REF], in which bow-tie models are used to assess the probability of core damage frequency (CDF) (in 1/reactor•year). These large and complex models are of the order of hundreds to thousand basic events and several hundreds of minimal cutsets considering the different hazards that could lead to loss of system and consequently core damage.

Let's take the external flooding hazard group as an example. In this hazard group, the external flooding is considered as the hazard leading to the initiation of events within the plant that would possibly result in a core damage.

The model is constructed considering the different equipment and systems that could be affected by water flooding in the NPP at different water heights. The different scenarios of the water arrival at the platform (with different heights) are built and propagated to understand their effect on the core. In this study, the probability of losing an equipment is calculated assuming that the equipment is directly lost once the water reaches the bottom of the equipment. In other words, the probability of losing an equipment equals to the probability that the level of water at the platform reaches the bottom of equipment.

The probability of having different water levels due to floods using a combined hydraulic/hydrologic method.

First, data regarding the topography, hydrological and physical characteristic of the river basin were collected from the site of the NPP of interest. These data, allows calculating the water flowrate needed to obtain a specific water height at the platform of the NPP. Then, the data of the millennial flowrates of the river, were used to extrapolate to calculate the "return period" (average time needed for a river flood to occur) and then, extrapolate it to assess the frequencies of river flowrates on which no data are available. In other words, the data regarding the flowrate frequencies and the physical and hydrological nature of the basin, allow evaluating the frequencies of having given heights of water at the platform of the reactor. Therefore, it allows calculating the probability of equipment failures due to water flooding. Other intermediate events are also presented in the PRA models to represent the propagation of the initiating events and the different possible responses from the safeguard systems or the operators till the reactor core meltdown [START_REF] Bani-Mustafa | Strength of Knowledge Assessment for Risk Informed Decision Making[END_REF].

In the original work of EDF, the uncertainty propagation was implemented, but only the mean values of the probability distributions of the risk were considered in MHRA and used for comparison to the safety criteria. However, due to confidentiality reasons, real values cannot be presented. Instead, we disguise the risk distribution, considering also the parametric uncertainty for illustration purposes, as shown in Figure 3.

Evaluation of level of trustworthiness

Evaluation of the attributes weights

As illustrated in Section 3, the first step for evaluating the level of trustworthiness is to determine the relative importances (weights) of the trustworthiness attributes. The weights of the attributes are evaluated using the DST-AHP technique. Here, for explanation purposes, the sub-attribute "modeling fidelity" (𝑇 1 ) is taken as an example to illustrate how to obtain local weights through pairwise comparisons and DTS-AHP.

I. Constructing pairwise comparison matrices

As shown in Section 3, the first step in the DST-AHP technique is to construct the pairwise comparison matrix.

Take the daughter attributes of modeling fidelity as an example. In this example, a 4 × 4 pairwise comparison matrix is constructed in Table 5.

Table 5 Pairwise comparison matrix (knowledge matrix) for comparing modeling fidelity "daughter" attributes Modeling fidelity

{𝑇 1,1 } {𝑇 1,2 } {𝑇 1,3 } Θ = {𝑇 1,1 , 𝑇 1,2 , 𝑇 1,3 } {𝑇 1,1 } 1 0 0 1/2 {𝑇 1,2 } 0 1 0 5/2 {𝑇 1,3 } 0 0 1 4 {𝑇 1,1 , 𝑇 1,2 , 𝑇 1,3 } 2 2/5 1/4 1
Please note that the zeros that appear in the matrix indicate that there is no need to compare the individual criteria directly: they are compared indirectly through comparing the individual criteria to the universal set Θ [START_REF] Dezert | Multi-criteria decision making based on DSmT-AHP[END_REF].

𝑇 1,1 represents the Quality of application, 𝑇 1,2 represents the Suitability of the model, 𝑇 1,3

represents the robustness of the results

In this matrix, the expert has considered four groups of focal sets: three for individual criteria and one containing all the criteria in order to consider the uncertainty in the evaluation. Choosing focal sets like this means that to the best of their knowledge, the experts believe that the aforementioned focal sets can be favorably compared to the universal set Θ.

II. Computing the weights

In the previous example, the expert was asked to fill the pairwise comparison matrix to express his/her preference of a criterion over another. In this step, the weights of the focal sets are derived using the conventional AHP technique, where the normalized principal eigenvector of the matrix represents the weights. This can be directly done by normalizing each column in the matrix individually and, then, averaging the elements in each row to obtain that weight.

III. Reliability discounting

After computing the BBA for each expert matrix, the weights need to be discounted based on the reliability of each expert. For illustration purposes, the reliability 𝛿 of the expert who made the assessment is assumed to be 0.60.

From Eq. ( 2), the discounted weights are found as the following:

𝑚 0.60 (𝑇 1,1 ) = 0.6 × 0.10 = 0.06

Similarly, for 𝑚 0.60 (𝑇 1,2 ) = 0.16, & 𝑚 0.60 (𝑇 1,3 ) = 0.19.

Finally, 𝑚 0.60 (Θ) is found as the following:

𝑚 0.60 (Θ) = (1 -0.60) + 0.6 × 0.32 = 0.59

Please note that the BBAs (weights) sum to one before and after the discounting.

IV. Combination of experts opinions

In this case study, three experts have been invited to evaluate the weights; their assigned BBAs are summarized in Table 7 (the BBAs are calculated following the steps in Section 3.2). The combination of the experts judgments is conducted sequentially. Table 8 shows the procedures for combining the judgments of the first two experts.

𝑚 𝛿 (𝑇 1,1 ) 𝑚 𝛿 (𝑇 1,1 ) 1 𝜙 1 𝜙 2 𝑚 𝛿 (𝑇 1,1 ) 2 𝑚 𝛿 (𝑇 1,2 ) 𝜙 3 𝑚 𝛿 (𝑇 1,2 ) 1 𝜙 4 𝑚 𝛿 (𝑇 1,2 ) 2 𝑚 𝛿 (𝑇 1,3 ) 𝜙 5 𝜙 6 𝑚 𝛿 (𝑇 1,3 ) 1 𝑚 𝛿 (𝑇 1,3 ) 2 𝑚 𝛿 (𝑇 1,1 , 𝑇 1,2 , 𝑇 1,3 ) 𝑚 𝛿 (𝑇 1,1 ) 2 𝑚 𝛿 (𝑇 1,3 ) 2 𝑚 𝛿 (𝑇 1,3 ) 2 𝑚 𝛿 (𝑇 1,1 , 𝑇 1,2 , 𝑇 1,3 ) 1
*Please note that the element 𝑖𝑗 in the Table represent the multiplication of the elements 1𝑗 × 𝑖1 , e.g., 𝑚 𝛿 (𝑇 1,1 ) × 𝑚 𝛿 (𝑇 1,1 ) = 𝑚 𝛿 (𝑇 1.1 ) 1 𝑚 𝛿 (𝑇 1,1 ) × 𝑚 𝛿 (𝑇 1,1 , 𝑇 1,2 , 𝑇 1,3 ) = 𝑚 𝛿 (𝑇 1.1 ) 2 From Eq. ( 4), 𝐾 = 0,17.

From Eq. (3):

𝑚 1,2 𝛿 (𝑇 1,3 ) = 0,26 1 -0.17 = 0.31

The same steps are repeated for the other mass functions and presented in Table 9. Finally, the new results obtained from the combination of the two experts are further recombined with the BBAs from the third matrix. The results are presented in Table 9.

V. Pignistic probability transformation

Then, the pignistic mass function is found by Eq. ( 5):

𝑤 1,2,3 𝛿 (𝑇 1,1 ) = 𝑚 1,2,3 𝛿 (𝑇 1,1 ) + 𝑚 1,2,3 𝛿 (𝑇 1,1 , 𝑇 1,2 , 𝑇 1,3 ) 3 = 0.05 + 0.06 3 = 0.07
The steps are repeated for the other mass functions and found to be:

𝑤 1,2,3 𝛿 (𝑇 1,2 ) = 0.42 𝑤 1,2,3 𝛿 (𝑇 1,3 ) = 0.51
Note that the three mass functions on the pignistic level sum to one. These pignistic mass functions represent the relative "believed weights" of the three criteria under modeling fidelity after the reliability discounting and transformation. The same steps are repeated for all the criteria. Then, the weights need to be evaluated with respect to the top-level goal: the trustworthiness. As illustrated previously, this can be done easily by multiplying the weight of the daughter attribute by the weight of the upper parent attributes in each level. For simplicity reasons, only the weights of the "leaf" attribute with respect to the top level attribute i.e., trustworthiness, are presented in Tables 10 and11 (see Section 5.2.2). Note that the weights of the 27 leaf-attributes with respect to the top goal sum to one

∑ 𝑊 𝑖 = 1 27 𝑖=1
.

Evaluation of the attributes scores

The next step is to evaluate the attributes score for the hazard group, given the scoring guidelines in Appendixes A-B. Some information regarding the risk assessment process is extracted from the PRA report to support the trustworthiness assessment:

• The heights (water levels) at the plant's platform at which the water can lead to a failure of a specific element were defined.

• The water flowrate that would result in a given water height at the NPP platform in a defined interval of time was predicted.

• The flow-rate was multiplied by a safety factor of 130%.

• The "return period" for each flowrate was obtained from the data of the millennial flooding flowrate of the river of interest and the data were extrapolated to assess the frequencies of extreme flowrates.

• The river flooding is considered as a predictable phenomenon and the probability of failure of transition into the emergency state (i.e., normal shutdown and cooling with steam generator, residual heat removal system, etc.) is assumed to be the intrinsic probability of failure.

• It is assumed that river overflow is the only source of external flooding.

• A combined hydraulic/hydrologic method is adopted, given the special hydrological and physical characteristics of the basin.

• It is assumed that once the water reaches the bottom of the equipment, the equipment fails.

• It is assumed that failing to close the valves (ensuring the volumetric protection sealing-water proofing)

causes the total loss of Emergency Feedwater System (EFWS).

• It is assumed that clogging inevitably occurs if the flooding occurs.

• The analysis and model calculation for this hazard group is taken with a specific cutoff error of 10 -14 .

Based on the excerptions from the report, it can be seen that:

• In this example, the risk analysis and assessment steps follow the IAEA recommendations.

• The calculation of flowrates and flow frequencies are calculated using solid deterministic models.

However, extrapolation of the data to obtain the frequencies of floods with extreme flowrates is still doubtful.

• The river overflow is a predictable phenomenon and does not happen suddenly. However, the river overflow is not the only source of flooding. For example, a rupture in the river dikes might also lead to sudden, unpredictable flooding.

• The application of a combined hydraulic/hydrologic method on the flooding studies of nuclear sites allows a more realistic evaluation of the flooding level and to estimate more precisely the return periods.

• The assumption that the water will fail the equipment directly if it touches its bottom level is conservative.

• Feedback data show that clogging due to river flooding has occurred before in the nuclear industry (see, for example, USNRC General Electric Advanced Technology Manual for more information (NRC, 2011)). However, claiming that each flooding would surely lead to clogging is still questionable and needs to be studied in details, taking into account the different influencing parameters (hydraulic, geometrical and topographical properties) of the area (see [START_REF] Gschnitzer | Towards a robust assessment of bridge clogging processes in flood risk management[END_REF]).

• In case of failing to close the valves ensuring the volumetric protection, the probability that water will go back through the drainage system is not identified and assumed to be one (𝑃 = 1), though there are no relevant calculations. Moreover, once the water enters the physical protection locations, the safetyrelated equipment is assumed to be lost. Both assumptions are conservative to increase the safety margin.

Based on the above observations, the leaf attributes in Figure 1 can be evaluated. For example, quality assurance attribute is evaluated to be five (𝑇 1,3,4,2 = 5), since the PRA is conducted following the IAEA recommendations.

The accuracy of the calculation is evaluated to be five (𝑇 1,3,2 = 5), since the cutoff error is apparently very low. The combined hydraulic/hydrologic models used for the flooding studies are able to capture the special hydrological and physical characteristics of the basin, which makes them suitable for the study. Hence, a score of four (𝑇 1,2,2 = 4) is given for the suitability of the model. The assumptions presented above are mostly conservative and unrealistic.

Therefore, a score of one (𝑇 1,3,3,1 = 1) is given for the plausibility of the assumptions. The other attributes are scored in the same way. The results are represented in Tables 10 and11. The level of trustworthiness for the external flooding is, then, calculated by Eq. ( 1): The trustworthiness for internal events hazard group (𝑇 𝑖𝑛𝑡 ) was calculated in the same way and, the result is 𝑇 𝑖𝑛𝑡 = 4.414. These results confirm the expectations that the PRA for internal events is considered relatively mature and well established [START_REF] Epri | An Approach to Risk Aggregation for RisK-Informed Decision-Making[END_REF] in contrast to the PRA of external hazards, which is considered less mature with several limitations (EPRI, 2012).

𝑇 𝑒𝑥𝑡 = ∑ 𝑊 𝑖 • 𝐴 𝑖 27 𝑖=1 = 3.260.

Risk assessment considering the level of trustworthiness

Determining the probability of trust in the PRA results

In this step, the decision maker is asked to assign a probability that represents the belief that the risk assessment model output is correct (hereafter called probability of trust), based on the certainty equivalent approach presented in Section 4.2. In this example, we assume that the decision maker exerts a risk-prone behavior and generates the results in Table 12. The data in Table 12 are extrapolated and fitted to a function, as shown in Figure 4. Then, the probability that the decision maker trusts each hazard group PRA given their trustworthiness is calculated from the fitted model in Figure 4. The probability of trust for the external flooding 𝑝 𝑒𝑥𝑡 is found to be 𝑝 𝑒𝑥𝑡 = 0.783. The probability of trust for the internal events 𝑝 𝑖𝑛𝑡 is found to be 𝑝 𝑖𝑛𝑡 = 0.957.

Risk assessment of a single hazard group considering the level of trustworthiness

The level of trustworthiness is integrated with the PRA results for both hazard groups following Eq. ( 6). The results are presented in Figures 5 and6, respectively. As illustrated in Figure 5, the mean risk value considering the trustworthiness is 1.088 × 10 -1 (𝑟𝑒𝑎𝑐𝑡𝑜𝑟 • 𝑦𝑒𝑎𝑟) -1 for external flooding compared to 1.589 × 10 -6 (𝑟𝑒𝑎𝑐𝑡𝑜𝑟 • 𝑦𝑒𝑎𝑟) -1 without considering the level of trustworthiness. For internal events, the mean risk value is 2.149 × 10 -2 (𝑟𝑒𝑎𝑐𝑡𝑜𝑟 • 𝑦𝑒𝑎𝑟) -1 considering the trustworthiness compared to 3.322 × 10 -8 (𝑟𝑒𝑎𝑐𝑡𝑜𝑟 • 𝑦𝑒𝑎𝑟) -1 without considering it for internal events, as illustrated in Figure 6. It can be seen from the Figures that considering the level of trustworthiness will lead to a larger spread out of the probability distribution of the risk. For further explanation, let's take Figure 5 as an example. In panel (a), which represents the risk analysis considering the parametric uncertainty propagation, the spread-out of the risk distribution is limited to the risk interval [4.626 × 10 -11 , 7.738 × 10 -6 ] . On the other hand, the interval of the risk distribution increases to [3.019 × 10 -6 , 2.169 × 10 -1 ] when the level of trustworthiness is considered in the risk analysis (see panel (b)).

This comes out as a result of accounting for the disbelief in the risk analysis that reflects the ignorance about the real value of risk. Hence, the spread of the risk distribution becomes wider, leading to a higher mean value of the risk In other words, real values of risk can fall in reality in ranges of risk wider than that obtained by the initial analysis and does not consider the level of trustworthiness.

Multi-Hazards risk aggregation

Finally, the overall risk given the level of trustworthiness can be calculated using Eq. ( 8). The results are presented in Figure 7. The empirical probability density function of the risk is evaluated through a Monte-Carlo simulation of 10 5 samples. As a comparison, the MHRA is also conducted using the conventional methods by adding the risk indexes from the two hazard groups directly, without considering the trustworthiness, as shown in The mean value of the total risk from the two hazard groups considering the level of trustworthiness is found to be 1.303 × 10 -1 (𝑟𝑒𝑎𝑐𝑡𝑜𝑟 • 𝑦𝑒𝑎𝑟) -1 compared to 1.622 × 10 -6 (𝑟𝑒𝑎𝑐𝑡𝑜𝑟 • 𝑦𝑒𝑎𝑟) -1 without considering the level of trustworthiness. As discussed earlier, the aggregation of the risks from the two hazard groups needs to consider the different levels of trustworthiness to yield a mathematically appropriate process and a physically meaningful results. In fact, considering the level of trustworthiness in the analysis means that we are accounting for the disbelief, shortcoming, and lack of knowledge in the analysis, which leads to a broader spread-out of the distributions and a larger risk interval. The increase of the interval, in which the risk can fall, represents in fact a more realistic risk analysis as it accounts for the ignorance in the model. The increase in the spread out of probability distribution of risk leads to a higher mean value of risk, as it takes into account the fact that the PRA models of the two hazard groups are based on different levels of trustworthiness. finally, leaf attributes. The total trustworthiness is calculated using a weighted average of the attributes, where the weights are calculated using DST-AHP method. Where AHP method is used to calculate the relative weights of the attributes using experts elicitations, whereas DST method is used to account for the uncertainty in their elicitations.

A MHRA method is, then, developed to aggregate the risk from different hazard groups with different levels of trustworthiness, based on a "weighted posterior" method. An application to a case study of a NPP shows that the developed method allows aggregating risk estimates with different degrees of maturity and realism from different risk contributors.

The current framework represents a systematic way for enhancing the risk assessment and representing a mathematically more appropriate risk aggregation process. This is done by considering the different levels of realism on which the risk analyses of the aggregated hazard groups are based and integrating it in the risk analysis. From a practical point of view, the framework is developed in systematic and practical, procedural steps that facilitate the application of the framework to real life cases. In addition, it represents an illuminating point to better inform riskbased decision making, as it represents the degree of realism of the analysis.

However, a possible of the framework that DST is used only to account for the uncertainty in the experts' elicitations of the relative weights of the attributes and not the scores. Therefore, further studies need to be conducted to integrate DST method to also account for the uncertainty in the evaluation of the attributes scores of each given model.

Also, another possible possible limitation of the framework that the conventional safety criteria cannot be directly applied to the new risk estimates considering the trustworthiness. Therefore, future work is further needed for developing new safety criteria that corresponds with the new risk estimates that consider the trustworthiness to better inform the decision maker. Completeness of data refers to the degree to which the collected data contains the needed information. For components and systems, data completeness is characterized by the following criteria (IAEA, 1991):

1. The data should contain baseline information, which covers the design data and conditions of a component at its initial state.

2. The data should contain the operating history, which covers the service conditions of systems and components including transient and failure data.

3. The data should contain the maintenance history data, which covers the components monitoring and maintenance data.

For more details on how each of the previous attributes is identified, see (IAEA, 1991). However, it should be noted that the completeness features are defined differently depending on the problem. For example, data required for quantifying to a component failure frequency is different from that for quantifying a natural event. General scoring guidelines for evaluating 𝑇 2321 are given, based on the degree to which criteria are satisfied, as shown in Table B.3.2.

  Figure 1 Hierarchical tree for trustworthiness evaluation

  standards and obtains the acceptance from different analysts Level of sophistication (𝐿𝑜𝑆 = 𝑇 1,3,5 ) The degree of treatment of the problem, and amount of effort and details invested in the problem given its requirement (requirement and complexity) Number of known hazards (𝑁𝐻 = 𝑇 2,1,1 )

Figure 2

 2 Figure 2 Main steps for MHRA considering the trustworthiness of the PRA

Figure 3

 3 Figure 3 Probability distribution of the risk considering parametric uncertainty: (a) external flooding risk, (b) internal events

Figure 5

 5 Figure 5 Updated risk estimates after considering the level of trustworthiness for external flooding (a) original risk estimate from the PRA, (b) Risk estimates after integrating the level of trustworthiness

Figure 6

 6 Figure 6 Updated risk estimates after considering the level of trustworthiness for internal events (a) original risk estimate from the PRA, (b) Risk estimates after integrating the level of trustworthiness

Figure 7

 7 Figure 7 Results of the MHRA, (a) conventional aggregation, (b) considering the level of trustworthiness

  Figure A.1-3 illustrate the different score for each corresponding scenario.

Figure A. 3 . 1

 31 Figure A.3.1 Evaluation of the conservatism in the light of the level of maturity (conservatism VS Best estimate)

  the value-ladenness score is calculated by averaging the scores over all the attributes in this table.

Table 1

 1 Definition of trustworthiness attributes (Level 1)

Table 2

 2 Definition of trustworthiness attributes (Level 2)

	Attribute	Definition
	Robustness of the results (𝑅𝑜𝑅 =	The capability of the PRA results to remain unaffected by small variations in model
	𝑇 1,1 )	parameters or model assumptions
	Suitability of the model (𝑆𝑜𝑀 =
	𝑇 1,2 )

Table 3

 3 Definition of trustworthiness attributes (Level 3)

Table 4

 4 Definition of trustworthiness attributes (Level 4)

Table 6

 6 Normalized pairwise comparison matrix (knowledge matrix) of modeling fidelity "daughter" attributes

	Modeling fidelity	{𝑇 1,1 }	{𝑇 1,2 } {𝑇 1,3 }	Θ = {𝑇 1,1 , 𝑇 1,2 , 𝑇 1,3 }	Weight (BBA)
	{𝑇 1,1 }	0.33	0	0	0.06	0.10
	{𝑇 1,2 }	0	0.71	0	0.31	0.26
	{𝑇 1.3 }	0	0	0.8	0.5	0.32
	{𝑇 1,1 , 𝑇 1,2 , 𝑇 1,3 }	0.67	0.29	0.2	0.13	0.32

Table 7

 7 Discounted basic belief assignment from the three experts

	Focal sets of the	Expert 1	Expert 2	Expert 3
	criteria	𝑚 𝛿 (𝐴)	𝑚 𝛿 (𝐴)	𝑚 𝛿 (𝐴)
	{𝑇 1,1 }	0.06	0.16	0.02
	{𝑇 1,2 }	0.16	0.24	0.38
	{𝑇 1,3 }	0.19	0.24	0.46
	{𝑇 1,1 , 𝑇 1,2 , 𝑇 1,3 }	0.59	0.36	0.14

Table 8

 8

		Dempster's rule of combination matrix
	Expert 2	𝑚 𝛿 (𝑇 1,1 )	𝑚 𝛿 (𝑇 1,2 )	𝑚 𝛿 (𝑇 1,3 )	𝑚 𝛿 (𝑇 1,1 , 𝑇 1,2 , 𝑇 1,3 )
	Expert 1

Table 9

 9 Mass function combinations from the experts

		Combined mass from	Combined mass from
	Focal sets of the criteria	experts 1 and 2	experts 1, 2 and 3
		𝑚 𝛿 (𝐴)
	𝑚 1,2 𝛿 (𝑇 1,1 )	0.15	0.05
	𝑚 1,2 𝛿 (𝑇 1,2 )	0.29	0.40
	𝑚 1,2 𝛿 (𝑇 1,3 )	0.31	0.49
	𝑚 1,2 𝛿 (𝑇 1,1 , 𝑇 1,2 , 𝑇 1,3 )	0.25	0.06

  Table 10 level-3 leaf attributes weights 𝑊 and scores 𝑆 for external flooding hazard group

	𝑨𝒕𝒕	MS IoA RM	S	HU	Cv	AoC NH	AR	EK	YE	NE	Ac	In	AD
	𝑾	0.012 0.026 0.025 0.158 0.070 0.025 0.012 0.022 0.032 0.054 0.034 0.017 0.105 0.105 0.065
	𝑺𝒄𝒐𝒓𝒆	2	2	3	4	3	4	5	2	2	3	3	4	3	3	3

Table 11

 11 level-4 leaf attributes weights 𝑊 and scores 𝑆 for external flooding hazard group

	𝑨𝒕𝒕	Pl	VL	Ag	QA	LoG	NoA	LoD	C	Co	V	T	Ac
	𝑾	0.037	0.029	0.025	0.066	0.006	0.005	0.004	0.017	0.011	0.009	0.011	0.017
	𝑺𝒄𝒐𝒓𝒆	1	4	4	5	4	4	4	3	3	3	3	3

Table 12

 12 Probability of trust given the level of trustworthiness

	Trustworthiness	Probability of trust
	1	0.05
	2	0.50
	3	0.75
	4	0.90
	5	1.00

Figure 4 Fitted probability of trust in the PRA given the trustworthiness

Attributes under "quality of application" consequently

	𝑇 122 = 1 if the selected	𝑇 122 = 3 if the selected	𝑇 122 = 5 if the selected
	model is not usually	model is usually used	model is usually used
	used for achieving	for achieving objectives	for achieving objectives
	objectives similar to the	similar to the required	similar to the required
	required ones or it is not	ones or it is suitable for	ones and it is suitable
	suitable for the problem	the problem settings but	for the problem settings
	Suitability of the tool 𝑇 122
	settings and cannot	doesn't capture entirely	in a way that captures
	capture all the important	the important aspects of	entirely the important
	aspects of the problem	the problem	aspects of the problem
			in a way that makes it
			suitable to represent
			reality
	𝑇 123 = 1 if the selected	𝑇 123 = 3 if the selected	𝑇 123 = 5 if the selected
	tool is new or has never	tool is a new updated	tool is quite common
	proved its successful use	version of a tool that has	tool that has proved its
	before, or if it is a new	proved its successful use	successful use in
	version of the tool that is	before	different problem
	Historical use 𝑇 123
	quite different from the		settings, or if it is a
	old one		slightly updated version
			of an old common one
			that proved it successful
			use
	Appendix A.3:

decision making assumptions and parameters.

Table A .

 A 3.1 Scoring guidelines for the quality of the application

	True value (high
	confidence, 𝑃 ≥ 90%)
	based on strong
	knowledge
	Score
		1	3	5
	Attribute
		𝐾 131 = 1 if the setting of	𝐾 131 = 3 if the setting	𝐾 131 = 5 if the setting
		accuracy is chosen to be low	of accuracy is chosen to	of accuracy is chosen to
		and high degree of error is	be acceptable with a	be high and errors are
	The accuracy of	accepted in the calculations.	tolerable degree of	conservatively accepted
	the calculation	For example, the cutoff error	errors. For example, the	in the calculations. For
	𝑇 132	(the chosen value of	cutoff error is set to be	example, the cutoff
		parameters at which lower	quite low and a	error is set at to be small,
		values are ignored) is set to	sufficient number of	and a high number of
		be large, and a low number of	trials are performed	trials are performed
		trials are performed

Table A .

 A 3.2 Scoring guidelines for quality of assumptions[START_REF] Boone | NUSAP: a method to evaluate the quality of assumptions in quantitative microbial risk assessment[END_REF]

	Score
	1	3	5
	Attribute
	𝐾 1331 = 1 if the assumption	𝐾 1331 = 3 if the	𝐾 1331 = 5 if the
	is not realistic (over	assumption is based on	assumption is
	conservative or over	existing simple models	plausible: it is
	Plausibility of
	optimistic), or the available	and extrapolated data	grounded on well-
	assumptions 𝑇 1331
	information is not sufficient		established theory or
	for assessing the quality of		abundant experience
	the assumptions		on similar systems, and
			verified by peer review

Note: If multiple assumptions are involved in the assessment, the final score for 𝑇 1331 is obtained by averaging the scores of all the assumptions.

Table A

 A

.3.3 Scoring guidelines for the value-ladenness of the assessors

Table A

 A

	Number of		the quality standards 𝑇 1342 = 1 if there is a	quality standards and 𝑇 1342 = 3 if there is a	entirely and 𝑇 1342 = 5 if there is a
	approximations	and recommendations large number of	recommendations set by the moderate number of	conservatively the low number of
	𝑇 1352		set by the PSA approximations and the	PSA community e.g., ASME approximations or the	quality standards and approximations and
			community e.g., ASME aggregate of the	standards, NRC regulatory aggregate of the	recommendations set the aggregate of the
			standards, NRC approximations affects	guides, IAEA approximations affects	by the PSA approximations does
			regulatory guides, significantly the	recommendations moderately the output	community e.g., not affect, or affects
			IAEA recommendations output		ASME standards, insignificantly the
					NRC regulatory output
	Level of details			guides, IAEA
					recommendations
	1
	2	Table A.3.5 Scoring guidelines for leaf attributes under the level of sophistication
	Score	1	3	5
	Attribute
	Level of granularity	𝑇 1341 = 1 if the level	𝑇 1341 = 3 if the analysis is	𝑇 1341 = 1 if the level
	𝑇 1351		of analysis is performed	performed in to a sufficiently	of analysis is zoomed
			abstractly and coarsely	fine level that regards the	in to the level of
			on the level of systems	small components of a	component's small
			or level the level of	system or a small factors of a	constituting parts e.g.,
	1		.3.4 Scoring guidelines for leaf attributes under verification large components problem considering the small
	Score	1	3	5 constituting parts of a
	Attribute				manual (i.e., valve,
	Agreement among	𝑇 1341 = 1 if some	𝑇 1341 = 3 if some experts	𝑇 1341 = 1 if most of the body, bonnet,
	peers		experts hold strongly	questions on the assumptions,	the experts agree on ports etc.) when
	𝑇 1341		conflicting views on the	but do not have strongly	the assumptions building the physical
			assumptions	conflicting views	model for calculating
	Quality assurance	𝑇 1341 = 1 if the	𝑇 1341 = 3 if the analysis	𝑇 1341 = 5 if the the failure rate of a
	𝑇 1342		analysis does not follow	follows moderately the	analysis follows manual valve

  Table B.1.1 Scoring guidelines for leaf attributes under known potential hazards Table B.2.1 Scoring guidelines for phenomenological understandings' leaf attributes

		hazards, as well as other	of analysis and hazards,	type of analysis and
		types of problem, in a way	as well as other types of	hazards, as well as
		that prevents him from	problem, in a way that	other types of
		imagining new unknown	allows him to imagine	problem, in a way that
		types of hazards	new unknown types of	allows him to imagine
			hazards	most of the unknown
	Score			types of hazards
		1	3	5
	Attribute
	𝑇 211 = 1 if there is only a Appendix B.2: Attributes under "phenomenological understanding" 𝑇 211 = 3 if there is a	𝑇 211 = 5 if there is a
		few number of known	moderate number of	high number of
	Number of known
		relevant hazards that are	known relevant hazards	known relevant
	hazards
		considered in the analysis	that are considered in the	hazards that are
	𝑇 211
			analysis	considered in the
				analysis
		𝑇 212 = 1 if there is no	𝑇 212 = 3 if there is only	𝑇 212 = 5 if there is
		past experience and	a few past experience	abundancy of past
		technical reports that	and technical reports that	experience and
		explain and cover in	explain and cover in	technical reports that
	Availability of accident	details the timing, causes	details the timing, causes	explain and cover in
	reports	and different sequences of	and different sequences	details the timing,
	𝑇 212	abnormal activities,	of abnormal activities,	causes and different
		incident or accident	incident or accident, or if	sequences of
			there is abundancy of	abnormal activities,
			reports that covers	incident or accident
			accidents without details
	Experts knowledge	𝑇 213 = 1 if the expert has	𝑇 213 = 3 if the expert	𝑇 213 = 5 if the expert
	about hazards	a low experience in such a	has a moderate degree of	has a high degree of
	𝑇 213	type of analysis and	experience in such a type	experience in such a

Table B

 B

		.3.2 scoring guidelines for data reliability
	Score
	1	3	5
	Attribute
	Completeness

In this setting, the conservatism is evaluated in the light of three criteria: (i) types of risk index estimates (best judgment, true value with a high confidence and true value with a low confidence); (ii) context of decision making;

(iii) the effect of conservatism on the perception of the problem compared to best or true estimates or true and

Appendix B.3: Evaluation guidelines for leaf attributes under "Data"

Amount of data 𝑇 231 is measured by a numerical metric, Years of Experience (YoE), defined by the number of related events recorded during a specific period.

YoE =length of the data collection period (in years) × sample size of the data

The amount of data is scored based on the criteria in Table B.3.1. The validity of data is evaluated by the following criteria:

1. The integrity of data is carefully managed.

2. Databases are well organized and formatted in a common way, and easily retrieved and manipulated.

3. Data should be collected and entered in the database by well-trained maintenance personnel, and modern computer techniques should be used for data storage, retrieval, and manipulation.

4. The data collection and entering process should include an appropriate quality control mechanism.

Based on the four criteria the evaluation guidelines of 𝑇 2323 can be defined in Table B.3.3. [START_REF] Popek | Sampling and analysis of environmental chemical pollutants: a complete guide[END_REF]. Since the data involved in nuclear PRA are mostly related to the number of failures or degradations and are usually collected digitally from different sources, systematic errors in the data are very small. This means that the accuracy of data is primarily determined by random errors. Since the error margin of the confidence interval is widely accepted as a good indicator of the random errors, it can be used as a measure of the data accuracy. Error factor may be defined based on the upper and lower bounds of confidence interval:

where 𝑈 𝑙 and 𝐿 𝑙 are the upper and the lower bounds of confidence intervals. The accuracy of data is, then, scored based on the value of error factors, following the guidelines in


Static characteristics of 5 kV SiC BJTs and Darlington's B. Asllani 1) , D. Planson 2) , H. Morel 2) and T. Lagier 1) 1) SuperGrid Institute, 23 rue Cyprian CS 50289, 69628 Villeurbanne Cedex -France

2) Univ Lyon, INSA Lyon, Univ. Claude Bernard, ECL, CNRS, F-69621, France E-mail: besar.asllani@supergrid-institute.com

High voltage low current SiC BJT transistors are interesting devices for applications like active filters, current limiters and active snubbers where high voltage blocking capabilities are required. Few attempts [START_REF] Singh | Switching and robustness analysis of 10 kV SiC BJTs[END_REF]- [START_REF] Zhang | 10 kV, 10 A bipolar junction transistors and Darlington transistors on 4H-SiC[END_REF] have proved that such devices can be manufactured and fulfill the requirements in terms of breakdown voltage, beta and reliability. SuperGrid Institute has designed 10 kV -5A SiC BJTs as a proof of concept with the aim to increase the nominal current rating to 50 A. This devices are intended to operate in the abovementioned applications and also to be employed as a power switch in HVDC converters. This abstract evaluates the static performances of the first run of 10 kV -5 A BJTs from SuperGrid. The switching performance will be assessed and reported in the full paper. Four different designs have been manufactured on three 4" SiC wafers. The epitaxial parameters and the peripheral protections are the same as the ones used for the 10 kV PiN diode reported at ECSCRM2018 [START_REF] Asllani | Advanced Electrical characterisation of high voltage 4H-SiC PiN diodes[END_REF]. Several parameters have been varied in order to test the impact on beta. After the fabrication, devices where tested on wafer and 40 samples have been selected and packaged in ISOPLUS packaging from DeepConcept for further testing purposes. Packaged devices are shown in Fig. 1.

Designs #2 and #3 which have the best betas are compared in figure 3. Design #3 has an average beta of approximately 10 at room temperature which drops at 6 at 150° (Fig. 4). Four devices with very similar characteristics have been selected and packaged into two paralleled BJTs and four others in Darlington configuration. As shown in Fig. 5 and Fig. 6, both these configurations have achieved to double the current rating without deteriorating the static performance and give promising results. Future monolithic Darlington designs can be used to further increase the current amplification and achieve higher current density. Such a configuration could also infer an improved thermal stability of the current amplification. Fig. 7 compares the output characteristics of designs #2 and #3. Both these devices seem to handle well linear operation. The blocking capability has to be improved to reach 10 kV rating as avalanche occurs just above 5 kV.

A specific driver will be designed and used to test the switching performance of the presented devices for the final paper. This work could yield devices that can compete against SiC MOSFETs for higher voltages then 10 kV.

Fig. 1 .Fig. 3 .Fig. 4 .

 134 Fig. 1. Isoplus packaging from DeepConcept

Fig. 6 .

 6 Fig. 6. Comparison of output characteristics of Paralleled and Darlington BJTs.

Fig. 7 . 2 IB

 72 Fig. 7. Output characteristics of the all of the measured single BJTs. Fig. 8. Blocking capability of BJTs.

Mo-P-45 ICSCRM2019 © ICSCRM2019