K E Bemmami

P David

Managing the use of simulation in Systems Engineering: an industrial state of practice and a prioritization method

Keywords: Model-based Systems engineering (MBSE), Simulation, Decision Support, Analytic Hierarchy Process (AHP)

Model-based systems engineering (MBSE) has been proposed as an approach to manage the complexity of modern product development through the continuous use of models. The use of model simulation is a core principle of the MBSE approach. In the early stages of projects, it for example supports defining the expected system features, when in the later phases it can be used to estimate the dynamic behavior. Simulation is pushed to obtain results earlier and cheaper than with testing and prototyping. However, the development of simulation can be a very tedious and expensive task. Simulation opportunities are numerous, but the project managers must identify the more relevant for their project. This paper aims at documenting the current state of practice on the usage of simulation in MBSE processes. Then it aims at exploring decision support opportunities for simulation use in MBSE projects. The paper presents a survey conducted amongst French companies, on how they apply MBSE, Verification Validation & Testing (VVT), and simulation. The perceived benefits, barriers, and the parameters influencing VVT strategies, and the use of simulation are alternately analyzed. The results of the survey are used to propose a Prioritization of Simulation Efforts Methodology (PSEM) to assists managers in choosing the right functions, or systems' elements to be simulated.

Introduction

Mastering the use of simulation tools in Systems engineering (SE) processes is a crucial subject for engineers. The digital world brings numerous advantages in terms of information management, processing, and visualization, but demands careful attention for their relevant and efficient utilization. The use of models proposed in the frame of the MBSE (Model-Based Systems Engineering) is helping to decrease project risks, shorten project, decrease development costs, and ensure higher product quality. These practices contribute to develop more complex systems, often combining disparate technologies and elements [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF]. The tools for computerized design have considerably evolved and given rise to some very powerful systems engineering tools. The potential to increase verification activities through simulation is a key point for the future of SE. SE teams have access to numerous tools and verification techniques as simulation, test, expertise … [START_REF] Chapurlat | UPSL-SE: A model verification framework for Systems Engineering[END_REF]. For its promoters, the application of simulation techniques can significantly increase design knowledge and support failure correction in an early product development stage. It increases the confidence of the developers in the design and fosters the correction of intrinsic conceptual design failures that usually cause high rework costs if found during system testing [START_REF] Hoppe | SysTest: Improving the verification, validation, and testing process-Assessing six industrial pilot projects[END_REF]). The combination of the simulation and verification tools help the system designer to understand the trace results given by model checkers, ultimately enhancing the use of the formal verification method [START_REF] Seidner | Simulation and Verification of [Dys] functional Behavior Models: Model Checking for SE[END_REF]. A significant amount of work has been devoted to develop the modeling infrastructure so that multidomain knowledge can be centrally managed and shared among stakeholders. Quintana et al. [START_REF] Quintana | Will Model-based Definition replace engineering drawings throughout the product lifecycle? A global perspective from aerospace industry[END_REF] showed the efforts on Model-Based definition in the aerospace industry. Hehenberger et al., showed the panorama of available methods for cyber-physical systems design [START_REF] Hehenberger | Design, modelling, simulation and integration of cyber physical systems: Methods and applications[END_REF]). However, the simulation capacity needed to experiment a full system dynamic behavior is rarely available at companies that still often use traditional systems engineering practices [START_REF] Zeigler | MBSE with/out Simulation: State of the Art and Way Forward[END_REF]). Many companies are still reluctant in deploying simulation benches, mostly since doubts remain on the return on investment. Moreover, project leaders still have difficulties to get clear evidence on whether or not simulation is an appropriate practice in their project context. This paper aims at contributing to develop an understanding of current practices in the industry regarding simulation and Verification in SE projects. Then, it aims at proposing a method to make relevant and efficient use of the simulation techniques within MBSE projects. It is assumed that full modeling and simulation of complex systems is technically irrelevant and that prioritization shall be made in modeling and simulation investments in projects. The research methodology used in this paper is first to establish an overview of the state of practices on simulation usage in SE/MBSE. Secondly, it is to identify some barriers to MBSE/simulation fulfillment. From these observations, the outline of a method to support decisions related to the use of simulation in SE/MBSE projects is given. The proposed method relies on prioritization suggested from the survey analysis. The relevance of the survey is evaluated by the comparison of some results with existing surveys in the literature. The external validation of SE methods is hard to obtain, since it necessitates conducting concurrent similar projects with and without the developed methodology in representative companies and project teams. To overcome this difficulty, this work is based on a consistent analysis of current state of practice and on practitioner judgement to design its proposal. Therefore, it is assumed that the proposal is aligned with SE community expectations and priorities. The remaining of the paper is organized as follows: Section 2 provides background information on MBSE practices, simulation and modeling activities, and on the link between simulation and VVT (Verification, Validation & Test) activities. Section 3 presents the survey conducted within CRRA1 (a regional branch of the French chapter of INCOSE2 ). Section 4 proposes a methodology to prioritize the use of simulation within projects based on the survey observations. Section 5 presents the application of the proposed methodology on a test project and comments on its output interpretation. Section 6 concludes the paper.

Deployment of MBSE and simulation

MBSE emerged to support the specification, analysis, design, and verification of complex systems using integrated system models with dedicated software tools [START_REF] Maccalman | Integrating external simulations within the model-based systems engineering approach using statistical metamodels[END_REF]. Companies and organizations have recognized the potential of MBSE and foresee benefits in terms of higher quality and shorter lead time [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF]. However, according to INCOSE SE Vision 2025 "MBSE has grown in popularity as a way to deal with the limitations of document-based approaches but is still in an early stage of maturity similar to the early days of CAD/CAE" [START_REF] Beihoff | A World in Motion-Systems Engineering Vision 2025[END_REF]. This status is a result of several barriers to the transition from the traditional SE approach to MBSE. For example, there exist required changes from legacy practices relied on documentation, in organizations and their network of suppliers and clients [START_REF] Friedenthal | A practical guide to SysML: the systems modeling language[END_REF]. Other barriers are the high cost for such technical transition, the lack of strong management structures to support and guide the implementation, the lack of knowledge to integrate a model-based approach with current business processes, and a lack of available trained resources [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF]). During the system development lifecycle, a variety of models and simulations that represent different domains are used. These domains may include operational efficiency, life cycle cost, physical behavior, reliability, and much more. The type of domain models ranges from simple analytical equations and spreadsheet models to simulation models that capture the dynamic complexities of a system over time. Each model is an abstraction of reality that represents a unique perspective of the system [START_REF] Maccalman | Integrating external simulations within the model-based systems engineering approach using statistical metamodels[END_REF]) and simulation refers to the application of computational models to the study and prediction of physical events or the behavior of engineered systems [START_REF] Oden | Revolutionizing engineering science through simulation[END_REF]. For supporters of MBSE, the superposition of modeling and simulation in SE provides a prospect that is an integral part of a cost-effective process to meet user requirements and needs [START_REF] Kossiakoff | Systems engineering: Principles and practices[END_REF]. Simulation provides the system designer with a fair insight upon its behavior, which is particularly needed in the early design phases of a complex system [START_REF] Seidner | Simulation and Verification of [Dys] functional Behavior Models: Model Checking for SE[END_REF]. For example, analyzing simulation model output results are one way a systems engineer can early verify and/or validate a system requirement (MacCalman at al. 2016). In many instances, the modeling, simulation, and analysis activities are interwoven, particularly during the development, verification, and validation (VV) phases [START_REF] Nasa | STANDARD FOR MODELS AND SIMULATIONS[END_REF]. Simulation techniques are also seen as an essential element for early verification, validation, and testing activities (VVT). In the context of the pilot project conducted by [START_REF] Hoppe | SysTest: Improving the verification, validation, and testing process-Assessing six industrial pilot projects[END_REF], simulation techniques were used more intensively, trying to enhanced VVT planning and execution in the first phases of the project. In this study, failures and intensive VVT in the latter phases were avoided, resulting in an overall VVT cost reduction of 18%, which was found significant. They also found a reduction in test redundancy and an improved coordination with the pilot project engineering services. But these results are only obtained on a pilot project.

Despite the importance of the simulation contributions, project managers still have some hesitations when it comes to its deployment. These hesitations may be due to a lack of trust, when in some way the simulated model is seen as abstract or simplistic [START_REF] Schamai | Model-based verification of dynamic system behavior against requirements: Method, language, and tool[END_REF]. Additionally, the heavy workload and the cost of deployment of engineering tools represent obstacles that are largely based on the feelings of project managers as reflected in practices studies [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF], [START_REF] Laing | Questioning integration of verification in model-based systems engineering: an industrial perspective⋆[END_REF]. In this respect, the NASA standard 7009A (2016) proposes a set of questions to examine before simulation deployment. These questions are related to the pedigree (and quality) of the data used to develop the model, the uncertainty characterization of the model, and the correctness of the model implementation per their requirements/specifications.

During the early stages of design, system engineers may rely on simulation models when operational testing is not feasible because the system does not exist. "However, systems engineering tools are not generally applied in a wider context to support collaborative environments to integrate different technical domains. The consequence will most likely be that the full potential of introducing an MBSE approach is not reached, and the effects are typically constrained to the systems engineering domain" [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF]. So, better integration of simulation in MBSE processes is to be thought. But, simulation is still perceived by managers and many project leaders as an expensive task with an unpredictable return on investment. A way to advance in this point would be to prioritize modeling and simulation efforts to the more relevant part of the system. Many parameters can be considered to assess the necessity of modeling and simulation tasks. The cost of modeling, skill availability, level of confidence in the estimation, reusability of the model, may be criteria to analyze. This work aims at contributing to the accurate use of simulation in MBE projects. To support these techniques deployment, it is proposed to accompany the project manager in identifying the more relevant part of the system/behavior to study by simulation. To construct a proposal, a state of practice to deal with concepts and priorities that sound relevant to SE practitioners is first carried out. This state of practice aims at understanding enablers and barriers to simulation utilization in companies. It is also used to identify the primary criteria used in companies to decide when modeling and simulation efforts might be made within projects. This work then proposes a multi-criteria decision support approach that uses the identified criteria to formalize this decision.

Survey on industrial practices in Simulation, Verification, and MBSE

Survey description

The survey was conducted during a workshop on MBSE & VV organized by the CRRA chapter of AFIS3 in January 2019.. To analyze the representativity of the provided survey, it is compared with 3 previous surveys on MBSE adoption: [START_REF] Vogelsang | Should I stay or should I go? On forces that drive and prevent MBSE adoption in the embedded systems industry[END_REF], [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF] and [START_REF] Chami | A Survey on MBSE Adoption Challenges[END_REF]. [START_REF] Vogelsang | Should I stay or should I go? On forces that drive and prevent MBSE adoption in the embedded systems industry[END_REF]) surveyed companies from embedded systems industry that tested MBSE to understand the drivers and hinders to MBSE adoption. [START_REF] Chami | A Survey on MBSE Adoption Challenges[END_REF] surveyed MBSE practitioners on their opinion on a list of MBSE adoption challenges. [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF] propose a survey to examine the current state of practice of MBSE and the ambitions of companies on that topic. The focal point of the presented survey is different from these 3 works since it is on simulation deployment, but it shares a common part on MBSE adoption. This comparison enables to qualify the surveyed sample on their behavior on MBSE adoption and on their perception of MBSE challenges. The obtained result let think that the sample can be representative of a wider community.

The workshop gathered 21 participants from France belonging to 18 different organizations from various fields. The respondents were not filtered. Their participation in the survey denotes their interest in sharing knowledge and thoughts on MBSE. Among the represented organizations, 60% of companies have more than 5 years' experience in setting up strong SE processes. It is also to be mentioned that 68% of participants have more than 5 years' experience in the SE field. The full panel description is provided in table 1. The interviewed panel can be compared to the one used in [START_REF] Vogelsang | Should I stay or should I go? On forces that drive and prevent MBSE adoption in the embedded systems industry[END_REF]) that gathered 20 respondents in a study on MBSE adoption. The used panel and the one from [START_REF] Vogelsang | Should I stay or should I go? On forces that drive and prevent MBSE adoption in the embedded systems industry[END_REF] are smaller than the one from [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF]) (66 respondents) or [START_REF] Chami | A Survey on MBSE Adoption Challenges[END_REF]) (42 respondents), but the last two studies have been performed through an online questionnaire. The panel used in this work is complementary to the one of [START_REF] Vogelsang | Should I stay or should I go? On forces that drive and prevent MBSE adoption in the embedded systems industry[END_REF] or [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF] as it covers a new geographical area (e.g. France, where [START_REF] Vogelsang | Should I stay or should I go? On forces that drive and prevent MBSE adoption in the embedded systems industry[END_REF]) covered Germany, [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF] covered the US and Sweden). [START_REF] Chami | A Survey on MBSE Adoption Challenges[END_REF] did not provide the geographical information. It is also the most heterogeneous panel from the companies' domain perspective, aerospace and defense were representing more than 70% of (Huldt & Stenius 2018) panel when 60% of [START_REF] Vogelsang | Should I stay or should I go? On forces that drive and prevent MBSE adoption in the embedded systems industry[END_REF] panel is from avionics or automotive industry. In [START_REF] Chami | A Survey on MBSE Adoption Challenges[END_REF], 31% of companies operates in consultancy or teaching domains and 24% from aerospace & defense. The panel of this study gathers companies from the electrical power management domain (16%, the highest part), defense, civil nuclear industry, or oil & gas. The objectives of the survey are triple: Analyzing SE, MBSE, and Simulation practices in participants' organization; eliciting barriers and enablers for MBSE and Simulation adoption; finding the relevant criteria for Simulation deployment in projects. More details on the survey can be found in [START_REF] Bemmami | State-ofpractice survey in industry on the deployment of simulation in systems engineering[END_REF]. Participants were consulted with a questioner organized into 40 questions presented in 4 sections:

• The status of current SE and MBSE practices in companies (maturity level, challenges, expected benefits). • The status of current practices of VVT in companies (VVT strategy, criteria for VVT deployment, practices). • The use of computer-based simulation in companies (use, criteria for deployment, satisfaction/barriers). • The use of multi-criteria decision methods in companies.

The questions were developed in a way that cross-referenced the factors governing the practices of the VVT activities and those of simulation, for the close link that exists between the two activities. Some questions were inspired from the survey conducted by [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF], to analyze the link between the MBSE approach and simulation activities, and trying to better understand the use of simulation and their impact on certain practices in SE field. The questionnaire has been sent by email to other MBSE user community in France that permit to gather 4 more answers, integrated to the final results.

The provided survey is complementary to the preceding ones in the literature by the covered geographical area and industrial domains. It is also addressing a complementary topic by providing a deeper analysis of the VVT and simulation practices, which were not targeted by previous state of practices. A comparison of the common part of our study to the results observed in the literature is provided before analyzing the complementary topics. As the results obtained on MBSE practices are compliant and complementary to the previous works, it is assumed that the results obtained on VVT and simulation analysis are representative of a typical SE practitioner community beyond our analyzed sample.

Survey main results

Observed MBSE practices

The results observed in this topic can be compared to the ones observed in the previous surveys on MBSE adoption: [START_REF] Vogelsang | Should I stay or should I go? On forces that drive and prevent MBSE adoption in the embedded systems industry[END_REF], [START_REF] Chami | A Survey on MBSE Adoption Challenges[END_REF] and [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF]. The first set of questions revealed that 80% of the respondents declare using an MBSE approach in their organizations. Nevertheless, it appears that respondents judge that the MBSE approach is not fully mastered in their organizations as more than 65% of participants express that their MBSE process is under development or not formalized. These figures are comparable to [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF] sample for which only 10% of participants' company never developed projects with MBSE, and only 37% applied MBSE often or on almost all projects, letting think of a relative lack of process formalization. [START_REF] Vogelsang | Should I stay or should I go? On forces that drive and prevent MBSE adoption in the embedded systems industry[END_REF]) and [START_REF] Chami | A Survey on MBSE Adoption Challenges[END_REF] only questioned MBSE practitioners and did not analyze process maturity.

Surveyed companies use MBSE to improve the quality of the developed product or system and knowledge management (according to 28% of respondents), to improve communication and information exchange (according to 23% of respondents), and to increase the ability to manage complexity (according to 32% of respondents). These results are compliant to [START_REF] Vogelsang | Should I stay or should I go? On forces that drive and prevent MBSE adoption in the embedded systems industry[END_REF]) observations that placed handling complexity, early feedback, and increase in quality in the top 5 motivations of pulling MBSE. Motivations for pulling MBSE were not investigated in [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF] and [START_REF] Chami | A Survey on MBSE Adoption Challenges[END_REF].

Participants raised several challenges to overcome to be better deploy MBSE. Mainly mentioned are, the difficulties of integrating the MBSE with other existing processes according to 83% of participants, the need to improve skills according to 74% of participants, as well as the way tools are used according to 52% of participants. The investment costs also seem to be a barrier to the use and evolution of the MBSE according to 48% of participants, which constitutes generally an important criterion to convince stakeholders in the MBSE deployment decision. These results are coherent with those obtained by [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF] but our study pushes the importance of skills far more than what Huldt and Stenius found. In [START_REF] Vogelsang | Should I stay or should I go? On forces that drive and prevent MBSE adoption in the embedded systems industry[END_REF]) study, the ROI uncertainty and the skill of employees were by far the 2 most important hinders to MBSE development. [START_REF] Chami | A Survey on MBSE Adoption Challenges[END_REF] results show that the human factor (awareness and change resistance) is the most recognized issue, just before the method definition and tools integration. The performed survey is thus confirming previous results on the difficulties on MBSE adoption. It complements the observation by addressing a complementary sample of company. Another interest is that, as it is done in [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF], the sample of consulted companies is not exclusively composed of MBSE practitioners. The next two sections present results on topics that were not investigated in the previous literature. The results obtained in this section show that our sample behave in a comparable manner as what was found in the literature. This gives credits to the subsequent observations made on VVT practices and on simulation deployment.

Observed VVT practices

More than 80% of participants report having a fairly clear VVT strategy in their organizations and 60% affirmed having a VVT strategy from the early phases of the project. As reported by 71% of participants, performing or not VVT actions is related to the assessment of the benefits that these actions will bring, and the risks taken in the opposite case. This evaluation considers different criteria such as the importance of the requirements verified/tested, the complexity of the system to be tested, the impact of undetected errors, the reachable result confidence, and the benefits of detecting errors. When pointing the important lifecycle phases for VVT actions, it was noticed that the functional architecture design and system requirements analysis are seen as the most critical for VVT activities for more than 80% of respondents. This reveals the relevance of addressing new techniques to support the early VVT deployment. To succeed in such an approach, the main impediments to early VVT pointed by respondents are time and cost (according to 47% of participants), and technical resources availability (according to 33% of participants).

Observed simulation practices

In the majority of respondents' organization (70%) simulation is used in more than 40% of projects, and mainly as a support to VVT activities. Various criteria are used to determine whether to use or not a simulation and to justify and convince the manager of the need to use it. For more than 40% of participants, it is important to consider the cost of deployment, the duration of model development and exploitation, the complexity of the system to be verified or tested, and the complexity of the system behavior or the environment in which it operates. According to more than 80% of participants, an improvement in the quality of certain practices has been observed following the use of simulation, mainly in architecture and design, requirements analysis, and verification and validation. The feedback from participants reinforces the global view regarding the use of simulation in projects, saying that the use of simulation improves the quality of tests performed and delivered projects. Despite the various benefits of simulation, barriers such as cost (for 70% of respondents), available skills (70%), and model complexity (40%) are delaying a wider deployment of simulation in SE/MBSE projects. The impact of simulation use on project cost is highly debated in the panel where 44% think it decreases the project costs when 43% think it increases it.

To better integrate MBSE into the practices of organizations, more than 50% of respondents agreed that it is necessary to overcome barriers such as the 1) difficulties of integrating the MBSE with other existing processes, 2) the need to improve skills in SE practices and modeling activities, and 3) reduce the investment costs.

Survey main conclusion

From the survey, several research directions may be sketched. First, work is still to be done in simulation software to convince users of their user-friendliness and to lower the required skill to use them. Secondly, analysis methods are needed to fully understand the costs and savings brought by the introduction of simulation in MBSE. This complex question has to be tackled to clarify the decision of deploying simulation in a company or a project. Direct costs are straightforward, but many benefits are hard to quantify when it is about avoiding a design error or providing a high-quality specification. Finally, the decision to use simulation has many facets. Diverse features are to be considered, on the kind of project, the kind of systems, the importance of requirements, or safety. Thus, it sounds interesting to elaborate on multi-criteria decision techniques to deepen the decision that can be taken in projects. Decisions on introducing simulation have to be made at all levels: Strategic, tactical, and operational. It could be summed up as:

1. Do investments in simulation frameworks in our organization have to be made? 2. On which kind of project shall simulation be used? 3. To which phase of the project shall simulation be used? 4. To which part (structure or behavior) of the system shall simulation be used?

To address these questions the set of criteria identified in this survey should help, namely for questions 2, 3, and 4. Criteria for deciding VVT actions and criteria for deploying simulation were identified in the survey and can be merged to investigate these questions. It appears that deciding to invest in creating simulation models is made on many common criteria with deciding to perform a VVT activity. The complexity of the system to verify and of its environment, as the importance of related requirements are of main importance. The decision to select simulation is partially based on selecting a VVT technique, therefore it is obvious to see operative criteria as the cost and deployment time. But it is hard to quantify the induced benefits of simulation as communication enhancement or emergent behavior discovering.

In the last part of the interview process, the respondents were asked about the use of multi-criteria decision methods to support decisions related to SE and MBSE projects. 50% felt it would be valuable to use this kind of technique and the remaining 50% does not have an opinion on it (none were against its use). The experience of organizations with multicriteria decision method in this context is quite poor (see table 2). This leads us to think that attempts and pilot projects should be done on the use of multicriteria decision method to support simulation decision in MBSE projects.

Table 2. Use of multi-criteria decision methods

To what extent are you using multi-criteria decision support methods to support decisions on

A methodology to prioritize simulation development in projects

The proposed method is set up to address the need to better integrate the use of simulation within the MBSE process. It focuses on assisting decisions related to its deployment. It is proposed to place this approach at the functional architecture design level of the SE process. This choice is made as [START_REF] Laing | Questioning integration of verification in model-based systems engineering: an industrial perspective⋆[END_REF] pointed out that it is the most important phase to improve the V&V activities efficiency. It is assumed that complex projects are characterized by the creation of functional architectures comprising a high number of interrelated functions. It is admitted by survey participants that using simulations of systems models can help to achieve a better system quality and a better project mastering if the modeling efforts are managed. A full simulation of the model is most of the time not achievable nor relevant from an economic point of view. It is thus needed to provide a selection of the system elements (subsystems or functions) to be simulated to enhance the simulation and V&V strategy. The provided methodology is set up to formalize this decision and to give evidence to project managers that the right priorities are defined in the development phases. In the remaining of the methodology presentation, it is assumed that the methodology addresses the functional architecture definition phase so only functions analysis will be presented. The analysis can be straightforwardly transposed to logical or physical subsystems.

In this context, the proposed methodology works with:

• Alternatives: The functions of the system, extracted from the functional architecture model that could be simulated. • Criteria: Decision parameters used to evaluate the alternatives. They are extracted from the industrial survey analysis.

The expected results from this methodology are, in the first place, a ranking of functions that should be considered in a simulation model. In the second place, information about the priority of each function for each criterion, this information aims to give a detailed view on each choice (for example: what is the function impacting the most important requirements).

The Prioritization of Simulation Effort Methodology (PSEM) proposed in this work is based on a multicriteria method. In the remaining of this section, the choice of a multicriteria method is explained. Then, a description of the adaptation of this method to improve its use is given. Finally, the developed methodology is detailed through the explanation of the criteria evaluation techniques and the comparison process description.

Choice of a multicriteria decision method

Multicriteria decision method selection

To compare the multicriteria methods, descriptors defined in the literature and specifications related to the context of this work are used. Table 3 gives the utilized descriptors in column 1, their definition in column 2, and the expected realization for the PSEM in column 3, and the suitable multi-criteria methods for each PSEM need in column 4. As described in column 3, the PSEM application needs a method to perform a total ranking of solutions (here a set of function) that can grow up to 100 of alternatives. It is needed to work in industrial context with possibly lacking information on criteria. The criteria used for the ranking are of diverse nature and are not equal in importance in the decision. It is preferred to use a computerizable method, that could be easy to set up in various industry. Finally, it is important to have a dynamic use of the method, since ranking may evolve along the project life. Inter comparison between project and reuse of the decision structure is also important to set stabilized processes in industry. The suitable method is selected based on the expressed PSEM need for each descriptor. The work of (Al-Shemmeri, Al-Kloub and Pearman, 1997; [START_REF] Pirlot | A common framework for describing some outranking methods[END_REF][START_REF] Guitouni | L'ingenierie du choix d'une procedure d'agregation multicritere[END_REF] regarding the comparison between the various multi-criteria methods permits to identify in column 4 the suitable method regarding each expected value of multi-criteria method descriptors. By considering table 3, it can be observed that the AHP method is considered as suitable for almost all PSEM needs. The main drawback is that it need adaptation to treat high number of alternatives. It appears that in comparison to the multicriteria methods mentioned earlier, the AHP method best fits with expressed needs for the benefits below:

• It provides a global evaluation of alternatives considering the weights of each criterion while integrating personal judgment on criteria importance. • It exists flexibility in the type of decision-making issues to address (choice or ranking) • It is easy to set up, precise, and have fast processing (based on matrices) • Many software tools are available to support the method or easy to develop • It is possible to reuse the decision model (e.g. criteria interrelation) • It is easy to modify criteria or to add new ones. The evaluation can be retaken over the project and can be adapted to different types of systems.

AHP is frequently applied in a variety of decision-making contexts [START_REF] Ho | Integrated analytic hierarchy process and its applications-A literature review[END_REF] and specifically in SE processes as in [START_REF] Lin | Using AHP and TOPSIS approaches in customer-driven product design process[END_REF] for design alternative evaluation in customer-driven product design process.

It is the information on the need to express the preferential relationships between criteria

Have to take in consideration criteria judgment

AHP, TOPSIS, PROMETHE 1&2, PAMSSEM 1&2

Type of treatment [START_REF] Guitouni | L'ingenierie du choix d'une procedure d'agregation multicritere[END_REF]

This is actually the type of processing used for aggregation

Mathematical

AHP description

The AHP method converts pairwise comparisons on a semantic scale into a priority vector. It is based on three principles:

1. Disaggregate the problem by identifying important elements. In this phase, the work is to determine the entities involved in the decision, the limits of the evaluation; to define which alternatives (or scenarios) are to be evaluated, and which criteria (e.g. economic, qualitative) are relevant to compare them.

2. State comparative judgments on the identified elements. It consists in structuring the hierarchy of decision criteria and alternatives in a tree-like form (decision tree) with several levels. The criteria and alternatives are successively compared two by two at each level of the decision tree. The pairwise comparison of the criteria gives the decision pattern (relative importance of criteria with each other). The alternatives are compared with each other at the last level of the tree structure. These comparisons are made via preference matrices. Traditionally, preferences are attributed according to a 9-level qualitative scale (Saaty, 1980), linking numerical values to verbal expressions: 1 means criterion A of equal importance to criterion B, up to 9 which means criterion A is absolutely more important than criterion B.

3. Deduce measures of relative importance from the pairwise comparison matrices to determine priorities for action. Once the local priorities for all criteria in the hierarchy have been determined, AHP calculates an overall evaluation score attached to each of the identified alternatives. This step makes it possible to consider the relative priority of each criterion to obtain a ranking of the alternatives according to the identified objectives.

In a classic AHP deployment, the user performs the following steps:

1. Decompose the decision-making situation hierarchically into interrelated elements (Decisionmaking criteria and Alternatives). Each criterion is considered as a node composed of alternatives.

2. Perform pairwise comparisons between decision-making criteria (For n criteria, n(n-1)/2 pairwise comparisons must be performed).

3. Then based on each criterion, perform pairwise comparisons between the solution alternatives (For k alternatives, k(k-1)/2 pairwise comparisons must be performed).

4. The relative importance of the compared elements is determined by calculating the eigenvectors corresponding to the different comparison matrices. This vector informs on the consistency of the comparisons.

5. Aggregate the relative weights of the decision elements to establish the relative performance of each alternative. The weight vector resulting from the aggregation is used to sort the alternatives.

The main disadvantage of the AHP method is that it needs many comparisons between different alternatives for each of the criteria one by one. A large number of elements to be compared can cause an explosion in the number of pairwise comparisons. In the studied case, the developed systems are often complex and contain a huge number of functions, components, or subsystems. To be able to use the AHP method in the proposed PSEM this constraint must be overcome. This is why adaptation and criteria evaluation routines are defined in the following sections.

AHP method settings and adaptation

An adaptation of this method is designed to overcome the above-mentioned constraint, mainly to reduce the analysis time. It is proposed to accelerate the alternatives pairwise comparison by preferring individual alternative evaluation regarding each criterion and then to generate the alternatives' pairwise comparison. To do so, the users are invited to give their alternative estimation value for each criterion. [START_REF] Saardchom | Credit Scoring Model by Analytic Hierarchy Process (AHP)[END_REF] in regard to its coherency, the facility of adaptation, and to avoid ambiguity in the interpretation of judgments in further process. Section 4.3 presents the translation from natural criteria estimation to marks expressed on this scale. Afterwards, to perform the pairwise comparison of alternatives regarding each criterion, a matrix is created and filled with the values of the gap between the estimation given for each alternative on the specific criterion. This matrix comprises positive and negative values that expresses the difference between alternatives ratings on the analyzed criteria. The absolute value of this difference is noted "x". A transformation of these values is needed to fit to AHP method to build the final comparison matrix. The adaptation is performed as follows: Each positive value "x" is replaced with "x+1", and each negative value "-x" is replaced with "1/(x+1)". For instance, if alternative A has an importance level of "3", and alternative B of "5", then in the matrix of comparison for the considered criterion the value for the pairwise comparison between A & B is "1/3". This means in the AHP "language" that alternative A in less performant than alternative B in regards of the selected criterion. The applied "+1" is due to the AHP evaluation system for which the value "1" codes that the two alternatives have an equal performance regarding the analyzed criterion. Therefore, if x = 0 the element of the alternatives' comparison matrix must be equal to 1. This procedure has 3 main advantages. First, the CR (Consistency Ratio), representing the measurement of the overall consistency under the AHP, was observed to be always under 10%. According to Saaty's rule proposed in 1980 [START_REF] Saardchom | Credit Scoring Model by Analytic Hierarchy Process (AHP)[END_REF], the more the CR is under 10% the more our judgment is consistent. Secondly, it allows a considerable time saving, due to the automation of the inter-alternative preference computation. Finally, this procedure aims to be more simple and easier to use. In the remaining of this section, the criteria and their evaluation are presented and the whole alternatives comparison process is presented.

Criteria definition and evaluation

The decision criteria utilized in the method are selected from the industrial survey. The criteria are the ones that have been cross-referenced from the most commonly used for the definition of the VVT strategy and the ones identified for simulation prioritization. The retained criteria are as follows:

• Importance of requirements: This criterion is related to the importance given to the requirements that are satisfied by the function or structure being studied.

• Complexity level: This criterion refers to the complexity of the function, or structure under study. Often hard to define, knowing that there is no well-established method for the measurement of this parameter.

• Benefits of detecting errors: This criterion concerned the benefit of early detection of error in the studied function and its impact on the system.

• Cost of model development and exploitation: This criterion is related to the cost estimation of the simulation model development for the concerned structure or function, and its exploitation.

• Duration of model development and exploitation: This criterion is related to the time estimation needed for the development of a simulation model for the concerned structure or function, and the time required to operate this model. It namely measures the impact of the decision on the project schedule.

The criteria defined in this section can be divided into two groups. The first one concerns those that can be estimated based on computation methods or with an identified procedure from the literature ("Importance of requirement" and "Complexity level" are retained in that category). The second category is composed of criteria that are still hard to estimate with computerized methods ("Benefits of detecting errors", "Cost of model development and exploitation" and "Duration of model development and exploitation" are retained).

A methodology of estimation for computerizable criteria

In order to provide an efficient realization on the procedure, some criteria can be automatically estimated with computerized methods. Such computerization enables to obtain objective evaluation of alternative in short analysis time. Computerization are proposed for the criteria: Importance of requirements and Complexity level.

Importance of requirements. In [START_REF] Sotelo | Avez-vous identifié toutes les parties prenantes?[END_REF]) the authors presented a method that rates the importance of a requirement based on the analysis of its relevance and the influence of the stakeholder holding it. It is proposed to consider this method for measuring the importance of requirements in this work. Each function or structure under study is the result of requirements expressed by stakeholders or transformed from their expressed need. To give an estimation to a function about the importance of requirements criterion, all its related requirements are listed, and for each one, the following indicators are defined:

• 𝐼𝑟(𝑟): This indicator is related to the estimation of the relevance of requirement relying on a scale of 1 to 3 (1: indifferent; 2: relevant; 3: extremely relevant).

• 𝐼𝑝𝑝(𝑟): This indicator is related to the decision-making authority of the stakeholder source of this requirement. They are judged on a scale of 1 to 3 (1: Can sometimes influence decisions and determine the actions to be taken; 2: Can make decisions and/or determine the actions to be taken; 3: Can make decisions and/or determine the actions to be taken)

To support the PSEM it is assumed that this way of treating the requirements should be used in the projects. It is assumed that in an MBSE approach, these 2 indicators are parameters of each requirement and are retrievable from the project' models. Fixing these parameters shall be made during a classic requirements analysis in the first steps of projects (see for example Stakeholder Needs and requirements definition and System Requirement definition processes of IEC 15288). In a well-defined SE process each function or component have a set of related requirements since they exist in the architecture to reach stakeholder needs. Thus, it is possible to retrieve for each function or component under study its related requirements and their Ir and Ipp indicator. It is then possible to compute the importance of this requirement set for each function or component. This importance is seen as a combination of requirements' relevance and the authority of the stakeholders owning them. The used formula measures the proportion in the importance of the requirements set related to the studied function over the full set of requirements importance:

∀ 𝑓 𝑖 (𝐹𝑜𝑛𝑐𝑡𝑖𝑜𝑛 i)𝜖𝐹, 𝐼𝑚𝑝𝑅𝑒𝑞(𝑓 𝑖 ) = ∑ 𝐼𝑟(𝑟) * 𝐼𝑝𝑝(𝑟) 𝑟∈𝑅 𝑖 ∑ 𝐼𝑟(𝑟) * 𝐼𝑝𝑝(𝑟) 𝑟∈𝑅 𝑅 = 𝑆𝑒𝑡 𝑜𝑓 𝑟𝑒𝑞𝑢𝑖𝑟𝑒𝑚𝑒𝑛𝑡𝑠; 𝑅 𝑖 = 𝑆𝑒𝑡 𝑜𝑓 𝑟𝑒𝑞𝑢𝑖𝑟𝑒𝑚𝑒𝑛𝑡𝑠 𝑜𝑓 𝑓 𝑖 . 𝐹 = 𝑆𝑒𝑡 𝑜𝑓 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛𝑠 𝑢𝑛𝑑𝑒𝑟 𝑠𝑡𝑢𝑑𝑦.
The obtained score is then automatically translated to the 7-level scale for its suitable use in the AHP process (see table 4). Complexity level. To measure the complexity level of the function or component under study, it is assumed, as part of an MBSE approach, that computerized architecture models are available. In the literature, indicators as NodeRank [START_REF] Bhattacharya | Graph-based analysis and prediction for software evolution[END_REF]) are proposed to measure the complexity of elements of an architecture. This measure aims at assigning a numerical weight to each node in a graph (The graph is derived from the architectural model under study), to measure the relative complexity of each node in the system. This rank is inspired from PageRank [START_REF] Brin | The anatomy of a large-scale hypertextual web search engine[END_REF] which represents the stationary distribution of the graph interpreted as a Markov chain. The analogy is made with often called functions or subsystems that influence a lot of surrounding subsystems. Several ways exist to define and compute the NodeRank. Here, the following recursive calculation is used. It is assumed the architecture is defined by an oriented graph. For example, the functions are the node and their oriented interfaces with other functions define the oriented graphs. For a node 𝑖, let 𝑁𝑅(𝑖) be its NodeRank, 𝑁𝑅 -(𝑖) its internal NodeRank and 𝑁𝑅 + (𝑖) its external NodeRank. The set INi contains all nodes 𝑣 that have an outgoing edge to 𝑖, and the set OUTi contains all the nodes 𝑣 that have an ingoing edge to 𝑖.

𝑁𝑅(𝑖) = 𝑁𝑅 -(𝑖) + 𝑁𝑅 + (𝑖) | 𝑁𝑅 -(𝑖) = ∑ 𝑁𝑅 -(𝑣) 𝑂𝑢𝑡𝐷𝑒𝑔𝑟𝑒𝑒(𝑣) 𝑣∈𝐼𝑁 𝑖 (1) | 𝑁𝑅 + (𝑖) = ∑ 𝑁𝑅 + (𝑣) 𝐼𝑛𝐷𝑒𝑔𝑟𝑒𝑒(𝑣) 𝑣∈𝑂𝑈𝑇 𝑖 (2) 𝑂𝑢𝑡𝐷𝑒𝑔𝑟𝑒𝑒(𝑣) = 𝐶𝑎𝑟𝑑(𝑂𝑈𝑇 𝑣 ) | 𝐼𝑛𝐷𝑒𝑔𝑟𝑒𝑒(𝑣) = 𝐶𝑎𝑟𝑑(𝐼𝑁 𝑣 )
Equal 𝑁𝑅 -and 𝑁𝑅 + values are initially assigned to all nodes. The operation (1) and ( 2) are iterated until values converge. The higher the NodeRank of a vertex "𝑖" is, the more complex "𝑖" it is in the system.

Table 5 is used for the transition to the estimations value used in AHP. The two indicators defined in this section can be automatically evaluated for each function in the model. This is a high gain of time for the analysis.

Estimation for hardly computerizable criteria

When no computerizable method exist to estimate criteria, experts' estimation is to be used. Their evaluation is left to experts because of their subjective nature or of the difficulty to find a relevant computation technique. In these cases, it exists some conventional methods that allow giving a judgment. In order to gain in precision in the evaluation of these criteria several approaches may be used. Similarities may be searched with preceding projects and if large database of projects exist it should be important to deploy treatment from the Big Data technologies. The second type of approach is to use formalized evaluation processes. Each alternative shall be analyzed through the same examination process. Such procedure should reinforce the quality and homogeneity of the evaluation. Finally, group decision techniques can be used or combined with the two above mentioned approaches as: consensus technique, majority rule or Brainstorming (participatory problem-solving method based on the spontaneous creativity of the participants). The use of this kind of method contributes to obtain consistent estimation of alternative performance regarding the selected criteria.

In the described method it is assumed that a grade on the 7-level scale is given for each criterion and alternative. This means that the performed evaluation is translated to common scale making relevant the multi-criteria usage. The concerned criteria in this proposal are: Benefits of detecting errors, Cost and Duration of model development and exploitation.

As examples of good practices to consistently evaluate the given criteria, several methods can be quoted.

For the "benefits of detecting errors" it can be imagined to use risk analysis as FMEA to evaluate the criticality of the explored alternatives (e.g. system functions or parts). For the "cost and duration of model development and exploitation", techniques derived from software cost estimation field can be used. As presented in [START_REF] Boehm | Software development cost estimation approaches-A survey[END_REF], some are based on previous projects analysis, other are based on expert judgment. Work as [START_REF] Jørgensen | A review of studies on expert estimation of software development effort[END_REF] shows that expert judgements are relevant when processes are supporting the evaluation to reduce the human biases. In [START_REF] Jørgensen | A review of studies on expert estimation of software development effort[END_REF], 12 principles to support the cost estimation process are proposed based on empirical evidence.

It is structured in 2 main steps:

Method setting: (see the first part of Figure 1). The first step of the AHP method is to structure the preference of the company towards the selection problem. It is done by realizing a pairwise comparison of criteria importance regarding the decision. This step is to be done once for every study. As an example, our tool includes a first version of this comparison. This comparison relies on the results of the survey mentioned in section 3. The rate of participants supporting the use of a given criterion in decisions has been used to set criteria comparison. The difference in rating between criteria has been computed to set the dominance of a criterion over another. The transition from the different rate values to the Saaty scale is made following table 6. These criteria comparison may be readapted for each company if needed. This choice shall be made regarding company strategy and type of project. The user may also need to add new criteria compared to the 5 earlier proposed. In this case, the initial matrix is updated and has to be enhanced with personal estimation for pairwise criteria comparison regarding these new criteria. The AHP method is very flexible for criteria adding and modification.

At the end of this part, the PSEM computes the performance vector related to the criteria, putting forward the importance of each criterion. Alternatives comparison: (see the second part of Figure 1). In this step, alternatives are retrieved from the system model. Alternatives can be a set of functions, components, or subsystems for which it is necessary to identify if it is important to further detail them and to treat them through a simulation process.

The user is asked to estimate each alternative against each criterion as described in section 4.3. The user can modify his estimates throughout the process. Once all alternatives estimations are complete, the PSEM tool computes the Pairwise Alternative Comparison Matrix (PACM), which describes the balance between alternatives for each criterion.

The proposed methodology allows the user to take a manual modification of PACM. This step aims at giving more freedom to the user and to highlight his personal experience in the relevant cases. One of the objectives of the PSEM at this step is the flexibility regarding the user's level of knowledge of alternatives, facilitating the use and avoiding falling into inconsistent judgments if the user is not an expert in the field.

At the end of this step, the PSEM tool constructs a new matrix which includes Priority Vectors of Alternatives for the various Criteria (PVAC).

Using the criteria performance vector mentioned in the first part and PVAC, the tool computes the proposed ranking of alternatives. The proposed PSEM tool aims at helping managers to clarify their judgment and at supporting their decision making for the use of simulation, but it cannot in any way replace the human who retains the primary responsibility for decision-making.

Example of PSEM application

To exemplify the proposed PSEM, a demonstrator tool has been developed and tested on an example project.

Project quick description

The project studies the development of a physical operations management simulation platform. The purpose of this platform is to reproduce an industry 4.0 environment for research and teaching. To this end, mobile robots are developed to serve in the production plant. The PSEM is proposed to be used on the system functional architecture model of the robot, to study the functions to simulate in priority and to further detail.

After analyzing the system functional architecture model of the robot (see figure 3), the following functions have been identified: Decide Destination, Navigate, Sense Environment, Communicate State, Set Operating Mode, Manage Energy, Move and Register Activity. The primary setting of criteria comparison has been defined though the exploitation of the industrial survey results (see table 7). The importance of each criteria has been defined by the survey participants and the figures given in table 7 are synthetizing their opinions. As explained in section 4.4, the more popular a criterion is in the survey the more important it is in the decision. Then, the alternatives are estimated for each criterion, as described below:

• First category of criteria: o Importance of requirement: All the requirements related to the functions are listed as the first step. Then the methodology described in section 4.3 is applied, by computing the importance degree of each function (see table 8). o Complexity level: Before applying the methodology described in section 4.3, it is necessary to draw the alternative interaction graph (figure 2) derived from the functional architecture model under study (made with a SysML Internal Block Diagram, see figure 3). This graph sums up the information flow between the defined functions and facilitates the enumeration of the IN and OUT degrees of each alternative.

Results

After computation, the results (see table 11) suggest prioritizing the study of the communicate state function. The detailed information given by the PSEM solver permits to trace how the priority of each function is defined. In the case where a significant advantage is given to the importance of requirement criterion, the navigate function would be suggested as a priority considering its rate of 41.9% of priority in this criterion. The results are based on the criteria inter-comparison and on the alternative evaluation regarding criteria. Changings in the evaluations may obviously change the proposed ranking. The robustness of the evaluation is based on the consistency of the performed evaluations.

These results are first based on the criteria importance vector that is drawn from the survey results mentioned in section 3. This vector is expressing companies' priority in the decision. It was therefore very important to be able to create a first relevant version of the vector based on multiple opinions to obtain a generic proposal. The exploitation of the field study was thus important to tune the PSEM. This criteria importance vector can be adapted by company willing to work with their own preferences by performing the cross comparison of each utilized criteria. The robustness of the obtained results regarding criteria weight can be analyzed using techniques as weight-sensitivity analysis as in [START_REF] Chen | The spatial framework for weight sensitivity analysis in AHPbased multi-criteria decision making[END_REF]).

To understand the robustness of results, it shall be considered that the judgment of the second group criteria relies on a human estimation and can vary depending on the user of the PSEM. This underlines the importance to work with an objective measurement method of criteria. For computerized estimated criteria, the robustness of the estimates is ensured by its objectivity. For the other criteria, the consistency of the measurement method is important. For such criteria, formalized evaluation procedure should be used or guidelines as the ones given in (Jorgensen 2004) applied.

In this proposed setting, a variation of the non-computerized criteria has been tested to evaluate the impact on the 2-top ranked alternatives and thus the robustness of the provided priorities. A variation of + or -1 is tested to simulate a hesitation of grade around the selected value. Table 12 shows the evolution of the ranking of Navigate and Communicate State functions when their scores for the experts rated criteria vary. In the tested variations, the direction and amplitude of evolution of the cost and the duration of model development and exploitation have been taken the same since they are linked. This is done to reduce the set of presented variation combination. Regarding the homogeneity of the obtained results, this reduction of scenario number is felt relevant. It can be observed that in any case, the functions remain in the top 2 demonstrating the relevance and robustness of the given ranking.

Conclusion and Discussion

The results of the CRRA survey have shown a need for further development in MBSE for a better integration with the existing SE process. It also raised a need to further improve skills in SE and modeling activities. The performed survey points out that simulation is perceived to increase the quality of tests performed and delivered projects, it is also considered as a key element for VVT strategies.

According to respondents, an improvement in the quality of certain practices has been observed following the use of simulation, mainly in architecture and design, requirements analysis, and verification and validation. The survey conducted by [START_REF] Huldt | State-of-practice survey of model-based systems engineering[END_REF] put forward a level of improvement in the quality of practices regarding the use of an MBSE approach, mainly in architecture and design, as well as the analysis of requirements. By analyzing these two results, it can be noticed that the MBSE approach and simulation have a considerable impact on the practices mentioned earlier (e.g. architecture and design, requirements analysis). Nevertheless, the studies revealed that the return on investment of such practice must be proven to convince managers of their potential benefits. Moreover, it is admitted that actions are to be taken to master the MBSE and simulation deployment costs. The proposition highlighted in this work is to better use simulation to overcome some barriers to MBSE deployment. The specific point treated in this work is on prioritizing simulation actions as it is impossible to simulate everything in projects.

In this context, a prioritization methodology (PSEM) is set up to assist project leaders in taking decisions related to the use of simulation. The methodology is tailored to treat a set of criteria highlighted in the CRRA survey and has been tuned to reveal the estimation of their importance found in the survey. The PSEM application in this paper is restricted to deal with the analysis of the functional architecture model of systems. A procedure for automatically estimating the two first criteria relevant to this SE step has been developed. The proposed PSEM uses standard estimation done by experts for criteria with no wellestablished method for their estimation (e.g. benefits of detecting errors during the first phases of system development, cost and time needed for the development and deployments of a simulation). Future research can expand the framework to look further into the procedures or methods for these criteria estimation. These methods should be integrated into an MBSE environment and could, for example, come from machine learning over a set of past projects. The methodological framework presented by [START_REF] Madni | Economic analysis of model-based systems engineering[END_REF] represents a good basis for complexity estimation. The amount of knowledge and information required to develop a function or a system can be considered as key parameters for a complexity estimation method. The economic analysis of MBSE implementation presented in the framework mentioned earlier can also be inspiring for the development of a methodology to analyze and estimate the potential gains from the simulation.

In future work, evaluations of the framework on real industrial cases should be realized to identify possible hinders in PSEM utilization and to quantify the benefits of its use. Building such experiments is clearly a hard point from resource availability perspective since it would necessitate concurrent engineering teams working with and without the PSEM on an identical project.

Another axis to advance this work is to use the same kind of approach to drive the modeling effort throughout the projects. It is a hard task to give evidence on how to decide the part of the model that has to be refined. It is about giving evidence on the relevant effort to put on modeling in projects. Such a contribution would be valuable to convince managers that MBSE activities are mastered and provide an unambiguous return on investment.

Finally, it is also interesting to work on the integration of the proposed PSEM with existing modeling platforms, for the automation of the alternative estimation process within the functional architecture model of the system.

Table 1 : Workshops panel composition (WS) ID Industrial Sector Expert Profile Employees Business model Company experience in SE

 1

	Respondent
	experience in
	SE

Table 3 : Descriptors definition, and suitable multi-criteria methods

 3

Table 4 : Transition table for the estimation value of the importance of requirement criteria Importance of requirement Estimation scale [0; 0.05[ 1 [0.05; 0.10[ 2 [0.10; 0.15[ 3 [0.15; 0.20[ 4 [0.20; 0.25[ 5 [0.25; 0.30[ 6 >=0.30 7

 4

Table 5 Transition table for the estimation value of complexity level criteria

 5

	NodeRank Value	Estimation scale
	[0; 0.4[	2
	[0.4; 0.8[	3
	[0.8; 1.2[	4
	[1.2; 1.6[	5
	[1.6; 2.0[	6
	>= 2.0	7

Table 6 : Transition table from difference rate value to Saaty scale

 6

	Difference rate value %	Saaty scale
	0	1
	[0; 5[	2
	[5; 10[	3
	[10; 15[	4
	[15; 20[	5
	[20; 25[	6
	>=25	7

Table 7 : Criteria comparison Criteria Importance of requirements Complexity level Benefits of detecting errors Cost of model development and exploitation

 7

	Duration of
	model development	Eigenvector
	and exploitation

Table 8 : Importance of Requirement

 8

		Function requirement group	Ir	Ipp	Importance Degree	Importance Estimation Value
	Decide Destination	E-F4 E-F5	3 2	3 2	0.188	4
		E-F1	3	2
	Navigate	E-F3 E-F2	3 1	3 1	0.362	7
		E-F4	3	3
	Sense Environment	E-F6	3	2	0.087	2
	Communicate State	E-F9	3	3	0.130	3
	Set Operating Mode	E-O9	2	2	0.058	2
	Manage Energy	E-I3	1	2	0.029	1
	Move	E-F6	3	2	0.087	2
	Register Activity	E-I6	2	2	0.058	2

Table 11 : Alternatives ranking given by the PSEM Rank Alternatives Priority % Importance of requirement Complexity level Benefits of detecting errors Cost of model development and exploitation Duration of model development and exploitation

 11

	1	Communicate State	0,219	0,112	0,235	0,176	0,233	0,233
	2	Navigate	0,182	0,419	0,141	0,094	0,139	0,233
	3	Set Operating Mode	0,135	0,064	0,235	0,176	0,080	0,080
	4	Sense Environment	0,130	0,064	0,034	0,176	0,233	0,139
	5	Move	0,108	0,064	0,082	0,054	0,139	0,139
	6	Register Activity	0,084	0,064	0,082	0,176	0,080	0,080
	7	Manage Energy	0,080	0,037	0,141	0,094	0,048	0,048
	8	Decide destination system	0,062	0,174	0,051	0,054	0,048	0,048

Table 12 : Impact of the variation of estimations for non-computerized criteria on the rank of the top 2 alternatives Benefits of detecting errors Cost of model development and exploitation Duration of model development and exploitation Rank of the alternative in the final results

 12

		6	6	6	1
		5	6	6	1
		6	5	5	1
	Communicate State	6 4	4 6	4 6	2 1
		4	5	5	1
		5	4	4	2
		4	4	4	2
		5	5	6	1
		4	5	6	1
		5	4	5	2
	Navigate	5 3	3 5	4 6	2 1
		3	4	5	2
		4	3	4	2
		3	3	4	2

Published in Computers In IndustryVol.131 oct.2021

CRRA: Regional Chapter Rhone Alpes, http://afis.community/

INCOSE: International Council on Systems Engineering, https://www.incose.org

AFIS: French Association of Systems Engineering, http://afis.community/; French chapter of INCOSE

Alternative comparison methodology

The proposed methodology produces a ranking of system functions, showing the priority of studying them through simulation. The next step consists in computing the NodeRank of each function and to convert it to an estimation value using the 7-level scale as defined in table 2. The results are shown in table 9. • Second category of criteria: The criteria in this group rely on experts' estimation. In this work, a personal estimation is given for each alternative for the concerned criteria. The used estimations are shown in table 10.