Giovanni Lugaresi
email: giovanni.lugaresi@centralesupelec.fr

Sofia Gangemi

Giulia Gazzoni

Andrea Matta

Online Validation of Digital Twins for Manufacturing Systems

Keywords: digital twin, online validation, smart manufacturing, discrete event simulation

Digital twins are considered among the most important instruments to optimize production and support decision making. To benefit from their functionalities, it is essential to guarantee a correct alignment between the physical system and the associated digital model, as well as to assess the validity of the digital model online. This operation should be conducted rapidly and with a limited available data set, especially in highly dynamic contexts. Further, the whole time behaviour of a system may be of interest rather than the sole average performance. Traditional validation techniques are limited because of the restrictive assumptions and the need of large amounts of data. This work defines the problem of checking the validity of digital twins for production planning and control while the physical system is operating. A method describing the data and the types of validation is proposed including a set of algorithms to be used at different levels of detail. The congruence between the physical system and the corresponding digital model is measured by treating data as sequences, and measuring their similarity level with digitally-produced data exploiting a proper comparison technique. The numerical experiments show the potential of the proposed approach and its applicability in realistic settings.

Introduction

Recently, manufacturing systems have been characterized by disruptive innovations driven by the digitization efforts of the Industry 4.0 revolution. The change is enabled by the development of new technologies such as Internet of Things, big data analytics, cloud computing and artificial intelligence. These technologies allow to interconnect elements within the production facility, to collect and analyse large amounts of data as well as to reproduce and predict the behaviour of a system. The Digital Twin (DT) is a growing technology capable to improve the performance of a manufacturing system and support decision making.

With reference to manufacturing systems, the DT is defined as: "a virtual representation of a production system that is able to run on different simulation disciplines that is characterized by the synchronization between the virtual and the real system" [START_REF] Negri | A review of the roles of digital twin in cps-based production systems[END_REF].

Within the scope of production planning and control, the virtual entity can be represented by a Discrete Event Simulation (DES) model. The DT is a dynamic concept and it can be seen as the evolution of simulation, integrating modelling and optimization capability. Simulation has been used since many years for the design, validation, and testing of manufacturing systems as well as for improving decision making. These applications traditionally take place in the early life cycle stages of a system. However, simulation can be applied also in the operational stages for offline decision-making, proactive reasoning, and reaction definition [START_REF] Monostori | Cyber-physical systems in manufacturing[END_REF]. The DT is able to collect and use real-time data from the field. Differently from traditional simulation approaches, a bi-directional exchange of information and the synchronisation between the physical and the virtual components are essential features of a DT.

In addition, the DT capability of providing feedback actions to the physical system allows to improve promptness and efficiency of decision making. Because of the physical-digital fusion, the DT is capable of optimizing a system during the entire lifecycle allowing short-term decision-making [START_REF] Liu | Review of digital twin about concepts, technologies, and industrial applications[END_REF].

Indeed, short-term decision making requires a rapid detection of disruptions, and any delay in the implementation of corrective actions may cause a negative impact on production, reflecting on economic losses for the company. To prevent this scenario, it is essential to have a model coherently representing a real system along its whole life-cycle, adapting to possible changes. As a consequence, to benefit from the DT functionalities, it is essential to assess online the validity of the virtual component.

Existing digital twin applications do not discuss how the validity of the digital model can be assessed online. Also, significant contributions on the digital model validation for short-term applications are scarse. Traditional validation techniques rely on statistical methods which require large data sets and multiple independent replications due to the stochastic nature of the physical system [START_REF] Balci | Some examples of simulation model validation using hypothesis testing[END_REF]. These requirements affect the promptness with which disruptive events are detected: the most recent behaviour is averaged with historical data and potential deviations might not be detected. Moreover, since only one set of data can be acquired from the field, batching procedures have to be applied to avoid auto-correlation effects and satisfy the assumption of independence. Batching leads to data averaging and the performance trend might be lost. As a consequence, changes that do not impact on the average behaviour of the system might not be identified and corrected. Given these characteristics, it is possible to conclude that traditional statistical approaches are not suitable for online applications, especially for manufacturing systems that do not produce large data sets.

This work represents the first study that defines the problem of online validation of digital twins by discussing the technical issues emerging when the sets of data generated by the physical and digital system are compared for validation purposes. This work also aims to overcome the aforementioned limitations by providing a new methodology to validate online DES models representing a discrete event system.

A special focus is given to manufacturing systems, that embody one of the main DT application fields.

The method deals with a limited data set from a single simulation experiment, and it is capable to analyse the evolution in time of the system behaviour. The digital model validity is assessed measuring the similarity between real system data and digital model data. As example, Figure 1 shows the online collection of inter-departure times from a manual assembly station. These data are typically saved in the Fig. 1: Example -Comparison of data sequences generated by a real system composed by two manual assembly stations and its corresponding digital model. The inter-departure times are collected from the physical system (a) and measured also in the digital space (b). A similarity indicator is used to assess the validity of the digital model. form of time-series and it is possible to treat them as sequences. Hence, the proposed method measures the similarity level with digitally produced data by means of a proper sequence comparison technique.

The similarity level is then used to determine if the digital model is valid.

The rest of the paper is organized as follows. The related works and the research gap on online validation are discussed in section 2. The developed methodology to assess online the validity of a DES model is explained in section 3. Section 4 presents the numerical experiments to test the applicability of the proposed methodology. Finally, conclusions are drawn in section 6.

Related Literature

This section summarizes significant contributions in the literature that study the consistency between manufacturing systems and their digital models. Table 1 summarizes the main contributions and their characteristics. The focus is on methods that address the validation of the digital model, either offline or online. Finally, the research gap is highlighted.

Offline Validation of Digital Twins

Assessing the correctness of the digital model is a fundamental step of its development phase. Indeed, digital models are used to support decision making in the real world and it is important to guarantee the reliability of the obtained results. The process of valuation of the model's correctness and capability to reproduce a system's behaviour is typically called verification and validation (V&V). According to Sargent [START_REF] Sargent | Verification and validation of simulation models[END_REF], V&V of a simplified model development process includes four parts: (1) conceptual model validation, (2) computerized model verification, (3) operational validation, and (4) data validity. This work focuses on the operational validation procedure defined as "determining that the model's output behaviour has a satisfactory range of accuracy for the model's intended purpose over the domain of the model's intended applicability", that in the remainder of this paper is simply referred to as validation. It is important to underline that the model validity is always checked under specific experimental conditions and for specific purposes. Moreover, results of the validation process depend on the level of accuracy defined by the user [START_REF] Sargent | Verification and validation of simulation models[END_REF].

Balci [START_REF] Balci | Validation, verification, and testing techniques throughout the life cycle of a simulation study[END_REF] proposed a taxonomy on more than seventy-five V&V testing techniques composed of four main categories, each comprising a wide set of techniques, characterized by an increasing level of formality and complexity: (1) informal techniques based on human reasoning and subjectivity; (2) static techniques established on the model design and source code (i.e., no model execution is needed);

(3) dynamic techniques dependent on the model execution and comparison of the output behaviour; (4) formal techniques rely on a mathematical proof of correctness. As stated by Sargent [START_REF] Sargent | Verification and validation of simulation models[END_REF], model validation consists in comparing the output behaviour of the system with the one of the simulation model, this approach is in accordance with the techniques identified in the dynamic category of the taxonomy proposed by Balci [START_REF] Balci | Verification, validation, and testing[END_REF]. Results of the comparison determine the level of accuracy of the digital model.

Balci [START_REF] Balci | Verification, validation, and testing[END_REF] proposed that V&V should be conducted continuously throughout the entire lifecycle of a validation study in order to identify and rectify deficiencies. The following errors must be prevented: (1) type I error, which is committed when the simulation results are rejected when in fact they are sufficiently credible; (2) type II, which represents a situation in which invalid simulation results are accepted as valid; (3) type III error is when the wrong problem has been solved hence when the problem formulation does not contain the actual issue. When simulation is used to support decision-making, fatal consequences may arise from type II and III errors. For this reason, early detection and correction of errors is extremely important.

A model is always an abstraction of a system and a perfect representation cannot be achieved, hence validity is checked with respect to the application objectives. The outcome of V&V process should be expressed as a percentage value of credibility instead of a binary variable assessing the correctness in absolute terms. Due to time and cost constraints, it is not possible to test a model under all input conditions, hence the credibility required depends on the predefined objectives [START_REF] Banks | Handbook of simulation: principles, methodology, advances, applications, and practice[END_REF]. In addition, Balci [START_REF] Banks | Handbook of simulation: principles, methodology, advances, applications, and practice[END_REF] distinguishes between two types of simulation to obtain data for validation purposes.

In a Trace-Driven Simulation (TDS) the model is run using as input the same trace data collected from the system. In a self-driven simulation, the model is run by sampling input data (i.e., technically random variates) from probabilistic models. Usually, input data are modelled by fitting probability distributions to the observed data.

Similarities with model validation can be found in conformance checking approaches. Conformance checking is a technique belonging to the wider spectrum of process mining (van der Aalst, 2016). Process mining comprises a variety of techniques that, based on event logs, allow the analysis of business processes.

By means of data coming from the event logs, process mining techniques enable to discover, monitor, and improve processes. Within process mining there are three classes of techniques: discovery, conformance checking and enhancement. Conformance checking deals with comparing an available event log of the process with the discovered process model to assess its correctness [START_REF] Naderifar | A review on conformance checking technique for the evaluation of process mining algorithms[END_REF]. [START_REF] Camargo | Automated discovery of business process simulation models from event logs[END_REF] propose a technique for automated discovery of a business process simulation model using process mining. To measure the accuracy of the discovered model, a variant of the Damerau-Levenstein distance [START_REF] Bard | Spelling-error tolerant, order-independent pass-phrases via the damerau-levenshtein string-edit distance metric[END_REF] is used to compare traces1 in the event log. In addition, traces are paired using the Hungarian algorithm [START_REF] Kuhn | The hungarian method for the assignment problem[END_REF] to minimize their distance. The similarity between event logs is then assessed as the overall distance between paired traces. The concept behind this technique is similar to validation, nevertheless in the conformance checking field the model accuracy is assessed only once after model discovery, and not during operational phases.

According to [START_REF] Sargent | Verification and validation of simulation models[END_REF], the most common approaches used for model validation are graphical comparison, confidence intervals, and hypothesis tests. These methods encounter their limitations when put to work in a digital twin framework, where decisions must be taken within short time frames, hence the validity of a model has to be checked quickly and with limited need of human intervention. The limitations are evident. Graphical comparisons are characterized by subjective decisions which lead to difficulties in evaluating results. Also, a graphical comparison activity implies that the information flows cannot be fully automatized. On the other hand, confidence intervals and hypothesis tests are based on statistical assumptions that are not always satisfied, for example the availability of enough data points or specific requirements on the statistical distribution (e.g., normality of data). In addition, the comparisons are typically based on the average behaviour of the system. Hence, traditional methodologies are not suitable for the purpose of this work.

Online Validation of Digital Twins

Recently, new studies about online validation have been published: the most relevant are reported in the following paragraphs. [START_REF] Marquardt | Indolence is fatal: research opportunities in designing digital shadows and twins for decision support[END_REF] introduced the importance of online validation when dealing with a DT. In order to provide both strategic and operational decision support, it is fundamental for the digital model to be always updated with respect to the real system. In particular, the paper highlights the effects caused by a model not coherently representing reality, considering both inconsistencies of input parameters and model assumptions. Although explaining the importance of having a model accurately representing the real world, the paper lacks a feasible online validation approach. An attempt to tackle this issue has been proposed by [START_REF] Overbeck | Continuous validation and updating for high accuracy of digital twins of production systems[END_REF], that uses statistical indicators to compare the performances of the physical and digital system. This paper shows the growing interest in the topic but still focuses on the average behaviour of the two systems. [START_REF] Lugaresi | Real-time validation of digital models for manufacturing systems: A novel signal-processing-based approach[END_REF] presented a new methodology to perform online validation based on harmonic analysis. Compared to statistical techniques, this method is capable to achieve reliable results even when applied to a relatively small data set. The main idea behind this procedure is to consider Key Performance Indicators (KPIs) trends as signals. Using the power spectral density [START_REF] Howard | Principles of random signal analysis and low noise design: The power spectral density and its applications[END_REF], it is possible to detect potential deviations between the model and the real system. Namely, the difference between the power spectral densities of the two outputs is used to define a measure called spectral indicator, that can be used to assess the level of consistency of the model. A methodology to discriminate between two systems based on the Fourier transform of the output trajectory is proposed in [START_REF] Morgan | Fourier trajectory analysis for system discrimination[END_REF]. The weighted average of the Fourier coefficient magnitudes is used to detect differences between the two systems under analysis. What differentiates the latter two approaches from other methodologies is the focus on the whole trend of the performance measure rather than on the average value. Nevertheless, both approaches analyse the performance measure in the frequency domain, making them less likely to be applied in the industrial practice.

Recent approaches considered the idea of comparing data traces (or sequences) for validating in realtime digital models. [START_REF] Leroy | Trace comprehension operators for executable dsls[END_REF] introduced a collection of operators for comparing execution traces of state machines models. The operators consist of dynamic trace filtering, trace comparison with models. Also, the comparison methods do not allow to include a random number generator in the digital model, hence they cannot be used while the simulation is running. This is probably the reason why none of the aforementioned works has been applied to manufacturing systems. An algorithm based on sequence comparison techniques for online input validation is proposed in [START_REF] Lugaresi | Online validation of simulation-based digital twins exploiting time series analysis[END_REF] and successfully applied on a simple single server system.

To the best of the authors knowledge, few solutions to the online validation problem for manufacturing systems are present in literature. This work focuses on data that represent material flow events within a production facilities, for instance, the entrance of a part in a machine, the inter-departure times from a station, the beginning of a production operation. Compared to the existing literature, the work that shares most of these objectives is [START_REF] Muñoz | Using trace alignments for measuring the similarity between a physical and its digital twin[END_REF], which also proposes sequence comparison techniques to assess the alignment between a system and its digital counterpart. However, the following main differences can be identified: (1) the proposed approach assumes to use two data sequences taken with the same time steps, and to be able to compare each sequence point-by-point. This is hard to guarantee in complex manufacturing systems, where even the settings of the sensors could determine the misalignment of data sequences [START_REF] Schmetz | The time synchronization problem in data-intense manufacturing[END_REF]; (2) the validation approach does not consider digital models with a random number generation; hence it can directly compare the data sequences; (3) given the aforementioned points, the proposed approach is applied to other domains such as robot trajectory planning, and does not consider manufacturing systems applications. This paper aims at overcoming the aforementioned limitations by developing a methodology that can be applied to validate a digital twin by exploiting real-time data streams from a manufacturing shop floor while operating. Specifically, the approach (1) can be applied to any sort of input data from the real system including a possible misalignment of the traces, (2) allows to still use a random number generator within the digital model while validating it [START_REF] Lugaresi | Real-time validation of digital models for manufacturing systems: A novel signal-processing-based approach[END_REF]; (3) can be applied within manufacturing system applications.

The main contributions of this paper are twofold. First, the paper describes the problem of online validation of digital twins within the scope of production planning and control. It is worth to notice that the same problem can be present in several other contexts in which the physical system can be modeled as a discrete event system, such as logistics and transportation, warehouse management, and supply chain management. The second contribution is the proposal of a general methodology for online validation of digital twins that can be deployed at different levels of detail and in an automated way while the physical system is operating. The proper technique to detect similarities and deviations of the digital model from the physical system may depend on the specific application. In this work, three alternative techniques have been identified from the literature on sequence comparison problems, and adapted for the online validation problem.

Methodology

This work proposes a methodology to perform online validation of digital twins based on DES models.

The starting point of this study is to understand when a model can be defined a good representation of the physical system and what is the useful information to delineate the system behaviour. In general, the concept of model validity can be related with the similarity level between real and digital information.

To be able to assess the correctness of the whole evolution in time of a specific information, both real and digital information can be considered as sequences, and their similarity level can be determined exploiting sequence comparison techniques. Hence, the goal of our methodology is to obtain a similarity indicator (e.g., a value ranging between 0 and 1). The model is defined valid if the indicator is greater than a predefined acceptance threshold set by the user.

In smart manufacturing systems, it is reasonable to assume that an adequate sensor network has been put in place, and data collection is possible within the same level of detail expected by the user. In this context, both real and digital data may be treated as sequences. A sequence is defined as a succession of values ordered in time. The starting point for both real and digital data is the event log, a registry of the events occurring in the system with the associated time-stamps. The events stored in the real event log

depend on which type of information can be collected from the physical system. On the other hand, the digital event log can store all events occurring in the simulation model. In addition to events, the event log is useful to calculate KPIs. For instance, Figure 1 shows as example the extraction of the sequence of inter-departure times from an assembly station. Each time a part is done and it departs the station, the inter-departure time (IDT ) is computed as the time elapsed since the previous part has left the station.

This value is computed and recorded in the company information system. A derived KPI such as the system throughput can be calculated as 1/IDT . The same information can be computed within a digital model.

The following sections provide a deeper understanding on the validation methodology. In section 3.1, the two validation types addressed in this work are defined, namely input and logic validation. Then, in section 3.2 input data and their level of detail are discussed. In section 3.3 the proposed procedure to perform both input and logic validation is introduced. Finally, in section 3.4 the specific techniques that are used for comparing data sequences are presented.

Types of Validation

As introduced by Nelson et al. ( 2013), a system is described by its logic and input data. Hence, the logic validation of the model concerns the building of the model itself and its assumptions. The input validation ensures the correctness of the input data fed to the model. The uncoupling of logic and input validation is useful to understand where a specific deviation has generated, and allows for a faster resolution.

Traditional validation instead is usually performed in one single phase. As a consequence, when a model is declared as not valid it is difficult to understand if the reason is either a wrong logical assumption or the wrong modeling of input data. Following the aforementioned insights, the proposed methodology takes into account both logic and input validation: each time a validation is performed, both aspects are checked.

Levels of Validation

The validation procedure can be adapted to the type of system is being analysed, and consequently to the level of detail of the respective data. This concept is illustrated in Figure 2. A complete set of events must be considered to focus on the whole system behaviour, while a partial set of events may represent only a portion of the first set, depending on a specific user interest (for instance, considering only the start and finish events of a process). For example, referring to Figure 1, the time-stamp associated with the exit of a finished part from station 2 represents an event belonging to the larger set of material flow events. Meanwhile, the inter-departure time represents a local performance indicator that is derived from the aforementioned set of events. Therefore, the corresponding data set has a higher level of granularity.

Accordingly, two levels of validation procedures can be distinguished, respectively (1) event-level validation and (2) performance-level validation. Event-level validation investigates if the sequences of events produced by the physical and digital system are comparable. It is applied when a detailed view on the system behaviour is required. Specific sections of the manufacturing system could be investigated:

for example, the events produced by the process sequence in a manufacturing cell might be of interest within a validation procedure. An event is uniquely defined by the event identifier string. For example, the start of the process on a generic resource 1 can be indicated by the string "S1". Together with the sequence containing information about the event IDs, also the sequence with the associated time-stamps (t) can be used to measure the similarity. Performance-level validation compares the performance of the system with the ones estimated using digital experiments. In this case, only the sequence of KPI values is analysed to assess the similarity. It is possible to evaluate global KPIs, representing the general system performances (e.g., system time, inter-departure time, etc), or local KPIs, representing the performances of specific portions of the system (e.g., waiting time, buffer level, etc). This choice is done considering that performance-level validation is characterized by a lower level of detail. In addition, KPIs usually already contain information about time. For instance, the sojourn time in an assembly station is a performance indicator that is dependent on the time dimension. KPIs level validation focuses on the general performance of the system by comparing sequences of KPIs and it is characterized by a lower level of detail.

Fig. 2: Representation of the levels of detail achievable with the proposed validation methodology.

Online Validation Procedure

A model is defined valid if both its logic and inputs properly represent the ones of the real system. Based on this consideration, the validation procedure is divided in two parts, one for each aspect; Figure 3 shows an overview of the proposed validation procedure. First, logic validation checks the correctness of the model logical layout and assumptions. In this phase, digital data are obtained following the Trace Driven Simulation (TDS) [START_REF] Law | Simulation Modeling & Analysis[END_REF]. Namely, each parameter of interest is monitored on the real system (e.g., the processing time of a welding station). The sensors in the real manufacturing system collect these data, and each data point is made available to be reproduced in the digital model. Input validation, shown at the bottom of Figure 3, assesses the correctness of the input data distributions. In this case, random variates must be generated in a way that can be comparable with the data collected from the physical system. This is done using a particular procedure named quasi Trace-Driven Simulation [START_REF] Lugaresi | Real-time validation of digital models for manufacturing systems: A novel signal-processing-based approach[END_REF]. The validation procedure is conducted comparing a set of real and digital data to obtain a similarity indicator ranging between 0 and 1. Acceptance thresholds are defined in accordance to the user requirements. If data are characterized by an appropriate similarity level both in terms of logic and input, the model is considered a valid representation of reality and it can be used for decision making purposes. On the other hand, if the digital model is not capable to represent coherently the real system behaviour, an update is needed. The logic update implies a modification of the simulation logic model or even the generation of a new model, while an input update requires a modification of the input simulation models.

Sequence Comparison Techniques

This section presents the techniques that have been identified and adapted to calculate the similarity indicators in the logic and input validations. The choice of sequence comparison techniques has been driven by the need to use a limited amount of data for the validation. Indeed, the selected techniques have been historically developed to compare limited data arrays, hence they fit the requirements for shortterm validation. Starting from the techniques available in literature, proper normalization procedures have been done and additional parameters have been introduced to achieve an indicator capable to measure the similarity level of two sequences.

For coping with event-level validation, the sequence comparison techniques have been selected to be able to deal with sequence of strings. Moreover, in order to not declare equal two events significantly shifted over time, the associate sequence of time values has been added to the comparison. Regarding performance-level validation, sequence comparison techniques able to deal with sequences of real numbers are investigated. Methods applicable to both sequences of strings and numbers are included in the research, in order to compare both sequences of events and KPIs. Moreover, the techniques have been selected

taking into account that data sequences may be of different length and that there must not be limitations regarding the minimum data length. As a result, three techniques for the online validation of digital twins are proposed and listed in the following sections.

The following notation is used:

-a and b represent two sequences of data. For simplicity, the same notation is used in case the two sequences are made by numbers or strings (i.e., event identifiers). m and n represent the length of a and b, respectively.

a [i] represents the i-th point of the sequence a.

a [i:j] represents the portion of a sequence from the i-th to the j-th point.

t a,[i] represents the time-stamp associated with the i-th data point in the sequence a.

d(x, y) is the Euclidean distance between two values x and y.

Table 2 collects the notation used along the whole paper.

Longest Common Sub-sequence (LCSS).

This technique is introduced to deal with string comparison. It is mostly applied to bio-informatics and computational linguistics fields [START_REF] Bergroth | A survey of longest common subsequence algorithms[END_REF]. This technique aims at identifying the maximum number of matching elements in the same sequential order. The original version of this technique does not take into account a possible shift over time of the two sequences, a clear limitation in case the digital model is simply reproducing events with a constant delay respect to the physical system. In this section a modified version of the LCSS technique is exploited to compute the similarity measure. This modification allows to compare sequences of event identifiers together with their respective time-stamp sequences t. Namely, two events with identical identifiers which are produced within a time interval longer than a threshold value δ are not declared similar. The parameter δ is defined as the indifference threshold, since a deviation smaller than δ is not sufficient to say the digital model is not valid. The choice of the indifference threshold is problem-dependent according to the validation-on-purpose concept [START_REF] Sargent | Verification and validation of simulation models[END_REF]. In case of string identifiers, the distance d is equal to 0 if the strings are identical, 1 otherwise; for example, d("s1", "s1") = 0, while d("s1", "f 2") = 1.

The similarity measure is a matrix, where each element L [i,j] (a,b) corresponds to the comparison of the sequences a [0:i] and b [0:j] , as reported in Equation (1).

L [i,j] (a,b) =            0 if i = 0 ∨ j = 0 L(a i-1 , b j-1 ) + 1 if d(a i , b j ) = 0 ∧ d(t a,[i] , t b,[j] ) < δ max{L(a i-1 , b j ), L(a i , b j-1 )} else.
(1)

Table 2: Notation used in this paper.

Variables Explanation

Data sequences:

a
Vector of values representing a data sequence.

ta Vector of acquisition time-stamps of each data point in the sequence a.

ai i-th element of data sequence a.

a [i:j]
Portion of the data sequence a, from the i-th to the j-th value.

Indicators: τ2 Time to detect that the system is correct after the resolution of the disruption.

L [i,j] (
Then, the LCSS-based similarity indicator, ranging between 0 and 1, is obtained dividing

L [m,n] (a,b)
by the length of the shortest sequence as reported in Equation (2).

Φ LCSS (a,b) = L [m,n] (a,b) min(m, n) . ( 2
M [i,j] (a,b) =            0 if i = 0 ∨ j = 0 M (a i-1 , b j-1 ) + 1 if d(a i , b j ) < ϵ max{M (a i-1 , b j ), M (a i , b j-1 )} else.
(3)

Then, the mLCSS-based similarity indicator is ranging between 0 and 1. This is achieved dividing

M [m,n] (a,b
) by the length of the shortest sequence as reported in Equation ( 4).

Φ mLCSS (a,b) = φ mLCSS[m,n] (a,b) min(m, n) . (4)
3.4.3 Dynamic Time Warping (DTW). One of the most common techniques to measure the distance between two temporal sequences in the time domain analysis is Dynamic Time Warping [START_REF] Müller | Dynamic time warping. Information retrieval for music and motion[END_REF]. It is based on an algorithm able to identify the best alignment between two sequences and measure their distance. The procedure computes all possible alignments between each point of both sequences and then identifies, as the best alignment, the one that minimizes the distance. The achieved alignment is defined as warp path. The distance measure is a cumulated distance since it represents the sum of the distances between all matching points in the path. The DTW distance measure is a matrix, where each element D [i,j] (a,b) corresponds to the comparison of the sequences a [0:i] and b [0:j] , as reported in Equation

(5).

D [i,j] (a,b) =            0 if i = 0 ∧ j = 0 ∞ if i = 0 ∨ j = 0 d(a i , b j ) + min{D(a i , b j-1 ), D(a i-1 , b j ), D(a i-1 , b j-1 )} else.
(5)

An additional normalization procedure allows to achieve the DTW similarity indicator ranging between 0 and 1. The two sequences a and b are normalized with respect to their maximum element, as in equation ( 6).

where m and n are the length of the sequences a and b, respectively.

Equation ( 5) is used to compute the cumulative distance D [m,n] (a, b), which is normalized by dividing it by the maximum length of the two sequences. Finally, the DTW-based indicator is obtained as indicated in equation ( 7):

Φ DT W (a,b) = 1 - D [m,n] (a,b) max(m, n) . ( 7
)
4 Numerical Experiments

This section illustrates the experiments carried out to assess the applicability of the proposed methodology. The behaviour of each technique is tested with respect to the associated parameter settings. Two experimental campaigns are performed, respectively dealing with input validation and logic validation.

In both cases, performance-level validation is carried out with two different reference KPIs: system time and inter-departure time. The first measure is frequently used to monitor and control the waiting time in queues. The second measure is the inverse of the throughput rate. Event-level validation considers start and finish events of the processing of each station and it is used when a detailed validation of the digital model is needed. The system and the digital model used to conduct the experiments are DES models realised on Rockwell Arena Simulation Software. The online validation methodology described in section 3 has been implemented in python2 .

System Description

The experiments regard the five-station closed system illustrated in Figure 4. The system is characterized by four stations in a closed loop with an inner loop composed of an additional station. Stations are assumed to be perfectly reliable and the first-come-first-served dispatching policy is applied. a station is indicated with the number s, and p s represents the processing time on the s-th station measured in seconds per part. Each station has a stochastic processing time that is normally distributed with a mean µ and standard deviation σ. The distribution parameters and the utilisation of each station are reported in Table 3. The conveyors determine the length of the queue of pallets that wait in front of a station. The maximum number of pallets in the queue is fixed and results in buffers with a capacity equal to 4 pallets Fig. 4: Experiments -Manufacturing system used for the experimental campaign. for each station. The Blocking After Service (BAS) discipline is applied. The number of circulating pallets is equal to 15. To complete the production cycle, pallets must go through the inner loop two times. The maximum number of pallets circulating in the inner loop is equal to 5. This limit is controlled by allowing the exit of parts from station s = 1.

Design of Experiments

Two separate experimental designs have been executed to test the applicability of the proposed techniques for coping with both input validation and logic validation. In the experiments, the digital model is modified by introducing deviations with respect to the parameters of the physical model. The consequence of this modification is a bias in the estimated performance. The proposed methodology is applied to detect this artificially introduced bias. The following sections explain in detail how the experiments have been designed.

Input Validation.

Input validation is performed by applying the techniques proposed in section 3.4, where the sequence of data points coming from the real system is used to generate a sequence of 2019). The experimental design on input validation has been carried out in order to understand:

-How performance-level validation based on different KPIs reacts to deviations of the input distribution parameters of the digital model. Specifically, the mean and standard deviation of the processing time distribution of one specific station are modified on purpose.

-If the validation methodology is able to detect deviations also when the variations occur on a part of the system that does not strongly determine its performance (i.e., a non bottleneck station).

A categorical factor called station is used to identify which station is subjected to the parameter deviation. This factor is characterized by two levels, that represent if the corresponding station is the bottleneck of the system or not.

-How the ϵ and δ parameter values affect the system response. The indifference threshold ϵ is chosen as a percentage of the reference KPI used for validating the digital model. Experiments investigate the behaviour of the methods using for both ϵ and δ values 10%, 30% and 50% of the real mean inter-departure time.

The system response is measured as the value of the similarity indicator for which convergence 3 is reached. The time can be measured as the number of parts or events necessary to reach a stable indicator. Table 4 summarizes 3 Herewith convergence is defined as reached if the indicator does not vary more than the 1% of the value obtained with the whole set (i.e., 10,000 parts). -If logic validation can identify a change in the number of pallets. In the digital model, the number of pallets circulating in the system is modified on purpose (three levels).

-If logic validation can correctly identify a change in a production policy of the system. The station blocking policy is modified on purpose. Two levels are identified: blocking after service (BAS) and blocking before service (BBS) .

-If ϵ parameter value depends on the reference KPI used for performance-level validation. For the methods including ϵ and δ parameter, an additional factor is present with three levels.

Similarly to the experiments for input validation, the system response is measured as the indicator value for which convergence is reached.

Factors, levels, and values of each method are reported in Table 5.

Numerical Results

In this section the numerical results obtained from the experiments are described.

Input Validation Results

. The detailed results of Kruskal-Wallis test are reported in the Appendix, in Table 6 for the main effects and in Table 7 for the interactions. The main effects plot and interaction plots are illustrated in Figure 5. Independently from the sequence comparison method, a change of the processing time mean is significantly impacting the validity indicator, while the effect of the variation of the standard deviation is not significant. This means that it is possible to identify a variation of the former parameter, while the latter could remain unnoticed. The parameters δ and ϵ are significantly impacting the behaviour of the indicator, meaning their choice within a validation procedure is important.

Also, the indicator is significantly influenced by the location of the parameter variation. Hence, if the alteration occurs in a station that is not significantly impacting the performance of the system (i.e., not the bottleneck), the validation procedure could not identify this change. The interactions among the parameters for the input validation are not significant in the reported experiments. In the following, the most significant results for each developed indicator are reported. As the input parameters deviate with respect to the ones characterizing the system, the indicator drops. This behaviour highlights the capability of LCSS to detect deviations between the system and the model in terms of occurring events due to a wrong estimation of the input distribution parameters of the model. In these experiments, the logic of the digital model is not changed. Figure 6b instead shows the indicator evaluated for a model with not correct input distribution parameters for different values of δ, the indifference zone parameter (i.e. δ=10%, δ=30%, δ=50%). As expected, it is observed that the greater δ, the higher the indicator.

4.3.1.2 mLCSS-based Indicator. Figure 7a shows the mLCSS-based indicator using as reference KPI the inter-departure time. The indicator is reported as a function of the number of produced parts and the ϵ parameter is selected as 30% of the mean inter-departure time. The highest curve (blue curve, i.e. µ = 5, σ = 0.7) represents the similarity indicator associated to correct input distribution parameters of the model. In this case, the indicator is characterized by large values meaning the digital model is adequately representing the system. As the input parameters deviate from the ones characterising the system, the similarity indicator drops as represented by curve in the middle (orange curve, i.e. µ = 5.25, σ = 0.84)

and by the one on the bottom part of the graph (green curve, i.e. µ = 6.25, σ = 1.4). Similarly to the LCSS-based method, results prove the capability of the mLCSS-based indicator to detect deviations in terms of KPIs. Figure 7b reports the indicator associated to a model with a large deviation in the input distribution parameters for different values of ϵ parameter (i.e. ϵ=10%, ϵ=30%, ϵ=50%). The larger the parameter value, the more the indicator rises, meaning significant changes between the system and the digital model are more likely to be accepted. Results prove the capability of DTW to detect deviations between the system and the digital model in terms of KPIs due to a wrong estimation of the input distribution parameters. It is worth to notice that this is true also with 50 and 100 data points, which demonstrates that the indicator is sensitive also with relatively small data sets.

Logic Validation Results

. The results of Kruskal-Wallis test are reported in the Appendix, in Table 8 for the main effects and in Table 9 for the interactions. The main effects plot and interaction plots are illustrated in Figure 9. Independently from the sequence comparison method, the number of pallets is significantly impacting the validity indicator. The change of the blocking policy is correctly detected by the LCSS-and mLCSS-based indicators, while it is not discerned by the DTW-based approach. The parameters δ and ϵ are significantly altering the behaviour of the indicators, confirming that their choice is important for a validation procedure. The interactions among the parameters δ and ϵ are not significant for all methods, while the interactions among the number of pallets and the blocking policy are significant.

Hence, if the starting point is a situation in which the model is correct (for instance, in Figure 9b, the BAS policy) and another significant parameter is changed, the validation procedure is correctly identifying the

Remarks on Implementation

The experimental campaign presented in section 4 provides useful insights on the practical application of the methodology from Figure 3. The following list aims at exploiting the lessons learned toward the implementation of online validation in a manufacturing system.

Logic Validation Setting

All three tested techniques can be applied for logic validation, if the interest is to detect changes that strictly influence the performance of the system (e.g., number of circulating pallets). For less sensitive changes such as the change of a production policy, based on the results of the experiments it is encouraged the use of LCSS-or mLCSS-based indicators. For LCSS and mLCSS, the parameters ϵ and δ should be chosen depending on the indifference zone. Indeed, the experiments show that the parameters values influence the indicator: a higher parameter value will tend to have a higher validity indicator, hence a larger indifference zone results in a higher probability that the model will be declared as valid. Also, it is worth to notice that logic validation is not characterized by the intrinsic error of the random number generation. Consequently, for correct logical assumptions and a digital-digital comparison the indicator

Input Validation Setting

Input validation should be set together with the quasi-trace driven simulation [START_REF] Lugaresi | Real-time validation of digital models for manufacturing systems: A novel signal-processing-based approach[END_REF] on the input data from the real system. All three tested methods proved to correctly identify the changes on parameters that directly influence the performance of the system (e.g., change of mean processing time on bottleneck station). LCSS-and mLCSS-based indicators should be set with the proper values of ϵ and δ parameter values. Different KPIs react differently to the same variation of the model. Hence, the proper ϵ value for mLCSS-based indicators depends specifically on the KPI under investigation. An ad-hoc calibration is needed for each validation campaign. For the δ parameter, the most appropriate value increases with the increase of the system complexity.

Threshold Values Definition

In order to identify the validity of the digital model online, validity thresholds should be defined for the specific application. Given the different input data, logic and input validation should be assigned different respective thresholds. The definition of thresholds is a non trivial task, and it remains strictly application dependent. As a general guideline, initial experiments should be performed using historical data and valid models, in order to identify the upper bounds of the similarity indicators. Then, a maximum allowed deviation can be defined (e.g., a 5 % drop of the indicator with respect to the upper bound).

However, the threshold should always be revisited once new data are available and the noise of the real system can be apprehended, while being coherent with the allowed error probability.

Online Application Setting

Once the logic and input validation techniques have been set, the procedure of Figure 3 can be implemented in a real system. For such implementation, two main decisions have to be made concerning the quantity of data to be collected at each step, and the query update frequency. Despite the choice of these parameters is not the focus of this work, their influence and behaviour may be qualitatively addressed in the following:

-Data set size. In general, it is reasonable to assume that the faster the production pace, the smaller the parameters can be to obtain an adequate size data set. Besides, the size of the datasets will influence the computation time of the queries, which should be safely smaller than the update time interval.

-Update frequency. In general, the update frequency should be chosen in relation with the time to signal of interest, namely the time needed to identify a disruption that is desired to guarantee. The production pace will also directly influence this choice, as systems with a low cadency will tend to have a lower update frequency than faster ones. In [START_REF] Lugaresi | Online validation of simulation-based digital twins exploiting time series analysis[END_REF], an initial attempt has been made by applying DTW-based validation in a controlled laboratory environment, using a query time of 10 minutes and an update frequency of 1 minute.

The choice of the parameters has been based on the production pace of the system under analysis and on how fast deviations need to be identified. Hence, they should be identified case by case based on the specific application. Nevertheless, as the data set used to compute the indicator contains data acquired during the correct functioning of the system, some time is always required to detect the occurrence of a disruption. Therefore, the amount of data and the frequency with which the indicator is computed must be carefully chosen. Future research should be dedicated on designing specific experiments to further define the implementation of the online validation methodology.

Conclusions

This work proposes an innovative methodology to deal with online validation of simulation-based digital twins of manufacturing systems. Thanks to the proposed techniques, it is possible to guarantee that a digital model is always updated both in terms of inputs and logic, enabling short-term decision making.

The main advantages of this research are the possibility to achieve reliable results even with limited data sets and the capability to monitor the whole trend of information instead of the sole average behaviour.

The validation procedure allows to detect deviations rapidly and avoid possible erroneous alerts due to wrong data acquisition.

The proposed methodology is subject to several limitations that highlight the need of further research.

In a real-digital comparison, the logic validation indicator associated to a correct logical layout does not reach a value equal to 1, due to the fact that it is not possible to achieve a perfect modelling of the system.

Hence, a prior analysis on the system is still required to properly identify the acceptance thresholds. The definition of threshold and parameter values should be based on the system topology and characteristics, as well as its real-time state. Future research should be dedicated to introduce a learning procedure to automatically identify the acceptance thresholds based on the system dynamics and available historical data. To this end, the integration with relevant works in the same direction [START_REF] Cruz | A two-step machine learning approach for dynamic model selection: A case study on a micro milling process[END_REF][START_REF] Castano | Data-driven insights on time-to-failure of electromechanical manufacturing devices: A procedure and case study[END_REF]) could be explored. The accuracy level requested by the user should also be taken into account. Last but not least, this methodology is not capable to identify the specific cause for the anomaly behaviour of the model. A significant development would be to study methods and tools to automatically detect the reason for which the model is not valid.

In future work, it is necessary to study the validity of the proposed methodology within real manufacturing environments. A step in this direction can be represented by tests using model factories. For instance, in [START_REF] Lugaresi | Online validation of simulation-based digital twins exploiting time series analysis[END_REF], a lab-scale model is used to implement a validation technique within a digital twin architecture. Experiments in a physical laboratory can bring the proposed approaches closer to real environments. Further, in future research the application of the proposed methodology to other types of manufacturing system data should be explored. Indeed, most systems record physical quantities along the sequential processing of parts, such as temperature or pressure. It is therefore of interest to validate digital models of specific physical processes within the production cycle, such as a heating curve for a treatment oven. Theoretically it is possible to extend the proposed approach toward these data types, provided the setting of proper sampling and pre-processing techniques. Last but not least, the present work has been driven by the need of validating digital models with few data. However, some manufacturing systems do produce sufficiently large datasets even in the short term, typically when they present a high production cadency. Hence, the proposed methodology could be compared with a set of traditional techniques to identify specific situations where both could be applied.

Fig. 3 :

 3 Fig. 3: Proposed online validation procedure (TDS: Trace Driven Simulation; qTDS: quasi Trace Driven Simulation).

δ

  a,b) LCSS-based similarity measure comparing the two data sequences a [0:i] and b [0:j] . M [i,j] (a,b) mLCSS-based similarity measure comparing the two data sequences a [0:i] and b [0:j] . D [i,j] (a,b) DTW-based similarity measure comparing the two data sequences a [0:i] and b [0:j] . Φm(a,b) Model validity indicator evaluated according to the method m comparing the two data sequences a and b. Φ ∈ [0, 1]. Parameters: Indifference threshold, meaning a deviation smaller than δ is not significant enough to say the digital model is not valid. ϵ Time closeness threshold, two elements are declared similar if their time difference is smaller than ϵ. w Number of warning signals that do not trigger a model update action. tquery Length of the last interval of time used to collect data. t update Time interval determining the frequency with which the similarity indicator is computed. θm Threshold used to compare the indicator based on the method m. τ1 Time to detect a change in the system.

  factors, levels, and values of each method. Each experimental point has been replicated five times and the replications are independent. As a consequence, for the test of both LCSS-and mLCSS-based indicators, the experiments are characterized by 54 experimental points, and a total of 270 runs have been done. For the test of the DTW-based indicator, 18 experimental points are determined by the design, and a total of 90 runs have been done.

  Each configuration of factors has been replicated five times and the replications are independent. As a consequence, experiments to test LCSS-and mLCSS-based indicators are characterized by 18 experimental points, and a total of 90 runs have been done. The experiments to test DTW-based indicator are characterized by 6 configurations, hence 30 experiments have been done.

4. 3

 3 .1.1 LCSS-based Indicator. Figure 6a shows the LCSS-based indicator for different configurations of input distribution parameters as a function of the number of events considered. Results are associated to δ parameter equal to 50% of the mean inter-departure time. When the model is characterised by correct input distribution parameters, the similarity indicator shows high values (blue curve, i.e. µ = 5, σ = 0.7).

Fig. 5 :

 5 Fig. 5: Experiments -Input validation (KPI: inter-departure time): (a) main effects plot and (b) interaction plot for LCSS-based indicator, (c) main effects plot and (d) interaction plot for mLCSS-based indicator, (e) main effects plot and (f) interaction plot for DTW-based indicator.

Fig. 7 :

 7 Fig. 7: Experiments -Input validation at performance-level: 95% confidence intervals of (a) mLCSS-based indicator with ϵ = 0.3 and inter-departure time as reference KPI (b) mLCSS-based indicator with wrong input distribution parameters for ϵ = 0.1, ϵ = 0.3, and ϵ = 0.5.

Fig. 9 :

 9 Fig. 9: Experiments -Logic validation (KPI: inter-departure time): (a) main effects plot and (b) interaction plot for LCSS-based indicator, (c) main effects plot and (d) interaction plot for mLCSS-based indicator, (e) main effects plot and (f) interaction plot for DTW-based indicator.

Fig. 10 :

 10 Fig. 10: Experiments -Logic validation at event-level: 95% confidence intervals of LCSS-based indicator with δ = 0.3.

Fig. 11 :

 11 Fig. 11: Experiments -Logic validation at performance-level: 95% confidence intervals of mLCSS-based indicator (KPI: inter-departure time).

Fig. 12 :

 12 Fig. 12: Experiments -Logic validation at performance-level: 95% confidence intervals of the DTW-based indicator (KPI: system time).

Table 1 :

 1 Summary of main validation techniques in the literature and their limitations.

	References			Methods	Limitations
	Sargent (2013); Balci (1994);	Graphical Comparison	Subjective decisions; difficulties in evalu-
	Kleijnen (1998)				ating results: information flows cannot be
						fully automatized
					Confidence Intervals	Based on strong assumptions; need for the
						availability of lots of data
					Hypothesis Tests	Typically based on the average behaviour
						of the system
	Khan et al. (2018)		Model specification checking	Only for automata and petri net models,
						focus on model verification
	Schruben and Cogliano (1981);	Signal-processing techniques	Need to switch to the frequency domain
	Lugaresi et al. (2019)
	Camargo et al. (2020)		Conformance checking	No approaches applied during the opera-
						tional phases
	Leroy	et	al.	(2018);	Trace Analysis	Appropriate for deterministic applications
	Muñoz	et	al.	(2022);
	Gong et al. (2022)

(e.g., trajectory planning); currently not applied in manufacturing systems

  ) 3.4.2 Modified Longest Common Sub-sequence (mLCSS). A variation to the LCSS technique is applied to deal with sequences of real numbers. Differently from LCSS, a matching threshold distance ϵ is introduced to compare real numbers instead of strings. In this case, the information related to the sequence of time values is not considered in the comparison. The modification allows to declare similar two elements not equal but characterized by a Euclidean distance smaller than the ϵ parameter. The corresponding mLCSS-based similarity measure is a matrix, where each element M [i,j] (a,b) corresponds

to the comparison of the sequences a [0:i] and b [0:j] , as reported in Equation (3).

Table 3 :

 3 Experiments -Processing time distributions and utilisation of each station.

	Station s	Processing Time ps [s]	Utilization
	1	Norm(12, 0.07)	0.797
	2	Norm(5, 0.07)	0.996
	3	Norm(7, 0.07)	0.929
	4	Norm(12, 0.07)	0.797
	5	Norm(12, 0.07)	0.796

Table 4 :

 4 Experiments -Input Validation: factors and respective levels of each method.

	Indicators

Table 5 :

 5 Experiments -Logic Validation: factors and respective levels of each method. Logic validation is performed by applying the techniques listed in section 3.4, where the data points coming from the real system are directly used in the digital model respecting the Trace Driven Simulation principle. The logic validation experimental campaign has been done in order to understand:

	Indicators

Table 6 :

 6 Experiments -Input Validation: results of the Kruskal-Wallis test for main effects.

	Method	KPI Factor	DF	H-Value P-Value N Samples
	LCSS	IDT	Mean	2	29.08	0	90
			Std	2	2.82	0.244	90
			Parameter δ 2	142.6	0	90
			Station s	1	44.59	0	135
	mLCSS	IDT	Mean	2	55.96	0	90
			Std	2	4.86	0.88	90
			Parameter ϵ 2	112.53	0	90
			Station s	1	55.17	0	135
		ST	Mean	2	122.9	0	90
			Std	2	1.33	0.513	90
			Parameter ϵ 2	41.01	0	90
			Station s	1	72.96	0	135
	DTW	IDT	Mean	2	65.52	0	30
			Std	2	1.13	0.568	30
			Station s	1	9.6	0.002	45
		ST	Mean	2	43.27	0	30
			Std	2	0.3	0.863	30
			Station s	1	24.55	0	45

Table 7 :

 7 Experiments -Input Validation: results of the Kruskal-Wallis test for interactions.

	Method	KPI Factor	DF	H-Value P-Value N Samples
	LCSS	IDT	Mean*Std	1	1.28	0.258	40
			Mean*Parameter δ	1	0.21	0.648	40
			Mean*Station s	1	0.89	0.346	40
			Std*Parameter δ	1	0.04	0.832	40
			Std*Station s	1	0.79	0.373	40
			Parameter δ*Station s 1	0.07	0.799	40
	mLCSS	IDT	Mean*Std	1	2.61	0.106	40
			Mean*Parameter ϵ	1	0.49	0.485	40
			Mean*Station s	1	1.49	0.222	40
			Std*Parameter ϵ	1	0.03	0.866	40
			Std*Station s	1	0.59	0.441	40
			Parameter ϵ*Station s 1	1.17	0.279	40
	DTW	IDT	Mean*Std	1	5.16	0.023	20
			Mean*Station s	1	0.15	0.695	20
			Std*Station s	1	0.01	0.903	20

Table 8 :

 8 Experiments -Logic Validation: results of Kruskal-Wallis test for the main effects.

	Method	KPI	Factor	DF	H-Value P-Value N Samples
	LCSS	IDT	N pallets	2	33.3	0	30
			Policy	1	13.4	0	45
			Parameter δ 2	19.91	0	30
	mLCSS	IDT	N pallets	2	11.83	0.003	30
			Policy	1	11.69	0.001	45
			Parameter ϵ 2	38.36	0	30
		ST	N pallets	2	80.42	0	30
			Policy	1	0.22	0.637	45
			Parameter ϵ 2	1.89	0.388	30
	DTW	IDT	N pallets	2	18.23	0	10
			Policy	1	0.08	0.771	15
		ST	N pallets	2	25.95	0	10
			Policy	1	0.04	0.835	15

Table 9 :

 9 Experiments -Logic Validation: results of Kruskal-Wallis test for the interactions.

	Method	KPI Factor	DF	H-Value P-Value N Samples
	LCSS	IDT	N Pallet*policy	1	9.34	0.002	20
			N pallet*parameter δ	1	0.11	0.735	20
			policy*parameter δ	1	0.48	0.49	20
	mLCSS	IDT	N Pallet*policy	1	7.61	0.006	20
			N pallet*parameter ϵ	1	5.1	0.024	20
			policy*parameter ϵ	1	0.15	0.695	20
	DTW	IDT	N Pallet*policy	1	0.89	0.345	10

A trace is defined as a sequence of activities performed by an element in the system (van der Aalst,

2016).

Python codes repository: https://github.com/giovannilugaresi/digital_twin_validator

differential computation and visualization, and extraction of graph-based views to analyze cycles. The authors use the Levenshtein distance to measure the traces similarity. Gong et al. (2022) introduced a method for human-computer interaction that allows remote manipulation of life-size humanoid robots. The authors included a novel set of metrics for evaluating the similarity between the motion of a robotic arm and the one of a human in real-time. Muñoz et al. (2022) presented a method for aligning the traces of digital twins in the form of data sequences representing their progressive states. For this scope, the authors propose a modified version of the Needleman-Wunsch algorithm. Despite being promising approaches, these techniques do not allow for the distinguished validation of logic and inputs of the digital

Appendix

This section reports the detailed results of the experimental campaign described in section 4 by using the Kruskal-Wallis test.


Elmehdi Azzouzi
email: elmehdi.azzouzi@edf.fr

Daniel Bouskela
email: daniel.bouskela@edf.fr

Faïda Mhenni
email: faida.mhenni@supmeca.fr

Jean-Yves Choley
email: jean-yves.choley@supmeca.fr

A Multi-Stakeholders Co-Design Methodology Applied to an Insulin Pump Mechatronic System

Keywords: Cyber-Physical-Systems, mechatronics, stakeholders, system lifecycle, co-design, methodology, requirements

Nowadays Cyber-Physical-Systems (CPS) are becoming more and more complex, and involve a large number of stakeholders. They all have a different but significant role at a certain phase of the system lifecycle, and they all have an influence on the CPS specifications. This paper gives an overview of a co-design methodology that aims at assisting multiple stakeholders, having concerns about a common system, to agree on sets of objectives and mutual obligations. The objective here is to identify the requirements that fulfill the intrinsic goals of each stakeholder. The agreements among stakeholders are introduced into formal contracts to enable their formal verification. The methodology was successfully applied to a large energy system of systems. This paper discusses its application to a smaller system which is the insulin pump.

I. INTRODUCTION

Nowadays Cyber-Physical-Systems (CPS) are becoming more and more complex, and involve a large number of stakeholders, all of whom have a significant role at a certain phase of the system lifecycle. Thus, they all got the capacity to influence the requirements of a CPS. Given the nature of their role, their culture and their field of expertise, stakeholders have different views regarding the same system and have a hard time agreeing on common objectives. At early conceptual phases of CPS, each stakeholder that is involved or concerned by a system has intrinsic interests and goals that can widely differ regarding his viewpoint. In a context where everyone seeks to maximize its own profit, and where the benefits of a stakeholder are inversely correlated with others interests, we generally find ourselves facing contradictory expectations and different perceptions regarding the same system. Stakeholders have no means to assist them to overcome this challenge.

In this context, we have introduced a methodology that allows stakeholders to be coordinated for the sake of identifying a common ground to meet everyone's expectations.

This methodology is based on multiple modeling layers that allow having at the same time a view that is centered on stakeholders in parallel with a "System" view on the CPS. The idea is to have a clear definition of stakeholders' intrinsic interests, their interactions regarding others and at the same time to have means to work on a system level and make modeling and simulation. It would allow stakeholders to make well-informed negotiations.

This methodology was conducted as part of an industrial project in the energy sector. It was applied to a large complex multi-energy system including a large number of stakeholders [START_REF] Bouquerel | Requirements modelling to help decision makers to efficiently renovate energy systems of urban districts[END_REF]. Encouraging results were achieved within the application of the methodology.

We have later asked ourselves whether this methodology was appropriate to smaller mechatronic systems. Indeed, one can underestimate the complexity of the development of small mechatronic CPS such as insulin pumps. On the contrary, such systems also involve a large number of disciplines and stakeholders having different cultures and coming from independent entities. Thus, they have different goals and constraints. Agreeing on common objectives is also a considerable challenge The stakeholders need to mutually agree on the appropriate system specifications to have better chances to come up with a successful design that answers the real need of stakeholders.

In this paper, we take the example of an insulin pump or so-called "Artificial Pancreas".

The paper is organized as follows. Section II provides an overview of the related works. A description of the methodology is introduced in section III. It is then applied to the insulin case study in section IV. Finally, a conclusion is given in section V.

II. STATE OF THE ART

In literature, the idea of correctly capturing stakeholders' needs is commonly recognized as a critical success factor of complex systems. The stakeholder theory introduced by Freeman in [START_REF] Freeman | Strategic Management: A Stakeholder Approach[END_REF] emphasizes the necessity for a company to take into consideration ethical and social aspects in decision making. For that end, the company needs to analyze its impact on the external stakeholders and vise versa by classifying them into different categories and acting in consequence.

Systems Engineering (SE) repositories such as [START_REF] Walden | [END_REF] propose among the processes the "Stakeholder needs and requirements definition" as one of the first activities that need to be meticulously analyzed as it is considered as the basis of all the processes that follow.

State of the art methodologies such as [START_REF] Hoffmann | Systems Engineering Best Practices with the Rational Solution for Systems and Software Engineering[END_REF], [START_REF] Lykins | 4.4. 4 Adapting UML for an Object Oriented Systems Engineering Method (OOSEM)[END_REF], [START_REF] Ahrens | [END_REF] only considers the stakeholders of a system as a source of requirements which are identified in early design phases. Thereafter, the methodologies provide processes and activities that aim at refining the stakeholders' needs in order to design the system that best answers the stakeholders' intents. The limitation here is that the establishment of stakeholders' requirements are generally set up from a unique designer point of view in the initial design phases, and are rarely revisited. However, in an industrial context, stakeholders have requirements that evolve all along with system development. The requirements are constantly refined and reevaluated by stakeholders at different development steps, especially for large Systems of System (SoS) where negotiations between stakeholders can take place. A correct definition of stakeholders' needs is thus a challenge by itself and requires in-depth analysis.

III. METHODOLOGY OVERVIEW

A. Preamble

The methodology introduced aims to define a framework that assists CPS stakeholders to negotiate and agree on mutual contracts that will answer the needs of all parties.

Our methodology is based on multiple layers with a stakeholder-centric description, as shown in Fig. 1. They consist on the one side of modeling the elements that are intrinsic to each stakeholder, thus representing its goals, role and behavior. On the other side, they consist of modeling the relationships and interactions between the stakeholders. These elements aim essentially at precisely specifying the perimeter and obligations of each stakeholder. They afford a collaborative framework where stakeholders can make "well informed" negotiations.

Five modeling layers are essentially defined in the methodology. For the purpose of developing executable verification models, we use the concepts of "Observer" and "Binding" added for connecting the heterogeneous models from different layers, notably the formal requirements, architectural models, and behavioral models.

The following section introduces the five modeling layers.

B. Intentional level

In this methodology, and contrary to the classical approaches that take as a starting point the missions of a system and its technical constraints, we start here from the intentional level of stakeholders. In this level, we will capture the high-level objectives of the stakeholders.

In the most abstract level of stakeholders' goals, which refers to their main reason for being involved in a CPS, the requirements or interests come down to a small number of needs and can be described with only a few words. They are generally limited to financial aspects for companies, to the well-being of consumers, and social and political aspects for governments. Afterward, they are declined through multiple decomposition layers into more elementary concepts and means-ends in order to reach high-level goals.

Starting from such an abstract level allows identifying the real needs of stakeholders instead of jumping into the solution world. It also ensures the traceability for the large number of specifications that emerge all along the system development.

The definition of the intentional level of our methodology was widely inspired by the work of Eric et al. in [START_REF] Eric | Social modeling for requirements engineering[END_REF]. The latter puts forward a characterization using natural language blocks as well as a formal one. Our methodology only considers the natural language description blocks of the intentional level.

We have considered a different approach for their formal characterization.

The intentional level is based on two main models:

• The stakeholders' dependency model captures the dependencies between stakeholders where a stakeholder can be a "depender" when depending on another party, or a "dependee" when a party is depending upon him. The object of the dependency is called the dependum. Fig. 2 shows the four types of dependencies between a stakeholder X and a stakeholder Y. The direction of the arrays goes from the depender to the dependee. This means here that the stakeholder X depends on the stakeholder Y regarding the goal G, the task T, the resource R, and the soft-goal SG.

• The stakeholder rationale model can be considered as a justification model that describes the reasons behind the dependencies of a stakeholder regarding others.

The high-level goals of stakeholders and their roles and captured in this model and are decomposed through multiple steps using four elements Goal, Soft-Goal, Task, and Resource.

C. Properties formalizing level

In this level, the stakeholders' needs as well as the requirements related to the dependencies are formalized. Requirements formalizing is done using the requirement language named FORM-L (FOrmal Requirements Modeling Language) [START_REF] Nguyen | Formal Requirements and Constraints Modelling in FORM-L for the Engineering of Complex Socio-Technical Systems[END_REF]. The objective of this level is defining from the one side the Key Performance Indicators (KPIs) under which stakeholders will make their future decisions. These KPIs will be part of the stakeholder description and are not intended to be shared with others. From the other side, the objective is to formally define the obligations of each stakeholder regarding others. These obligations will be integrated into the formal contracts that bind them and will be subjected to modifications all along the negotiations. The formal contracts are considered here to be the interfaces between stakeholders. They define the degrees of freedom of each of its contractors. This is a strong property that will allow each stakeholder to make an abstraction of the real behavior of his environment and work separately. The contract guarantees an envelope covering all possible behaviors of other stakeholders.

D. Stakeholders interaction model

In this methodology, we have chosen to distinguish between the interactions among stakeholders and the interactions among the systems under their responsibility, even though they are strongly interdependent. Indeed, it seemed crucial in this methodology to have a dedicated model for the interactions between the stakeholders because some aspects such as financial transactions or administrative processes are more related to stakeholders than systems. Such aspects have a considerable influence on the choices that are made at the system level. It is thus important to identify the relationships between the two levels for a good decision making.

Fig. 2. Types of dependencies

This modeling level consists of a set of nodes representing the stakeholders and a set of arrays that represent what is the nature of the interaction between stakeholders, and in which direction. The interactions here must be characterized using four elements: the resource being exchanged, the means of exchange, the protocol of exchange and the rules of the exchange. Detailed modeling of these aspects can be developed as part of the behavioral modeling layer. This model stem from the intentional level where we identify all the dependencies between stakeholders. Indeed, the resource dependencies between stakeholders are translated and characterized more deeply in this level.

E. Architecture description

The architecture description layer was mainly introduced in order to have a "system" view of the developed CPS by bringing to the fore the dynamic interactions between the subsystems under the responsibility of different stakeholders. Similarly to the level above, the interactions are characterized using the four elements the resource being exchanged, the means of exchange, the protocol of exchange and the rules of the exchange. This view does not preclude the inclusion of the stakeholder aspect. In fact, we have introduced the concept of "perimeter" in the architecture description. The perimeter is used to define the scope of responsibility of each stakeholder. This perimeter is subjected to modifications all along the development phases of the system. Negotiations between stakeholders will have a considerable influence on the definition of the perimeter. The architectural description included in the perimeter of each stakeholder is later introduced into this stakeholder description model.

F. Behavioral modeling level

This level only deals with one dimension, the behavioral description of the system under the perimeter of each stakeholder. In this layer, stakeholders have the freedom to model their behaviors using different modeling languages and tools. Throughout the different steps of the engineering lifecycle, different kinds of behavioral models will be used by engineers. In preliminary studies, stochastic models are more likely to be used as only a few is known about the system. In design phases, static or dynamic behavioral models based on Differential-Algebric Equations (DAE) can be used for system sizing. In detailed design phases, 3D models, Computational Fluid Dynamics (CFD) and similar paradigms are used to have a system representation that is close to its real performances once produced.

We do not consider in this methodology that the behavioral models of stakeholders are shared with the others. We are aware that they represent a precious knowledge for parties. However, according to the context, stakeholders can share their models or make co-simulation for a better assessment of the overall system.

G. Verification Model

In order to find mutual agreements between stakeholders and set up formal contracts, parties need to check on the one side that the contract requirements satisfy their high-level need, on the other side, that they have the capacity to commit to their obligations. To that end, we have introduced a verification level into the methodology. This layer allows stakeholders to make all kinds of testing using modeling and simulation.

The architecture of the verification model used in this methodology was first introduced in [START_REF] Bouskela | Toward a rigorous approach for verifying cyber-physical systems against requirements[END_REF].

Two main concepts are used for setting a verification model: Observers and Bindings. The observers and bindings allow connecting the behavioral models, architectural models, and the formal requirement models for the purpose of making executable verification models. In fact, these models are from different natures and are generally developed by engineers from different disciplines with different backgrounds. The observers, as their name suggests, are used to observe and capture data from the dynamic behavior simulated using the disciplinary models. They also capture engineering data from the architectural description model. The observers have a second role which is transforming the data in order to be consistent with the inputs of the requirement model. The bindings allow connecting the heterogeneous models mentioned above including the observation models.

IV. CASE STUDY

In order to evaluate the appropriateness of our methodology regarding mechatronic systems, we have set up a reflection on the insulin pump example. The idea is to propose a framework for the concerned stakeholders to converge towards mutual agreements answering the needs of everyone.

A. Insulin pump: a CPS and mechatronic case study

An insulin pump, or "artificial pancreas", is a mechatronic device that contributes to solving diabetic patients' constraints [START_REF] Ahrens | [END_REF], [START_REF] Iacovacci | Design and Development of a Mechatronic System for Noninvasive Refilling of Implantable Artificial Pancreas[END_REF]. Indeed, in order to deal with diabetes, an artificial corrective process is necessary to improve a biological defective process, quantified by the glucose level, with an automated dosage of injected insulin. This closed-loop insulin delivery system consists of three main blocks, which are an "Insulin Dispensing Device" (pump, catheter, and cannula), a "Continuous Glucose Monitoring" (a sensor placed on the skin), and a control algorithm implemented in an embedded system. The system energy autonomy and its ergonomy are important. Moreover, it has to be connected to a smartphone, a computer, and a medical network. Thus, it is a CPS with important human, energetic, safety, and security concerns.

B. Methodology application

In the early conceptual phase of the insulin pump development, the high-level interests of each stakeholder are defined.

For this case study, we consider the following stakeholders:

• Patients: considered to be diabetic.

• Doctors: are in charge of the medical monitoring of the patients.

• Health insurers: are in charge to repay a part of the expenses to patients.

• Government: is in charge of making the medical laws and standards, and give certifications. • Insulin supplier: is in charge of the supply of insulin.

• Vendors: are in charge of selling insulin pumps to patients.

• Designer -Producer: is in charge of the development and the production of the insulin pump.

Although this list of stakeholders is not exhaustive, it already contains a considerable number of parties that need to agree on mutual requirements. In addition to these stakeholders, we consider the environmental constraints that will be taken into account in a specific contract. Once the stakeholders converge towards mutual contracts, we assume that the objectives of the methodology were attained.

1) Intentional level:

The first step of the methodology is to capture the high-level intentions of each stakeholder as well as stakeholder dependencies. In the following, we will describe the intentions of the patient and the designer.

Fig. 3 shows the strategic rational model of the patient. His main high-level goal is his well-being. This goal is achieved by the task "Manage diabetes" which in turn is broken down into three sub-tasks: "Get medical monitoring" by the doctor, "Undergo insulin treatment" and "Get health insurance" to help to support the induced medical expenses. "Manage diabetes" is associated with a soft-goal "Affordable". There are two alternatives for the patient to have insulin treatment: either to get the classical insulin injections or to purchase an insulin pump that will substitute the manual injections.

The insulin treatment induces two soft-goals: "Comfort", and "Trust". The final choice of the patient among the different alternative solutions will be guided by the abovementioned soft-goals.

The second step of the intentional level is the identification of the different dependencies between stakeholders. These dependencies will allow stakeholders to achieve their highlevel goals by relying on other parties to perform the tasks they cannot achieve on their own. Fig. 4 represents the different dependencies among the stakeholders. The dependencies related to the fulfillment of the patient's intentions are highlighted in red. For instance, the patient depends on the doctor to "Get medical monitoring" which is a sub-task derived from his high-level goal of "Well being". This dependency is represented by a goal dependency having an elliptic shape labeled "Medical monitoring" in Fig. 4. He also depends on the Designer/Producer to get a quality product that contributes to the fulfillment of the "Trust" and "Comfort" soft-goals.

2) Stakeholder interaction model: In this phase, the interactions between stakeholders are captured. Fig. 5 shows three types of exchanged elements between stakeholders, notably: financial transactions, data transfer, goods transfer. When comparing with Fig. 4, we can see that the resource dependencies between stakeholders are translated here into interactions.

3) Formalizing properties level: In this level, the properties defined in the intentional level are formalized by:

• deriving Key Performance Indicators (KPIs) from the high-level needs of stakeholder,

• establishing contracts containing formal requirements related to the dependencies between stakeholders. These formal requirements can be automatically verified later on.

In the intentional description of the patient, three main soft-goals were identified: "Affordable", "Comfort" and "Trust". Obviously, all patients do not have the same view and the same priorities. However, all kinds of patients need to be represented. The soft-goals above-mentioned form a decision triangle where the patients can set the cursor in different positions according to their view.

These aspects can be formalized with:

• Affordable = The extra amount to be paid for the pump must not exceed X percent of the medical fees of the standard treatment.

• Comfort = (i) The size of the pump must not exceed the dimensions A*B*C mm 3 (ii) the weight of the pump must be less than the weight M g, (iii) the frequency of battery change must be less than Nb times a year, (iv) the insulin refilling must be less than Ni times a month. From the designer side, the KPIs will be derived from the two soft-goals "Be profitable" and "Be competitive" and the goal "Gain market share".

• Be profitable = The annual revenue from the activity must be higher than P percent of the sales revenue

• Be competitive = The price of the insulin pump must be lower than at least C percent compared to its competitors when the product will be placed on the market.

• Gain market share = The market share of the insulin pump must increase with Z percent per year for the two first years.

We emphasize that the formal properties can be defined in different manners, and the choices of the KPIs can widely vary depending on the intentions of the stakeholder. For instance, the formal characterization of "Being competitive" could have been defined using a market share indicator.

Similarly to the patient and the designer, the formalizing was done for the other stakeholders.

In order to establish contracts between stakeholders, the dependencies between them were formalized. For instance:

• The goal dependency "Reasonable purchase price" of the vendors regarding the designer/Producer will induce the requirement: The purchase price of the insulin pump must be lower than the threshold Pmax

• The resource dependencies "Insulin pumps" and "Insulin" of the patients regarding the vendors will induce requirements regarding the availability of these resources.

4) Architecture description:

The architecture description level was developed in order to have a "System" view on the CPS by identifying the interactions between sub-systems and defining the perimeters of the concerned stakeholders.

In the current case study, the dynamic interactions between systems are mainly (i) the interactions between the components of the insulin pump, (ii) the interactions between the insulin pump and the body of the patient (iii) and the data exchanged between the insulin pump and the doctor monitoring device.

5) Behavioral modeling:

In this level, stakeholders develop the behavioral models of the system under their perimeter for subsequent verification.

In order to attain his goals, the designer needs to develop behavioral models that would allow him to assess whether if the development of the insulin pump satisfies his KPIs such as profitability. To that end, the designer will develop a preliminary financial model that considers the pre-sizing of the insulin pump and the conceptual solutions to be considered.

In the design phases, the designer will develop dynamic physical models coupled with I&C representing the behavior of the different components of the insulin pump. These models will allow him to make a detailed design, and they used latter in the verification model.

Similarly, other stakeholders develop behavioral models representing their systems in order to help them make the decisions that answer the most their needs.

6) Verification model: Verification models are built using elements from the different levels introduced earlier. From the designer's view, the first verification model to develop would aim to check if the preliminary architecture of his insulin pump is coherent with his KPIs and the contracts regarding other stakeholders. For that purpose, the designer gathers the following elements (i) the financial computation model (ii) the architecture description of the insulin pump that contains the first estimations of the elements costs, for different possible technologies, (iii) the KPIs model with the requirements that are related to the profitability, and (iv) the formal contracts with the other stakeholders including their requirements.

Other verification models are established throughout the other design phases including behavioral models that are representative of the reality and requirements that are more specific on what the system shall do and how it should perform.

V. CONCLUSION

In this paper, we have presented a co-design methodology that assists multiple stakeholders of a CPS to converge towards common specifications and objectives. The idea is to elaborate together formal contracts specifying the obligations of each of its signatories.

The particularity of this methodology is that it combines two complementary views: a view that is centered on stakeholders, and a view that allows having a "System" view on the CPS. Throughout its different modeling layers, the methodology offers means to have design traceability at every development phase with justification element all along.

The application of the methodology to the mechatronic system "insulin pump" was worthwhile as it allowed us to test whether the methodology can be applied to other domains than the energy sector for which it was initially built. This application have given us promising results to test the methodology on other sectors having the common challenge of stakeholders' coordination.

However, one of the methodology layers turned out to be not very accurate for mechatronic systems, notably the architecture description level. The goal of this level was above all to identify the perimeters of each stakeholder and the interactions between them in operational phases. In the insulin pump case study, only a few interactions between systems were identified. The systems being under the responsibility of different stakeholders. However, this does not question the adaptability of the approach for mechatronic systems.

Fig. 1 .

 1 Fig. 1. Methodology modeling layers

Fig. 3 .

 3 Fig. 3. Patient strategic rationale model

Fig. 4 .

 4 Fig. 4. Stakeholders dependency model

Fig. 5 .

 5 Fig. 5. Stakeholders interaction model • Trust= The insulin pump must have an accuracy of Pr mg and must not derive from it more than once a year. When it does, an alarm should be sent to the patient and the doctor.