Bruno Cognet
email: bruno.cognet.1@ens.etsmtl.ca

Jean-Philippe Pernot
email: jean-philippe.pernot@ensam.eu

Louis Rivest
email: louis.rivest@etsmtl.ca

Chritophe Danjou
email: christophe.danjou@polymtl.ca

Digital maturity models: comparing manual and semi-automatic similarity assessment frameworks

Keywords: Industry 4.0, smart manufacturing, digitalisation, maturity models, comparison framework, semi-automatic comparison, similarity assessment

The fourth industrial revolution is forcing companies to define their digital strategy, making it imperative that they assess their digital maturity as a basis for improvements. As a result, a variety of maturity models have emerged. However, it can be difficult to identify which one is most appropriate. This paper introduces a new methodology to compare a manual and a semiautomatic framework for assessing the similarity of digital maturity models. It allows identifying the most adequate framework for comparing maturity models. Both frameworks have been designed to identify correspondences between KPIs. The analysis of the matches and the obtained results are then used to tune the semi-automatic framework. The proposed comparison methodology has been validated using two digital maturity models and shows that the semi-automatic framework provides good results in a very efficient manner. Several insights have been derived and will help to develop a new maturity model.

Introduction

Manufacturing companies feel increasing pressure to adopt the Industry 4.0 paradigm to evolve and remain competitive on the world market [START_REF] Taisch | Recommendations for the Future of Manufacturing[END_REF][START_REF] Oztemel | Literature review of Industry 4.0 and related technologies[END_REF]. To support small and medium-sized enterprises (SMEs) with their digital transformation, seen as a pillar of Industry 4.0, several maturity models have been developed for evaluating the level of digital maturity of individual companies [START_REF] Mittal | A critical review of smart manufacturing & Industry 4.0 maturity models: implications for small and medium-sized enterprises (SMEs)[END_REF][START_REF] Scremin | Towards a Framework for Assessing the maturity of manufacturing companies in Industry 4.0 adoption[END_REF]. The results of these evaluations are then used to design and set up digital transformation plans. However, there are many different approaches to assess the digital maturity of a company, and it can be difficult to identify the most appropriate option as they do not always focus on the same set of criteria.

The current literature shows that each maturity model has its specific objectives, benefits and challenges. Therefore, a suitable solution may lie in the partial exploitation of some of their core advantages, making it imperative to clearly identify where the existing maturity models match and where they differ [START_REF] Cognet | Towards a novel comparison framework of digital maturity assessment models[END_REF]. It must be noted that most digital maturity models do indeed share some common characteristics and goals. For example, most of them use a set of questions to evaluate certain criteria that are grouped in dimensions, and possibly sub-dimensions. However, once formalised, criteria are not always evaluated in the same way. In some cases, an evaluation is performed by asking the user to self-assess the levels, or by using some black box mechanisms to compute a score from multiple user-specified answers. Regarding the questionnaires, some approaches refer to self-assessment, whereas others are focused on guided assessment without an automatic benchmarking system.

Maturity models are usually organised in sets of questions and answers. Some models identify "items" [START_REF] Schumacher | A maturity model for assessing industry 4.0 readiness and maturity of manufacturing enterprises[END_REF], while others use "factors" [START_REF] Samaranayake | Implementing industry 4.0a technological readiness perspective[END_REF] or "variables" [START_REF] Schwer | Digital maturity variables and their impact on the enterprise architecture layers[END_REF] that define digital maturity. When evaluating the digital maturity of a company, the so-called criteria can be considered as KPIs for monitoring the status of a company; this is the terminology that is used in this document. Hence, comparing two maturity models is conducted so as to identify the level of correspondence between two sets of KPIs.

The final objective of our work is to propose an approach to systematically compare existing digital maturity models. The comparison is based on the identification of matches between the digital maturity indicators of various maturity models. In this context, the objective of this paper is to propose a methodology to evaluate two frameworks for comparing digital maturity models and their indicators (considered KPIs). One framework is called 'manual', and the other, 'semi-automatic'. The two frameworks aim to assess the distance between two maturity models. The methodology aims to adjust and validate the semi-automatic comparison framework to efficiently identify potential matches between the maturity indicators of several models. Both the manual and the semi-automatic frameworks have the same goal: to help experts to match the KPIs. This paper aims to demonstrate how two distinct frameworks for comparing two digital maturity models can be compared and how the semi-automatic one can be fine-tuned in order to produce similar results to the expert-driven manual one. This paper's contribution is to explain how the overall methodology was developed. The outcome of applying the overall maturity assessment is intended to be useful to organisations that need to determine which maturity model to use.

The specific contributions of this paper are threefold:

1 A new methodology for comparing two similarity assessment frameworks, one manual and the other semi-automatic, that are themselves used to compare two maturity models. The comparison is performed at the level of keywords, which are associated with KPIs related to questions/answers that have been taken from questionnaires and categorised by dimension and sub-dimension. At this abstraction level, the work is conducted in a common space (the space of the keywords) to obtain comparable results. It shows that moving from the frameworks to more tangible data, i.e., the matching matrices in our case, can help to more efficiently compare them while introducing ad-hoc criteria.

2 A list of the KPIs reverse engineered from two digital maturity models. These are of direct interest to companies, which can incorporate them in their own KPI grids and use them to perform their own digital maturity assessments, and to consulting firms wishing to develop their own maturity models.

3 An analysis of the similarities (matches) found between the two frameworks to support the fine-tuning of the semi-automatic framework. This latter framework is much more efficient than the manual one and, once fine-tuned, can be reused to work directly on the comparison of a large number of maturity models (not included in this paper).

The comparison of the two frameworks is composed of several steps: the reverse engineering of the KPIs from existing models, the definition of keywords, the manual/semi-automatic identification of the matches, analysis of the matches found in the two frameworks, and the fine-tuning of the semi-automatic framework. The two similarity assessment frameworks, and the methodology used to compare them, have been tested with two well-known digital maturity models, IMPULS [START_REF] Lichtblau | IMPULS -Industrie 4.0 Readiness[END_REF] and PwC -PricewaterhouseCoopers - [START_REF] Geissbauer | Industry 4.0: Building the digital enterprise[END_REF]. This paper is organised as follows. After a summary of the background literature in Section 2, an overview of the whole comparison methodology is presented in Section 3. The manual and semi-automatic similarity assessment frameworks are introduced in Sections 4 and 5, respectively. The methodology used to compare the two frameworks and enhance the semi-automatic one is defined in Section 6. Section 7 discusses the results obtained following the proposed methodology validated on two maturity models, IMPULS and PwC. The last section concludes this paper and discusses the next steps.

Background literature

In general, the term "maturity" refers to a "state of being complete, perfect, or ready" [START_REF] Simpson | The Oxford English Dictionary[END_REF]. Maturity models are tools used to identify the best practices for the transformation of an organisation [START_REF] Schumacher | A maturity model for assessing industry 4.0 readiness and maturity of manufacturing enterprises[END_REF]. They provide a structured approach to initiate and accompany short-term operational projects, as well as medium-term tactical changes and long-term strategic change [START_REF] Felch | Maturity models in the age of industry 4.0 -do the available models correspond to the needs of business practice[END_REF].

Currently, there is a variety of digital maturity models to support companies in their digitalisation activities. Their common goal is to assess the digital maturity of an organisation and to provide an indication of the actions required to increase the maturity level. Existing studies have reviewed most of the common maturity models, in general [START_REF] Wendler | The maturity of maturity model research: a systematic mapping study[END_REF], as well as digital maturity models in particular [START_REF] Rossmann | Digital maturity: conceptualization and measurement model[END_REF].

According to the above and other related studies, the most common features for maturity models include their incorporation of maturity dimensions (usually 3-7 dimensions that are descriptive of the maturity to be assessed, and which are often divided into more detailed maturity criteria, descriptive of the related maturity dimensions), maturity levels and related maturity descriptions [START_REF] Akdil | Maturity and readiness model for Industry 4.0 strategy[END_REF]. Maturity dimensions, in general, can be divided into three broader categories: maturity of people/culture (e.g., skills, capabilities), processes/structures, and objects/technology (such as ICT tools). A recent literature review-based conceptual paper related to the broad concept of digital maturity [START_REF] Rossmann | Digital maturity: conceptualization and measurement model[END_REF] demonstrates that in current digital maturity studies, digital maturity has included aspects that can be divided into eight capability dimensions (i.e., broad digitalisation-related maturity categories): strategy, leadership, business model, operating model, people, culture, governance, and technology. It should be understood that the digital transformation of organisations concerns all their activities, and that digital technologies can be implemented to support this transformation [START_REF] Salkin | A conceptual framework for Industry 4.0[END_REF].

Several papers in the literature compare existing maturity models. Some authors perform an overall comparison of digital maturity models according to the following criteria: maturity level, dimensions, and scope of the study [START_REF] Akdil | Maturity and readiness model for Industry 4.0 strategy[END_REF]. [START_REF] Schwer | Digital maturity variables and their impact on the enterprise architecture layers[END_REF], on the other hand, use a 7-step comparison method that makes it possible to identify 147 'variables of digitisation' and classify them according to 6 dimensions. In both cases, the qualitative approaches make it possible to identify the scopes of the selected maturity models.

Earlier research [START_REF] Westerman | Leading Digital: Turning Technology Into Business Transformation[END_REF] presumed that the development of a specific set of the above types of digital capabilities leads to higher digital maturity, and moreover, that a higher degree of digital maturity can lead to superior corporate performance. Maturity models also provide the basis for guiding a digital transformation [START_REF] Schumacher | A maturity model for assessing industry 4.0 readiness and maturity of manufacturing enterprises[END_REF], but the development of a roadmap is necessary to ensure the actions will be performed in the right order [START_REF] De Carolis | Guiding manufacturing companies towards digitalization -a methodology for supporting manufacturing companies in defining their digitalization roadmap[END_REF]. However, such maturity models vary in terms of their structure, scope and industry focus [START_REF] Schwer | Digital maturity variables and their impact on the enterprise architecture layers[END_REF]. Furthermore, while Rossman's (2018) recent study has been among the first to present a more unified conceptualisations of the topic, the current research still reflects conceptual unclarity and fragmented views about the concept and the measurement frameworks for digital maturity. In general, the development of a maturity model, digital or not, requires a literature review to identify the existing models, along with a comparison of these models, facilitated by a group of maturity and digital experts [START_REF] Becker | Developing maturity models for IT management[END_REF].

Currently, there is no clear definition of what digital transformation really is. Many points of view can be observed in the literature [START_REF] Moeuf | The industrial management of SMEs in the era of Industry 4.0[END_REF][START_REF] Oztemel | Literature review of Industry 4.0 and related technologies[END_REF][START_REF] Pereira | A review of the meanings and the implications of the Industry 4.0 concept[END_REF]. This diversity is also reflected in digital maturity assessment models [START_REF] Gökalp | Development of an assessment model for Industry 4[END_REF][START_REF] Rossmann | Digital maturity: conceptualization and measurement model[END_REF]. An analysis of the digital maturity assessment models available in the literature shows that they do not all assess the same aspects [START_REF] Gökalp | Development of an assessment model for Industry 4[END_REF][START_REF] Mittal | A critical review of smart manufacturing & Industry 4.0 maturity models: implications for small and medium-sized enterprises (SMEs)[END_REF]. Instead, they only focus on an aspect of the modifications induced by the desire for digital transformation. Their evaluation may focus on the human aspect of the fourth industrial revolution, while others have a more technical dimension focused on the evaluation of installed technologies. However, the aggregation of all these digital maturity models should provide a more complete assessment of a company's digital maturity.

In addition, each model proposes its own system of dividing into dimensions and subdimensions [START_REF] Gökalp | Development of an assessment model for Industry 4[END_REF][START_REF] Mittal | A critical review of smart manufacturing & Industry 4.0 maturity models: implications for small and medium-sized enterprises (SMEs)[END_REF]. The issues of the different models partially overlap. Thus, in order to establish a complete list of KPIs, it is important to identify the similarities between the different questions rather than trying keep individual statements from the existing maturity models. The development of a maturity model requires a methodology to be followed. Comprehensive literature reviews show that this can be a lengthy process and suggest that the choices made at each stage should be justified [START_REF] Becker | Developing maturity models for IT management[END_REF][START_REF] Wendler | The maturity of maturity model research: a systematic mapping study[END_REF]. A review of the literature makes it possible to identify existing maturity models, and a comparison of the models selected can then serve as a basis for the new model. Thus, assessing the similarities of different maturity models is not straightforward, as it requires a deep understanding of and expertise in the widely varying domains and dimensions covered by the available multiple models. [START_REF] Cognet | Towards a novel comparison framework of digital maturity assessment models[END_REF] developed a manual similarity assessment framework that can manually identify the level of match between the KPIs extracted from two digital maturity models used as the inputs in their framework. The matches are identified with the help of several experts. This framework is time consuming, as the matches must be evaluated one by one.

This paper intends to overcome these limits and extend this work, using a more efficient semi-automatic framework that performs similarity assessment by comparing keywords associated manually to the KPIs. The literature shows that the assessment of similarity between two sentences can be performed and automated using a word analysis [START_REF] Liu | Assessing sentence similarity using wordNet based word similarity[END_REF]. The approach developed and detailed in this paper is a simplified version of what is observed in the literature. These methods also make it possible to quantify the similarity between two sentences and to limit subjective bias in the sentence analysis. Comparing a known number of words to a database that enables the classification of these same words according to their meaning or the desired grouping can give a numerical indicator for maturity indicator comparison.

The first results obtained with the semi-automatic process were slightly different than those obtained following the manual process, thus requiring a further fine-tuning step. This paper introduces a new methodology that compares the manual and semi-automatic frameworks and improves the semi-automatic framework accordingly.

The large variety of available digital maturity models justifies the need to develop a framework to assess their similarities in an efficient and accurate manner. The literature review shows that existing comparison approaches are qualitative. Our semi-automatic framework, on the other hand, is quantitative. This section introduced the background literature of maturity models, and the next section focuses on presenting the overall methodology utilised to compare two digital maturity model similarity assessment frameworks.

Overall comparison methodology

The literature review revealed that there is a large diversity between digital maturity models. The manual and semi-automatic similarity assessment frameworks were developed to compare maturity models for this study. In this paper, a methodology is developed to compare those two frameworks. This methodology is composed of several steps, illustrated in Figure 1 and detailed in the next sections:

• reverse engineering (RE) of the lists of KPIs (or criteria) 1 K c L and 2 K c
L of the two maturity models 1 K M and 2 K M , respectively, to be compared • similarity assessment using both the manual and the semi-automatic frameworks.

The output of the manual framework is a matching matrix 1,2 m M characterising how much the two maturity models match. The output of the semi-automatic framework is the matching matrix 1,2 a M • computation of the deviation matrix 1,2 ΔM characterising the deviations between the results obtained by two similarity assessment frameworks

• fine-tuning of the semi-automatic framework. The reverse engineering step is required for all maturity models, even if they provide their KPIs and information about models' KPIs. This step is designed to extract and formalise the list of KPIs that best characterises the criteria adopted by a given maturity model in order to assess the maturity levels, accomplished by using all the available resources describing the considered maturity model (e.g., online self-assessment tools, questionnaires, benchmarking reports, articles). How this list is determined has been detailed in [START_REF] Cognet | Towards a novel comparison framework of digital maturity assessment models[END_REF]. In short, the output list of KPIs results from consensual exchanges involving a pool of experts in the domain. During the evaluation process, experts are requested to focus on the explicitly available information rather than on more implicit data whose interpretation could be questionable. Following this process, the risk of bias due to reinterpretations is reduced, but cannot be fully disregarded.

The reverse engineering is a pre-processing step required to obtain the KPIs used as the inputs of the manual and semi-automatic similarity assessment frameworks. This is a prerequisite common to both frameworks and detailed in [START_REF] Cognet | Towards a novel comparison framework of digital maturity assessment models[END_REF]. Maturity models use dimensions and sub-dimensions to group the questions/answers by broad categories. The reverse engineering of KPIs homogenises the maturity models' questions/answers. For instance, IMPULS has broad questions for which multiple answers are expected, whereas PwC has more precise questions and only two sliders that need to be positioned to answer. Therefore, the KPIs are also grouped along these dimensions and sub-dimensions, since they are directly derived from the questions/answers. The other steps of the proposed methodology are detailed in sections 4 and 5.

Manual similarity assessment framework

The manual framework evaluates the level of matching between the KPIs of the two digital maturity models whose similarity is being assessed [START_REF] Cognet | Towards a novel comparison framework of digital maturity assessment models[END_REF].

A maturity model M K (with K ∈ {IMPULS,PwC,ADN,…}) makes use of

K c N criteria denoted as K i C (with [1.. ]) K c i N ∈ and grouped in K d N dimensions denoted as K j D (with [1.. ]) K d j N ∈ . The j th dimension K j D contains , K c j N criteria, which start at index K j
s and end at index K j e . The criteria can be gathered together in a single list

{ , [1.. ]} K K K c c i L C i N = ∈ , or in separate lists , { , [ .. ]} K K K K i j j d j L C i s e = ∈ associated to their respective dimensions and with [1.. ] K d j N ∈
. The following rules apply:

, 1

K d N K K c c j j N N = =  (1) 1 1 , 1 1, 2.. , K K K K K j j c j d s and j N s s N - -   = ∀ ∈ = +   (2) , 1 [1.. ], j K K K j d c k k j N e N = ∀ ∈ =  (3) The criteria K i C with [1 ] ( ) .. K c i N ∈
of a maturity model K M are the KPIs used to evaluate the maturity.

The similarity assessment of the maturity models is performed at the level of the KPIs. To characterise how much the KPIs match, three levels are introduced: Strong match, Partial match, and No match. Two KPIs are considered a Strong match if the experts involved in this process identify a strong similarity between the two. Conversely, if the two KPIs do not share any similar features, a No match is considered. In between these two situations, when the KPIs share some similar features, but also have dissimilarities, a Partial match is assigned. Such a three-level matching analysis presents a good trade-off between an under-segmentation, which would lead to a coarse analysis, and an over-segmentation that would complexify the comparison making it cumbersome and unworkable.

Therefore, the matching function f m , with m referring to the manual framework, evaluates the matching level of two KPIs, ( )

1 2 1 2 1 2 1 2 1 2 1 2 ,
Strong if and strongly match

Partial if and partially match

No otherwise

K K i i K K m i i K K i i C C f C C C C   =    (4)
This function is then called to fill in the matching matrix 1,2 m M containing the ( )

1 2 K K c c N N ×
values returned when applied with the KPIs of 1 K M and 2 K M . Clearly, due to this procedure, it is possible to observe that the matching function is symmetric, i.e., it returns the same matching level no matter the order of the arguments.

Here again, the assessment of the matching levels results from consensual exchanges involving a pool of experts. In a first individual phase, experts are asked to suggest a matching level for each KPI pair. Then, during a consensus phase, experts exchange information about their classifications and discuss the matching levels for which there are discrepancies. When the discussion fails to reach an adequate consensus, a simple majority rule can be used, giving greater weight to the choice(s) of the most experience experts. Ultimately, an additional expert may be considered to solve any residual conflicts. Thus, the matching process results strongly rely on the exchanges between the experts, and consequently on their knowledge and experience in the domain. Clearly, similar results could not really be obtained using simple text-based similarity analysis tools.

Semi-automatic similarity assessment framework

During the manual comparison of the IMPULS and PwC models, the experts compared the KPIs directly with one another. After this first comparison, two observations were noted:

1 working at the KPI level is time-consuming 2 the experts were concentrating on the important words of the KPIs.

Working at the KPI level is time-consuming because the experts must evaluate the matching level of each pair of KPIs, one-by-one. For instance, considering two maturity models based on an average of

1 2 K K c c N N =
= 30 KPIs each, the number of evaluations increases rapidly to 900. After testing with the manual framework, only 49 matches were identified out of the 825 KPI pairs formed by the two selected models.

The manual comparison also revealed that the experts were focusing on the important words of the KPIs. This observation led to the creation of the semi-automatic framework. The switch from KPI comparison (manual comparison) to keyword comparison (semiautomatic comparison) is also motivated by the number of KPIs to be compared (in this paper, 20 × 33 and then about 400 × 400).

The keywords work at a different level of abstraction and overcome the formulation. In other words, two KPIs can express the same idea but be written in different ways. The semi-automatic framework is composed of three steps: the definition of keywords for each KPI, the identification of matches between keywords, and an automatic scoring to evaluate the matches between KPIs. Thus, extracting keywords from each KPI and creating a matrix to classify the keywords then makes it possible to identify similar KPIs without focusing on the formulation.

In this framework, the similarity assessment is not based on the KPIs but on the keywords, which are defined to characterise the KPIs. Thus, for a maturity model

K M , experts define , K i q N keywords , K i h Q , (with [1.. ]) K i q h N ∈ in , K q i L which best characterise each KPI K i C (with [1.. ]) K c i N ∈ . Each maturity model leads to a list of keywords , { , [1.. ]} K K K q c q i L L i N = ∈
. These lists are then grouped into an overall list of keywords L q .

To perform an automatic correspondence between KPIs, the keywords of L q must be classified in a matching matrix Q which characterises the matches at the level of the keywords. Thus, each row of this matrix gathers keywords with a similar meaning. Thus, each row contains a distinct concept which could help in the development and structuring of a new maturity model in a later step.

This last step of the semi-automatic similarity assessment framework aims at automatically computing the scores characterising the matching levels of all the KPI pairs between two maturity models. This step takes as its input the lists of keywords of two maturity models ( )

K K c c N N ×
real values ranging from 0 (no match) to 1 (perfect match). This computation relies on a low-level counting function f a (where a stands for automatic) that can verify if a keyword Q matches at least one of the elements of a list of keywords L according to the matching matrix Q. This function is defined as follows:

1 if / and are on the same row in ( , ) . 0 otherwise

h h a Q L Q Q Q f Q L ∃ ∈  =   (5)
Thus, for each KPI pair ( )

1 2 , K K i j C C ∈ ( ) 1 2 , K K c c L L , with (i,j) ∈ 1 2 1.. 1.. K K c c N N   ×       , the two related lists of keywords 1 , K q i L and 2 , K q j
L are automatically analysed so as to compute their level of match and fill the term (i,j) of the matrix 1,2 a M as follows:

( ) ( ) ( ) , , 2 1 2 1 2 1 2 1 , 1,2 , , , , , , 1 1 1 , , K j K i q q N N K j K i K K a a a q j q i h h i j K j K i q q h k M fQ L fQ L N N = =     = +   +     (6)
The terms of this matrix are always between 0 and 1. Compared to the manual similarity assessment framework, the semi-automatic one helps to better characterise the so-called Partial matches. However, in the next section, the comparison of the two matrices 1,2 m M and 1,2 a M will not consider those real values, and all values in between 0 (no match) and 1 (perfect match) will be associated with a Partial match.

It is worth emphasising that, at its core, the manual framework acts at the KPI level rather than at the question/answer or dimension/sub-dimension level, whereas the semiautomatic framework acts at the keyword level to overcome the formulation of the KPIs. Focusing on KPIs and keywords reduces the subjectivity when comparing different models, since the keywords characterise the essence of the KPIs. This is the strategy followed in this paper. The alternative, i.e., working at the level of the dimensions would make the comparison significantly more difficult. Indeed, the dimensions are very abstract and are highly dependent on the design of each maturity model. Each model groups the questions into dimensions (and sometimes sub-dimensions) depending on its own vision of maturity assessment. Using keywords to compare KPIs is simpler than comparing KPIs directly, as the keywords convey the essence of the KPIs. The final list of keywords is the lowest common denominator and establishes the 'dictionary' that defines the common space for comparing maturity models.

Fine-tuning of the semi-automatic framework

This section details the two last steps of the overall comparison methodology, i.e., the comparison of the matching matrices 1,2 m M and 1,2 a M and its use to fine-tune the semiautomatic similarity assessment framework, as described in Figure 1.

The matrix 1,2 m M is the result of quite a long analysis (more than two days) by several experts involving complex discussions and consensus phases. As a result, it is assumed in this work to be the reference matrix that embeds the experts' knowledge and expertise, to be reproduced by the semi-automatic matrix during the fine-tuning process.

Comparison of the matching matrices

As briefly introduced at the end of section 4, the two matrices to be compared do not own comparable values directly. The elements of 1,2 m M can be of three types (Strong, Partial or No match), whereas 1,2 a M is filled with real values in between 0 and 1. Thus, to be able to compare the two matrices, and to observe the possible differences, a deviation matrix 1,2 ΔM is defined as follows:

( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) 1 ,2 1,2 , , 1 ,2 1,2 , , 1,2 , 1 ,2 1,2 , , 1
m a i j i j m a i j i j i j m a i j i j m a i j i j M M M M M M M M M  = =   - ≠ =  =  - = >    ≠ >  (7)
In other words, when an element of 1,2

ΔM

is equal to 0, it means that the related elements in 1,2 m M and 1,2 a M match, and consequently that the manual and semi-automatic frameworks result in the same conclusion at the level of the underlying KPIs. When an element of 1,2

ΔM

is equal to -2, it means that the two frameworks differ in their conclusion, and that the semi-automatic framework identifies a false-positive match, i.e., a match not found by the experts involved in the manual framework. Thus, in this case the semi-automatic framework over-estimates the match level of the underlying KPIs. Finally, when an element of 1,2

ΔM

is equal to -1, it means that the semi-automatic framework does not identify a match that has been found by the experts following the manual framework. In this case, the value returned by the semi-automatic framework is a false-negative, which is not desirable as it means that some matches are not properly captured.

It is important to understand that the deviation matrix is used only to fine-tune the semi-automatic framework and not used anymore afterwards.

Three types of change for fine-tuning of the semi-automatic framework

Given that the ultimate objective is to replace the manual similarity assessment framework by a semi-automatic one, it is important to fine-tune the latter so that it identifies at least the matches found manually by the experts. Following such a finetuning process, the false-positive matches would remain, while the false-negative ones would be removed. In other words, at the end of this process, the deviation matrix 1,2 ΔM should only contain the digits 0 or -2.

During the fine-tuning process, the KPI pairs whose match level is equal to -1 in the deviation matrix 1,2 ΔM are identified and treated one by one. Three types of changes can therefore be considered and are listed below in order of preference:

• First, the lists of keywords used to characterise the KPI pairs whose match level can be considered as a false-negative can be changed. Keywords can either be removed if found to be irrelevant, or added if those already used do not allow for a complete characterisation of the KPIs. New keywords can either be selected from the complete list L q , or added to it if none of the available keywords conveys the related meaning;

• Second, the matching matrix Q can be changed by either reconsidering the position of some keywords on the various rows, or by splitting or merging some rows; and

• Lastly, the matrix 1,2 m M , obtained by the experts at the end of the manual framework, may be changed and some of its values reconsidered during a new consensus discussion. This last solution is to be followed only if the first two changes did not produce a convergence.

Of course, the two first types of changes may affect other matches that were not initially considered as false-negative, i.e., the new value of -1 could appear in the deviation matrix. Thus, as soon as a list of keywords or the matching matrix is modified, all the values of matrices that may have changed must be revaluated. At the end of this process, the semi-automatic framework is fine-tuned, and it produces results similar to those of the manual framework, with no false-negatives.

Results and discussion

Even though there is a wide variety of maturity models available in literature, the proposed comparison frameworks have been tested and validated with two of the earliest maturity models: IMPULS and PwC. Those two maturity models were selected because they are both digital maturity self-assessment tools easily available online, and because their number of questions and number of dimensions are quite similar and reasonably low, making it simpler to validate the proposed comparison methodology. The resulting fine-tuned semi-automatic similarity assessment framework can be used on any other maturity model.

IMPULS and PwC maturity models

Considering the formalisation introduced above, the two maturity models can be characterised by the values presented in Table 1.

Table 1

Numerical characteristics of the two maturity models being compared {6,5,7,1,4,2} {6,6,5,6,6,4} This table shows that both maturity models have the same number of dimensions, whose details are given in Table 2. Clearly, this is a unique case, as there is no obvious reason to have

K K1=IMPULS K2=PwC K d N 6 6 K c N 25 33 , { , [1.. ]} K K c j d N j N ∈
IMPULS PwC d d N N =
. Table 2 indicates that there is no correspondence between the dimensions of the two selected models, which is why the comparison is based on smaller components, the KPIs. The table also reveals that each dimension is not evaluated with the same number of KPIs. For instance, the first dimension of IMPULS contains six KPIs, whereas its last dimension has only two. This might be a good indicator of the importance of a given dimension in the overall maturity model. Availability of add-on functionalities (e.g., memorisation, localisation, self-reporting) in the company's products in order to make them smart 20 Availability of, and customer integration with data-driven services that use data gathered during the production and usage phases 21

Capacity to collect and analyse data from the usage phase 22

Share of revenues derived from data-driven services 23

External share of data collected all along the product lifecycle 24 Levels of the employees' skills with respect to the future requirements (e.g., IT infrastructure, automation technology, data analytics) of Industry 4.0 25

Level of the company's effort to acquire new skills and train its employees Level of integration of sales channels used to sell the organisation's products 8

Level of integration of communication channels for customer interaction 9

Availability of digital tools and technologies to support the organisation's sales force 10 Degree of dynamic customisation of the prices based on customers' willingness to pay 11 Degree of customer data analysis to increase customer insight 12

Level of collaboration with partners regarding customers' access approach 13 Degree of digitisation of activities from product development to production 14

Ability to monitor production and to dynamically respond to changes in demand 15 Degree of integration of the end-to-end IT-enabled planning and steering process over the entire value chain 16

Degree of digitisation of the production equipment up to a virtual representation of the factory 17

Degree of digitisation of activities from customer's order to service 18 Degree of consideration of the digitisation and Industry 4.0 requirements in IT architecture 19

Level of use of a manufacturing execution system (MES) or of a similar system control the manufacturing process 20

Level of maturity of the IT and data architecture to gather, aggregate and interpret realtime manufacturing, product and client data 21

Importance of new technologies (social media, mobility, analytics and cloud computing) to enable business operations 22

Ability of the IT organisation to fulfil business requirements within the requested time, quality and cost 23

Level of IT integration with customers and partners 24 Degree of sophistication of the digital compliance policy 25 Levels of the organisation's IP protection and of the external IP consideration 26

Level of consideration of the digital product portfolio and production factory in the risk management 27

Level of management of the digital components of the organisation's value chain with respect to tax-related topics (IP location...) 28

Level of consideration of production in the organisation's IT security concept 29

Level of consideration of the service-partners or customers in the organisation's compliance and risk management 30

Ability to create value from data so as to optimise operations and foster new business models 31

Level of the organisation's capabilities and resources related to Industry 4.0 32

Level of involvement, support and expertise of the organisation's managers with regards to Industry 4.0 33

Level of collaboration of the organisation with external partners (e.g., academia, industry, suppliers, customers) on Industry 4.0 topics

The first step of the proposed methodology is designed to reverse engineer the KPIs of the maturity models to be compared (Figure 1). In [START_REF] Cognet | Towards a novel comparison framework of digital maturity assessment models[END_REF], it was decided to keep the KPIs already formalised in the models' documentation, even though they were sometimes quite generic, without considering the underlying dimensions and corresponding questions. However, at the first attempt with the manual framework, the experts encountered problems in understanding some of the indicators. In a second attempt, the experts decided to reverse engineer the KPIs and to disregard those that were formalised too synthetically. Table 3 contains 4. Starting from the available online self-assessment tools of IMPULS and PwC, each question and possible answers have been carefully analysed and discussed to come out with a consensual formalisation of the KPIs. This step is not straightforward and required several in-depth discussions to achieve a consensus. The main difficulty was to avoid over-interpretation of the online questionnaire and to remain as objective and factual as possible.

Initial matching matrices and deviation matrix

Once the lists of the IMPULS and PwC KPIs have been obtained, the two similarity assessment frameworks can be run separately.

Following the manual framework, the matching matrix 1,2 m M can be filled out while evaluating the matching levels between all the KPI pairs of the two maturity models. As mentioned in [START_REF] Cognet | Towards a novel comparison framework of digital maturity assessment models[END_REF], the experts felt the need to develop selfunderstandable KPIs that directly embed the context within their formulation. This was done during the reverse engineering step, and the identified KPIs of Table 3 and4 is presented in Table 5. Green colours correspond to Strong matches between two KPIs, and yellow colours to Partial matches, whereas no colour indicates No match. For instance, five KPIs from PwC have a strong match with five different KPIs from IMPULS. It is also clear that KPI 13 from IMPULS partly matches four KPIs from PwC, and so forth. To better understand the strong matches, the matching matrix of IMPULS compared to itself would be a square matrix with only green cells on its diagonal and with 100% of the KPIs strongly matched.

The semi-automatic framework was utilised next, and the matching matrix 1,2 m M completed and presented in Table 6. Experts used 133 keywords overall (i.e., card ( )

p L =133)
, with an average of 3 keywords per KPI. These keywords are sorted in the matching matrix Q, defined by 62 rows, which means that the adopted keywords cover 62 different concepts. This value is directly comparable to the number of KPIs in both maturity models (25 for IMPULS and 33 for PwC), which suggests that the KPIs do not overlap very much. In Table 6, each non-zero value is coloured orange, independently of whether it has a high or low matching level. As for 1,2 m M this matrix clearly highlights the KPIs that have no correspondence in the other maturity model. For instance, KPI 9 of PwC can be considered as specific to PwC as it does not match any of the IMPULS KPIs. ΔM is then computed and displayed in Table 7. It characterises the deviations between the two similarity assessment frameworks. A red colour corresponds to a value of -1, which means that a match has been identified with the manual framework but not with the semi-automatic one (i.e., a false-negative with respect to the semi-automatic framework). In another example, a purple colour represents a value of -2, which means that a match was revealed by the semi-automatic framework but was not been detected by the manual one (i.e., a false-positive). Of course, if the two frameworks both identify a match, or they identify no match between two KPIs, the colour remains white.

During the first test of the semi-automatic framework, some manual matches were not found (red cells or false-negatives). Indeed, Table 7 directly highlights the 31 potential conflicts (i.e., red cells) resulting from a first try of the methodology defined to compare the manual and the semi-automatic frameworks (Figure 1). This corresponds to only 3.8% of the complete list of 25×33 = 825 cells, which can be considered reasonably good for a first attempt. However, this result is not satisfactory, as the semi-automatic framework should detect all matches that would be detected with a manual assessment process. Thus, some fine-tuning is therefore needed, as presented next.

Comparison of the two frameworks and fine-tuning of the semi-automatic framework

As mentioned before, the last step of the proposed methodology aims at removing the undesired deviations revealed by the deviation matrix. Thus, particular attention must be paid to the fine-tuning of this semi-automatic framework. As discussed in section 6.2, three types of actions can be taken. Thus, some KPIs were not described by the right keywords (some keywords have thus been removed, added, or modified), some have been moved in the matrix Q, and as a last resort, the matching matrix 1,2 m M itself has been slightly modified. Some examples of the three types of actions are detailed in the following paragraphs to better understand these adjustments to the semi-automatic framework.

The experts' first task was to review all the KPIs and ensure they were characterised by the right keywords. This review made it possible to modify, add or delete keywords that had a broad meaning. For example, the KPI 32

PWC C

was characterised by the following three keywords: 'expertise', 'organisation's managers' and 'i4.0'. However, 'i4.0' is a broad keyword that could characterise many KPIs and be placed on several lines of the keyword matrix, so it was removed. The keyword 'expertise' evolved into 'expertise wrt i4.0' to be more precise, and 'organisation's managers' was replaced with 'managers support'. For the KPI 12 PWC C

, the different iterations made it possible to modify the keyword 'collaboration' (to 'external collaboration') to specify its meaning, keep the keyword 'partners' and add the keyword 'customers'.

At the end of the first test of the semi-automatic framework, the experts noticed that there were a lot of lines with only one keyword in the keyword matrix and decided to classify the keywords in a different way. For example, the first keyword matrix had 'i4.0 strategy', 'indicators' and 'strategy review' on three separate lines. These three keywords were grouped together on the same line in the matrix, as they are all related to the digital strategy. Another way is to split a line across multiple existing lines. In the first attempt, the experts grouped 'MES' (manufacturing execution system) and 'manufacturing process' on one line. When modifying the keyword matrix, 'MES' was placed on a line grouping digital tools, and 'manufacturing process', on a line dedicated to production digitalisation. Each time a keyword was modified, the experts made sure the keywords were still related to the KPIs they characterised.

The last way for the experts to tune the semi-automatic framework was to check on the manual matches. Manual matches can reveal a misunderstanding on the part of the experts or that the modification of statements for IMPULS KPIs has not been taken into account. For example, the KPIs 9

IMPULS

C

and 12 PWC C evolved between the time the manual framework was implemented and the time the semi-automatic framework was used. Therefore, these KPIs are not similar. The first KPI evaluates the organisation's capacity to create a digital model of the factory from production data, while the second KPI focuses on evaluating the collaboration between the organisation and partners. Thus, the experts decided to remove this match from the manual matching matrix.

Ultimately, 139 keywords have been used, with an average of 3 keywords per KPI, and distributed on 57 rows of the matrix Q. These modifications took place in a series of consensual discussions. An intriguing observation --when checking the matches in 1,2 m M the experts noticed that some of the manually identified matches were occurring because of some over-interpretations of the core meaning of the different KPIs --and this had to be corrected. At the end, the experts had modified 11 partial matches out of the 49 matches in the 1,2 m M . While there is a definite increase in the false-positive matches, as there are more purple cells in Table 10 than in Table 7 (57 vs. 46), it should be noted that the semiautomatic framework saves a significant amount of time. Indeed, the experts no longer need to evaluate all possible KPI couples (825), but only those recorded by the semiautomatic framework (95 matches, Table 9).

It should be emphasised that the number of false-positives increases between Tables 7 and 10 because the fine-tuning stage results in:

1 the aggregation of several lines in the keyword matrix, which increases the number of false-positives 2 the attribution of more keywords to each KPI, which increases the number of falsepositives as well.

However, the increase in false-positives remains acceptable since the semi-automatic framework automatically eliminates about 80% of the KPI pairs that have no link (truenegatives). This is particularly valuable when comparing more than two maturity models.

The new matrices are displayed in Tables 8, 9 and 10. Having reached the end of the process, the semi-automatic framework has been properly fine-tuned and the deviation matrix 1,2

ΔM

no longer reveals any false-negative matches (the red cells have been eliminated). The semi-automatic framework has been validated, since the results clearly demonstrate that it retrieves all matches identified by the manual framework. It therefore yields the same results as the manual framework, but in just a second, which was the primary objective.

Conclusions and future works

Digital maturity models help to characterise the maturity level of SMEs with respect to specific KPIs and dimensions, and consequently they provide important inputs with which to design improved digital transformation plans. Many countries and consulting firms have been engaged in developing their own models, and so it is important to understand the positioning of each model with respect to the others. It is possible to envision different strategies to identify similarities and differences between the available models within the Industry 4.0 paradigm. This paper has compared two similarity assessment frameworks: a manual framework and a semi-automatic framework. The manual framework relies on consensual discussions between experts who directly identify how much the KPIs used in one maturity model match the KPIs adopted by another. The semi-automatic framework also compares the KPI pairs, but through set of keywords associated to each KPI. Keywords are sorted in a matching matrix, which helps to characterise the level of matching of the KPI pairs to which they are associated. The manual and semi-automatic frameworks are compared in a common space, the space of the keywords. The KPIs are quantitatively compared based on matching scores obtained using Equation ( 7). This methodology can be reused to limit the subjective bias of the study when the input data are statements. Maturity models often do not explicitly propose roadmaps, and it is not the aim of this paper to do so either. However, from the reverse engineered KPIs, one can directly identify the important aspects required for a successful digital transformation. Thus, in a sense, the KPI grid can also give good hints as to what should be improved, even if it does not explain the roadmap steps to be put in place.

The comparison methodology was tested and validated with two maturity models, namely IMPULS and PwC. The results show that the proposed methodology was successful in fine-tuning the semi-automatic framework using results from the manual framework. As an outcome of this fine-tuning, the semi-automatic framework accurately captured the similarities and differences between the KPIs of two maturity models. A pool of four experts played a key role in the reverse engineering of KPIs and of keywords' definition steps. Once this fine-tuning was accomplished, the semi-automatic framework definitely accelerated the comparison process.

The analysis process revealed four valuable take-away lessons. First, mitigation measures had to be set up to avoid over-interpreting the KPIs. In this sense, experts were asked to focus on explicit and tangible information rather than on implicit information that could be open to interpretation. Second, KPIs should be as self-explanatory as possible in order to avoid having to go back to the dimensions or questions to clearly understand the context. Third, the definition of keywords for each KPI is particularly important. It is necessary to characterise all the ideas of the KPIs in order not to miss any possible matches. Fourth, the classification of keywords in the matching matrix should not be neglected, it is that classification which makes it possible to define a match between two KPIs.

This validated semi-automatic framework now needs to be exploited with additional maturity models. Comparing more maturity models will help to define a common kernel of KPIs shared by many models. In turn, this common kernel will likely help to specify a new maturity model, together with its KPIs and dimensions.

Figure 1

 1 Figure 1 Overall methodology for the comparison of manual and semi-automatic similarity assessment frameworks

  . It is defined as follows:

  obtained through a consensus workshop involving four experts. Similarly, the list PwC c L of the PwC c N =33 KPIs of PwC have been reverse engineered and are shown in Table

Table 2

 2 Dimensions of the two maturity models being compared

	j1	Dimension 1 IMPULS j D	j2	Dimension 2 PwC j D
	1	Strategy and organisation	1	Business models, product and service portfolio
	2	Smart factory	2	Market and customer access
	3	Smart operations	3	Value chains and processes
	4	Smart products	4	IT architecture
	5	Data-driven services	5	Compliance, legal, risk, security and tax
	6	Employees	6	Organisation and culture
	Table 3	KPIs reverse engineered from the IMPULS maturity model
	i1		Reverse engineered KPI 1 IMPULS i C
	1	Implementation status of Industry 4.0 strategy
	2	Operationalisation and review of Industry 4.0 strategy through a system of indicators
	3	IT and digital technologies used in the company
	4	Level of financial investment in the implementation of Industry 4.0 in various company sectors in the next 5 years
	5	Level of financial investment in the implementation of Industry 4.0 in various company sectors in the past 2 years
	6	Company sectors of systematic innovation management
	7	Availability of communication, control and interoperability functionalities of the equipment infrastructure
	8	Upgradability of communication, control and interoperability functionalities of the equipment infrastructure
	9	Level of digital modelling of the factory through the collection, storage and processing of data during production
	10	Digital tools used in the company's sectors
	11	Interface of the digital tools to the leading system
	12	Internal cross-sectors' level of information sharing
	13	External information sharing between sectors, and with customers and/or suppliers
	14	Level of deployment of workpiece self-guiding capacities through production
	15	Level of deployment of autonomous real-time response capacities to changes in production conditions
	16	Organisation/distribution of IT expertise across the company's departments
	17	Level of implementation of IT security solutions for internal/external data storage and data communication
	18	Use of cloud-based software, and of cloud services for data analysis and storage
	19

Table 4

 4 KPIs reverse engineered from the PwC maturity model

	i2	Reverse engineered KPI 2 PwC i C
	1	Contribution of digital features, products and services to the overall value creation of the organisation's portfolio
	2	Degree of digitisation of the organisation's products and/or services
	3	Possibilities for customer customisation of products
	4	Degree of digitisation of the product lifecycle phases
	5	Importance of data usage and analysis for the organisation's business model
	6	Intensity of the collaboration with external partners and clients for the development of products and services
	7

Table 5

 5

	Initial matching matrix	1,2 m M (see online version for colours)

Table 6

 6

	Initial matching matrix	1,2 a M (see online version for colours)

Table 7

 7 Δm (see online version for colours)

	Initial deviation matrix	1,2

Table 8

 8

	Final matching matrix	1,2 m M (see online version for colours)

Table 9

 9 Final matching matrix 1,2 a M (see online version for colours)

		6	24 25
		5	20 21 22 23
		4	19
	KPIs IMPULS	3	12 13 14 15 16 17 18
			10 11
		2	9
			8
			7
			6
			5
			4
		1
			3
			1 2	1 1	2	3	4	5	6	7 2	8	9	10	11	12	13 3	14	15	16	17	18 4	19	20	21	22	23	24 5	25	26	27	28	29	30 6	31	32	33	Automatic match.
																			KPIs PwC														Notes:

Table 10

 10 Final deviation matrix Δm 1,2 (see online version for colours)

		6	24 25
		5	20 21 22 23
		4	19
	KPIs IMPULS	3	12 13 14 15 16 17 18
			10 11
		2	9
			8
			7
			6
			5
		1	1 2 3 4	1 1	2	3	4	5	6	7 2	8	9	10	11	12	13 3	14	15	16	17	18 4	19	20	21	22	23	24 5	25	26	27	28	29	30 6	31	32	33	Automatic false-positive match.
																			KPIs PwC															Notes:

scope of the Industry 4.0.