domain: spi sdv

text:
Burak Kocak

Nathaniel Mercaldo

Bettina Baessler

Ilaria Ambrosini

Spyridon Bakas

Roberto Cannella

Alessandro Luca

Cappellini

Armando Ugo Cavallo

Leonid L Chepelev

Linda Chi

Hang Chu

Russell H The

Morgan

Nandita M Desouza

Matthias Dietzel

Salvatore Claudio Fanni

Andrey Fedorov

Laure S Fournier

Valentina Giannini

Rossano Girometti

Kevin B W Groot Lipman

Georgios Kalarakis

Brendan Kelly

Michail E Klontzas

Elmar Kotter

Yun Ho

Lee

Mario Maas

Luis Marti-Bonmati

Henning Müller

Nancy Obuchowski

Ekaterina Petrash

Elisabeth Pfaehler

Daniel Pinto

Santos

Andrea Ponsiglione

Sebastià Sabater

Francesco Sardanelli

Philipp Seeböck

Arnaldo Stanzione

Alberto Traverso

Martin Vallières

Lisanne V Van Dijk

Joost J M Van Griethuysen

Robbert W Van Hamersvelt

Peter Van Ooijen

Federica Vernuccio

Alan Wang

Stuart Williams

Zhongyi Zhang

Alex Zwanenburg

Tugba Akinci

D ' Antonoli

METhodological RadiomICs Score (METRICS): A quality scoring tool for radiomics

Purpose: To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies.

Methods:

We conducted an online modified Delphi study with a group of international experts.

It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated.

Results:

In total, 59 panelists from 18 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were, in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community.

Conclusion:

In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a wellconstructed framework for the key methodological concepts to assess the quality of radiomic research papers.

Clinical relevance statement:

A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and

Introduction

Radiomics is an evolving field of image analysis technique for extracting quantitative features from medical images with the premise of building predictive models and assisting clinical decision-making [1]. Since its introduction into medicine more than a decade ago, an exponential number of radiomics-related articles have been published yearly [2]. However, a growing translational gap exists between radiomics research and clinical practice [3,[START_REF] Pinto Dos Santos | A decade of radiomics research: are images really data or just patterns in the noise?[END_REF]. One of the main reasons for this issue is the poor quality of research methodology, including but not limited to, poor study design, inadequate description of image segmentation, feature extraction or model building methodology, lack of generalizability, lack of data, model and code sharing practices, all of which ultimately limit the reproducibility of the proposed radiomics models [3,[START_REF] Papanikolaou | How to develop a meaningful radiomic signature for clinical use in oncologic patients[END_REF][START_REF] Buvat | The Dark Side of Radiomics: On the Paramount Importance of Publishing Negative Results[END_REF][START_REF] Vallières | Responsible Radiomics Research for Faster Clinical Translation[END_REF][START_REF] Kocak | Transparency in Artificial Intelligence Research: a Systematic Review of Availability Items Related to Open Science in Radiology and Nuclear Medicine[END_REF].

In 2017, Lambin et al [START_REF] Lambin | Radiomics: the bridge between medical imaging and personalized medicine[END_REF] proposed the radiomics quality score (RQS), a set of assessment criteria covering the radiomics workflow to improve the quality of radiomics research. Since then, many systematic reviews have been published applying the RQS to published research to examine the quality of radiomics studies [START_REF] Spadarella | Systematic review of the radiomics quality score applications: an EuSoMII Radiomics Auditing Group Initiative[END_REF]. Nevertheless, some RQS item definitions may lead to ambiguity and the applicability of the items can be limited based on different characteristics of the study design, which may negatively affect the reproducibility of the score even among experts in the field [START_REF] Spadarella | Systematic review of the radiomics quality score applications: an EuSoMII Radiomics Auditing Group Initiative[END_REF][START_REF] Sanduleanu | Tracking tumor biology with radiomics: A systematic review utilizing a radiomics quality score[END_REF]. In addition, as shown previously [START_REF] Sanduleanu | Tracking tumor biology with radiomics: A systematic review utilizing a radiomics quality score[END_REF], a high RQS score does not always guarantee high quality of a study or lack of significant bias [START_REF] Welch | Vulnerabilities of radiomic signature development: The need for safeguards[END_REF]. Furthermore, this assessment system was developed by a small group of researchers and the development process was not detailed in-depth in terms of how it deals with the relative importance of each item that contributes to overall radiomics research quality. Thus, the need for an easy-to-use and transparent evaluation system developed by an international group of experts persisted.

Recently, the CheckList for EvaluAtion of Radiomics Research (CLEAR) guideline for reporting radiomics studies that covers the entire life cycle of optimal radiomics research, was published and endorsed by the European Society of Radiology (ESR) and European Society of Medical Imaging Informatics (EuSoMII) [START_REF] Kocak | CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII[END_REF]. The CLEAR reporting guideline has great potential to improve the quality of reporting in radiomics papers, which would ultimately lead to an improvement in research quality. Nevertheless, reporting guidelines are not assessment tools or instruments for measuring research quality [START_REF] Caulley | Reporting guidelines of health research studies are frequently used inappropriately[END_REF][START_REF] Logullo | Reporting guideline checklists are not quality evaluation forms: they are guidance for writing[END_REF]. Thus, the need remains for an easy-to-use, reproducible assessment system for radiomics research. In this paper, we propose a new quality assessment tool, METhodological RadiomICs Score (METRICS), which was developed by a large group of international experts in the field and is easy to use, aimed at improving research quality and closing the gap between research and clinical translation in radiomics and machine learning.

Material and Methods

Design and Development

As there is no guidance for developing scoring systems, the recommendations for developing reporting guidelines were followed [START_REF] Moher | Guidance for Developers of Health Research Reporting Guidelines[END_REF]. Therefore, a steering committee (T.A.D., B.K., and R.C.) was established first to organize and coordinate the development of METRICS.

To develop the METRICS tool, an online modified Delphi study with a group of international experts was planned. The process was organized in three stages. The steering committee members conducted the first stage (Stage#1), consisting of item preparation. The second stage (Stage#2) was held with the participation of a group of panelists from the EuSoMII Radiomics Auditing Group for discussion of the items to be voted on. The third stage (Stage#3) was carried out in 4 rounds by two separate groups of panelists to determine the METRICS items and their weights. The first three rounds of Stage#3 were aimed at determining which methodological items were eligible for METRICS. The items' weights were then determined in the final round of Stage#3. Following each round, the panelists received structured feedback on the preceding round to reconcile individual opinions.

The surveys were open for at least two weeks in each round in Stage#3, and a reminder e-mail was sent one week, three days, and one day before the deadline. When necessary (e.g., when overlapping with major conferences or holidays), deadlines were extended to ensure a reasonable number of panelists was achieved.

The modified Delphi surveys were carried out using a Computer Assisted Web Interviewing (CAWI) system, i.e., Google Forms (Google LLC). For online group discussions online platforms, i.e., Google Docs (Google LLC) or WhatsApp (Meta Platforms Inc.), were used.

To simplify the calculation of the METRICS score, the development of an online calculation tool was planned. A GitHub repository was also planned for providing updates and gathering community feedback.

Anonymity

Although the panelists voted independently, the voting rounds of the modified Delphi exercise were not anonymous to track panelists' participation. Only the organizers had access to the panelists' data, and they preserved the anonymity of the votes and their respective comments during and after the voting tasks (i.e., when feedback was provided after rounds).

Informed consent

At the start of the Delphi questions, participants' informed consent was requested using the same form. Participants may have opted out of the study at any time. Those who indicated a desire to decline the survey were to be deleted from future invitations. Only while the round was active, panelists could withdraw their votes.

Consensus criteria

The vote for "strongly agree" and "agree" accounted for agreement and "strongly disagree" and "disagree" accounted for disagreement. The "neutral" votes were not included in either decision. The consensus was defined a priori as either agreement (agreement ≥75%) or disagreement (disagreement ≥75%) [START_REF] Diamond | Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies[END_REF]. If there was no agreement or disagreement, it was referred to as "no consensus," and they were voted again. If "no consensus" items did not achieve agreement in the next voting, they were removed from the tool. The consensus items with disagreement were removed from the tool without further discussion.

Recruitment of participants

Individuals having significant experience in radiomics, machine learning, deep learning, informatics, or related editorial tasks from various countries were invited via an e-mail describing the development plan of the METRICS tool and explaining its purpose. Members of the EuSoMII Radiomics Auditing Group (Group#1 panelists) were assigned to discussion panels in Stage#2 and Round#3 of Stage#3. Other invitees (Group#2 panelists) were assigned to modified Delphi voting rounds (i.e., Round#1, Round#2, and Round#4 of Stage#3).

Modified Delphi

Stage#1 (preparation)

To identify potential items, a thorough and systematic literature review was conducted. Two members of the steering committee performed an independent literature search in PubMed using the following syntax to find the relevant checklists, guidelines, or tools: (radiomics) AND ((checklist) OR (guideline)). The search date was January 24th, 2023. All entries and related publications, if accessible by the readers, were assessed to determine the currently available tools. All eligible documents found were independently evaluated by the entire steering committee to develop the initial template of METRICS.

Participants were requested to consider the following principles: i, there should be no overlap between items; ii, an ideal study should be able to achieve a perfect score (i.e., all points available or 100%), meaning that items should not be mutually exclusive; iii, items must be objectively defined, to increase reproducibility; iv, not only hand-crafted but also studies based on deep learning should be considered and item conditionality should be assessed accordingly; v, since this is a methodological scoring system, the items should be mainly related to the "materials and methods" and "results" sections of a research paper; vi, while items should also aim at improving the methodological reproducibility and transparency of the studies, METRICS is not a reporting checklist; and vii, items should point out potential bias sources and help users to avoid them.

Considering the principles defined above, an initial draft was created with three organizers of the METRICS project. For any disagreement among the organizers, the decisions were made based on a majority vote.

Stage#2 (discussion with Group#1 panelists)

The items prepared by the organizers were presented to the EuSoMII Radiomics Auditing Group with the same principles and discussed online. This stage was an open discussion and not anonymous. The panelists were free to suggest adding, removing, merging, and modifying items.

Stage#3 (modified Delphi rounds)

Round#1 (item selection) On a 5-point Likert scale (strongly agree; agree; neutral; disagree; strongly disagree), the Group#2 panelists were asked to rate the extent to which they agreed with the inclusion of each item on the METRICS tool. With a text box, participants were further asked for suggestions on the item's name and definition. In addition, a text box was provided at the end of each section for participants to suggest additional items. After this round, the Group#2 panelists were provided with a statistical summary of each item from Round#1, along with anonymous comments.

Round#2 (continued for item selection)

The same panelists as in Round#1 were invited to participate in Round#2. Panelists who were invited but did not respond to Round#1 were also invited to participate in Round#2. Using the same structure as Round#1, panelists were also presented with items that reached no consensus as well as new item or items suggested in previous round.They were asked to use the same 5-point Likert scale to express their level of agreement with the inclusion of each item in the METRICS tool. No new item proposal was asked in this round. After Round#2, the same panelists were provided with a statistical summary of each item from Round#2, along with anonymized comments.

Round#3 (group discussion with EuSoMII Radiomics Auditing Group)

The purpose of Round#3 was to discuss the results of the previous rounds, modify if necessary, and finalize the items to be included in the METRICS tool. It was held on online platforms (Google Docs and WhatsApp Group). All Group#1 panelists were invited. The discussion included both agreed and unresolved topics. Any modification proposals were discussed and items were edited in consensus by the steering committee.

Round#4 (ranking of finalized items to determine the weights) Group#2 panelists who participated in at least one of the first two rounds (Round#1 and Round#2) were invited to this round. The panelists were asked to rank the categories and then all items within each category in order of their importance in radiomics research. After Round#4, the same panelists were provided with an anonymized statistical summary of each item and category.

Pilot testing

We invited Group#1 panelists to test the usability and understandability of the online checklist. Also, the final METRICS tool was tested on studies from the literature, including a sample of different pipeline designs and aims (i.e., handcrafted radiomics, deep radiomics and end-toend deep learning; lesion characterization and region of interest segmentation).

Statistical analysis

Descriptive statistics (i.e., median, interquartile range, percentage) were used to present the results. The ranks derived from hierarchical (i.e., multi-tiered) ranking with expert panel opinion were aggregated using their median value. Using the rank-sum method [START_REF] Roszkowska | Rank Ordering Criteria Weighting Methodsa Comparative Overview[END_REF][START_REF] Stillwell | A comparison of weight approximation techniques in multiattribute utility decision making[END_REF], median ranks were first converted to importance scores with the following formula: Score = (N+1) -Rank, where N is the total number of categories or total number of items within a category. The category weights were then rescaled to 1. The final weights of each item were computed as the product of the category and item weights (e.g., [weight of Category A] x [weight of Item#1 in Category A]). The items within the respective category went through the same rescaling procedure. The final METRICS score was calculated on a percentage scale, accounting for the conditionality of items and categories.

Results

All key points are summarized with a flowchart in Figure 1.

Modified Delphi

In total, the 3 steering committee members invited 61 experts to participate in this study, 56 of which accepted the invitation. In detail, 14 experts from the EuSoMII Radiomics Auditing Group (Group#1) accepted the invitation to participate in panel discussions (i.e., discussions at Stage#2 and Round#3 of Stage#3), together with the steering committee members. Furthermore, 42 experts (Group#2) accepted the invitation to perform Delphi voting (i.e., rating in Round#1 and Round#2; ranking in Round#4 of Stage#3). Country data of all participants is presented in Figure 2.

The literature search resulted in 58 publications. After independent evaluation of the content of these publications by steering committee members, 16 relevant checklists, guidelines or quality scoring tools were identified as potentially useful for designing a new quality scoring tool [START_REF] Lambin | Radiomics: the bridge between medical imaging and personalized medicine[END_REF][START_REF] Whiting | QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies[END_REF][START_REF] Bossuyt | STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies[END_REF][START_REF] Collins | Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement[END_REF][START_REF] Luo | Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research: A Multidisciplinary View[END_REF][START_REF] Martin | Critical Appraisal Checklist for Analytical Cross Sectional Studies[END_REF][START_REF] Vallières | Responsible Radiomics Research for Faster Clinical Translation[END_REF][START_REF] Mongan | Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers[END_REF][START_REF] Zwanenburg | The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping[END_REF][START_REF] Orlhac | Radiomics in PET Imaging:: A Practical Guide for Newcomers[END_REF][START_REF] Pfaehler | A systematic review and quality of reporting checklist for repeatability and reproducibility of radiomic features[END_REF][START_REF] Shur | Radiomics in Oncology: A Practical Guide[END_REF][START_REF] Sollini | PET/CT radiomics in breast cancer: Mind the step[END_REF][START_REF] Volpe | Machine Learning for Head and Neck Cancer: A Safe Bet?-A Clinically Oriented Systematic Review for the Radiation Oncologist[END_REF][START_REF] Jha | Nuclear Medicine and Artificial Intelligence: Best Practices for Evaluation (the RELAINCE Guidelines)[END_REF][START_REF] Hatt | Joint EANM/SNMMI guideline on radiomics in nuclear medicine : Jointly supported by the EANM Physics Committee and the SNMMI Physics, Instrumentation and Data Sciences Council[END_REF]. Based on the results of this literature review and previous experience, 33 items were initially drafted. These items were then reduced to 30 after discussion with the Group#1 panelists in Stage#2, as three were considered unclear or partly overlapping with other entries, with which they were merged.

The 30 items obtained after Stage#2 discussion were presented to the Group#2 panelists for the first round of the Delphi survey, which was completed by 40 of the 42 panelists. The consensus for an agreement was achieved for 26 items, while 4 items failed to achieve any consensus. No item reached the consensus threshold for disagreement. There was one new item proposal that was added to the list after discussion by the steering committee (item#17, robustness assessment of end-to-end deep learning pipelines). A summary of the votes in Delphi Round#1 is presented in Figure 3. The highest agreement (100%) was achieved by item#21 (i.e., consideration of uncertainty).

Following the Round#1, 4 items with no consensus and 1 newly proposed item were presented to the Group#2 panelists in Round#2 of the Delphi process. In this round, 41 of the 42 panelists participated. The consensus for an agreement was achieved for 30 items. There was no consensus on one item about prospective data collection, which was therefore removed from the list. There was no disagreement with consensus. A summary of the votes in Round#2 is presented in Figure 3.

All Group#1 panelists were invited to Round#3 for the panel discussion by the steering committee members. A small number of minor modifications were made to the item definitions at this time. The agreement was achieved for all 30 items within 9 categories.

The final Delphi round, Round#4, consisted of ranking of all 9 categories and the 30 items divided by category. This was performed by all 42 of the Group#2 panelists. A summary of the category and item ranks in Round#4 is presented in Supplementary file 1 Figure S1 and Supplementary file 1 Figure S2, respectively. According to their final weights, top 5 items with highest weights were as follows: item#3 (i.e., high-quality reference standard with a clear definition; weight, 0.092); item#27 (i.e., external testing; weight, 0.075), item#2 (i.e., eligibility criteria that describe a representative study population; weight, 0.074), item#11 (i.e., appropriate use of image preprocessing techniques with transparent description; weight, 0.062), and item#18 (i.e., proper data partitioning process; weight, 0.060). The lowest weights belonged to the three items of category "open science" and were as follows: item#28 (i.e., data availability, weight, 0.007), item#29 (i.e., code availability, weight, 0.007), and item#30 (i.e., model availability, weight, 0.007).

Weights calculated for categories and items are presented in

Anonymized individual votes and ranks obtained in the

Finalized METRICS tool

The final METRICS tool included 30 items within 9 categories and is presented in Table 1 with relative item weights. It also accounts for different study pipelines by including several conditional items. Figures 5 and6 present a flow diagram to exemplify their usage in practice.

A user-friendly online calculation tool was prepared to streamline the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html). It also allows printing (paper and PDF) and exporting (Excel spreadsheet). Supplementary file 3 (without explanation) and supplementary file 4 (with explanation) allow downloading the METRICS tool in table format. However, the use of the online tool mentioned above is highly recommended, as the final METRICS percentage score is based on the maximum achievable absolute score after accounting for item conditionality. This calculation can be performed automatically by the web-based tools (both online and offline versions). Supplementary file [START_REF] Papanikolaou | How to develop a meaningful radiomic signature for clinical use in oncologic patients[END_REF] includes evaluation examples from the literature, covering the use of METRICS on different radiomics pipeline designs.

A GitHub repository was also set up for the METRICS tool (https://github.com/metricsscore/metrics). The discussion function was also activated to receive community feedback to improve it in the future. Also, an offline version of the calculation tool can be downloaded from this repository, which requires no setup or installation but directly starts working on common web browsers such as Google Chrome (recommended; Google LLC). The online calculation tool and potential updates can also be accessed via this repository.

Total score categories

To improve the comprehensibility of the METRICS total score, we propose the use of 5 arbitrary categories as a representation of gradually increasing quality, namely total score between 0-20%, "very low"; 21-40%, "low"; 41-60%, "moderate"; 61-80%, "good"; and 81-100%, "excellent" quality. However, these categories should be validated through future systematic reviews using METRICS, and should be used as a complement of METRICS and not as a substitute for the quantitative score.

Discussion

In this work, we developed a scoring tool for assessing the methodologic quality of the radiomics research, i.e., METRICS, based on the input of a diverse and large international panel with 59 participants. Our study was conducted in 3 consecutive stages, with 4 rounds of the modified Delphi exercise in the last stage. Based on panelist ratings, 30 items within 9 categories were ultimately included in the METRICS tool. The weights of these items were then calculated using a hierarchical ranking of categories and items based on the rank-based assessment by the Delphi panelists. A web application was developed to automate the calculation of the METRICS score, and a repository was created to collect feedback from the radiomics research community.

There have been only few tools proposed to assess the methodological quality of radiomics research in the literature, e.g., the RQS [START_REF] Lambin | Radiomics: the bridge between medical imaging and personalized medicine[END_REF]. Despite the fact that the RQS was published as part of a review article, it has received so much attention from the community that it became the de facto standard for evaluating radiomics methodology [START_REF] Spadarella | Systematic review of the radiomics quality score applications: an EuSoMII Radiomics Auditing Group Initiative[END_REF]. Although it was developed and published by leading radiomics researchers, it lacked methodological transparency in terms of how it was developed and how the scores for each item were assigned. The first and most widely used version was designed to evaluate traditional radiomics and modeling in general, and thus does not apply to deep learning workflows.

Although not directly related to radiomics, the Must AI Criteria-10 (MAIC-10) checklist can be used to evaluate the quality of AI and medical imaging studies [START_REF] Cerdá-Alberich | MAIC-10 brief quality checklist for publications using artificial intelligence and medical images[END_REF]. It aims to simplify the process while overcoming some of the limitations of other published checklists in the fields of artificial intelligence and medical imaging. MAIC-10 is a very short and simple tool that covers a wide range of concepts. According to the authors of MAIC-10, unlike other checklists or quality scoring tools, it was designed to provide a quantitative, objective, and reproducible quality score with a broad scope of applications across studies on AI in medical imaging. MAIC-10 achieved a high correlation score to CLAIM [START_REF] Mongan | Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers[END_REF], a widely used 42-item reporting checklist, despite being tested on a small number of publications from the journal in which it was published. It was also proposed as the most reproducible checklist in terms of intraobserver reproducibility, with CLAIM taking second place. However, the MAIC-10 scores are unweighted, namely ignoring the relative importance of each item and simply assigning a score of 1 for adherence. Such a simple scoring strategy was also used for the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist as well [START_REF] Heus | Uniformity in measuring adherence to reporting guidelines: the example of TRIPOD for assessing completeness of reporting of prediction model studies[END_REF]. A recent radiomics-specific reporting checklist, the CLEAR checklist, was developed by an international initiative led by a group of experts and endorsed by ESR and EuSoMII [START_REF] Kocak | CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII[END_REF]. Although CLEAR was designed primarily as a reporting tool and not a methodological guide, it still provides useful information about the methodology. Furthermore, it has a shortened version called CLEAR-S that focuses solely on methodological aspects and open science, with no score or weights. There are also reporting checklists for AI and medical imaging that were not specifically designed for radiomics, such as CLAIM [START_REF] Mongan | Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers[END_REF]. CLAIM is a highly cited checklist that provides guidance for reporting and methodology. However, it was created by a relatively small group of scientists with no formal methodology for determining item eligibility, such as the Delphi method. Of note, a recent article provides a comprehensive review of available guidelines that can be used in AI research and medical imaging [START_REF] Klontzas | AI Reporting Guidelines: How to Select the Best One for Your Research[END_REF].

To develop the proposed scoring system, we used a modified Delphi method with an international group of panelists and defined weights of each item to present a more nuanced way of assessment. As a result, the category "Study Design" had the highest weight, and thus the biggest effect on the final score. This result is such that adhering to all items of the category may already allow a METRICS score ranging between 20% and 25%, considering all possible conditionals. It includes three items as follows: i, adherence to radiomics and/or machine learning-specific checklists or guidelines; ii, eligibility criteria that describe a representative study population; and iii, high-quality reference standard with a clear definition. The first item was introduced as a new concept in comparison to the RQS [START_REF] Lambin | Radiomics: the bridge between medical imaging and personalized medicine[END_REF] and MAIC-10 [START_REF] Cerdá-Alberich | MAIC-10 brief quality checklist for publications using artificial intelligence and medical images[END_REF] tools. The authors of the MAIC-10 checklist included the study design as a single item and defined it as a very broad concept. While most of their 10 items were discussed in at least half of the studies evaluated as part of the MAIC-10, the study design was not defined in any of the studies evaluated. Previously, the CLEAR checklist [START_REF] Kocak | CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII[END_REF] and, to a lesser extent, CLAIM [START_REF] Mongan | Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers[END_REF] drew attention to some of these concepts in terms of reporting.

It may appear surprising that the category related to open science practices had the lowest weight, and thus the lowest effect on the final score. As widely known, radiomic studies suffer from significant reproducibility issues, which have been mainly attributed to the lack of data, code, and model sharing practices leading to poor generalizability [START_REF] Kocak | Transparency in Artificial Intelligence Research: a Systematic Review of Availability Items Related to Open Science in Radiology and Nuclear Medicine[END_REF][START_REF] Gidwani | Inconsistent Partitioning and Unproductive Feature Associations Yield Idealized Radiomic Models[END_REF][START_REF] Zwanenburg | Radiomics in nuclear medicine: robustness, reproducibility, standardization, and how to avoid data analysis traps and replication crisis[END_REF][START_REF] Park | Reproducibility and Generalizability in Radiomics Modeling: Possible Strategies in Radiologic and Statistical Perspectives[END_REF]. This apparent discrepancy may be attributable to the panelists' consideration that proper study design "comes first". In other words, if the study's methodological steps are flawed, data and model availability becomes a secondary concern as these would still lack reproducibility. It should be noted that "reproducibility" refers to the ability to implement the same methods reported in a study on the same input data and obtain identical findings. This is entirely dependent on sharing of data, code and models and represents a guarantee of the correctness in reporting study results. On the other hand, "replicability" defines the ability of obtaining consistent results in relation to the same hypothesis when using a different patient population (and even partially different methodology). In practice, replicability is a better indicator of robustness of the conclusions drawn from a study in relation to a specific hypothesis and does not require meeting the criteria of the METRICS open science items, but only appropriate reporting of the study hypothesis and methodology. Finally, "generalizability" refers to the ability of a model to be applicable on a different patient distribution compared to the original study, and represents one of the major challenges facing radiomics and its translation to the clinical setting. More information on these topics can be found in [START_REF]Reproducibility and Replicability in Science[END_REF]. This apparently counterintuitive disconnection between open science practice and study quality can also be seen elsewhere in the recent literature. For instance, in a meta-research study about AI literature published in RSNA journals, it was reported that only 13% of the included literature shared data, 30% of the included literature shared code and only 11% of the shared code was actually reproducible [START_REF] Venkatesh | Code and Data Sharing Practices in the Radiology Artificial Intelligence Literature: A Meta-Research Study[END_REF]. Another recent literature review showed a similar trend, where the data sharing rate within randomly sampled AI publications from Q1-Q2 journals was only 1%, and proper model sharing (i.e., sharing premodeling, modeling, and post-modeling files at once) was observed in only 6% of the included studies [START_REF] Kocak | Transparency in Artificial Intelligence Research: a Systematic Review of Availability Items Related to Open Science in Radiology and Nuclear Medicine[END_REF]. Our belief is that papers scoring highly on METRICS will allow improved replicability and generalizability.

Even though an item focused on the role of prospective study design/data collection was initially included, the panelists were unable to reach an agreement on it, and it is not present in the final METRICS tool. The RQS, on the other hand, places a strong emphasis on prospective studies, particularly those registered in trial databases, and awards the studies with the highest score of the tool for this item [START_REF] Lambin | Radiomics: the bridge between medical imaging and personalized medicine[END_REF]. Based on the feedback of the panelists during Round#1 of Stage#3, the most likely reason for this would be that radiomics research requires large data sets, which are difficult to achieve with prospective studies when compared to retrospective design and data sets. Another issue raised by panelists was the potential penalization of large retrospective data sets in comparison to prospective studies with small data sets. Therefore, despite its undoubtedly high importance in clinical research, the role and added value of prospective data collection currently remain uncertain in radiomics and artificial intelligence research within the medical imaging domain, and could be secondary compared to other considerations on overall data labeling and management as established by the METRICS expert panel. It would be worthwhile to receive community feedback on this and other topics in the future, which may contribute to future revisions of METRICS.

Our work has several distinguishing features and strengths. First, the weights obtained in this work were not assigned arbitrarily but were the result of expert ranking. This was a primary goal of the study as there has been no previous work on radiomics quality scoring that has presented a transparent methodology for assigning item weights. Second, the METRICS tool considers not only hand-crafted radiomics but also deep learning-based radiomics. Third, both Group#1 and Group#2 had a large number of panelists. Furthermore, the panel was diverse in terms of country and domain expertise. This was necessary to reduce noise in calculations. Fourth, panelist participation in the Delphi rounds was also very high, with a minimum of 95% (40 of 42). Fifth, we created an easy-to-use web application to streamline scoring. This was crucial because METRICS contains conditional items that cover all aspects of radiomics, which may make the calculation difficult on paper. Finally, we established a living repository to discuss the METRICS tool and its content and receive feedback in order to improve them in the future.

There are however several limitations to declare. First, our modified Delphi procedure was not completely anonymous and the steering committee had access to identities, which was a significant deviation from the standard Delphi exercise. We chose this approach to ensure panelist participation. Nevertheless, we kept the votes and comments anonymous for other panelists. Second, the ranking in Round#4 of Stage#3 did not account for potential items of equal importance. An analytical hierarchy process and pairwise voting could have been an alternative approach that takes equality into account. However, by this method, the number of questions would have been doubled in Round#4, which might cause fatigue and had negative effects on the scoring process. Third, during tool development, the need for conditional items became apparent, even if their use may complicate the scoring process. In reality, radiomic research involves numerous methodological variations and nuances that could be overlooked with a fixed item list. However, the availability of online and offline automated calculation tools should help mitigate this limitation. Fourth, the conditionality of the items or categories was not taken into account when calculating weights. Dynamic weights would have necessitated calculations of all possible conditional combinations and, as a result, multiple rankings, which is impractical and of limited value as differences are expected to be small compared to the current METRICS tool. Fifth, the number of items in each category varied. Nonetheless, the weighting process accounted for this to avoid biases in the final tool due to item number within categories. Sixth, the order of the items and categories in the Delphi rounds was fixed, which may have an influence on ranking and introduce bias. Alternatively, the order of these could have been randomized during voting, and this could have been done independently for each panelist as well. Finally, the reproducibility of the METRICS was not evaluated. Such an analysis necessitates a dedicated study design by incorporation of other tools for comparison, which should be performed in a future investigation.

In conclusion, we developed a scoring tool for a comprehensive assessment of the methodologic quality of the radiomics research, i.e., METRICS, with a large international panel of experts and by using a modified Delphi protocol. With its flexible format to cover all methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of the radiomic research papers. A web application was developed to help with the calculation of the METRICS score, and a repository was created to collect feedback from the radiomics community. We hope that the researchers would benefit from this tool when designing their studies, assessing the methodological quality of papers in systematic reviews, and that journals would adopt the METRICS quality scoring tool for peer review. Comments and contributions to this tool are welcome through its repository to improve it in the future.

Tables Table 1: METRICS tool. 1 Conditional for studies including region/volume of interest labeling. 2 Conditional for studies using fully automated segmentation. 3 Conditional for the hand-crafted radiomics. 4 Conditional for tabular data use. 5 Conditional on the use of end-to-end deep learning. 6 Score is simply the weight if present and 0 otherwise. Proposed total score categories: 0-20% = very low, 21-40% = low, 41-60% = moderate, 61-80% = good, and 81-100% = excellent.

Categories

No. Use of uni-parametric imaging or proof of its inferiority >>> Use of a single imaging set (such as a single MRI sequence rather than multiple, or a single phase in a dynamic contrast-enhanced scan) should be preferred, as multi-parametric imaging may unnecessarily increase data dimensionality and risk of overfitting. Therefore, in the case of multi-parametric studies, uni-parametric evaluations should also be performed to justify the need for a multi-parametric approach by formally comparing their performance (e.g., DeLong's or McNemar's tests). This item is also intended to reward studies that experimentally justify the use of more complex models compared to simpler alternatives, in regard to input data type.

Items

#24

Comparison with a non-radiomic approach or proof of added clinical value >>> Whether a non-radiomic method that is representative of the clinical practice is included in the analysis for comparison purposes. Non-radiomic methods might include semantic features, RADS or RECIST scoring, and simple volume or size evaluations. If no nonradiomics method is available, proof of improved diagnostic accuracy (e.g., improved performance of a radiologist assisted by the model's output) or patient outcome (e.g., decision analysis, overall survival) should be provided. In any case, the comparison should be done with an appropriate statistical method to evaluate the added practical and clinical value of the model (e.g., DeLong's test for AUC comparison, decision curve analysis for net benefit comparison, Net Reclassification Index). Furthermore, in case of multiple comparisons, multiple testing correction methods (e.g., Bonferroni) should be considered in order to reduce the false discovery rate provided that the statistical comparison is done with a frequentist approach (rather than Bayesian).

#25

Comparison with simple or classical statistical models >>> Whether a comparison with a simple baseline reference model (such as a Zero Rules/No Information Rate classifier) was performed.

Use of machine learning methods should be justified by proof of increased performance. In any case, the comparison should be done with an appropriate statistical method (e.g., DeLong's test for AUC comparison, Net Reclassification Index). Furthermore, in case of multiple comparisons, multiple testing correction methods (e.g., Bonferroni, Benjamini-Hochberg, or Tukey) should be considered in order to reduce the false discovery rate provided that the statistical comparison is done with a frequentist approach (rather than Bayesian).

Figure 4 .

 4 For categories, the highest and lowest weights belonged to study design and open science, respectively.

  Round#1, Round#2, and Round#4 of the Stage#3 are presented in Supplementary file 2.

Figures

  Figures

Figure 1 :

 1 Figure 1: Key steps in the development of METRICS. Boxes related to stages and rounds are color-coded based on the main group of panelists involved. Dotted lines indicate the participation of organizers in the discussions in the relevant rounds as panelists. *Including organizers (i.e., steering committee members).

Figure 2 :

 2 Figure 2: Country of panelists. a, World map for distribution of 59 panelists including three organizers by country. b, Countries by groups. Group#1, EuSoMII auditing group including three organizers; Group#2, voters participated in Round#1, Round#2, and Round#4 of Stage#3. In case of multiple countries, the country of the first affiliation was considered.

Figure 3 :

 3 Figure 3: Rates from modified Delphi Round#1 and Round#2 of Stage#3. The number of the items matches those of the final METRICS tool. Item#X, i.e., prospective data collection, stands for the excluded item from the final METRICS tool. Please note Item#17 is missing in Round#1, which is the proposed item in Round#1 to be voted in Round#2.

Figure 4 :

 4 Figure 4: Weights of METRICS categories and items.

Figure 5 :

 5 Figure5: Use of conditions for the "Segmentation" section. Please note, the term "segmentation" is referred to either fine (e.g., semantic or pixel-based) or rough (e.g., cropping or bounding box) delineation of a region or volume of interest within an image or image stack for model training or evaluation. Studies can also be performed without such annotations, for example, using class labels that are assigned either to the entire image, volume, exam or patient or with unsupervised approaches that require no labeling at all (e.g., clustering models).

Figure 6 :

 6 Figure 6: Use of conditions related to the sections "Image Processing and Feature Extraction'' and "Feature Processing". Please note the flowchart assumes a single pipeline is used in a given study. However, different techniques might coexist in a single study. For instance, a study might include both hand-crafted feature extraction and end-to-end deep learning for comparison purposes, in such a case, all conditions can be selected as "Yes".

  Whether the model is tested on an independent data set that is sampled from the same source as the training and/or validation sets. 0.037

  confounding factors were analyzed, identified if present, and removed if necessary (e.g., if it has a strong influence on generalizability). These may include different distributions of patient characteristics (e.g., gender, lesion stage or grade) across sites or scanners. Use of appropriate performance evaluation metrics for task >>> Whether appropriate accuracy metrics are reported, such as AUC for Receiver Operating Characteristics (ROC) or Precision-Recall (PRC) curves and confusion matrix-derived accuracy metrics (e.g., specificity, sensitivity, precision, F1 score) for classification tasks; MSE, RMSE, and MAE for regression tasks. For classification tasks, the confusion matrix should always be included, to allow the calculation of additional metrics. If training a DL network, loss curves should be presented. measures are included in the analysis, such as 95% confidence interval (CI), standard deviation (SD), or standard error (SE). Report on methodology to derive that distribution (ie. bootstrapping with replacement, etc).

			instance, in the case of large slice thickness (e.g., ≥5 mm), extreme and a resampling technique (e.g., cross-validation) is used instead, test
			upsampling (e.g., 1 x 1 x 1 mm3) of the volume might be inappropriate. data should always be handled separately from this.
		#19	In such a case, 2D feature extraction could be preferable, ensuring in-plane isotropy of the pixels. On the other hand, achieving isotropic Handling of confounding factors >>> Whether potential 0.030
			voxel values should be targeted in 3D feature extraction, to allow for
			texture feature rotational invariance. Also, whether gray level
			discretization parameters (bin width, along with resulting gray level
			range, or bin count) are described in full detail. Description of different
	Metrics and	#20	image types used (e.g., original, filtered) should also be included (e.g.,	0.035
	Comparison		high and low pass filter combinations for wavelet decomposition, sigma values for Laplacian of Gaussian edge enhancement filtering). If the
			image window is fixed, it should be clarified.
		#12	Use of standardized feature extraction software 3	0.031
			>>> Whether a standardized software (e.g., compliant with IBSI) was
			used for feature extraction, including information on the version
			number.
		#13	Transparent reporting of feature extraction parameters, otherwise	0.041
		#21	providing a default configuration statement >>> Whether feature types (e.g., hand-crafted, deep features) and Consideration of uncertainty >>> Whether uncertainty 0.023
			feature classes (for hand-crafted) are described. Also, if a default
			configuration statement is provided for the remaining feature extraction
			parameters. A file presenting the complete configuration of these
		#22	settings should be included in the study materials (e.g., parameter file Calibration assessment	0.044 0.018
			such as in YAML format, screenshot if a dedicated file for this is not >>> Whether the final model's calibration is assessed.
		#5 #23	Clinical translatability of the imaging data source for radiomics analysis available for the software). In the case of DL, neural network architecture along with all image operations should be described.	0.029
	Feature	#14	>>> Whether the source of the radiomics data is an imaging technique Removal of non-robust features 4	0.020
	Processing		that reflects established standardization approaches, such as
			acquisition protocol guidelines (e.g., PI-RADS specifications).
		#6	Imaging protocol with acquisition parameters	0.044
			>>> Whether the image acquisition protocol is clearly reported to
			ensure the replicability of the method.
		#7	The interval between imaging used and reference standard	0.029
			>>> Whether the time interval between the diagnostic imaging exams
		#15	(used as an input for the radiomics analysis) and the outcome Removal of redundant features 4	0.020
			measure/reference standard acquisition is appropriate to validate the >>> Whether dimensionality is reduced by selecting the more
			presence or absence of target conditions of the radiomics analysis at informative features such as with algorithm-based feature selection
			the moment of the diagnostic imaging exams. (e.g., LASSO coefficients, Random Forest feature importance),
	Segmentation 1	#8	Transparent description of segmentation methodology univariate correlation, collinearity, or variance analysis. The specific	0.034
			>>> Whether the rules or the method of the segmentation are defined methods used should be clearly presented, with specific results for
			(e.g., margin shrinkage, peri-tumoral sampling, details of segmentation each component in multi-step feature removal pipelines.
		#16	regardless of whether manual, semi-automated or automated methods Appropriateness of dimensionality compared to data size 4	0.030
			are used). In the case of DL-based radiomics, the segmentation can >>> Whether the number of instances and features in the final training
			refer to the rough delineation with bounding boxes or cropping the data set is appropriate according to the research question and
			image around a region of interest. modeling algorithm. This should be demonstrated by statistical means,
		#9	Formal evaluation of fully automated segmentation 2 empirically through consistency of performance in validation and	0.022
			>>> If a segmentation technique that does not require any sort of testing, or based on previous evidence in the literature.
		#17	human intervention is used, examples of the results should be Robustness assessment of end-to-end deep learning pipelines 5	0.020
			presented and a formal assessment of its accuracy compared to >>> Whether DL pipeline consistency of performance has been
			domain expert annotations included in the study (e.g., DICE score or assessed in a test-retest setting, for example by a scan-rescan
			Jaccard index compared with a radiologist's semantic annotation). Any approach, use of segmentations by different readers, or stability
			intervention to the annotation in terms of volume or area should be analysis [i.e., image perturbations]. The specific methods used should
			considered as the use of a semi-automated segmentation technique. be clearly presented.
	Preparation for	#18	This item also applies to the use of segmentation models previously Proper data partitioning process	0.060
	Modeling		validated on other datasets. >>> Whether the training-validation-test data split is done at the very
		#10	Test set segmentation masks produced by a single reader or beginning of the analysis pipeline, prior to any processing step. Data	0.011
			automated tool split should be random but reproducible (e.g., fixed random seed),
			>>> Whether final segmentation in the test set is produced by a single preferably without altering outcome variable distribution in the test set
			reader (manually or with a semi-automated tool) or an entirely (e.g., using a stratified data split). Moreover, the data split should be on
			automated tool, to better reflect clinical practice. the patient level, not the scan level (i.e., different scans of the same
	Image	#11	Appropriate use of image preprocessing techniques with transparent	0.062
	Processing and		description
	Feature		>>> Whether preprocessing of the images is appropriately performed
	Extraction		considering the imaging modality (e.g., gray level normalization for
			MRI, image registration in case of multiple contrasts or modalities) and
			feature extraction techniques(i.e., 2D or 3D) that are used. For

>>> Whether unstable features are removed via test-retest, reproducibility analysis by analysis of different segmentations, or stability analysis [i.e., image perturbations]. Instability may be due to random noise introduced by manual or even automated image segmentation or exposed in a scan-rescan setting. The specific methods used should be clearly presented, with specific results for each component in multi-step feature removal pipelines. patient should be in the same set). Proper data partitioning should guarantee that all data processing (e.g., scaling, missing value imputation, oversampling or undersampling) is done blinded to the test set data. These techniques should be exclusively fitted on training (or development) data sets and then used to transform test data at the time of inference. If a single training-validation data split is not done

CEO and shareholder of Quibim SL. Editorial Board Member of Insights into Imaging. EK: Speaker fees for Siemens Healthineers, Speaker fees for abbvie, member of the scientific advisory board and shareholder of contextflow GmbH, Vienna. Member of the scientific advisory board Gleamer. RoC: Support for attending meetings from Bracco and Bayer; research collaboration with Siemens Healthcare; co-funding by the European Union -FESR or FSE, PON Research and Innovation 2014-2020 -DM 1062/2021. Editorial Board Member of Insights into Imaging. LSF: General Electric Healthcare (Honoraria), Median Technologies (Honoraria), Sanofi (Honoraria), Guerbet (conference funding), Bristol-Myers Squibb (research grant). MEK: Meeting attendance support from Bayer. LMB: Editor-in-Chief of Insights into Imaging, member of the non-profit Scientific Advisory Boards of Quibim SL and the Girona Biomedical Research Institute. DPdS: Editorial Board Member of Insights into Imaging. Speaker fees for Bayer AG, Advisory Board for cook medical, Author fees for AMBOSS GmbH. AP: Editorial board member of Insights into Imaging. FV: None related to this study; received support to attend meetings from Bracco Imaging S.r.l., This study was

Data generated or analyzed during this study are presented with this manuscript .

Data availability 0.007 #29 Code availability 0.007 #30 Model availability 0.007 Total METRICS score (should be given as percentage) Data availability 0.007 #29 Code availability 0.007 #30 Model availability 0.007 Total METRICS score (should be given as percentage)

Electronic Supplementary Materials

Supplementary File 1: Rank statistics. Supplementary file 3: METRICS tool without explanations. 1 Conditional for studies including region/volume of interest labeling. 2 Conditional for studies using fully automated segmentation. 3 Conditional for the hand-crafted radiomics. 4 Conditional for tabular data use. 5 Conditional on the use of end-to-end deep learning. Proposed total score categories: 0-20% = very low, 21-40% = low, 41-60% = moderate, 61-80% = good, and 81-100% = excellent.

Categories

No.

Items

0.007

Total METRICS score (should be given as percentage) 1 Conditional for studies including region/volume of interest labeling. 2 Conditional for studies using fully automated segmentation. 3 Conditional for the hand-crafted radiomics. 4 Conditional for tabular data use. 5 Conditional on the use of end-to-end deep learning. 6 Score is simply the weight if present and 0 otherwise. Proposed total score categories: 0-20% = very low, 21-40% = low, 41-60% = moderate, 61-80% = good, and 81-100% = excellent.

Supplementary file 5:

Evaluation examples for the demonstration of how to use METRICS. The table reports whether each of the example studies fulfill ("yes") or not ("no") the item requirements or if the item is not applicable to the study design ("n/a"), as appropriate. Please note that conditional item weights do not influence the maximum obtainable total score in case of non-applicability.

Items


Laura Willot
email: laura.willot@cyu.fr

Dan Vodislav

Valerie Gouet-Brunet

Livio De Luca

Adeline Manuel

Clustering for the Analysis and Enrichment of Corpus of Images for the Spatio-temporal Monitoring of Restoration Sites

Keywords: Information systems → Clustering and classification, Similarity measures, • Computing methodologies → Matching Cultural Heritage, Semantic analysis, Multimodality, Data enrichment

come   L'archive ouverte pluridisciplinaire

INTRODUCTION 1.Context

On 15 April 2019, a fire started in the Notre-Dame de Paris cathedral, destroying most of its roof when the spire collapsed. At first, generating a deep emotion from people all around the world, it quickly became an unprecedented and unique opportunity for the scientific community to study the building.

The research group for the study of Notre-Dame de Paris has been organized around 9 thematic Working Groups (WGs) with more than 175 researchers from disciplines as various as archaeology, history, architecture, computer science, physics, chemistry, etc. These researchers produce a massive quantity of heterogeneous data which led to the introduction of the n-dame digital ecosystem [START_REF] De | ERC n-Dame_Heritage[END_REF] developed to bring together new tools and methods for the analysis of these data, with regards to their spatial, temporal and morphological characteristics [START_REF] De | Vers des biens communs numériques pour construire des cathédrales de données et de connaissances pluridisciplinaires sur le patrimoine européen[END_REF][START_REF] De Luca | L'écosystème numérique n-dame pour l'analyse et la mémorisation multidimensionnelle du chantier scientifique Notre-Dame-de-Paris[END_REF].

In this work, we intend to approach the problem of analyzing, structuring and enriching massive corpus of images of Cultural Heritage (CH) restoration sites, with the help of the construction of similarity links relying on different modalities and clustering methods, applied to all three dimensions identified here -temporal, spatial and semantic -and to their visual content.

Challenges

One of the definitions of Big Data involves three distinctive features: volume, velocity and variety. Although not applicable as such in the humanities (this is particularly true for the velocity property) [START_REF] Schöch | Big? Smart? Clean? Messy? Data in the Humanities[END_REF], this definition highlights three characteristics that can be found in the data produced by the researchers working on the study of Notre-Dame de Paris and represent different challenges to overcome.

First, the massive quantity of data (the volume property, relatively to what humanities produce) makes it nearly impossible for a human being to comprehend and analyse all the data available. Automatic methods and tools are needed to allow any user to have a clear picture of what is the data and how each entry interacts with the others. Second, the continuous production of data (even though not as constant and instantaneous as in other domains, this production can be considered as a reduction of the velocity property to its incremental nature) needs to be addressed by the implementation of methods that can be applied to an ever-increasing amount of data. And third, the data produced is particularly heterogeneous (the variety characteristic) and of a 'real world' nature: each sample, description, image, etc. is a human-produced element in the specific context of a restoration site, in our case the restoration of Notre-Dame de Paris. Hence the common digital framework the Digital Data WG is working on [START_REF] De | ERC n-Dame_Heritage[END_REF].

Objectives

This work focuses on the photographs acquired for the spatiotemporal monitoring of the cathedral's restoration. This specific item has been chosen for its inherent transversal quality, through time and disciplines. Indeed, many pictures have been taken before and after the fire, allowing a comparison between the different states of the cathedral, including the evolution of the restoration work done in the building. Furthermore, images are the carrier for many more information than the plain visual content they bare: scientists use pictures to keep track of samples' extraction sites, to identify the differences and similarities between several elements of a similar nature, across time and space, etc. Hence, images record activities, observations, steps of workflows or documentation. Exploring the potential of this implied information is at the core of this study. We want to make this information explicit or at least to uncover the potential in terms of analysis and processing for enrichment.

In that respect, images can be enriched through the use of diverse metadata, falling under one of the following dimensions: time, space and semantics. Thus, we need to consider not only the visual content of the images, but all associated metadata that allow us understand more than what the plain visual content of the images. With an appropriate common framework structuring the data, it is possible to develop and integrate automatic methods to ease the analysis and enrich the corpus.

We will shortly introduce work related to the structuring and analysis of massive corpus of heterogeneous CH data in Section 2. Then, we will present in Section 3 the specific features of the data we focus on. Sections 4 and 5 address the methods used to analyse our data -namely, clustering methods -and the results we obtained. Finally, we conclude and present our future work in Section 6.

STATE OF THE ART

The documentation of CH objects [START_REF] Letellier | Recording, Documentation and Information Management for the Conservation of Heritage Places: Illustrated Examples[END_REF] and processes [START_REF] Carboni | Data Provenance in Photogrammetry through Documentation Protocols[END_REF] is a major concern for the CH community, especially as the volume of data increases. A well-documented digital CH item can be easily found and used, which is essential for a good data management. However the task of manually adding metadata to the assets can be timeconsuming, hence the development of automatic processes for CH data enrichment. These methods can be divided into two groups: the learning-based techniques and the 'handcrafted' ones.

Several of the methods imply the use of Machine Learning (ML) or Deep Learning (DL) to enrich datasets, whether supervised (with pre-determined labels) or unsupervised (without those labels). The literature on the subject is abundant. [START_REF] Rei | Multimodal metadata assignment for cultural heritage artifacts[END_REF] introduces and adapts a multimodal ML approach to predict some chosen properties for digital artefacts. [START_REF] Belhi | Leveraging Known Data for Missing Label Prediction in Cultural Heritage Context[END_REF] proposes a classification and annotation method based on Convolutional Neural Networks (CNN) for visual features learning subsequently merged with textual metadata. [START_REF] Llamas | Classification of Architectural Heritage Images Using Deep Learning Techniques[END_REF] presents as well an application of CNNs for the automatic classification of images of architectural heritage in order to help researchers and historians in the documentation process. Nevertheless, these supervised, learning-based approaches require efforts and a predefined list of labels to be applied to the images. This can be proved very inefficient when dealing with data still unexplored. Therefore, data exploration with unsupervised tools is a necessary first step to understand the data.

Clustering, generally considered as a specific case of classification, is an unsupervised task used to attribute unknown labels to the elements. The labeling is done with regards to a certain similarity shared among the elements [START_REF] Saxena | A review of clustering techniques and developments[END_REF]. This type of task can be of great use to gather some understanding of the data in the early stages of exploration [START_REF] Richard O Duda | Chap 11: Unsupervised Learning and Clustering[END_REF] especially as clustering is often expected to produce interpretable results [START_REF] Rai | A Survey of Clustering Techniques[END_REF]. As highlighted by [START_REF] Rai | A Survey of Clustering Techniques[END_REF][START_REF] Saxena | A review of clustering techniques and developments[END_REF][START_REF] Xu | Survey of clustering algorithms[END_REF], because of a lack of consensus on the definition of 'cluster', several classifications and taxonomies of clustering methods have been developed and express the variety of techniques that can be used to create clusters. The choice of technique depends on both the type of data available and the expected results. Therefore, a good understanding of the characterization of the data at hand is necessary before turning to an exploration of the relations between these data.

DATA: IMAGE CHARACTERISTICS

In this section, we propose to briefly inventory the data and metadata characterizing the images of the corpus, starting with their visual content and going on with the three other dimensions of interest. To introduce these data, we will begin with a description of the data sources for our case study.

Data Sources

All along their life cycle, from their production to their archiving, images are enriched through diverse methods (for instance, manual tagging to describe an image content with keywords).

In our case study, most of these data come from Aïoli, "a realitybased 3D annotation platform, which allows a multidisciplinary community to build semantically-enriched 3D descriptions of heritage artefacts from simple images and spatialized annotations" [START_REF] Abergel | Aïoli: A reality-based 3D annotation cloud platform for the collaborative documentation of cultural heritage artefacts[END_REF]. In this platform, a user can create a project, usually limited to a single object or an area of interest, from a set of pictures acquired following a photogrammetric protocol. From these pictures, a point cloud is reconstructed and presented in a 2D/3D viewer, along with the spatialized pictures whose position and orientation are now known. The user can then annotate any picture by drawing a region of interest on it which is then re-projected on any other image on which this region appears [START_REF] Abergel | Aïoli: A reality-based 3D annotation cloud platform for the collaborative documentation of cultural heritage artefacts[END_REF]. An Aïoli annotation is composed of user-defined elements: the region drawn on the image, a name, and a description sheet. The name belongs to a hierarchy created by the user to organise the different regions of a project in different categories. The description sheet is generated by various contents (such as files, free texts, etc.) chosen by the user to contextualise the regions. Figure 1 presents these elements.

In our case study, the regions of interest and related descriptions (ie, the annotations and their images) come from the digital version of the condition report provided by professionals in the field and were digitized in Aïoli [START_REF] Roussel | An Approach to Build a Complete Digital Report of the Notre Dame Cathedral after the Fire, using the Aïoli Platform[END_REF]. Thus, the annotation tool provides both images and qualitative contextualized information in the form of experts' annotations. The resulting dataset reflects the practice and knowledge in the field of conservation and restoration.

Image characteristics

Visual Content.

The visual content of an image is the first accessible data when no metadata is available. It is a low level data that can be automatically analyzed with many approaches coming from image analysis, indexing, computer vision and more generally AI. Note that we can analyze and describe the whole content of the image as well as focus on parts or details.

The visual content is not always enough to describe the images and build a structured corpus of images. Thus, all metadata associated with the images (as presented in the following subsections) is a valuable information that can be used, sometimes in conjunction with the visual content, to deduce new information from the study of the corpus of images.

Time.

Any image possesses a date of acquisition resulting from the acquisition itself. All images in the corpus is therefore part of a timeline including all known acquisition dates for all these images (Figure 2). An acquisition date is usually composed of a day and an hour of acquisition, allowing different degrees or levels of analysis: all images taken the same day can be compared to each other, with regards to their hours of acquisition. But the days of acquisition inscribe the images in a larger temporality: one can consider days, weeks or even months to be a more appropriate scale to compare the images to each other. The periods can be identified as specific activities that took place in the cathedral: for instance, a picture of the nave taken during the process of cleaning the space has a date (and time) of acquisition placing this image in the specific day it was taken, but also in the period during which various people came to clear the floor of the nave by recovering the debris that littered it.

Space.

The spatial dimension may be represented in different ways; in our case, each image can be associated to a 6-dimensional vector (3 values for the position and 3 for the orientation) defined in a reference system common to all the images in the corpus. It is possible to consider the image by itself, and thus, get an idea of where it was taken, but it is also possible to look at its position in the corpus, that is, to compare it with the other images' positions. It is also interesting to consider the scale of the restoration project: as the building is particularly huge, it has been "divided" into several spaces (such as the nave, the choir, etc.) which are in turn divided into smaller spaces (Figure 3).

Semantics.

Here, the semantic dimension is understood as any string of characters associated with an image describing or enhancing it.

As explained in Section 3.1, we consider the semantic data coming from the Aïoli database from which we identified two types of textual data:

• free descriptive texts: user-defined texts from the description sheet associated to a region drawn by the user (Figure 1 (d)) • hierarchical categorization: the tree-like structure of data to which belong the name of the annotation (Figure 1 (c))

Similarly to the temporal and spatial dimensions, different scales of semantic data can be explored. First, descriptive texts are composed of single words giving insights on what's really about the image. But those words can also be part of structures such as controlled vocabularies, glossaries, thesauri and ontologies. These different The semi-structured hierarchical categorization of the regions is another interesting data structure to characterize the images and allows different entry point to explore the data related to an image: one could be interested in the leaf node level or any higher level in the hierarchy.

METHODS: CLUSTERING

In this section, we present a selection of clustering methods based on the different modalities chosen to explore the corpus of images. These well-proven methods serve as illustrations of image clustering for spatio-temporal monitoring of restoration sites.

Visual Clusters

The aim of visual clustering is to group images by visual content similarity (Figure 4) to directly create links between the images or to be the basis of further analysis of the other modalities. The data considered is the image itself.

Several algorithms and methods have been developed throughout the years to match images based on their content. Some of these techniques rely on learning (for instance, DELF [START_REF] Noh | Large-Scale Image Retrieval with Attentive Deep Local Features[END_REF]) whereas others are considered "hand-crafted": SIFT [START_REF] David | Distinctive Image Features from Scale-Invariant Keypoints[END_REF], SURF [START_REF] Bay | SURF: Speeded Up Robust Features[END_REF] or ORB [START_REF] Rublee | ORB: An efficient alternative to SIFT or SURF[END_REF], to name but a few. These image matching algorithms and techniques can be used to create clusters of images sharing similar visual contents.

However, as the learning-based methods might not work as well on our dataset because of the specificity of our domain (CH restoration sites), for this exploratory work we decided to focus on hand-crafted methods: the SIFT and ORB algorithms (Figure 5). Clusters can subsequently be created based on the similarity shown by the matching keypoints between each pair of images.

Temporal Clusters

Clustering the images by date of acquisition groups images close in time even though they may not share the same visual features.

To achieve this temporal clustering, we consider the day and hour of acquisition of each image. A quick visualization of the data (as in Figure 6) allows us to get an idea of the clusters that can be identified. As a one-dimensional variable, it may not be necessary to use powerful methods such as k-means and related n-dimensional clustering techniques. However, a simple histogram technique is not suitable either as the bins' width is constant whereas we need to be able to create different-sized bins. Thus, we decided to use a visualisation tool for a first exploration of the temporal clusters that could be created with our data. RawGraphs [START_REF] Mauri | RAWGraphs: A Visualisation Platform to Create Open Outputs[END_REF] offers several charts for data with a temporal dimension. A beeswarm plot displays items as dots over a horizontal line and uses the vertical space to avoid overlaps. The clusters are then deduced from a visual inspection of the resulting chart.

Spatial Clusters

The aim of spatial clustering is to group images together by geographical proximity, and for this purpose we have access to the position and orientation of the images.

Based on the division of the cathedral into spaces, we chose to first study the position of the images, regardless of their orientation. Using a Principal Components Analysis as a visualization tool to empirically determine clusters, the data (𝑥, 𝑦, 𝑧 coordinates of the images) is first centered and reduced. Thus, we get a preliminary visualization of the principal components of the position data (Figure 7). As for the temporal clustering, the spatial clusters are determined from a visual inspection of the resulting PCA.

Semantic Clusters

Semantic clustering aims to highlight the similarities between images that might not be visible directly on the images: in the context of a restoration site, two images might present different artefacts with the same types of alterations and still be visually dissimilar.

For this exploratory semantic clustering, we focused on the semistructured hierarchical categorization of the regions drawn on the images, that is any sub-tree of the general tree of alteration patterns presented on Figure 9 (examples of sub-trees are given in Figure 11). Therefore, the resulting clusters take into account only part of the semantic data available in Aïoli. But as a complex data structure used to characterize an image, it is a good entry point to study semantic clusters.

To create clusters based on these hierarchical categorizations, a tree-like data structure (a custom Python class) is built for each image. These trees are then compared with a Tree Edit Distance (TED), an adaptation of the string edit distance to ordered labeled trees [START_REF] Tai | The Tree-to-Tree Correction Problem[END_REF]. The editions considered are: a label modification, a node insertion, and a node deletion. Various algorithms have been developed to compute this TED [START_REF] Bille | A survey on tree edit distance and related problems[END_REF]. We decided to use the Python implementation of the APTED algorithm [START_REF] Pawlik | Efficient Computation of the Tree Edit Distance[END_REF][START_REF] Pawlik | Tree edit distance: Robust and memory-efficient[END_REF] for its efficiency and ease of use. Each tree is thus compared to all other trees to create a 𝑛 * 𝑛 distance matrix, 𝑛 being the total number of images. Each coefficient of the matrix is the TED computed between the trees (hierarchical categoriization) of a pair of images. This matrix is then used to create the clusters with Agglomerative Clustering from the Scikit-learn library [START_REF] Pedregosa | Scikit-learn: Machine Learning in Python[END_REF], a hierarchical clustering method with a "bottom-up" approach [START_REF] Rai | A Survey of Clustering Techniques[END_REF]. This algorithm takes several parameters in input: a number of clusters and a linkage method (ie: the distance to use between sets of observation). We decided to limit the number of clusters to the number of projects because a smaller number would likely not be enough to differentiate the images on this criterion, and a higher one would probably create too many groups and hinder the purpose of creating links between the pictures. Following the literature [START_REF] Angur | Hierarchical Cluster Analysis: Comparison of Single linkage,Complete linkage, Average linkage and Centroid Linkage Method[END_REF], we chose the most used linkage method, namely the average one.

EXPERIMENTS & RESULTS

In this Section, we aim to show how the different clustering approaches, used to explore the corpus, can help create an understanding of the data and gather new information.

Dataset

After three years of work, 240 projects -as defined in the Aïoli platform (cf. Section 3.1) -have been created with more than 9,000 annotations with an average of 49 annotations per annotated project [START_REF] Roussel | An Approach to Build a Complete Digital Report of the Notre Dame Cathedral after the Fire, using the Aïoli Platform[END_REF]. From these data, we selected 5 representative projects and created a dataset with more than 1000 images and 300 distinct annotations. The statistics for this dataset are presented in the Table 1. This selection of data allowed us to experiment with the clustering analysis methods presented here, while reducing computing times.

Analysis of the clustering methods

Visual Clusters.

The image matching techniques, such as presented in Figure 5, show potential to create visual links between the images. Automating the creation of such links will provide a strong basis to build upon, especially when considering the other dimensions. Indeed, the exploration of visual (dis)similarity can improve the analysis of temporal, spatial, and semantic clusters by providing low-level information on the images grouped together. By taking advantage of the visual links created between the images, an enrichment process can be set up to propagate data from image to image and contribute to an image-level enrichment of pictures with no or few metadata.

On a corpus-level enrichment, preliminary analysis of temporal, spatial, and semantic clusters is necessary to determine where they benefit from these visual links. Hence, the analysis of the other clustering methods that follows.

Temporal Clusters.

The visual analysis of the Beeswarm plot presented in Figure 2 shows three main clusters roughly corresponding to the months of February 2019, July 2019 and around June/July 2020. The colors correspond to four different acquisition phases of the cathedral's spaces, two of which took place in a few days in July 2020 (on the 9 th and 11 th ). From what we know (cf. Table 1), we can also see that there were probably two major phases for the acquisitions that took place after the fire: the first one in 2019 for the outside, and the second one in 2020 for the interior spaces. This actually corresponds to a practical reality: right after the fire, the inside of the cathedral was not accessible due to the debris that fell into the nave. Thus, the acquisition process mainly focused on the areas outside the cathedral at that time, before the indoor spaces were cleared and workers were allowed inside.

Based on the dates, we can infer new information: periods of time separated by events such as the start or ending of an acquisition process. We can also consider the different scales of time: one could be interested in the daily acquisition processes whereas another might want to get a more global view of the recording of the cathedral on a monthly or even yearly scale, for instance. These information can then be used to enrich the corpus of images by characterizing their temporality with higher-level concepts.

As for now, this clustering analysis is just visual and only validated by what people know of the worksite, but it could be improved by automatic methods to precisely define the boundaries of the clusters. Furthermore, these dates cover only the acquisition process and no physical changes that might occur in the cathedral. Thus, it could be interesting to cross this modality with the spatial and visual aspects of the images to determine whether a modification of the spaces happened between the acquisitions of two images. This change might denote that a new cluster boundary is needed and thus create a new level of clusters.

Spatial Clusters.

As shown on Figure 7, the PCA provides an illustration of the potential clusters that can be drawn from the analysis of the positions (𝑥, 𝑦, 𝑧 coordinates of the images). We can visually distinguish four main clusters, validated by prior knowledge of the corpus.

Cluster 𝐵 could be further divided into three smaller clusters. These images are part of the project "IE_NF_CB_ExtradosEst" (the extrados of the vaults) and the three sub-clusters actually correspond to the lines followed by the rope access technicians who carried out the acquisitions of this type of difficult-to-access spaces (Figure 8). This is a peculiarity that is not seen in the other projects and differentiates it from these.

Similarly, cluster 𝐶 encompasses both of the exterior projects: E_NF_TA_TAC_CouverturesNord (the roof covering between the aisles 29 an 39) and E_NF_T31_ContrefortF31 (the buttress F31 along the aisle 31). The pictures of these two projects were actually taken in the same zone of the cathedral: the buttress F31 is included in the space between the aisles 29 to 39. This is an illustration of the complexity of the structuring of the spaces of the cathedral and how they are related to each other. In this example, the PCA highlights the closeness between several images which are indeed located in the same range.

Thus, from the images' positions, we can deduce new information in the form of clusters that can be structured into several layers of hierarchy: levels of spaces included in one another. Those spaces can then be used to characterize the images by providing higher-level information tan their spatial coordinates.

Semantic Clusters.

For the purpose of this demonstration we selected a few annotations from the condition report and aligned the expressions used in this report with a few terms defined in glossaries of stone alteration phenomena and sources [START_REF] Bromblet | Guide "Altérations de la pierre[END_REF][START_REF]EwaGlos -European Illustrated Glossary of Conservation Terms for Wall Paintings and Architectural Surfaces[END_REF] (Figure 9). The statistics for this sample of the whole dataset is presented in the Table 2.

As explained in Section 4.4, we only considered the hierarchical categorisation of the regions drawn on the images. This gave us some preliminary results to build upon (Figure 10). The images of a given project mostly belong to several clusters, indicating a variety of alteration patterns in a given space of the cathedral. Likewise, the clusters are mainly distributed over several projects: even though pictures are located in different parts of the cathedral (inside or outside the cathedral, in the nave or in the choir, at the roof level, etc.) or are visually dissimilar, they do share some alteration patterns (Figure 11). The first two columns indicate the name of the project and the day of acquisition (dd/mm/yy) of the images constituting the projects; the third column states whether the images were taken inside (I) or outside (E) the cathedral; the following two columns specify the location and type of space/element; then, a few statistics are given (total number of images, number of images with known acquisition time, number of images with known location, number of regions and number of images with at least one region) per project. In order to use the Agglomerative Clustering function, we had to set the number of clusters. This is not an easy choice, especially when the aim of this clustering is also to explore the data. One This is the global tree-like structure of the terms considered. Thus, hierarchical categorization trees of any image will be a sub-tree of this one potential solution to this could be to take advantage of the "bottomup" nature of this clustering algorithm and compute the whole dendrogram of clusters instead of stopping it when the given number of clusters is reached. This could be helpful for the analysis of the corpus and create further hierarchical knowledge about the images.

Another improvement to this exploration lies in the use of more semantic data. Taking into account the number and size of each type of regions per image to build the trees and compute the distance between two of them might help refine this distance. Indeed, two images might have similar trees (ie, the same types of regions appear on both pictures) but one of the region type could be more numerous or bigger on one of the images, suggesting that the alteration is more present on this image than on the other one.

Global analysis

Due to time constraints, only prior knowledge has been used to evaluate the relevance of the visual and hierarchical clustering of the data. Nevertheless, these different methods of clustering show interesting results for each of the dimensions. They allow to discover new information that can be used to enrich the corpus of images, and can further be utilized to create and answer more complex queries. These exploratory tools are complementary and serve the same purpose, namely, the analysis and enrichment of the corpus. Such techniques highlight similarities between images that would not have been shown if the visual aspect had been considered alone.

From different types of values characterizing an image (dates, positions, annotations, etc.), it has been possible to create groups of similar images (sharing the same features) and thus, observe them along different axes, understand better their complexity and infer new information which, in turn, can be layered to depict the different scales that can be adopted to study the building. This is further noticeable with a hierarchical construction of the clusters: each level of clusters brings its own information on the images grouped together at that level.

On an image level (as opposed to a corpus level where we consider the information related to the whole corpus or groups of images) the visual links can be further taken advantage of, in conjunction with other types of clusters, to propagate data from image to image: when introducing a new image with no or few metadata to the corpus, the visual similarity between this new image and the ones in the corpus, organised in temporal, spatial and semantic clusters, can be used to deduce metadata to enrich the new image.

CONCLUSION & FURTHER WORK

This paper explored a corpus of images based on clustering methods. We showed that, through the clustering of different dimensions (temporal, spatial, semantic, and visual), it is possible to observe them along different axes, to understand better their complexity and then to infer new information to enrich the corpus of images. The analyses of the different dimensions were developed separately to evaluate their own potential to enrich the data. The visual analysis of the visual content, the acquisition dates of the images and the Principal Component Analysis (PCA) of the positions, as well as the agglomerative clustering based on a distance matrix of Tree Edit Distance (TED) computed on the annotations have composed this statistical exploration of the dataset. In order to automatize the data enrichment, the multimodal statistical analysis is a mandatory step to test and validate the workflow and results that will further be enriched by a validation method such as the Silhouette Score. Albeit low level data type, the images in the dataset function as a foundational layer: they confirm or infirm the results visually for the user. The next step is to go from multidimensionality (separate modalities) to combined variables (multimodality and fusion).

Figure 1 :

 1 Figure 1: Example of Aïoli data (a) types of regions appearing on the image (b) image from an acquisition process in one of the chapels with the regions in color (c) extract of the hierarchical categorization used to organize the annotations in this project (d) extract of the description sheet associated to the annotation CA19_ECAIL9

Figure 2 :

 2 Figure 2: Timeline of the date of acquisition of some of the images

Figure 3 :

 3 Figure 3: Spatialization of the images (a) example of an image's coordinates and location in one of the Aïoli chapel projects (b) location of the same project in the cathedral (in green)

Figure 4 :

 4 Figure 4: Example of visually alike images from the projects IE_CH_CA19 and IE_NF_CA25

Figure 5 :

 5 Figure 5: Examples of hand-crafted matching algorithms (left) SIFT (right) ORB The colored lines correspond to the pairs of matching keypoints

Figure 6 :

 6 Figure 6: Beeswarm plot of the images' dates of acquisitions (from February 2019 to July 2020)Each dot corresponds to an image with a known date of acquisition. The colors correspond to four of the five projects of the dataset (see Section 5.1 for more details on the selected projects).

Figure 7 :

 7 Figure 7: PCA on the 𝑥, 𝑦, 𝑧 coordinates of the images The colors correspond to the five projects for which the positions of the images are known

Figure 8 :

 8 Figure 8: Screenshot of the point cloud and spatialized pictures of the Extrados of the Vaults

Figure 9 :

 9 Figure 9: Stone alteration patterns

Figure 10 :Figure 11 :

 1011 Figure 10: Clusters of images based on hierarchical categorizationThe numbers of images are represented per project (lines), per cluster (columns)

Table 1 :

 1 Description of the dataset used for the experiments

	project name	acq day I/E broad loc	typology	#img #acq time #spatial #regions #annot im
	IE_CH_CA19	11/06/20	I	choir	chapel	93	93	93	92	93
	IE_NF_CA25	09/06/20	I	nave	chapel	94	94	94	106	94
	E_NF_T31_ContrefortF31	03/02/19 E	nave	buttress	16	16	16	31	16
	IE_NF_CB_ExtradosEst	--	I	nave	roof	443	0	443	84	433
	E_NF_TA_TAC_CouverturesNord 06/07/19 E	nave	roof terrace 596	200	596	18	593
						1242	403	1242	331	1229

Table 2 :

 2 Statistics for the selected regions of interest

		#regions #annot img
	total	331	1229
	selection	166	996

ACKNOWLEDGMENTS

This work benefits from a State aid managed by the French National Research Agency (Agence Nationale de la Recherche) under the future investment programme integrated into France 2030, with reference ANR-17-EURE-0021 -Paris Seine Graduate School of Humanities Creation Heritage (École Universitaire de Recherche Paris Seine Humanités, Création, Patrimoine) -Foundation for Cultural Heritage Science (Fondation des Sciences du Patrimoine) All pictures of the cathedral Notre-Dame de Paris presented in this work are credited under the © Chantier Scientifique Notre-Dame de Paris / Ministère de la Culture / CNRS.