Santiago Marro
email: santiago.marro@inria.fr

Theo Alkibiades Collias

Elena Cabrio
email: elena.cabrio@inria.fr

Serena Villata
email: serena.villata@inria.fr

On the Automatic Assessment of Natural Language Expert Explanations in Medicine

Keywords: NLP, Named Entity Recognition, Healthcare

The importance of explanations in decision-making, particularly in the medical domain, has been widely recognized. However, the evaluation of the quality of these explanations remains a challenging task. In this work, we propose a novel approach for assessing and evaluating the reasons provided in explanations about clinical cases. Our approach leverages an external knowledge base and a defined prevalence function to score each reason based on its pertinence in the domain. By applying a deterministic prevalence function, we ensure total transparency of the reasons' assessment, facilitating a precise explanation of the rationale behind the scoring hierarchy of each reason. We demonstrate the effectiveness of our approach in clinical cases, where medical experts explain the rationale behind a specific diagnosis and why other potential diagnoses are dismissed. Our methodology provides a nuanced and detailed evaluation of the explanation, contributing to a more comprehensive understanding of the decision-making process.

Introduction

Explainable Artificial Intelligence (XAI) [START_REF] Adadi | Peeking inside the black-box: A survey on explainable artificial intelligence (xai)[END_REF][START_REF] Došilović | Explainable artificial intelligence: A survey[END_REF] has emerged as a central topic in contemporary AI research, given the predominance of black box methods on the one hand, and their application to sensitive domains such as medicine and education on the other hand. AI systems support human decision-making like in medical diagnosis. Nonetheless, the efficacy of these systems is dependent on their capability to deliver explanations that are comprehensible and significant to the user [START_REF] Reiter | Natural language generation challenges for explainable ai[END_REF][START_REF] Fox | Argumentation-based inference and decision making-a medical perspective[END_REF]. Recent work shows how the best-known XAI approaches fail to provide sound explanations, or that alternatively find explanations that can exhibit significant redundancy [START_REF] Marques-Silva | Delivering trustworthy ai through formal xai[END_REF].

To address this challenging open issue, we propose a novel approach for an assessment and evaluation of the reasons employed in explanations, which satisfies transparency as well. More precisely, our goal is to automatically evaluate the relevance of all conceivable reasons that could explain a particular event, and subsequently compare them with the reasons invoked by the explainer.

When applied to the medical field, our approach scrutinizes explanations provided in medical examinations, wherein medical residents elucidate a specific diagnosis of a patient, given the context (i.e., a clinical case detailing the patient's condition) and their medical expertise. Consequently, we generate an assessment that identifies the reasons employed in the explanation and evaluates them against the relevance scoring produced by our approach. Our approach leverages an external knowledge base, the Human Phenotype Ontology (HPO) [START_REF] Köhler | The human phenotype ontology in 2017[END_REF], and a deterministic prevalence function to score each reason based on its pertinence in the domain. This function allows to elaborate the resulting reasons' scores, in a transparent way. We evaluate our approach on the Antidote Casimedicos dataset [START_REF] Agerri | Hitz@antidote: Argumentation-driven explainable artificial intelligence for digital medicine[END_REF], a unique resource comprising 621 clinical case descriptions, each with a set of potential diagnoses, an indicator of the correct answer, and a detailed explanation of the decision-making process provided by medical professionals. The results obtained on this dataset show the effectiveness of the proposed approach.

While our methodology is assessed on a use case from the medical domain, it is abstract enough to be applied to any domain. We envision two potential scenarios where our methodology could be particularly beneficial: AI for education and online medical fora.

In the context of AI for education, our approach can assist medical resident students in learning how to solve medical cases and develop a logical and explainable reasoning process in order to explain a diagnosis. By providing a systematic and transparent way of evaluating the reasons given in explanations, we can help students understand the rationale behind a specific diagnosis and why other potential diagnoses are dismissed.

In online medical fora, our approach can help online users to distinguish good explanations from bad explanations present there. Users often discuss diagnoses and share their experiences, but the quality of these discussions can vary widely. With our approach, we can provide a systematic and transparent way of evaluating the reasons given in these discussions, helping users and moderators identify high-quality explanations and promote more informed discussions.

The research presented in this paper is driven by the necessity for a systematic and transparent methodology to assess the pertinence of reasons used in medical explanations. To the best of our knowledge, this is the first approach that leverages an external knowledge base, the Human Phenotype Ontology (HPO), and a deterministic prevalence function to evaluate the reasons for the potential diagnoses based on their relevance in the context of a specific clinical case and grounding on the HPO knowledge base.

The paper is organized as follows: after a comparison with the related work, we first describe our data preprocessing heuristics (Section 3.1), followed by the extraction and encoding of reasons from the clinical case into HPO terms (Section 3.2). Next, we describe the computation of the pertinence score for each reason using the prevalence function (Section 3.3) and discuss its deterministic nature. Then, we illustrate the sentence-matching approach employed to align the extracted reasons with those found in the explanation (Section 3.5). Finally, we demonstrate the generation of a pertinence assessment of the reasons following a template-based generation technique (Section 3.6).

Related Work

In this section, we discuss the related work on explanation selection and then we focus on the medical domain application scenario.

Explanation Selection. In the process of explanation selection, individuals choose what they perceive to be the most relevant causes from a larger set of causes for a particular event. This selection is not arbitrary and is guided by criteria such as temporality, abnormality, intention, and the differences between a fact and a foil [START_REF] Miller | Explanation in artificial intelligence: Insights from the social sciences[END_REF]. Hilton [START_REF] Hilton | Social attribution and explanation[END_REF] sustains this is due to the fact that causal chains are often too large to comprehend.Research shows that the primary way individuals select explanations is by contrasting a fact and a foil. The fact refers to the actual state of affairs, while the foil represents an alternative state that did not occur. The contrast between the fact and the foil forms the basis for explanation selection, with the explanation that highlights the greatest number of differences between the fact and the foil deemed to have the highest explanatory power [START_REF] Hesslow | The problem of causal selection, Contemporary science and natural explanation: Commonsense conceptions of causality[END_REF]. Contrastive explanation is a concept that further elaborates on this idea. It posits that the differences between two events form the basis for explanation. This theory has garnered support from experimental research in cognitive science, which suggests that people perform causal inference, explanation, and generalization based on contrastive cases [START_REF] Rehder | A causal-model theory of conceptual representation and categorization[END_REF][START_REF] Rehder | When similarity and causality compete in category-based property generalization[END_REF]. Abnormality also plays a crucial role in explanation selection. Hilton and Slugoski [START_REF] Hilton | Knowledge-based causal attribution: The abnormal conditions focus model[END_REF] propose the abnormal conditions model, arguing that abnormal events are key in causal explanation. This model suggests that individuals use their perceived background knowledge to select conditions that are considered abnormal. This model has been supported by subsequent experimental studies [START_REF] Mcclure | Implicit and explicit processes in social judgments: The role of goal-based explanations[END_REF][START_REF] Samland | Do social norms influence causal inferences?[END_REF][START_REF] Hilton | The course of events: counterfactuals, causal sequences, and explanation[END_REF]. In this paper, we introduce an approach that not only evaluates the relevance of each potential explanation for a given event but also incorporates the principles of abnormality and contrastive explanation into the calculation of the relevance score.

XAI for the medical domain. The importance of explanations in AI systems, particularly in the medical domain, has been extensively studied [START_REF] Holzinger | Measuring the quality of explanations: the system causability scale (scs) comparing human and machine explanations[END_REF][START_REF] Panigutti | Doctor xai: an ontology-based approach to black-box sequential data classification explanations[END_REF][START_REF] Goebel | Explainable ai: the new 42?[END_REF]. In the context of medical diagnosis, explanations often involve identifying the key reasons or symptoms that led to a specific diagnosis. The Human Phenotype Ontology (HPO) [START_REF] Köhler | The human phenotype ontology in 2017[END_REF] provides a standardized vocabulary of phenotypic abnormalities encountered in human disease, which can be used to facilitate the assessment of explanations in this domain. Our work builds upon this ontology by developing an approach that assesses the selected reasons in explanations. The National Institutes of Health (NIH) Undiagnosed Diseases Program (UDP) [START_REF] Gall | Defining disease, diagnosis, and translational medicine within a homeostatic perturbation paradigm: The national institutes of health undiagnosed diseases program experience[END_REF] has also investigated the use of HPO in the context of diagnosing and evaluating patients with conditions that have eluded diagnosis. The clinical features of a patient are encoded into HPO terms, which are then used to retrieve a list of candidate diseases that might explain the patient's phenotype. This list is then examined by a clinician to identify the most likely diagnosis. Our methodology extends this approach by not only using HPO to facilitate diagnosis but also to evaluate the reasons given in explanations.

Assessing Reasons used in Explanations

Our approach to assessing the reasons used in explanations is visualized in Figure 1. We start with a clinical case of a patient, supplemented by an explanation provided by a medical expert, which elaborates on the specific diagnosis attributed to the patient. The objective is to evaluate the identified reasons in the clinical case and the external knowledge, and compare them with the reasons invoked by the expert to justify the medical diagnosis. To achieve this, we compute a pertinence score using a deterministic prevalence function (see Section 3.3), which ensures complete transparency, allowing us to explain why each reason is more or less pertinent than the others with respect to the given case.

The HPO Ontology [START_REF] Köhler | The human phenotype ontology in 2017[END_REF] and, more precisely, its associated knowledge base (KB) will serve as our external knowledge source by providing a standardized vocabulary of phenotypic abnormalities, namely symptoms and findings encountered in human diseases. The HPO contains over 13,000 terms describing phenotypic abnormalities seen in human disease. It uses a directed acyclic graph structure to represent the relationships between terms, allowing for flexible descriptions. Most terms have textual definitions and synonyms. The ontology terms describe clinical abnormalities at different levels of specificity, from general (e.g. Abnormal ear morphology) to very precise (e.g. Chorioretinal atrophy). This KB facilitates the evaluation of the reasons in the explanation by providing standardized vocabulary and semantic relationships between phenotypic concepts relevant to human disease.

Our approach consists of two main steps: (i) the reasons given in the explanation are extracted from the clinical case and encoded into HPO terms (following the approach of Marro et al. [START_REF] Marro | Natural language explanatory arguments for correct and incorrect diagnoses of clinical cases[END_REF]), in which a medical Named Entity Recognition (NER) step is performed, to then align them into HPO terms. This allows us to retrieve all the standardized information the ontology contains, such as the definition and the frequency of occurrence of that term in actual patient cases for each possible disease; (ii) the pertinence score for each reason is computed using the prevalence function, which takes into account the relevance of each reason in the context of the specific clinical case and the knowledge base.

Dataset and preprocessing

The foundation for our work is the Antidote Casimedicos dataset [START_REF] Agerri | Hitz@antidote: Argumentation-driven explainable artificial intelligence for digital medicine[END_REF]. CasiMedicos1 is a community and collaborative project where each member, developer or contributor adds something unique related to medical exams with the objective of enriching the information that will be available to students. In casiMedicos resource, the MIR exams were chosen and commented on by Spanish medical doctors in a voluntary effort to provide answers and explanations to the MIR exams. In the MIR exams source, there are 953 commented questions (153K words) that have been extracted from the MIR exams held between the years 2005, 2014, 2016, 2018, 2019, 2020, 2021 and 2022.

This dataset is unique in the medical domain as it consists of 621 theoretical, fictional clinical cases that provide real expert-crafted explanations, making it a valuable resource for students. Each clinical case is paired with a set of potential diagnoses, an indicator of the correct answer, and a detailed explanation of the decision-making process provided by medical professionals 2 . What sets this dataset apart from other Question Answering (QA) datasets in the medical domain is its inclusion of explanatory arguments for both the correct diagnosis or treatment and the reasons why other options are incorrect. These explanations, written by medical doctors, offer a rich source of information for research in XAI.

To prepare the data, we first enhance contextual information in the explanations by expanding abbreviated diagnosis references. It is common for the explainer to refer to diagnoses as "Answer 1", thus we implement a string replacement with the corresponding answer. Subsequently, as delineated in [START_REF] Agerri | Hitz@antidote: Argumentation-driven explainable artificial intelligence for digital medicine[END_REF], the dataset encompasses various types of questions. For the purpose of evaluating the explanations provided by the experts, we manually filter out cases that solely discuss potential diagnoses of the patients, yielding a total of 206 clinical cases.

Identification and Alignment of Potential Causes

The initial phase of our approach (Figure 1) consists in identifying all potential causes within the given context. In our medical scenario, the context is represented by clinical cases, and we regard all symptoms as potential causes that could explain the patient's diagnosis. To address this, we utilize the clinical information extraction pipeline proposed by Marro et al. [START_REF] Marro | Natural language explanatory arguments for correct and incorrect diagnoses of clinical cases[END_REF], which performs two key steps:

(i) Medical named entity recognition: Marro et al. first identify medical concepts and abnormalities described in the clinical case text using a named entity recognition (NER) system. Their NER module detects mentions of symptoms, findings, and other phenotypic concepts, labelling them with semantic tags like "Sign or Symptom" or "Finding. " Marro et al. trained this module on a dataset of 314 annotated clinical cases, achieving promising results for symptom detection with an F1-Score of 0.86 using a transformer-based model. The NER model can accurately label symptoms described in layperson terminology, an important capability since clinical cases are often written for student exams, where terms may not appear standardised.

(ii) Alignment to ontology terms: Next, Marro et al. map the detected semantic concepts to the standardized terminology of the Human Phenotype Ontology (HPO) [START_REF] Köhler | The human phenotype ontology in 2017[END_REF]. This mapping enables linking colloquial descriptions like "shortness of breath" to the formal HPO concept of "Dyspnea. " They compute contextual embeddings for each identified symptom and find the closest HPO match via cosine similarity. Their top-performing alignment approach attained 0.53 accuracy in aligning symptoms to equivalent ontology terms in the top 5 matches.

The output we obtain from their pipeline is a list of symptoms extracted from the case text and aligned to ontology concepts. We select the Sign or Symptom and No Sign or Symptom entities, as these denote the presence or absence of abnormalities relevant for diagnosis. This aligned set of reasons extracted from the specific clinical case serves as our starting point. We later expand this list by retrieving additional related symptoms from the HPO knowledge base.

Ultimately, Marro et al.'s techniques provide us with an initial set of ontology-grounded reasons consisting of the aligned symptoms and no symptoms identified within the context of the particular clinical case description. We subsequently feed this encoded evidence as input to the prevalence function, which will evaluate and score the relevance of these reasons from the case alongside other reasons derived from the external HPO knowledge.

Prevalence Function

The Prevalence Function 3 is a central component of our approach, designed to systematically assess the pertinence of each possible reason that could explain a given event. This function is inspired by cognitive processes involved in explanation selection, aiming to replicate these processes in a transparent and replicable manner. The function takes into account the possible reasons found in the clinical case (see Section 3.2) and a KB that serves to find other possible reasons outside of the context but still relevant to the case. It then evaluates all possible reasons based on a set of predefined conditions, each of which contributes to the final prevalence score of the key reason. These conditions include whether the key reason is linked to the correct or incorrect diagnosis, its occurrence rate, and whether it is unique to the correct diagnosis, or shared among all possible diagnoses. In line with the abnormal conditions model, our function assigns a higher score to key reasons that are unique to the correct diagnosis and have a low occurrence rate. This reflects the idea that abnormal conditions, i.e., conditions that do not usually occur, are more likely to be the cause of an event. Moreover, our approach integrates the concept of contrastive explanation. For instance, if a symptom associated with an incorrect diagnosis has a high occurrence rate and does not appear in the clinical case, it can be invoked to discard the incorrect diagnosis. This aligns with the idea that the differences between two events form the basis for the explanation.

The computation of the prevalence function starts with the acquisition of the set of potential reasons to be evaluated. In the context of medical diagnosis, these reasons correspond to symptoms, which can be identified either within the clinical case or within the Human Phenotype Ontology (HPO) as symptoms associated with each potential diagnosis. The ontology provides not only a list of symptoms for each diagnosis but also pertinent information about each symptom, such as its occurrence rate, definition, and synonyms. This information facilitates the definition of three distinct sets of reasons, which serve as the basis for the computation of the Prevalence function:

• 𝑆𝑦𝑚𝑝𝑡𝑜𝑚𝑠𝑂𝑓 𝐶𝑜𝑟𝑟𝑒𝑐𝑡𝐷𝑖𝑎𝑔𝑛𝑜𝑠𝑖𝑠: symptoms that belong to the correct disease; • 𝑆𝑦𝑚𝑝𝑡𝑜𝑚𝑠𝑂𝑓 𝐼𝑛𝑐𝑜𝑟𝑟𝑒𝑐𝑡𝐷𝑖𝑎𝑔𝑛𝑜𝑠𝑖𝑠: symptoms that belong to all the incorrect diseases; • 𝑃 𝑟𝑒𝑠𝑒𝑛𝑡𝑆𝑦𝑚𝑝𝑡𝑜𝑚𝑠: symptoms found in the case description.

The Prevalence Function is then used in conjunction with the additional disease information and symptom sets obtained from the HPO to produce a list of key reasons and their calculated prevalence scores. This allows us to provide a robust and transparent framework for assessing the quality of the reasons on which the explanations are grounded.

Algorithm explanation

The Prevalence Score Function, as outlined in Algorithm 0, is designed to assess the relevance or pertinence of a given key reason in the context of a clinical case (CC) and a knowledge base (KB). The function operates by assigning a score to the key reason based on its presence in the correct and incorrect diagnoses, its occurrence rate, and its presence in the clinical case.

The function begins by initializing the score to zero and setting several boolean variables to false (lines 2-6). It then retrieves the symptoms associated with the correct and incorrect diagnoses from the knowledge base (lines 7-8) and identifies the symptoms present in the clinical case using Named Entity Recognition (NER) (line 9).

Then, it checks if the key reason is present in the symptoms of the correct diagnosis (lines 11-15). If it is, the function increments the score and sets the variable linkedToCorrectDiagnosis to true. If not, linkedToCorrectDiagnosis is set to false.

Next, the function checks if the key reason is present in the symptoms of the incorrect diagnoses (lines 17-20). If it is, the variable linkedToIncorrectDiagnosis is set to true. If not, it is set to false.

The function then checks the occurrence rate of the key reason (lines 22-29). If the key reason has a high occurrence rate (more than 70%), the variable hasHighOccurrenceRate is set to true, and if it is linked to the correct diagnosis, the score is incremented. If the key reason has a low occurrence rate (less than 30%), the variable hasLowOccurrenceRate is set to true.

It then checks if the key reason is unique to the correct diagnosis or shared with other diagnoses (lines 31-40). If the key reason is unique to the correct diagnosis and has a low occurrence rate, the score is incremented twice. If the key reason is shared with other diagnoses, the score is decremented.

Finally, the function checks if the key reason is present in the symptoms of the incorrect diagnoses but not in the present symptoms (lines 42-48). If the key reason has a high occurrence rate, the score is incremented. Otherwise, the score is decremented.

The final score represents the prevalence of the key reason in the context of the specific clinical case, providing a measure of its relevance or pertinence.

Algorithm 1 Prevalence Function

Reason Alignment via Sentence Matching

A crucial step in our approach is the alignment of potential causes (i.e., reasons) identified in the clinical case with those actually invoked in the expert's explanation. This alignment (visualized as the "Reasons alignment module" in Figure 1) is achieved through a sentence matching technique. The objective of this step is to discern which of the potential reasons identified were actually utilized by the experts in their explanation, thereby enabling subsequent suggestions of modifications to enhance the explanation's pertinence.

Our approach to sentence matching is inspired by the work of Lu et al. [START_REF] Lu | NapSS: Paragraph-level medical text simplification via narrative prompting and sentence-matching summarization[END_REF], particularly their creation of an intermediate dataset using a distance metric for fine-tuning their sentencematching model. In their work, Lu et al. [START_REF] Lu | NapSS: Paragraph-level medical text simplification via narrative prompting and sentence-matching summarization[END_REF] employ the Jaccard distance to identify sentences with high similarity between complex and simplified texts. We adapt this strategy to our context, aiming to locate similar reasons between the clinical case and the explanations provided by the explainer. In our adaptation of Lu et al. 's approach, we aim to identify similar reasons between the clinical case and the explanations provided by the experts. However, our methodology diverges in two key aspects: the choice of distance metric, and the preprocessing of the texts for comparison. Instead of employing the Jaccard distance, we opt for a process that begins with the detection of medical-named entities within both texts. Following this, the texts are segmented into individual sentences.

Subsequently, we compute sentence embeddings using only the identified named entities. This computation leverages the Sentence Transformers method [START_REF] Reimers | Sentence-bert: Sentence embeddings using siamese bert-networks[END_REF], using various pre-trained models specialized in scientific text. To align sentences from the clinical case with those in the explanations, we employ cosine similarity. A match is considered valid only when the cosine distance is sufficiently close, ensuring that only highly similar sentences are matched, thereby enhancing the precision of our reason alignment process. Our embedding calculation method is also sensible to negation agents by representing the absence of a symptom in a clinical case with the label "No Sign or Symptom" in the NER stage. As an example of this, the sentence "A 62-year-old man with no history of alcohol abuse. . . " will have a different numerical representation than "A 62-year-old man with a history of alcohol abuse. . . ".

Template-Based Explanation Generation

In the final step of our pipeline, we employ a template-based generation approach to articulate the pertinence of each reason. This approach allows us to generate natural language explanations that are understandable by human users. Each template is designed to address a specific combination of features associated with a reason, and the appropriate template is selected based on the values of these features for each reason. The features considered in our approach are:

• uniqueToCorrectDiagnosis indicates whether the reason is unique to the correct diagnosis.

• sharedToOtherDiagnosis indicates whether the reason is shared with other diagnoses.

• hasLowOccurrenceRate indicates whether the reason has a low occurrence rate.

• hasHighOccurrenceRate indicates whether the reason has a high occurrence rate.

• linkedToCorrectDiagnosis indicates whether the reason is directly linked to the correct diagnosis.

• linkedToIncorrectDiagnosis indicates whether the reason is linked to an incorrect diagnosis.

• presentInClinicalCase indicates whether the reason is present in the clinical case.

Based on the values of these features, a template, like the following, is selected to generate the explanation: This template-based generation approach allows us to generate explanations that are informative and specific to the context of each reason, thereby ensuring the interpretability of the proposed approach.

Evaluation

In this section, we first present the experimental setting we propose to assess our approach, and then we discuss the obtained results. Finally, we apply our approach to a clinical case to discuss the final outcome of the pipeline.

Experimental Setting.

The main experimental component of our task is the named entity-based sentence matching. This task can be decomposed into two subtasks: first, the generation of tuples of similar sentences, and second, the fine-tuning of Language Models (LMs) using the generated dataset. For tuple generation, we implemented two approaches:

Baseline string distance method : As a baseline, we employed a simpler method based on string distance between full sentences, similar to the work of Lu et al. [START_REF] Lu | NapSS: Paragraph-level medical text simplification via narrative prompting and sentence-matching summarization[END_REF]. We first separated each sentence from the clinical case and associated explanation. We then computed the Levenshtein distance between all combinations of case and explanation sentences. If the distance between a pair surpassed a defined threshold (0.5), we kept that tuple as a match. Otherwise, we discarded it. This baseline relies only on the full sentence text, without considering named entities.

Named entity-based method : Subsequently, we generated tuples using the medical-named entities identified by Marro et al.'s pipeline [START_REF] Marro | Natural language explanatory arguments for correct and incorrect diagnoses of clinical cases[END_REF]. Their system detects not only symptom mentions but also their absence, allowing us to model the presence or lack of Signs/Symptoms differently. The identified entities are joined into sentences and embedded using Sentence-BERT [START_REF] Reimers | Sentence-bert: Sentence embeddings using siamese bert-networks[END_REF]. We compute cosine distances between all case and explanation sentence pairs, keeping those above a threshold (0.975) as matches.

For both cases, a dataset composed of (𝑐𝑎𝑠𝑒_𝑖𝑑, 𝑎𝑛𝑠𝑤𝑒𝑟_𝑠𝑒𝑛𝑡𝑒𝑛𝑐𝑒 𝑖 , 𝑐𝑎𝑠𝑒_𝑠𝑒𝑛𝑡𝑒𝑛𝑐𝑒 𝑗 , {𝑇 𝑟𝑢𝑒|𝐹 𝑎𝑙𝑠𝑒})

tuples is generated. For the fine-tuning of the LMs, we employ the PyTorch implementation provided by Hugging Face [START_REF] Wolf | Transformers: State-of-the-art natural language processing[END_REF]. The experiments were conducted with a batch size of 8, a maximum sequence length of 256, and a learning rate of 2.5e-5 over 4 epochs. We selected all-mpnet-base-v2 [START_REF] Reimers | Sentence-bert: Sentence embeddings using siamese bert-networks[END_REF] as our baseline, and fine-tuned models such as BioBERT v1.2 [START_REF] Lee | Biobert: a pre-trained biomedical language representation model for biomedical text mining[END_REF], S-PubMedBert-MS-MARCO [START_REF] Deka | Improved methods to aid unsupervised evidencebased fact checking for online health news[END_REF], and BioBERT-mnli-snli-scinli-scitail-mednli-stsb [START_REF] Deka | Evidence extraction to validate medical claims in fake news detection[END_REF] as more domain-specific LMs under the same experimental setting. Despite each transformer model achieving its best results with a different cosine similarity threshold for performing the named entity-based matching, we kept a threshold value of 0.975 to ensure the matching of sentences with the highest possible semantic coherence.

Results.

In this section, we present the obtained results on the Casimedicos dataset. We evaluate the quality of explanations written by experts, highlighting both successful and unsuccessful examples. The performance of the sentence matching task is quantified in terms of macroaverage precision, recall, and F1 score, as shown in Table 1.

We adopt the all-mpnet-base-v2 model as our baseline, being it the state-of-the-art for sentence embedding computation across various domains. However, our results indicate that domain-specific models outperform this baseline across all metrics. In particular, the model based on PubMedBert [START_REF] Gu | Domain-specific language model pretraining for biomedical natural language processing[END_REF][START_REF] Deka | Improved methods to aid unsupervised evidencebased fact checking for online health news[END_REF] demonstrates superior performance, achieving the highest scores in precision, recall, and F1 score (highlighted in bold in Table 1). These results underline the importance of domain-specific models in achieving high-quality sentence matching. To illustrate the outcome of our approach, we present a full clinical case, the expert's explanation, and the assessment of reasons from the CasiMedicos dataset. We consider a clinical case where the correct diagnosis is Porphyria cutanea tarda. The other potential diagnoses considered are Epidermolysis bullosa acquisita, Acute intermittent porphyria, and Ulerythema ophryogenesis.

Clinical Case: "A 62-year-old man with a history of significant alcohol abuse, carrier of hepatitis C virus, treated with Ibuprofen for tendinitis of the right shoulder, goes to his dermatologist because after spending two weeks on vacation at the beach he notices the appearance of tense blisters on the dorsum of his hands. On examination, in addition to localization and slight malar hypertrichosis. "

Expert's Explanation: "Porphyria Cutanea Tarda: 60% of patients with PCT are male, many of them drink alcohol in excess, women who develop it are usually treated with drugs containing estrogens. Most are males with signs of iron overload, this overload reduces the activity of the enzyme uroporphyrinogen decarboxylase, which leads to the elevation of uroporphyrins. HCV and HIV infections have been implicated in the precipitation of acquired PCT. There is a hereditary form with AD pattern. Patients with PCT present with blistering of photoexposed skin, most frequently on the dorsum of the hands and scalp. In addition to fragility, they may develop hypertrichosis, hyperpigmentation, cicatricial alopecia, and sclerodermal induration. "

Assessment of Reasons:

The generated explanations for the top and bottom scoring reasons are as follows:

• You should consider invoking the reason Elevated urinary delta-aminolevulinic acid since it is a symptom with a high occurrence rate for the incorrect disease Acute intermittent porphyria, and it does not appear in the clinical case, supporting discard Acute intermittent porphyria as the correct diagnosis. • You should also consider invoking the reason Abnormal hair morphology since it is a symptom with a high occurrence rate for the incorrect disease Epidermolysis bullosa acquisita, and it does not appear in the clinical case, supporting discard Epidermolysis bullosa acquisita as the correct diagnosis. • You should consider invoking the reason Alcoholism since it is a symptom unique to the correct diagnosis and present in the clinical case. • The symptom Contact dermatitis does not meet the criteria for a strong reason in this case. • The symptom Dry skin does not meet the criteria for a strong reason in this case.

• The symptom Dermal atrophy does not meet the criteria for a strong reason in this case. The Expert's Explanation for this case attributes the patient's condition to Porphyria Cutanea Tarda (PCT), citing factors such as the patient's gender, alcohol abuse, and the presence of blistering on photoexposed skin. These align with our top-scoring reasons in the Assessment of Reasons, demonstrating the agreement between the expert's explanation and our assessment. In the expert's explanation, the symptom "Abnormal hair morphology" is not mentioned. However, our methodology identifies it as a significant reason that could enhance the explanation. This symptom is common in the incorrect disease Epidermolysis bullosa acquisita, but it is not present in the clinical case. Therefore, its absence provides a strong reason to discard Epidermolysis bullosa acquisita as the correct diagnosis. This additional information could potentially enhance the expert's explanation by providing further evidence to support the correct diagnosis and rule out other alternatives. This demonstrates the capability of our approach not only to validate the reasons used by the expert but also to suggest new pieces of information that could enrich the explanation.

Concluding remarks

In this paper, we present a novel approach to recognizing the intricate cognitive processes that underpin the selection of explanations and strive to emulate these processes in a methodical and transparent manner. By incorporating the principles of abnormality and contrastive explanation, we ensure that our approach is attuned to the subtleties of explanation selection in real-world contexts, with a particular focus on the medical domain. Moreover, by leveraging the Human Phenotype Ontology and medical named entity recognition, we are able to identify and assess potential reasons in a systematic and data-driven manner. This not only allows us to assess the explanations that are consistent with the expert's perspective but also to suggest additional pieces of information that could enhance the explanation itself.

Our work provides a significant step forward in the development of AI systems that can automatically assess explanations both in a cognitively plausible and contextually sensitive manner. It is driven by the necessity for a systematic and transparent approach to assessing the pertinence of reasons used in explanations, particularly in the medical domain. The deterministic nature of the prevalence function ensures the transparency of our approach.

A key advantage of our methodology is its modularity, with distinct components that can be modified or replaced to adapt the approach to new domains or tasks. The prevalence function encapsulates the reasoning model for scoring explanations, while the sentence matching technique, matching strategy, and knowledge base instantiate components tailored to assessing clinical rationales. Each of these modules could be swapped with alternatives better suited for a different domain or application. For example, the prevalence function could be re-designed to match commonsense rather than medical reasoning, while more general lexical resources like WordNet could replace domain-specific ontologies. This flexibility broadens the applicability of our techniques to assess explanations in a wide range of settings. Future work includes the application of this approach to education in medicine, by providing a systematic and transparent way of evaluating the reasons given in medical residents' explanations, and to online discussions in medical fora by helping users and moderators to identify high-quality and low-quality explanations. Additionally, our current implementation is limited to evaluating explanations based solely on symptoms. However, physicians may consider other types of clinical information, such as lab findings, family history, or patient demographics. To address this, we plan to expand the scope of our approach by extracting and encoding additional clinical entities, such as test results, into ontology concepts like HPO. This will enable assessing explanations across a broader range of relevant clinical data.

Figure 1 :

 1 Figure 1: Overview of our approach for the automatic assessment of explanation's reasons.

•

  Template 1: "You should consider invoking the reason [reason] since it is unique to the correct diagnosis, has a high occurrence rate, and is present in the clinical case. " • Template 2: "You should also consider invoking the reason [reason] since it is a symptom with a high occurrence rate for the incorrect disease [disease], and it does not appear in the clinical case, supporting discard [disease] as the correct diagnosis. " • Template 3: "You should consider removing the reason [reason] since it is a common symptom alongside all possible diagnoses. "

Table 1

 1 Results for named entity-based matching in macro multi-class precision, recall, and F1-score.

	Tuples method	Model	P	R	F1
	String Distance all-mpnet-base-v2	0.80 0.84 0.82
	String Distance BioBERT cased v1.2	0.81 0.82 0.82
	String Distance BioBERT-mnli-snli-scinli-scitail-mednli-stsb 0.82 0.84 0.83
	String Distance S-PubMedBert-MS-MARCO	0.82 0.83 0.83
	Named Entity	all-mpnet-base-v2	0.84 0.70 0.74
	Named Entity	BioBERT cased v1.2	0.84 0.76 0.80
	Named Entity	BioBERT-mnli-snli-scinli-scitail-mednli-stsb 0.85 0.79 0.82
	Named Entity	S-PubMedBert-MS-MARCO	0.89 0.85 0.87

https://www.casimedicos.com/

https://github.com/ixa-ehu/antidote-casimedicos

Acknowledgements

This work has been supported by the French government, through the 3IA Côte d'Azur Investments in the Future project managed by the National Research Agency (ANR) with the reference number ANR-19-P3IA-0002. This work was supported by the CHIST-ERA grant of the Call XAI 2019 of the ANR with the grant number Project-ANR-21-CHR4-0002.

end if 16: if 𝑆𝑦𝑚𝑝𝑡𝑜𝑚 is in 𝑆𝑦𝑚𝑝𝑡𝑜𝑚𝑠𝑂𝑓 𝐼𝑛𝑐𝑜𝑟𝑟𝑒𝑐𝑡𝐷𝑖𝑎𝑔𝑛𝑜𝑠𝑖𝑠 then 17:

𝑙𝑖𝑛𝑘𝑒𝑑𝑇 𝑜𝐼𝑛𝑐𝑜𝑟𝑟𝑒𝑐𝑡𝐷𝑖𝑎𝑔𝑛𝑜𝑠𝑖𝑠 = 𝑇 𝑟𝑢𝑒


Alberto Ranavolo
email: a.ranavolo@inail.it

Giorgia Chini

Francesco Draicchio

Alessio Silvetti

Tiwana Varrecchia

Lorenzo Fiori

Antonella Tatarelli

Patricia Helen Rosen

Sascha Wischniewski

Philipp Albrecht

Draicchio

Fiori

Rosen Ph

Cherubi- Ni A

Human-Robot Collaboration (HRC) Technologies for Reducing Work-Related Musculoskeletal Diseases in Industry 4.0

Keywords: HRC, biomechanical load, workplace rehabilitation, return to work, ergonomic, WMSDs

est destinée au dépôt

Introduction

The new scenarios of a connected and digital world provide opportunities for the integration of new tools into everyday life and workplaces, the so-called Industry 4.0.

In the workplace, the use of sensor networks and human-robot collaboration (HRC) technologies are coming more and more to the fore as an opportunity for both biomechanical overload risk mitigation and for the adoption of new return-to-work strategies. On these grounds, the European Union's Horizon 2020 research and innovation program funded, under Grant Agreement No. 871237, the Socio-physical Interaction Skills for Cooperative Human-Robot Systems in Agile Production (SOPHIA) project with the aim to develop a new generation of HRC technologies and sensors networks. Sensor networks, through the continuous and real-time monitoring of worker's physiological and biomechanical parameters (by measuring kinematic, kinetic and muscular activity, etc.), can be used to control the robots through specific interfaces, evaluate the efficacy of ergonomic interventions and provide vibro-tactile/acoustic/visual stimuli to the workers to execute the task in a less overloading way to reduce the risk of developing work-related musculoskeletal disorders (WMSDs). HRC technologies include wearable assistive robots (WearBots, Exoskeletons) and collaborative robots (CoBots) able to act on the base of the real needs of the worker, support him during the working activity with the aim of minimize the biomechanical load and reduce the probability of WMSDs insurgence. Commonly used IMUs can be accelerometers (uni-/bi-or tri-axial), gyroscopes and magnetic sensors. Typically, the probes are equipped with three orthogonal accelerometers and three orthogonal gyroscopes to measure linear acceleration and angular velocity, respectively, along three orthogonal axes. These devices are particularly suitable for use in the workplace since they are portable, easy to wear for the users, monitored remotely and able to provide a direct feedback to the end-users [START_REF] Ranavolo | Wearable monitoring devices for biomechanical risk assessment at work: Current status and future challenges-A systematic review[END_REF][START_REF] Wang | Interactive wearable systems for upper body rehabilitation: a systematic review[END_REF][START_REF] Cuesta-Vargas | The use of inertial sensors system for human motion analysis[END_REF][START_REF] Ullah | A comprehensive survey of wireless body area networks[END_REF][START_REF] Patel | A review of wearable sensors and systems with application in rehabilitation[END_REF].

The forces exchanged by workers with the environment can be measured by highly reliable [START_REF] Holt | Hand-held dynamometry strength measures for internal and external rotation demonstrate superior reliability, lower minimal detectable change and higher correlation to isokinetic dynamometry than externally-fixed dynamometry of the shoulder[END_REF], easy to use, portable and low-priced hand-held dynamometers. These devices are placed between a fixed point and the subject's body part to assess the isometric muscle (or muscle group) strength to investigate changes in the functional status of trunk, lower and upper limbs [START_REF] Park | Reliability and Validity of a New Method for Isometric Back Extensor Strength Evaluation Using A Hand-Held Dynamometer[END_REF][START_REF] Jackson | Intrarater reliability of handheld dynamometry in measuring lower extremity isometric strength using a portable stabilization device[END_REF][START_REF] Karthikbabu | Hand-Held Dynamometer is a Reliable Tool to Measure Trunk Muscle Strength in Chronic Stroke[END_REF][START_REF] Andersen | Between-day reliability of a hand-held dynamometer and surface electromyography recordings during isometric submaximal contractions in different shoulder positions[END_REF][START_REF] Stark | Hand-held dynamometry correlation with the gold standard isokinetic dynamometry: A systematic review[END_REF]. For a given hand size, forces are also recorded by superior grip dynamometers, by instrumented gloves (i.e., equipped by force sensitive resistors) or by force sensor mats applied to handles [START_REF] Kong | Optimal cylindrical handle diameter for grip force tasks[END_REF][START_REF] Seo | The effect of torque direction and cylindrical handle diameter on the coupling between the hand and a cylindrical handle[END_REF][START_REF] Seo | Investigation of grip force, normal force, contact area, hand size, and handle size for cylindrical handles[END_REF][START_REF] Kong | Evaluation of handles in a maximum gripping task[END_REF][START_REF] Kong | Evaluation of meat-hook handle shapes[END_REF][START_REF] Hall | External pressure at the hand during object handling and work with tools[END_REF][START_REF] Radwin | External finger forces in submaximal static prehension[END_REF]. To adapt to a wide variety of handle sizes and geometries multi-dimensional grip dynamometers are a valid alternative. In the end, we can also mention the use of the haptic tools, physical bendable strips that enable the users to manipulate and apply deformations to digital surfaces and to move and rotate virtual objects [START_REF] Ranavolo | Wearable monitoring devices for biomechanical risk assessment at work: Current status and future challenges-A systematic review[END_REF]. sEMG sensors are used to investigate the muscle activity during the execution of manual handling activities. Single-or double differential bipolar recordings using wet electrodes are the most widely used sEMG measurement methods in the workplace, since in this case the probes do not interfere with the typical movements performed by workers thanks to the miniaturization process and wireless communication protocols.

Wearable robotic technologies

Exoskeletons are wearable devices that help people during the execution of specific tasks by applying forces and/or torques on one or more joints. They were first developed in the clinical setting for motor rehabilitation [START_REF] Colombo | Treadmill training of paraplegic patients using a robotic orthosis[END_REF] and to support people with motor disabilities [START_REF] Ortiz | XoSoft-iterative design of a modular soft lower limb exoskeleton[END_REF][START_REF] Xosoft | XoSoft[END_REF], in military [START_REF] Kazerooni | On the control of the berkeley lower extremity exoskeleton (BLEEX)[END_REF] and sport [START_REF]RoamRobotics[END_REF] fields.

More recently, the use of exoskeletons has also been extended to the industrial sector, as they can be an additional tool for reducing biomechanical risk in the workplace.

The exoskeletons essentially differ based on how the torques / forces applied to the human joints are generated and therefore the first distinction is between active and passive exoskeletons.

Active exoskeletons are those that generate forces/torques with powered actuators, such as electric motors, pneumatic or battery-operated exoskeletons. The action of the exoskeleton is controlled by a computer program based on information acquired through a series of sensors applied to the body of the subject who uses it (e.g.: sEMG, accelerations, angular velocities). Since the functioning of these exoskeletons is based on the online processing of biomechanical parameters, these exoskeletons they follow the movement with greater precision than the passive ones for the tasks in which they can contribute. In addition, those with battery are also easier to move, and therefore more comfortable to use at workplace, than the tethered ones. On the other hand, however, these exoskeletons are heavier and less manageable than the passive ones [START_REF] Toxiri | Back-support exoskeletons for occupational use: an overview of technological advances and trends[END_REF]. Passive exoskeletons use elastic elements such as coil springs, compact rotational springs, integrated gas springs or elastic bands [START_REF] Toxiri | Back-support exoskeletons for occupational use: an overview of technological advances and trends[END_REF] for the generation of joint forces/torques. . Within each of these two macro categories, the exoskeletons are then divided into soft, rigid, or mixed.

Collaborative Robots (Cobots)

Collaborative robots, also called cobots [START_REF] Colgate | Robots for Collaboration with Human Operators[END_REF] are robots based on HRC systems and represent a natural evolution of industrial robot, because they can solve existing challenges in industry, i.e., they can help workers to perform physically heavy tasks, thanks to the ability to physically interact with humans in a shared workspace; moreover, they are designed to be easily reprogrammed even by non-experts to be used for different roles [START_REF] Guerin | A framework for end-user instruction of a robot assistant for manufacturing[END_REF]. Furthermore, the greater convenience of collaborative systems is their flexibility [START_REF] Matheson | Human-Robot Collaboration in Manufacturing Applications: A Review[END_REF]. HRC systems were introduced primarily for occupational health (ergonomics and human factors) reasons [START_REF] Matheson | Human-Robot Collaboration in Manufacturing Applications: A Review[END_REF]. The use of cobots can contribute to economic growth and the creation of better, healthier, and more attractive working environments for the future workforce since cobots can simultaneously increase productivity and reduce WMSDs, which represent the single largest category of work-related disease in industrial countries. Anyhow, several technologies must be in place to enable humans and robots to work together to achieve shared goals. So, it is important to distinguish the different ways of interaction. Müller et al. [START_REF] Müller | Skill-based dynamic task allocation in Human-Robot-Cooperation with the example of welding application[END_REF] proposed a classification for the different methodologies in which humans and robots can work together. They distinguish among: i) coexistence, when human and robot are in the same environment, but they do not interact, ii) synchronized if human and robot work in the same space at different time, iii) cooperation, when human and robot work in the same workspace at the same time, but they perform different tasks and iv) collaboration, when human and robot perform the task together. Regardless of the type of the human-robot interaction, to make it effective, it is fundamental to ensure a correct information exchange between the operator and the cobot. This requires suitable interfaces that monitor human behavior-to properly plan the execution of the collaborative task-and strategies that increase the mutual awareness of the human-robot couple [START_REF] Ajoudani | Smart collaborative systems for enabling flexible and ergonomic work practices[END_REF]. To this scope, as mentioned above, kinematic, kinetic and physiological sensors networks are available. In addition, other devices are also available that can enhance the sensory experience when using a cobot, such as wearable haptic systems to provide the user with the sense of touch [START_REF] Bianchi | A fabric-based approach for wearable haptics[END_REF], or augmented reality systems, in which components of the digital world may be superimposed upon or composed with the real world and used in teleoperation [START_REF] Andaluz | Transparency of a bilateral tele-operation scheme of a mobile manipulator robot[END_REF].

3

The SOPHIA project activities

Standardization

To provide definitions and guidelines for the safe and practical use of cobots in industry, several standards are already available [START_REF]Machine safety, general design principles, risk assessment, and risk reduction[END_REF][START_REF]Robots and equipment for robots, Safety requirements for industrial robots, Part 1: Robots[END_REF][START_REF]Robots and equipment for robots, Safety requirements for industrial robots, Part 2: Systems and integration of robots[END_REF][START_REF]Robots and robotic devices, Collaborative Robots[END_REF]. Moreover, none of the ergonomics standards [START_REF]Manual Handling Part 1: Lifting and Carrying[END_REF][START_REF]Manual Handling Part 2: Pushing and Pulling[END_REF][START_REF]Manual Handling Part 3: Handling of Low Loads at High Frequency[END_REF][START_REF]Application Document for ISO Standards on Manual Handling (ISO 11228-1/2/3) and Static Working Postures (ISO 11226)[END_REF][START_REF]Evaluation of Static Working Postures[END_REF][START_REF]Manual Handling of People in the Healthcare Sector[END_REF], neither the traditional methods listed within them cover the biomechanical risk detection when collaborative technologies are used. This gap, together with the need to strengthen the scientific basis upon which the standards are based [START_REF] Armstrong | Scientific basis of ISO standards on biomechanical risk factors[END_REF], represents the reasons that existing standards should be supplemented or revised or, if necessary, that new standards should be developed [START_REF] Ajoudani | Smart collaborative systems for enabling flexible and ergonomic work practices[END_REF].

Literature [START_REF] Alberto | Wearable monitoring devices for biomechanical risk assessment at work: Current status and future challenges-A systematic review[END_REF] already evidenced some critical issues such as: their observational nature, subjectivity, susceptibility to the restrictions of the equations and parameters, insufficient accuracy, precision and resolution, unclear choices of the preferred methods of risk assessment over others. New sensor-based tools for biomechanical risk assessment will be used for quantitative "direct instrumental evaluations" to obtain the rating in standard methods, when applicable, to measure some parameters otherwise measured with poor precision and accuracy, necessary to obtain the level of risk. In this light, SOPHIA is going to work on the need of a revision of current ergonomics standards to also include the use of these tools for biomechanical risk assessment.

Reduction of biomechanical risk

Bio-electrical activity, skeletal joint kinematics and kinetics data will be used to assess the worker motor capacity and how it varies over time thus providing a musculoskeletal model that can predict muscle fatigue and injury based on worker movements. Next step will be the investigation of the interaction between worker and external systems (wearable and robots) and to analyze how these impact on biomechanical load. These data could also be used to develop an online instrumental-based tool for monitoring and classifying the biomechanical risk in manual handling activities when standardized protocols cannot be used or for a confirmation of the rating of observational data with standard protocols. One more target is to develop wearable devices to monitor human-motor variables and to render haptic stimuli to specific areas of the worker's body (e.g. shoulder, lower back, ankle, knee, etc.) to inform the users about the inappropriateness of the posture adopted guiding them towards ergonomic postures and a safe action execution.

HRC in work environment

The European Union (EU) recognizes to HRC technologies a high relevance for the economic growth and for population health care. EU has planned a Strategic Research Agenda to provide a strategic overview and a technical guide aimed to identify medium term research and innovation goals [START_REF]Multi-Annual Roadmap[END_REF][45] and promotes standardization activities for a better market adoption and to develop a single digital market [START_REF] Vanderborght | Unlocking the Potential of Industrial Human-Robot Collaboration[END_REF].

In this light, the SOPHIA project aims to achieve successful and robust HRC through the process of data from different sensors and to publish a software library and an open access dataset for benchmarking HRC solutions in collaborative scenarios. It will be also developed the overall cognitive decision frameworks allowing the human and robot to collaborate considering human and environment to guarantee health (ergonomics) and safety (collision avoidance). In this light it is critical to develop the social interaction principles (human-centered) to ensure a fluent communication between workers and HRC technologies.

To improve the flexibility of Fellow-Assistant robots, SOPHIA project includes activities focused on the development of stable hierarchical interaction controllers. This will enable CoBots to reconfigure the collaborative task frame, to simultaneously ensure human ergonomics and safety requirements and adapt task parameters by optimizing the required multi-task criteria. Multi-task and multi-person optimization will be central to the development of CoBot control framework in real environmental scenario.

HRC and work rehabilitation

SOPHIA project aims to validate the HRC technologies also in the healthcare sector and in return-to-work rehabilitation of neurological patients with motor disorders and to develop miniaturized wearable devices to monitor human-motor parameters and treat specific areas of the worker's body with tactile stimuli. To achieve these outcomes the European consortium is developing myoelectric HRC interfaces to study the interaction among hybrid work environments and workers with the aim to highlight their specific residual abilities and unfulfilled potential. Furthermore, the project aims to design training plans on sEMG based technique for broaden the audience of experienced professionals in multifactorial movement analysis. Neurological disease patients can receive remarkable rehabilitation results from the use of HRC technologies. Ongoing monitoring of sEMG parameters such as muscle activation timing, amplitude and fatigue play a significant role in the design of innovative active exoskeleton controller systems. The main issue with using sEMG to control collaborative wearable trunk and upper limb devices designed to assist neurological patients, concerns the algorithms applied in human-robot interfaces. Just few years ago the application of these algorithms was limited due to their inaccuracy in recognizing the high subjective movement variability of neurological patients. But now, thanks to machine learning algorithms, these limits have been overcome and HRC technologies are enhanced and optimized also for people with severe upper and lower limb disabilities [53]. Hence, SOPHIA project will develop algorithms for the HRC to recognize specific movement pattern to predict patient's movement intention.

2 Wearable sensors and robotic technologies: state of the art

	2.1 Wearable sensors
	Recently, commercial and research miniaturized wearable wireless sensors were in-
	troduced in the workplace to monitor workers during their activities.

These sensors include Inertial Measurement Units (IMUs) to measure bodies' kinematics, dynamometers to evaluate subjects' force and surface electromyography (sEMG) sensors to analyze the muscle behaviors.


Wanchen Li

Robin Passama

Vincent Bonnet

Andrea Cherubini

A comparison of human skeleton extractors for real-time human-robot interaction

Keywords: human-robot interaction, activity recognition, 3D skeleton detection, computer vision

Modern industrial manufacturing procedures gradually integrate physical Human-Robot interaction (pHRI) scenarios. This requires robots to understand human intentions for effective and safe cooperation. Vision is the most commonly used sensor modality for robots to perceive human behavior. In this paper, a comparison of existing vision-based human skeleton extraction frameworks is made, to provide guidance for the design of human-robot interaction applications. A dataset consisting of consecutive images that records 14 actions conducted by different users acquired by a kinect camera is used for human skeleton extraction. The work justifies our choice of skeleton extractors according to pHRI constraints.

I. INTRODUCTION

A. Motivation for human activity recognition using skeleton data

Modern manufacturing processes are integrating more and more human-robot cooperation scenarios, where robot and humans share the same workspace. To successfully perform a cooperation task, robots need to embed intelligence to be able to understand the intention of humans, by recognizing human activity. This is first of all for safety considerations: since the robot is located in proximity to the human, the movement of the robot should be guaranteed to be harmless to human. For this reason, the robot needs to dispose information on human body posture. Moreover, since robots and human are sharing the same workspace, the two can perceive each other's behavior and even physically interact directly or indirectly. The capability of understanding human activity enables robots to collaborate with humans in an intuitive and efficient manner. Apart from improving productivity, human robot cooperation should also enhance human comfort and prevent potential musculoskeletal disorders. With clever definition of a taxonomy of activities, activity recognition provides a basis for online ergonomic evaluation of human postures [START_REF] Vianello | Latent ergonomics maps: Real-time visualization of estimated ergonomics of human movements[END_REF].

Currently, one of the most popular data sources for human activity recognition is vision based data, since visual modality has been continuously studied and significant advances in both computer science and hardware have been made in recent years. With recent release of cost-effective depth sensors, the problem of loss of information in projection from 3d to W.Li, R.Passama, A.Cherubini are with LIRMM, University of Montpellier, CNRS, Interactive Digital Humans group, 161 rue Ada, 34095, Montpellier, France. firstname.lastname@lirmm.fr;

V.Bonnet is with LAAS-CNRS, INSA-Toulouse, 7 Av. du Colonel Roche, 31400 Toulouse, France. firstname.lastname@laas.fr 2d can be recovered. Depth cameras give easy access to 3d human body data at a high resolution frame, therefore, this data acquisition mode is close to human-being perception. The low-cost and ease of installation of depth camera, compared to motion capture systems such as xsens, vicon makes it more friendly for use cases in the complex industrial humanrobot cooperation context. The depth camera is used in a non invasive manner, which does not require workers to carry addition loads. Therefore, we pay special attention to human activity recognition technology based on 3d visual perception.

In [START_REF] Aggarwal | Human activity recognition from 3d data: A review[END_REF], the authors compared human activity recognition from different data sources of range sensors, i.e. single devices that can capture 3D data from one point of view, such as depth camera. The comparison was made between 3D silhouettes, skeletal data, spatio-temporal features, 3D occupancy features extracted from sensors, and 3D optical flow data. The review showed that using 3D silhouettes for activity recognition can be only limited for atomic actions recognition and occlusion can gravely degrade the silhouettes extraction. This is not suitable for human-robot scenarios including complex actions and interactions with objects. Using 3d optical flow data is computationally costly and not suitable for real time application. Using skeletal data for feature building is invariant to camera location and subject appearance, and it is better at modeling finer activities. Indeed, skeleton-based recognition exploits the position and orientation changes of human joints between frames. Since the joints and the segments structure can be translated into a kinematic model of human body, the kinematic properties of different body parts can provide abundant features for activity recognition. Of course, skeletal data can't provide information on external objects involved in the human activities, and any incompleteness of skeleton data will largely degrade the recognition.

B. Previous work on 3d skeletal data acquisition from depth camera

Skeleton data is an auxiliary output of raw visual data for human activity recognition. It generates a vast literature and tons of astonishing outbreaks of technology improvements in recent years. Skeletal data can be acquired from pure RGB data or in combination with depth information.

Initial skeleton extraction methods merged from the works on body part extraction, identification and detection using classifiers on monocular images [3] [5]. In [START_REF] Shotton | Real-time human pose recognition in parts from single depth images[END_REF], the technology of joint position identification using a deep randomized decision forest based on body part proposal from depth image was proposed, this became the core technology embeded in Microsoft kinect Xbox. Another notable work was using joint regressors based on random forests algorithm for estimation 2d skeleton positions [START_REF] Dantone | Human pose estimation using body parts dependent joint regressors[END_REF]. Afterwards, with the advancement on machine learning technology, skeleton extraction by learning showed great results [START_REF] Tompson | Joint training of a convolutional network and a graphical model for human pose estimation[END_REF] [START_REF] Toshev | Human pose estimation via deep neural networks[END_REF]. Another side effect of research on machine learning is the explosive growth of data volume. Many free-access large datasets have been created for launching different algorithmic challenge so as to promote technical advancement in machine learing. Among them, for the topic of human pose estimation (or skeleton recognition), famous datasets are MPII Human Pose [START_REF] Andriluka | 2d human pose estimation: New benchmark and state of the art analysis[END_REF], OCHuman [START_REF] Zhang | Pose2seg: Detection free human instance segmentation[END_REF], Human3.6M [START_REF] Ionescu | Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments[END_REF]. The MPII dataset is composed of images taken from online videos covering hundreds of human activities, with person's 16 joints positions manually annotated. The Human3.6M is a motion capture dataset, images are acquired from motion capture systems with accurate 3D joint positions recording. The OCHuman is a dataset that focuses on occluded human images, which makes it the most challenging dataset for human detection. Among these 3, the MPII dataset is the most coherent to our scope of discussion. We browsed the relevant papers on human skeleton recognition developed based on this dataset. We screened several frameworks that are most likely to be suitable for real-time human-robot collaboration applications for comparison. Our selection criteria are as follows:

• The proposed framework for skeleton extraction has open access code source. In addition, the code source should provide simple APIs for direct downstream exploitation. It should be portable for arbitrary image or video inference. The input image or video for inference does not carry additional key-point cues or multiple view-angle cues.

• The proposed framework should be able to extract skeleton from images at a considerable speed. Because for the pHRI scenario, the human activity recognition task must be fast in order to timely control the robot. In our specific pHRI scenario, we tend to use skeleton extraction libraries to obtain Cartesian coordinates of joint positions of human body, then apply inverse kinematic analysis on the acquired skeleton information to obtain joint angles, finally use joint angles to recognize human activity [START_REF] Singh | A framework for recognizing industrial actions via joint angles[END_REF]. In this complete pipeline, to assure its real time compatibility, the skeleton extraction algorithm should run at about 10 Hz. The algorithm should at least be capable of providing skeleton recognition information of limbs and torso, to assure a successful inverse kinematic analysis on human body model. • Our pipeline is developed in a C++ implementation framework, ideally the skeleton extraction library should allow easy interfacing with the current framework.

In fact, the first restriction has already excluded a large part of skeleton extraction frameworks. In the following section, we will discuss several well developed frameworks that are compatible with the first restriction. We discuss their skeleton extraction principles and their execution performances. For this, we test each library on a video recorded during an experimentation for SOPIHA project at University of Montpellier. In this video, a person used a brush to deburr a gear in cooperation with BAZAR robot, which simulates an industrial activity in a pHRI scenario. There is the presence of different objects, a single person and 2 robotic arms in the video. The test was performed on a computer equipped with 1 GPU GeForce GTX 1060 and 1 CPU Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz. Our selected frameworks for skeleton extraction are Detectron2, Mediapipe, Yolov7, Alphapose and Openpose. All of these frameworks rely on deep learning method for skeleton prediction.

II. COMPARISON OF HUMAN SKELETON EXTRACTION FRAMEWORKS

A. Detectron2

Detectron2 is a library that provides human detection and segmentation algorithms for computer vision inputs such as image or video. It is the next generation of Detectron library, developed by Facebook AI Research (FAIR). The library is implemented in Python upon Pytorch engine, it embeds various variant algorithm of Mask R-CNN model developed also by FAIR [START_REF] He | Mask rcnn[END_REF]. Its advantages with respect to prior technology Faster-RCNN, is the high-quality on instance segmentation, by adding a branch in the network for providing object mask with the existing branch for bounding box recognition. The skeleton detection is an extended functionality. The framework regards each key-point of human pose as a one-hot mask, the prediction of key-points on human skeleton simply becomes the prediction of K object masks on the human body zone, one for each key-point located on the joint position. These keypoint prediction is made without knowledge on human body modeling structure. As such, this is a one shot detection method applicable for multi-person cases. The more the presence of human in the visual input, and the more key-points to be detected, the longer the algorithm needs for calculation.

The given model in the library for key-point detection was trained on COCO trainval35k. The framework offers possibility for customized training. For keypoint detection, the output of the framework is the bouding box position of detected person, his joint positions, the segmentation mask and the confidence level of the prediction. We run the framework on our customised video input, the prediction runs at around 3.57 fps on the tested computer. We observe strong oscillation on the predicted joint positions between frames. Keypoint prediction is missing on occluded human body part. The result certainly fails to meet our needs for real time skeleton recognition in terms of computing speed and accuracy.

B. Mediapipe

Mediapipe is a library for body pose tracking in RGB video frames using a 33 points topology. The algorithm is based on BlazePose [START_REF] Bazarevsky | On-device Real-time Body Pose tracking[END_REF], it is a two-step encoder-decoder method. A detector firstly locates the region of interest within the frame, then a tracker locates the joint positions and tracks them in the derived ROI in the upcoming frames. With this principle, the detection only work with the presence of person face because the person detector is built on the face detection proxy, upon assumption that the head of person should always be visible. In addition, the detection of the person does not happen on each frame in order to gain a rapid execution speed. These principles make it a light-weight detection library but only applicable for vision streaming input which contains a single person whose movement does not involve sudden changes between frames. The framework is based on Tensorflow engine, written in C++ with possible APIs in Python via Pybind library and in C++ via Bazel building.

Mediapipe is trained on closed-source dataset, it can provided 3d inference of joint positions, the z direction coordinates are inferred from the pairing of detected x, y joint positions with Ghum 3D model (Generative 3D Human Shape and Articulated Pose Models). The library achieves a processing speed on about 17 fps on our video input. Due to its inference principle, we observe that the detected key-point positions are largely biased from actual joint positions when the person is moving fast in the video. Keypoint detection is available on occluded body parts, but once again, the accuracy of these detected positions is questionable.

C. Yolov7

Yolov7 is an architecture capable of object detection, object segmentation and human skeleton detection, which differs it from previous YOLO models. The algorithm optimized the training cost, the re-parameterized method, dynamic label assignment strategy and model scaling techniques with respect to previous YOLO algorithms, this enables faster inference speed and higher detection accuracy [START_REF] Wang | YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[END_REF]. It is a single stage detector implementing a heatmap free approach for keypoint detection. It can provides one-shot detection for multi-person. The framework is implemented in Python based on Pytorch which allows customized training and the provided model for keypoint detection was trained on COCO dataset, which contains annotations on body segment but not on hand, face and foot. The processing speed of this architecture on our video is 11.04 fps. The prediction results on joint positions are stable between frames. The algorithm can even output keypoint predictions on occluded body part.

D. Alphapose

Alphapose is an architecture proposed in 2018 and the development community remains actively updated today in 2023. The pose detection algorithm follows the top-down strategy. Compared to other top-down skeleton extraction algorithms, its difference is using a lower detection confidence for human detection, with this the algorithm can obtain more candidates for pose estimation. Pose estimation candidates are then eliminated by parametric pose non-maximumsuppression. For dealing with scaling problem across different body part keypoint estimation, the algorithm adopts symmetric integral keypoints regression method to localize keypoints in different scales. The algorithm also introduce a pose-aware identity embedding to conduct identity tracking duing pose estimation [START_REF] Fang | Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time[END_REF]. The framework is implemented in Python based on Pytorch, and different pretrained models by COCO, PoseTrack and Halpe-FullBody datasets are proposed for either torso or whole body pose estimation including face and hands. We tested different pretrained models in the Alphapose library on our video. Our obtained best performance in terms of execution speed is using Fast Pose DUC model for 17 keypoints estimation in COCO format at 9.74 fps, using Fase Pose model for 26 keypoints estimation in Halpe format at 8.01 fps, and ??? for 136 keypoints estimation. The pose estimation results oscillate between adjacent frames, and pose estimation result is missing on occluded body part.

E. Openpose

Openpose is a skeleton extraction library for multi-person detection of body, hand, foot and facial keypoints. It follows a buttom-up approach. The keypoint estimation on face and hand; the 3d triangulation for all detected keypoints are both relied on multi-camera system, for our use case with one single camera, the library can only offer 25 keypoints estimations in 2d on body and foot. For this, the library uses a feedforward network to predict a confidence map of body part locations and a set of part affinity fields. Then these confidence maps and part affinity fields are parsed by greedy inference in the global context of detection to output 2d keypoints [START_REF] Cao | Realtime multiperson 2d pose estimation using part affinity fields[END_REF]. This core model for body-foot keypoint detection is trained on COCO and MPII datasets, plus a small subset of foot instances out of COCO datasets with 6 additional annotated keypoints.

The library is implemented based on caffe deep learning engine, in C++ language. We test its core functionality of bodyfoot keypoint preciction on our custom video, the processing speed has reached 8.7 fps for outputting the result at default resolution 1x368. At this resolution level, the output keypoints have slight ocsilations between frames but the global skeleton prediction result remains convincing. We can further speed up the processing by tuning output resolution to a lower value with sacrifice of detection accuracy. We found that when the output resolution has been lower to 1x320, the skeleton output remains convincing without strong ocsilation between frames, and the processing speed at this resolution is about 9.91 fps.

III. CONCLUSIONS

Apart from above 5 libraries, another open-source library with complete implementation and interfaces for arbitrary images and videos skeleton extraction is Mmpose library. This library has integrated a wide spectrum of skeleton extraction algorithms. We have also tested this library on our video but the processing speed was about 2.155 fps. Due to its inefficiency, we didn't include it in the library comparison. ArtTrack and LightTrack libraries are also excluded for the same reason, MeTRAbs is excluded because of its close dependency on a 3d display rendering software.

We summary our comparison between abovementioned libraries in the below table I, we quantify their corresponding processing speed. Our running test is carried out on an old For our project, the desired pipeline which implements the skeleton extraction framework is written in C++ language. Therefore, in this comparison we care about the easiness of interfacing the skeleton extraction library with c++ coding project. In the above 5 libraries, the most adapted one should be Openpose. However, this is a constraint unique for our project. Broadly speaking, for general pHRI applications, YOLOv7, Alphapose and Openpose libraries are all worth considering. To further compare these 3 libraries, we will focus on their skeleton extraction accuracy.

It is difficult to compare the accuracy from the visual feedback of the skeleton detection results of different libraries(see Fig1), they presumably all give credible skeleton predictions. We need a quantitative accuracy comparison of these skeleton extraction libraries, this is done in the next section between YOLOv7, Alphapose and Openpose libraries. Detectron2 library is excluded because of its inefficiency, and we also excluded Mediapipe because its keypoint detection is limited in single person use case with no significant movement, while these constraints can be easily violated in industrial application.

  (a) Yolo output. (b) Alphapose output.(c) Openpose output.

Fig. 1 :

 1 Fig. 1: Skeletion estimation output from YOLO, Alphapose, Openpose on a same image.

TABLE I

 I graphics card, if it is configured on a advanced graphics card, there is still room for processing speed improvement for each library.

	Skeleton extractors	Tracking	Multi-person	Foot keypoints	Hand keypoints	Facial	Easy C++	Framerate
			detection			keypoints	interfacing
	Detectron2	×	✓	×	×	ears,eyes,nose	×	3.57 fps
	Mediapipe	✓	×	✓	✓	ears,eyes,	×	17 fps
						nose,mouth
	YOLOv7	×	✓	×	×	ears,eyes,nose	×	11.04 fps
	Alphapose	✓	✓	×	×	ears,eyes,nose	×	9.74 fps
	Openpose	×	✓	✓	×	ears,eyes,nose	✓	9.91 fps

model

ACKNOWLEDGMENT

This research was carried out in the context of the SOPHIA project, which received funding from the EU Horizon 2020 research and innovation program under Grant Agreement No. 871237.


Characterizations of the MIL-100(Fe) paper membrane

Dual-fibre concept

The mechanical properties of the 75 wt% MIL-100(Fe) loaded paper membrane with different ratio of fibres (cellulose:NFC) were investigated by evaluating their tensile strength as well as their flexibility.

Strain-stress curves Force vs bending angle curves

NFC fibre interacting at micron level with the particles

Long cellulose fibre

These results highligth a synergistic effect of the fibres on the mechanical properties of the paper membrane. Conventional cellulosic fibres provide flexibility to the composite, while NFC enhances the mechanical strength by improving inter-fiber bonding and intermolecular bonds with MOF particles.

Cellulose flexibility

Breakage

NFC limited flexibility

Cellulose brittle in tensile strength

NFC enhancing tensile strength

Abundant and inexpensive binder used Rapid, simple, sustainable and scalable process High MOF loading (>70wt%)

Capture properties identical to the powder

The paper membrane met all the specifications for a good adsorbent composite and showed high selectivity for the capture of polar VOCs. The next step is to study its seperation performance.

MOFs are obtained in loose powder form requiring a shaping method for their practical use. Problems are often observed with beads or pellets, monoliths such as structural changes, pore blocking and/or low mass transport properties. To avoid these drawbacks, MOFs can be shaped as a paper.

Reaching a high MOF loading (>70wt%) while keeping good mechanical properties remains very challenging. We combined cellulose fibres with Nano-Fibrillated Cellulose (NFC) to improve the inter-fibre bonding. A 75wt% MIL-100(Fe) loaded paper without pore blockage and mechanically stable was obtained.

[1] : H. Katsuzawa et al. J. Fiber Soc. Japan. 50, 452-457. 1994

  isothermXRD patterns of the MOF particles before and after shaping remain identical confirming that the process doesn't alter the cristallinity of the particles. The paper membrane degrades starting from 240°C (TGA not shown here). The N 2 adsorption isotherm indicates a fully preserved porosity with a S BET of 1540 vs 1900 m 2 .g -1 for the MOF powder. Finally, nano-CT scan revealed a homogeneous spatial distribution of the particles and the fibres in the volume. the paper membrane based on the CT scanning data (collab M. Pinto, S. Carvalho, Lisbon) (Fe) as powder has been recently identified to selectively trap traces of organic acids through a chemisorption mechanism on Lewis acid sites[2]. We made a set-up to determine the adsorption capacity of the paper membrane with different VOCs (acetic, acid, formic acid, acrylic acid and furfural) in presence of moisture.