MD Paul L Den Exter

MD Lucia J M Kroft

Carole Gonsalves

Grégoire Le Gal

MD Cornelia M Schaefer-Prokop

MD, MSc, FRCPC Marc Carrier

Frederikus Menno V Huisman

F Beenen

MD | Carol Gonsalves

| Gregoire

MD Le Gal

| Menno

MD V Huisman

MD Frederikus A Klok
email: f.a.klok@lumc.nl

Dr Susan Kahn

Gregoire Le Gal

Establishing diagnostic criteria and treatment of subsegmental pulmonary embolism: A Delphi analysis of experts

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Essentials

• Establishing a subsegmental pulmonary embolism diagnosis is challenging.

• Surveys were conducted among expert thoracic radiologists and clinical thrombosis specialists.

• Delphi analysis provided uniform diagnostic criteria to diagnose subsegmental pulmonary embolism.

• Anticoagulant treatment was favored for prior venous thromboembolism, antiphospholipid syndrome, pregnancy, cancer, and proximal deep vein thrombosis.

| INTRODUCTION

Multidetector computed tomographic pulmonary angiography (CTPA) has evolved as the imaging test of choice in the diagnostic workup of clinically suspected pulmonary embolism (PE). [START_REF] Huisman | Pulmonary embolism[END_REF] As CTPA allows better visualization of peripheral pulmonary arteries compared to previously used imaging techniques, small emboli isolated to subsegmental branches of the pulmonary artery tree are increasingly being detected. [START_REF] Stein | Multidetector computed tomography for acute pulmonary embolism[END_REF][START_REF] Anderson | Computed tomographic pulmonary angiography vs ventilation-perfusion lung scanning in patients with suspected pulmonary embolism: a randomized controlled trial[END_REF][START_REF] Carrier | Symptomatic subsegmental pulmonary embolism: to treat or not to treat?[END_REF] The rate of these so-called subsegmental PE (ssPE) diagnoses has been reported to have almost doubled to 9%, when earlier studies were compared to more recent studies using computed tomography (CT) scans with more detector rows. [START_REF] Carrier | Subsegmental pulmonary embolism diagnosed by computed tomography: incidence and clinical implications. A systematic review and meta-analysis of the management outcome studies[END_REF] Considering that a proportion of ssPE diagnoses would have gone undetected and thus left untreated with former imaging techniques, the clinical significance of these findings is subject to debate. Some observational case series have indeed provided some ground for leaving ssPE untreated in the setting of low risk of (recurrent) venous thromboembolism (VTE) and in the absence of deep vein thrombosis (DVT) of the leg. [START_REF] Donato | Clinical outcomes in patients with isolated subsegmental pulmonary emboli diagnosed by multidetector CT pulmonary angiography[END_REF][START_REF] Goy | Sub-segmental pulmonary embolism in three academic teaching hospitals: a review of management and outcomes[END_REF] On the other hand, unselected patients with ssPE have been described to have a risk of recurrent VTE comparable to that of patients with more proximal PE. [START_REF] Den Exter | Risk profile and clinical outcome of symptomatic subsegmental acute pulmonary embolism[END_REF] Establishing a radiological diagnosis of ssPE is challenging, as demonstrated by a retrospective cohort study in which 11% of routinely established ssPE diagnoses were reinterpreted to be without any evidence of PE, while on the other hand, 37% of patients initially diagnosed with ssPE were reinterpreted to have more proximal PE. [START_REF] Pena | Difference in interpretation of computed tomography pulmonary angiography diagnosis of subsegmental thrombosis in patients with suspected pulmonary embolism[END_REF] As a false-positive ssPE diagnosis may come with the consequence of unnecessary anticoagulant treatment and its associated potential harms, establishing a correct diagnosis is of utmost importance. [START_REF] Swan | The problem of under-diagnosis and over-diagnosis of pulmonary embolism[END_REF] Moreover, clinical trials aimed at assessing the optimal management of ssPE would be invalid should the diagnosis not be reproducible.

One of the possible explanations of the poor interobserver reliability for the diagnosis of ssPE is the lack of widely accepted and reproducible diagnostic criteria as well as the existence of related terms such as incidental and isolated. We therefore sought to establish uniform diagnostic criteria for ssPE, guided by systematic review of the literature and consensus of expert thoracic radiologists. Second, we aimed to propose these criteria to clinical VTE specialists and to establish consensus of current best practice with regard to the therapeutic management of ssPE.

| METHODS

| Study design

The Delphi method was used to assess expert consensus among a panel of thoracic radiologists and VTE specialists. This is a well-accepted and widely used structured process to build consensus through opinions and feedback from a group of informed experts. [START_REF] Jones | Consensus methods for medical and health services research[END_REF][START_REF] Humphrey-Murto | The use of the Delphi and other consensus group methods in medical education research: a review[END_REF][START_REF] Waggoner | Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research[END_REF] It is an anonymous process where a predefined research problem, usually lacking empirical evidence, is expressed to an expert panel in the form of a questionnaire. Responses are collated and analyzed, and the process usually continues for several rounds until the best possible level of consensus is achieved. [START_REF] Hasson | Research guidelines for the Delphi survey technique[END_REF] The experts are allowed to adjust their answers in subsequent rounds, based on how they interpret the results of the expert panel as whole, which are provided to them.

The Conducting and Reporting Delphi Studies guidelines were followed for the conducting and reporting of this Delphi study. [START_REF] Jünger | Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review[END_REF]

| Selection of experts

This Delphi analysis consisted of two parts. The first part was designed to establish radiological criteria for a CTPA diagnosis of ssPE, and the second to propose these criteria to clinical VTE specialists and to establish consensus of current best practice about the treatment of ssPE.

For the first part of the Delphi analysis, expert radiologists were selected based on the following criteria: (i) leaders in the field of clinical thoracic radiology as demonstrated by a strong publication track

Conclusion:

The results of this analysis provide standard radiological criteria for ssPE that may be applicable in both clinical trials and practice.

K E Y W O R D S

anticoagulant treatment, Delphi analysis, diagnosis, pulmonary embolism, subsegmental record, (ii) preferably being member of the Fleischner society, and (iii) currently practicing and/or involved in the training of resident radiologists. For the second part, clinical VTE specialists were selected based on the following criteria: (i) leaders in the field of diagnosis and management of VTE as demonstrated by a strong publication track record, (ii) preferably being member of ISTH or the American Society of Hematology, and (iii) currently practicing and/or involved in the training of resident clinicians.

Both expert panels were selected to represent a wide geographic area and to include both females and males. The experts completed the questionnaire anonymously and were unaware of the identity of the other involved experts. proved its final version. This resulted in seven questions that were constructed to survey their clinical management approach. In addition, the final results of Part I of the Delphi analysis were provided to the clinical expert panel, to assess whether they would agree with the proposed diagnostic criteria. Subsequent rounds were planned until consensus would have been reached. Consensus in both Parts I and II of the Delphi analysis was defined as a minimum level of agreement of 70%, in line with previous Delphi reports. [START_REF] Mahler | American College of Chest Physicians consensus statement on the management of dyspnea in patients with advanced lung or heart disease[END_REF][START_REF] Blasi | The most appropriate therapeutic strategy for acute lower respiratory tract infections: a Delphi-based approach[END_REF][START_REF] Zafar | Consensus-based standards for best supportive care in clinical trials in advanced cancer[END_REF][START_REF] Brown | Recommendations for musculoskeletal ultrasonography by rheumatologists: setting global standards for best practice by expert consensus[END_REF] We planned any number of consecutive Delphi rounds to achieve this consensus. The Delphi process was undertaken with the use of an online survey tool (SurveyMonkey). Responses were filed at the experts' discretion until a given deadline. Two reminders were sent before reaching the deadline.

| Delphi process and steering committee

| Hypothesis

The normal anatomy of the pulmonary artery follows that of the bronchial anatomy. The main pulmonary artery arises at the base of the right ventricle and extends superiorly for approximately 5 cm before dividing in into the right and left pulmonary artery (Figure 1).

The right main pulmonary artery usually divides into an ascending upper lobe artery and a descending interlobar artery that gives rise to the middle and right lower lobe arteries. The left main pulmonary artery, after giving rise to the left upper lobe artery, usually descends and gives rise to the (segmental) lingual arteries and left lower lobe artery. Pulmonary arteries divide by dichotomous branching.

Overall, approximately 17 of these divisions are usually present. The further branching of lobar, segmental, and subsegmental pulmonary arteries show considerable variation. To identify a specific lobar or segmental artery branch, it is usually necessary to identify its associated bronchus.

The subsegmental pulmonary arteries may have a diameter of up to 6-7 mm in diameter just after the initial branch, decreasing to <1.5 mm in the peripheral lung. [START_REF] Chen | Measurement of the diameter of segmental and sub-segmental pulmonary arteries of the right inferior lobe by multislice spiral CT angiography[END_REF][START_REF] Sakuma | Epoprostenol infusion therapy changes angiographic findings of pulmonary arteries in patients with idiopathic pulmonary arterial hypertension[END_REF] Vessels smaller than this range are labeled arterioles and are not visible on standard CTPA images, which normally allows detection of arteries as small as 2 mm in diameter. A prerequisite to identify subsegmental PE is the clear and doubtless identification of the subsegmental vessels with an intraluminal contrast defect. The contrast defect can be local and surrounded by intravascular contrast or lead to a complete obstruction and thus cutoff of the vessel. It has been demonstrated that 98% of subsegmental arteries could be identified using 64-row multidetector computed tomography scanners with 1.2 mm collimation and 0.5 mm ultrathin reconstructed effective slice thickness. [START_REF] Niemann | Detectability of subsegmental pulmonary vessels in 64 MDCT-pulmonary angiography[END_REF] Anatomic regions that are not adequately depicted correspond mostly to paracardiac anatomic locations that are exposed to the motion of the beating heart muscle with resulting artefacts.

Consequently, we hypothesized that radiological criteria for ssPE should involve the following aspects: (i) technical quality of the scan (eg, contrast timing, artefacts, identification of all subsegmental arteries), (ii) anatomic location (ie, correlation to bronchus and number of side branches from the lobar artery), and (c) artery diameter. This hypothesis was the basis of the first part of the Delphi analysis.

| Analysis

Descriptive statistics were calculated using SPSS version 25.0.0 (IBM, Armonk, NY, USA).

| RESULTS

| Literature search

The systematic literature review was performed on October 30, 2017, and resulted in a total of 280 eligible articles. Based on the titles and abstracts, 32 articles were selected for complete review.

Diagnostic criteria for a CTPA diagnosis of ssPE could not be extracted from any of the articles.

| Delphi Part I: Radiological definition

In October 2018, the first round of the Delphi questionnaire was distributed among 40 expert radiologists. Of those, 29 consented to participate and completed the questionnaire (73%). The expert panel included radiologists practicing in 12 different countries; 27 of 29 participants completed the full Delphi procedure.

Following compilation of the results of the online survey, consensus was reached on 8 of 16 statements (Figure 2). A vast majority of experts (89%) agreed that the diagnostic criteria for ssPE should be based on the anatomic location within the pulmonary arterial system alone and not on the vessel diameter of the pulmonary artery affected. This was reinforced by another statement in which the expert panel agreed (78%) that the diameter of a certain affected pulmonary artery branch does not contribute to the definition of ssPE.

Technical specifications of the CT scanner, a minimum Hounsfield Unit, or maximal collimator width were not considered to be relevant either. Also, 82% of the experts agreed that the term isolated ssPE refers to a solitary embolus in a single subsegmental artery without the presence of other ssPEs or additional emboli more centrally.

The statements for which consensus was not achieved were discussed within the Delphi steering committee, modified, and sent for a secondary survey round. In this second round, respondents were able to see which of the multiple-choice options achieved the highest level of consensus. The second survey consisted of nine statements with multiple-choice answers and were answered by all responders of the first round. Eight of the nine statements achieved consensus (Figure 3). Based on the results of the Delphi, we constructed the following diagnostic criteria for ssPE on CTPA: "A contrast defect in a subsegmental artery, ie, the first arterial branch division of any segmental artery independent of artery diameter, visible in at least two subsequent axial slices, using a CT scanner with a desired maximum collimator width of ≤1 mm." The agreement on specific recommendations for confirming and reporting of ssPE is summarized in Table 1. A total of 96% of the participants approved these diagnostic criteria and recommendations. They also agreed on the statement that the diameter of the affected vessel does not contribute to the diagnosis or treatment of ssPE (97% consensus). Of the respondents who did not agree with the diagnostic criteria, three commented that the definition isolated ssPE should include the absence of (proximal) DVT. Two respondents commented that "sufficient contrast" should be specified. One suggested that the number and location of subsegmental F I G U R E 1 Schematic overview of anatomy of the pulmonary artery; green branches represent main and interlobar arteries, blue branches lobar arteries, red branches segmental arteries, and gray branches subsegmental arteries Yes, peripheral one third of segment.

| Delphi

Yes, peripheral 50% of segment.

Should the anatomic location (e.g., peripheral one third of the segment) be taken into the account for ssPE determination? Does perfusion imaging (dual energy or subtraction) help in reliably diagnosing ssPE?

Can the diameter of a certain pulmonary artery branch be used to diagnose ssPE? Should the confidence in the diagnosis of ssPE be included into the report? What are the best window settings to evaluate ssPE?

this may decrease to 74% when considering single segmental or subsegmental pulmonary emboli. [START_REF] Kligerman | Radiologist performance in the detection of pulmonary embolism: features that favor correct interpretation and risk factors for errors[END_REF] Interobserver agreement between radiologists in diagnosing ssPE has been shown to be low (κ = 0.38; 95% CI, 0.0-0.89). [START_REF] Ghanima | Multidetector computed tomography (MDCT) in the diagnosis of pulmonary embolism: interobserver agreement among radiologists with varied levels of experience[END_REF] In another study, reinterpretation by five radiologists showed that at least one radiologist disagreed with the initial interpretation of ssPE in 60% of the cases. [START_REF] Miller | Small pulmonary artery defects are not reliable indicators of pulmonary embolism[END_REF] This might in part be explained by the high level of expertise that is necessary to diagnose ssPE. Indeed, it has been demonstrated that the sensitivity of ssPE diagnosis is higher among thoracic radiologists than among nonthoracic radiologists and that the diagnosis is more often correctly made among radiologists than among radiology residents. [START_REF] Shaham | CT pulmonary angiography for the detection of pulmonary embolism: interobserver agreement between on-call radiology residents and specialists (CTPA interobserver agreement)[END_REF][START_REF] Joshi | Reliability of on-call radiology residents' interpretation of 64-slice CT pulmonary angiography for the detection of pulmonary embolism[END_REF][START_REF] Verweij | Interobserver agreement between on-call radiology residents and radiology specialists in the diagnosis of pulmonary embolism using computed tomography pulmonary angiography[END_REF] In contrast to our hypothesis, the expert radiologists were resolute in their opinion that only the anatomic of the contrast filling defect accounted for the definition of ssPE. The pulmonary artery diameter, which is known to be variable based on the anatomic location and dependent on anatomic variants and hemodynamic status, was not considered to be relevant. The reason why we explicitly included the arterial diameter in the first Delphi round was that the anatomic definition of subsegmental refers to quite a wide range of arterial diameters from the fourth generation (main-lobar-segmental-subsegmental) of more centrally located arteries to very small peripheral arteries. This indeed allows for a more straightforward definition of ssPE when excluding artery diameter as input criteria, which represents a variable and difficult to reliably measure parameter. Of note, whether this definition leads to higher interobserver agreement and a decline in false-positive ssPE diagnoses remains to be demonstrated.

Image quality was also rejected as being relevant for the definition of ssPE. The expert radiologists did reach consensus on the statement that "the observer should verify whether and to which branching of the subsegmental arteries contrast enhancement is sufficient to identify contrast defects at the subsegmental level before evaluation of the presence of subsegmental pulmonary embolism can be reliably performed." This implies that the diagnosis cannot The desired max. reconstructed slice thickness to reliably diagnose ssPE is ≤1 mm.

The preferred approach of reading a CTPA to diagnose ssPE is systematically, going from the center to the periphery.

An observer should verify whether and to which branching of subsegmental arteries contrast resolution is sufficient to identify contrast defects on subsegmental level before evaluation of the presence of ssPE can be reliably performed.

To diagnose ssPE with certainty, the presence of substantial motion artefacts should be excluded. If the observer is not sure of the diagnosis, this should be stated. The observer should verify to which branching of the subsegmental arteries (eg, proximal, distal) contrast enhancement is sufficient to identify contrast defects on subsegmental level before evaluation of the presence of subsegmental pulmonary embolism can be reliably performed.

To diagnose subsegmental pulmonary embolism with certainty, the presence of substantial motion artefacts should be excluded, preferably in the "lung window," in addition to affirmation of sufficient contrast enhancement. If the observer is not sure of the diagnosis, this should be stated in the report.

The term isolated subsegmental pulmonary embolism is reserved for a subsegmental pulmonary embolism in one subsegmental artery, and the term multiple subsegmental emboli refers to PE in two or more subsegmental arteries (or distal branches)."

be reliably assessed in a certain segment if "sufficient contrast enhancement" is not reached. In such cases, assuming either an abnormal D-dimer test or a high or likely clinical pretest probability, 29 a repeat CTPA scan can be considered if it is likely that technical aspects and not limitations of the clinical status of the patient caused the limited quality of the examination. Alternatively, a scintigraphic examination can be considered. Importantly, consensus among radiologists was reached to include the level of confidence into the diagnosis of isolated ssPE into the report, allowing for outweighing the risk of treatment versus a potentially false-positive diagnosis.

A remarkable observation in this study is that expert radiologists uniformly refer to "isolated ssPE" when it involves a single subsegmental embolus. This definition is more distinct than the one used in clinical studies, where isolated usually refers to one or multiple ssPEs without radiological evidence for more proximal PE or DVT. [START_REF] Carrier | Subsegmental pulmonary embolism diagnosed by computed tomography: incidence and clinical implications. A systematic review and meta-analysis of the management outcome studies[END_REF][START_REF] Carrier | Symptomatic sub-segmental pulmonary embolism: what is the next step?[END_REF] This difference in interpretation of ssPE terminology further strengthens the need for criteria that are universally applicable. Importantly, this new definition was approved by a majority of the clinical VTE experts. The fact that isolated refers to a single embolus means that irrespective of its location on a central or peripheral subsegmental artery, its resulting perfusion defect will be limited to a subsegmental area. This can be also seen as a further justification of the fact that the generation of a subsegmental artery and thus the diameter are not considered in the definition.

Until recently, anticoagulant treatment was the management approach of choice for any patient diagnosed with PE, regardless of its anatomic location. Whether patients with ssPEs benefit from anticoagulant treatment is increasingly challenged by reports of patients with ssPEs with an uncomplicated course in the absence of treatment. [START_REF] Bariteau | Systematic review and meta-analysis of outcomes of patients with subsegmental pulmonary embolism with and without anticoagulation treatment[END_REF] In addition, although this should be regarded as indirect evidence, population studies have demonstrated that the introduction of modern CT scanners has led to an increased incidence of PE diagnoses without influencing the mortality rate or the 3-month risk of VTE among patients left untreated following a negative CTPA. [START_REF] Wiener | Time trends in pulmonary embolism in the United States: evidence of overdiagnosis[END_REF] Conversely, diagnostic strategies raising the threshold for performing CTPA in the setting of suspected PE are associated with a lower prevalence of ssPE. [START_REF] Van Der Pol | Lower prevalence of subsegmental pulmonary embolism after application of the YEARS diagnostic algorithm[END_REF][START_REF] Van Der Hulle | Simplified diagnostic management of suspected pulmonary embolism (the YEARS study): a prospective, multicentre, cohort study[END_REF] Based on these findings, the latest guidelines of the American College of Chest Physicians have suggested that in patients with ssPEs and no proximal DVT in the legs who have a low risk for recurrent VTE, clinical surveillance is preferred over anticoagulation. [START_REF] Kearon | Establishing diagnostic criteria and treatment of subsegmental pulmonary embolism: A Delphi analysis of experts[END_REF] Definite evidence for the safety of such an approach has yet to arrive from a prospective single-arm trial (NCT01455818) and a planned randomized controlled trial (NCT04263038). While Incidental ssPE should be left untreated.

What is the maximum diameter size a ssPE may be left untreated?

The treatment duration of ssPE should be the same as for more proximal PE in the setting of a comparable thromboembolic/bleeding risk profile. Strengths of this study include the fact that broad and worldwide expert panels were surveyed. Second, by selecting a second expert panel of VTE specialists, we were able to validate consensus for the provided diagnostic criteria in an independent sample of expert clinicians. Third, the results are strengthened by a high level of consensus among the different statements (16/17). In both expert panels, participation initial rates were <80%, which forms a limitation of this study. Still, consensus was reached easily within both panels, with need of few consecutive survey rounds.

In conclusion, this study has provided radiological criteria for ssPEs that may be applied in both clinical trials and practice.

Moreover, we also report best practice for the therapeutic manage- A Going from the center to the periphery per lobe.

ment
B Going caudo-cranially or cranio-caudally.

C Other, namely: [free text] D No systematic approach is necessary for reading CTPA for subsegmental pulmonary embolism. 14.9%)

The term isolated subsegmental pulmonary embolism refers to:

A A solitary embolus in a single subsegmental artery without the presence of additional emboli more centrally.

B The presence of one or multiple subsegmental emboli without the presence of additional emboli more centrally.

Should the confidence into the diagnosis of subsegmental pulmonary embolism be included into the report?

A Yes, in this way: [free text] B No.

17. What is the best (systematic) standard of reporting the results of CTPA in the setting of suspected pulmonary embolism with regard to the subsegmental arteries?

[free text]

A

  multinational Delphi steering committee of eight members was established to oversee the process: six clinical experts on venous thromboembolism (PE, MC, CG, FK, MH, GG) and two thoracic radiological experts (LK and CSP). In cooperation with a trained medical librarian, a comprehensive literature review (Appendix 1) was performed by two members (PE and FK) to search for current recommendations in diagnosing PE and, in particular, ssPE. Based on the findings of this review and input from the steering committee, a first version of the Delphi questionnaire was drafted by two members (PE and FK). All members of the steering committee, both the clinical VTE experts and the expert radiologists, provided feedback on the questionnaire and approved its final version. For Part I of our Delphi study, this resulted in a total of 16 statements with primarily multiple-choice questions (Appendix 2). Each question included a free text box for comments. A final open question was included, which allowed the expert panel to provide input on the best systematic approach for reporting the results of CTPA in the setting of suspected pulmonary embolism with regard to the subsegmental arteries. Subsequent rounds were planned until consensus would have been reached. Following completion of Part I, a second survey (Part II) was constructed with an aim to survey clinical VTE experts. Again, a first version of the Delphi questionnaire was drafted by two members (PE and FK). All members of the steering committee provided feedback on the questionnaire and ap-

  Part II: Therapeutic management In February 2019, the first round of the Delphi questionnaire was distributed among 51 clinical VTE specialists. Of those, 40 consented to participate and completed the questionnaire (78%). The expert panel included clinicians practicing in 11 different countries; 39 of 40 participants completed the full Delphi procedure. The results of Part II of the Delphi analysis are depicted in

Figure 4 .

 4 Figure 4. A majority of 83% of the experts agreed with the proposed diagnostic criteria derived from the first part of the Delphi.

F I G U R E 2

 2 should always be included in the report. For one respondent, it was unclear why the phrase "first arterial branch" was needed in the definition. One final respondent suggested to add: "A contrast defect in a subsegmental or lower/smaller/further out artery."Over two Delphi rounds, the second round answered by all the responders of the first round, agreement of >70% was achieved that incidental and symptomatic ssPE may be left untreated in the absence of the following five conditions: previous VTE, antiphospholipid syndrome, pregnancy, active malignancy, and asymptomatic or symptomatic proximal DVT. The number of subsegmental emboli, that is, one or multiple, was not found to be a discriminative factor for the decision whether to start treatment. Therefore, the identified conditions necessitating anticoagulant treatment apply to both single or multiple subsegmental pulmonary emboli. In case treatment was indicated, the expert panel agreed (77%) that the treatment duration of ssPE should be the same as for more proximal PE in the setting of a comparable thromboembolic risk profile. It was agreed that the preferred type of anticoagulant (eg, direct oral anticoagulants, vitamin K antagonists, or low-molecular-weight heparins) for ssPE should be the same as for more proximal PE.Based on input from the expert panel during the first Delphi round, we additionally assessed the implications of the bleeding risk for the decision of whether to treat ssPE. It was agreed (90%) that the risk of bleeding complications should carry a greater weight in the decision of whether to treat a patient with ssPE with anticoagulants, and for how long, than for comparable patients with more proximal PE.4 | DISCUSSIONThe results of this Delphi analysis have established a clear radiological definition of ssPE with a high level of consensus among both thoracic radiological and clinical VTE experts. These criteria provide guidance in diagnostic and therapeutic decision making. Based on the second part of the Delphi, we also report best practice for the management of ssPE in specific settings as agreed upon by the expert clinicians.Reported difficulties in diagnosing ssPE stress the need for uniform diagnostic criteria. Although clinical trials have reported the sensitivity of diagnosing ssPE on CTPA to be as high as 82%-100%, Statements reaching consensus in the first round of the first part of the Delphi analysis. CTPA, computed tomographic pulmonary angiography; ssPE, subsegmental pulmonary embolismThe definition of subsegmental pulmonary embolism on CTPA be based on:

F I G U R E 3

 3 Statements reaching consensus in the second round of the first part of the Delphi analysis. CTPA, computed tomographic pulmonary angiography; ssPE, subsegmental pulmonary embolism filling defect in a subsegmental artery visible in at least two subsequent axial slices.

1

 1 Specific recommendations for confirming and reporting subsegmental pulmonary embolism

F I G U R E 4

 4 awaiting the results of this trial, our study provides more clarity on which conditions, as established by clinical VTE specialists, could determine the decision on whether to treat ssPEs. The following conditions were considered to necessitate anticoagulant treatment in case of either single or multiple subsegmental pulmonary emboli: Summary of best clinical practice with regard to management of ssPE resulting from the two rounds of the second part of the Delphi analysis. CTPA, computed tomographic pulmonary angiography; DOAC, direct oral anticoagulant; LMWH, low-molecular-weight heparin; PE, pulmonary embolism; ssPE, subsegmental pulmonary embolism; VKA, vitamin K antagonist; VTE, venous thromboembolism Disagree The type of anticoagulant (e.g. DOAC, VKA, LMWH) for SSPE should be be the same as for more proximal PE. Patients with previous VTE and a new diagnosis of ssPE should be treated with anticoagulant therapy.

  antiphospholipid syndrome, pregnancy, active malignancy, and asymptomatic or symptomatic proximal DVT.

  of ssPEs in specific settings as agreed on by the expert cli-Other, namely: [free text] 12. What are the best window settings to evaluate subsegmental pul-A At least 4-detector-row CT is sufficient. B At least 16-detector-row CT is sufficient. C At least 64-detector-row CT is sufficient. D Yes, but other, namely: [free text] E Technical specifications of the CT scanner are not relevant for the diagnosis of subsegmental pulmonary embolism.

	DEN EXTER ET al.	| 1261
	A PPE N D I X 1 6. What is the desired maximum collimator width to reliably diagnose	C The technical quality of the CTPA.
	subsegmental pulmonary embolism?	D Other, namely: [free text]
	ELECTRONIC DATABASE SEARCH
	PUBMED A < 1 mm.	2. What is in your opinion correct? monary embolism?
	("subsegmental pulmonary emboli"[tw] OR "subsegmental pulmo-B 1 mm.	A pulmonary artery is called subsegmental
	nary embolism"[tw] OR "subsegmental pulmonary embolisms"[tw] C 2 mm.	A A constant CT angiography window.
	OR "subsegmental pulmonary embolus"[tw] OR "subsegmental D 3 mm.	A After the first arterial branch has divided off the segmental ar-B The window/ settings should be adjusted to the intravascular
	pe"[tw] OR "subsegmental pes"[tw] OR "subsegmental pulmo-E Other, namely: [free text]	tery irrespective of its size (following standard anatomy; Figure contrast.
	nary thromboembolism"[tw] OR "subsegmental sized pulmonary F The collimator width does not contribute to the diagnosis of sub-	1). C The window settings should be modified going from the central
	emboli"[tw] OR "sub segmental acute pe"[tw] OR "sub segmental segmental pulmonary embolism.	B As soon as any arterial branch has divided off the segmental ar-arteries to the peripheral arteries.
	embolisms"[tw] OR "sub segmental pe"[tw] OR "sub segmental pul-	tery irrespective of its size. D Other, namely: [free text]
	monary embolism"[tw] OR (("subsegmental"[tw] OR subsegment*[tw] 7. What is the desired Hounsfield Unit in the main pulmonary artery	C Other, namely: [free text] E Window settings do not contribute to the diagnosis of subseg-
	OR "sub segmental"[tw] OR sub segment*[tw]) AND ("pulmo-to reliably diagnose subsegmental pulmonary embolism?	mental pulmonary embolism.
	nary emboli"[tw] OR "pulmonary embolism"[tw] OR "pulmonary	3. If a pulmonary artery originates from a division of a clearly identifi-
	embolisms"[tw] OR "pulmonary embolus"[tw] OR "pe"[ti] OR A >100 HU.	able segmental artery (following standard anatomy; Figure 1), should the 13. Should technical specifications of the CT scanner be considered
	nicians. Future studies should assess to what extent our diagnostic "pes"[ti] OR "pulmonary thromboembolism"[tw] OR "Pulmonary B >150 HU.	vessel diameter be taken into account for the determination whether it is for the diagnosis of subsegmental pulmonary embolism?
	criteria improve the sensitivity and interobserver reliability of diag-Embolism"[Mesh])) OR (((small branch*[tw] OR smaller branch*[tw] C >200 HU.	subsegmental or not? If yes, what is the maximum diameter size a pulmo-
	nosing ssPEs. OR smallest branch*[tw]) AND ("Pulmonary Artery"[Mesh] OR D Other, namely: [free text]	nary branch may be called subsegmental?
	"Pulmonary Artery"[tw] OR "Pulmonary Arteries"[tw])) AND ("pul-E The Hounsfield Unit in the main pulmonary artery does not con-
	monary emboli"[tw] OR "pulmonary embolism"[tw] OR "pulmo-tribute to the definition of subsegmental pulmonary embolism.	A ≤2 mm.
	nary embolisms"[tw] OR "pulmonary embolus"[tw] OR "pe"[ti] OR	B ≤3 mm.
	"pes"[ti] OR "pulmonary thromboembolism"[tw] OR "Pulmonary 8. Should CT images be routinely reviewed in multiplanar reconstruc-	C ≤4 mm.
	Embolism"[Mesh])) NOT ("Animals"[mesh] NOT "Humans"[mesh]) tions to reliably diagnose subsegmental pulmonary embolism?	D ≤5 mm.
	NOT ("Case Reports"[ptyp] OR "case report"[ti]) AND (english[la]	E ≤6 mm.
	OR "dutch"[la])) NOT ("subsegmental pulmonary emboli"[ti] OR A Always.	F Other, namely: [free text]
	"subsegmental pulmonary embolism"[ti] OR "subsegmental pulmo-B Never.	G The diameter of the affected vessel does not contribute to the
	nary embolisms"[ti] OR "subsegmental pulmonary embolus"[ti] OR C On demand, namely: [free text]	definition of subsegmental pulmonary embolism.
	"subsegmental pe"[ti] OR "subsegmental pes"[ti] OR "subsegmental
	pulmonary thromboembolism"[ti] OR "subsegmental sized pulmo-9. Should CT images be routinely reviewed using maximum intensity	4. If a pulmonary artery originates from a division of a clearly identifi-
	nary emboli"[ti] OR "sub segmental acute pe"[ti] OR "sub segmental projections to reliably diagnose subsegmental pulmonary embolism?	able segmental artery (following standard anatomy; Figure 1), should be
	embolisms"[ti] OR "sub segmental pe"[ti] OR "sub segmental pulmo-	the anatomic location (eg, peripheral one third of the segment) taken into
	nary embolism"[ti] OR (("subsegmental"[ti] OR subsegment*[ti] OR A Always.	the account for the determination whether it is subsegmental or not? If
	"sub segmental"[ti] OR sub segment*[ti]) AND ("pulmonary emboli"[ti] B Never.	yes, what would be your preference?
	OR "pulmonary embolism"[ti] OR "pulmonary embolisms"[ti] OR C On demand, namely: [free text]
	"pulmonary embolus"[ti] OR "pe"[ti] OR "pes"[ti] OR "pulmonary	A Peripheral one third of the segment.
	thromboembolism"[ti] OR "Pulmonary Embolism"[majr])) OR (((small 10. Does perfusion imaging (dual energy or subtraction) help in reli-	B Peripheral 50% of the segment.
	branch*[ti] OR smaller branch*[ti] OR smallest branch*[ti]) AND ably diagnosing subsegmental pulmonary embolism?	C The anatomic location does not contribute to the definition of
	("Pulmonary Artery"[majr] OR "Pulmonary Artery"[ti] OR "Pulmonary	subsegmental pulmonary embolism.
	Arteries"[ti])) AND ("pulmonary emboli"[ti] OR "pulmonary embolism"[ti] A Always.	D Other, namely [free text]:
	OR "pulmonary embolisms"[ti] OR "pulmonary embolus"[ti] OR "pe"[ti] B Never.
	OR "pes"[ti] OR "pulmonary thromboembolism"[ti] OR "Pulmonary C Only in selected cases, namely: [free text]	5. As a general rule, can the diameter of a certain pulmonary artery
	Embolism"[majr])) NOT ("Animals"[mesh] NOT "Humans"[mesh]) NOT	branch be used to diagnose subsegmental pulmonary embolism? If yes,
	11. What are the imaging criteria for the presence of subsegmental	at what diameter size should a pulmonary branch always be called
	A PPE N D I X 2 pulmonary embolism (several answers are possible)?	subsegmental?
	DEPLHI QUESTIONNAIRE SSPE, EXPERT A Contrast defect in a subsegmental artery visible in a single axial	A ≤1 mm.
	RADIOLOGISTS, FIRST ROUND slice.	B ≤2 mm.
		C ≤3 mm.
		D ≤4 mm.

1. In your opinion, should the definition of subsegmental pulmonary embolism on CTPA consist of the following parameters (multiple answers possible, please check all that apply)?

A The anatomic location within the pulmonary arterial system. B The diameter of the pulmonary artery affected. E Other, namely: [free text] F The diameter of the affected vessel does not contribute to the definition of subsegmental pulmonary embolism. B Contrast defect in a subsegmental artery visible in at least two subsequent axial slices. C Contrast defect in a subsegmental artery visible in at least two directions (axial and coronal or sagittal).

D Sudden contrast drop in a subsegmental artery while arteries of comparable diameter in the surrounding show a higher contrast. E 14. What is the best (systematic) approach of reading a CTPA scan that should be used to diagnose subsegmental pulmonary embolism?

ACKNOWLEDGMENTS

Funding information

G. Le Gal is the recipient of an Early Researcher Award from the Province of Ontario, a Heart and Stroke Foundation of Canada Ontario Mid-Career Investigator Award, and holds the Chair on Diagnosis of Venous Thromboembolism from the Department of Medicine, University of Ottawa.

RELATIONSHIP DISCLOSURE

The authors declare no conflicts of interests.

AUTHOR CONTRIBUTIONS

PLE and FAK performed the literature review, designed the questionnaires, performed the analyses, drafted the manuscript and designed the study. LJMK, C G, CLG, CMS, MC, and MV designed the questionnaires and critically reviewed the manuscript. All authors approved the final version of the manuscript.


Nicolas Kaczmarek

Astrid Mignon

teaching and research institutions in France or abroad, or from public or private research centers.

Investigation of the impact of relational framing on self-determination toward a distress tolerance task

Introduction

Investigation of the impact of relational framing on self-determination toward a distress tolerance task

N. Kaczmarek a et b (ORCID: 0000-0001-7341-2813), A. Mignon c (ORCID: 0000-0002-2163-5078), & A. Rousseau d (ORCID: 0000-0001-8660-3176) c a ULR 4072 -PSITEC Laboratory, University of Lille, Lille (59 000, France); b Clinique de la Mitterie, Lomme (59 160, France); c UMR 9193 -SCALab Laboratory, University of Lille, Lille (59 000, France); d EA7411 -CERPPS Laboratory, University of Toulouse, Toulouse (31 000, France).

Contact: nicolas.kaczmarek2.etu@univ-lille.fr

Methods & Results

Discussion

Behaviors could be framed in various ways. Noteworthy, they could be related to desired consequences through conditional framing (e.g., « If I practice yoga regularly, then I'll be in good shape ») or related to meaningful values through hierarchical framing (e.g., « Practicing yoga is a way of acting with openneness ») [1][2][3][4]. It is assumed that hierarchical framing would induce a higher quality motivation than conditional framing, and therefore promote behavioral commitment.

To date, only one study has tested this assumption [5]. It has shown that, compared to participants who followed conditional instructions (i.e., focus on the results), those who followed hierarchical (i.e., focus on the process) or a mixed of conditional and hierarchical instructions (i.e., focus on both) had better performance in a distress tolerance task, emotional experience, and task evaluation. However, it did not assess motivation's quality directly, through variables like self-determination [6][7].

The current study has two aims: 1) to replicate those findings [5] and 2) extend them by evaluating the impact of instruction framing on self-determination indicators.

References Contrary to expectations, this study did not replicate the effects of instructions framing on distress tolerance, emotional experience and task evaluation. However, the manipulation check revealed that instructions framing might have been unsufficient to modify in depth participants goal framing, which could explain those results.

Even though, this study revealed an unexpected impact of instructions framing on some self-determination indicators, with a superiority tendency of both conditional and hierarchical framing over mixed framing. It also revealed that, irrespective of their type, instructions framing might have some deleterious effects if they are mismatched with participants current goal framing. However, as goal framing was measured in post-test only, it is impossible to rule out some alternative interpretations.

Further studies are required to draw conclusion on those points. It seems crucial to develop more effective interventions to change participants goal framing, and to take into consideration the potential moderating effects of participants initial goal framing.

Manipulation check: Results revealed the negative impact of the target distress task on emotional experience (W < 7500, p ≤ .001, r rb > .30). However, it revealed no significant impact of instructions framing on participants conscious goal framing (χ² < 4.70, p > .09).

Evaluation of the impact of instructions framing on variables of the replicated study [5]: Results

revealed no significant impact of instructions framing on performance in the side and target distress tolerance task, emotional experience, and and task evaluation (χ² < 4.60, p > .1).

Evaluation of the impact of instructions framing on self-determination indicators :

Results revealed a significant impact of instructions framing on three of seven self-determination indicators (χ² > 6.8, p < .05; see Figure 1).

Evaluation of the interactive effects of instructions framing and participants goal framing:

Among others, results revealed that the impact of instructions framing on outcomes depends on participants conscious goal framing (see Figure 2 and 3 for examples).

Pre-test:

• Emotional experience (Mood Reactivity Scale and PA-NAS).

• Side distress tolerance task (a difficult anagram task).

Experimental phase:

• Target distress tolerance task (PASAT-C) with instructions using {Hierarchical; Conditional; Mixed} framing, depending on experimental condition.

Post-test:

• Conscious conditional and hierarchical goal framing.

• Emotional experience (similar to pre-test).

• Side distress tolerance task (similar to pre-test).

• General self-efficacy (GSE).

• Willingness to complete similar tasks.

• Task evaluation (comfort, positivity, perceived success and easiness)

  : [1] Barnes-Holmes, S. C. H. D., & Roche, B. (2001). Relational frame theory : A post-Skinnerian account of human language and cognition. Springer Science & Business Media. [2] Plumb, J. C., Stewart, I., Dahl, J., & Lundgren, T. (2009). In search of meaning : Values in modern clinical behavior analysis. The Behavior Analyst, 32(1), 85-103. [3] Torneke, N. (2010). Learning RFT : An introduction to relational frame theory and its clinical application. New Harbinger Publications. [4] Villatte, M., Villatte, J. L., & Hayes, S. C. (2015). Mastering the clinical conversation : Language as intervention. Guilford Publications. [5] Murthy, V. E., Villatte, M., & McHugh, L. (2019). Investigating the effect of conditional vs hierarchical framing on motivation. Learning and Motivation, 65, 33-42. [6] Deci, E. L., & Ryan, R. M. (2000). The" what" and" why" of goal pursuits : Human needs and the self-determination of behavior. Psychological inquiry, 11(4), 227-268. [7] Ryan, R. M., & Deci, E. L. (2019). Brick by brick : The origins, development, and future of self-determination theory. In Advances in motivation science (Vol. 6, p. 111-156). Elsevier. Participants: 167 participants (Age: M = 23.3, SD = 3.7; 54.49% of women) took part in this online study. Procedure: Open science: The material, data and analyses of this study are available on OSF, via the following QR-Code. The article will be linked to this page as soon as it is published.

Figure 1 .

 1 Figure 1. Effects of instructions on self-determination indicators. Legends. *p < .01 ; ** p < .05.

•Figure 2 .

 2 Figure 2. Correlations between hierarchical goal

Figure 3 .

 3 Figure 3. Correlations between conditional goal


Abdelahad Chraibi
email: abdelahad.chraibi@alicante.fr

David Delerue

Julien Taillard

Chaib Draa

Régis Beuscart

Arnaud Hansske

A Deep Learning Framework for Automated ICD-10 Coding

Keywords: Medical informatics, Natural Language Processing, Deep Learning, Automated Coding

The International Statistical Classification of Diseases and Related Health Problems (ICD) is one of the widely used classification system for diagnoses and procedures to assign diagnosis codes to Electronic Health Record (EHR) associated with a patient's stay. The aim of this paper is to propose an automated coding system to assist physicians in the assignment of ICD codes to EHR. For this purpose, we created a pipeline of Natural Language Processing (NLP) and Deep Learning (DL) models able to extract the useful information from French medical texts and to perform classification. After the evaluation phase, our approach was able to predict 346 diagnosis codes from heterogeneous medical units with an accuracy average of 83%. Our results were finally validated by physicians of the Medical Information Department (MID) in charge of coding hospital stays.

Introduction

To standardize and uniformize the protocol of reimbursement demands' treatment, the French insurance companies adopt the ICD-10, one of the most widely used classification system for classifying diagnoses and procedures. This system provides a codification of diseases and procedures, including a wide variety of signs, symptoms, and causes of injuries or diseases. ICD is used to translate diagnoses of diseases or other health problems into alphanumeric codes to facilitate data storage, research and analysis [1]. One EHR is attributed to each patient. At the end of an episode of care, all medical information of a patient result in textual documents such as discharge summaries, operative reports and progress notes, based on which diagnosis codes and procedure codes are assigned manually by human coders [2]. In this work, we considered the discharge summary (DS) which is a free-text document edited in the moment of patient departure. It describes the main health information about a patient during his hospitalization and provides final diagnosis, main interventions, treatments, etc. These free text notes have the advantage to be valuable sources of information and generally not subject to redaction's rules. Consequently, the use of abbreviations and synonyms, the grammatical errors, the spelling mistakes, the semantic ambiguities, the negation sentences, etc. entails issues and causes imprecision when manually matching ICD codes to those descriptions. The automated coding methods become more attractive to assist the physicians when reporting diagnoses codes to become more productive, efficient, accurate, and consistent. It can be considered as a classification problem corresponding to a variety of NLP approaches that transform narrative text of DSs into structured data frames before applying Machine Learning (ML) or DL techniques to determine the appropriate assignment of codes. Since 1990, several researchers have been interested to creating an automated system for assigning ICD codes to clinical reports. Depending on the type of data, multiple methods have been applied ranging from simple regression to advanced DL approaches with the objective to maximize performance and improve the quality of cares [START_REF] Farkas | Automatic construction of rule-based ICD-9-CM coding systems[END_REF]. In the last decade, Authors in [START_REF] Dermouche | Supervised topic models for diagnosis code assignment to discharge summaries[END_REF] experimented probabilistic topic models on collected DSs within urology and hematology services by comparing models issued from both classical ML approaches (Decision Tree, Naïve Bayes, and SVM) and modern NLP approaches (supervised Latent Dirichlet Allocation (LDA) and labeled LDA). In [START_REF] Xu | Multimodal Machine Learning for Automated ICD Coding[END_REF] authors presented a multimodal machine learning model to cope with different type of data including unstructured text, semi-structured text and structured tabular data for which Text Convolutional Neural Network (CNN), Bidirectional LSTM and decision trees were respectively applied. [START_REF] Baumel | Multi-label classification of patient notes: case study on ICD code assignment[END_REF] present a Hierarchical Attention bidirectional Gated Recurrent Unit for multi-label ICD-10 code assignment. Compared it to SVM-based one-vs-all model, continuous bag-of-words model and CNN model, the HA-GRU yield the most promising results when combined with careful data preparation steps. More recently, Almagro et al. [START_REF] Almagro | ICD-10 Coding of Spanish Electronic Discharge Summaries: An Extreme Classification Problem[END_REF] compared algorithms based on binary outputs, groups of subsets and extreme classification to automatically code Spanish electronic discharge summaries. [START_REF] Sonabend | Automated ICD coding via unsupervised knowledge integration (UNITE)[END_REF] considered 6 principal diseases and proposed an unsupervised knowledge integration algorithm to analyze clinical narrative notes via semantic relevance assessment. [START_REF] Zhou | Construction of a semi-automatic ICD-10 coding system[END_REF] conceived a semi-automatic ICD-10 coding system based on regular expressions with promising results. The aim of this work is to assist hospitals in coding their stays by proposing an intelligent system based on NLP and DL approaches to automatically and accurately translates the free-text diagnosis descriptions into ICD codes. The rest of the paper is organized as follows: section 2 presents the dataset and methods. Section 3 and 4 give experimental results and discussion. Finally, section presents conclusions and perspectives.

Methods

This study has been conducted using a database with more than 134K EHRs. The selected DSs correspond to 108K hospital admissions between 01-01-2016 and 31-03-2019 for which ICD-10 was used to assign diagnosis codes. All reports have been automatically de-identified using NLP technics including entity recognition techniques and regular expressions to constitute a large corpus of raw French text covering multiple medical specialties. This step has no influence on the accuracy of the final classification since even with the original corpus the results are the same. Following MID's recommendations, the DSs has been split in four groups according to the Length of Stay (LoS): group_0 (LoS = 0 night), group_1_2 (1 night <= LoS <= 2 nights), group_3_4 (3 nights <= LoS <= 4 nights) and group_5+ (LoS >= 5 nights). Some statistics related to the used dataset are presented in table 1, where for each group of stays the number of available DSs, the number of distinct used codes in the studied period and the number of initial selected codes for classification being coded at least 30 times over the period are presented in the second, third and fourth columns, respectively. To produce the classification model, we adopt a pipeline involving some major steps: text preprocessing, feature engineering, data splitting, parameter tuning, and creation of classification model. A first training iteration is performed using a list of ICD codes being selected according to their occurrence frequency in the corpus of DSs (at least 30 occurrences) to have representative data for each target in the training and the test datasets. Other code selection iterations are applied at the end of each 'model validation' phase based on the F1-Score threshold to keep the top predictable codes. For a classification, almost all machine learning algorithms need to take input in numerical inputs and have not the capacity to interpret other types of data if not transformed. In order to transform the text into numerical values, we used a hybrid Bag of Words (BoW) considering frequency of ICD thesaurus in the text. The thesaurus is a list of medical concepts created from the official description of ICD-10 diagnoses by extracting unique terms. Then, exact and partial string matching (i.e., similarity computation) for thesaurus terms is applied to each DS and a BoW vector is created based on terms frequency.

Feature selection methods are used to cope with popular data problems by reducing the data dimension with minimal loss of information. In this work, the variance threshold method is used to remove features with variation below a fixed cutoff without considering the relationship of features with the target variable. This cutoff value is manually fixed after several trials (i.e., training and evaluation cycles) and depends on the input data (e.g., in our case, it took a different value for each of the four groups). Artificial Neural networks (ANN) are popular classification methods been widely applied by many researchers to classify textual documents with different types of feature vectors. We adopted a multilayer ANN from DL4J library with backpropagation architecture, one of the most popular neural networks needing several parameters' configuration in the training phase [START_REF] Hecht-Nielsen | Theory of the backpropagation neural network[END_REF][START_REF] Kamath | Comparative study between traditional machine learning and deep learning approaches for text classification[END_REF]. In medical field, backpropagation is widely used to deal with different problem and data type (i.e., image, text, etc.) [START_REF]Backpropagation neural network for prediction of heart disease[END_REF][START_REF] Azmi | Breast cancer prediction based on backpropagation algorithm[END_REF][START_REF] Gupta | Medical diagnosis using back propagation algorithm[END_REF]. DL4J

proposes Arbiter [15], a framework dedicated to hyperparameter optimization using grid search and random search methods. The use of Arbiter made the hyperparameters optimization an automatable task to guaranty optimal training performances. To evaluate the performance of our models on the test dataset, we used the standard evaluation metrics of accuracy, precision, recall and F-score.

Results

A set of experiments were carried out in to evaluate the effectiveness of the proposed approach. Our IA models were trained and evaluated on retrospective French data considering just the principal diagnosis code as label for the whole dataset to address the problem as a multi-class classification task. Table 2 summaries the obtained results for each group. The first indicator is the number of predictable codes which represents the final selection after all training iterations with a F1-Score threshold of 50%. The performances are then reported in terms of accuracy, precision, recall, and F-measure for each group. Average values are all superior to 80% indicating a well prediction effectiveness and a good capacity to adapt to different specialty and different LoS. Table 3 represents the number of predicted codes per group of LoS in four intervals of F1-Score. We can note that 62% of the identified codes have a F1-score above 80% which is very high and encourages further investigation. We can observe the degressive predictive performance while increasing the LoS which could be explained by the correlation between the LoS and the number of diagnoses. In fact, the number of codes corresponding to short stays is limited. On the contrary, for long stays, the number of possible diagnoses is higher, as well as the number of associated diagnoses.

Discussion

Our proposed automated coding models work properly by predicting a principal diagnosis ICD-10 code to each patient's stay according to its LoS. The returned prediction could be a diagnosis code or a null value in the case if the code is not recognized in the list of supported ICD-10 codes by the model, or if the DSs raw text is not enough explicit to distinguish the diseases or the reasons of hospitalization. In these situations, a human intervention is needed to code the stay. The starting step was to limit the study to ICD-10 codes with at least 30 occurrences to reduce the noise and bias. If we look a little more closely to group_0 in Table 1, we retained approximatively 9% of codes (i.e., 145 of 1661) covering more than 82% of cases (i.e., 29005 DSs). The fixed objective was to retain the maximum of ICD codes while ensuring high performances. Thus, an iterative learning approach is adopted to prune the list of retained ICD codes to focus on the well predictable one and ignore the rest. For group_0, this represents 5% of codes and only 34% of the volume of the data set. The prediction of these codes seems to be more interesting for coders as they are more time consuming to manually code. It is difficult to compare our results with the literature since the problem definition and the finality are not always the same. For instance in terms of F1-Score, [START_REF] Almagro | ICD-10 Coding of Spanish Electronic Discharge Summaries: An Extreme Classification Problem[END_REF] obtained 46% for Spanish text classification, [START_REF] Dermouche | Supervised topic models for diagnosis code assignment to discharge summaries[END_REF] reached 74% in hematology unit with 30 diagnosis, [START_REF] Farkas | Automatic construction of rule-based ICD-9-CM coding systems[END_REF] attained 88% in radiology with 45 diagnosis, [START_REF] Sonabend | Automated ICD coding via unsupervised knowledge integration (UNITE)[END_REF] obtained 76% with 6 diagnosis, [START_REF] Zhou | Construction of a semi-automatic ICD-10 coding system[END_REF] reached 46% with 500 diagnosis while in our work, we obtained 83% with 346 diagnosis.

Conclusion

The application of automated systems for coding diagnosis has recently been renewed using DL methods. In this work we constructed a model based on DL and NLP approaches to automate the ICD-10 coding process from French raw text in discharge summaries. Several ANN were trained corresponding to four categories of stay duration on a real-life dataset. From the tens of thousands of ICD-10 codes, we studied only 613 that are referenced at least 30 times and finally select 346 having the higher predictive effectiveness. The performance and the relevance of our models has been approved by physicians of the MID in a French hospital. The major limitation of our work is the imbalanced and mislabeled data which really depends on the quality and size of available databases. Further works will investigate these limitations and try the use of Convolution Neural Network, considering corpus features as image pixels, and compare it to our previous results. This opens another path of research concerning the use of GPU servers instead of CPU to try to reduce the computation time and eventually accelerate the convergence of our models.

Table 1 .

 1 Statistics related to the dataset

	Group of reports	Number of DS	Count of distinct codes	Initial selected codes
	Group_0	35308	1661	145
	Group_1_2	20257	1866	109
	Group_3_4	14489	1675	111
	Group_5+	38475	2459	248

Table 2 .

 2 Evaluation of ICD-10 classification models

	Type of Stay	Predictable Codes	Accuracy	Precision	Recall	F1-Score
	Group_0	79	89,54%	90,04%	89,03%	89,30%
	Group_1_2	74	87,76%	85,37%	83,76%	83,92%
	Group_3_4	70	82,84%	83,67%	81,41%	81,70%
	Group_5+	123	76,70%	78,99%	76,25%	76,23%
	Average		84,21%	84,52%	82,61%	82,79%

Table 3 .

 3 Evaluation of ICD-10 classification models

	Type of Stay	F1 = 100%	F1-Score 80% <= F1 < 100%	50% <= F1 < 80%	F1 < 50%
	Group_0	5	56	18	0
	Group_1_2	4	48	22	0
	Group_3_4	3	36	31	0
	Group_5+	3	61	59	0