Guillaume Chassagnon

Maria Vakalopoulou

Enzo Battistella

Stergios Christodoulidis

Trieu-Nghi Hoang-Thi

Severine Dangeard

Eric Deutsch

Fabrice Andre

Enora Guillo

Nara Halm

Stefany El Hajj

Florian Bompard

Sophie Neveu

Chahinez Hani

Ines Saab

Aliénor Campredon

Hasmik Koulakian

Souhail Bennani

Gael Freche

Maxime Barat

Aurelien Lombard

Laure Fournier

Hippolyte Monnier

Téodor Grand

Jules Gregory

Yann Nguyen

Antoine Khalil

Elyas Mahdjoub

Pierre-Yves Brillet

Tran Ba

Valérie Bousson

Ahmed Mekki

Robert-Yves Carlier

Marie-Pierre Revel

Nikos Paragios

AI-driven quantification, staging and outcome prediction of COVID-19 pneumonia

Keywords: COVID 19 pneumonia, Artifial Intelligence, Deep Learning, Staging, Prognosis, Biomarker Discovery, Ensemble Methods

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

COVID-19 emerged in December 2019 in Wuhan, China [START_REF] Zhu | A novel coronavirus from patients with pneumonia in china[END_REF] caused by the SARS-Cov-2 virus, and it could lead to respiratory failure due to severe viral pneumonia [START_REF] Zhou | Clinical course and risk factors for mortality of adult inpatients with covid-19 in wuhan, china: a retrospective cohort study[END_REF]. The disease spread worldwide leading the World Health Organization to declare it as a pandemic in March 2020. One of the important actions to handle the pandemic is the fast and robust use of imaging along with clinical and biological comorbidities for the quantification and staging of patients upon their hospital admission. Being able to identify patients that need intubation upon admission is very important and essential for the management of a hospital's resources and the most optimal management of patients.

Moreover, a robust staging of the patients could also facilitate proper selection of patients for different treatments, reducing the unnecessary use of the hospital's intensive care units. To the best of our knowledge, currently the staging of the patients is mainly based on clinical and biological biomarkers such as age, sex and other comorbidities [START_REF] Zhou | Clinical course and risk factors for mortality of adult inpatients with covid-19 in wuhan, china: a retrospective cohort study[END_REF]Li et al., 2020a;[START_REF] Yuan | Association of radiologic findings with mortality of patients infected with 2019 novel coronavirus in wuhan, china[END_REF][START_REF] Tang | Abnormal coagulation parameters are associated with poor prognosis in patients with novel coronavirus pneumonia[END_REF][START_REF] Onder | Case-fatality rate and characteristics of patients dying in relation to covid-19 in italy[END_REF][START_REF] Guo | Diabetes is a risk factor for the progression and prognosis of covid-19[END_REF][START_REF] Terpos | Hematological findings and complications of covid-19[END_REF], while the role of imaging is mainly focusing on an estimation of the disease extent from CT scans. This estimation is mainly done by medical experts and hence suffers from inter-and intra-observer variability.

In this study, we investigated an automatic method (Figure 1) for COVID-19 disease quantification and staging that extracts and selects image characteristics directly from the CTs and fuse them with known clinical and biological markers. A variety of image characteristics are used providing insights about their use on patient staging and better disease understanding. The contributions of this study are three-folds: (i) a tool for automatic disease quantification based on 2D & 3D deep convolutional neural networks (CNNs) is developed, facilitating severity estimation for optimal patient care, (ii) a COVID19-specific holistic, highly compact multi-omics patient signature integrating imag-1 Guillaume Chassagnon & Maria Vakalopoulou & Enzo Battistella have equally contributed to this work 2 Corresponding author: n.paragios@therapanacea.eu ing, clinical, and biological data and associated comorbidities for automatic patient staging is presented, (iii) short and longterm prognosis for clinical resources optimization offering alternative/complementary means to facilitate triage are reported.

To the best of our knowledge this is among a few systematic efforts to quantify disease extent, to discover low dimensional and interpretable imaging biomarkers and to integrate them to clinical variables into short and long term prognosis of COVID-19 patients.

The paper is organized as follows: we first review related work mainly focusing on interstitial lung diseases (ILDs) diseases, which is followed by a description of all the components and implementation details of our method. We then present the acquired multi-center dataset, the evaluation setting, and the results of our experiments. Furthermore, we discuss in detail similarities and differences of our method with other recently proposed methods for quantification and staging of COVID-19.

Lastly, we present possible directions for future research.

Related Work

In this section, we provide a short review of previous studies on quantification of ILDs since COVID-19 and ILDs share a lot of similarities due to their diffuse pathological manifestations, such as ground glass opacities, band consolidations, and reticulations. Furthermore, we elaborate on studies that tackle severity or treatment response for such types of disease.

ILD Quantification

There are numerous studies proposed the last years on automatic quantification of ILD diseases using CT scans. The main goal of these studies is to develop models that are able to identify one or more types of different pathological lung tissue in ILD cases (such as ground glass, consolidation, honeycombing, etc) and successfully separate them from the healthy tissue. Initial efforts were mainly based on classification schemes. In particular, small patches including only a single tissue type were extracted and described using a number of handcrafted features focusing mainly on texture, then these features were used to train different machine learning classifiers [START_REF] Gangeh | A texton-based approach for the classification of lung parenchyma in ct images[END_REF][START_REF] Huber | Texture feature ranking with relevance learning to classify interstitial lung disease patterns[END_REF]. Following recent advances in deep learning and especially the success of convolutional neural networks (CNNs), researchers have recently employed such tools also in thoracic imaging tasks [START_REF] Chassagnon | Deep learning: definition and perspectives for thoracic imaging[END_REF], with ILD quantification being among them. The main advantage of CNNs is their ability to generate features automatically from the input, and create meaningful representations for the studied per time problems. In particular, a patch-based framework using a convolutional architecture is presented in [START_REF] Anthimopoulos | Lung pattern classification for interstitial lung diseases using a deep convolutional neural network[END_REF] for the automatic quantification of 5 different ILD patterns. Similarly, in [START_REF] Gao | Holistic classification of ct attenuation patterns for interstitial lung diseases via deep convolutional neural networks[END_REF] a patch-based approach is adapted to classify them in 6 different ILD patterns. Even if the method reported higher performance than other methods based on handcrafted features, the use of patches, besides being time consuming and ineffi-cient, does not exploit the texture of the entire lung.

Many of the already proposed CNNs have further been adapted to perform the task of semantic segmentation in an endto-end fashion instead of only image classification. Semantic segmentation refers to the task of infering a class for each of the pixels of an image instead of a single class for an image. Such models can be found in literature both in 2D [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF][START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] and 3D (C ¸ic ¸ek et al., 2016) and have also been used for ILD quantification. The authors of [START_REF] Vakalopoulou | Atlasnet: Multi-atlas non-linear deep networks for medical image segmentation[END_REF] present the coupling of 2D fully convolutional networks with deformable registration for the automatic quantification of systemic sclerosis disease.

Moreover, in [START_REF] Anthimopoulos | Semantic segmentation of pathological lung tissue with dilated fully convolutional networks[END_REF]

ILD Staging

Staging of patients with ILDs is very important as it could greatly help clinicians with their daily practice, while choosing treatment options [START_REF] Kolb | Staging of idiopathic pulmonary fibrosis: past, present and future[END_REF]. There have been a number of studies recently that try to identify and extract biomarkers from CT scans and associate them with the severity and treatment of ILD patients. These biomarkers are usually enhanced with clinical and physiological information to provide a scoring system as survival predictor. Among the variety of biomarkers, disease extent is one of the most powerful ones providing strong associations with severity and mortality [START_REF] Cottin | Interstitial lung disease associated with systemic sclerosis (ssc-ild)[END_REF][START_REF] Tomassetti | Staging systems and disease sever-ity assessment in interstitial lung diseases[END_REF]. Visual scoring of the disease extent on CT can be time-consuming [START_REF] Robbie | Evaluating disease severity in idiopathic pulmonary fibrosis[END_REF] highlighting the need for tools for automatic disease quantification. Moreover, except the disease extent, the location of the disease is also very important for the staging.

In [START_REF] Depeursinge | Automated classification of usual interstitial pneumonia using regional volumetric texture analysis in high-resolution ct[END_REF][START_REF] Christe | Computer-aided diagnosis of pulmonary fibrosis using deep learning and ct images[END_REF] the quantification of the disease is performed on different lung regions providing descriptive information about the severity of the ILD patients.

A variety of works report that radiomics, quantitative features extracted from the images, provide valuable information about the severity and response to treatment for different diseases including cancer [START_REF] Sun | A radiomics approach to assess tumour-infiltrating cd8 cells and response to anti-pd-1 or anti-pd-l1 immunotherapy: an imaging biomarker, retrospective multicohort study[END_REF]. These features could also provide very good tools for monitoring disease progression and therapeutic response [START_REF] Wu | Computed tomographic biomarkers in idiopathic pulmonary fibrosis. the future of quantitative analysis[END_REF]. In particular, in [START_REF] Bocchino | Performance of a new quantitative computed tomography index for interstitial lung disease assessment in systemic sclerosis[END_REF] intensity-based characteristics such as skewness and kurtosis were used together with disease extent to distinguish between systemic sclerosis patients with and without ILD diseases. Moreover, in [START_REF] Lafata | An exploratory radiomics approach to quantifying pulmonary function in ct images[END_REF] a variety of image radiomics and their relationship with the pulmonary function were investigated. Their results indicate that high-throughput radiomics data extracted from the lungs may be associated with pulmonary function as measured by common PFT metrics.

Methodology

In this section, we describe our AI driven scheme for the quantification of CT scans for patients suffering from COVID-19 pneumonia. Furthermore, we provide a method for the automatic selection and combination of multi-modal variables towards a holistic signature designed for the COVID-19 triage. In the following parts of this section, we provide details for all the different components of the system.

Lung, breast and heart segmentation

Segmentation of the heart and breast were extracted by using the software ART-Plan (TheraPanacea, Paris, France the final segmentation is obtained by using the inverse transformation (T ´1 i ) to back-project to the original anatomy, while a majority voting scheme is used to produce the final projection, combining the results of the different networks.

For the registration of the CT scans to the templates, an elastic registration framework based on Markov Random Fields was used, providing the optimal displacements for each template [START_REF] Ferrante | Deformable registration through learning of context-specific metric aggregation[END_REF]. In particular, the registration is performed by a non-linear transformer T , corresponding to the operator that optimizes in the continuous domain Ω the following energy,

EpT ; S , A i q " ij Ω k ÿ j"1 w j ρ j pS ˝T, A i qdΩ `α ij Ω ψpT qdΩ (1)
where ρ j corresponds to the different similarity metrics (sum of absolute difference, normalised cross correlation, etc) used to compare the source 3D volume to the target anatomy, w j are linear constraints factorizing the importance of the different metric functions and ψp¨q is a penalty function acting on the spatial derivatives of the transformation.

Concerning the details of the architecture, in our experiments each C i consists of a SegNet [START_REF] Badrinarayanan | Segnet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF] based architecture. More specifically, for the CovidE2D models the CT scans were separated on the axial view. Each network included 5 convolutional blocks, each one containing two Conv-BN-ReLU layer successions. Maxpooling layers were also distributed at the end of each convolutional block for the encoding part. Upsampling operators were used on the decoding part to restore the spatial resolution of the slices together with the same successions of layers.

CovidE3D component

To fully exploit the 3D nature of our dataset, the second component of our proposed CovidENet is based on a 3D fully convolutional network similar to 3D-UNet (C ¸ic ¸ek et al., 2016). In order to train this model, 3D sub-volumes of the CT scan that fully included without any downsampling either the left or right lung were extracted. The corresponding sub-volumes were also extracted from the ground truth annotation masks. To this end, we trained the model with the CT scan sub-volume as input and the annotation as target. As far as the architecture is concerned, the model consisted of five blocks with a down-sampling operation applied every two consequent Conv3D-BN-ReLU layers.

Additionally, five decoding blocks were utilized for the decoding path, were at each block a transpose convolution was performed in order to up-sample the input. Skip connections were also employed between the encoding and decoding paths. The dimensions of the input that corresponded to the spatial dimensions of the CT scan and consequently the spatial dimensions of the features maps were not bound to some fixed dimension in order to feed the entire left/ right lung volumes. As such, 3D volumes of arbitrary spatial dimensions could be fed to the network and thus the batch size was fixed to 1.

Holistic Multi-Omics Profiling & Staging

In order to combine disease extent with disease characteristics and patients commodities, we investigate a variety of imaging characteristics extracted using disease, cardiac and lung segmentations. These imaging characteristics (radiomics) were then combined with meaningful clinical and biological indicators that have been reported to be associated prognosis of COVID-19. Patient charts were reviewed to assess short term (4 days after the chest CT) and long term prognosis (31 days after the chest CT). For the staging task, patients were divided in 2 groups: those who died, or required mechanical ventilation either at the initial or at a subsequent admission as severe cases (S), and the rest as non-severe cases (NS). For the prognosis task, three distinct subpopulations were defined: those who had a short term negative (SD = short-term deceased) outcome (deceased within 4 days after admission), those who didn't recover (LD= long-term deceased) within 31 days after the chest CT (either died after day 4 or still intubated at day 31) and those who recovered (LR= long-term recovered). The last two groups formed the short intubated (SI) group of patients.

Features Extraction

Radiomics features were extracted from the CT scans using the previously described segmentations of the disease, lung and heart. As a preprocessing step, all images were resampled by cubic interpolation to obtain isometric voxels with sizes of 1mm. Subsequently, disease, lung and heart masks were used to extract 107 radiomic features for each of them (left and right lung were considered separately both for the disease extent and entire lung). These features included first order statistics (maximum attenuation, skewness, 90th percentile etc), shape features (surface, maximum 2D diameter per slice, volume etc) and texture features (GLSZM, GLDM, GLRLM etc). For the extrac-

tion, the open source Pyradiomics library was used [START_REF] Van Griethuysen | Computational radiomics system to decode the radiographic phenotype[END_REF].

Two other image indexes were also calculated, namely disease extent and fat ratio.

Holistic Biomarker Selection

Using all the calculated attributes (clinical, biological, imaging) we constructed a high dimensional space of size 543, including clinical/biological variables. A min-max normalization of the attributes was performed by calculating the minimum and maximum values for the training and validation cohorts. The same values were also applied on the test set.

To prevent overfitting and discover the most informative and robust attributes for the staging and prognosis of the patients we propose a robust biomarker selection process. Feature selection is very important for classification tasks and has been used

widely in literature especially for radiomics [START_REF] Sun | A radiomics approach to assess tumour-infiltrating cd8 cells and response to anti-pd-1 or anti-pd-l1 immunotherapy: an imaging biomarker, retrospective multicohort study[END_REF].

COVID-19 Multi-Omics Profiling Signature

Using the aforementioned selection method, we have extracted 15 different radiomics features. These features belong to: features from imaging and in particular from the disease regions (5 features), lung regions (5 features) and heart features

(5 features). On these radiomics features biological and clinical data were added (6 features: age, sex, high blood pressure (HBP), diabetes, lymphocyte count and CRP level) and image indexes (2 features: disease extent and fat ratio). At the end our biomarker consisted of 23 features in total.

Regarding imaging features, we identified the following features as more important for the prognosis of the COVID-19 patients. These features include both first and second order statistics together with some shape features.

• Length on the Axis.

The selected disease area features capture both disease extent and disease textural heterogeneity. Disease textural heterogeneity is associated with lesions, the presence of which generates imaging pattern more complex than pure ground glass opacities usually found in mild disease. The selected lung features capture the dispersion and heterogeneity of lung densities, both of which may reflect the presence of an underlying airway disease such as emphysema but also the presence of sub-radiological disease. Lastly, the selected heart features can be seen as a surrogate for cardiomegaly and coronary calcifications.

Staging Mechanism

The staging/prognosis component was addressed using an ensemble learning approach. Similarly to the biomarker extraction, the training data set was subdivided into training and validation sets on the principle of 80%-20%. This subdivision was performed such that the distribution of classes between the two subsets was identical to the observed one. We determine the optimal outcome. The weights granted to each selected classifier determined according to the rank of this classifier on validation regarding balanced accuracy weighted with a higher importance the best performing algorithms. Then, the selected classifiers were retrained using the entire training set,

and their performance was reported on the external test cohort.

Prognosis Mechanism

To perform the short-term deceased (SD), long-term Deceased (LD), long term recovered (LR) classification task, a SD/SI (SI: intubated at 4 days) classifier and a LD/LR classifier was applied in a hierarchical way, performing first the short-term staging and then the long-term prognosis for patients classified as in need of mechanical ventilation support. More specifically, a majority voting method was applied to classify patients into SD and SI cases. Then, another hierarchical structure was applied on the cases predicted as SI only to classify them into the ones who didn't recover within 31`days of mechanical ventilation (LD) and the ones who recovered with 30 days on mechanical ventilation (LR).

Implementation Details

Deep Learning Segmentation

In order to train all the models, each CT scan was normalized by cropping the Hounsfield units in the range r´1024, 300s.

A variety of hyperparameters including loss functions, learning rates, optimizers had been tested and in this section we report the ones with the best performance for each compo- the implementation of all the models Scikit-learn library was used [START_REF] Pedregosa | Scikitlearn: Machine learning in python[END_REF].

Dataset

This retrospective multi-center study was approved by our In- 3). In addition to the CT examination -when available -patient sex, age, and body mass index (BMI), blood pressure and diabetes, lymphocyte count, CRP level and D-dimer level were also collected (Table 3).

For short-term outcome assessment, patients were divided into 2 groups: those who died or were intubated in the 4 days following the CT scan composed the severe short-term outcome subgroup, while the others composed the non-severe short-term outcome subgroup. For long-term outcome, medical records were reviewed from May 7th to May 10th, 2020 to determine if patients died or had been intubated during the period of at least one month following the CT examination. The data associated with each patient (holistic profiling), as well as the corresponding outcomes both in terms of severity assessment as well as in terms of final outcome and readers assessment will be made publicly available. Annotation consisted of manual delineation of the disease and manual segmentation of the lung without using any preliminary lung segmentation. Furthermore, 3 radiologists, an internationally recognized expert with 20`years of experience in thoracic imaging (Reader A ), a thoracic radiologist with 7`years of experience (Reader B ) and a resident with 6-month experience in thoracic imaging (Reader C ) were asked to perform a triage (severe versus non-severe cases) and for the severe cases (short-term deceased versus short-term intubated) prognosis process to predict the short-term outcome.

Experimental results

Statistical Analysis

The dice similarity score (DSC) was calculated to assess the similarity between the 2 manual segmentations of each CT exam of the test dataset and between manual and automated segmentations. The Hausdorff distance (HD) was also calculated to evaluate the quality of the automated segmentations in a similar manner. Disease extent was calculated by dividing the volume of diseased lung by the lung volume and expressed in percentage of the total lung volume. Disease extent measurement between automated and manual segmentations were compared using paired Student's t-tests. Similarly, correlation between disease extent measurents from Covid2D, Covid3D, CovidENet and manual segmentations were compared using Spearman correlation coefficient.

For the stratification of the dataset into the different categories, classic machine learning metrics, namely balanced accuracy, weighted precision, and weighted specificity and sensitivity were utilized.

Disease Quantification

The evaluation of CovidENet together with its components and the comparison with the 2 independent experts is summarised in Table 4. CovidE2D component performed better than the CovidE3D for the segmentation of COVID-19 disease. This is indicated by the higher DSC and HD values achieved by the CovidE2D component (Figure 4). However, their fusion 4). The mean/median DSCs between the Examples of disease segmentations are presented in Figure 6.

One can observe that the segmentations provided by CovidENet are very close to the ones generated by the experts. In particular, the algorithm detects the diseased regions even in the case that they are relatively small capturing all the different opacities of COVID-19 such as ground glass and consolidation.

COVID-19 Holistic Multi-Omics Profiling & Staging

The holistic COVID-19 pneumonia signature is presented in (Table 5) along with the correlations with outcome. The average signature for the severe and non-severe case in the test set are presented in Figure 7. Consensus ensemble learning through majority voting was used to determine the subset of AI methods that have robust, reproducible performance with good generalization properties. Human "reader+++" was used as a reference through consensus among three chest radiologists (resident, 7`years of experience, 20`years of experience in thoracic imaging). Our method aiming to separate patients with S/NS outcomes had a balanced accuracy of 70% (vs 67% for human readers consensus), a weighted precision of 81% (vs 78%), a weighted sensitivity of 64% (vs 70%) and specificity of 77% (vs 64%) and outperformed the consensus of human readers (Figure 7, Table 6). Our method successfully predicted 81% of the severe/critical cases opposed to only 61% for the consensus reader. The superiorty of our approach is also indicated by the higher AUC reported (0.76), in comparison with the one achieved by the different readers (0.69). Severe cases as depicted in Figure 7 referred to diabetic men, with higher level of volume/heterogeneity of disease and C-reactive protein levels. Moreover, as indicated in Figure 7 the non-uniformity on GLRLM for both lung and disease together with the disease extent seems to contribute considerable to the classification of the patients to NS versus S cases.

Prognosis & Staging

The COVID-19 pneumonia pandemic spiked hospitalizations, while exerting extreme pressure on intensive care units.

In the absence of a cure, staging and prognosis is crucial for clinical decision-making for resource management and experimental outcome assessment, in a pandemic context. Our objective was to predict patient outcomes prior to mechanical ventilation support. The proposed ensemble classifier aiming to predict the SD/(LD or LR) had a balanced accuracy of 88%

(vs 81% for human readers consensus), a weighted precision of 94% (vs 87%), a weighted sensitivity of 94% (vs 88%) and specificity of 81% (vs 75%) and outperformed consensus of hu- man readers (Table 6). Our method for prognosis of SD/ LD/ LR had a balanced accuracy of 71%, a weighted precision of 77%, a weighted sensitivity of 74% and specificity of 82% to provide full prognosis (Figure 8). Concerning the performance of our method for the classification of LD and LR patients (Table 7), our ensemble classifier reports a balance accuracy of 69%, a weighted precision of 76% a weighted sensitivity of 74% and a weighted specificity of 65%. As indicated also in Figure 8 the performance of our method reach an AUC of 0.86 for the SD, a 0.86 for the LR and 0.76 for the LD classes. Moreover, the age, HBP and lung non uniformity on the GLSZM seems to associate better for this task.

Moreover, in order to assess the impact of each feature category on the implemented models we performed an ab- and percentage of opacity. Again, a direct comparison could not be reported, as the evaluation of the method was not performed using DSC or HD, since the performance was measured on the ability to regress the proposed scores.

Assessing the severity of COVID-19 patients is also a very quickly evolving topic in the medical community with some methods being currently under review. Extracting valuable information from the imaging using recent advances is very important and could potentially facilitate the clinical practice.

Starting with, disease extent is known to be associated with severity (Li et al., 2020a;[START_REF] Yuan | Association of radiologic findings with mortality of patients infected with 2019 novel coronavirus in wuhan, china[END_REF] directly from CT scans. In [START_REF] Bai | Predicting covid-19 malignant progression with ai techniques[END_REF], the authors proposed a deep learning pipeline based on LSTMs using 2D CT slices and a fusion of imaging and clinical information to assess the severity and progression of COVID-19 patients. The proposed method reports an accuracy of 89.1% on a test cohort of 80 patients, outperforming classical machine learning techniques. Besides having a smaller test cohort, our method explores features that are interpretable helping better understanding of the disease and providing additional information for the staging of the patients. Moreover, recently [START_REF] Lassau | Aibased multi-modal integration of clinical characteristics, lab tests and chest cts improves covid-19 outcome prediction of hospitalized patients[END_REF] proposed the assessment of severity using a deep learning tool achieving an AUC of 0.76 on a completely independent cohort. Again even if we can not perform a direct comparison our method reports similar performance in a completely independent cohort, while it is based on interpretable features extracted from different regions. Finally, in [START_REF] He | Synergistic learning of lung lobe segmentation and hierarchical multi-instance classification for automated severity assessment of covid-19 in ct images[END_REF] a 2D deep learning based approach using a multi-task learning is presented in order to separate COVID-19 patients to severe and non severe cases.

Clinical Impact

To the best of our knowledge this study is the first to have developed a robust, holistic COVID-19 multi-omics signature for disease staging and prognosis demonstrating an equivalent/superior-to-human-reader performance on a multicentric data set. Our approach complied appropriate data collection and methodological testing requirements beyond the existing literature [START_REF] Mei | Artificial intelligence-enabled rapid diagnosis of patients with covid-19[END_REF]. The proposed holistic signature harnessed imaging descriptors of disease, underlying lung, heart and fat as well as biological and clinical data. Among them, disease extent is known to be associated with severity (Li et al., 2020a;[START_REF] Yuan | Association of radiologic findings with mortality of patients infected with 2019 novel coronavirus in wuhan, china[END_REF], disease textural heterogeneity reflects more the presence of heterogenous lesions than pure ground glass opacities observable in mild cases. Heart features encode cardiomegaly and cardiac calcifications. Lung features show patients with severe disease having greater dispersion and heterogeneity of lung densities, reflecting the presence of an underlying airway disease such as emphysema and the presence of sub-radiological disease. Among clinical variables, a higher CRP level, lymphopenia and a higher prevalence of hypertension and diabetes were associated with a poorer outcome, consistent with previous reports [START_REF] Zhou | Clinical course and risk factors for mortality of adult inpatients with covid-19 in wuhan, china: a retrospective cohort study[END_REF][START_REF] Guo | Diabetes is a risk factor for the progression and prognosis of covid-19[END_REF][START_REF] Terpos | Hematological findings and complications of covid-19[END_REF]. Interestingly, age was less predictive of disease severity than of poor outcome in severe patients. This is linked to the fewer therapeutic possibilities for these generally more fragile patients. Lastly, the average body mass index 8. An ablation study of the different selected features. A leave-one-out method has been applied by removing one feature sequentially to test the features importance and the performance robustness. Note: a) D0: disease extent, b) D1: disease variables that are shape/geometry related, c) D2: disease variables that are tissue/texture, d) O1: heart/lungs variables that are shape/geometry related, e) O2: heart/lungs variables that are tissue/texture, f) B1: age, gender, biological/obesity/diabetes/fat/high blood pressure. LD = long-term-deceased; LR = long-term deceased; NS = non severe; S = severe; SI = short-term intubation; SD = short-term deceased.

with severe outcomes [START_REF] Huang | Serial quantitative chest ct assessment of covid-19: Deep-learning approach[END_REF][START_REF] Chaganti | Quantification of tomographic patterns associated with covid-19 from chest ct[END_REF] and an editorial described the measurement of anthropometric characteristics as crucial to better estimate the risk of complications [START_REF] Stefan | Obesity and impaired metabolic health in patients with covid-19[END_REF]. However a meta-analysis showed that whereas being associated with an increased risk of COVID-19 pneumonia, obesity was paradoxically associated with reduced pneumonia mortality [START_REF] Wynants | Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal[END_REF]. Overall, the combination of clinical, biological and imaging features demonstrates their complementary value for staging and prognosis.

Future Work

In conclusion, artificial intelligence enhanced the value of chest CT by providing fast accurate and precise disease extent

Fig. 1 .

 1 Fig. 1. Overview of the method for automatic quantification, staging and prognosis of COVID-19. Our study includes 8 independent cohorts, resulting in 693 Covid-19 patients in total. A variety of clinical and biological attributes were collected and combined with imaging biomarkers for short and long term prognosis of COVID-19 patients. Our study is composed by three different steps: (i) Proposing a state-of-the-art deep learning based consensus of 2D & 3D networks for automatic quantification of COVID-19 disease, reaching expert-level annotations, (ii) A radiomics study integrating interpretable features extracted from disease, lung and heart regions. A consensus-driven COVID-19 low dimensional bio(imaging)-holistic profiling and staging signature has been proposed using robust machine learning algorithms, fusing imaging, clinical and biological attributes. & (iii) An ensemble of robust linear & non-linear classification methods for the proper identification of patients that need intubation.

Fig. 2 .

 2 Fig. 2. Correlation between body mass index (BMI) and fat ratio.

  The disease extent was calculated as the percentage of lung affected by the disease in respect to the entire lung volume. The disease components were extracted by calculating the number of individual connected components for the entire disease regions. The fat ratio, calculated as an indicator of obesity, was used as a surrogate of the body mass index and calculated by dividing the volume of thoracic fat by the volume of the thorax. The index was defined in an unsupervised manner. To obtain fat segmentation, CT scans were smoothed using a Gaussian kernel with a standard deviation of 2. Then, a threshold of the densities in the range of [29, 130] was applied on the smoothed CTs to isolate the fat regions. Fat masks were calculated starting from the highest to the lowest part of the lungs. In order to avoid gender bias, we used breast segmentation to exclude breast fat. Then the volume of the fat segmentation was divided by the body volume. To validate this morphometric measurement we assessed its correlation with BMI in the 362 patients for which BMI was available and we found a strong correlation using Pearson correlation (r= 0.64; pă 0.001; Figure 2).

First

  , the training data set was subdivided into training and validation on the principle of 80%-20% maintaining the distribution of classes between the two subsets identical to the observed one. To perform features selection, we have created 100 subdivisions on this basis and evaluated variety of classical machine learning -using the entire feature space -classifiers such as Decision Tree Classifier, Linear Support Vector Machine, XG-Boosting, AdaBoost and Lasso. These classifiers were trained and validated to distinguish between severe (S) and non severe (NS) cases. In addition to these 5 classifier-based feature selection approaches, we also considered statistics-based approaches based on Mutual Information, Chi-squared statistics and Univariate linear regression tests. Each of these methods was used to assess the importance of the features regarding outcome prediction. Features were ranked according to their prevalence, the total number of splits they were selected in, for each of the methods. Our experiments indicated that different classifiers highlight different attributes as important. In order to take advantage of the different feature selection properties, we adopted a consensus feature selection method by selecting features with the highest sum of prevalence over all methods. Besides, to maintain structural properties, we selected the features in the top 5 prevalence in each region.

  Disease areas: Non-Uniformity of the Gray Level Dependence Matrix (GLDM), Dependence Non-Uniformity of the GLDM, Mesh Volume, Voxel Volume, Non-Uniformity of the Gray level Run Length Matrix (GLRLM). • Lung areas: Kurtosis, Mean Absolute Deviation, Zone Emphasis of the GLSZM, Non-Uniformity of the GLSZM, Variance of the GLSZM. • Heart areas: Maximum 2D diameter Slice, Non-Uniformity of the GLSZM, Sphericity, Flatness, Minimum

  have used 10-fold cross validation on this basis and evaluated the average performance of the following supervised classification methods: Nearest Neighbor, {Linear, Sigmoid, Radial Basis Function (RBF), Polynomial Kernel} Support Vector Machines (SVM), Gaussian Process, Decision Trees, Random Forests, AdaBoost, XGBoosting, Gaussian Naive Bayes, Bernoulli Naive Bayes, Multi-Layer Perceptron & Quadratic Discriminant Analysis. These classifiers have been trained using the identified holistic signature. For each binary classification task a consensus model was designed selecting the top 5 classifiers with acceptable performance, ą 60% in terms of balanced accuracy, as well as coherent performance between training and validation, performance decrease ă 20% for the balanced accuracy. The selected models were trained and combined together through a weighted winner takes all approach to

Fig. 3 .

 3 Fig. 3. Training and validation curves for one template/ atlas (A i ) of Co-vidE2D and the CovidE3D.

  tients) were included. Data of 536 patients from 5 centers (A, B, C, D and H) were used for training and those of 157 patients from 3 other centers (E, F and G) composed an independent test set (Table

Fifteen

  radiologists (GC, TNHT, SD, EG, NH, SEH, FB, SN, CH, IS, HK, SB, AC, GF and MB) with 1 to 7 years of experience in chest imaging participated in the data annotation which was conducted over a 2-week period. For the training and validation set for the COVID-19 radiological pattern segmentation, the whole CT examinations were manually annotated slice by slice. On each of the 50 cases (23, 423 axial slices) composing this dataset, all the COVID-19 related CT abnormalities (ground glass opacities, band consolidations, and reticulations) were segmented as a single class. Additionally, the whole lung was segmented to create another class (lung). To facilitate the collection of the ground truth for the lung anatomy, a preliminary lung segmentation was performed with Myrian XP-Lung software (version 1.19.1, Intrasense, Montpellier, France) and then manually corrected. For the test cohort, 20 CT slices equally spaced from the superior border of aortic arch to the lowest diaphragmatic dome were selected in a total of 130 patients composing a 2, 600 images dataset. Each of these images were systematically annotated by 2 out of the 15 participating radiologists who independently performed the annotation.

Table 4 .

 4 Fig. 4. Box-Plot in terms of DSC and HD between CovidENet and its individual components, Obs1 & Obs2. One can observe that CovidENet (blue) performs better and closer to Obs1-Obs2 (red) DSC and HD metrics than its individual components CovidE2D & CovidE3D.

Fig. 5 .

 5 Fig. 5. Plots indicating the correlation between disease extent automatically measured and the average disease extent measured from CovidE2D, CovidE3D and CovidENet respectively and the manual segmentation. Disease extent is expressed as the percentage of lung affected by the disease. The red line shows a perfect correlation (Spearman R =1). Spearman correlation coefficients are displayed for each comparison.

Fig. 6 .

 6 Fig. 6. Qualitative analysis for the comparison between manual and the proposed CovidENet disease quantification. Delineation of the diseased areas on chest CT in different slices of COVID-19 patients. From left to right: Input, CovidENet-segmentation, Obs1-segmentation, Obs2-segmentation.

Table 5 .

 5 Correlation between outcome and the 23 features of the holistic COVID19 signature. Note: GLSZM =Gray Level Size Zone Matrix, GLRLM = Gray Level Run Length Matrix, GLDM = Gray Level Dependence Matrix, LD = long-term-deceased, LR = long-term deceased, NS = non severe, S = severe, SI = short-term intubation , SD = short-term deceased.

Fig. 7 .

 7 Fig. 7. COVID-19 Holistic Multi-Omics Signature & Staging: Spider chart representing average profiles (average values of the variables after normalization between 0 and 1) with respect to severe versus non-severe separation are shown along with prevalence of biomarkers (diameter of the circle). The prevalence of the biomarker corresponds here to the number of selections of the biomarker during the feature selection process. Classification performance, confusion matrices and area under the curve with respect to the proposed method and the consensus of expert readers (reader+) are reported on the right side. Selective associations of features with outcome (NS/S) are shown at the top right of the figure (box plots).

Fig. 8 .

 8 Fig. 8. Short & Long Term Prognosis. Spider chart representing average profiles (average values of the variables after normalization between 0 and 1) with respect to the short deceased (SD), long deceased (LD) and long recovered (LR) classes are shown along with their correlations with the outcome (diameter of the circle). The presented correlation corresponds to Pearson Correlation for LR/LD outcome (Table5). Classification performance, confusion matrices and area under the curve with respect to the proposed method and -when feasible -the consensus of expert readers (reader+) are reported on the right side. ROC curves correspond to one-vs-all classification of the SD/LR/LD patients. Selective associations of features with final outcome (LD/LR) are shown at the bottom of the figure (box plots).

  quantification and by helping to identify patients with severe short-term outcomes. This could be of great help in the current context of the pandemic with healthcare resources under extreme pressure. Beyond the diagnostic value of CT for COVID-19, our study suggests that AI should be part of the triage process. Our methodology provides a deep learning-based pipeline that provides disease quantification comparable to the human experts, while it explores image characteristics, fusing them

Table 1 .

 1 Acquisition and reconstruction parameters of the dataset used in this study. Note: For quantitative variables, data are presented as mean ˘standard deviation, and numbers in brackets indicate their range. CT = Computed Tomography ; CTDIvol = volume Computed Tomography Dose Index ; DLP = Dose Length Product.

	important motion artifacts. No patient was intubated at the time
	of the CT acquisition. A total of 693 patients, after all the ex-
	clusion criteria were applied, formed the full dataset (321, 360
	CT slices).
	Chest CT exams were acquired on 4 different CT models
	from 3 manufacturers (Aquilion Prime from Canon Medical
	Systems, Otawara, Japan; Revolution HD from GE Health-
	care, Milwaukee, WI; Somatom Edge and Somatom AS+ from
	Siemens Healthineer, Erlangen, Germany). The different acqui-
	sition and reconstruction parameters are summarized in Table 1.
	CT exams were mostly acquired at 120 (n=481{693; 69%) and
	100 kVp (n=186{693; 27%). Images were reconstructed us-
	ing iterative reconstruction with a 512 ˆ512 matrix and a slice
	thickness of 0.625 or 1 mm depending on the CT equipment.

stitutional Review Board (AAA-2020-08007) which waived the need for patients' consent. Patients diagnosed with COVID-19 from March 4th to April 5th from eight large University Hospitals were eligible if they had positive reverse transcription polymerase chain reaction (PCR-RT) and signs of COVID-19 pneumonia on unenhanced chest CT. Only the CT examination that was performed at initial evaluation was included in our dataset. Exclusion criteria were (i) contrast medium injection and (ii) Only the lung images reconstructed with high frequency kernels were used for analysis. For each CT examination, dose length product (DLP) and volume Computed Tomography Dose Index (CTDIvol) were collected. tients; E: 50 patients, F: 30 patients) were included to compose the test dataset (Table 2). The patients from the training cohort were annotated slice-by-slice, while the patients from the testing cohort were partially annotated on the basis of 20 slices per exam covering in an equidistant manner the lung regions. The proportion between the CT manufacturers in the datasets was pre-determined in order to maximize the model generalizability while taking into account the data distribution. Patient characteristics for the automatic quantification of COVID-19 disease. Note: For quantitative variables, data are presented as mean ˘standard deviation, and numbers in brackets indicate their range. CT = Computed Tomography; CTDIvol = volume Computed Tomography Dose Index; DLP = Dose Length Product.

Table 3 .

 3 Patient characteristics for the automatic staging and prognosis tools. Note: For quantitative variables, data are presented as mean standard deviation, and numbers in brackets indicate their range. For qualitative variables, data are numbers of patients, and numbers in parentheses are percentages. CT = Computed Tomography, CTDIvol = volume Computed Tomography Dose Index; DLP = Dose Length Product. *Available clinical data: n " 692 for diabetes and high blood pressure(leading to 0.19% of missing data on the training set), n " 674 for lymphocyte count (leading to 2.05% and 5.10% of missing data on the training and test sets respectively), n " 654 for CRP (leading to 4.66% and 8.92% of missing data on the training and test sets respectively), n " 362 for Body Mass Index, and n " 339 for D-dimers.

			Training/Validation Dataset	Test Dataset
			(Centers A+B+C+	(Centers E+F+G;	p value
			D+H; n=536)	n=157)
	Age (y)		63 ˘16 [22-98]	62 ˘17 [17-98]	0.495
	No. of Men	374(70)	103(78)	0.321
	High	blood	235 (44)	71 (45)	0.773
	pression*
	Diabetes*	97 (18)	37 (24)	0.888
	Body mass in-	27.7 ˘5.1 [17.0-44.1]	27.1 ˘5.1 [14.5-42.7] 0.390
	dex (kg{m 2 )*
	Lymphocyte	1.3 ˘2.7 [0.1-48.5]	1.3 ˘3.3 [0.23-41.0]	0.915
	count
	(ˆ10 9 {L)*
	CRP (mg/L)*	104.3 ˘82.9 [1.0-430.7]	94.2 ˘74.8 [2.0-342] 0.166
	D-dimers (mi-	2458 ˘6533 [181-86248]	815 ˘924 [168-6138] ă 0.001
	crog/L)*
	Disease	ex-	22.2 ˘18.4 [0.0-89.8]	24.0 ˘18.7 [1.1-89.8]
	tent**
	Fat ratio on CT 18.6 ˘5.9 [1.7-42.3]	18.3 ˘5.5 [2.7-30.6]	0.589
	Short-term			0.994
	outcome
	Deceased	28(5)	8(5)
	Intubated	80(15)	23(15)
	Alive and Not	428(80)	126(80)
	Intubated
	Follow-up du-
	ration
	Worsening			0.554
	during follow-
	up***
	Deceased	69(13)	17(11)
	Intubated	68(13)	22(14)
	DLP		181 ˘115 [43-755]	218 ˘106 [ 43-679 ]	ă 0.001
	(mGy.cm)
	CTDIvol		4.9 ˘3.2 [1.0-19.8]	6.1 ˘3.0 [1.2-18.0]	ă 0.001
	(mGy)

**Percentage of lung volume on the whole CT. ***Data available for 688 patients.

  lation study by successively removing one category of features from the 6 categories defined for each classification task. Results are presented in Table8. The feature cate-

	gories were identified as follows: a) D0: disease extent, b)
	D1: disease variables that are shape/geometry related, c) D2:
	disease variables that are tissue/texture, d) O1: heart/lungs
	variables that are shape/geometry related, e) O2: heart/lungs
	variables that are tissue/texture, f) B1: age, gender, biologi-
	cal/obesity/diabetes/fat/high blood pressure. One can observe
	that the Clinical Only category contributes a lot for the separa-
	tion of SD/LD/LR while for the NS/S cases their contribution
	is marginal, in contrary to the other imaging characteristics.
	6. Discussion
	AI-enhanced imaging, clinical and biological information
	proved capable to identify patients with severe short/long-term
	outcomes, bolstering healthcare resources under the extreme
	pressure of the current COVID-19 pandemic. The information

obtained from our AI staging and prognosis could be used as an additional element at admission to assist decision making.

Table 6 .

 6 Prognosis of medical experts and their consensus for the Non Severe (NS) versus Severe (S), Intubated (SI) versus Deceased (SD) and NS/SI/SD patients Note: Classification Performance Reader A (Senior), Reader B (Established), Reader C (Resident), Reader ```( Consensus among Human Readers), Reader ´´´( Average performance of Human Readers).

		Balanced	Weighted	Weighted	Weighted
		Accuracy	Precision	Sensitivity	Specificity
	NS/SI/SD
	Reader A	0.62	0.77	0.68	0.69
	Reader B	0.59	0.75	0.67	0.65
	Reader C	0.61	0.76	0.68	0.62
	Reader ```0 .63	0.77	0.70	0.67
	Reader ´´´0 .61 ˘0.01 0.76 ˘0.01 0.68 ˘0.01 0.66 ˘0.03
	Proposed	0.67	0.81	0.63	0.80
	NS/S
	Reader A	0.69	0.79	0.70	0.67
	Reader B	0.66	0.77	0.70	0.62
	Reader C	0.65	0.76	0.70	0.60
	Reader ```0 .67	0.78	0.70	0.64
	Reader ´´´0 .67 ˘0.01 0.77 ˘0.01 0.70 ˘0.01 0.63 ˘0.03
	Proposed	0.70	0.81	0.64	0.77
	SI/SD
	Reader A	0.81	0.87	0.88	0.75
	Reader B	0.79	0.84	0.84	0.74
	Reader C	0.81	0.87	0.88	0.75
	Reader ```0 .81	0.87	0.88	0.75
	Reader ´´´0 .81 ˘0.01 0.87 ˘0.01 0.87 ˘0.01 0.75 ˘0.03
	Proposed	0.88	0.94	0.94	0.81

as well as that the disease textural heterogeneity reflects more the presence of heterogeneous lesions than pure ground glass opacities observable in mild cases. In

[START_REF] Li | Automated assessment of covid-19 pulmonary disease severity on chest radiographs using convolutional siamese neural networks[END_REF]

, the authors proposed the use of Siamese networks for the severity assessment of COVID-19

Table 7

 7

		Balanced	Weighted	Weighted	Weighted
	Classifier	Accuracy	Pecision	Sensitivity	Specificity
		Train Test Train Test Train Test Train Test
	L-SVM	0.77	0.62 0.81	0.7	0.74	0.63 0.81	0.61
	S-SVM	0.63	0.69 0.71	0.76 0.56	0.63 0.7	0.74
	AdaBoost 0.82	0.69 0.84	0.76 0.8	0.74 0.83	0.65
	Decision Tree	0.7	0.72 0.8	0.78 0.6	0.68 0.81	0.76
	Ensemble Classifier	0.82	0.69 0.84	0.76 0.8	0.74 0.83	0.65

. Performance for the Deceased (LD) and Recovered (LR) in the long-term outcome for each of the selected classifiers and their ensemble. Note: P-SVM = Support Vector Machine with a polynomial kernel; S-SVM = Support Vector Machine with a sigmoid kernel.

Acknowledgments

We thank Mihir Sahasrabudhe and Norbert Bus for their valuable comments. This work was partially supported from the ART-AI Horizon 2020.

with clinical and biological data in order to perform staging of the patients to non severe, needed intubation and deceased. Our prognosis and staging method achieved state of the art results through the deployment of a highly robust ensemble classification strategy with the use of image characteristics and patients' characteristics available within the image' metadata. In terms of future work, we are planning to investigate and generate tools for the multiclass disease segmentation and investigate in depth the characteristics of each class and their association with severity. Our findings could have a strong impact in terms of (i) patient stratification with respect to the different therapeutic strategies, (ii) accelerated drug development through rapid, reproducible and quantified assessment of treatment response through the different mid/end-points of the trial, and (iii) continuous monitoring of patient's response to treatment. ) and Deceased (SD) patients in the short-term outcome outcome for each of the top-5 selected classifiers and their ensemble presented in Section 5. Note: P-SVM = Support Vector Machine with a polynomial kernel.

Appendix A. Tables for the prognosis and staging