Éloïse Grossiord

Hugues Talbot

Nicolas Passat

Michel Meignan

Laurent Najman

Automated 3D Lymphoma Lesion Segmentation from PET/CT Characteristics

Keywords: Positron Emission Tomography, lymphoma segmentation, multimodality, machine learning, mathematical morphology, component-tree, random forest

Positron Emission Tomography (PET) using 18 F-FDG is recognized as the modality of choice for lymphoma, due to its high sensitivity and specificity. Its wider use for the detection of lesions, quantification of their metabolic activity and evaluation of response to treatment demands the development of accurate and reproducible quantitative image interpretation tools. An accurate tumour delineation remains a challenge in PET, due to the limitations the modality suffers from, despite being essential for quantifying reliable changes in tumour tissues. Due to the spatial and spectral properties of PET images, most methods rely on intensity-based strategies. Recent methods also propose to integrate anatomical priors to improve the segmentation process. However, the current routinely-used approach remains a local relative thresholding and requires important user interaction, leading to a process that is not only user-dependent but very laborious in the case of lymphomas. In this paper, we propose to rely on hierarchical image models embedding multimodality PET/CT descriptors for a fully automated PET lesion detection / segmentation, performed via a machine learning process. More precisely, we propose to perform random forest classification within the mixed spatial-spectral space of component-trees modeling PET/CT mages. This new approach, combining the strengths of machine learning and morphological hierarchy models leads to intelligent thresholding based on high-level PET/CT knowledge. We evaluate our approach on a database of multi-centric PET/CT images of patients treated for lymphoma, delineated by an expert. Our method provides good efficiency, with the detection of 92% of all lesions, and accurate segmentation results with mean sensitivity and specificity of 0.73 and 0.99 respectively, without any user interaction.

INTRODUCTION

Positron Emission Tomography (PET) using 18-fluorine fluorodeoxyglucose ( 18 F-FDG) is recognized as the modality of choice for lymphoma, due to its high sensitivity and specificity. PET imaging is now routinely used, not only to detect tumors, but also to assess their metabolic activity, allowing for diagnosis, staging and treatment response evaluation. In particular, PET provides useful information about the metabolic activity of lesions, characterized by the intensity of an injected radiotracer.

In this context, it is essential to develop quantitative image interpretation tools, that are both accurate and reproducible, especially for tumour segmentation. However this problem is not trivial and remains challenging considering the limitations the modality suffers from: limited spatial resolution compared to morphological imaging; partial volume effects and intrinsic noise that lowers the quality of reconstructed images; and physiological artifacts. Besides, the segmentation of lymphoma is a difficult task.

The most common procedure in clinical routine is a local relative thresholding of standard uptake value (SUV) which requires important user interaction. This process is not only time-consuming but also user-dependent; hence this task can be especially laborious for lymphomas where lesions are numerous, located in multiple sites of the body and with multiple intensity distributions and contrasts to surrounding tissues. Therefore, the development of efficient and easy-to-use image processing and analysis tools for clinical practice, relying on automated lesion detection and segmentation is highly relevant.

Most methods rely on intensity-based strategies. Variants of SUV thresholding (mainly local fixed, adaptive or relative) remain popular methods [START_REF] Nestle | Comparison of different methods for delineation of 18 F-FDG PET-positive tissue for target volume definition in radiotherapy of patients with non-small cell lung cancer[END_REF]. However, in order to tackle the limitations of such approaches, alternative strategies relying on more sophisticated paradigms have been proposed (see [START_REF] Shepherd | Comparative study with new accuracy metrics for target volume contouring in PET image guided radiation therapy[END_REF][START_REF] Foster | A review on segmentation of positron emission tomography images[END_REF], for recent surveys) based on region-growing [START_REF] Li | A novel PET tumor delineation method based on adaptive region-growing and dual-front active contours[END_REF], classification (FLAB [START_REF] Hatt | A fuzzy locally adaptive Bayesian segmentation approach for volume determination in PET[END_REF], FCM [START_REF] Lelandais | Segmentation of biological target volumes on multi-tracer PET images based on information fusion for achieving dose painting in radiotherapy[END_REF][START_REF] Lapuyade-Lahorgue | SPEQTACLE: An automated generalized fuzzy C-means algorithm for tumor delineation in PET[END_REF], FHMC), or level sets, active contours or random walk [START_REF] Bagci | A graph-theoretic approach for segmentation of pet images[END_REF]. Practically, such methods generally lead to interactive tools in clinical routines, where the expert user provides regions of interest (e.g., bounding-boxes) and/or seeds, and tunes threshold values in order to delineate lesions; and are limited to the PET modality.

These methods use few a priori information; as a consequence, they can lead to inaccurate segmentation results where lesions are mixed-up with hyperfixating organs. A growing number of strategies have been designed to integrate additional information in order to improve the segmentation process. We can mention here the use of shape priors [START_REF] Grossiord | Hierarchies and shape-space for PET image segmentation[END_REF], texture [START_REF] Buvat | Tumor texture analysis in PET: Where do we stand?[END_REF] or anatomical (spatial or functional) context, in particular by considering multimodal images (e.g., PET/CT or MRI/PET) to collect supplementary anatomical information, and rely on non-interactive but time-consuming strategies (e.g., multi-valued deformable models [START_REF] Naqa | Concurrent multimodality image segmentation by active contours for radiotherapy treatment planning[END_REF], optimization schemes [START_REF] Wojak | Joint variational segmentation of CT-PET data for tumoral lesions[END_REF], graph-based strategies [START_REF] Ju | Random walk and graph cut for co-segmentation of lung tumor on PET-CT images[END_REF][START_REF] Cui | Primary lung tumor segmentation from PET-CT volumes with spatial-topological constraint[END_REF][START_REF] Bagci | Joint segmentation of anatomical and functional images: Applications in quantification of lesions from PET, PET-CT, MRI-PET, and MRI-PET-CT images[END_REF][START_REF] Han | Globally optimal tumor segmentation in pet-ct images: a graph-based cosegmentation method[END_REF] or PET/CT textural characterization [START_REF] Markel | Automatic segmentation of lung carcinoma using 3D texture features in 18-FDG PET/CT[END_REF]).

In this paper, our purpose is to propose an automated segmentation of lymphoma using machine learning on PET/CT characteristics. Our approach combines an intuitive and real-time paradigm, based on intensity and thresholding, and the ability to embed additional information that improve the accuracy of segmentation, in particular by discriminating actual lesions from hyperfixating organs. To achieve that goal, we use hierarchical image models, developed in the framework of mathematical morphology, recently shown as relevant data-structures for the representation of PET images and the developement of segmentation methodologies [START_REF] Grossiord | Hierarchies and shape-space for PET image segmentation[END_REF][START_REF] Alvarez Padilla | Multicriteria 3D PET image segmentation[END_REF]. More precisely, we combine component trees [START_REF] Salembier | Anti-extensive connected operators for image and sequence processing[END_REF], modeling PET images from both spatial and spectral points of view, with a random forest strategy [START_REF] Breiman | Random forests[END_REF], in order to automatically classify the image structure using multimodal PET/CT descriptors. Such coupling allows us to take advantage of the decomposition of the PET images into relevant regions, namely the connected components of each threshold set, to compute and embed specific information related to the shape, intensity or texture of these regions, thus boosting the performances of random forest classification.

This paper is structured as follows: Sec. 2 exposes each step of the proposed methodology; Sec. 3 presents the experiment performed in this study and the obtained results; Sec. 4 presents our conclusions and perspectives.

MATERIAL AND METHODS

The proposed framework, described in Fig. 1, relies on the following steps. First, a hierarchical representation of the PET image is built, as a component-tree [START_REF] Salembier | Anti-extensive connected operators for image and sequence processing[END_REF]. This data-structure is a lossless image model, and allows for the representation of all the regions of the PET image as nodes of a tree, with specific space and intensity properties. Then, a set of PET/CT description features of each region is computed and stored as attributes of its corresponding node within the component-tree. A random forest classifier is then trained on these features to predict the class membership for each node of the component-tree. In this paper, we focus on 3 specific classes: lesions, organs and "non-relevant" structures. The adopted approach preserves the region of interest with the highest membership probability to the tumour class. The segmentation result can then be reconstructed from the selected set of nodes.

Clinical Dataset

Our database consists of a series of 43 multi-centric PET/CT exams of patients, treated for lymphoma, presenting significant FDG-PET tumor radioactivity uptake before radiotherapy. The acquisitions were carried out on several PET/CT scanners (Philips ALLEGRO, Siemens BioGraph TruePoint (Model 1093/1094), GE DISCOV-ERY ST/STE/LS) with their associated reconstruction algorithms (RAMLA, FORE/OSEM). They were acquired following standard protocol for PET cancer imaging: PET acquisition using a scan time of 3 minutes per bed position, between the base of the skull and mid-tigh, one hour after the peripheral intravenous injection of an 18 F-FDG dose of 4 to 6 MBq/kg in patients, fasted for at least 6 hours before FDG injection. PET images were obtained with CTbased attenuation correction and their spatial resolution (FWHM) varied from 2.73 × 2.74 × 3.27 mm 3 to 5.3 × 5.3 × 2 mm 3 . Each voxel from PET exam was converted into standardized uptake value (SUV) for comparisons accross patients. The SUV is a standardized decay-corrected value of 18F-FDG activity per unit volume of body weight.

Each patient exam was associated with 2 corresponding groundtruths. First, tumour segmentations, performed by an expert as a threshold at 41% of SUVmax in manually placed VOIs, were considered as Tumour Ground-Truth. Second, the same methodology was applied to create the Organ Ground-truth, constituted of 5 hyperfixating organs (brain, heart, kidneys and bladder). By contrast with usual delineation of organs on CT scans, the segmentation of organs was performed on the PET modality because our image representation is based on the PET image and it was then mandatory that we could associate the delineated organ components to actual nodes in the representation. A precise anatomical volume of organs was not necessary for the process.

Component-Trees and Feature-Based Segmentation

Due to the contraints of clinical routine, the segmentation process should be carried out in reasonable time to be integrated in such practice. Hence, the underlying processes should present efficient algorithmic complexities. To tackle this issue, we propose to consider connected operators [START_REF] Salembier | Connected operators. A review of region-based morphological image processing techniques[END_REF] from mathematical morphology [START_REF]Mathematical Morphology: From Theory to Applications[END_REF]. More precisely, we recently proposed the use of component-trees, introduced by [START_REF] Salembier | Anti-extensive connected operators for image and sequence processing[END_REF][START_REF] Jones | Connected filtering and segmentation using component trees[END_REF], as a relevant data-structure for the representation and segmentation of PET images (based on shape priors [START_REF] Grossiord | Hierarchies and shape-space for PET image segmentation[END_REF].

On the one hand, this structure decomposes the image into basic elements (i.e. the connected components of each binary level-sets of an image), that can be associated with descriptive features [START_REF] Jones | Connected filtering and segmentation using component trees[END_REF]. This decomposition is especially well-fitted for handling 3D images where the structures of interest correspond to extremal intensity values, as for angiographic [START_REF] Dufour | Filtering and segmentation of 3D angiographic data: Advances based on mathematical morphology[END_REF], or PET images [START_REF] Grossiord | Hierarchies and shape-space for PET image segmentation[END_REF]. On the other hand, this data-structure allows for the development of efficient (real-time) segmentation methods, since it can be built and handled [START_REF] Passat | Interactive segmentation based on component-trees[END_REF] in linear or quasi-linear time [START_REF] Carlinet | A comparative review of component tree computation algorithms[END_REF].

Once all the connected components of each binary (thresholded) image are extracted and organized in a tree structure [START_REF] Carlinet | A comparative review of component tree computation algorithms[END_REF] with respect to the standard inclusion relation on sets, it is possible to describe them with a set of characteristics. Then, a feature-based segmentation allows for the discarding of nodes that do not correspond to lesions, with respect to the high-level information provided in each node. (It is then important to provide good descriptive characteristics of regions in order to be able to select those corresponding to lesions.) Since the component-tree is a lossless model of the image, the segmentation result can be reconstructed straightforwardly from the remaining set of nodes.

Multimodal Region Description

The ultimate goal of this work is to be able to reproduce automatically what experts can achieve in clinical pratice. To reach that goal, a large number of features are computed for the characterization of each node. Indeed, considering that tumours have metabolic processes different from normal tissues, and that experts are able to differentiate various tissues in images based on textures, shape or locations on PET and CT images, it was important to be as exhaustive as possible towards the extracted features, and offer a full description of regions. We could expect that those differences between normal and abnormal areas would translate in those characteristics computed on PET and CT. Hence, the random forest classifier would be able to combine features and provide an accurate "description" of lesions, and efficiently discriminate abnormal from normal hyperfixations. One of the main advantages of the random forest is its ability to consider a large number of descriptors -correlated or not -and avoid over-fitting. Note, however, that we selected features that previously shown to provide good tissue discrimination in other tissue characterization studies.

The features chosen for characterization fall into four categories, computed on the PET and/or on the CT image. It was mandatory for the extracted spatial features to be comparable; thus, all patient exams were registered to correspond to a same anatomical space: all CT exams were registered to a reference CT exam, chosen randomly among all patients, and the registration matrices were propagated / applied to the corresponding PET exams and associated groundtruths. Additionnaly, the CT images were downsampled to the spatial resolution of the PET exams.

(1) Intensity and histogram-based statistics -These features rely on information derived from the distribution of intensities within each node, computed on both PET and CT modalities. The measurements are statistical characteristics, and include max, min, mean, contrast, variance, TLG, skewness or kurtosis.

(2) Shape descriptors -First, the shape of the 3D region is described by its metabolic volume (MTV). In addition, shape descriptors are derived from the eigenvalues of the covariance matrix computed on voxels coordinates within the considered region, among which elongation, flatness, eccentricity, sphericity, aspect ratio, compacity, and sparsity.

(3) Textural features -Textural features allow for the analysis of the heterogeneity of intensity distribution within the considered region. We considered 6 second-order textural features proposed by Haralick et al. [START_REF] Haralick | Textural features for image classification[END_REF], computed on both PET and CT. These features are based on the grey-level co-occurence matrix (GLCM) reflecting spatial grey-level dependencies, and more especially the joint probability ditribution of each combination of pairs of grey-level values in each direction. To compute the GLCM, intensity range within each region was quantified within 64 bins. The texture features were then computed for every direction covering 26-connected neighborhood, and averaged to obtain rotation-invariant information. Features used for this work include energy, entropy, inertia, inverse difference moment, cluster shade and cluster prominence. (4) Spatial Information -The intuition of anatomical knowledge is one of the major characteristics that enable experts to discriminate normal from abnormal hyperfixations. We considered two types of spatial information, in order to locate each considered region within the body. First, the spatial location of the considered regions is described by their barycenter coordinates. In addition, we introduced an information of distance towards specific hyperfixating organs (brain, heart, kidneys, and bladder). For that purpose, 10 patient exams were selected among our database, to be considered as "reference exams", such that those organs of interest were especially visible on PET image. Then, for each node, we computed its distance to the 5 organs of interest for the 10 reference exams (i.e., for each node, 5 × 10 distances).

Considering all these features, a total of 88 features were extracted for each node.

Region Classification by Random Forest Classifier

Ground-Truth Tree-Nodes Labeling

The learning procedure considered here is a 3-class classification, with tumour, organ and non-relevant structure classes. In order to feed the learning process, which needs to be trained on the multiple feature values of lesions, organs and non-relevant regions, described in the previous section, it is mandatory to be able to identifty tree nodes corresponding to each class from the 3D PET images.

To reach that goal, we rely on the pre-defined expert segmentations of organ and tumour regions. The difficulty lies in the fact that labeled regions may not completely fit actual connected components of the tree, as they were constructed in limited bounding boxes. As a consequence, those defined PET regions are relevantly associated to the closest set of nodes (sub-branches) of the component-tree. This process consists of maximizing the spatial inclusion of a node within a labeled ground-truth region, while minimizing its spatial overlapping. The remaining nodes, that were not classified as tumour nor organ, were tagged as "non-relevant" regions.

The outcome of this labeling procedure was the classification of the 76 214 nodes of the component-trees associated to the 43 PET images as 16 330 lesion nodes (21.4%) , 15 532 hyperfixating organ nodes (20.4%) and 44 352 non-relevant structures (58.2%).

Random Forest Training and Region Classification

The random forest (RF) method is a machine learning technique initially proposed by Breiman [START_REF] Breiman | Random forests[END_REF], which intensively uses intuitive classification and regression trees (CART) [START_REF] Breiman | Classification and Regression Trees[END_REF]. The principle of construction of a random forest can be summarized in three steps: 1) generating new "training" databases by re-sampling with replacement the original database; 2) building a classification tree of each new "training" database; 3) finally agregating the predictions given by each classification tree.

Hence, Random Forests are an ensemble classifier consisting of a collection of tree-structured classifiers, and each tree provides a unit vote for the most popular class. The different predictions coming from these independent trees are then aggregated to provide a consensus result. In this study, all tree nodes with their associated features are considered as the original database of the random forest classifier.

Random Forests presents several advantages. First, they tend not to overfit data. Second, they can deal with datasets containing more variables (i.e. descriptors) than observations (i.e. patient nodes). Finally, they can handle correlated or redundant variables.

EXPERIMENTS AND RESULTS

Clinical Experiments

Our image segmentation method was perfomed on the clinical database described in Sec. 2.1. More precisely, the experiments were made of the following steps:

(1) Component-tree construction for all the PET images.

(2) Computation of multimodal PET/CT features (described in Sec. 2.3) for every nodes of the component-trees.

(3') Node labeling procedure based on ground-truth (described in Sec. 2.4.1).

(4') Random forest construction.

(3) 3-class classification of all tree nodes with the random forest.

(4) Reconstruction of the segmented images from the tree nodes predicted as tumour regions by the random forest.

Given the small size of the considered database, our goal was to learn as much as possible from the data. Consequently, we used the same database for training (Steps (1, 2, 3', 4')) and evaluating (Steps [START_REF] Nestle | Comparison of different methods for delineation of 18 F-FDG PET-positive tissue for target volume definition in radiotherapy of patients with non-small cell lung cancer[END_REF][START_REF] Shepherd | Comparative study with new accuracy metrics for target volume contouring in PET image guided radiation therapy[END_REF][START_REF] Foster | A review on segmentation of positron emission tomography images[END_REF][START_REF] Li | A novel PET tumor delineation method based on adaptive region-growing and dual-front active contours[END_REF]) the random forest model, using a leave-onepatient-out (LOPO) cross-validation strategy to avoid biaising the results. LOPO cross-validation consists of extracting all the data related to one of the N = 43 patients considered in the dataset, and to build a random forest from the samples of the remaining N -1 patient images. This experiment was repeated for all the patient images constituting the dataset. Our calculations were performed using the scikit-learn 1 package of Python. The number of trees used to train the random forest was set to 10 3 . We quantitatively evaluated our methodology by validating first node classification results from the random forest (results of Step (3)), and then volumic results from the reconstructed images (results of Step ( 4)).

Random Forest Validation

Based on the LOPO cross-validation, we quantitatively assessed the quality of the multi-class classification step (tumour, organ and nonrelevant structure classes). The confusion matrix summarizing these results is presented in Table 1. We can observe that 61 630 nodes were correctly classified (over a total number of 76 214 nodes). Therefore, we obtain a multi-class overall accuracy of 81%. More precisely, for the classification of nodes belonging to the tumour class, we obtain an overall sensitivity and specificity of 0.65 and 0.92 respectively, with an overall accuracy of 0.86.

Tumour Segmentation Results

To evaluate our approach, we measured the differences between the ground-truth segmentations and our reconstructed tumour regions. An example of result is illustrated in Fig. 2. In terms of localization of tumours, our method automatically locates 92% of all the manually delineated tumours for all the patients. This is very satisfying and encouraging result.

In terms of volumic results, we obtained a mean sensitivity and specificity of 0.74 and 0.99 respectively, with a mean accuracy of 0.99. These results are very promising, meaning that we correctly retrieved almost 75% of all the tumoral volume automatically.

However, it should be noted that our overall detected tumoral volume overestimates the ground-truth tumoral volume by 35% in average. This overestimated volume may take its origin in two phenomena: on the one hand, parts of properly detected lesions had their volumes overestimed by the segmentation process, and on the other hand, false positive structures were detected as tumours compared to the ground-truth. Some of these structures are remaining hyperfixating organs, such as kidneys or part of the heart. Besides, it should be noted that, sometimes, some of these structures also correspond to real lesions that had not been correctly labeled by the expert when designing the ground-truth.

CONCLUSION AND PERSPECTIVES

We have shown how hierarchical approaches, combined with machine-learning and multimodality descriptors, can automatically and efficiently detect lymphoma lesions in 3D. Our results suggest that we detect 92% of all lesions, but with a volume overestimate by 35%. This is a very encouraging result; indeed the ability to detect automatically more than 90% of all lesions is a precious help in clinical routine.

This volume overestimation may be problematic for an actual use in practice. However, the volumic overestimation deriving from the overestimation of correctly classified tumour regions may be solved by considering a learning procedure applied on a larger database, or with the selection of the most important feature descriptors, leading to more accurate classification results. Second, the hyperfixating organs classified as tumour areas (e.g. heart and kidneys) may be eliminated beforehand by using an anatomical atlas. Besides, some of the remaining false positives were actual lesions that had not been labeled by the clinical expert. In the context of an automatic pre-segmentation, this detection could offer the possibility for the clinical expert to consider such lesions or not.

Additionally, we could consider taking advantages of the hierarchical structure of the component-tree, in order to improve the segmentation process, for instance by injecting tree-related structural information (number and length of sub-branches, . . . ) or considering whole labeled sub-branches as unitary structures of the segmentation process, instead of nodes. This will be the purpose of future work. We are currently integrating our framework within an ImageJ plugin dedicated to PET/CT viewing [START_REF] Kanoun | Beth Israel Plugin for FIJI: logiciel gratuit et open source pour la recherche scientifique[END_REF].

Fig. 1 .

 1 Fig. 1. Segmentation global framework. Boxes (1), (2) and (3) refer to Secs. 2.2, 2.3 and 2.4, respectively.

1

  http://scikit-learn.org

Fig. 2 .

 2 Fig. 2. PET image (maximum intensity projection) surimposed with ground-truth (in green, on the left) and segmentation result (in blue, on the right).

Table 1 .

 1 Confusion matrix of nodes after random forest classification (T, O and NR correspond to tumor, organ and non-relevant structure classes, respectively).

			Predicted Class
		Class	T	O	NR
	Actual Class	T O NR	10 568 875 12 222 1 008 3 656 1 856 38 840 4 754 2 435