Berihu Kibrom

Louis Girum

Anne-Ségolène Rebaud

Michel Cottereau

Jérôme Meignan

Laetitia Clerc

Olivier Vercellino

Franck Casasnovas

Catherine Morschhauser

Irène Thieblemont

Buvat

Ph.D Kibrom B Girum
email: kibrom.girum@curie.fr

Louis Rebaud

Anne-Ségolène Cottereau

Michel Meignan

Jérôme Clerc

Laetitia Vercellino

Olivier Casasnovas

Franck Morschhauser

Catherine Thieblemont

Dr Irène Buvat
email: irene.buvat@u-psud.fr

8F-FDG PET maximum intensity projections and artificial intelligence: a win-win combination to easily measure prognostic biomarkers in DLBCL patients

Keywords: Artificial intelligence, DLBCL, 18 F FDG PET/CT, dissemination, metabolic tumor volume

teaching and research institutions in France or abroad, or from public or private research centers.

INTRODUCTION

Diffuse large B-cell lymphoma (DLBCL) is the most common type of non-Hodgkin lymphoma. In clinical practice, 18 F-FDG PET/CT is a standard-of-care for staging and assessing response in DLBCL patients (1).

The prognostic value of the total metabolically active tumor volume (TMTV) measured from the whole-body 18 F-FDG PET/CT images performed before treatment has been widely demonstrated in lymphoma, especially in DLBCL (2)(3)(4)(5)(6). The disease dissemination reflected by the largest distance between two lesions in the baseline whole-body 18 F-FDG PET/CT image (Dmax) has been recently shown to be a complementary early prognostic factor compared to TMTV (7,8). TMTV and Dmax calculations require tumor volume delineation over the whole-body three-dimensional (3D) 18 F-FDG PET/CT images, which is prone to observer-variability and complicates the use of these quantitative features in clinical routine.

To address this problem, automated lesion segmentation approaches using convolutional neural networks (CNN) have been proposed (9,[START_REF] Blanc-Durand | Fully automatic segmentation of diffuse large B cell lymphoma lesions on 3D FDG-PET/CT for total metabolic tumour volume prediction using a convolutional neural network[END_REF]. These methods require high computational resources to be developed but have shown promising results, despite missing small lesions (7). Yet, results from CNN still need to be validated and adjusted by an expert before using them for further analysis (7,8). This implies a thorough visual analysis of all 3D 18 F-FDG PET/CT images and delineation of the lesions missed by the algorithm. Consequently, developing a pipeline that would speed-up this checking/adjustment process is highly desirable in clinical practice.

Nuclear medicine physicians commonly use two-dimensional (2D) PET Maximum Intensity Projection (MIP) views for visual interpretation as a synthetic representation of the 3D distribution of the tracer over the whole-body. However, to the best of our knowledge, the prognostic value of PET parameters extracted from 2D MIP has never been explored. We hypothesized that tumor burden and spread could be automatically evaluated from only two PET MIP images corresponding to coronal and sagittal views. This would have two advantages: first, result checking and adjustment would be faster from two MIP views compared to the whole-body 3D 18 F-FDG PET/CT images, typically including more than 200 transaxial slices. Second, a deep learning model for segmenting MIP images would involve much fewer parameters than when segmenting the whole-body 3D 18 F-FDG PET images. It is less computationally expensive and might require less data for training.

The purpose of this study was to investigate whether TMTV and Dmax biomarkers could be replaced by surrogate biomarkers automatically calculated using an artificial intelligence (AI) algorithm from only two MIP of the whole-body 18 F-FDG PET images. We then determined the prognostic values of the surrogate biomarkers in terms of progression-free survival (PFS) and overall survival (OS).

MATERIALS AND METHODS

Patient Cohorts

The study population included DLBCL patients who had a baseline (before treatment initiation) PET/CT scan from two independent trials: REMARC (NCT01122472) and LNH073B (NCT00498043). The characteristics of these cohorts have been published elsewhere ((6) for REMARC, [START_REF] Casasnovas | FDG-PET-driven consolidation strategy in diffuse large B-cell lymphoma: final results of a randomized phase 2 study[END_REF] for LNH073B) [START_REF] Nioche | Lifex: A freeware for radiomic feature calculation in multimodality imaging to accelerate advances in the characterization of tumor heterogeneity[END_REF].

PFS and OS as defined following the revised National Cancer Institute criteria [START_REF] Cheson | Revised response criteria for malignant lymphoma[END_REF] were recorded. Flow diagrams for the datasets and the study design are summarized in Figure 1. All data were anonymized before analysis. The institutional review board approval, including ancillary studies, was obtained for the two trials, and all patients provided written informed consent. The demographics and staging of the patients used for the survival analysis are summarized in Table 1.

Measurements of Reference TMTV and Dmax

For the REMARC cohort, the lymphoma regions were identified in the 3D PET images as described in (6,[START_REF] Capobianco | Deep-learning 18 F-FDG uptake classification enables total metabolic tumor volume estimation in diffuse large B-Cell lymphoma[END_REF], while the LNH073B lesions were segmented as explained in (7). In all cohorts, physicians removed the regions corresponding to physiological uptakes and added pathological regions missed by the algorithm. The supplementary material (section A) provides the details. Expert-validated 3D lymphoma regions were used to compute the reference TMTV and Dmax (based on the centroid of the lymphoma regions), as shown in Figure 1(B) (8).

Calculation of the PET MIP Images and 2D Reference Lymphoma Regions

For each patient whole-body 3D 18 F-FDG PET images and associated 3D lymphoma regions, two 2D MIP views and associated 2D lymphoma regions were calculated (Figure 2). The 3D PET image was projected in the coronal and sagittal directions, 90° apart (Figure 2), setting each pixel value of the projection to the maximum intensity observed along the ray normal to the plane of projection. Similarly, MIP of the expert-validated 3D lymphoma regions were calculated, resulting in binary images of 2D lymphoma regions (Figure 2), hereafter called MIP_masks. As described in the following section, these MIP_masks were then used as a reference output to train a CNN-based fully automatic lymphoma segmentation method.

Fully Automatic Lymphoma Segmentation on PET MIP Images

Deep Learning Model Inputs and Architectures. To automatically segment the lymphoma lesions from the sagittal and coronal PET MIP images, we adapted a previously published supervised 2D deep learning model [START_REF] Girum | Learning with context feedback loop for robust medical image segmentation[END_REF]. The sagittal and coronal PET MIPs were independent input images during training.

The corresponding MIP_mask was the output image. The deep learning model was trained to transform a given sagittal or coronal PET MIP image to the corresponding MIP_mask with pixels of lymphoma regions set to one and pixels of the non-lymphoma regions set to zero.

Training, Validation, and Testing Configurations. First, using the REMARC cohort (298 patients), a five-fold cross-validation technique was used to train and evaluate the model. Patients were randomly split into five groups, and then five models were trained on 80% of the population and the remaining 20% was used for validation. The detailed network architecture [START_REF] Girum | Learning with context feedback loop for robust medical image segmentation[END_REF][START_REF] He | Deep residual learning for image recognition[END_REF] and the training procedures are fully described in the supplementary material (section B, Supplemental Figure 1) [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], following the CLAIM guidelines [START_REF] Mongan | Checklist for artificial intelligence in medical imaging (CLAIM): A guide for authors and reviewers[END_REF] and SNMMI AI Task force recommendations [START_REF] Bradshaw | Nuclear medicine and artificial intelligence: Best practices for algorithm development[END_REF]. The deep learning model will be publicly available upon publication [GitHub, Anonymous].

Secondly, we tested the model trained from the REMARC cohort (298 patients) on the independent LNH073B cohort (174 patients) to characterize its generalizability and robustness. The REMARC and LNH073B cohorts were acquired from two different trials. The REMARC study data was a double-blind, international, multicenter, randomized phase III study, which started inclusion in 2010. In contrast, the LNH073B data was a prospective multicenter, randomized phase II study that started including patients in 2007.

Calculation of Surrogate TMTV and Surrogate Dmax

The surrogate TMTV (sTMTV) and Dmax (sDmax) were defined and computed from the MIP_masks automatically segmented from the coronal and sagittal PET MIP images using the deep learning method.

Tumor Burden Analysis. To characterize tumor burden, we defined a surrogate tumor volume sTMTV from the MIP_mask as the number of pixels belonging to the tumor regions in MIP_mask multiplied by the pixel area. For a given patient, sTMTV was calculated from the coronal and the sagittal MIP_masks as sTMTV = sTMTVcoronal + sTMTVsagittal.

Tumor Dissemination Analysis. The spread of the disease was analyzed by estimating the largest distance between the tumor pixels belonging to the MIP_mask, using a new robust largest distance estimation approach. First, we separately calculated the sum of pixels along the columns and the rows of MIP_mask, yielding x and y profiles (Supplemental Figure 2). Second, in each of these two profiles, the distances between the 2% percentile and the 98% percentiles (x2% and x98% in the x profiles, y2% and y98% in the y profiles) were calculated, yielding (𝑥 98% -𝑥 2% ) and (𝑦 98% -𝑦 2% ), respectively. These percentiles were chosen to improve the robustness of the calculation to outliers. The largest distance was defined as 𝑠𝐷𝑚𝑎𝑥 𝑠𝑎𝑔𝑖𝑡𝑡𝑎𝑙/𝑐𝑜𝑟𝑜𝑛𝑎𝑙 = (𝑥 98% -𝑥 2% ) + (𝑦 98% -𝑦 2% ). For a given patient, the surrogate tumor dissemination sDmax was the sum of the coronal and sagittal disseminations using 𝑠𝐷𝑚𝑎𝑥 = 𝑠𝐷𝑚𝑎𝑥 𝑠𝑎𝑔𝑖𝑡𝑡𝑎𝑙 + 𝑠𝐷𝑚𝑎𝑥 𝑐𝑜𝑟𝑜𝑛𝑎𝑙 .

Statistical Analysis

Using the MIP_masks obtained from the expert-delineated 3D lymphoma regions (Figure 2) as a reference, CNN's segmentation performance was evaluated using the Dice score, sensitivity, and specificity. The difference between the CNN-based segmentation results and the expert-delineated 3D lymphoma regions were quantified using Wilcoxon statistical tests. Univariate and multivariate survival analyses were performed. For all biomarkers, we calculated a time-dependent area under the receiver operating characteristics curve (AUC) [START_REF] Heagerty | Time-Dependent ROC curves for censored survival data and a diagnostic marker[END_REF]. Bootstrap resampling analysis was performed to associate confidence intervals to the Cox model hazard ratio and the time-dependent AUC. See the supplementary material (section C) for details. Test results were considered statistically significant if the two-sided P-value was <0.05.

RESULTS

A total of 475 patients from two different cohorts were included in this study, of which 93 patients were excluded from the biomarker and survival analysis because the provided baseline 18 F-FDG PET/CT images were not suitable to analyze all biomarkers (no PET segmentation by an expert or less than 2 lesions). Summary statistics of patients are presented in Table 1.

Lymphoma Segmentation

The performance of the proposed segmentation method was evaluated patient-wise. The CNN segmentation method achieved a 0.80 median Dice score (interquartile range [IQR]: 0.63-0.89), 80.7% (IQR: 64.5%-91.3%) sensitivity, and 99.7% (IQR: 99.4%-0.99.9%) specificity on the REMARC cohort. On the testing 174 LNH073B patients, the CNN yielded a 0.86 median Dice score (IQR: 0.77-0.92), 87.9% (IQR: 74.9.0%-94.4%) sensitivity, and 99.7% (IQR: 99.4%-99.8%) specificity. In the LNH073B data, the CNN yielded a mean Dice score of 0.80 ± 0.17 (mean ± SD) on the coronal view and 0.79 ± 0.17 on the sagittal view. Figure 3 shows segmentation result examples from experts (MIP_masks) and CNN. See Supplemental Figure 3 for more segmentation results. The Dice score was not significantly different (P >0.05) between the coronal and sagittal views, both for the REMARC and LNH073B cohorts (p>0.05).

In both cohorts, there was a significant correlation between ranked TMTV and Dmax values and the associated surrogate values obtained using CNN. For REMARC, TMTV was correlated with sTMTV (Spearman r = 0.878, p<0.001), and Dmax was correlated with sDmax (r = 0.709, p<0.001). Out of 144 patients who had TMTV greater than the median TMTV (242 cm 3 ), 121 (84.02%) patients had also sTMTV greater than the median sTMTV (174 .24 cm 2 ). 144 patients had Dmax greater than the median Dmax (44.8 cm), and 113 (78.5%) of these patients also had sDmax greater than the median sDmax (98.0 cm).

For LNH073B, TMTV was correlated with sTMTV (r =0.752, p<0.001), and Dmax was correlated with sDmax (r = 0.714, p<0.001). Out of 48 patients who had TMTV greater than the median TMTV (375 cm 3 ), 42 (87.5%) patients had also sTMTV greater than the median sTMTV (307.2 cm 2 ). 48 patients had Dmax greater than the median Dmax (44.1 cm), and 39 (81.3%) of these patients also had sDmax greater than the median sDmax (116.4 cm). Table 2 shows the descriptive statistics for the surrogate PET features.

Survival Analysis

The time-dependent AUC and hazard ratios (HR) with 95% confidence interval of the metabolic tumor volume and tumor spread are shown in Table 3 for the REMARC and LNH073B data. All PET features extracted from the baseline 3D 18 F-FDG PET/CT images and using AI (sTMTV and sDmax) were significant prognosticators of the PFS and OS.

Combining TMTV and Dmax (or their surrogates), three risk categories could be differentiated in the REMARC data (Figure 4): using the 3D features, category 1 corresponded to low TMTV (≤ 222 cm 3 ) and low Dmax (≤ 59 cm) (low risk, n=108); category 2 corresponded to either high Dmax or high TMTV (intermediate risk, n=112); category 3 corresponded to both high Dmax and high TMTV (high risk, n=67). This stratification was similar when using the MIP-features-based categories using AI (Figure 4). The accuracy of the CNN-based classification into three categories with respect to the 3D-biomarkers-based classification was 71.4%.

In the LNH073B cohort, combining TMTV and Dmax (or their surrogates), three risk categories could be differentiated (Figure 5). Using the 3D features, category 1 was defined as low TMTV (≤ 468 cm 3 ) and low Dmax (≤ 60 cm) (n=45); category 2 corresponded to either high Dmax or high TMTV (n=37); category 3 corresponded to both high Dmax and high TMTV (n=13). Out of the 13 patients classified as high risk, 9 (69.2%) patients had less than 4-years of OS, and 10 (76.9%) patients had less than 4-years of PFS. This stratification was similar when using the CNN-based results. The sTMTV cut-off value was 376 cm 2 , the sDmax cut-off value was 122 cm. There were 38 patients in category 1, 35 in category 2, and 22 in category 3. Out of the 22 patients classified as a high risk, 19 (77.3%) patients had less than 4-years of OS, and 19 (86.4%) patients had less than 4-years of PFS. The accuracy of the AI-based classification into three categories with respect to the 3D-biomarkers-based classification was 64.2%. All patients classified as high risk using the 3D biomarkers were also classified as high risk using the CNN, except one patient who had an OS of 36.6 months. Out of the nine patients classified as high risk when using the CNN but not when using the 3D biomarkers, 8 (88.9%) patients had less than 4-years of OS, and the remaining one (11.1%) patient had 21.95 and 57.99 months of PFS and OS respectively.

In Supplemental Figure 4, the confusion matrices show the agreement between the 3D-based biomarkers and the surrogate MIP biomarkers in the LNH073B data. The percentage of the data classified into high, low, and intermediate risk is also shown. Using a classification in two groups based on one biomarker only (either tumor burden or dissemination biomarkers), the AI-based classification had a 79% accuracy compared to the 3D-based classification.

DISCUSSION

We developed and evaluated a new framework to calculate surrogate metabolic tumor volume (sTMTV) and surrogate tumor dissemination (the largest distance between lymphoma sites) (sDmax) features from 2D PET MIP images. The motivation for considering tumor delineation on 2D MIP views instead of the 3D volume was twofold: first, checking lymphoma regions on 2D PET MIP images is much faster than on the 3D 18 F-FDG PET/CT volumes. Second, training an automated AI tumor segmentation algorithm is easier in 2D than in 3D from a practical point of view (fewer parameters to be tuned, less data to be used for training, and less computational cost). We thus investigated the prognostic values of these surrogate biomarkers using two independent retrospective cohorts of DLBCL patients with baseline 18 F-FDG PET/CT. Characterizing tumor burden and its dissemination was feasible using features extracted from the 2D PET MIP images. TMTV and Dmax were highly correlated with sTMTV and sDmax, respectively.

Developing automatic and robust lymphoma segmentation methods on PET MIP images could cost less data and less computational resources than when using the whole-body 18 F-FDG PET images. It could allow AI experts to quickly investigate appropriate segmentation approaches to tackle the challenging lymphoma segmentation task and reduce inter-center and inter-expert variations in lymphoma delineation.

Experts can validate and correct, if necessary, AI results on 2D MIP images easier and faster than on their 3D volume counterparts. We also showed that a convolutional neural network (CNN) could segment lymphoma lesions fully automatically from the given 2D PET MIP image with high accuracy compared with expert readers. This result was confirmed on the independent LNH073B cohort. The proposed CNN-based segmented regions enabled features extraction with predictive values comparable to when these features are calculated from the areas delineated by experts in the 3D image. The main strength of this work was that we validated our findings using an external cohort from a different retrospective trial. However, training the proposed deep learning model from an increased training sample size, preferably from different centers and acquisition parameters, might further improve its performance. No correlations were observed between the segmentation errors made by the model and lesion size. Previous lymphoma segmentation methods used the whole-body 18 F-FDG PET/CT images (9,[START_REF] Blanc-Durand | Fully automatic segmentation of diffuse large B cell lymphoma lesions on 3D FDG-PET/CT for total metabolic tumour volume prediction using a convolutional neural network[END_REF]. Most of these methods involved complex preprocessing, CT and PET image alignment, and did not investigate whether both TMTV and Dmax remained good prognosticators when calculated from the automated segmentation. Recent studies have also demonstrated that CNN-based results need corrections by experts (7,8). Correction of results on 3D volume could be time-consuming, observer-dependent, and difficult. In contrast, corrections, and validations (if necessary) could be easier and faster on 2D PET MIP images, allowing easy use of these features in clinical routine.

Interestingly, the surrogate biomarkers automatically calculated using AI (sTMTV and sDmax) had strong prognostic values regarding PFS and OS, comparable to the prognostic importance of TMTV and Dmax obtained from the 3D volumes. The classification of patients into the three risk groups using the 3D TMTV and Dmax agreed with the patient's classification based on the 2D sTMTV and sDmax (71.4% and 64.2% respectively in REMARC and LNH037 cohorts). Patients classified as high-risk using 3D-based biomarkers and low-risk (or vice versa) using 2D-based biomarkers had values close to the cut-off values.

Visual assessment of the segmentation results suggested that the 2D-based biomarkers tend to perform well compared to the 3D-based biomarkers when the patient had lesions spread over the body and performed less well when the patient had a large bulky lesion.

In this work, we defined and calculated the surrogate biomarkers from both the coronal and sagittal PET MIPs. However, experiments showed that characterizing the lymphoma disease using sTMTV and sDmax calculated from either coronal or sagittal also had good predictive values, comparable to these features obtained from 3D volumes. The same cut-off values were used to analyze the PFS and OS. The cohorts were from two independent studies with varying stages of cancer (Table 1). Thus the (s)TMTV cutoff values were different between the two cohorts. Interestingly, the cut-off values to characterize the lesion dissemination (Dmax and sDmax) in DLBCL patients on baseline PET images were almost identical on the independent cohorts. Dmax and sDmax were defined empirically, yet a recent study has shown that the distance between lesions calculated using different distance measurement methods (namely Euclidean, Manhattan, and Chebyshev) in 3D yielded similar results in predictions of the outcome [START_REF] Cottereau | New approaches in characterization of lesions dissemination in DLBCL patients on baseline PET/CT[END_REF].

Our study has limitations. Although we validated the CNN on two independent retrospective studies, validating the proposed CNN in larger multicenter cohorts will be required to develop it into a clinical tool. In addition, although the CNN results can be easily visually checked, they should ideally be provided with a confidence level, that could be turned into a confidence associated with the risk classification.

CONCLUSION

In this study, we introduced biomarkers extracted from PET MIP as surrogates of the total metabolic tumor burden and tumor dissemination. To our knowledge, this is the first study showing that PET parameters extracted from 2D MIP images are predictive of outcome in a large series of patients with DLBCL, with results comparable to these features calculated from the 3D 18 F-FDG PET/CT images. We demonstrated that surrogate TMTV and Dmax calculated from lymphoma regions automatically delineated on PET MIP images using artificial intelligence have strong prognostic values in stratifying patients with DLBCL. This result might considerably facilitate the calculation and usage of these features in clinical practices.

DISCLOSURE

Tables

  Kibrom B. Girum and Irène Buvat disclosed a research grant given to the Institut Curie by ANR (ANR-19-SYME-0005-03). Louis Rebaud disclosed employment by Siemens Medical Solutions. No other potential conflicts of interest relevant to this article exist. KEY POINTS QUESTION: Are surrogate tumor burden and dissemination features calculated from PET maximum intensity projection (MIP) images prognostic biomarkers in diffuse large B-cell lymphoma (DLBCL) patients and can they be automatically measured using an AI? PERTINENT FINDINGS: Surrogate total metabolically active tumor volume (sTMTV) and dissemination feature (sDmax) calculated from MIP of whole-body 18 F-FDG PET images are predictive of progressionfree survival (PFS) and overall survival (OS) in DLBCL patients from two independent cohorts. A convolutional neural network (CNN) could segment lymphoma lesions from 2D PET MIP images automatically and the resulting CNN-based sTMTV and sDmax estimates were predictive of PFS and OS in two independent cohorts. IMPLICATIONS FOR PATIENT CARE: Surrogate tumor burden and dissemination features automatically calculated using AI from only two PET MIP images are prognostic biomarkers in DLBCL patients.

Figures and Tables FIGURE 1 .

 1 Figures and Tables

FIGURE 2 .

 2 FIGURE 2. An example of 18 F-FDG PET MIP images (left) and associated lymphoma regions (right) based on the expert delineation of the 3D 18 F-FDG PET images.

FIGURE 3 .

 3 FIGURE 3. 18 F-FDG PET MIP images and segmentation results (blue color overlapped over the PET MIP images) by experts (MIP_masks) and by the CNN for four patients: (A) from the REMARC cohort, and (B) from the LNH073B cohort.

FIGURE 4 .

 4 FIGURE 4. Kaplan-Meier estimates of overall survival (OS) and progression-free survival (FPS) on the REMARC cohort according to 3D 18 F-FDG PET/CT image-based features TMTV (cm 3 ) and Dmax (cm) (A, C), and according to PET MIP image-based features (sTMTV (cm 2 ) and sDmax (cm)) estimated from AI (B, D).

FIGURE 5 .

 5 FIGURE 5. Kaplan-Meier estimates of overall survival (OS) and progression-free survival (FPS) on the LNH073B cohort according to 3D 18 F-FDG PET/CT image-based features TMTV (cm 3 ) and Dmax (cm) (A, C), and according to PET MIP image-based features (sTMTV (cm 2 ) and sDmax (cm)) estimated from AI (B, D).

TABLE 1 .

 1 Population characteristics.

	Characteristic	REMARC	LNH073B
	No. of patients	287	95
	Sex
	No. of men	165 (57.5%)	42 (44%)
	No. of women	122 (42.5%)	53 (56%)
	Median age (y)	68 [64.0-73.0]	46 [33.25-55.0]
	Median weight (kg)	72 [63.0-84.2]	68 [58.0-80.0]
	Median height (cm)	167.5 [160.0-175.0]	173 [140.0-193.0]
		(1 case missed)
	Ann Arbor stage
	<I	1 (0.4%)	0 (0%)
	>=II	286 (99.6%)	95 (100%)
	Performance status
	0	115 (40%)	0 (0%)
	1	121 (42%)	27 (28.4%)
	2	42 (14.6%)	43 (45.3%)
	3	2 (0.7%)	20 (21.1%)
	4	2 (0.7%)	5 (5.3%)
	Missing	5 (1.7%)	NA

Note: data in brackets are interquartile ranges (quartile one to quartile three).

TABLE 2 .

 2 Statistics for Surrogate TMTV and Surrogate Dmax.

	Cohort	sTMTV/sDmax	Mean	SD	Minimum	Q1 (25%)	Median	Q3 (75%)	Maximum
	REMARC	sTMTV (cm 2 )	252.27	245.75	0.48	77.04	174.24	350.56	1339.36
		sDmax (cm)	100.16	49.89	0.40	66.20	98.0	135.0	225.20
	LNH073B	sTMTV (cm 2 )	388.12 249.91 63.68	224.48	307.2	450.08	1186.24
		sDmax (cm)	121.82 41.10	43.20	92.00	116.40	145.60	222.40

TABLE 3 .

 3 Results of the Univariate Analyses for Progression-free Survival (PFS) and Overall Survival (OS) using Timedependent Area Under the Receiver Operating Characteristics Curve (AUC) analysis and Cox Models (Hazard Ratios (HR)) on the REMARC Data.

				3D 18 F-FDG PET/CT estimates	2D PET MIP estimates
	Data	PFS/OS Metrics	TMTV	Dmax	sTMTV	sDmax
	REMARC PFS	AUC	0.67	0.65	0.65	0.68
				(0.60-0.73)	(0.58-0.72)	(0.58-0.72)	(0.62-0.75)
			HR	11.24	9.0	11.81	12.49
				(2.10-46.20)	(2.53-23.63)	(3.29-31.77)	(3.42-34.50)
		OS	AUC	0.67	0.62	0.67	0.68
				(0.58-0.76)	(0.53-0.71)	(0.58-0.76)	(0.59-0.76)
			HR	16.43	8.60	22.14	22.79
				(2.42-77.29)	(1.47-28.33)	(4.73-69.06)	(3.80-79.21)
	LNH073B PFS	AUC	0.62	0.56	0.66	0.58
				(0.49-0.75)	(0.39-0.72)	(0.53-0.80)	(0.41-0.74)
			HR	13.79	32.83	9.24	16.79
				(0.45-86.80)	(0.4-220.8)	(0.95-37.94)	(0.69-86.41)
		OS	AUC	0.65	0.51	0.64	0.50
				(0.46-0.82)	(0.31-0.72)	(0.45-0.82)	(0.29-0.72)
			HR	64.30	49.21	14.17	20.39
				(0.74-384.80)	(0.07-258.3)	(0.59-67.02)	(0.08-93.66)

The REMARC and LNH073B clinical studies and analyses were sponsored by the Lymphoma Academic Research Organization (LYSARC) of France. This study has received funding from ANR (ANR-19-SYME-0005-03).

Supplementary material

A. Measurements of Reference TMTV and Dmax

For the REMARC cohort, the lymphoma regions were automatically identified in the 3D PET images as described in (1,2). A SUVmax 41% threshold segmentation was then applied on these regions, and the results were visually checked by an expert nuclear medicine physician to exclude physiological lesions and to manually add missed lesions whenever needed as described in (3).

The LNH073B data were processed by a nuclear medicine physician using the LIFEx software (4): hypermetabolic regions were first automatically detected by selecting all voxels with an SUV greater than 2 included in a region greater than 2 mL, and a 41% SUVmax thresholding of the resulting regions was used. Like in the REMARC cohort, the expert removed the regions corresponding to physiological uptakes and added pathological regions missed by the algorithm.

For both cohorts, the physicians were blinded to the patient outcomes. The 3D lymphoma regions validated by experts were used to compute the baseline TMTV and Dmax (based on the centroid of the lymphoma regions) (5).

B. Final Network Architecture and Training

The deep learning model was trained from the REMARC data using a five-fold cross-validation technique. It was then tested on another independent cohort, LNH073B. The architecture of the network was inspired by (6). The model consists of an encoder and a decoder network with a skipped connection between the two paths and external fully connected network-based feedback. Lymphoma regions are often scattered over the whole body, and information could easily be lost in the successive convolution and pooling operations. To alleviate this scenario, we have used residual CNN as a building block (7) in all encoder and decoder components of the deep learning model (Figure 1). It can ease training and facilitate information propagation from input to the output of the network architecture. The input and output dimensions of the network were 128×256×1. was followed by batch normalization and activation function. We have used the exponential linear unit (ELU) activation function, except it was a sigmoid activation function at the output layers. After the convolutional building block in the encoder, we applied a 2x2 max pooling operation with stride 2 for downsampling.

Supplemental

Before the convolutional building block, we used a 2x2 up-convolutional layer in the decoder. The deep learning model will be publicly available upon publication [GitHub].

Pre-processing. All available 3D PET images and the corresponding expert-validated 3D lymphoma segmented regions were resized in to 4 x 4 x 4 mm 3 voxel size. The resized 3D images were then padded or cropped to fit into a 128x128x256. The resized and cropped image were projected into sagittal and coronal views. The input and output image dimensions to the network were 128x256x1.

Training. The model was trained with a batch size of 32 for 1000 epochs and 300 early stop criteria.

Different augmentation techniques, including flipping and rotation, were considered and tested but did not improve the results, so we did not use any data augmentation to produce the final model. The deep learning model neural network weights were updated using a stochastic gradient descent algorithm, ADAM optimizer (8), with a learning rate of 1e-4. All other parameters were Keras default values. A sigmoid output activation function was used to binarize the image into the lymphoma region and non-lymphoma region. We used the average of the Dice similarity coefficient (LossDice) and binary cross-entropy (Lossbinary cross-entropy) as a loss function defined by:

The model was implemented with Python, Keras API, and Tensorflow backend. The data was processed using the Python 3.8.5 package, including Numpy, Scipy, Pandas, and Matplotlib. We did not apply any post-processing method for the segmentation metrics. To compute the surrogate biomarkers from the AIbased segmented images, regions with less than 4.8 cm 2 were removed. The deep learning model will be publicly available upon publication at [GitHub]. Supplemental Figure 2. Illustration of the calculation of the tumor dissemination feature. For the given PET MIP image, we created two profiles corresponding to the sum of the signal in the x and y directions, respectively. The horizontal line shows the distances between the 2% percentiles and the 98% percentiles.

was the same for the sagittal PET MIP image. Pixel positions with zero total number of tumor pixels (often at the beginning and end of the pixel positions) are not considered for the percentile calculation.

C. Statistical Analysis Details

Univariate Analysis. For all biomarkers, we calculated a time-dependent area under the receiver operating characteristics curve (AUC) (9). Bootstrap resampling analysis was performed to associate confidence intervals to the Cox model hazard ratio and the time-dependent AUC. The bootstrapping involved 10,000 random samplings of the data with replacement. All statistical comparisons, except the Kaplan-Meier analysis, were made without discretizing the continuous values.

Multivariate Analysis. We estimated the survival functions using Kaplar-Meier estimates. For each PET-derived feature, we selected the optimal cut-off values for PFS and OS at the values that yielded the smallest P-value in the log-rank test between categories of a given study population. The cut-off values were constrained to be between the interquartile ranges of the TMTV or Dmax values. This procedure was the same for all measurements, namely for the 3D 18 F-FDG PET-based biomarkers (TMTV and Dmax) and PET MIP-based biomarkers from the deep learning method (sTMTV and sDmax). A receiver-operatingcharacteristics (ROC) analysis was also used to define the optimal cut-off values that predict the occurrence of an event (progression-free survival or overall survival) by maximizing the sensitivity plus specificity minus one (i.e., sensitivity + specificity -1). It yielded nearly the same results as calculating the cut-off values using the log-rank test approach. For the TMTV, we obtained a cut-off value of 222 cm 3 , which is close to the published values of 220 cm 3 (1). For uniformity of the comparison of the 2D and 3D PET features, we followed the same procedures for all features to compute the cut-off values. Supplemental Figure 3. 18


Aliza Abraham
email: aliza.abraham@univ-amu.fr

Thomas Leweke

Néstor Ramos-García

Jens Nørkaer Sørensen

Tip vortex instability triggered by rotor asymmetry for accelerated wake recovery

Keywords: Tip vortex instability, wake recovery, passive flow control

The breakdown of tip vortices in the wake of a wind turbine is the crucial first step of wake mixing and recovery. Capitalizing on inherent instabilities in the vortex system to promote interaction between adjacent vortices can accelerate this breakdown process. One way to trigger vortex instability is to introduce an asymmetry to the rotor by modifying one or two of the blades. The current study uses water channel experiments and simulations to investigate the effects of different types of rotor asymmetry on the wake. Results show that rotor asymmetry successfully triggers vortex instabilities, though some types of asymmetry are more effective than others. These findings suggest rotor asymmetry could be used as a type of passive flow control to mitigate the detrimental effects of wakes on downstream turbines in a wind farm.

Introduction

Previous studies have shown that wind turbine wake effects are responsible for some of the largest losses in wind plant energy production [START_REF] Lee | An overview of wind-energy-production prediction bias, losses, and uncertainties[END_REF] and significant increases in fatigue loading on downstream turbines [START_REF] Lee | A numerical study of atmospheric and wake turbulence impacts on wind turbine fatigue loadings[END_REF]. The near wake, within 1-4 rotor diameters of the turbine depending on the inflow turbulence level, is characterized by coherent vortical structures shed from the turbine tower, nacelle, blade roots, and blade tips. Mixing between the wake and the surrounding flow increases substantially once these structures start to break down, particularly those shed from the blade tips [START_REF] Lignarolo | Tip-vortex instability and turbulent mixing in wind-turbine wakes[END_REF]. This mixing re-energizes the flow in the wake, providing more available power to downstream turbines. In addition to their role in delaying wake recovery, coherent structures can be responsible for strong structural loading events on downstream turbines [START_REF] Lee | A numerical study of atmospheric and wake turbulence impacts on wind turbine fatigue loadings[END_REF]. Therefore, accelerating the breakdown of such structures can help minimize the detrimental impacts of wind turbine wakes on downstream machines.

Lignarolo et al. [START_REF] Lignarolo | Tip-vortex instability and turbulent mixing in wind-turbine wakes[END_REF] show that tip vortex breakdown occurs after pairing between adjacent vortices. This pairing is caused by the mutual inductance instability in the helical vortex system, which occurs when filaments are close enough to experience the induced velocity field of their nearest neighbors. This instability can be triggered by introducing a slight asymmetry to the rotor as a form of passive flow control to offset the tip vortices from their baseline position, demonstrated for a two-bladed rotor by Quaranta et al. [START_REF] Quaranta | Local and global pairing instabilities of two interlaced helical vortices[END_REF]. The current study uses water channel experiments and simulations with a multi-fidelity vortex solver to investigate the impact of rotor asymmetry on the wake of a three-bladed rotor, the configuration usually used for horizontal axis wind turbines.

Methods

The experiments were conducted in a recirculating free-surface water channel with a 150 cm × 38 cm × 50 cm (length × width × height) test section, the same as that used in [START_REF] Quaranta | Local and global pairing instabilities of two interlaced helical vortices[END_REF]. The three-bladed rotor was constructed using additive manufacturing, with separate parts for the hub, nose, and blades, so each blade could be removed and replaced independently. It had a radius of 9.0 cm and a tip chord of 2.3 cm, with blades designed using the Glauert method [START_REF] Glauert | Airplane Propellers, Div. L of Aerodynamic Theory IV[END_REF] for the outer 50% of the radius. The NACA2414 airfoil was used for the blade cross-section as a compromise between low-Reynolds number performance and a minimum necessary thickness for additive manufacturing. In the experiments, the rotor rotated at 3 Hz, with a flow speed of 56 cm/s, yielding a tip speed ratio of 3 and a tip-chord-based Reynolds number of 40,000. The wake of the rotor was measured using particle image velocimetry (PIV) and dye visualizations. For each case, one or two of the rotor blades were removed and replaced with blades that were modified in some way, such as a length increase, an out-of-plane deflection, or a combination of both.

To complement the experiments, simulations of a few select cases will be conducted using the MIRAS code, which models the vortices shed from a wind turbine rotor using a hybrid filament-mesh method [START_REF] Ramos-García | Hybrid vortex simulations of wind turbines using a three-dimensional viscous-inviscid panel method[END_REF]. These simulations will provide additional information that is not available in the experiments, such as the downstream evolution of the wake beyond the field of view of the water channel test section. Velocity fields from planes positioned where a downstream turbine would be located will elucidate the impact of the perturbed wake on other turbines within a wind farm.

Results and Conclusions

Experimental results show that rotor asymmetry causes vortex pairing to occur earlier (further upstream) than it would without the perturbation. The location of this pairing can be observed in the swapping of the axial positions of adjacent helices, whose downstream distance is indicated by the arrow on the dye visualizations of the blade tip vortices depicted in figure 1. The type of rotor asymmetry affects the onset of pairing, with some perturbations significantly more effective than others of the same magnitude. For example, the white blade in figure 1(b) is longer than the other two blades by 5% of the distance between neighboring vortex loops, h, and deflected upstream by 5% of h, while the white blade in figure 1(c) is lengthened and deflected by the same magnitude, but in the downstream direction. Clearly, the perturbation shown in figure 1(c) induces vortex pairing much earlier than that in figure 1(b). The difference in effectiveness of these two asymmetries of the same magnitude is related to the direction of the mutually-induced strain in a system of co-rotating vortices [START_REF] Leweke | Dynamics and instabilities of vortex pairs[END_REF]. With 14 different rotor asymmetries tested, many additional conclusions can be drawn about the most effective perturbations for triggering the mutual inductance instability in the helical tip vortex system. These findings can be used to design turbines with wakes that break down faster, thereby reducing their negative impact on downstream machines within a wind farm. With the increasing interest in segmented blade technology [START_REF] Peeters | The concept of segmented wind turbine blades: A review[END_REF], a blade tip generating the desired vortex perturbation could be implemented on new or existing turbines. To explore the feasibility of implementation, investigation into the impact on the loading of the asymmetric turbine itself and quantification of the effect of the perturbed wake on a downstream turbine are currently underway.

Figure 1 :

 1 Figure 1: Dye visualizations of blade tip vortices in the wake of a three-bladed rotor, including (a) the baseline symmetric case with an entirely black rotor, (b) the case with one blade (in white, pointing upwards) extended and deflected upstream relative to the other two blades, and (c) the case with one blade (in white, pointing upwards) extended and deflected downstream. The white arrows in (b) and (c) indicate the distance from the rotor plane to the location of vortex pairing.


PhD Abhinav K Jha
email: a.jha@wustl.edu

Tyler J Bradshaw

Irène Buvat

Mathieu Hatt

Chi Liu

Nancy F Obuchowski

Babak Saboury

Piotr J Slomka

John J Sunderland

Richard L Wahl

Zitong Yu

Sven Zuehlsdorff

Arman Rahmim

Ronald Boellaard

Nuclear Medicine and Artificial Intelligence: Best Practices for Evaluation (the RELAINCE guidelines)

Keywords: Artificial intelligence, Clinical task-based evaluation, PET, SPECT, proof of concept, technical efficacy, clinical utility, post-deployment monitoring, generalizability

Noteworthy:

• We propose a four-class framework to evaluate AI algorithms for nuclear-medicine imaging.

• We provide the RELAINCE (Recommendations for Evaluation of AI for Nuclear Medicine) guidelines to evaluate promise, technical efficacy, clinical utility, and post-deployment efficacy of AI algorithms. • We outline key elements that should be specified as the output of an AI-algorithm evaluation study.

team was tasked with outlining best practices for evaluation of AI methods for nuclear-medicine imaging. This report has been prepared by this team.

In medical imaging, images are acquired for specific clinical tasks. These tasks can be broadly classified into three categories: classification, quantification, or a combination of both. An oncological PET image may be acquired for the task of tumor-stage classification or for quantification of tracer uptake in tumor. However, current Al-algorithm evaluation strategies are often task agnostic. For example, AI algorithms for reconstruction and post-processing are often evaluated by measuring image fidelity to a reference standard using figures of merit (FoMs) such as root mean square error. Similarly, AI-based segmentation algorithms are evaluated using FoMs such as Dice scores. However, recent studies show that these evaluation strategies may not correlate with clinical task performance [START_REF] Yang | CT-less Direct Correction of Attenuation and Scatter in the Image Space Using Deep Learning for Whole-Body FDG PET: Potential Benefits and Pitfalls[END_REF][START_REF] Yu | AI-based methods for nuclear-medicine imaging: Need for objective task-specific evaluation[END_REF][START_REF] Li | Task-based performance evaluation of deep neural network-based image denoising[END_REF][START_REF] Zhu | Comparing clinical evaluation of PET segmentation methods with reference-based metrics and no-gold-standard evaluation technique[END_REF][START_REF] Kc | Deep neural networks-based denoising models for CT imaging and their efficacy[END_REF]. One study observed that a reconstruction algorithm for whole-body FDG-PET using fidelity-based FoMs indicated excellent performance, but on the lesiondetection task, the algorithm was yielding both false negatives and positives due to blurring and pseudo-lowuptake patterns, respectively [START_REF] Yang | CT-less Direct Correction of Attenuation and Scatter in the Image Space Using Deep Learning for Whole-Body FDG PET: Potential Benefits and Pitfalls[END_REF]. Similarly, Yu et al. observed that evaluation of an AI-based denoising method for cardiac SPECT using fidelity-based FoMs suggested significantly improved performance compared to without denoising. However, when evaluated on the task of detecting cardiac perfusion defects, the performance of the AI-based denoising method was equivalent, if not worse, to that obtained without applying denoising [START_REF] Yu | AI-based methods for nuclear-medicine imaging: Need for objective task-specific evaluation[END_REF]. Such findings demonstrate that task-agnostic approaches to evaluate AI methods have major limitations. Thus, evaluation strategies that measure performance on clinical tasks are needed.

Evaluation studies should also quantitatively describe the generalizability of the AI algorithm to different population groups and to different technical factors, such as scanners, acquisition, and analysis protocols. Finally, evaluations should yield quantitative measures of performance to enable clear comparison with standard-of-care approaches and other methods and provide guidance for clinical utility. To account for these factors, we recommend that an evaluation strategy for an AI algorithm should always produce a claim consisting of the following components (Fig. 1):

-A clear definition of the task -Patient population(s) for whom the task is defined -Exact definition of the imaging process (acquisition, reconstruction and analysis protocols) -The process to extract task-specific information -Figure of merit to describe task performance, including process to define reference standard We describe each component in Sec. B. To produce such a claim, we propose an evaluation framework in Sec. C. The framework categorizes the evaluation strategies into four classes: proof-of-concept, technical, clinical and post-deployment evaluation. This framework will guide AI-developers to conduct the evaluation study that provides evidence to support their intended claim. In Sec. C, we also provide the best practices for conducting evaluations for each class. The report finally provides the RELAINCE (Recommendations for Evaluation of AI for Nuclear Medicine) guidelines, that enlist these best practices.

In this paper, the terms "training", "validation" and "testing" will be used according to their usual meaning in the AI literature. More specifically, training, validation and testing will denote the building of a model on a specific dataset, the tuning/optimization of the model parameters, and the evaluation of the optimized model. The focus of this paper is on evaluation. The development of AI-based algorithms using the training and validation procedures is described in a companion paper [START_REF] Bradshaw | Nuclear Medicine and Artificial Intelligence: Best Practices for Algorithm Development[END_REF].

B. Components of the claim

The claim provides a clear and descriptive characterization of the performance of the AI algorithm. The components of a claim are shown in Fig. 1 and described below.

B.1. Definition of the clinical task:

In this paper, the term "task" refers to the clinical goal for which the image was acquired. Broadly, in NM imaging, tasks can be grouped into three categories: classification (which can include lesion detection), quantification, or a combination of both. In the classification task, the patient image is used to classify the patient into a category. For example, identifying if cancer is present or absent, or the cancer stage from an oncological PET image. Similarly, using an AI algorithm to predict whether a patient is expected to respond to therapy or not would be a classification task. In a quantification task, the patient image is used to quantify some parameter, for example, quantifying standardized uptake value (SUV) in an oncological PET image.

B.2: Patient population for whom the task is defined:

The performance of an imaging method can be affected by the physical and statistical properties of the imaged patient population. Results for one population may not necessarily translate to others [START_REF] Zech | Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study[END_REF][START_REF] Noor | Can we trust AI not to further embed racial bias and prejudice?[END_REF]. Thus, the patient population should be clearly defined in the claim. This includes aspects such as sex, ethnicity, age group, geographic location, disease stage, social determinants of health, and other disease and application-relevant biological variables. Providing these elements in the claim will inform the generalizability of the method.

B.3. Definition of imaging process:

The imaging system, acquisition protocol, and reconstruction and analysis parameters may affect task performance. For example, an AI algorithm evaluated for a high-resolution PET system may not apply to systems with lower resolution, since the method may rely on high-frequency features captured by the high-resolution system [START_REF] Reuzé | Prediction of cervical cancer recurrence using textural features extracted from 18F-FDG PET images acquired with different scanners[END_REF]. Depending on the method, specific acquisition-protocol parameters may need to be specified or the requirement to comply with a certain accreditation standard, such as SNMMI-Clinical Trial Network, RSNA QIBA profile, and the European Association of Nuclear Medicine Research Ltd (EARL) standards. For example, an AI-based denoising approach for ordered subsets expectation maximization (OSEM)-based reconstructed images may not apply to images reconstructed using filtered-backprojection or even for a different number of OSEM iterations since noise properties change with the number of iterations. Thus, depending on the application, these parameters should be specified in the claim. Further, if the algorithm was evaluated across multiple scanners, or with multiple protocols, that should be specified. This would strengthen confidence in generalizability of the algorithm.

B.4. The process to extract task-specific information:

Task-based evaluation of an AI imaging algorithm requires a strategy that extracts task-specific information from the images. For classification tasks, a typical strategy is to have human observer(s) read the images, detect lesions and classify the patient or each detected lesion into a certain class (e.g., malignant or benign). Here, the competency of the observer (multiple trained radiologists/one trained radiologist/resident/untrained reader) will impact task performance. Further, the choice of the strategy may provide more confidence about the validity of the algorithm. This is also true for quantification and joint classification/quantification tasks. Thus, this strategy should be specified in the claim.

Patient population

• Demographics including sex, age, ethnicity should be stated AUC WashU

Imaging process

• Imaging system(s)

• image-acquisition protocol(s) • Single/multi center (CIs), which provide a measure of uncertainty in the performance. To obtain the FoM, a reference standard is needed. The process to define the reference standard should be stated.

By providing all the components of a claim, developers will describe the generalizability of the method. Figure 2 presents a schematic that shows how different levels of generalizability can be established. Some key points from this figure are as listed below: -Providing evidence for generalizability requires external validation. This is defined as validation where some portion of the testing study, such as the data (patient population demographics) or the process to acquire the data, is different from that in the development cohort. Depending on the level of external validation, the claim can be appropriately defined.

-For a study that claims to be generalizable across populations, scanners, and readers, the external cohort would be from different patient demographics, with different scanners, and analyzed by different readers than the development cohort, respectively. -Multi-center studies provide higher confidence about generalizability compared to single-center studies since they typically include some level of external validation (patients from different geographical locations/different scanners/different readers).

C. Methods for evaluation:

The evaluation framework for AI algorithms is provided in Fig. 3. The four classes of this framework are differentiated based on their objectives, as briefly described below, with details provided in Sec. C.1-C.4. An example for an AI low-dose PET reconstruction algorithm is provided. Fig. 3 contains another example for an AI-based automated segmentation algorithm. A detailed example of using this framework to evaluate a hypothetical AI-based transmission-less attenuation compensation method for SPECT [START_REF] Garcia | SPECT attenuation correction: An essential tool to realize nuclear cardiology's manifest destiny[END_REF] is provided in Supplemental section A.

-Class 1: Proof-of-concept (POC) evaluation: Shows the novelty and promise of an algorithm proposed using task-agnostic metrics. Suitable for method-development studies. Provides promise for further clinical task-specific evaluation.

Example: Evaluating the AI PET reconstruction algorithm using root mean square error. -Class 2: Technical evaluation: Quantifies technical performance of an algorithm on a clinical task using measures such as accuracy, repeatability, and reproducibility. In the subsections below, for each class of evaluation, we provide the key objectives, the best practices for study design, including determining study type, data collection including sample-size considerations, defining a reference standard, and choosing FoMs (Fig. 4), and finally a generic structure for the claim.

C.1 Proof-of-concept (POC) evaluation

Objective: Quantitatively demonstrate the technological innovations of newly developed AI algorithms using task-agnostic FoMs and provide evidence that motivates further clinical task-specific evaluation. Clinical or task-specific technical claims should not be put forth based on POC evaluation.

Rationale for task-agnostic objective:

A newly developed AI method may be suitable for multiple clinical tasks. For example, a segmentation algorithm may be applicable to radiation-therapy planning, estimating volumetric (e.g., metabolic tumor volume (MTV)) or radiomic features, or monitoring therapy response.

Evaluating the algorithm on all these tasks would require multiple studies. Further, the developer may not have the necessary resources (such as a large, representative dataset) to conduct these studies. Thus, a taskagnostic objective facilitates timely dissemination and widens the scope of newly developed AI methods.

Study design:

The following are recommended best practices to conduct POC evaluation of an AI algorithm. Best practices to develop the algorithm are covered in the companion paper [START_REF] Bradshaw | Nuclear Medicine and Artificial Intelligence: Best Practices for Algorithm Development[END_REF]. a. Data collection: In POC evaluation, the study can use realistic simulations, physical phantoms, and/or retrospective clinical or research data, usually collected for a different purpose, e.g., routine diagnosis.

The data used for evaluation may come from the development cohort, i.e., the same overall cohort that the training and validation cohorts were drawn from. However, it is important that there is no overlap between these data. Public databases, such as those available at TCIA [START_REF] Clark | The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository[END_REF] and from challenges * can also be used. b. Defining reference standard: For POC evaluations conducted with simulation and physical phantoms, the ground truth is known. For clinical data, curation by readers may be used. The curation quality need not be of the highest quality. For example, curations by single reader may be sufficient. c. Testing procedure: The testing procedure should be designed to demonstrate promising technological innovation. The algorithm should thus also be compared against a reference or standard of care, and preferably other state-of-the-art algorithms.

d. Figures of merit:

While the evaluation is task-agnostic, the FoMs should be carefully chosen to show promise for progression to clinical task evaluation. For example, evaluating a new denoising algorithm that overly smooths the image at the cost of resolution using the FoM of contrast-to-noise ratio may be misleading. In those cases, a FoM such as structural similarity index may be more relevant. For this reason, we recommend evaluation of the algorithms using multiple FoMs. The list of these FoMs is provided in supplemental table 1.

Output claim of the POC study:

The claim should state the following:

-The application (e.g., segmentation, reconstruction) for which the method is proposed.

-The patient population.

-The imaging and image-analysis protocol(s).

-Process to define reference standard -Performance as quantified with a task-agnostic evaluation metric. We re-emphasize that since this is a POC study, the claim should not be interpreted as an indication of the algorithm's expected performance in a clinical setting. Example claim: Consider the evaluation of a new segmentation algorithm. The claim could read as follows:

"An AI-based PET segmentation algorithm evaluated on 50 patients with locally advanced breast cancer acquired on a single scanner with single-reader evaluation yielded mean Dice scores of 0.78 (95% CI 0.71-0.85)."

C.2 Technical efficacy evaluation

Objective: To evaluate the technical performance of an AI algorithm on specific clinically relevant tasks such as those of detection and quantification using FoMs that quantify aspects such as accuracy (discrimination accuracy for detection task and measurement bias for quantification task) and precision (reproducibility and repeatability). The objective is not to assess the utility of the method in clinical-decision making, since clinicaldecision making is a combination of factors beyond technical aspects, such as prior clinical history, patient biology, other patient characteristics (age/sex/ethnicity) and results of other clinical tests.

Study design: Given the goal of evaluating technical performance, the evaluation should be performed in controlled settings. Practices for designing such studies are outlined below. A framework and summary of tools to conduct these studies is provided in Jha et al [START_REF] Jha | Objective task-based evaluation of artificial intelligence-based medical imaging methods: Framework, strategies and role of the physician[END_REF].

a. Study type: A technical evaluation study can be conducted through the following mechanisms: 1) Realistic simulations are studies conducted with anthropomorphic digital phantoms simulating patient populations, where measurements corresponding to these phantoms are generated using accurately simulated scanners. A specific class referred to as virtual clinical trials (VCTs) can be used to obtain population-based inferences [START_REF] Abadi | Virtual clinical trials in medical imaging: a review[END_REF][START_REF] Yu | A physics and learning-based transmission-less attenuation compensation method for SPECT[END_REF]. 2) Anthropomorphic physicalphantom studies are conducted on the scanners with devices that mimic the human anatomy and physiology. 3) Clinical-data-based studies where clinical data is used to evaluate the technical performance of an AI algorithm. For example, repeatability study of an AI algorithm measuring MTV in test-retest PET scans. The tradeoffs with these three study types are listed in Table 1. Each study type can be single or multiscanner/center studies, depending on the claim:

-Single-center/single-scanner studies are typically performed with a specific system, image acquisition and reconstruction protocols. In these studies, the algorithm performance can be evaluated for variability in patients, including different demographics, habitus, or disease characteristics, while keeping the technical aspects of the imaging procedures constant. These studies can measure the sensitivity of the algorithm to patient characteristics. They can also study the repeatability of the AI algorithm. Reproducibility may be explored by varying technical factors such as reconstruction settings. -Multi-center/multi-scanner studies are mainly suitable to explore the sensitivity of the AI algorithm to acquisition variabilities, including variability in imaging procedures, systems, reconstruction methods and settings, and patient demographics if using clinical data. Typically, multi-center studies are performed to improve patient accrual in trials and therefore the same in-and exclusion criteria are applied to all centers. These studies can identify AI algorithms that are sensitive to variations in scanner performance and reconstruction protocols. Further, multicenter studies can help assess the need for harmonization of imaging procedures and system performances.

b. Data collection:

• Realistic simulation studies: To conduct realistic simulations, multiple digital anthropomorphic phantoms are available [START_REF] Kainz | Advances in Computational Human Phantoms and Their Applications in Biomedical Engineering-A Topical Review[END_REF]. In virtual clinical trial-based studies, the distribution of simulated image data should be similar to that observed in clinical populations. For this purpose, parameters derived directly from clinical data can be used during simulations [START_REF] Leung | A physics-guided modular deep-learning based automated framework for tumor segmentation in PET[END_REF]. Expert reader-based studies can be used to validate realism of simulations [START_REF] Liu | Observer study-based evaluation of a stochastic and physics-based method to generate oncological PET images[END_REF].

Next, to simulate the imaging systems, tools such as GATE [START_REF] Jan | GATE: a simulation toolkit for PET and SPECT[END_REF], SIMIND [START_REF] Ljungberg | The SIMIND Monte Carlo program[END_REF], SimSET [START_REF] Lewellen | The SimSET program[END_REF],

PeneloPET [START_REF] España | PeneloPET, a Monte Carlo PET simulation tool based on PENELOPE: features and validation[END_REF], and other tools [START_REF] Jha | Objective task-based evaluation of artificial intelligence-based medical imaging methods: Framework, strategies and role of the physician[END_REF] can be used. Different system configurations, including those replicating multi-center settings, can be simulated. If the methods use reconstruction, then clinically used reconstruction protocols should be simulated. Simulation studies should not use data that was used for training/validation of the algorithm. • Anthropomorphic physical-phantoms studies: For clinical relevance, the tracer uptake and acquisition parameters when imaging these phantoms should be similar to that in clinical settings. To claim generalizable performance across different scanner protocols, different clinical acquisition and reconstruction protocols should be used. A phantom used during training should not be used during evaluation irrespective of changes in acquisition conditions between training and test phases. • Clinical data: Technical evaluation studies will typically be retrospective. Use of external datasets, such as those from an institution or scanner not used for method training/validation, is recommended. Public databases such as TCIA may also be used. Selection criteria should be defined.

c. Process to extract task-specific information:

• Classification task: Performance of AI-based reconstruction or post-reconstruction algorithms should ideally be evaluated using psychophysical studies by expert readers. Methods such as two alternative forced choice (2-AFC) tests and ratings-scale approaches could be used. When humanobserver studies are infeasible, numerical observers, such as the channelized Hotelling observer, could be used [START_REF] Barrett | Model observers for assessment of image quality[END_REF][START_REF] Abbey | Human-and model-observer performance in ramp-spectrum noise: effects of regularization and object variability[END_REF][27]. This is a better choice than using human observers with limited training, who may yield misleading measures of task performance. AI algorithms for optimizing instrumentation/acquisition can be evaluated directly on projection data, which provides the benefit that the evaluation would be agnostic to the choice of the reconstruction and analysis method [START_REF] Gross | Optimizing a multiple-pinhole SPECT system using the ideal observer[END_REF][START_REF] Rong | Optimization of energy window for 90Y bremsstrahlung SPECT imaging for detection tasks using the ideal observer with model-mismatch[END_REF]. In this case, observers that are optimal in some sense, such as the ideal observer (which yields the maximum possible area under the receiver operating characteristics (ROC) curve (AUC) of all observers) should be used [START_REF] Barrett | Model observers for assessment of image quality[END_REF]. The ideal observer can be computationally challenging to obtain in clinical settings, and to address this, different strategies are being developed [START_REF] Clarkson | Fisher information and surrogate figures of merit for the task-based assessment of image quality[END_REF][START_REF] Li | Use of Sub-ensembles and Multi-template Observers to Evaluate Detection Task Performance for Data That are Not Multivariate Normal[END_REF]. An example of evaluating a hypothetical AI method for improving timing resolution in a time-of-flight PET system is presented in Jha et al [START_REF] Jha | Objective task-based evaluation of artificial intelligence-based medical imaging methods: Framework, strategies and role of the physician[END_REF]. • Quantification task: The task should be performed using optimal quantification procedures to ensure that the algorithm evaluation is not biased due to a poor quantification process. Often, performing quantification requires an intermediate manual step. For example, the task of regional uptake quantification from reconstructed images may require manual delineation of regions of interest. Expert readers should perform these steps. NM images are noisy and corrupted by image-degrading processes. Thus, the process of quantification should account for the physics and statistical properties of the measured data. For example, if evaluating a segmentation algorithm on the task of quantifying a certain feature from the image, the process of estimating that feature should account for the image-degrading processes and noise [START_REF] Jha | Objective task-based evaluation of artificial intelligence-based medical imaging methods: Framework, strategies and role of the physician[END_REF]. If only using the measurements and not incorporating any prior information on the parameters that are quantified, maximum-likelihood estimation methods are an excellent choice [START_REF] Carson | A Maximum Likelihood Method for Region-of-Interest Evaluation in Emission Tomography[END_REF]. If using prior information, estimators that yield maximum-a-posteriori [START_REF] Whitaker | Estimating random signal parameters from noisy images with nuisance parameters: linear and scanning-linear methods[END_REF] and posterior-mean [START_REF] Liu | A Bayesian approach to tissue-fraction estimation for oncological PET segmentation[END_REF] estimates could be used. In several cases, measuring quantitative features directly from projection data may yield more reliable quantification [START_REF] Carson | A Maximum Likelihood Method for Region-of-Interest Evaluation in Emission Tomography[END_REF] and can be considered. • Joint classification/quantification task: These tasks should again be performed optimally. If manual inputs are needed for the classification or quantification component of the task, these should be provided by expert readers. Numerical observers such as channelized scanning linear observers [START_REF] Tseng | Combination of detection and estimation tasks using channelized scanning linear observer for CT imaging systems[END_REF] and those based on deep learning [START_REF] Li | Supervised learning-based ideal observer approximation for joint detection and estimation tasks[END_REF] can also be used.

d. Defining a reference standard:

For simulation studies, the ground-truth is accurately and precisely known. Experimental errors may arise when obtaining ground truth from physical-phantom studies, and preferably, these should be modeled during the statistical analysis. For clinical studies, ground truth is commonly unavailable. A common workaround is to define a reference standard. The quality of curation to define this standard should be high. When the reference standard is expert defined, multi-reader studies are preferred where the readers have not participated in the training of the algorithm, and where each reader independently interprets images, blinded to the results of the AI algorithm and the other readers [START_REF] Miller | Gold standards and expert panels: a pulmonary nodule case study with challenges and solutions[END_REF]. In other cases, the reference standard may be the current clinical practice. Finally, another approach is to use no-gold-standard evaluation techniques, which have shown ability to evaluate algorithm performance on quantification tasks even without any ground truth [START_REF] Hoppin | Objective Comparison of Quantitative Imaging Modalities Without the Use of a Gold Standard[END_REF][START_REF] Jha | A no-gold-standard technique for objective assessment of quantitative nuclear-medicine imaging methods[END_REF][START_REF] Jha | Practical no-gold-standard evaluation framework for quantitative imaging methods: application to lesion segmentation in positron emission tomography[END_REF].

e. Figures of merit:

A list of FoMs for different tasks is provided in Supplemental Table 2. Example FoMs include AUC to quantify accuracy on classification tasks, bias, variance and ensemble mean square error to quantify accuracy, precision and overall reliability on quantification tasks, and area under the estimation ROC (EROC) curve for joint detection/classification tasks. For a multicenter study, variability of these FoMs across centers, systems and/or observers should be reported.

Output claim from evaluation study:

The claim will consist of the following components:

-The clinical task (detection/quantification/combination of both) for which the algorithm is evaluated.

-The study type (simulation/physical phantom/clinical).

-If applicable, the imaging and image-analysis protocol or accreditation standards that need to be adhered to for this claim to hold. -If clinical data, process to define ground truth.

-Performance, as quantified with task specific FoMs.

Example claim: Consider the same automated segmentation algorithm as mentioned in Sec. C.1, being evaluated to estimate MTV. The claim could be: "An AI-based fully automated PET segmentation algorithm yielded MTV values with a normalized bias of X% (provide 95% confidence intervals) as evaluated using physical-phantom studies with an anthropomorphic thoracic phantom conducted on a single scanner in a single center."

C.3 Clinical utility evaluation

Objective: Evaluate the utility of an AI algorithm on making clinical decisions, including diagnostic, prognostic, predictive and therapeutic decisions. While technical evaluation was geared towards quantifying the performance of a technique in controlled settings, clinical evaluation investigates clinical utility in a practical setting.

Study design:

a. Study type: Following study types can be used: -Retrospective study: A retrospective study employs existing data sources. In a blinded retrospective study, readers analyzing the study data are blinded to the relevant clinical outcome. Retrospective studies are the most common mechanism to evaluate AI algorithms. Advantages of these studies include low costs and quicker execution. These studies can provide considerations for designing prospective studies. With rare diseases, these may be the only viable mechanism for evaluation. However, these studies cannot conclusively demonstrate causality between the algorithm output and the clinical outcome. Also, these studies may be affected by different biases such as bias in patient selection.

-Prospective observational study: In this study, the consequential outcomes of interest occur after study commencement, but the decision to assign participants to an intervention is not influenced by the algorithm [START_REF] Berger | Prospective Observational Studies to Assess Comparative Effectiveness: The ISPOR Good Research Practices Task Force Report[END_REF]. These studies are often secondary objectives of a clinical trial. -Prospective interventional study: In a prospective interventional study of an AI algorithm, the decision to assign the participant to an intervention depends on the AI-algorithm output. These studies can provide stronger evidence for causation of the AI-algorithm output to clinical outcome. The most common and strongest prospective interventional study design are randomized control trial (RCTs), although other designs such as non-randomized trials and quasi-experiments are possible [START_REF] Thiese | Observational and interventional study design types; an overview[END_REF]. RCTs are considered the gold standard of clinical evaluation but are typically logistically challenging, expensive, and time consuming, and should not be considered as the only means to ascertain and establish effective algorithms. -Real-world post-deployment evaluation studies: These studies make use of real-world data (RWD) from AI algorithms that have received regulatory clearance 43 . Such studies have the potential to provide information on a wider patient population that may be difficult to obtain through the prospective interventional study. Moreover, the RWD can not only be leveraged to improve upon the performance of the initially cleared AI device but also be used to evaluate new AI medical applications that require the same/similar data as the initially cleared AI-module, thus saving time and cost. However, critical to this type of study is that its study design be carefully crafted with a study protocol and analysis plan defined prior to retrieving/analyzing the RWD [START_REF] Sherman | Real-World Evidence -What Is It and What Can It Tell Us?[END_REF][START_REF]Use of real-world evidence to support regulatory decision-making for medical devices[END_REF]. Special attention should be paid while designing these studies to negate bias [START_REF] Tarricone | What type of clinical evidence is needed to assess medical devices?[END_REF].

Choosing the study type:

This is a multi-factorial decision (Fig. 5). To decide on the appropriate study type, we make a distinction between AI algorithms that make direct interventional recommendations (prescriptive AI) and those that do not (descriptive AI): -A purely descriptive AI algorithm does not make any direct interventional recommendations but may alter clinical decision making. The algorithms can be further categorized into those that provide a description about the present (e.g., for diagnosis, staging, therapy response assessment) vs. those that predict the future (e.g., prognosis of therapy outcome, disease progression, overall survival).

There are close links between these two categories, and the line between them will likely be increasingly blurred in the era of AI: e.g., more-refined AI-derived cancer staging that is trained with outcome data and therefore becomes highly predictive of outcome. A well-designed blinded retrospective study is sufficient to evaluate a purely descriptive AI system. However, if clinical data for a retrospective study do not exist, a prospective observational or real-world study is required. -A prescriptive AI algorithm makes direct interventional recommendation(s). It may have no autonomy (i.e., only making a recommendation to a physician) or full autonomy (no supervision), or grades in between. For a prescriptive AI algorithm that is not autonomous, a prospective interventional study is recommended. A well-designed real-world study may be used as a substitute. However, for a fully autonomous prescriptive AI system of the future (e.g., fully automated therapy delivery), such a study may be required. Future studies and recommendations are needed for autonomous prescriptive AI systems, as the field is not mature enough. Thus, we limit the scope of this section to only those systems that have expert physician supervision.

b. Data collection

An AI algorithm yielding strong performance using data from one institution may perform poorly on data from other institutions [START_REF] Zech | Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study[END_REF]. Thus, we recommend that for clinical evaluation, test data should be collected from different, and preferably multiple, institutions. Results from external institutions can be compared with internal hold-out samples (data from the same institution not used for training) to evaluate generalizability. To avoid variation due to site selection used for the external validation, or random bias in internal sample selection, leave-one-site repeated hold-out (for example 10-fold cross-validation) strategy can be used with a dataset that is completely independent from the training and validation dataset.

To demonstrate applicability over a certain target population, the collected data should be representative of that population in terms of demographics. When the goal is studying performance on a specific population subset (e.g., patients with large body mass indices) or check sensitivity of the method to certain factors (e.g., patients with metallic implants), the other criteria for patient selection should be unbiased. This ensures that the evaluation specifically studies the effect of that factor.

In studies that are retrospective or based on real-world data, once a database has been set up corresponding to a target population using existing datasets, patients should be randomly selected from this database to avoid selection bias.

Sample-size considerations:

The study must have a predefined statistical analysis plan [START_REF] Hemingway | Ten steps towards improving prognosis research[END_REF]. The sample size is task dependent. For example, if the claim of improved AUC with the use of the AI method vs. a non-AI approach or standard clinical analysis is studied, then the sample size will be dictated by the detection of the expected change between the two ROC areas. Inputs required for the power-analysis to compute sample size may be obtained from the POC and technical evaluation studies. Pilot studies could also be conducted to estimate sample sizes. c. Defining reference standard: For clinical evaluation, the reference standard should be carefully defined. This requires in-depth clinical and imaging knowledge of the data. Thus, medical experts should be involved in defining task-specific standard. Some reference standards are listed below: -Clinical outcomes: Eventually the goal of imaging is to improve clinical outcomes. Outcomes such as overall survival, progression-free survival, major clinical events, and hospitalization, could thus serve as gold standards, especially for demonstrating clinical utility in predictive and prognostic tasks. A decrease in the use of resources as a result of the AI tool with comparable outcomes could also be a relevant and improved outcome (e.g., fewer non-essential call back tests with AI). -External standard: For disease diagnosis tasks, when available, an external standard such as invasive findings, e.g., biopsy-pathology or invasive coronary angiography, or some other definitive diagnosis (derived from other means than the images utilized) should be considered. -Trained-reader-defined clinical diagnosis: For diagnostic tasks, expert reader(s) can be used to assess the presence/absence of the disease. Similar best practices as outlined in Sec. C.2 should be followed to design these studies. However, note that, unlike technical evaluation, where the goal was restricted to defect detection, here the goal is disease diagnosis. Thus, the readers should also be provided other factors that are used to make a clinical decision, such as the patient age, sex, ethnicity, other clinical factors that may impact disease diagnosis, and results from other clinical tests.

Note that if the reference standard is defined using a standard-of-care clinical protocol, it may not be possible to claim improvement over with this protocol. In such a case, agreement-based studies can be performed and concordance with this protocol results could be claimed within certain confidence limits. For example, to evaluate the ability of an AI-based transmission-less attenuation compensation algorithm for SPECT/PET, we may evaluate agreement of the estimates yielded by this algorithm with that obtained when a CT is used for attenuation compensation [START_REF] Shi | Deep learning-based attenuation map generation for myocardial perfusion SPECT[END_REF].

d. Figure of merit:

These are summarized in supplemental table 2. To evaluate performance on diagnosis tasks, the FoMs of sensitivity, specificity, ROC curves, and AUC can be used. In well-defined populations with known disease prevalence, parameters such at the PPV and NPV may be operationally significant, as well. Since the goal is to demonstrate clinical utility, sensitivity and specificity may be clinically more relevant than ROC analysis. To demonstrate clinical utility in predictive and prognostic tasks, in addition to AUC, FoMs that quantify performance in predicting future events such as Kaplan-Meier estimators, prediction risk score and median time of future events can be used.

Output claim from clinical evaluation study:

The claim will state the following: -The clinical task for which the algorithm is evaluated.

-The patient population over which the algorithm was evaluated.

-The specific imaging and image-analysis protocol(s) or standards followed.

-Brief description of study design: Blinded/non-blinded, randomized/non-randomized, retrospective/prospective/post-deployment, observational/interventional, number of readers. -Process to define reference standard and figure of merit to quantify clinical utility.

Example claims:

i.Retrospective study: The average AUC of 3 experienced readers on the task of detecting obstructive coronary artery disease from myocardial perfusion imaging (MPI) PET scans improved from X to Y, representing an estimated difference of 𝛥 (95% CI for 𝛥), when using an AI-based computer aided diagnosis (CAD) tool compared to not using this tool, as evaluated using a blinded retrospective study. ii.Prospective observational study: Early change in MTV measured from FDG-PET images using an AIbased segmentation algorithm yielded an increase in AUC from X to Y, representing an estimated difference of 𝛥 (95% CI for 𝛥) in predicting pathological complete response in patients with stage II/III breast cancer, as evaluated using a non-randomized prospective observational study. iii.Prospective interventional study: Changes in PET-derived quantitative features using an AI algorithm during the interim stage of therapy were used to guide treatment decisions in patients with stage III NSCLC. This led to an X% increase (95% CI) in responders than when the AI algorithm was not used to guide treatment decisions, as evaluated using a randomized prospective interventional study.

C.4. Post-deployment evaluation

Objective: Post-deployment evaluation has multiple objectives including monitoring algorithm performance post clinical deployment, off-label evaluation, and collecting feedback for proactive development (Fig. 6).

Evaluation strategies: a. Monitoring: Critically important in post-deployment monitoring of an AI method is quality and patient safety. It is imperative to monitor devices and follow reporting guidelines (such as adverse events), recalls and corrective actions. Fortunately, applicable laws and regulations require efficient processes in place.

Often, logging is used to identify root causes for equipment failure. However, the concept of logging can be expanded: advanced logging mechanisms could be employed to better understand use of a particular AI method. A simple use case is logging the frequency with which an AI algorithm is used in clinical workflow.

Measuring manual intervention for a workflow step that was designed for automation could provide a first impression of the performance in a clinical environment. However, more complex use cases may include the aggregation of data on AI-method performance and how this impacted patient and disease management. For wider monitoring, developers should also seek feedback from customers, including focus groups, customer complaint and inquiry tracking, and ongoing technical performance benchmarking [START_REF] Larson | Regulatory Frameworks for Development and Evaluation of Artificial Intelligence-Based Diagnostic Imaging Algorithms: Summary and Recommendations[END_REF]. This approach may provide additional evidence on algorithm performance and could assist in finding areas of improvements, clinical needs not yet well served or even deriving a hypothesis for further development.

Advanced data logging and sharing must be compliant with applicable patient privacy and data protection laws and regulations. Routinely conducted image-quality phantom studies provide a mechanism for post-deployment evaluation, in particular as sanity checks to assess that the AI algorithm was not affected by a maintenance operation such as a software update. These studies could include assessing contrast or SUV recovery, absence of non-uniformities or artifacts, and cold-spot recovery, and other specialized tests depending on the AI algorithm. Also, tests can be conducted to assure that there is a minimal or harmonized image quality as required by the AI tool for the configurations as stated in the claim.

AI systems likely will operate on data that is generated in non-stationary environments with shifting patient populations and where clinical and operational practices change over time [START_REF] Finlayson | The Clinician and Dataset Shift in Artificial Intelligence[END_REF]. Post-deployment studies may be needed to identify these dataset shifts and assess if recalibration or retraining of the AI method may be necessary to maintain performance [START_REF] Davis | A nonparametric updating method to correct clinical prediction model drift[END_REF] [START_REF] Feng | Learning to safely approve updates to machine learning algorithms[END_REF]. Monitoring the distribution of various descriptors of the patient population, including the demographics and the prevalence of the disease can provide cues for detection of dataset shifts. In case of changes in these demographics, the output of the AI algorithm can be verified by physicians for randomly selected test cases. A possible solution to data shift is continuous learning of the AI method [START_REF] Baweja | Towards continual learning in medical imaging[END_REF]. In supplemental section B, we discuss strategies [START_REF] Díaz-Rodríguez | Don't forget, there is more than forgetting: new metrics for Continual Learning[END_REF][START_REF] Goodfellow | An empirical investigation of catastrophic forgetting in gradient-based neural networks[END_REF][START_REF] Chaudhry | Riemannian walk for incremental learning: Understanding forgetting and intransigence[END_REF] to evaluate continuous-learningbased methods.

b. Off-label evaluation: Typically, an AI algorithm is trained and tested using a well-defined cohort of patients, in terms of patient demographics, applicable guidelines, practice preferences, reader expertise, imaging instrumentation, and acquisition and analysis protocols. However, the design of the algorithm may suggest that an algorithm may exhibit acceptable performance in patient groups outside of the intended scope of the algorithm. Here, a series of cases is appropriate to collect preliminary data that may suggest a more thorough trial. An example is a study where an AI algorithm that was trained on patients with lymphoma and lung cancer [START_REF] Sibille | 18F-FDG PET/CT Uptake Classification in Lymphoma and Lung Cancer by Using Deep Convolutional Neural Networks[END_REF] showed reliable performance in patients with breast cancer [START_REF] Weber | Just another "Clever Hans"? Neural networks and FDG PET-CT to predict the outcome of patients with breast cancer[END_REF]. Evaluation of AI algorithms in off-label cohorts can provide evidence of clinical utility beyond the settings initially targeted.

c. Collecting feedback for proactive development: Medical products typically have a lifetime longer than a decade. This motivates proactive development and maintenance to ensure that a product represents state of the art throughout its lifetime. This may be imperative for AI where technological innovations are expected to evolve at a fast pace in the coming years. A deployed AI algorithm offers the opportunity to pool data from several users. Specifically, registry approaches enable cost efficient pooling of uniform data, multi-center observational studies, and POC studies that can be used to develop a new clinical hypothesis or evaluate specific outcomes or particular disease.

Figures of merit:

We provide the FoMs for the studies where quantitative metrics of success are defined.

• Monitoring study with clinical data: Frequency of clinical usage of the AI algorithm, number of times the AI-based method changed clinical decisions or affected patient management. • Monitoring study with routine physical-phantom studies: Since these are mostly sanity checks, similar FoMs as when evaluating POC studies (Sec. C.1) may be considered. However, in case task-based evaluation is required, FoMs as provided in Sec. C.2 may be used. • Off-label evaluation: Similar FoMs as when evaluating technical efficacy and clinical utility.

D. Discussions and Summary

The key recommendations from this manuscript are summarized in Table 2. These are referred to as the RELAINCE (Recommendations for Evaluation of AI for Nuclear Medicine) guidelines, with the goal of improving the reliance of AI for clinical applications. Unlike other guidelines for the use of AI in radiology [START_REF] Dikici | Integrating AI into radiology workflow: levels of research, production, and feedback maturity[END_REF][START_REF] Mongan | Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers[END_REF][START_REF] Omoumi | To buy or not to buy-evaluating commercial AI solutions in radiology (the ECLAIR guidelines)[END_REF], these guidelines are exclusively focused on best practices for AI-algorithm evaluation.

This report advocates that an evaluation study should be geared towards putting forth a claim. The objective of the claim can be guided by factors such as the degree of impact on patient management, level of autonomy, and the risk that the method poses to patients. Risk categories have been proposed for medical software by the International Medical Device Regulators Forum (IMDRF) and subsequently adopted by the FDA [START_REF]Software as a medical device (SaMD): Clinical evaluation[END_REF]. The proposed risk categories range from 1 (low risk) to 4 (highest risk) depending on the vulnerability of the patient and the degree of control that the software has in patient management. The pathway that a developing technology will take to reach clinical adoption will ultimately depend on which risk category it belongs to, and investigators should assess risk early during algorithm development and plan accordingly [START_REF]Factors to consider when making benefit-risk determinations in medical device premarket approval and de novo classifications: Guidance for industry and Food and Drug Administration staff[END_REF].

In this report, we have proposed a four-class framework for evaluation. For clinical adoption, an algorithm may not need to pass through all classes. Further, not all these classes may be fully relevant to all algorithms. For example, an AI segmentation algorithm may require technical but not necessarily clinical evaluation. The types of studies required for an algorithm will depend on the claim. A developer may choose to report POC, technical, and clinical evaluation in the same multi-part study.

These evaluation studies should be multidisciplinary, and include computational imaging scientists, physicians, physicists, and statisticians right from the study-conception stage. In particular, physicians should be closely involved since they are the end users of these algorithms. Previous publications have outlined the important role of physicians in evaluation of AI algorithms [START_REF] Rubin | Artificial Intelligence in Imaging: The Radiologist's Role[END_REF], including for task-based evaluation of AI algorithms for nuclear medicine [START_REF] Jha | Objective task-based evaluation of artificial intelligence-based medical imaging methods: Framework, strategies and role of the physician[END_REF].

Class of evaluation Recommendation

Proof of concept evaluation

Ensure no overlap between development and testing cohort.

Check that ground-truth quality is reasonable.

Provide comparison with conventional and state-of-the-art methods.

Choose figures of merit that motivate further clinical evaluation.

Technical evaluation

Choose clinically relevant tasks: Detection/quantification/combination of both.

Determine the right study type: Simulation/phantom/clinical.

Ensure that simulation studies are realistic and account for population variability.

Testing cohort should be external.

Ground truth should be high quality and correspond to the task Define an optimal strategy to extract task-specific information Choose figures of merit that quantify task performance.

Clinical evaluation

Determine study type: Retrospective, prospective observational, prospective interventional, or post-deployment real-world studies Testing cohort must be external.

Collected data should represent the target population as stated in the claim.

Reference standard should be high quality and be representative of clinical utility.

Post-deployment evaluation

Monitor devices and follow reporting guidelines.

Consider designing phantom studies as sanity checks to assess routine performance.

Periodically monitor data drift.

For off-label evaluation, follow recommendations as in clinical/technical evaluation depending on objective.

Table 2: RELAINCE guidelines

In summary, AI-based technologies present an exciting toolset for advancing nuclear medicine. We envision that following these best practices for evaluation will assess suitability and provide confidence for clinical translation of these methods, and provide trust for clinical application, ultimately leading to improvements in quality of healthcare.

E. Disclosures

Sven Zuehlsdorff is a full-time employee of Siemens Medical Solutions USA, Inc. Nancy Obuchowski is a contracted statistician for QIBA. Tyler Bradshaw receives research support from GE Healthcare. Ronald Boellaard is (unpaid) scientific advisor for the EARL PET/CT accreditation program. Piotr Slomka has research grant from Siemens Medical Solutions, is a consultant for IBA, and receives royalties from Cedars-Sinai for nuclear cardiology software.

B. 5 .

 5 Figure of merit (FoM) to evaluate task performance: FoMs quantitatively describe a method's performance on the clinical task, enabling comparison of different methods, comparison to standard of care, and helping define quantitative metrics of success. FoMs should be accompanied by confidence intervals

Fig. 1 :•

 1 Fig. 1: The components of a claim

Fig. 2 :

 2 Fig. 2: increasing levels of rigor of evaluation, and how they in turn provide increased confidence in the generalizability

Fig. 4 :

 4 Fig. 4: Elements of study design for each class of evaluation

Fig. 3 :

 3 Fig. 3: Framework for evaluation of AI-based algorithms. The left of the pyramid provides a brief description of the phase, and the right provides an example of evaluating an AI-based segmentation algorithm on the task of evaluating metabolic tumor volume (MTV) using this framework.

Fig. 5 :

 5 Fig. 5: Flowchart to determine the clinical evaluation strategy

Fig. 6 :

 6 Fig. 6: An eye chart showing the different objectives of post-deployment monitoring, grouped as a function of the scope and goal of the study

Figure

  Figure of merit should reflect performance on clinical decision making.

Table 1 :

 1 Comparison of different data types and associated tradeoffs and evaluation criteria to evaluate technical efficacy

	Simulation	Physical	Clinical
	studies	phantoms	studies

F. Acknowledgements

The taskforce members thank Kyle J. Myers, PhD for helpful discussions and Bonnie Clarke for all her help throughout this project.