MD, DM, FACC, FASE Partho P Sengupta

MSc Sirish Shrestha

PhD Béatrice Berthon

MD Emmanuel Messas

MD Erwan Donal

MD, PhD Geoffrey Tison

MD James K Min

Jens-Uwe Jan D'hooge

Joel Voigt

Dudley

MD Jan D'hooge

MD Jens-Uwe Voigt

PhD Joel Dudley

MD, PhD Johan Verjans

PhD Khader Shameer

PhD Kipp Johnson

PhD Lasse Løvstakken

PhD Mahdi Tabassian

PhD Marco Piccirilli

PhD Mathieu Pernot

MS, PhD Naveena Yanamala

PhD Nicolas Duchateau

MD, PhD Nobuyuki Kagiyama

PhD Olivier Bernard

PhD Piotr Slomka

MD, PhD Rahul Deo

MD Rima Arnaout

Proposed Requirements for Cardiovascular Imaging Related Machine Learning Evaluation (PRIME) Checklist Reviewed by the American College of Cardiology Health Care Innovation Council

Keywords: Machine Learning, Artificial Intelligence, Reporting Items, Checklist, Cardiovascular imaging AI, Artificial intelligence ML, Machine learning

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

In 2016, during a two day Think Tank meeting, The American College of Cardiology's Executive Committee and Cardiovascular Imaging Section Leadership Council initiated a discussion regarding the future of cardiovascular imaging among thought leaders in the field (1).

One of the goals was focused on machine learning (ML) tools and methods that embrace datadriven approaches for scientific inquiry. The 2016 document stressed the creation and adoption of standards, the development of registries, and the use of new techniques in bioinformatics.

Furthermore, the imaging community's unfamiliarity with the approach was cited as a potential barrier to widespread adoption. In the recent years, the field of cardiac imaging has seen a remarkable burst of innovation with the use of ML, demonstrating powerful algorithms that start impacting the ways clinical and translational research is designed and executed (2)[START_REF] Dey | Artificial Intelligence in Cardiovascular Imaging: JACC State-of-the-Art Review[END_REF][START_REF] Winther | ν-net: Deep Learning for Generalized Biventricular Mass and Function Parameters Using Multicenter Cardiac MRI Data[END_REF][START_REF] Tan | Fully automated segmentation of the left ventricle in cine cardiac MRI using neural network regression[END_REF][START_REF] Betancur | Deep Learning for Prediction of Obstructive Disease From Fast Myocardial Perfusion SPECT: A Multicenter Study[END_REF][START_REF] Madani | Fast and accurate view classification of echocardiograms using deep learning[END_REF][START_REF] Krittanawong | Artificial Intelligence in Precision Cardiovascular Medicine[END_REF][START_REF] Zheng | A machine learning-based framework to identify type 2 diabetes through electronic health records[END_REF][START_REF] Dawes | Machine Learning of Three-dimensional Right Ventricular Motion Enables Outcome Prediction in Pulmonary Hypertension: A Cardiac MR Imaging Study[END_REF][START_REF] Lang | Recommendations for cardiac chamber quantification by echocardiography in adults: an update from the American Society of Echocardiography and the European Association of Cardiovascular Imaging[END_REF][START_REF] Zhang | Fully Automated Echocardiogram Interpretation in Clinical Practice[END_REF][START_REF] Ouyang | Video-based AI for beat-to-beat assessment of cardiac function[END_REF][START_REF] Fahmy | Automated Cardiac MR Scar Quantification in Hypertrophic Cardiomyopathy Using Deep Convolutional Neural Networks[END_REF][START_REF] Bai | Automated cardiovascular magnetic resonance image analysis with fully convolutional networks[END_REF].

ML is a subfield of artificial intelligence (AI) where an algorithm automatically discovers patterns of data in the datasets without using explicit instructions. Several recent state-of-the-art review articles have focused on providing introductory concepts regarding ML algorithm applications for general cardiologists [START_REF] Dey | Artificial Intelligence in Cardiovascular Imaging: JACC State-of-the-Art Review[END_REF][START_REF] Krittanawong | Artificial Intelligence in Precision Cardiovascular Medicine[END_REF][START_REF] Johnson | Artificial Intelligence in Cardiology[END_REF][START_REF] Al'aref | Clinical applications of machine learning in cardiovascular disease and its relevance to cardiac imaging[END_REF]. While ML is creating headlines in medical journals, congress, and on the web, considerable uncertainty and debate have arisen around topics such as problems with real-world data sources, the inconsistent availability of labeled data and outcome information, bias injection, inaccurate measurements, reproducibility, lack of external validation, and insufficient reporting, which contribute to hindering the reliable assessment of prediction model studies and reliable interpretations of the results by clinicians. This Proposed Recommendations for Cardiovascular Imaging Related Machine Learning Evaluation (PRIME) Checklist aims to provide general framework as reference in guiding scientific work for investigators, data scientists, authors, editors, and reviewers involved in machine learning research in cardiovascular imaging. The goal of the PRIME Checklist document is to standardize the application of artificial intelligence (AI) and ML, including data preparation, model selection, and performance assessment. The document provides a set of strategic steps towards developing a pragmatic checklist (Table 1), which may allow consistent reporting of machine learning models in cardiovascular imaging studies. To further determine its ease of use and applicability, we illustrate the application of checklist developed in PRIME Checklist for two recent articles that developed ML models in cardiovascular imaging (Supplementary Table S1).

Designing the Study Plan

Defining the goal of the analysis is a key first step that informs many downstream decisions as to whether to use machine learning at all including the presence or absence of labeled data for supervised or unsupervised learning. These informed decisions can alter the approach to model training, model selection, development, and tuning.

Determining the appropriateness of machine learning to the dataset

The first question researchers should address is whether the ML approach could be applicable and beneficial to their study. There is overlap between traditional statistics and ML, but they differ regarding the extent of the assumptions and the formulation of the methods to either predict or make inferences. If the dataset is relatively small (i.e., fewer than hundreds of sample per class for "average" modeling problems), then overfitting becomes a much bigger concern resulting in models that fail to generalize well for unseen instances [START_REF] Ghojogh | The Theory Behind Overfitting, Cross Validation, Regularization, Bagging, and Boosting: Tutorial[END_REF]. Similarly, if variables that are important to modeling the data are missing or if the model is too simple, the resulting ML model may underfit the data, thus producing less than optimal results [START_REF] Ghojogh | The Theory Behind Overfitting, Cross Validation, Regularization, Bagging, and Boosting: Tutorial[END_REF]. Statistical analyses rely on simpler models that are not necessarily optimized to the specific data under observation, and are therefore less prone to overfitting, at the price of lower performance if the problem is complex [START_REF] Dhurandhar | Improving Simple Models with Confidence Profiles[END_REF]. On the contrary, datasets with large numbers of features, on the order of thousands or those having many irrelevant or redundant features, with regard to a given task may just as easily lend to overfitting. ML is especially useful where the data are unstructured, feature selection or exploratory analyses are preferred to identify meaningful insights. As such, the learning algorithm may find patterns in the data to generate a homogenous fraction and identify relationships in a data-driven manner beyond the a priori knowledge or existing hypotheses [START_REF] Dey | Artificial Intelligence in Cardiovascular Imaging: JACC State-of-the-Art Review[END_REF]. While the use of advanced machine learning algorithms may be better suited for handling big or heterogeneous datasets, it comes at the cost of the interpretability, complexity, and the ability to draw a causal inference. Caution should be taken against causally interpreting results derived from models designed primarily for prediction. For tasks where the goal is to establish causality, the techniques that are commonly used in "traditional" biostatistics, including statistical analyses methods such as propensity score matching, or bayesian inference, maybe better suited; however, newer methods involving ML algorithms are being developed for causal inferences [START_REF] Blakely | Reflection on modern methods: when worlds collide-prediction, machine learning and causal inference[END_REF][START_REF] Leng | Reconstructing directional causal networks with random forest: Causality meeting machine learning[END_REF].

Understanding and Describing the Data

Irrespective of whether ML tools or statistical analysis methods are used, it is crucial to understand and describe the data available for analysis to draw appropriate conclusions, whether it is tabular, images, time-series data, or a combination. Important considerations about the data include the availability of data that is representative of the target population, the method used to obtain data, and the resultant biases that may influence the conclusions that can be drawn from the data. Describing the data can also help understand the relevance to the target population. The method of data collection, including the sampling method, is also important, as bias may be introduced from systematic error, coverage error, or selection. Various guidelines and associated checklists for medical research have been established to aid in the reporting of relevant details about the data, depending on the study design [START_REF] Vandenbroucke | Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration[END_REF]. Clearly describing the data preprocessing or data cleaning methods used is essential to enable reproducibility.

It should be acknowledged that all ML or statistical algorithms are guided by basic data assumptions; an independent and identical distribution is an important assumption where the random variables are mutually independent and have the same statistical distribution and properties. Methods to check for the model assumptions, such as learning curve [START_REF] Cohen | Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective[END_REF], diagnosing bias and variance [START_REF] Mehta | A high-bias, low-variance introduction to Machine Learning for physicists[END_REF] or error analyses, may be required.

Defining the process

When building ML models, it is crucial to specify the inputs (e.g., pixels in images, a set of parameter values, and patient information), and desired outputs (e.g., object categories and the presence or absence of disease, an integer representing each category, the probability for each category, the prediction of a continuous outcome measurement, transformed pixel data) that are required. While defining outputs is essential for supervised learning approaches, unsupervised learning approaches may also benefit from defining the output that is desired for the task to select an appropriate model. Some tasks, once well-defined, can only be achieved using certain types of algorithms. For example, image recognition tasks from raw pixel/image data may require the extraction of the optimal features from the data, which is intrinsically performed by deep neural networks and goes beyond the use of hand-crafted features as input. At the conceptual level, deep learning works by breaking a complex task (e.g., identifying a tumor or other abnormalities in organs/tissue) into simple fewer abstract tasks. For example, if the task is to identify a square from other geometric shapes in an image, this task can be divided further into smaller nested sub-steps i.e., by first checking if there are four lines associated with a shape or not. Alternately, one could check if lines are inter-connected and perpendicular to each other and whether they are closed or not and so on in a step-by-step hierarchical fashion. After the consecutive hierarchical identification of complex task, deep-learning approaches automatically find out the features that are important for solving the problem. It is generally preferable to start by defining the overall broader analysis, necessary to accomplish the task of interest, and dividing it into several sub-tasks. Defining the problem or task as precisely as possible can also help guide the data annotation strategy and model selection. Once the data analysis objective has been identified and the inputs/outputs have been defined for each task, it is easier to determine the appropriate models for the analysis pipeline (Figure 1).

Strategic steps for developing the checklist:

•

Identify and assess if machine learning could be appropriate.

• Define the objectives of machine learning to achieve the overall goal.

• Understand and describe the data.

• Identify input and target variables.

• Describe the baseline data and understand biases that may exist

Data Standardization, Feature Engineering and Learning

Data preparation, standardization and feature extraction is key to the success of model development. It ensures that the data format is appropriate for machine learning, the utilized variables carry relevant information for solving the problem at hand and the learning system is not biased towards a subset of the variables or categories in the database.

Data format

To analyze the data of N patients (also called 'observations'), each with M different measurements (also called 'variables' or 'dimensions'), e.g., ejection fraction, body mass index (BMI), and image pixels/voxels, by using an ML algorithm, a data matrix X should first be constructed such that the rows of this data matrix correspond to the observations and the columns correspond to the variables (Figure 2). Depending on the database and the problem at hand, X can be either a 'wide' (Figure 2a) or a 'tall' (Figure 2b) data matrix. In the former case, the number of observations is much smaller than the number of variables (N << M; Figure 2c), while in the latter, there is a large group of observations, but each observation has only a few variables (N >> M; Figure 2d).

Generating a data matrix from cardiac images can be performed either on entire images or on selected regions of the images, depending on the learning purpose. When the goal is to use a learning algorithm for modeling the global characteristics of the images, hand-crafted features (e.g., radiomics) extracted based on all the pixels of selected regions of a given image are considered to be the variables of one observation (Figure 2c), which typically leads to a wide data matrix. On the contrary, to model regional image characteristics, however, a region of interest (ROI) or patch consisting of a small group of pixels, thus more easily yielding a tall data matrix (Figure 2d). Further, a series of techniques (e.g., convolution, max pooling or patch-based methods) can further be applied in reducing the size of the image-data matrix to highly informative elements, which often results in a tall/thin matrix, either for classification or for pattern recognition task to identify key regions of interest [START_REF] Madani | Fast and accurate view classification of echocardiograms using deep learning[END_REF][START_REF] Anwar | Medical Image Analysis using Convolutional Neural Networks: A Review[END_REF]. For example, a data matrix generated by using, say, a single frame from longitudinal, 4 chamber or short-axis view of the heart that is 512x512 pixels in size for a total of N patients would result in creating a N x (512 2 ) size matrix for input into deep learning algorithms.

Data preparation

To analyze cardiac images in a ML framework, some preprocessing stages are usually carried out. The irrelevant areas of the images can be removed in a 'cropping' stage to focus on learning from useful regions and to prevent learning from extraneous regions (which can also contribute to leakage, as discussed below). If the images that are acquired from a group of subjects have different sizes, they typically should be 'resized' first [START_REF] Gonzalez | Digital Image Processing[END_REF] to a reference image size to construct a data matrix with the same number of variables. More advanced techniques from computational atlases are also necessary to align the anatomy-based data of each subject to a common geometry and temporal dynamics, as spatiotemporal misalignment of input images will increase variance in the input of neurons of a neural network thereby slowing its ability to learn relevant features [START_REF] Duchateau | Which reorientation framework for the atlas-based comparison of motion from cardiac image sequences?[END_REF][START_REF] Duchateau | A spatiotemporal statistical atlas of motion for the quantification of abnormal myocardial tissue velocities[END_REF]. Another common preprocessing stage is 'noise removal', which helps a learning algorithm to better model the essential characteristics of the images. When the acquired images have poor contrast, a 'histogram equalization' [START_REF] Gonzalez | Digital Image Processing[END_REF] technique can be used to adjust the intensities of the pixels and to increase the contrast of a low contrast region, thus facilitating its interpretation and analysis. The pixel intensities can also be manually adjusted during image acquisition. An example is the changing of the dynamic range of echocardiographic images by an operator. Techniques may also be applied to correct the differences in slice thickness, grey level distribution or even image resolution and different imaging protocols (contrast / non contrast; low dose / high dose). Deep learning techniques, which are robust with regards to image quality, may need to be employed. For CT and MRI images a particular window and level settings may be applied before the deep learning training and normalized to the full intensity ranges.

In cardiovascular imaging, data samples are often represented in 3D and 4D formats which currently may present challenges for the deep learning techniques due to the constraints posed by computing resources related to image sampling as well as large number of input variables. Nevertheless, efficient deep learning techniques have been developed for video analysis and these can be potentially adapted to the direct interrogation of 3D or 4D data often seen in echography, nuclear cardiology or CT [START_REF] Ullah | Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features[END_REF]. An alternative approach to deal with complex multidimensional data is to provide an intermediate simplified image representation. In cardiac imaging, often a bull's eye representation (aka 'polar map') is used for the projection of 3D or 4D image data into a simple 2D (or 2D+time) format. For example, a bull's-eye representation approach has successfully been implemented for deep learning of nuclear cardiology studies [START_REF] Betancur | Deep Learning for Prediction of Obstructive Disease From Fast Myocardial Perfusion SPECT: A Multicenter Study[END_REF][START_REF] Betancur | Deep Learning Analysis of Upright-Supine High-Efficiency SPECT Myocardial Perfusion Imaging for Prediction of Obstructive Coronary Artery Disease: A Multicenter Study[END_REF][START_REF] Spier | Classification of Polar Maps from Cardiac Perfusion Imaging with Graph-Convolutional Neural Networks[END_REF]. It allows data normalization from multiple scans and disparate sources such as motion/thickening, perfusion or flow. Such approaches could also be applied to other cardiovascular modalities.

Thus, data preparation step aids in normalizing and compressing images, with respect to their intensities, anatomical representation and viewpoint etc., across a given study prior to their entry into the ML framework. This process ensures that the algorithm spends more time and capacity learning the features that are important, rather than trying to rectify issues related to intensities or noise levels in the image data. Size of the images can also be reduced in order decrease memory requirements.

Feature engineering and learning

The next stage after data preparation is extracting a set of 'features' from the data matrix to be later used as the input to the learning methods. Feature extraction helps to overcome the following two main problems that can limit the efficient performance of a learning framework:

(i) Curse of dimensionality: When the data matrix is wide, the variable/feature space of the data can be referred to as 'high-dimensional'. This may lead to an algorithm which fails to learn essential characteristics of the data due to its complexity and poor generalization power when dealing with unseen data -a phenomenon that is referred to as the 'curse of dimensionality' [START_REF] Bishop | Pattern recognition and machine learning[END_REF][START_REF] Hastie | The Elements of Statistical LearningData Mining, Inference, and Prediction[END_REF]. To tackle these problems, the number of observations should increase significantly with the data dimensionality. For example, in pattern recognition, a typical rule of thumb is that there should be at least 5 training examples for each uncorrelated dimension in the representation [START_REF] Koutroumbas | Pattern recognition[END_REF].

Moreover, it has been previously suggested that the sample to feature ratio should be between 5-10 depending upon the complexity of the classifier [START_REF] Somorjai | Class prediction and discovery using gene microarray and proteomics mass spectroscopy data: curses, caveats, cautions[END_REF][START_REF] Dernoncourt | Analysis of feature selection stability on high dimension and small sample data[END_REF]. However, a significant increase in the number of observations is not always possible, especially for medical data/studies, given that it necessitates the collection of data from a large group of patients. This curse of dimensionality is one of the main reasons why having a large database is desirable to build an efficient learning algorithm.

(ii) Correlated variables: When a database includes correlated variables, a subset of the variables that are mutually uncorrelated may be sufficient to learn the data characteristics effectively [START_REF] Bishop | Pattern recognition and machine learning[END_REF]. Indeed, adding correlated variables to a database will only bring redundant information and does not help the learning algorithm to achieve a better understanding of the data. For the imaging data for example, neighboring pixels typically have similar values and are highly correlated [START_REF] Hyvärinen | Natural Image Statistics: A Probabilistic Approach to Early Computational Vision[END_REF].

Given the curse of dimensionality, learning process of algorithms cannot work effectively in data with too many features. Techniques to reduce the number of variables while retaining the most relevant information are critically important; a process called 'feature extraction' and 'dimensionality reduction'. It can be performed either manually using expert knowledge or by algorithms such as principal component analysis or multifactor dimensionality reduction [START_REF] Bishop | Pattern recognition and machine learning[END_REF][START_REF] Hastie | The Elements of Statistical LearningData Mining, Inference, and Prediction[END_REF][START_REF] Koutroumbas | Pattern recognition[END_REF]. The result of the feature extraction process should be a compact set of (ideally uncorrelated) features or variables in the form of a tall matrix that encodes the essential characteristics of the data.

The available approaches for extracting features from the image data can be divided into the following three broad categories (Figure 3): (i) handcrafted methods (e.g., local binary patterns (LBP) [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF] and scale invariant feature transform (SIFT) (39)), (ii) classic ML methods for dimensionality reduction (e.g., PCA [START_REF] Jolliffe | Principal Component Analysis[END_REF], independent component analysis (ICA) [START_REF] Hyvärinen | Independent component analysis: algorithms and applications[END_REF], or ISOMAP [START_REF] Tenenbaum | A global geometric framework for nonlinear dimensionality reduction[END_REF]) and (iii) deep learning methods [START_REF] Lecun | Deep learning[END_REF]. The methods in the first two categories are manually designed to extract specific types of features from the data, while in the last one, the features are learned from the database itself. Moreover, in the case of deep-learning, techniques such as max-pooling provide effective ways of down-sampling the image size in each layer, eliminating the need for feature extraction and improving overall training performance [START_REF] Anwar | Medical Image Analysis using Convolutional Neural Networks: A Review[END_REF].

Nevertheless, the classical feature learning algorithms have some limitations in the data modeling approaches like linearity, sparsity, or lack of hierarchical representation. The deep learning techniques, on the other hand, can learn complex features from the data at multiple levels and do not have limitations of the classical algorithms. However, they need a large-scale database to achieve efficient learning of the data characteristics. To train a deep learning algorithm with a smaller database, the following two main strategies can be used: (i) data augmentation (e.g., by using different types of data/image transformations) ( 44) and (ii) transfer learning, which works by fine-tuning a deep network that has been pretrained with a different large database (e.g., natural images) [START_REF] Shin | Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning[END_REF][START_REF] Bengio | Deep Learning of Representations for Unsupervised and Transfer Learning[END_REF]. However, the field of transfer learning for medical imaging data is currently at its early stages, and more critical insights into its actual relevance will be known in the coming years.

Variable normalization

For a database that is composed of several variables of different nature (e.g., anthropometric or imaging-derived measurements), the values of the variables lie in different ranges. Direct usage of these variables may bias the learning system towards the characteristics of the variables with larger values despite the usefulness of the variables with smaller values in solving a given problem. To deal with such challenges, a 'variable normalization' approach can be used to transform the variables such that they all lie in the same range prior to entering the learning phase [START_REF] Hastie | The Elements of Statistical LearningData Mining, Inference, and Prediction[END_REF][START_REF] Koutroumbas | Pattern recognition[END_REF]. Variable normalization is especially helpful for a deep learning algorithm, as it helps achieve faster convergence of a deep neural network [START_REF] Ioffe | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[END_REF].

Missing variable estimation

Machine learning algorithms often need complete datasets. If data are missing, the options are to exclude those subjects, encode them as missing or to impute missing values [START_REF] Bengio | Deep Learning of Representations for Unsupervised and Transfer Learning[END_REF][START_REF] Troyanskaya | Missing value estimation methods for DNA microarrays[END_REF][START_REF] Tabassian | Handling missing strain (rate) curves using K-nearest neighbor imputation[END_REF]. In cardiovascular imaging, 2D images are normally collected from multiple views, e.g., for volumetric measurements, and 3D images are composed of multiple 2D slices. These images can also be acquired throughout the cardiac cycle. When some of the 2D views are not accessible or when a group of 2D images at some points during the cardiac cycle or in a 3D volume are artefactual/missing, an imputation technique can estimate these images or the parameters extracted from them [START_REF] Tabassian | Handling missing strain (rate) curves using K-nearest neighbor imputation[END_REF]. Thanks to development of the new deep learning algorithms, such as generative adversarial networks (GAN) [START_REF] Goodfellow | Generative Adversarial Nets[END_REF], missing images can often be estimated [START_REF] Shang | VIGAN: Missing View Imputation with Generative Adversarial Networks[END_REF] based on the available data, although the physiological relevance of their content is not fully guaranteed. However, it should be acknowledged that most of the imputation methods assume that the missing observations occur at random, are missing completely at random, or are missing not at random [START_REF] Rhodes | Improving Disparity Research by Imputing Missing Data in Health Care Records[END_REF].

Researchers should consider whether the missing observations carry any specific biases (e.g., selection bias or immortal time bias). While there is no clear guidance or cutoff on what proportion of missing data warrants the use of data imputation techniques [START_REF] Madley-Dowd | The proportion of missing data should not be used to guide decisions on multiple imputation[END_REF] and given that this answer depends on the complexity of the addressed problem, missing value imputation is best utilized whenever possible as it provides evidence about the robustness of the learned models regardless of its impact on model performance [START_REF] Liu | An Overview and Evaluation of Recent Machine Learning Imputation Methods Using Cardiac Imaging Data[END_REF].

Feature selection

An important phase in designing a classic ML system is to determine the optimal number of preserved features. This determination can be performed by using a 'feature selection' technique where a larger than required set of features is first extracted and then a subset with discriminative information is selected [START_REF] Koutroumbas | Pattern recognition[END_REF][START_REF] Inza | Filter versus wrapper gene selection approaches in DNA microarray domains[END_REF]. When a deep learning algorithm is used, the optimal features are automatically learned during the end-to-end training of the algorithm, and utilizing an independent feature selection method is often not required [START_REF] Lecun | Deep learning[END_REF][START_REF] Arnaout | Deep-learning models improve on community-level diagnosis for common congenital heart disease lesions[END_REF].

Outliers

An observation is considered as an outlier if its values deviate substantially (or significantly, if looked through a statistical test) from the average values of a database, which may be attributed to measurement error, variability in the measurement, or abnormalities due to disease [START_REF] Hastie | The Elements of Statistical LearningData Mining, Inference, and Prediction[END_REF][START_REF] Koutroumbas | Pattern recognition[END_REF].

Even though outliers can negatively influence and mislead the training process resulting in less accurate models, they may carry relevant information related to the given task. Thus, outliers should be carefully examined to see if this comes from improper measurements that should be repeated, or not. With the existence of outliers, learning algorithms and/or a performance metrics that are robust to outliers can be useful alternatives. Methods robust to outliers including but not limited to decision trees and k-nearest neighbor (KNN) should be employed as much as possible [START_REF] Hastie | The Elements of Statistical LearningData Mining, Inference, and Prediction[END_REF]. If no other solution exits, the removal of outliers, using an outlier detection approach (56), may be considered and the selection criterion along with the proportion of samples removed should be reported.

Class imbalance

A significant imbalance in data classes (e.g., healthy vs. diseased) is quite common in medical datasets because, on the one hand, the majority of subjects in a database are usually healthy and, on the other hand, because collecting patient data for some rare diseases is difficult and is not always possible. As a result, the performance of the learning algorithm might be skewed, as it only learns the characteristics of the larger sized categories. This problem is referred to as 'class imbalance' and can be dealt with in the following three established ways: (i) rebalancing the categories using 'under-sampling' or 'over-sampling' (i.e., making the different classes similarly sized by omitting samples from the larger class or by up-sampling the data in the smaller class),

(ii) giving more importance (i.e., weight) to the samples of smaller categories during the learning process [START_REF] Koutroumbas | Pattern recognition[END_REF], and iii) utilizing synthetic data generation methods, such as the synthetic minority over-sampling technique (SMOTE) [START_REF] Chawla | SMOTE: Synthetic Minority Oversampling Technique[END_REF]. While the random over-sampling generates new data by duplicating some of the original samples of the minority class or category, SMOTE interpolates values using a k-nearest neighbor technique to synthesize new data instances [START_REF] Last | Oversampling for Imbalanced Learning Based on K-Means and SMOTE[END_REF] . Recent advances in deep generative techniques, such as GAN or variational autoencoders or the use of loss functions that are robust to data imbalance [START_REF] Rezende | Variational Inference with Normalizing Flows[END_REF], have made it possible to tackle complicated imbalanced data based on the learning strategies.

Data shift

Data shift is a common problem that afflicts the ML models in cardiovascular imaging in which the distribution of the database that is used for testing the performance of the learning models or systems may differ from the distribution of the training data. This may occur when the data acquisition conditions or the systems that are used for collecting the test data change from when the training dataset was acquired, and could induce i) a covariate shift -a shift in the distribution in the covariates, ii) a prior probability shift -a difference in the distribution of the target variable, or iii) a domain shift -a change in measurement systems or methods. It is imperative to assess and treat the shifts that may occur in the dataset prior to evaluating a model [START_REF] Subbaswamy | Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport[END_REF].

Data leakage

Data leakage is a major problem in ML, in which data outside of the training set seeps into the model while building the model. This event could lead to error-prone or invalid ML models.

Data leakage could occur if the same patient's data is used in the training and testing sets and is generally a problem in complex datasets, such as time series, audio and images, or graph problems.

Strategic steps for developing the checklist:

• The data format for training a machine learning algorithm should be large and the ratio of the observations/measurements (i.e., N/M) should be at least five.

• When the data matrix is wide, a feature extraction/learning algorithm or dimensionality reduction technique should be used.

• Redundant features should be removed, and variables should be normalized.

• Outliers should be addressed/removed

• Missing features should be imputed using relevant methods.

• Dataset shift, leakage and class imbalance are common pitfalls and should be evaluated.

Selection of Machine Learning Models

Model selection is the process of identifying the model that yields the best resolution and generalizability for the project and can be defined at multiple levels, i.e., learning methods, algorithms, and tuning hyperparameters. Learning methods include supervised, unsupervised, and reinforcement learning. Importantly, supervised learning is a method that learns from labeled data, i.e., data with outcome information to develop a prediction model, while unsupervised learning aims to find patterns and association rules in data that do not have labels (Figure 4).

Common algorithms, such as regression or instance-based learning, often handle highdimensional data well and tend to perform better or equivalent to complex algorithms on small datasets while retaining the interpretability of the model. To achieve better performance, simple algorithms, or weak learners, may be combined in various ways using ensemble methods, such as boosting, bagging, and stacking, which sacrifice the interpretability. More complex algorithms that are also difficult to interpret, including neural networks, can outperform simpler models given an adequate amount of data. A subset of neural networks, known as deep convolutional neural networks [START_REF] Madani | Fast and accurate view classification of echocardiograms using deep learning[END_REF][START_REF] Anwar | Medical Image Analysis using Convolutional Neural Networks: A Review[END_REF], are particularly useful for finding patterns in image data without the need for feature extraction [START_REF] Wang | Detecting Cardiovascular Disease from Mammograms With Deep Learning[END_REF][START_REF] Litjens | State-of-the-Art Deep Learning in Cardiovascular Image Analysis[END_REF][START_REF] Retson | Machine Learning and Deep Neural Networks in Thoracic and Cardiovascular Imaging[END_REF][START_REF] Avendi | A combined deep-learning and deformablemodel approach to fully automatic segmentation of the left ventricle in cardiac MRI[END_REF]. The implementation of an algorithm can vary significantly in terms of the size and complexity (e.g., the size and number of features in a random forest decision tree, the number and complexity of kernels applied in an SVM, and the number and type of nodes and layers in a neural network) of the algorithms.

Regardless of the choice of the algorithms, it is imperative to perform hyperparameter tuning and model regularization to produce the optimal performance [START_REF] Kostoglou | A Novel Framework for Estimating Time-Varying Multivariate Autoregressive Models and Application to Cardiovascular Responses to Acute Exercise[END_REF][START_REF] Al'aref | Determinants of In-Hospital Mortality After Percutaneous Coronary Intervention: A Machine Learning Approach[END_REF]. These processes may be more important than selecting the types of algorithms that could impact the interpretability, simplicity, and accuracy. When performing model selection (especially involving large-scale data and/or deep learning methods), hardware constraints (e.g., memory size, cache, parallelism etc.) are often a key limiting factor beyond model performance.

However, recent advances towards developing hardware accelerators (e.g., graphical processing units, tensor processing units) and growing convenience/abstraction by cloud computing could Finally, an essential factor in algorithm selection is the need for the interpretability of the model's decisions, i.e., an understanding of which input features caused the model to make the decision it made. Interpretability may be extremely important for certain learning tasks and less important for others. Regression, decision trees, and instance-based learning methods are generally highly interpretable, while methods to interpret the function of deep neural networks are still evolving; saliency mapping, class activation, and attention mapping are some example methods for neural network interpretation and visualization [START_REF] Gilpin | Explaining Explanations: An Overview of Interpretability of Machine Learning[END_REF][START_REF] Fong | Interpretable Explanations of Black Boxes by Meaningful Perturbation[END_REF]. New mechanisms for understanding the workings of machine learning models (69-72), and approaches for probabilistic deep learning [START_REF] Lake | Human-level concept learning through probabilistic program induction[END_REF][START_REF] Gal | Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning[END_REF] further provide an opportunity to develop models to balance between both inference and prediction.

Strategic steps for developing the checklist:

• For the initial model development, always select the simplest algorithm that is appropriate for the available data.

• The size of the dataset and the complexity of the employed algorithm should be considered to achieve a good compromise between 'bias' and 'variance' in the estimations.

• Complex algorithms must be benchmarked to the performance of the initial simple model across several metrics.

• Tune the hyperparameters to optimize the models and to increase performance.

Model Assessment

The next step after selecting a learning model is to evaluate the generalizability by applying it to new data, i.e., assessment of its performance on unseen data. Ideally, model assessment should be performed by randomly dividing the dataset into a 'training set' for learning the data characteristics, a 'validation set' for tuning the hyperparameters of the learning model, and a 'test set' for estimating its generalization error, where all the three sets have the same probability distribution (i.e., the statistical characteristics of the data in these three sets are identical).

However, in many domains, including cardiovascular imaging, having access to a large dataset is often difficult, thus preventing model assessment using three independent data subsets. As mentioned in the previous section, the ratio of the training samples to the number of measured variables should be at least five to ten [START_REF] Koutroumbas | Pattern recognition[END_REF][START_REF] Somorjai | Class prediction and discovery using gene microarray and proteomics mass spectroscopy data: curses, caveats, cautions[END_REF], depending on the dataset and complexity of the classifier [START_REF] Raudys | Statistical and neural classifiers: an integrated approach to design[END_REF], to learn the data characteristics properly. If this criterion is not met, the data are called scarce. In this situation, the data may be divided into two subsets for training and final validation of the learning algorithm. However, the results may depend on the random selection of the samples. Therefore, the training set can then be further partitioned into two subsets, but this process is repeated several times by selecting different training and testing subjects to obtain a good estimate of the generalization performance of the learning algorithm [START_REF] Hastie | The Elements of Statistical LearningData Mining, Inference, and Prediction[END_REF]. This method of model assessment can be performed via 'cross-validation' or 'bootstrapping', as further explained below. These techniques ensure that (i) the learning model is trained properly given that the majority of the data samples can be used in the training process, (ii) the learning model is not biased towards the characteristics of a subset of the data and (iii) the optimal values of the hyperparameters of the learning model (e.g., the number of layers in a neural network and the neurons in each layer) can be determined [START_REF] Hastie | The Elements of Statistical LearningData Mining, Inference, and Prediction[END_REF].

Cross-validation

This technique works by dividing the data into multiple nonoverlapping training and testing subsets (also called folds) and using the majority of the folds for training a learning model and the remaining folds for evaluating its performance [START_REF] Bishop | Pattern recognition and machine learning[END_REF][START_REF] Hastie | The Elements of Statistical LearningData Mining, Inference, and Prediction[END_REF][START_REF] Koutroumbas | Pattern recognition[END_REF]. The cross-validation process can be implemented in one of the following ways.

i. k-fold cross-validation:

The data is randomly partitioned into k folds of roughly equal sizes, and in each round of the cross-validation process, one of the folds is used for testing the learning algorithm and the rest of the folds are used for its training (Figure 5).

This process is repeated k times such that all folds are used in the testing phase and the average performance on the testing folds is computed as an unbiased estimate of the overall performance of the algorithm [START_REF] Bishop | Pattern recognition and machine learning[END_REF][START_REF] Hastie | The Elements of Statistical LearningData Mining, Inference, and Prediction[END_REF].

ii.

Leave-one-out cross-validation:

In this technique, the number of the folds is equal to the number of the observations in the database, and in each round, only one observation is used for testing the learning algorithm.

iii.

Monte-Carlo cross-validation:

In this method of cross-validation, there is no limit to the number of the folds, and a database can be randomly partitioned into multiple training and testing sets. The training samples are randomly selected 'without replacement', and the remaining samples are used for the testing group (Figure 6I) [START_REF] Kuhn | Applied Predictive Modeling[END_REF].

Bootstrapping

This method works by randomly sampling observations from a database 'with replacement' to form a training set whose size is equal to the original database. As a result, some of the observations can appear several times in the training set, while some may never be selected. The latter observations are called 'out-of-bag' and are used to test the learning algorithm. This process is repeated multiple times to estimate the learning method's generalization performance (Figure 6 II) [START_REF] Hastie | The Elements of Statistical LearningData Mining, Inference, and Prediction[END_REF][START_REF] Kuhn | Applied Predictive Modeling[END_REF]. While bootstrapping tends to drastically reduce the variance, it often tends to provide more biased results, more importantly when dealing with small sample sizes.

Model Evaluation

The reporting of accuracy in ML is closely linked to the reporting of summary statistics, and the same background and assumptions apply. While a review of statistical theory is out of scope for the PRIME Checklist, we encourage the readers to obtain a clear understanding of the statistics for classification and prediction [START_REF] Wheelan | Naked statistics. stripping the dread from the data[END_REF][START_REF] Mlodinow | The Drunkard's Walk: How Randomness Rules Our Lives: Vintage[END_REF][START_REF] Wasserman | All of statistics : a concise course in statistical inference[END_REF][START_REF] Urdan | Statistics in Plain English[END_REF][START_REF] Cohen | Empirical methods for artificial intelligence[END_REF][START_REF] Box | Statistics for Experimenters: Design, Innovation, and Discovery[END_REF][START_REF] Sabo | Statistical Research Methods: A Guide for Non-Statisticians[END_REF]. Most of the following section applies to supervised learning algorithms, for which labels are used in the definition of the performance measures.

Unsupervised learning is more difficult to evaluate but should also evaluate the relevance of the output data representation and the stability of the results against the data and model parameters.

Strategic steps for developing the checklist:

• Model assessment should be performed by randomly dividing the dataset into training, validation and testing data when applicable.

• When data is inadequate or scarce, model assessments using cross-validation and/or bootstrapping techniques should be performed to obtain a good estimate of the generalization performance of the learning algorithm.

• Typical numbers for k in a k-fold cross-validation should be 5 and 10.

• Consider using leave-one-out cross validation as an appropriate choice when the data is small.

For classification tasks, the accuracy is the percentage of data that is correctly classified by the model, which could be influenced by the quality of the expert annotations. The balance of classes in the training data is also a known source of bias. As such, a prerequisite for reporting accuracy measures is to provide a clear description of the data material used for training and validation.

We further suggest balancing the class data according to prevalence when possible, or that balanced accuracy measures are reported [START_REF] Wainer | An empirical evaluation of imbalanced data strategies from a practitioner's point of view[END_REF].

The model parameters (e.g., initialization scheme, number of feature maps, and loss function), regularization strategies (e.g., smoothness and dropouts), and hyperparameters (e.g., optimizer, learning rate, and stopping criterion) also play a part in the model performance. A second prerequisite is, therefore, to provide a clear description of how the ML model was generated. We further suggest that the certainty of the accuracy measure is reported where applicable, for instance, by estimating the ensemble average and variance from several models generated with random initialization. Additionally, cross-validation analysis should be added to underline the robustness of the model, especially for limited training and test data (see the previous section). Furthermore, to assess the generalizability of the algorithm, it is necessary to report the accuracy of the model by testing the data from different geographical locations with similar statistical properties and distributions [START_REF] Abazeed | Walking the tightrope of artificial intelligence guidelines in clinical practice[END_REF].

A report of the accuracy for ML algorithms in cardiovascular imaging will depend on the method and problem. For instance, the classification of disease from image features differs from the classification of image pixels in semantic segmentation, both in terms of the measures reported and of the risk in use.

For multiclass/label classification, we suggest using a statistical language close to the clinical standard. For instance, the report sensitivity, specificity, and odds ratio should be used instead of the precision, recall, and F1 score. This will also ensure that true negative outcomes are considered [START_REF] Tharwat | Classification assessment methods[END_REF]. Nonetheless, for classification tasks, the confusion matrix should normally be included but could be supplementary material. For image segmentation problems, we suggest reporting several measures to summarize both the global and local deviations, such as the mean absolute error (MAE), the Dice score to summarize the average performance and the Hausdorff distance metric to capture local outliers.

When the output of the regression or segmentation algorithms are linked to clinical measurements (e.g., ejection fraction), we suggest Bland-Altman plots as for conventional evaluation of the image measurements, and we stress the importance of comparing the performance with several expert observers for both intra-and inter-expert variability.

For the classification of disease from image features, the cost of misclassification should be clearly conveyed, e.g., rare diseases may not be properly represented in the dataset. The balance of classes should reflect the prevalence of the disease of interest, and scoring rules based on estimated probability distributions should be used for the accuracy reporting when possible, instead of direct classification. The choice of the scoring rule used for the decision, e.g., mean squared error, Brier score, and log-loss, should be rationalized. The common classification scores (sensitivity, specificity, positive-and negative predictive value) should include a full ROC analysis to provide a more in-depth evaluation of the detection performance. It is also relevant to include benchmark results from alternative ML methods as well as more traditional techniques, such as logistic regression.

Best Practices for Model Replicability

The reproducibility of scientific results is essential to make progress in cardiac medicine. The ability to reproduce findings helps to ensure the validity and correctness, as well as enabling others to translate the results into clinical practice. However, there are several complementary definitions of reproducibility. We focus here primarily on technical replicability (87); i.e., the ability to independently confirm published results of a model by inspecting and executing data and code under identical conditions. Technical replicability is especially important in ML projects, which often involve custom software scripts, the use of external libraries, and intensive or expensive computation. Actions taken at any point in an ML workflow, from quality control and data preparation into suitable data structures to algorithm development to the visualization of

Strategic steps for developing the checklist:

• Use a statistical language close to the clinical standard and introduce new measures only when needed.

• Balance the classes according to prevalence where available or report balanced accuracy measures.

• Estimate the accuracy certainty, e.g., from an ensemble of models, to strengthen the confidence in the values reported.

• Include Bland-Altman plots when machine learning is linked to clinical measurements.

• Include an inter-/ intra-observer variability measures as a reference where possible.

• The risk of misclassification should be conveyed, and appropriate scoring rules for decisions may be needed for the classification of a disease. results, are often based upon heuristic judgments, and there are potentially numerous justifiable analytic options. Ultimately, these selections may significantly alter the results and conclusions.

The first step for making ML projects reproducible could be the release of all the original code written for a project. There are several options for the publication of code. When possible, Although the availability of code is required for technical replicability, equally important is the availability of the data used in the project [START_REF] Tharwat | Classification assessment methods[END_REF]. Clinical data should be anonymized, or if anonymization is not possible (as in the case of some genetic data), then data should be made available to other researchers with appropriate IRB approval. Other options include the generation of synthetic datasets with the same statistical properties as the original dataset, a field of study called differential privacy. Manuscripts must state where both the raw and improving the technical replicability. Moreover, documentation generation tools (e.g., Sphinx, https://www.sphinx-doc.org/en/master/) or easy-to-launch demos (e.g. through Jupiter python notebooks) should be employed.

Reporting Limitations, Biases and Alternatives

"All models are wrong, but some are useful" is a well-known statistical aphorism attributed to George Box. Accurate reporting and acknowledgement of limitations are required for manuscripts incorporating ML (ML). Any statistical model or ML algorithm incorporates some assumptions regarding the data. All model assumptions should be affirmatively identified and checked with the dataset utilized in the manuscript, and the results should be reported in the manuscript or supplementary material. The algorithms used in computational research efforts span a large spectrum of complexity. Generally, more basic models and algorithms should first be investigated before additional complexity is incorporated into models or different algorithms are selected. Deep learning models should be benchmarked against simpler models whenever possible, especially when applied to tabular data. Statistical or ML models incorporating large numbers of variables (e.g., polygenic risk score models) should be benchmarked against standard clinical risk prediction models using more traditional clinical variables.

Concordant findings from multiple, independent datasets dramatically increase the scientific value of manuscripts, since it decreases the likelihood that the algorithms have been erroneously overfitted to the idiosyncratic features of a certain dataset. Deep learning models are especially notorious for harnessing spurious or confounding features of the dataset to perform well. For example, Zech and Badgeley et al. reported a case where a convolutional neural network trained on a health system's chest X-rays used the presence of a "PORTABLE" label on X-ray images to predict cardiomegaly with high accuracy [START_REF] Zech | Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study[END_REF]. Furthermore, in the case of supervised ML involving human-annotated variables or outcomes, it should be noted that ML algorithms will recapitulate the underlying biases of the humans who constructed the dataset.

Strategic steps for developing the checklist:

• Affirmatively identify and check relevant model assumptions and report the findings.

• Benchmark complex algorithms against simpler algorithms and justify the use of more complex models.

• Benchmark algorithms, incorporating high-dimensional data or novel data sources, against standard clinical risk prediction models In a short and wide data matrix, the number of observations is much smaller than the number of variables (N<<M). Considering different regions of interest (ROI) on the whole image of a given patient, the extraction of hand crafted features (e.g., radiomics features) may lead to a short and wide data matrix, as the number of features extracted for each ROI per image is typically larger than the number of samples or patients (c). To make a tall and thin data matrix from the image data, an image can be divided to many (overlapping) ROIs or patches, each with a small number of pixels (d). The extraction of pixel data as features from each patch per patient may be much smaller in size than the total number of patients. A) In Supervised learning method, after the hyperparameter tuning, data can be applied to two different tasks: classification or regression, depending on the type of the outcome. If the outcome is a category, then classification can be performed whereas if the outcome variable is a numeric value, regression may be applied for prediction. B) Unsupervised learning method, the data in which the data are either utilized for clustering, topical modeling, or representing the data distribution while reducing the dimensionality of the data according to the problem to be solved. Central Illustration: Steps for building a machine learning pipeline and the reporting items in a checklist. This illustration provides the principal requirements for building a checklist (at every step of the model building process) to enable the precise application of predictive modeling, consistent reporting of model specifications and results in the field of cardiovascular imaging. CV = cross validation; GPL = general public license; LOOCV = leave one out cross validation; ML = machine learning; S/W = software.

Introduction

Artificial Intelligence

Branch of computer science concerned with building intelligent systems that automatically behave based on the data and experience they learn. Typically, there are two types of AI: general and applied AI. General AI is a self-sufficient system that can possess a cognition capability comparable to, or even surpassing, that of humans. Applied AI is a functional system that is specialized for a purpose. Natural language processing and ML are forms of AI.

Machine Learning

A subfield of applied AI that concerns algorithms, statistical modeling, and data analysis and that learns from the data to detect patterns and make assessments with minimal human intervention. There are 3 main to apply ML: supervised, unsupervised, and re-enforcement learning. Semi-supervised learning is also considered a class of ML.

Designing the study plan

Model Interpretability

A degree to which an individual can comprehend the reason and decision made by the algorithm. Higher the interpretability of the ML model, easier it is for an individual to comprehend the reason to a decision reached by an algorithm.

Model Complexity

Refers to the number of features/covariates/variables and the transformations and linearity (or non-linearity) of the features that are included in the predictive model. It can also refer to the learning and computational complexity of a given learning algorithm.

Bayesian inference

Process to interpret probability and represent the confidence given an occurrence of an event. It is based on Bayes' statistical method that assigns probabilities to events or parameters based on the current data but updates as the probabilities as more data is obtained.

Systematic error

The errors that are produced from measurement errors. This may occur if the measurement unit is faulty, incorrectly used, or the data incorrectly interpreted or entered.

Sparsity

In numerical terms, sparsity is estimated as the number of zerovalued elements divided by the total number of elements in a data array or matrix. Input data will be considered sparse when most of the elements (> 50% of data) of a data array or matrix are zero i.e., when its sparsity is greater than 0.5.

Selection of Machine Learning Models

Supervised

A class of ML that learns from labeled data to predict or classify the outcome of interest. The most common supervised learning tasks are regression and classification.

Unsupervised

A class of ML that learns from unlabeled data. The most common unsupervised learning tasks are clustering and dimensionality reduction.

Semi-supervised Learning

A class of ML that learns from labeled and unlabeled data in the dataset. Typically, the proportion of unlabeled data is higher than the proportion of labeled data.

Overfitting

A case when the model fits the training data too closely, but the generalization of the model is unreliable.

Underfitting

A case when the model neither fits the training data nor generalizes to new data. An underfit model results in low generalization and unreliable predictions.

Clustering

An unsupervised ML task in which the fractions of data with some notion of similarity are grouped together while keeping the others separate. K-means and agglomerative hierarchical clustering are popular clustering algorithms.

Ensemble learning

An ML process to combine multiple models to obtain better predictive performance compared to the performance from a single model. Bagging, boosting, and stacking are three main methods of ensemble learning.

Bagging

Bagging also known as 'bootstrap aggregating' is an ensemble technique that is designed to improve the stability and the accuracy of ML algorithms. The objective of bagging is simple, generate several random training sets and "average" their predictions in order to obtain a model with a lower variance. Thus, bagging reduces variance and helps to avoid overfitting.

Boosting

In ML boosting is also an ensemble technique that attempts to create a strong learner from a number of weak learners. The main objective of boosting is to increase the complexity of models that suffer from high bias especially in the case of underfitting. In simple terms, boosting tries to reduce the error in predictions and hence reduces bias.

Saliency mapping

Saliency can be seen as a kind of image segmentation. In computer vision, a saliency map is an image that shows each pixel's unique quality. The goal of a saliency map is to simplify and/or change the representation of an image into something that is more meaningful and easier to analyze.

Model Assessment

Model

The mathematical representation of the process that is governed by the data. The data is partitioned into training or testing, and the training set is generally used to generate the model that can be tested for its generalizability in ML. For example, a model is presented as 𝑦 " = 𝛽 % 𝑥 % + 𝛽 ) 𝑥 ) … 𝛽 + 𝑥 + in linear regression.

Model Parameters

Model parameters are internal configuration variables such as coefficients in linear regression (e.g., β1, β2, etc. in the equation above) that are estimated from the data and describe the model.

Model Evaluation

Classification A supervised ML technique that attempts to classify data into binary scores or categories.

Regression

A supervised ML technique that attempts to predict a continuous value.

Section

  improve the overall process of model training/selection/optimization. Moreover, selecting the best model often involves the relative comparison of performance between different models.Therefore, the purposeful selection of loss function and the metric that represents it (e.g., absolute error, mean-squared error etc.), which in turn is heavily influenced by the model choice, dataset, and particular problem/task to solve, becomes fundamental in selecting an appropriate model.The size and complexity of algorithms should be chosen carefully to minimize the bias, the model error on the training dataset, and the variance, the model error on the validation dataset. Simpler models may underfit the data; they may generalize better (lower variance) at the cost of lower accuracy (higher bias). Further, overfitting (high variance and low bias) may come from a too complex model or insufficient representative training samples. Several considerations including size, complexity/dimensionality, number of features and nonlinear relationships among variables in the dataset guides the choice of the initial algorithm and its complexity, but the final algorithm design (including the choice of hyperparameters) is determined empirically or by specific optimization and cross-validation.

  we suggest uploading source code with software and packages' version information as supplementary material alongside the manuscript. Other options include permanent archival on a lab website, or per-project archival on open source and public source code repositories if permitted by investigator's institution. Manuscripts should explicitly state where and how the code may be downloaded and under what license. Although there are numerous open source licenses available, in most cases, either of two licenses will suffice: the Massachusetts Institute of Technology/Berkeley Software Distribution (MIT/BSD) licenses (https://opensource.org/licenses/MIT) (the MIT and BSD licenses are essentially equivalent) and the GNU General Public License (GNU GPLv3). The MIT/BSD licenses allow published code to be distributed, modified, and executed freely without liability or warranty; the GNU GPLv3 license allows the same with the additional restriction that all software-based upon the original code must also be freely available under the GNU GPLv3 license, meaning others cannot reuse the original code in a closed-source product.

Figure 1 :

 1 Figure 1: Machine learning pipelineSchematic diagram of a general ML pipeline. The data section consists of project planning, data collection, cleaning, and exploration. The modelling section describes the model building, in which hyperparameter tuning and the dimensionality reduction process, such as feature selection and engineering, model optimization and selection, and evaluation, are included. Finally, the reporting segment consists of the reporting mechanisms of the analysis, including reproducibility and maintenance, and a description of the limitations and alternatives.

Figure 2 :

 2 Figure 2: Schematic demonstration of short/wide (a) and tall/thin (b) data matrices and the way that they can be created from the image data.In a short and wide data matrix, the number of observations is much smaller than the number of variables (N<<M). Considering different regions of interest (ROI) on the whole image of a given patient, the extraction of hand crafted features (e.g., radiomics features) may lead to a short and wide data matrix, as the number of features extracted for each ROI per image is typically larger than the number of samples or patients (c). To make a tall and thin data matrix from the image data, an image can be divided to many (overlapping) ROIs or patches, each with a small number of pixels (d). The extraction of pixel data as features from each patch per patient may be much smaller in size than the total number of patients.

Figure 3 :

 3 Figure 3: The main approaches for feature engineering and learning.The hand-engineering approaches are manually designed to extract certain types of features from the data; for example, Local Binary Pattern (LBP) and Scale-Invariant Feature Transform (SIFT) derives the properties from the image such as object recognition or edge detection. The classic learning techniques use data samples to learn their characteristics for dimensionality reduction, but they have limitations in their data modeling techniques, such as linearity, sparsity or lack of hierarchical representation. Principal component analysis (PCA) applies orthogonal transformation to produce linear combination of uncorrelated variables that best explains the variability of the data whereas independent component analysis (ICA) transforms the dataset into independent components to reduce dimensionality. Deep learning methods, however, can learn complex features from the data at multiple levels in various hidden layers.

Figure 4 :

 4 Figure 4: Model selection processIllustration of the model selection process, which consists of identifying the two classes of ML. A) In Supervised learning method, after the hyperparameter tuning, data can be applied to two different tasks: classification or regression, depending on the type of the outcome. If the outcome is a category, then classification can be performed whereas if the outcome variable is a numeric value, regression may be applied for prediction. B) Unsupervised learning method, the data in which the data are either utilized for clustering, topical modeling, or representing the data distribution while reducing the dimensionality of the data according to the problem to be solved.

Figure 5 :

 5 Figure 5: Schematic illustration of the k-fold cross-validation process. Data are randomly partitioned into k distinct folds, and in each round,(k-1) folds are used for training the learning algorithm, and the kth fold is used for testing its performance. This process is repeated k times such that all folds are used in the testing phase.

Figure 6 :

 6 Figure 6: Schematic illustration of the monte-carlo and bootstrap resampling methods. I) Monte-Carlo cross-validation performed in k rounds.In each round, the training and testing samples are randomly selected without replacement from the original data. II) The bootstrapping process, which can be performed in B rounds. In each round, the training data is generated by randomly sampling from the original data with replacement. The samples that are not included in the training dataset (i.e., out-of-bag samples) form the testing dataset.

1the study plan 1 . 1 5

 115 Designing Describe the need for the application of machine learning to the Describe the overall steps of machine learning workflow pg 3, pg 4, Fig

pg 4 pg 5 3 10 3. 2 5 3. 3 6 4 . 1 - 6 4. 2 5 . 1 9 5. 2 5 . 3 class pg 5 pg 5 5. 4 6 X 6 , 7 61 6 . 2 6 . 3 -7 pg 5 7 7 . 2

 51025364162519253546676263572 Selection of Machine Learning Model3.1 Explicitly define the goal of the analysis e.g., regression, Identify the proper learning method used (e.g., supervised, reinforcement learning etc.) to address the problem pg 1, pg 5 pg Provide explicit details on the use of simpler, complex, Provide details on whether the model is interpretable pg 2Provide a clear description of data used for training, validation, and testing pg 7 pg 5Describe how the model parameters were optimized (e.g., optimization technique, number of model parameters etc.) Provide the metric(s) used to evaluate the performance of the Define the prevalence of disease and the choice of the scoring rule used X X Report any methods used to balance the numbers of subjects in each Discuss the risk associated to misclassification pg Best Practices for Model Replicability 6.1 Consider sharing code or scripts on public repository with appropriate copyright protection steps for further development and non-commercial use pg 5-7 pg 5Release data dictionary with appropriate explanation of the Document version of all software and external libraries pg 5Reporting limitations, biases and alternatives 7.1 Identify and report the relevant model assumptions and findings X X If well performing models were tested on a hold-out validation dataset, detail the data of that validation set with the same rigor as that of training dataset (see section 2 above) X X

Table 1 . Checklist for Standardized Reporting of Machine Learning Investigations

 1

	Section Checklist item

  Describe how the data were processed in order to make it clean, uniform, and consistent Describe whether class imbalance existed and which method was applied to deal with it

		2, Table 1	pg 4,7
	2 Data standardization, feature engineering, and learning
	2.1 Fig 1, Fig 3, Table 1, pg	pg 4
		5, pg 6
	2.2 Describe whether variables were normalized and if so, how	pg 5-7	pg 5
	this was done
	2.3 Provide details on the fraction of missing values (if any) and	pg 5-7	N/A
	imputation methods
	2.4 Perform and describe feature selection process	pg 5-7	pg 5
	2.5 Identify and describe the process to handle outliers, if any	N/A	N/A
	2.6

Acknowledgement:

We thank Jai K. Nahar, MD, MBA and James E. Tcheng, MD for reviewing and providing constructive feedback for the document.

manipulated/transformed data may be obtained and justify any restrictions to data availability.

All data should also be accompanied by a codebook (also known as a data dictionary) containing clear and succinct explanations of all variables and class labels along with detailed description of their data types and dimensions.

Finally, we note that even in the case of freely available data and open-source code, it can be difficult to reproduce the results of published work due to the complexities of software package versioning and interactions between different computing environments. We, thus, suggest that authors make the entire analyses automatically reproducible through the use of software environments (e.g., Docker containers, https://www.docker.com/). Analyses in software containers may be freely downloaded and run from beginning to end by other scientists, greatly

Strategic steps for developing the checklist:

• Release the code and upload data as supplementary information alongside the manuscript when possible for non-commercial use; otherwise, consider making the code and data available via an academic website for non-commercial use as permissible.

• Use the MIT/BSD or GPLv3 license to release open-source code.

• Release a codebook (data dictionary) with clear and succinct explanations of all variables.

• Document the exact version of all external libraries and software environments.

• Consider the use of Docker containers or similar packaging environments such as Sphinx for straightforward technical reproducibility and to generate reliable code/software manuals.

Summary and Future Directions

As artificial intelligence and ML technologies continue to grow, three specific areas of opportunities will need further consideration for future standardization. First, there has been growing enthusiasm in the use of automated machine learning (auto-ML) platforms that democratize machine-learning strategies. Second, using the 'multiomics-approach', clinical and other data like smart-phones based health-data could be integrated with imaging variables to provide more algorithmic sophistication and objectivity to the existing taxonomy of risk factors and cardiac diseases [START_REF] Natarajan | Diagnostic Accuracy of Community-Based Diabetic Retinopathy Screening With an Offline Artificial Intelligence System on a Smartphone[END_REF][START_REF] Rajalakshmi | Automated diabetic retinopathy detection in smartphone-based fundus photography using artificial intelligence[END_REF][START_REF] Wasserlauf | Smartwatch Performance for the Detection and Quantification of Atrial Fibrillation[END_REF]. Finally, sophisticated algorithms and variations of GAN will be increasingly used to synthesize data that closely resemble the distribution of the input data [START_REF] Suarez | You Do the GAN GAN[END_REF][START_REF] Wang | AT-GAN: A Generative Attack Model for Adversarial Transferring on Generative Adversarial Nets[END_REF][START_REF] Chang | Knowledge-Guided Generative Adversarial Networks[END_REF]. This approach may be particularly fruitful for the field of simulation and in-silico clinical trials, which were recently recognized by the Food and Drug Administration (FDA) as key new directions to validate novel devices and therapies [START_REF] Morrison | How Simulation Can Transform Regulatory Pathways[END_REF]. In this context, recent studies have combined computational modeling with ML for synthetic data generation or tracking a disease course [START_REF] Kagiyama | Artificial Intelligence: Practical Primer for Clinical Research in Cardiovascular Disease[END_REF]. Moreover, the ML research presents unprecedented opportunities for restructuring the industry, research, and medical alliance. This fact is well recognized by the FDA, which has mandated the standardization and applications of ML software as medical devices [START_REF]Software as a Medical Device (SaMD): Key Definitions[END_REF]. With the advancement of organizations and cardiac medicine and imaging towards the actualization of precision medicine, the PRIME Checklist would need to be updated continuously as ML algorithms continue to transform cardiovascular imaging practice over the next decade.

Coverage error

The bias that is introduced in the data when all components of the population are not adequately covered. It may occur if a component is included more than actual estimate from the survey (over-coverage), failure to include adequately (under-coverage), or misclassified.

Selection bias

The bias in the data that occurs when the individual or group of data for analysis is not collected with proper randomization, and, therefore, is not representative enough of the population of interest.

Hyperparameters

Parameters that are used to modulate how the model learns from the data and typically tuned before the learning process begins, or during the cross-validation process.

Neural Networks ML models that are inspired from the neurons of the human brain. A network has input layers and output layers and may have one or more intermediate layers. Some networks have only one or a few layers, such as perceptron, or multiple layer perceptron, which only feeds the data forward. Deep learning relies on a network with several layers in the architecture and has a backward data feed to minimize error.

Deep neural network / Deep Learning

A type of ML based on neural networks made of several (in general, at least 3) intermediate layers. It is used for supervised and unsupervised tasks but requires large amount of data compared to traditional shallow learners. It is also known as deep neural network. Convolutional neural network, generative adversarial network, recurrent neural network are some examples popular in computer vision and medical imaging.

Independent and Identical Distribution

If the data or a random variable is collected from the same probability distribution as all others and is mutually independent such that the outcome of the data is not dependent on the previous observation.

Data standardization, feature engineering and learning

Matrix

Mathematical object that contains numeric values, symbols, or expressions arranged in a rectangular array of columns and rows.

Histogram equalization

A method in image processing to adjust the contrast of the image using the image's histogram.

Pixel intensity

The value of a pixel's brightness. In normal medical images, pixel intensities range from 0 to 255 where greater values indicate brighter colors.

Data Annotation

A process of labeling the data available in various formats (e.g., image, video, audio or text) to make it usable for ML.

Data augmentation

A technique to add data or variability to the data.

Transfer learning

Transfer learning is a method of ML in which the knowledge gained from another pretrained model is applied to learn from another set of data. This method can reduce the need for data by orders of magnitude.

Dimensionality Reduction

An unsupervised ML technique to find a new representation of the data while reducing the features that describe the data. This method is useful for avoiding overfitting and producing simpler models.

Multicollinearity

An issue in statistics and ML regression models in which two or more variables provide the same information due to their close relationship.

Generative Adversarial Network (GAN)

Generative models using deep neural network architecture that consists of the following two stages: the generator stage, which generates new examples, and the discriminator stage, which classifies if the generated examples are real (with the same distribution as the problem domain) or fake. Importantly, generative models became popular due to their ability to generate highly realistic images (in terms of appearance, not necessarily physiological content).

Hand-crafted features

In image-based classification tasks, "hand-crafted" features refer to properties derived using certain manually predefined algorithm based on the expert knowledge using the information present in the image itself. Some examples of hand-crafted features include local binary patterns, scale invariant feature transform and histogram of oriented gradients etc.

Variational Autoencoders

In deep learning, variational autoencoders (VAE) are a class of neural networks that can learn to compress data (i.e., image, text or sequence) and perform dimensionality reduction completely in an unsupervised manner. Instead of letting a neural network learn from an arbitrary function, VAEs learn from the parameters of a probability distribution by modeling input data.

Linearity

In statistical terms, linearity means that the response variable is a linear combination of the parameters of independent variables (i.e., regression coefficients) and the predictor/dependent variables.

Loss function

A method to measure error made by the model in prediction for a single training example. Mean squared error, likelihood loss, mean absolute error are some of the common loss functions in evaluating the model. It is also known as error function.

Regularization

Technique to reduce complexity and avoiding overfitting by adding penalty term which can e.g., shrink or eliminate the coefficients, to smoothen the estimated trend or classification boundary.

Precision

Proportion of samples that were classified or identified as positive.

𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 + 𝐹𝑎𝑙𝑠𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒

Recall

Proportion of positive samples classified or identified correctly. 9:;< =>?@A@B< 9:;< =>?@A@B<CDEF?< G<HEA@B< F1 score Weighted average of the precision and recall where 1 is the best value and 0 is the worst.

𝐹1 = 2 × 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑅𝑒𝑐𝑎𝑙𝑙 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙

Brier score

Method to verify the accuracy of a classification where 0 is for 100% accurate and 1 is 100% inaccurate. The most common formulation of the Brier score is

where N is the number of instances, ft is the forecast probability (i.e. 25% chance) and ot corresponds to the outcome (1 if it happened, 0 if it didn't).

Software Engineering Best Practices and Data Availability for Reproducibility

Source code repository A central file storage location used by version control system to manage and store several versions of source codes.

Practical illustration of how to apply the checklist developed in the PRIME guidelines

The finalized PRIME reporting guidelines are shown in Table 1. The guidelines checklist is structured to correspond with the various facets of the building machine learning systems (data preprocessing, analysis, model development etc) and is intended to capture critical information to enable correct application of machine learning (ML) models and allow consistent reporting of model results in cardiovascular imaging studies.

Basically, checklist is a set of essential items for authors to consider when reporting results from ML models for publication in cardiovascular imaging. The checklist begins with an item that relates to providing a detailed description of the overall study plan. It is very important to clearly define and establish the need for the application of ML approaches to a given problem, as it becomes the foundation for all the other steps in ML workflow. Model selection, algorithm complexity, performance evaluation and assessment all are interlinked and often depend on the task that one is trying to solve. Similarly, we also recommend listing and reporting items related to limitations, biases and alternative analysis strategies, to satisfy the need for precision, completeness, and transparency in reporting results of modeling studies. Apart from these, we also created separate checklist items for each process in the ML workflow to provide a list of reporting items to be included in a research articles and reports.

To further determine its ease of use, applicability and understand how well it captures all the practical steps of developing predictive models, we gathered information from two recent articles that developed ML models in cardiovascular imaging. The entire manuscripts were searched thoroughly to extract any information reported that was in reference to the PRIME recommendation's checklist items. A complete checklist along with page numbers referring to each item is provided in Table S1 (see below). Both of these studies implemented deep learning models either to improve the diagnosis of congenital heart disease or augment the computer-assisted echocardiographic interpretation. Almost 90% of the items in the PRIME reporting checklist were addressed or satisfied by the two studies considered, indicating that these reports/articles were executed with PRIME-analogous guidelines in mind. Certain missing checklist items such as "Benchmark complex models against more simplistic models if possible" were in fact not applicable due to the specific nature of the selected ML model and the nature of the study (e.g., deep learning applied to video or image analysis. Thus, the guidelines listed here could be effective in capturing the reporting requirements of studies employing machine learning approaches in cardiovascular imaging.

Supplemantary Table S1