domain: info scco

text:
Rafael Rodrigues
email: rafael.rodrigues@ubi.pt

Lucie Lévêque

Jesús Gutiérrez

Houda Jebbari

Meriem Outtas

Lu Zhang

Aladine Chetouani

Shaymaa Al-Juboori

Maria Martini

Antonio M G Pinheiro

Objective quality assessment of medical images and videos: Review and challenges

Keywords: Quality assessment, Objective metrics, Medical imaging, Task-based quality

Quality assessment is a key element for the evaluation of hardware and software involved in image and video acquisition, processing, and visualization. In the medical field, user-based quality assessment is still considered more reliable than objective methods, which allow the implementation of automated and more efficient solutions. Regardless of increasing research in this topic in the last decade, defining quality standards for medical content remains a non-trivial task, as the focus should be on the diagnostic value assessed from expert viewers rather than the perceived quality from naïve viewers, and objective quality metrics should aim at estimating the first rather than the latter. In this paper, we present a survey of methodologies used for the objective quality assessment of medical images and videos, dividing them into visual quality-based and task-based approaches. Visual quality based methods compute a quality index directly from visual attributes, while taskbased methods, being increasingly explored, measure the impact of quality impairments on the performance

Introduction

Medical images and videos provide clinical information from the human body with reduced invasiveness, but also structural and functional outcomes which could not be obtained by other means. Nowadays, medical imaging plays an inexorable part in diagnosis, treatment planning, and patient monitoring. Over the last decades, medical imaging techniques have been constantly developed, updated, and extensively used in numerous medical specialties. The World Health Organization (WHO) estimated around 3.6 billion diagnostic procedures performed worldwide, per year, between 1997 and 2007 [START_REF]Communicating radiation risks in paediatric imaging: information to support health care discussions about benefit and risk[END_REF]. Radiology is the leader in the production of medical imaging content [START_REF] Krupinski | [END_REF], with a vast number of imaging modalities. Data from the Organization for Economic Co-operation and Development (OECD), which considers Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET) scans performed each year, further reveals a clear upward trend in the last decade, with over 253 million exams reported by OECD countries [START_REF]Health care utilisation[END_REF]. Moreover, and besides other relevant imaging modalities, medical images and videos are now also transmitted in real time for telemedicine applications. Therefore, large amounts of content from different acquisition methods are continuously created in the medical practice.

MEDICAL IMAGE & VIDEO QA

Task-based methods

Visual quality-based methods

Full Reference (FR)

No Reference (NR) Reduced Reference (RR) Localization Detection and Classification Segmentation

Fig. 1 Overview of the paper organization (QA: quality assessment).

Several impairments can affect the quality of the end visual signal, impacting the quality perceived by the viewers (e.g., clinicians, radiologists, etc.), as well as their performance on clinical tasks. This spectrum of impairments highly depends on a wide variety of acquisition and reconstruction-related factors, which are often specific to each imaging modality. Furthermore, external and patient-related factors may induce artifacts in the acquired content, as well. Images and videos may also be subject to different processing, encoding/compression, transmission, and visualization methodologies. For example, in telemedicine applications, video content and real-time interactions are key features, which lead to a high demand of hardware resources and network bandwidth for data storage and transmission [START_REF] Lévêque | [END_REF]5]. Thus, measuring image and video quality in health applications is both a necessity, towards improving methodologies throughout the clinical workflow, and a wide open challenge, given the diversity of content, applications, and impairments. This paper aims at providing researchers with a broad sense of current boundaries and future opportunities in this field.

Overview of Image and Video Quality Assessment

The quality perceived by the users of image and video content is one of several factors influencing their Quality of Experience (QoE), defined by EU Cost Action 1003 Qualinet [START_REF] Callet | Output from the Fifth Qualinet Meeting[END_REF] as "the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations concerning the utility and enjoyment of the application or service in the light of the user's personality and current state".

Typically, image and video quality is evaluated from a perceptual perspective, either subjectively or objectively, with each approach bearing its own motivations, advantages, and drawbacks. Subjective quality assessment (QA) relies on human observers to analyze images or videos, and rate their quality following specific methodologies (e.g., [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF][START_REF]Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment[END_REF]). The output of subjective tests is then statistically analyzed, usually by computing the Mean Opinion Score (MOS) or Differential MOS (DMOS) [START_REF] Nasr | [END_REF]. Depending on the type of content and application, user expertise may be a requirement for an accurate and meaningful evaluation. In most healthcare applications, the quality perception of end-users is likely to be strongly influenced by the clinical utility of the content, rather than strictly aesthetic criteria. Therefore, subjective QA methods may have some limitations in this context, concerning the availability of expert observers, in addition to the inherent intra-and inter-subject scoring variability [10].

While a recent review of subjective QA of medical images and videos was published by Lévêque et al. [11], our paper focuses on objective QA methods for medical images and videos. Objective QA relies on the use of image processing and analysis algorithms towards automated quality estimation methods [START_REF] Nasr | [END_REF]. Although objective methods may be less reliable than subjective assessment, they are more cost-effective, less time-consuming, and reduce observer variability and bias [10]. In another review paper published in 2016, Chow et al. discussed both subjective and objective methods [12], with a primary focus on MRI, CT, and ultrasound imaging. More recently, Raj et al. [13] presented a survey of objective QA methods for fundus images. In our article, we will mainly focus on more recent research, and on imaging modalities not included in these last two papers.

The most common design principle in objective image and video QA applications is the computation of a quality index from certain visual and/or structural attributes of the content. On the contrary, task-based QA methods assess the quality in terms of a specific goal of the content, by measuring its influence on the performance of certain tasks (e.g., diagnosis or localization of an anatomical structure [14]). This kind of approach has been increasingly explored in the development of quality metrics that are specific for medical images and video. Consequently, we propose to divide the discussed QA methodologies into two major categories, i.e., visual quality-based, and task-based methods, as described in Figure 1.

Visual Quality-based Methods

Visual quality-based methods are commonly categorized as Full Reference (FR), No Reference (NR), or Reduced Reference (RR), depending on the availability of an undistorted reference (preferably the original image or video). FR metrics predict the quality by directly comparing the reference and its distorted versions, contrary to NR metrics which assess the quality only using the distorted image. Finally, RR metrics compare features representative of the distorted and reference images. The performance of objective methods is usually evaluated through a statistical comparison with subjective results [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF][START_REF]Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models[END_REF]. The following subsections present an overview of visual quality-based objective QA methods for medical images and video, according to the aforementioned categorization.

Full-Reference Approaches

Full-reference QA of medical images and videos had focused almost exclusively on adapting metrics designed for non-medical content to obtain quality predictions for compressed and/or artificially distorted medical content. Table 1 summarizes the works presented in this section.

Magnetic Resonance Imaging

FR quality assessment studies using MRI includes the works of Chow et al. [START_REF] Chow | Alzheimer's Disease Neuroimaging Initiative[END_REF] and Mason et al. [10]. Both works tested a large set of metrics, e.g. [28][29][30][31][32][33] on datasets with different compression settings (Discrete Cosine Transform (DCT) [34], JPEG [35], JPEG2000 [36], wavelets) and simulated artifacts (e.g. Rician noise, Gaussian blur or motion artifacts). The best average correlations with the subjective scores were achieved for Noise Quality Measure (NQM) with Pearson Linear Correlation Coefficient (PLCC) and Spearman Rank-order Correlation Coefficient (SROCC) around 0.94 [START_REF] Chow | Alzheimer's Disease Neuroimaging Initiative[END_REF], and Visual Information Fidelity (VIF) [10].

Computed Tomography

In an early work from 2003, Zhou et al. [START_REF] Zhou | International Conference on Neural Networks and Signal Processing[END_REF] estimated the quality of CT images with both DCT and Wavelet compression, using a back-propagation neural network with Peak Signal-to-noise Ratio (PSNR) and Structural Similarity index (SSIM), and the Blockwise Distortion Measure [START_REF] Fränti | [END_REF] as inputs. The reported agreement rates with the obtained subjective scores ranged between 85.71% and 96.88%. Kowalik et al. [START_REF] Kowalik-Urbaniak | International Conference Image Analysis and Recognition[END_REF] also used CT images with compression -JPEG and JPEG2000, and studied the Receiver Operating Characteristic (ROC) curves of SSIM, Signal-to-noise Ratio (SNR), VIF, Mean Squared Error (MSE), and Visual Signal-to-noise Ratio (VSNR) [37] in classifying images as having non-noticeable or non-acceptable distortions, after subjective annotation by experts. The largest Area Under the Curve (AUC) was obtained with SSIM (0.99 for brain images and 0.96 for body images), closely followed by VIF.

Ultrasonography

In [START_REF] Panayides | [END_REF], the authors evaluated the performance of a H.264 encoding framework for atherosclerotic plaque ultrasound videos, which resulted in enhanced performance in noisy environments. WSNR obtained the best correlation with the subjective scores (PLCC = 0.69 and SROCC = 0.72). The authors also proposed minimum settings for several parameters, including the frame rate, bit rate, and PSNR in the Regions of Interest (ROI).

The authors of [21] and [22] assessed the quality of ultrasound video excerpts compressed with High Efficiency Video Coding (HEVC) [38], using a set of 7 common FR metrics, as well as the proposed content-specific Cardiac Ultrasound Video Quality Index (CUQI), in [22]. CUQI estimates the diagnostic quality of cardiac ultrasound video, using motion and edge information. In terms of correlation with DMOS, all metrics yielded good PLCC and SROCC coefficients in both studies. In [22], CUQI outperformed the other tested metrics, with PLCC and SROCC after DMOS nonlinear regression of 0.94 and 0.93, respectively. Nonetheless, all the tested metrics achieved correlations above 0.9, and SSIM achieved the best result in terms of PLCC without nonlinear regression of DMOS.

Endoscopic / Laparoscopic videos

In [START_REF] Kumcu | Medical Imaging 2014: Image Perception[END_REF] and [5], the authors performed objective QA of laparoscopic and surgical videos, respectively, compressed with H.264. SSIM, High Dynamic Range Visible Difference Predictor v2 (HDR-VDP-2), and Video Quality Metric (VQM) were used in both studies, but a larger set of metrics was used in [5], including, for example, PSNR-HVS and PSNR-HVS-M [START_REF] Egiazarian | Proceedings of the Second International Workshop on Video Processing and Quality Metrics[END_REF], MSE,Multi-scale SSIM (MS-SSIM) [START_REF] Wang | The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers[END_REF], and two NR metrics -Natural Image Quality Evaluator (NIQE) [START_REF] Mittal | [END_REF] and Blind/Referenceless Image Spatial Quality Elevator (BRISQUE) [42]. In the latter study, MSE, SSIM, MS-SSIM, and BRISQUE achieved good correlations with MOS (PLCC > 0.9).

Usman et al. [24] were the first to investigate the quality of wireless capsule endoscopy videos. Videos were compressed with HEVC, and 10 metrics were computed to estimate video quality. VIF, Pixel Visual Information Fidelity (VIFP), and Information Fidelity Criterion (IFC) showed good correlation with the obtained MOS (between 0.88 and 0.92), with VIFP outperforming the other two, both in statistical significance and in computation time.

Multiple imaging modalities

Other interesting works were published in [START_REF] Kumar | ternational Conference on Signal Processing Systems[END_REF] and [START_REF] Kumar | IEEE International Conference on Image Information Processing[END_REF], where the authors studied the performance of PSNR and SSIM on CT, MRI, and ultrasound images with Set Partitioning in Hierarchical Trees (SPIHT) compression [43]. Both PSNR and SSIM quality estimations correlated well with subjective scores. In [START_REF] Kumar | IEEE International Conference on Image Information Processing[END_REF], Kumar et al. proposed an improved version of SSIM, which outperformed PSNR with correlation coefficients of 0.99 (MRI), 0.98 (CT), and 0.98 (ultrasound).

Several variations of SSIM were also tested by Renieblas et al. [START_REF] Renieblas | [END_REF], to estimate the quality of planar Xray and MRI, with simulated Gaussian blur and noise, and JPEG and JPEG2000 compression. Best overall correlation with subjective scores were obtained with 4-MS-G-SSIM (PLCC ≥ 0.75 overall, and ≥ 0.86 for MR images), which considers four-component regionbased weighting (4-), multiscale (MS-), and gradientbased (G-) computation. The authors concluded that MS-approaches generally improve the performance of single-scale counterparts, and that r * (structural component only) showed a slight advantage over the complete SSIM index.

Reduced-Reference Approaches

A few works, described in Table 2, addressed the use of RR metrics to measure the quality of medical images and video. One of the most popular is the RR version of VQM, which, as aforementioned, was used in [START_REF] Kumcu | Medical Imaging 2014: Image Perception[END_REF], showing a reasonable performance (PLCC=0.97 and SROCC=0.94) in comparison to other FR metrics. Other approaches are based on the comparison of certain attributes of the images or videos, through the computation of a similarity score. For example, Lee and Wang used the similarity between the intensity histograms of reference and distorted fundus images to provide an estimation of their quality [44]. For the same image modality, the authors in [START_REF] Lalonde | Proceedings of vision interface[END_REF] proposed a similarity metric based on the comparison of the distribution of edge magnitudes and the local intensity distribution for distorted and reference images. In both cases, the performance of their approaches was not reported with objective measures, but through a compar-ison with SNR and through qualitative results, respectively. The results show that the proposed approaches are useful to discriminate between good and bad images. In addition, the proposed approaches can be extended to other types of images.

Finally, another set of RR metrics is based on the use of watermarking, i.e., adding a payload to the content, and then using similarity metrics to compare the reference and the distorted image or video. An example of this approach was proposed by Planitz and Maeder [START_REF] Planitz | Digital Image Computing: Techniques and Applications[END_REF], using SSIMto measure the degradations and watermarking capacity. This study demonstrated that more robust watermarking techniques can be used in less visually sensitive areas, while lighter techniques should be applied in more sensitive areas. Nasr and Martini [START_REF] Nasr | [END_REF] used PSNR and Mean SSIM (MSSIM) to measure the quality of medical ultrasound videos with simulated Gaussian, and JPEG2000 and HEVC compression. A predefined reduced-size logo that shares the same features of the original frame of interest (i.e., same organ and layout). Considering that the logo does not depend on the specific content of the sequence, this method can be considered, to some extent, as NR. The authors concluded that the proposed technique did not require the original frame, whilst achieving a high PLCC with the subjective results, with reported average correlations above 0.97, for both PSNR and MSSIM.

No-Reference Approaches

Tables 3 and4 summarize the articles further presented in this section.

Magnetic Resonance Imaging

Liebgott et al. [START_REF] Liebgott | IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)[END_REF] proposed a new method to assess the quality of 2D MRI using active learning. A descriptor including features based on contrast, resolution, texture, and intensity information was reduced using Principal Component Analysis (PCA) [START_REF] Abdi | [END_REF] and classified using an Support Vector Machine (SVM) model [59]. Subjetive socres on a 5-level scale were used to train and validate the model. Results showed that active learning reduces the need for training data by around 50%, compared to a previous method by the same authors.

Chow and Rajagopal proposed a modified version of BRISQUE [START_REF] Chow | [END_REF], trained with mean-subtracted contrastnormalized MRI and the corresponding DMOS. This proposed metric was evaluated on two separate datasets with both T1-weighted (T1w ) and T2w brain MRI, one with unknown artifacts, and another with the same range of distortions of [START_REF] Chow | Alzheimer's Disease Neuroimaging Initiative[END_REF]. The proposed modified The authors in [49] applied the Bayes theorem to calculate the posterior probability of entropy, given three quality attributes, i.e., contrast, sharpness, and standard deviation. A global quality index for 2D MRI was obtained by averaging those probability values, which were first separately computed for low-and high-entropy feature images. The evaluation dataset included ten T1w MRI of the brain acquired with bias fields, and twenty-one images acquired without perceived distortions (T1w, T2w, Proton Density and Fluid Attenuated Inversion Recovery (FLAIR)). Twenty different levels of Rician noise and motion blur were induced in the second subset. Five radiologists performed subjective QA, which showed that the predicted quality decreased consistently across the twenty distortion levels of noise and blurring. Correlation with MOS was assessed separately for each modality and distortion level using SROCC. The reported coefficients were globally above 0.6, with a tendency to be higher for lower distortion levels.

In [50], the authors presented ENMIQA, an entropybased metric for the objective QA of MRI, which expresses local intensity differences after non-maximum suppression at various threshold levels. A dataset of T2w MRI was used, but no information was provided in the paper on the type of artifacts or noise present in the images. The performance of the proposed metric in quality prediction was compared against a large set of NR, and ENMIQA outperformed them all in terms of correlation with MOS. However, overall correlation coefficients were quite low, especially for rank correlations (PLCC = 0.65, SROCC = 0.35). PLCC was also reported for each anatomical structure separately, with wrist and knee images yielding coefficients near 1 and 0.9, respectively.

In a recent work, Chabert et al. [51] proposed using a set of features reduced using PCA, which included SNR, Contrast-to-noise Ratio (CNR), Foreground-Background Energy Ratio (FBER), sharpness in fat, uniformity, the Wang index [START_REF] Wang | Proc. of the International Conference on Image Processing[END_REF] and Shannon entropy [START_REF] Hadjidemetriou | [END_REF], among others, to predict the quality perception of neuroradiologists for lumbar T1w and T2w MRI, with simulated Gaussian noise and blurring, and contrast manipulation. A few classifiers were tested, with SVM showing a superior overall performance (accuracy above 73%). Despite relying on user interaction for ROI-based feature extraction, the results seem promising for online monitoring of the image quality, at the moment of acquisition.

The appearance of deep learning models has been establishing a new paradigm in NR medical image quality assesment. Esses et al. [52] trained a Convolutional Neural Networks (CNN) model based on the AlexNet architecture [62] to screen the diagnostic quality of T2w liver MRI. The obtained results indicate a 79% and 73% concordance between CNN predictions and experts 1 and 2, respectively. Moreover, the CNN showed good negative predictive values in the identification of nondiagnostic image quality (94% and 86% for each of the experts).

Sujit et al. implemented a branched custom CNN model to assess the image quality of 3D T1w brain MRI [53]. Each branch received a different imaging plane (i.e., coronal, sagittal, and axial) as input. The global quality score was obtained by averaging the scores of each plane. Training and validation data were collected from the Autism Brain Imaging Data Exchange (ABIDE) multicentric dataset [63], which is publicly available with binary subjective quality scores (i.e., "acceptable" or "unacceptable"). The proposed ensemble model obtained an AUC of 0.90 on a 20% test split. The model was also tested on a separate test set, from the CombiRx dataset [64], yielding a lower AUC of 0.71. The authors argued that the lower results may be due to differences in cohorts, as CombiRx data was not included in training.

Ma et al. [START_REF] Ma | IEEE 17th International Symposium on Biomedical Imaging (ISBI)[END_REF] investigated the use of CNN to estimate the diagnostic quality of abdominal MRI. A custom 4-layer acCNN and a ResNet-10 model, were trained to classify images on both binary (nondiagnostic vs. diagnostic) and 3-level (non-diagnostic (0), diagnostic (1), or excellent ( 2)) quality scales. The proposed 4-layer CNN outperformed ResNet-10, with an accuracy of 84% in binary classification (AUC = 0.72) and 65% in 3-label classification (AUC 0 = 0.77, AUC 1 = 0.69, AUC 2 = 0.83). However, the authors argued that the high disagreement between human observers, as well as the probability of label unreliability, could influence the results. Moreover, the activation maps suggest that low-level features have higher discriminative power, whereas deeper features do not have a great impact on these classification tasks, i.e., deeper models may not necessarily improve the results in similar applications.

Retinal fundus images / Ophthalmology images

Köhler et al. [START_REF] Köhler | Proceedings of the 26th IEEE international symposium on computer-based medical systems[END_REF] extended the work of [65], where a global quality index Q was obtained from patchwise quality indices q(P ). These were based in the singular value decompositions of G, a patch-wise gradient matrix. In [START_REF] Köhler | Proceedings of the 26th IEEE international symposium on computer-based medical systems[END_REF], the authors implemented a spatially weighted version (Q v ), assigning larger weights to patches around blood vessels, according to a vesselness measure. The proposed metric was tested as a PSNR and SSIM, on images from the DRIVE database, with simulated Gaussian blur and noise. Q v outperformed Q, with an overall SROCC of 0.89 (PSNR) and 0.91 (SSIM). In a second experiment, the authors performed binary quality prediction (acceptable vs. nonacceptable), considering annotations from experts. Q v outperformed other NR metrics -Q, Cumulative Probability of Blur Detection (CPBD), and an anisotropy measure -, with an AUC of 0.89.

The authors in [START_REF] Wang | [END_REF] proposed an Human Visual System (HVS)-based feature extraction algorithm, which relies on multi-channel sensation, Just Noticeable Blur (JNB), and the Contrast Sensitivity Function, to detect illumination and color distortions, blur, and low contrast in retinal fundus images. Extracted features were classified with an SVM and a decision tree, to predict image quality, either based on each of the three aforementioned properties, or globally. A joint dataset contained the 3 partial binary annotations, given by ophthalmologists. Average AUC of 0.97, 0.96, and 0.68 for illumination/color, JNB, and color, respectively, was obtained with the decision tree, and 0.93, 0.93, and 0.88 with the SVM. In terms of overall quality, the SVM performed best, with sensitity/specificity of 0.92/0.87.

In [START_REF] Remeseiro | International Joint Conference on Neural Networks (IJCNN)[END_REF], the authors used texture features from RGB and CIELab channels, including Wavelet and Gabor filters, and features from the Gray-level Co-occurrence Matrix (GLCM) [66] to compute the quality of retinal fundus images. Several feature selection methods and classifiers were tested for the binary classification of the image quality (good / poor quality). However, no information on the quality impairments nor the subjective evaluation setup is provided in the paper. The best performance in quality prediction was obtained with GLCM features from CIELab channels, filtered using correlation-based feature selection, and classified with an SVM classifier (Accuracy = 99.09%).

More recently, Coyner et al. [67] studied the use of CNN models to measure the quality of retinal fundus images, considering their usefulness for a confident detection of retinopathy of prematurity. The model was based on the InceptionV3 architecture, and was trained to classify an independent set of thirty images as having acceptable or non-acceptable quality. The testing set was previously ranked by six experts, and the CNN probability output showed good correlation with the consensus ranking from the experts (SROCC = 0.90). SROCC ranged from 0.86 to 0.93, when considering individual rankings.

Focusing on eye fundus images for the diagnosis of diabetic retinopathy, Raj et al. [68] presented a new multivariate regression-based CNN model to predict the image diagnostic quality. The proposed model incorporates four different backbones -InceptionV3, DenseNet-121, ResNet-101, and Xception -, trained against subjective scores for six quality parameters, i.e., visibility of the optic disc, macula, and blood vessels, color, contrast, and blur. The top of the model provides a global diagnostic quality score based on the computed features. Results showed a strong correlation with the subjective scores for overall diagnostic quality, with values 0.94, 0.95, 0.85, and 0.40 obtained for SROCC, PLCC, Kendall Rank Correlation Coefficient (KRCC), and Root Mean Square Error (RMSE) respectively. The classification accuracy was 95.66% over the FI-QuA dataset, presented in the same paper, and 98.96% and 88.43% respectively over the two publicly available datasets DRIMDB [69] and EyeQ [START_REF] Fu | International Conference on Medical Image Computing and Computer-Assisted Intervention[END_REF].

In [START_REF] Shen | [END_REF], the authors proposed a CNN-based model with three modules for the quality assessment of retinal images. The model extracts global and local features (optic disc and fovea) using a VGG-16 backbone, which provides an overall quality score and three partial scores for different factors -artifacts, clarity and field definition. Visual feedback from class activation mapping is also provided to ophthalmologists. ROI detection is performed by a separate module, using a ResNet-50 backbone for center detection and a VGG-16 local encoder for iterative ROI refinement. A third module implements an unsupervised Adversarial Discriminative Domain Adaptation (ADDA) method [START_REF] Tzeng | IEEE Conference on Computer Vision and Pattern Recognition (CVPR)[END_REF], using a Generative Adversarial Network architecture [START_REF] Goodfellow | [END_REF] to address domain shifts between training and test data. Several variations of the proposed model were tested to classify images as adequate or inadequate for the diagnosis of diabetic retinopathy, and the best result was obtained with the full described model, with an AUC of 0.95.

Niwas et al. [74] developed a metric to assess the quality of Anterior Segment Optical Coherence Tomography (AS-OCT) images using Local Binary Patterns (LBP) in the complex Wavelet domain. The image is first decomposed into 32 wavelet decomposition levels, using the Double Density Dual Tree-complex Wavelet transform, and the LBP histogram is obtained for each sub-band. The Minimum Redundancy Maximum Relevance (mRMR) method is then applied to select the most relevant features to classify the image in terms of quality, with three levels considered. Several classifiers were tested, including SVM, random forest, decision tree, and an AdaBoost classifier, but the best results were achieved using a Naïve Bayes classifier with mRMR feature selection. These were compared to the ground truth given by experts, yielding an overall weighted accuracy of 82.9%.

Endoscopic / Laparoscopic videos

Based on a new publicly available database of 2D laparoscopic videos (LVQ) [START_REF] Khan | Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment[END_REF], a residual network-based method was proposed in [START_REF] Khan | IEEE International Conference on Image Processing (ICIP)[END_REF] to predict the quality score and detect the type of distortion. Particularly, to tackle the problem of limited data, a ranking based pre-training approach has been proposed. The proposed model with a ResNet-18 backbone, obtained a better performance in terms of SROCC with the established quality ranking (0.69), when compared to another deep learning-based method, i.e. RankIQA [START_REF] Liu | Proceedings of the IEEE International Conference on Computer Vision[END_REF] (0.57). It should be noted that the quality ranking was inferred directly from the severity levels of simulated distortions (i.e., defocus and motion blur, Gaussian noise, uneven illumination, and smoke). Simultaneously, the authors attempted image classification into 20 different labels, considering every distortion-level pair. Different ResNet depths were tested, with ResNet-50 achieving the best accuracy (87.3%).

In a very recent paper, Ali et al. [START_REF] Ali | [END_REF] proposed an integrated deep learning approach for both QA and frame restoration in video endoscopy. In the context of this paper, the QA method relied on detecting six different types of impairments, i.e., bubbles, blur, contrast, specularity, saturation, and miscellaneous artifacts, in near real-time, using a YOLOv3 architecture. The dataset included a set of frames from both normal bright field and narrow-band imaging videos of different patients. After a bounding box detection stage, the framework also performed a finer segmentation of the artifacts, with the best results being obtained with DeepLabv3+ spatial pyramid pooling.

Ultrasonography

Abdi et al. [77] trained a total of 430 CNN models to attempt the automated QA of transthoracic echocardiograms, using Particle Swarm Optimization (PSO) for hyperparameter optimization. Subjective scoring was based on the visibility of several anatomical structures (Manual Echo Score (MES)). The network resulting from PSO featured three convolutional layers, fed to two fully connected layers, and was trained independently three times. The final model performance was measured by the mean absolute error between the predicted score and the MES, with average reported values of 0.71 ± 0.58. The authors further analyzed the obtained CNN feature maps, which suggested the visibility of the septum and lateral walls to be important factors for a higher predicted image quality.

Fused images

Tang et al. published two papers on the quality assessment of Multimodal Medical Image Fusion (MMIF).

In both papers, the authors tested eight image fusion algorithms, namely, Nuclear Norm Minimization (NNM), Laplacian Pyramid and Sparse Representation (LP-SR), Cross-scale Coefficient Selection (CSCS), Guided Filtering (GFF) Pulse-coupled Neural Network with Modified Spatial Frequency based on Nonsubsampled Contourlet Transform (NSCT-PCNN-SF), Improved Sum-Modified-Laplacian (ISML), Convolutional Sparse Representation (CSR), and Discrete Tchebichef Moments and Pulse-coupled Neural Network (DTM-PCNN), with different pairs of imaging modalities, namely, CT and MRI, T1w and T2w MRI, B-mode Ultrasound and Single-photon Emission Computed Tomography (SPECT), MRI and PET, and MRI and SPECT. In [78], the quality index was obtained by pooling the phase congruency, which measures the correlation between salient features of the source and fused images, and the standard deviation, which provides a measure of sharpness and clarity. In [79], the authors further proposed using a PCNN with NSCT subimages (both high-and low-frequency components).

An overall quality index was then given by pooling the obtained components. The performance of the proposed metrics was compared to several state-of-theart metrics, including gradient, structure, edge and entropy-based metrics, considering their correlation to the MOS of twenty radiologists. In [78], the proposed metric achieved average a SROCC near 0.8 and KRCC near 0.7, whilst in the more recent paper, the proposed metric yielded the following performance measures: PLCC=0.79, SROCC=0.73, KRCC=0.61, and RMSE=0.27. Although both metrics largely outperformed the baseline metrics, suggesting their relevance in MMIF QA, the obtained correlations could be further improved.

Multiple imaging modalities

In [START_REF] Outtas | International Symposium on Signal, Image, Video and Communications[END_REF], the authors proposed to modify NIQE [START_REF] Mittal | [END_REF], a NR metric originally developed for natural images, and used for the QA of medical images in some studies. The authors improved its perceptual evaluation, using a frequency-domain analysis inspired by the Blind Image Quality Evaluator based on Scales (BIQES) metric [START_REF] Saha | [END_REF]. The ratio kurtosis/standard deviation of the log amplitude of the Fourier spectra was taken as weighting factor. In a first experiment, ultrasound images were corrupted with simulated noises, i.e. Sattar's noise and speckle noise, and then filtered using a median filter. NIQE-K and BIQES achieved comparable performances in terms of quality ranking. Nonetheless, NIQE-K ranked a noisy image (Sattar's noise) better than the original ultrasound image, despite their close quality indices. In the second experiment, three radiologists were asked to the detect and localize multiple sclerosis lesions in MRI. Their performance was quantified by the Jack-knife Alternative Free-response Receiver Operating Characteristic (JAFROC). Result analysis showed the consistency of NIQE-K for ultrasound image quality, and that, for 40% of the images, the behavior of NIQE-K is the same as that of the radiologists.

Task-based Methods

Task-based objective QA methods, also often referred to as numerical observers (NO) or model observers (MO), are designed to approach the performance of human observers (i.e., medical experts) in a given task, as opposed to visual quality-based approaches, which typically aim at predicting the MOS of human observers [84]. The underlying paradigm is to quantify the quality of medical images and videos by their effectiveness with respect to its intended purpose [START_REF] Zhang | Sixth International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM)[END_REF]. In this section, we review task-based methods proposed for different tasks, as well as the corresponding Figures of Merit (FOM) for performance evaluation. Table 5 is a summary of these works (note that we focus on works published after 2013, since an overview was already published then [START_REF] Zhang | [END_REF]).

Detection and classification tasks

The most common task is the detection task, in which a decision is made in favor of one of two hypotheses, i.e., signal present (H1) or absent (H2). Low-dose reconstruction methods using iterative algorithms in tomography introduced new challenges related to extraction of image information. In any diagnosis, a good compromise must be found between the dose level and the resulting quality of the medical image, in order to reduce the exposure dose, while allowing the detection of low contrast textures. In [87], the authors used a Channelized Hotelling Observer (CHO) [START_REF] Zhang | [END_REF]88] and developed an internal noise model to compare detectability indices (d ) in low-dose CT images. The d index, chosen as FOM, was calculated from the distribution of signal and noise decision variables (i.e., sum of channel outputs). The experiment was performed with five observers, using CT phantom images, and showed that Iterative Model Reconstruction (IMR, Philips) enables at least a 67% dose reduction comparatively to Filtered Back-projection (FBP).

Racine et al. [89] conducted research to objectively evaluate the low contrast detectability in CT, with different radiation doses (CTDI vol of 5, 10, 15, and 20 mGy). Images of a QRM 401 phantom containing 5and 8-mm diameter spheres, with a contrast level of 10 and 20 HU, were acquired and reconstructed using three algorithms, i.e., FBP, Adaptive Statistical Iterative Reconstruction (ASIR), and Model-based Iterative Reconstruction (MBIR). A CHO model with Dense Difference-of-Gaussian (D-DOG) channels was used to evaluate the image quality for every combination of the aforementioned parameters. The performance of the CHO model was compared with that of six medical students, who provided detectability levels for the test images in four-alternative forced choice tests. A high correlation was found between the results of the human observers and the CHO model, independently of the dose levels or the signals considered, suggesting it might be used to predict expected detectability levels, and ensure the diagnostic quality of low-dose CT acquisitions. PLCC was 0.98 for MBIR and 0.93 for FBP. MBIR gave the highest overall detectability index, particularly for low CTDI vol .

Greffier et al. [90] used the imQuest software to assess the quality of low-dose CT scans, comparing the performance of four manufacturers (i.e., GE, Philips, Siemens, and Canon), with different reconstruction algorithms (i.e., FBP, MBIR and Hybrid/Statistical Iterative reconstruction (H/SIR)), and five dose levels (i.e., 0.5, 1.5, 3.0, 7.0, and 12.0 mGy). A Non-Prewhitening Matched Filter (NPWMF) model observer with an eye filter [91] was used to calculate the detectability index, considering two detection tasks: a large mass in the liver, and a small calcification. The d index was obtained from the Noise Power Spectrum (NPS) and the Task Transfer Function (TTF), which were measured on a ACR QA phantom with acrylic inserts. The reported results showed that the use of an optimization algorithm (either H/SIR or MBIR) improves d for low-dose acquisition, when compared to FBP. When directly comparing H/SIR and MBIR, the second led to an increase in d (measured at 3 mGy), as well as potential dose reductions for Siemens, GE, and Philips systems, with features sizes. Potential dose reduction using Philips (IMR) reached 62% and 78% for the small and large feature, respectively. Supervised learning-based approaches for implementing model observers have become possible substitutes for numerical observers, particularly through the use of CNN. Li et al. [92] evaluated different CNN-based denoising methods on simulated planar scintigraphy images. The images were simulated for Signal Known Exactly / Background Known Statistically (SKE/BKS) lesion detection, with a Gaussian signal and a lumpy object as random background. Mixed Poisson and Gaussian noise was added to the simulated images, both signal absent and signal present, to produce the final dataset. Three denoising encoder-decoder networks were tested, i.e., a linear CNN (without ReLU activation layers), a non-linear CNN with MSE loss, and a ResNet-based model with a perceptual loss function. Their performance was then assessed using several observers for the detection of the lesion signals, i.e., a Bayesian Ideal Observer (IO) [93,[START_REF] Cormack | Handbook of Image and Video Processing (Second Edition)[END_REF], a CNNbased observer, a Hotelling Observer (HO) [START_REF] Zhang | [END_REF]93], a CHO, a Regularized HO (RHO), and a NPWMF. The authors performed ROC analysis and further computed a detection efficiency measure, given by e ≡ AU C denoised /AU C noisy , observing that, while an increase in network depth improved SSIM and RMSE measures, the performance of the detection task generally dropped, thus suggesting that denoising methods might cause a loss of statistical information in the image. For the CNN-based observer, HO and RHO, the performance decreased with the increase in network depth. It is also argued that using non-task-based loss functions to optimize CNN-based denoising models might play an important part in that loss of information.

The authors in [START_REF] Kopp | [END_REF] tested two antropomorphic model observers to detect liver lesions, one based on softmax regression (SR-MO) and a CNN-MO. A phantom with contrast targets of different diameters was scanned on a CT scanner, at different X-ray exposures, and the images were reconstructed using both FBP, which was used for the test dataset, and iterative reconstruction, used in training and validation. One radiologist provided confidence levels considering the de-tection task for a total of 7488 images. Model performance evaluation relied on computing the JAFROC for the reader study, a CHO with Gabor channels, and the two proposed models, which were trained using two strategies, i.e., separated models for each lesion size, or a common model for all diameters. The PLCC, the χ 2 goodness-of-fit, and the mean absolute percentage difference (MAPD) were then used to compare the models' AUC values. The authors concluded that the CNN-MO can accurately approximate the human performance. This model outperformed the CHO (PLCC = 0.95, MAPD = 2.2%) using the first training strategy (PLCC = 0.98, MAPD = 1.2%), and the SR-MO with both strategies. With the second strategy, the CNN-MO achieved a PLCC of 0.92, and MAPD of 3.0%.

Alnowami et al. [97] aimed at studying the minimum detectable contrast in mammography screenings, using a deep learning-based MO. They first trained a datadriven CNN model to classify image patches, from clinical routine screenings, as normal tissue or containing a lesion. In this SKS detection task, the model achieved a sensitivity of 0.90 and a specificity of 0.92 on the test set. In a second experiment, its performance was compared with two groups of human observers (experts and non-experts), using a four-alternative forced choice setup with simulated images to assess the minimum detectable contrast. Contrast threshold values obtained with the proposed MO approximated the human performance on spherical targets, and even outperformed it for 4mm targets. For lesion targets, human performance was better, but the CNN-MO still achieved a comparable performance (12% difference).

Zhou et al. [96] used deep learning methods to approximate an IO and a HO for binary signal detection tasks, i.e., a CNN and a Single Layer Neural Network (SLNN), respectively. Computer-simulated images were generated using continuous-to-discrete mapping with a Gaussian kernel, and considering four binary signal detection tasks: a Signal Known Exactly / Background Known Exactly (SKE/BKE) task, where the IO and HO are analytically determined, two SKE/BKS tasks, with lumpy background and clustered lumpy background object models, and a Signal Known Statistically / Background Known Statistically (SKS/BKS) task with a lumpy background. Overall, the CNN-IO and SLNN-HO closely approximated the results of the IO and HO, in terms of AUC. In the case of the SKE/BKE task, the obtained AUC with IO and CNN-IO was 0.89, whereas with HO and SLNN-HO it was 0.83. For the SKE/BKS task with lumpy background, the IO was computed using Markov-chain Monte Carlo (MCMC) techniques [104], and achieved an AUC of 0.91. CNN-IO closely approximated this performance,

Localization tasks

An interesting work was published in [98], where the authors proposed a localization-based approach to assess the quality of acquisition of fetal ultrasound scans. First, the authors implemented a localization CNN (L-CNN) using pre-learned AlexNet low-level cues, which identifies a ROI containing the fetal abdomen. A sliding window strategy was used to feed local inputs to the L-CNN and obtain a ROI probability map throughout the entire scan. Image quality was then assessed by considering the quality of depiction of both the stomach bubble (SB) and umbilical vein (UV) within the ROI. These two structures may be labeled as satisfactory, not good or absent. While the first label translates to S SB or S U V =1, the other two led to 0. From the combinations of these two binary scores, a 4-class output was obtained by either human observers or a classification CNN (C-CNN), with knowledge transferred from the L-CNN to its encoding layers. The final quality score of the Fetal Ultrasound Image Quality Assessment (FUIQA) scheme is given by the sum of S SB , S U V and S ROI , which is also a binary score, determined by the ROI to field of view ratio. The authors provided an extensive discussion on the proposed methods, for example comparing the AUC for SB and UV detection with different input layers, based on local phase analysis. The reported quality outputs of the described FUIQA model were highly coherent with those from 3 radiologists, with agreement values of 0.91, 0.89, and 0.88. Zhou et al. [99] extended their proposed supervised learning method [96] to approximate an IO in joint detection and localization tasks. The LROC curves produced by the proposed method were compared with those of traditional observers when computationally feasible. The signal can be localized in nine different locations and the considered signal detection-localization tasks are BKE task, BKS with a lumpy background model and BKS with a clustered lumpy background model. Overall, the CNN-IO was able to come close to the analytical IO for the BKE task, the MCMC-IO for the BKS task with a lumpy background, and outperformed the scanning HO for the BKS task with CLB since MCMCs have not been reported to date for this type of background. Note that approximating an IO using supervised learning requires a large amount of training data, which can be a challenge when only a limited amount of experimental data is available.

Lorente et al.

[100] proposed a MO based on U-Net [105] for defect localization on simulated images, with three levels of correlated noisy backgrounds. Two network configurations (i.e., kernel sizes 3×3 and 5×5) and two loss functions (i.e., MSE and binary crossentropy) were tested, and their accuracy compared with a human observer. Accuracy was described as the ratio between correctly located defects, i.e. within a 5 pixel distance of the actual defect, and the total number of images. The authors concluded that the models trained with a binary cross-entropy loss function provided results closer to the human observer. For example, for the third level of correlated noisy background, the accuracy of the human observer was 0.8, and the 3x3 and 5x5 MO trained with MSE loss yielded an accuracy of 0.92 and 0.91, respectively. The MO trained with binary cross-entropy loss, got accuracy values of 0.89 and 0.85, respectively.

Segmentation tasks

In [101], the authors studied a segmentation-based QA framework for retinal images. Unsupervised vessel segmentation was performed on a dataset of 800 images from the UK Biobank [106], using QUARTZ (Quantitative Analysis of Retinal Vessel Topology and Size) [107]. The algorithm relies on a multi-scale line detector, based on the average gray-level intensities around each target pixel, complemented with high-intensity pixel thresholding, masking of the fovea, and suppression of small objects. Image quality was then measured by extracting three features from the binary segmentation image, selected to mimic manual QA, i.e., area, fragmentation, and complexity, which were finally classified using both a SVM classifier with a radial basis function kernel and an ensemble decision tree classifier. The dataset had been labeled by two observers as adequate or inadequate for epidemiological studies. The SVM achieved the best performance in classification, with an AUC of 0.98. Although method validation is not directly related with the human segmentation performance, the method uses the segmentation task outcome to predict image quality.

The authors in [102] studied the correlation between texture-based muscle segmentation in Dixon MRI and a set of NR quality metrics, i.e., Variance, Laplacian, Gradient, Autocorrelation, Frequency Threshold metric (FTM), Marziliano Blurring metric (MarzBM), HP metric, Kurtosis-based metric, and Riemannian Tensorbased metric (RTBM), to assess the feasibility of using texture segmentation methods as a content-specific quality measure for MRI. The authors implemented a pixel-wise binary segmentation method, using Ad-aBoost [108], with a local texture descriptor, consisting of the histogram of oriented gradients (HOG) [109], 3level Haar Wavelet coefficients, and statistical measures from the original grayscale image and the Laplacian of Gaussian (LoG) filter [110]. The Dice overlap coefficient with manual segmentations was chosen as FOM. Overall, the segmentation output showed reasonable correlation with the variance metric (PLCC = 0.72, SROCC = 0.74) and RTBM (PLCC = 0.71, SROCC = 0.73). Considering only cases with poor segmentation (Dice < 0.7), where all metrics performed better, even though it should be noted that this analysis relied on a smaller number of points. RTBM (PLCC = 0.93, SROCC = 0.86) and FTM (PLCC = 0.84, SROCC = 0.96) yielded the best correlations.

More recently, Alais et al.

[103] proposed a CNN that segments the macular region, which consists of 3×3 convolutional layers only, has very few parameters, and makes decisions considering a threshold t to obtain a binary image. If the obtained area is greater than a value A, then the algorithm considers that the macula is visible, and the image quality is considered sufficient for macula location. The authors extracted 6098 eye fundus images from the e-ophtha database [111], with more than half of the images containing a visible macula. The chosen parameters, t and A, that offer a trade-off between sensitivity and specificity, gave an accuracy of 96.4% on the test set. The proposed CNN wrongly predicted that the macula was visible in four images among 304 images (i.e., 1.3% of false positives), whereas U-Net had nine false positives. Regarding the fovea localization, the authors obtained an average error of 0.95 pixels for their network versus 1.22 pixels for U-Net. The same tests, conducted on the ARIA database [112], revealed a prediction of the fovea with 1.4 pixels of mean error (0.1mm) and 6 pixels of maximum error. Finally, the network was tested on pathological images, with one unsuccessful detection due to the presence of macular hemorrhages or exudates.

Discussion and Future Work

Medical images and video refer to a wide variety of different acquisition methods, clinical applications, and quality issues, as it is clear from the papers reviewed in this paper. Therefore, it is almost impossible to establish meaningful comparisons between the objective methodologies used, i.e., their merits and drawbacks. Nonetheless, some global considerations may be drawn, particularly on future research directions.

Regarding FR and RR visual quality-based metrics, all reviewed papers reported the use of metrics that were originally designed for natural content, or variations of those metrics. PSNR and SSIM were the most common across these studies, and both were used in a vast majority of them [5,[START_REF] Nasr | [END_REF]10,[START_REF] Chow | Alzheimer's Disease Neuroimaging Initiative[END_REF][START_REF] Zhou | International Conference on Neural Networks and Signal Processing[END_REF]21,22,[24][START_REF] Kumar | ternational Conference on Signal Processing Systems[END_REF][START_REF] Kumar | IEEE International Conference on Image Information Processing[END_REF]. [START_REF] Renieblas | [END_REF] also reported the use of SSIM, along with several variations of that metric. Other metrics, such as VIF and NQM were commonly tested as well. The only FR metric specifically designed for medical imaging was proposed by [22], who described a QA metric for cardiac ultrasound videos, based on cardiac motion and structural information. Considering the reviewed NR visual quality-based approaches, most methods are tailored for specific imaging modalities and/or artifacts. From our analysis, it is also clear that deep learning methods are becoming a staple in NR medical image and video QA, as most recent papers used CNN instead of handcrafted features, with some interesting results [52-54, 67, 68, 71, 75-77].

As stated in [11], there seems to be a lack of publicly available databases with subjective quality annotations in the medical imaging domain. Some notable examples are described in [63,[START_REF] Khan | Medical Imaging 2020: Image Perception, Observer Performance, and Technology Assessment[END_REF]113,114]. In order to develop new reliable objective quality metrics for the medical imaging field, more training data is needed. With a view to address this issue, human-in-the-loop machine learning techniques could be considered (e.g., as described by [115]). In this sense, although automated and semi-automated techniques have been proposed for segmentation purposes [101-103], there is still a lack of annotated databases and studies to support the development of reliable methods, for example incorporating models of how clinicians perform diagnosis from images and videos [116]. Artificial intelligence promises strong breakthrough in medical imaging objective QA.

Another common challenge in objective QA studies for medical images and videos has to do with artifact simulation. While some of the reported visual qualitybased studies used databases containing images or video with real artifacts (e.g., [45, 47, 52-54, 56, 71, 76]), others used simulated distortions and/or artifacts over real data assumed to be clean (e.g., [START_REF] Nasr | [END_REF]10,[START_REF] Chow | Alzheimer's Disease Neuroimaging Initiative[END_REF][START_REF] Renieblas | [END_REF]49,51,[START_REF] Köhler | Proceedings of the 26th IEEE international symposium on computer-based medical systems[END_REF][START_REF] Khan | IEEE International Conference on Image Processing (ICIP)[END_REF][START_REF] Outtas | International Symposium on Signal, Image, Video and Communications[END_REF]). Collecting data with real artifacts may not always be possible, or ends up being impractical. On the other hand, simulated artifacts are often generic and limited in range, which might hinder the application of the developed QA methods to real clinical data [117]. Recently, some efforts towards the simulation of content-specific and realistic artifacts, to be applied in healthcare QA research, have been reported (e.g., [117][118][119][120]), and future work will likely approach this question more often, leveraging on the continuous development of deep-learning methods, such as Generative Adversarial Networks (GAN).

A similar question can be raised regarding taskbased QA research. Traditional model observers are based on the (total or partial) knowledge of statistical characteristics of the images (i.e., signal and background). Hence, many studies (e.g., [87,89,90,96,99,100,121]) are based on simulated or phantom images, since real medical images suffer from a lack of statistical information. Currently, there is no evidence that QA studies conducted on simulated images ensure sufficient confidence to draw relevant conclusions on real images. Deep learning methods could allow to go through this weakness. Indeed, any task that could be carried out with deep learning provides an assessment of the quality. In fact, the more efficiently the task is done, the better the quality. There is still room for improvement in the field, the whole purpose being to define the most relevant tasks, i.e., those that can be reliably delegated to a model. In task-based QA, the key problem lies in modeling the task performed by human observers. So far, to our knowledge, existing models are limited in task range. For example, no model observer has been proposed for characterization tasks, which focus on analyzing certain properties of abnormalities (e.g., contour or texture) for differential diagnosis, and generally involve a linguistic response (e.g., benign vs. malignant), given its high complexity [START_REF] Zhang | [END_REF]. Other tasks could also be further explored in future work, such as estimation tasks (or joint detection / classification / estimation tasks), which aim at determining a scalar value or range of values from a given object to be used in diagnosis (e.g., tumor diameter or radiotracer uptake [14]).

Although 3D visualization of medical images and videos is emerging, research on relevant quality assessment aspects is still behind. Stereoscopic medical imaging and, more recently, light field medical imaging, open new opportunities, for instance in surgery training, also at a distance [122]. Compression [123] and transmission [122] of 3D stereoscopic medical images and videos, as well as of light field medical data, require suitable metrics for the assessment of their performance. Recently, studies on quality assessment for light field medical images have started (e.g, [124]). However, objective metrics in this domain are still missing, or their development is still ongoing. Future research might focus on assessing whether existing metrics, developed for generic 3D images and videos (e.g., [125] ), are suitable to assess the quality of medical data represented in these formats. While efforts are ongoing in this direction, the availability of wider datasets of med-ical data in stereoscopic 3D and light field 3D formats would definitely be useful towards this effort.

Several of the reported studies considered coding distortions [5, 9, 10, 16, 17, 19-27, 46, 48, 80]. There is a number of applications, mostly based on telemedicine applications, where lossy compression of medical images or videos might be acceptable. However, in most cases, medical imaging applications cannot rely on images with lossy compression, as no one can be sure of the influence that those losses can have in a diagnosis. Multiple times, radiologists use almost imperceptible textures to define their diagnosis and, in such cases, lossy compression can have a major impact. Hence, several studies on medical image quality are quite questionable, as they use medical modalities where no radiologist would accept any kind of lossy coding.

Conclusion

In this article, we presented an extensive review of the literature on the objective quality assessment of medical images and video. More precisely, we covered both visual quality-based and task-based methods, considering various imaging modalities and application purposes. Furthermore, we aimed at covering a wide range of approaches to the quality assessment of medical images and videos content, including the application of preexisting metrics for natural images and the development of content-specific metrics, either based on handcrafted features or using deep learning-based models.

  , [126], [127], [128]), and light field data (e.g., [129], [130], [131], [132]

Table 1

 1 Overview of visual quality-based approaches to the quality assessment of medical images and video (FR metrics). In some works, NR metrics were also tested, which are marked with * .

	Reference	Imaging modality	Quality impairments / Processing	Validation	Objective metrics
	[16]	MRI	Rician and Gaussian noise, and Gaussian blur (5 levels); DCT, JPEG and JPEG2000 compression (5 ratios)	MOS: 4 medical observers (SDSCE)	SNR, PSNR, SSIM, MSSIM, FSIM, ICF, NQM, WSNR, VIF, VIFP, UQI, IW-PSNR, IW-SSIM
	[10]	MRI	White and Rician noise, Gaussian blur, motion, undersampling and wavelet compression	MOS: 5 radiologists	RMSE, PSNR, SSIM, MS-SSIM, IW-SSIM, GMSD, FSIM, HDRVDP, NQM, VIF
			Compression with DCT	MOS: Common
	[17]	CT	and Wavelet using different compression	persons, clinic doctors, and	PSNR, SSIM, distortion measure from [18].
			ratios	radiologists
	[19]	CT	JPEG and JPEG2000 compression (5 ratios)	MOS: 6 radiologists (Double-stimulus DCR)	MSE, local MSE, SNR, SSIM, VSNR, VIF
			H.264 Compression with
	[20]	Ultrasound videos	different quantization parameters and	MOS: 2 medical experts	PSNR, SSIM, VSNR, VIF, VIFP, IFC, NQM, WSNR
			packet-loss rates
	[21]	Ultrasound videos	HEVC compression (8 quantization levels)	MOS: 4 medical experts and 16 naïve observers (DSCQS)	PSNR, SSIM, UQI, VQM, NQM, VIF, VSNR
	[22]	Ultrasound videos	HEVC compression (8 quantization levels)	DMOS: 4 medical experts (DSCQS)	PSNR, SSIM, UQI, VQM, NQM, VIF, VSNR, CUQI (proposed metric)
	[23]	Laparoscopic surgery videos	H.264 compression (4 bit rates)	MOS: 9 laparoscopic observers (SSCQE) surgeons and 16 naïve	VQM, HDR-VDP-2, PSNR
					SSIM, UQI, PSNR, WSNR,
	[5]	Endoscopic surgery videos	H.264 compression (11 ratios)	MOS: 14 medical observers (DSCQS)	VSNR, HDR-VDP, IFC, MSE, MS-SSIM, PSNR-HVS, PSNR-HVS-M, VIF, VIFP,
					NIQE * and BRISQUE *
	[24]	Endoscopic videos	HEVC compression (8 quantization levels)	MOS: 6 medical observers and 19 naïve observers (DSCQS)	MSE, PSNR, SSIM, MS-SSIM, VSNR, IFC, VIF, VIFP, UQI, NQM, WSNR
	[25]	CT, MRI	Varying bitrates (from 0.1 to 2.0) using SPIHT compression	MOS: 6 medical observers	PSNR, SSIM
	[26]	CT, MRI, Ultrasound	Varying bitrates (from 0.02 to 2.0) using SPIHT compression	MOS: 5 medical observers	PSNR, SSIM, proposed SSIM variation
	[27]	X-ray, MRI	Gaussian noise and Gaussian blur (5 levels); JPEG and JPEG2000 compression (5 ratios)	MOS: 3 radiologists (Double-stimulus DCR)	16 combinations of SSIM components (multiscale, gradient-based and structural component)

Table 2

 2 Overview of visual quality-based approaches to the quality assessment of medical images and video (RR metrics).

	Reference	Imaging modality	Quality impairments / Processing	Validation	Objective metrics
	[44]	Retinal fundus photography	Reduction of brightness, contrast, and SNR	Comparison with SNR and agreement with human perception	Similarity metric based on intensity histograms (RR)
	[45]	Retinal fundus photography	Image acquisition impairments (out of focus, poor illumination, etc.)	Qualitative assessment: 1 ophthalmologist	Similarity metric based on the distribution of edge magnitudes and the local intensity distribution (RR)
	[46]	CT, MRI	Compression with DCT and Least Significant Bit (LSB) techniques and ratios different compression	Objective measures of classification and pixels percentage of damages	Structural comparison metric (RR).
	[9]	Ultrasound videos	Gaussian noise, JPEG2000 compression (9 compression (8 QP) ratios), HEVC	DMOS: 4 medical observers (DSCQS) experts, 16 naive	PSNR, SSIM, MSSIM

BRISQUE, the original counterpart, and a JPEG-based model were tested on the first dataset, and the correlation with MOS was obtained with the proposed metric (PLCC of 0.96 and SROCC of 0.93). The proposed BRISQUE also outperformed its original version, in terms of correlation with two FR metrics -NQM and Feature Similarity Index (FSIM) -, for the majority of the simulated distortions in the second dataset.

Table 3

 3 Overview of visual quality-based approaches to the quality assessment of medical images and video (NR metrics I).

	Reference	Imaging modality	Quality impairments / Processing	Validation	Objective metrics
	[47]	MRI	Different imaging sequences, contrast weights and subsampling strategies	5-level qualitative scale: 5 observers	Contrast, resolution, texture and intensity-based features with SVM classifier
			Rician and Gaussian
	[48]	MRI	noise, and Gaussian blur (5 levels); DCT, JPEG and JPEG2000	MOS: 10 radiologists; FR metrics: NQM and FSIM	BRISQUE (modified and original), JPEG-based model
			compression (5 ratios)
	[49]	MRI	Bias fields, motion blur, Rician noise (20 levels)	MOS: 4 radiologists MRI reader and 1 experienced	Entropy posterior probability
					ENMIQA (proposed
					entropy-based metric), BPRI,
	[50]	MRI	Not specified	MOS: 31 radiologists	dipIQ, IL-NIQE, MEON, Q index, S index, QENI,
					SNR-ROI, SSEQ and
					SISBLIM
					Whole image and ROI-based
					features (e.g. SNR, CNR,
	[51]	MRI	Gaussian noise, Gaussian blurring, contrast manipulation	MOS: 3 neuroradiologists	uniformity, Wang index, Shannon entropy) with different classifiers (LDA,
					QDA, SVM, MLP, Logistic
					Regression)
			Not specified (motion
	[52]	MRI	artifacts and inhomogeneous fat supression mentioned in	Binary quality label: 2 radiologists	AlexNet
			some cases)
				Qualitative binary
	[53]	MRI	Motion artifacts	score: 2 expert	6-layer CNN
				observers
				Qualitative binary
	[54]	MRI	Motion artifacts	score and 3-level	4-layer CNN and ResNet-10
				scales: 2 radiologists
	[55]	Retinal fundus photography	Gaussian blurring (2 levels) and Gaussian noise (20 levels)	FR metrics: PSNR and SSIM; MOS (no. of observers not given)	Local gradient-based metrics (normal and vessel-guided), CPBD and anisotropy
	[56]	Retinal fundus photography	Uneven illumination, blurring, low contrast and color distortion	Composite binary scoring: 3 ophthalmologists	HVS-based feature matrix (multiple channel sensation, JNB, contrast sensitivity function)
					Texture features from RGB
					and CIELab channels
	[57]	Retinal fundus photography	Not specified	MOS (no. of observers not given)	(Butterworth, Gabor and Wavelet filters, Gaussian
					Markov random fields,
					GLCM)

Table 4

 4 Overview of visual quality-based approaches to the quality assessment of medical images and video (NR metrics II).

	Reference	Imaging modality	Quality impairments / Processing	Validation	Objective metrics
				MOS: 3
				ophthalmology
	[67]	Retinal fundus photography	Not specified	experts (cross-validation set)	InceptionV3
				+ 6 ophthalmology
				experts (test set)
	[68]	Retinal fundus photography	Not specified	Qualitative 3-level scale from feature-based MOS: 15 ophthalmologists	Multivariate regression-based CNN, with feature extraction using InceptionV3, DenseNet-121, ResNet-101, and Xception
	[71]	Retinal fundus photography	Artifacts (e.g. shadows, lens stains), image clarity and field definition	Binary quality label and factor-specific qualitative scores: 3 ophthalmologists	CNN model with global and local feature extraction and ADDA, DeepIQA, DRIQC, MEON, MFIQA
				Qualitative 3-level	DDDT-CWT based LBP
	[74]	AS-OCT	Not specified	scale: 2 medical	features with a naïve Bayes
				experts	classifier
	[75]	Laparoscopic surgery videos	Defocus and motion blur, Gaussian noise, uneven illumination, smoke	Labels inferred from simulated distortions	ResNet (18, 34, 50 layers), RankIQA
	[76]	Endoscopy video	Blur, bubbles, contrast, specularity, saturation, misc. artifacts	MOS: 3 radiologists	YOLOv3 with DeepLabv3+ spatial pyramid pooling
	[77]	Ultrasound (Echocardiogram)	Not specified	MES (6-level scale): 1 cardiologist	PSO optimized CNN model: 3 convolutional layers
			8 image fusion algorithms:		Pooling of phase congruency
		Fused images	NNM, LP-SR, CSCS,		and standard deviation
	[78]	(MRI, CT, US,	GFF, NSCT-PCNN-SF,	MOS: 20 radiologists	(proposed), Gradient-based
		SPECT, PET)	ISML, CSR, and		and Structure-based metrics,
			DTM-PCNN		Edge information, RSFE, MI
					PCNN in NSCT (proposed),
	[79]	Fused images (MRI, CT, US, SPECT, PET)	8 image fusion algorithms [78]	MOS: 20 radiologists	baseline metrics of [78], Phase congruency, Entropy, OSSI, Tsallis entropy-based metric,
					BMPRI, MEON
	[80]	MRI, Ultrasound	Noise artifacts, median filtering, JPEG 2000 (3 ratios)	Direct comparison of metrics; JAFROC: 5 radiologists (MRI)	NIQE, BiQES, NIQE-K (proposed modified NIQE)

Table 5

 5 Overview of task-based approaches to the quality assessment of medical images and video.

	Reference	Imaging modality	Quality issue	Model	Task	Figure of merit
	[87]	CT phantom	Reconstruction: IMR, FPB/ low dose radiation	CHO	Detection	Detectability (d')
	[89]	CT phantom	Reconstruction: FBP, ASIR, MBIR/ low dose radiation	CHO (D-DOG)	Detection	Detectability (d')
	[95]	CT phantom	Detection of lesions in CT with different X-ray exposures	SR-MO, CNN-MO, CHO (Gabor)	Detection	ROC / AUC
			Reconstruction: Asir,
			Asir-V, IMR, iDose
	[90]	CT phantom	SAFIRE, ADMIRE, AIDR3D, First, FPB,	NPWMF (imQuest tool)	Detection	Detectability (d')
			H/SIR and MBIR / low
			dose radiation
			Approximation of the IO
		Computer-	and the HO by supervised
	[96]	simulated	learning methods to	CNN-IO, SLNN-IO	Detection	ROC / AUC
		images	evaluate image quality for
			detection tasks
	[97]	Mammography	Study the minimum detectable contrast in mammography scans	CNN-MO	Detection	Contrast threshold
		Simulated
	[92]	Planar Scintigraphy	Comparison of CNN-based denoising methods	IO, CNN-MO, HO, RHO, NPWMF	Detection	ROC / AUC
		images
	[98]	Ultrasound	Quality assessment of acquisition of fetal ultrasound scans	Custom CNN-based model	Localization	Multiple structure visibility
			Approximation of the IO
		Computer-	and the HO by supervised	Analytical IO,
	[99]	simulated	learning methods to	Scanning HO,	Localization	LROC
		images	evaluate image quality for	MCMC-IO, CNN-IO
			localization tasks
		Computer-	Approximation of a human
	[100]	simulated	observer in defect	U-Net	Localization	Accuracy
		images	localization tasks
	[101]	Retinal fundus photography	Quality assessment of acquisition of retinal images	Measures from classifier segmented areas with SVM or decision tree	Segmentation	ROC / AUC
	[102]	MRI	Compare the performance of the segmentation model with several no-reference quality metrics	Texture-based features with AdaBoost classifier	Segmentation	Dice overlap coefficient
	[103]	Retinal fundus photography	Image quality in macular region	Custom CNN, U-Net	Segmentation, Localization	Macula visibility
	with AUC=0.91. The traditional HO and SLNN-HO	not be compared to the IO, with the authors arguing
	yielded an AUC of 0.81. A similar outcome was ob-	that MCMC application to clustered lumpy background
	served in the SKE/BKS task with clustered lumpy	object models had not been reported to date. Finally,
	background for HO and SLNN-HO (AUC=0.85). CNN-	for the SKS/BKS task, the performance of both the
	IO achieved an AUC of 0.89, but its performance could	traditional HO and SLNN-HO are close to a random

Acknowledgements

R. Rodrigues and A. Pinheiro acknowledge Fundação para a Ciência e a Tecnologia (FCT/MCTES) for funding this research, under the doctoral grant SFRH/BD/130858/2017, and the project UIDB/50008/2020, respectively. They would also like to acknowledge the project CENTRO-01-0145-FEDER-000019 -C4 Cloud Computing Competence Centre. The work of J. Gutiérrez was partially supported by the Juan de la Cierva fellowship (IJC2018-037816).


domain: spi chim phys

text:
est destinée au dépôt

Principle of Ohmic Heating

The product is placed in an isolated treatment cell closed by two electrodes connected to an electricity generator. An alternative electric current circulates through the product which acts like an electrical resistance. The whole volume of the product is thus heated up according to Ohm's law. The homogeneity of the thermal treatment is achieved by homogenizing:

-the electric field parallel electrodes -the velocity field batch configuration -the product liquid products

Reactor conception

CONCLUSION and PERSPECTIVES

The ohmic reactor allows reaching a large range of temperatures -ambient to 140°C -during a wide range of times -5 seconds to several minutes -thus covering pasteurization to UHT sterilization conditions. The thermal history of the product has been proven to have a good reproducibility. The sampling device combined to a very efficient cooling should allow producing reliable experimental data for studying the kinetics of chemical reaction under HTST conditions.

THERMAL TREATMENTS

Thermal treatment in 3 steps

Holding step with sampling

The heating kinetic can be adjusted by an exponential equation with a relative standard deviation of ± 3 % for a and b, for given initial conductivity and electrical power inlet and independently of the initial and final temperatures.

Cooling step

The holding step is also very reproducible. The results presented for each temperature level are average of 3 to 20 trials. Sampling causes temperature artifacts which appear to be non significant as relative standard deviations do not exceed 1 %.

The cooling step is very fast and is independent on initial temperature: it takes about 1 second for all samples to cool down to 100°C and several minutes to decrease to 20 °C. stephanie.roux@univ-montp2.fr ; mathilde.courel@agroparistech.fr ; jppain@polytech.univ-montp2.fr

  Conception of an ohmic heating reactor to study the kinetics of chemical reactions at high temperature in liquid food products Stéphanie Roux, Mathilde Courel, Jean-Pierre Pain To cite this version: Stéphanie Roux, Mathilde Courel, Jean-Pierre Pain. Conception of an ohmic heating reactor to study the kinetics of chemical reactions at high temperature in liquid food products. First European Food Congress: Food Production, Nutrition, Healthy Consumers, Nov 2008, Lubljana, Slovenia. EFFoST 2008. mnhn-03922902v2 A BATCH OHMIC REACTOR Objective Development of a tool to study the kinetics of chemical reactions in high temperature short time (HTST) thermal treatments of liquid food products Specifications -Reach sterilization temperatures above 100 °C without boiling by maintaining pressure -Produce reliable kinetic data during very short treatment times -Guarantee a thorough control of the temperature because of strong thermal dependence of the chemical reactions

-

  Treatment cell: 100 mL ; ULTEM -Electrodes: DSA coated titanium -Tension generator: 50 Hz; 0 -320 V; 0 -20 A -Thermocouples: type K -Labview program for recordings -Thermal regulation: simulator control and data acquisition -5 sampling valves of 15 mL each PRELIMINARY MEASUREMENTS Electrical conductivity of the sample Electrical conductivity σ (Siemens/m) is a key variable in Ohm's law, which determines the aptitude of the product for ohmic heating: σ : electrical conductivity (S/m) e : distance between the electrodes (m) S : electrode surface (m²) I : current intensity (A) U : electrical tension (V) The conductivity can be measured experimentally by heating the product in the ohmic prototype and recording temperature, tension and intensity 0.05 < σ <100 S/m is optimum for ohmic heating σ < 0.05 S/m and σ > 100 S/m is not appropriate for ohmic heatingTemperature homogeneity in the reactorThe thermal homogeneity of the reactor has been checked by recording the temperature of 5 thermocouples placed at different distances from the center of the reactor.A dispersion of maximum 2 % around the average value of the targeted temperature shows a good thermal homogeneity of the reactor I

  1. Heating : from 50 °C to sterilization T (100 to 140 °C) 2. Holding : sterilization temperature is maintained 3. Cooling : cooling from sterilization T to 25 °C Sampling Example: ohmic treatment of a semi-skimmed milk at 140°C with five sampling Schematic representation of the ohmic heating reactor REPRODUCIBILITY Heating step

  Conception of an ohmic heating reactor to study the kinetics of chemical reactions at high temperature in liquid food products Stéphanie ROUX a , Mathilde COUREL b , Jean-Pierre PAIN a a UMR Qualisud, Université Montpellier II, Place Eugène Bataillon, 34095 Montpellier, France b UMR 1145 Génie Industriel Alimentaire (AgroParistech, Cemagref, INRA), 1 av. des Olympiades, 91744 Massy, France

samples can be taken during one thermal treatment, during heating or holding steps. Sampling time can be very short: 5 seconds.Sampling may produce an artifact: in some cases an increase in temperature can be observed; it stays very brief and the impact on the whole thermal treatment can be neglected.