Member, IEEE Ali Ak

Emin Zerman

Maurice Quach

Senior Member, IEEE Aladine Chetouani

Aljosa Smolic

Senior Member, IEEE Giuseppe Valenzise

Fellow, IEEE Patrick Le Callet

BASICS: Broad Quality Assessment of Static Point Clouds in a Compression Scenario

Keywords: Point cloud quality, 3D models, point cloud compression, subjective quality assessment, dataset

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

I. INTRODUCTION

D IGITAL imaging technologies have revolutionized the ca- pability to capture real-world environments and recreate them in different temporal or spatial contexts. This capacity has extended to the realm of 3D scenes through the integration of computer graphics and photogrammetry techniques. Presently, we can capture intricate 3D objects and scenes using an array of tools, ranging from solely RGB cameras to RGB cameras supplemented with additional sensors [START_REF] Collet | High-quality streamable freeviewpoint video[END_REF]- [START_REF] Rosnell | Point cloud generation from aerial image data acquired by a quadrocopter type micro unmanned aerial vehicle and a digital still camera[END_REF]. Within the domain of 3D modeling, two primary representations have gained prominence: colored point clouds and textured 3D meshes [START_REF] Zerman | Textured mesh vs coloured point cloud: A subjective study for volumetric video compression[END_REF]. In this paper, our focus is directed toward point clouds, a representation widely used in numerous applications, in particular augmented and virtual reality. The acquisition of point clouds is typically accomplished through diverse means, including stereo-camera arrays [START_REF] Schreer | Capture and 3D video processing of volumetric video[END_REF], LiDAR sensors [START_REF] Roynard | Paris-Lille-3D: A large and high-quality ground-truth urban point cloud dataset for automatic segmentation and classification[END_REF] and conventional cameras [START_REF] Rosnell | Point cloud generation from aerial image data acquired by a quadrocopter type micro unmanned aerial vehicle and a digital still camera[END_REF]. Point cloud acquisition produces huge amounts of data, calling for compression techniques for efficient transmission and storage.

However, evaluating the performances of compression algorithms is time consuming and expensive. While the field of research in this area has witnessed notable expansion in recent years [START_REF] Zerman | Textured mesh vs coloured point cloud: A subjective study for volumetric video compression[END_REF]- [START_REF] Alexiou | PointXR: A toolbox for visualization and subjective evaluation of point clouds in virtual reality[END_REF], there are still many open questions and problems.

Existing datasets in the field have notable shortcomings in terms of diversity, scale, consistency, and data accessibility. These limitations constitute substantial challenges for research into learning-based approaches. The absence of diversity is frequently attributed to several factors, including the recurrent use of the same point clouds across various datasets, limited geometric complexity and semantic categories, as well as the utilization of the same compression algorithms as the sole source of distortion types. This lack of diversity is further compounded by the relatively modest scale of the existing datasets, rendering them ill-suited for the development of learning-based quality metrics. Consistency issues also can be observed in existing datasets, stemming from factors such as the utilization of unnormalized point clouds and inconsistent rendering specifications. A detailed overview of the existing datasets is presented in Table I, which summarizes the aforementioned deficiencies. Another substantial shortcoming is the limited data availability and incompleteness, e.g., due to copyright constraints or repository management practices, as well as the absence of individual opinion scores, standard deviations and confidence intervals, which are typically needed to develop more sophisticated metrics. The accessibility of data in existing datasets is summarized in Table II.

In conclusion, the need for a new dataset is evident due to aforementioned deficiencies in existing datasets. Our proposal seeks to address these shortcomings comprehensively, providing a dataset that encompasses all essential characteristics required for the development of more accurate quality metrics. Furthermore, it provides further insights into the performance of point cloud quality metrics when applied to point clouds encoded with learning-based compression algorithms.

The contributions of this work are threefold:

Dataset

Point Clouds Subjective Annotations BASICS (proposed) ✓ Individual Scores vsenseVVDB2 [START_REF] Zerman | Textured mesh vs coloured point cloud: A subjective study for volumetric video compression[END_REF] ✓ Individual Scores Perry et al. [START_REF] Perry | Quality evaluation of static point clouds encoded using MPEG codecs[END_REF] Broken URL ✗ Su et al. [START_REF] Su | Perceptual quality assessment of 3D point clouds[END_REF] ✓ (D)MOS only da Silva Cruz et al. [START_REF] Da Silva Cruz | Point cloud quality evaluation: Towards a definition for test conditions[END_REF] ✗ ✗ Yang et al. [START_REF] Yang | Predicting the perceptual quality of point cloud: A 3D-to-2D projection-based exploration[END_REF] ✓ (D)MOS only NBU-PCD1.0 [START_REF] Hua | VQA-CPC: a novel visual quality assessment metric of color point clouds[END_REF] ✗ ✗ SIAT-PCQD [START_REF] Wu | SIAT-PCQD: Subjective point cloud quality database with 6DoF headmounted display[END_REF] Broken URL ✗ Subramanyam et al. [START_REF] Subramanyam | Comparing the quality of highly realistic digital humans in 3DoF and 6DoF: A volumetric video case study[END_REF] ✗ Individual Scores PointXR [START_REF] Alexiou | PointXR: A toolbox for visualization and subjective evaluation of point clouds in virtual reality[END_REF] ✓ Individual Scores

• We compare the performances of various state-of-theart methods for point cloud compression (as outlined in Section IV). • We provide an exhaustive benchmark of the state-of-theart point cloud quality metrics, including both point-based and image-based assessment (as outlined in Section V). The BASICS dataset has been made publicly available 1 under the Creative Commons Attribution-NonCommercial-ShareAlike (CC BY-NC-SA 4.0) license with the aim of fostering continued research within the field. The repository contains individual and mean opinion scores, as well as pristine and compressed point clouds. Additionally, the dataset's GitHub page 2 provides the scripts required for the evaluation of point cloud quality metrics. The BASICS dataset has been successfully employed in the grand challenge on point cloud quality assessment 3 organized by some of the authors at the 2023 IEEE International Conference on Image Processing (ICIP2023).

II. THE BASICS DATASET

The BASICS dataset has been designed and collected with two main objectives in mind: on one hand, providing diverse and extensive data to train learning-based point cloud quality assessment metrics; and on the other hand, creating challenging test conditions to benchmark existing quality metrics. In this section, we describe the various stages of the dataset generation process.

A. Material selection

In order to construct a point cloud quality assessment dataset that encompasses semantic diversity tailored for telepresence applications, we gathered a total of 75 point clouds, distributed across three semantic categories. Despite the broad scope of these categories, they remain highly relevant within the context of telepresence applications. The categories we have defined are "Humans & Animals (HA)", "Inanimate Objects (IA)", "Buildings & Landscapes (BL)", collectively covering a comprehensive set of subjects such as animals, humans, everyday items, vehicles, architectural structures and natural landscapes. Figure 1 offers a visual glimpse into each of these categories.

For the data collection and material selection part, our primary objective was to collect a comprehensive repository of publicly available point clouds that could be freely redistributed. However, numerous data sources impose strict restrictions on redistribution. Consequently, we acquired the 3D models from two sources: collaborator studios (i.e., V-SENSE studio 4 and XD Productions5 ) as well as an online repository for 3D model sharing called SketchFab 6 . Even in this case, availability of point clouds was somewhat limited. To address this, we gathered 3D meshes and generated point clouds via sampling the mesh surface, as detailed in Section II-B.

In total, 104 models were handpicked by three authors of this paper considering the semantic categories described above. Following the removal of materials that exhibited significant similarity, models with highly reflective surfaces and imperfect texturing, as well as those with loose semantic associations to their respective categories, the total count of models was reduced to a final selection of 75. For each model, information regarding its source (a SketchFab URL if applicable), file format, statistics, and licensing details can be accessed in the dataset repository 1 .

B. Pre-processing

In order to mitigate potential distortions stemming from other sources, the collected 3D models require a preprocessing and a conversion into point clouds before any fur- ther processing. This was necessary due to the heterogeneous formats in which these models were originally available. The scripts used for pre-processing are provided as a toolbox and available on Github 7 .

Models that were already in point cloud PLY format required minimal attention, except for voxelization. Conversely, 3D meshes underwent through several steps. These steps are further discussed below.

1) Making 3D meshes uniform: The collected 3D mesh models were in various formats. All 3D meshes were converted into OBJ format, to streamline the pre-processing chain, using Blender8 and Meshlab 9 .

2) Cleaning 3D meshes: Some of the 3D meshes had either parts that had transparent or reflective properties (e.g., glasses in some models), which could not be replicated well during point cloud rendering. Some other meshes had incomplete parts (e.g., trees, some of the building façades) which would decrease the users' quality of experience and introduce other sources of distraction and distortion. To avoid such effects, these parts were removed or cleaned in Blender.

The mesh files were then unified into a single OBJ file, so that the sampling process in the pipeline could be done with ease. Next, the material properties (which are described in the .mtl files) are checked to eliminate any other reflective properties of the materials, which could not be reproduced correctly in the point cloud format. After all these operations, the 3D meshes were ready for the point cloud sampling step.

3) Sampling point clouds: Using CloudCompare10 , point clouds were sampled from the 3D meshes' surfaces. During this operation 10 million points were sampled on the surfaces of the said meshes. The sampling operation extracted the location, color, and normal attributes for each point in the point clouds. At the end of this stage, all 3D models were in (originally or after conversion) point cloud format.

4) Point cloud voxelization: We used 10-bit quantization for point cloud voxelization, as higher bit-depths showed no notable benefits. That is, the spatial coordinates are normalized such that they are integers between 0 and 1023. Only unique points remained after voxelization step. This has two main advantages: first, the coordinates are in a range that is predictable for point cloud processing but also with respect to rendering and second, we use voxelized coordinates in combination with cube based rendering to improve stability, predictability and quality of renderings.

C. Compression

Compression is a crucial stage for various point cloud storage and transmission applications, including streaming and telepresence. It is also the most realistic distortion source for point clouds. In this context, one needs to compress point clouds to transfer them to the remote receiver under current bandwidth limitations.

We selected four compression algorithms:

• GPCC-Octree-Predlift (noted as GPCC-Predlift),

• GPCC-Octree-RAHT (noted as GPCC-Raht),

• VPCC, and • GeoCNN. VPCC and GPCC were selected as they are part of MPEG standardization efforts, and they are among the most commonly used compression methods. GeoCNN was selected to introduce artifacts of a learning-based compression to BASICS database. Sample results are shown in Figure 2.

GPCC [START_REF] Graziosi | An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC)[END_REF] compresses the point cloud using the octree structure which can find the occupancy of the points in 3D space without projection to 2D space. GPCC uses either octree or trisoup (Triangle soup) approaches -based on a pruned octree. As octree and trisoup only focus on geometry compression, attributes (such as color) are compressed using region- adaptive hierarchical transform (RAHT) and predicting/lifting transform (indicated as pred/lift or predlift).

For GPCC, it was noticed during the pilot tests that the trisoup generated uneven structure and holes on the reconstructed point clouds. Therefore, in this database, only octree is used for GPCC geometry coding, together with the two attribute coding methods: Octree-Predlift and Octree-Raht. For compression, the six quality parameter (QP) values from common test conditions (CTC) [START_REF] Schwarz | Common test conditions for point cloud compression[END_REF] were used. The worst quality level (i.e., r1) was discarded in the pilot tests because its quality was too bad and this would affect the subjective quality experiment adversely by changing the rating scale and the participants' votes. With this change, the number of quality levels for GPCC became five: r GP CC ∈ {2, 3, 4, 5, 6}.

VPCC [START_REF] Graziosi | An overview of ongoing point cloud compression standardization activities: video-based (V-PCC) and geometry-based (G-PCC)[END_REF] is the video-based point cloud compression approach, in which the point cloud is projected to the sides of a cube and the projection is coded using traditional video compression methods, such as HEVC/H.265 or VVC/H.266. The projection is done for both color information and also the depth information (i.e., the inherent 3D structure of points in 3D). Utilizing the inherent temporal coding capabilities of traditional video codecs, VPCC can effectively compress dynamic point cloud sequences.

For VPCC, we used the "all-intra" mode, and the compression levels were taken from the "longdress" sequence QP as determined in the CTC. It should be noted that the compression levels were not optimized for each content, as this was out of scope of this study.

Nevertheless, it was noticed that the given bitrates in the CTC seemed to yield a much higher quality than GPCC, therefore, another quality level below the ones given in CTC was added. This quality level is called rate_zero with gQP = 36 and tQP = 47. With this change, the number of quality levels for VPCC became six: r V P CC ∈ {0, 1, 2, 3, 4, 5}.

GeoCNN [START_REF] Quach | Improved deep point cloud geometry compression[END_REF] compresses voxelized point clouds by first performing block partitioning. Then, each block is passed to a variational autoencoder where the encoder transforms the input binary occupancy voxel grid to a latent space. The latent space is then quantized and entropy coded using a learned entropy model. After entropy decoding the bitstream, the latent space is transformed back to a voxel grid containing predicted occupancy probabilities. The probabilities are then thresholded to binary values which yields the decoded block. With the result of each block, the entire decompressed point cloud is obtained. Four different quality levels were used for GeoCNN. r GeoCN N ∈ {1, 2, 3, 4}.

III. SUBJECTIVE QUALITY ASSESSMENT

We conducted a large-scale subjective experiment using the Prolific [START_REF]Prolific[END_REF] crowdsourcing platform with 3654 participants. 1494 processed point clouds (PPC) from 75 original point clouds (SRC) were generated for the experiment with each PPC were evaluated by 60 unique participants on average. Consequently, we successfully accumulated approximately 90,000 subjective opinion scores. This section describes the details regarding the crowdsourcing study.

A. Generating Visual Stimuli

In a voxelized point cloud, a voxel is occupied if at least one point of the point cloud is within the voxel. Each voxel is rendered as a cube spanning its volume. This is different from the more common point-based method ("point" as OpenGL point primitive) where points are rendered as screen-aligned squares of a given window space size. The main drawback of the point-based rendering is its susceptibility to scaling issues: as the viewer zooms in or out, the points can either become smaller or larger, impacting the perceived density of the point cloud. Furthermore, point-based rendering often results in flicker artifacts, particularly when spatial overlaps occur during perspective changes. Cube-based rendering addresses these issues, ensuring consistent rendering quality across various perspectives and effectively eliminating flicker artifacts.

In practice, we specify cube sizes. For octree-based methods, the size of the cube is defined based on the number of removed octree levels n r . With n l bit quantization, the maximum is n l levels. Thus, we specify the size of the cube as 2 n l -nr : that is, a size of 1 when lossless, a size of 2 when removing one octree level, a size of 4 when removing two octree levels, etc. . . For other methods, the cube size is determined empirically to ensure water-tight rendering.

We employed a helix-like rendering trajectory, as illustrated in Figure 3. The frontal orientation of each point cloud was manually designated, and the rendering trajectory consistently started from this predefined front side. To ensure that the front side of the point cloud remained visible either at the beginning or at the end of the rendered video, we introduced a slight overlap between the trajectory's starting and ending points. Certain point clouds are unnatural to observe from lower angles (e.g., landscape, buildings). Therefore, each point cloud was individually categorized as either "low", "mid" or "high" to determine the initial elevation of the rendering trajectory. While moving on the rendering trajectory, the camera was always oriented toward the center of the point cloud.

B. Methodology

Subjective quality assessment of point cloud content can be broadly categorized as interactive and passive [START_REF] Alexiou | Subjective and objective quality assessment for volumetric video[END_REF]. In the interactive paradigm, observers have the liberty to examine the point cloud from any desired point of view, often within the context of augmented reality or virtual reality applications. In contrast, the passive approach involves rendering point clouds in the form of a conventional video with a predetermined camera trajectory. Although both paradigms have their own advantages and drawbacks, there is no statistically significant difference between the subjective opinions collected with each approach [START_REF] Viola | On the impact of vr assessment on the quality of experience of highly realistic digital humans: A volumetric video case study[END_REF]. In order to minimize the variance between observer opinions and facilitate a more practical data collection through crowdsourcing, we opted for the passive approach [START_REF] Nehmé | Exploring crowdsourcing for subjective quality assessment of 3D graphics[END_REF].

Several methodologies can be found in the literature and recommendations for subjective quality assessment of traditional image and video sequences [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF]. Commonly employed methodologies include, but are not limited to, Absolute Category Rating (ACR), Double Stimulus Impairment Scale (DSIS), Two-Alternative Forced Choice (2AFC). Several studies compared the accuracy and reliability of each methodology for diverse types of multimedia content. In the realm of traditional images and videos, it is shown that the pair comparison methodology tends to be more accurate due to straightforward experiment procedure and there is no statistically significant difference between ACR and DSIS methodology [START_REF] Mantiuk | Comparison of four subjective methods for image quality assessment[END_REF]. However, it's worth noting that the pair comparison methodology may become impractical when dealing with a substantial number of test conditions due to the exponential increase in required comparisons [START_REF] Zerman | The relation between MOS and pairwise comparisons and the importance of cross-content comparisons[END_REF]. On the other hand, the recent study by Nehme et al. [START_REF] Nehmé | Comparison of subjective methods, with and without explicit reference, for quality assessment of 3D graphics[END_REF] suggests that the DSIS method is more accurate than ACR for 3D graphical content. The rationale behind this assertion is that in ACR experiments, participants unfamiliar with the pristine models may struggle to discern various types of distortions. DSIS methodology enhances accuracy by presenting both the reference and the distorted model before the rating phase, allowing for a more informed evaluation. In line with these findings, we adopted the DSIS methodology with a side-by-side presentation format, as recommended by Nehme et al. [START_REF] Nehmé | Comparison of subjective methods, with and without explicit reference, for quality assessment of 3D graphics[END_REF]. A screenshot depicting the stimuli presented to the participants is featured in Figure 4.

C. Test Procedure

Previous studies have indicated that crowdsourcing experiments can yield results of comparable accuracy to traditional laboratory experiments across various Quality of Experience (QoE) tasks and with diverse experiment designs [START_REF] Nehmé | Exploring crowdsourcing for subjective quality assessment of 3D graphics[END_REF], [START_REF] Goswami | Reliability of crowdsourcing for subjective quality evaluation of tone mapping operators[END_REF]. In light of these findings, we chose to leverage the Prolific crowdsourcing platform to recruit participants and conduct our subjective experiment. Prolific ensures transparency and ethical participation by clearly communicating to participants that they are taking part in a research study. The experi- ment requirements are thoughtfully balanced to benefit both researchers and participants alike [START_REF] Palan | Prolific.ac-a subject pool for online experiments[END_REF]. This approach not only provides access to a broad pool of participants but also expedites the data collection process.

Test sessions & Duration: In crowdsourcing settings where participants lack supervision during the experiment, it is essential to limit both the number of stimuli and the duration of the test compared to laboratory experiments [START_REF] Goswami | Reliability of crowdsourcing for subjective quality evaluation of tone mapping operators[END_REF]. To accommodate this requirement, we divided the experiment into 60 sessions, each containing 25 stimuli and 2 "dummies". Sample frames from the two dummy stimuli are presented in Figure 5. For training purposes, one dummy from the highly compressed stimuli and one dummy from the least compressed stimuli were uniformly presented to every participant. These dummy stimuli remained consistent for all participants, and participants were not informed that these stimuli were included for training purposes. In total, each participant rated 27 stimuli of 10-seconds rendered videos. With unlimited voting time after each stimulus presentation, the average duration of the test sessions amounted to approximately 5 minutes and 30 seconds.

Participants & Requirements: We recruited 60 participants (50% female -50% male) on average per session, 3654 participants in total. The age of the participants range from 18 to 70. Each participant was compensated for their time and effort in line with the Prolific requirements. Moreover, to uphold the integrity of the experiment and guarantee that all stimuli were presented as intended, participants were constrained to use specific browsers operating in full-screen mode at a resolution of 1080p. Additionally, participants were required to meet specific qualifications, namely completing at least 200 submissions on the Prolific platform with a 100% approval rate. These stringent prerequisites helped ensure the reliability and commitment of the participants.

IV. SUBJECTIVE EXPERIMENT RESULTS

This section presents our analysis on the collected subjective quality scores. In Section IV-A, we discuss the observer reliability and provide an overview of the results obtained from the observer screening tools we have applied. Additionally, in Section IV-B, we present our findings on the performance of compression algorithms.

A. Observer Screening

In addition to the recruitment requirements of the participants (see Section III-C), the "dummy stimuli" described above were also used as trap questions to detect participants engaging in malicious behavior. Moreover, post-experiment observer screening tools were employed to enhance the reliability of the subjective opinion scores.

As described in Section III, the dataset underwent evaluation by a total of 3,654 participants, divided into 60 smaller playlists, each receiving an average of 60 participants. Within each playlist, we included 2 dummy stimuli (depicted in Figure 5) designed to calibrate participants' expectations regarding the extent of distortions present in the experiment. Participants were not informed that these initial 2 stimuli were for training purposes and evaluated them like regular stimuli. Source and processed point cloud renderings were displayed side-by-side like the rest of the stimuli in the experiment (see Figure 4 for what is displayed to observers). We selected the first dummy with the highest level compression in the dataset (resulting in a low-quality PPC), and the second dummy with the lowest level of compression (resulting in a high-quality PPC). This deliberate selection allowed us to expose each participant to both ends of the quality scale, aligning their expectations regarding the extent of degradation. Notably, if a subject rated the first dummy as "imperceptible" due to its clear distortions, we excluded them from the experiment, removing a total of 22 subjects.

As post-experiment observer screening, we applied the two common ITU standards as well as a recently proposed model called ZREC 11 [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF], [START_REF] Itu-R | Methods for the subjective assessment of video quality, audio quality and audiovisual quality of internet video and distribution quality television in any environment[END_REF], [START_REF] Zhu | ZREC : robust recovery of mean and percentile opinion scores[END_REF]. Our objective in employing these methodologies was to assess the validity of the collected subjective opinion scores and establish reliable MOS. While the three methods differ fundamentally, their primary aim is to reduce confidence intervals (CI) by identifying and, if necessary, excluding outliers from the collected data. Although there is a high correlation between the MOS obtained through each method, the average 95% CIs differ significantly.

Initially, we calculated the average 95% CI by taking the mean of the 95% CI of all stimuli without applying any postscreening methods, referred to as raw-MOS. Subsequently, we implemented the outlier rejection as recommended in ITU.BT500 [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF]. Notably, BT500 identified only 43 subjects as outliers out of the total 3,633 participants. As a result, the impact on both the MOS and the 95% CI was minimal, with the 95% CI (0.1982) remaining almost as high as the raw-MOS 95% CI. Conversely, we observed a more substantial shift in the acquired MOS when using P913-12.6 and ZREC, which yielded much lower confidence interval values. Specifically, we calculated an average 95% CI of 0.1697 and 0.1463 for P913-12.6 and ZREC, respectively. Based on these findings, we utilized the MOS acquired via ZREC due to the lower CI. To promote further investigation, we provide both the raw opinion scores and estimated MOS using ZREC in the public repository of the dataset.

B. Performance of compression algorithms

Although BASICS dataset contains 4 compression algorithms, in this analysis, we excluded GeoCNN. Since GeoCNN only compresses geometry and is complemented with uncompressed color information for visualization, it is not fair to directly compare it with GPCC-Predlift, GPCC-Raht and VPCC, which code both geometry and color.

When comparing the performance of GPCC-Predlift, GPCC-Raht and VPCC codecs, we employed two complementary approaches that analyze and visualize the MOS vs Bit-Per-Point (BPP) behavior of each compression algorithm. Due to large differences over BPP values per SRC, we first normalized all BPP values per SRC. We achieved this by dividing each PPC with the highest BPP value of the PPC across all compression algorithms that belongs to the corresponding SRC. Consequently, we could operate within a BPP range of 0 to 1, allowing a relative comparison of the compression algorithms.

The first approach draws inspiration from DenseLines [START_REF] Moritz | Visualizing a million time series with the density line chart[END_REF], a method originally devised to visualize vast quantities of time series data. To construct a Mean Opinion Score-Bit-Per-Point (MOS-BPP) curve density map for each compression algorithm, we partitioned the grid into square cells. Subsequently, for a given compression algorithm and SRC, we assigned a value of 1 to each cell intersecting with the polygon defined by the MOS and 95% CI values of each PPC. For each SRC, we normalized the marked cell values by dividing them by the total number of intersecting cells, effectively illustrating the effect of lower 95% CI and BPP values. Then, for each compression algorithms, we aggregated the normalized cell values from all SRCs and performed a linear mapping to scale the results into [0,1] range. Outputted values then plotted as overlaying heatmaps (with increasing transparency from 0 to 1), as showcased in Figure 6.

In the second approach, we employed a 2D Gaussian distribution to model each compression level within each compression algorithm. Subsequently, we extracted the centers of these 2D Gaussian distributions and utilized them to generate lines that captured the average MOS-BPP curves for each compression algorithm. These results were then plotted as lines superimposed upon the curve density heatmaps, providing a concise representation of each compression algorithm's performance.

As depicted in Figure 6, there is no significant difference between the GPCC-Predlift and GPCC-Raht. At lower BPP values, VPCC exhibits superior performance compared to both GPCC variants. However, as BPP values increase, the distinction between these compression algorithms becomes less pronounced. It's worth noting that while there may be occasional exceptions to these observations, they are minor in nature and do not substantially deviate from the trends observed in the analysis. We encourage interested readers to check the dataset public repository, where the MOS-BPP plots for each SRC is provided.

V. BENCHMARK OF OBJECTIVE QUALITY METRICS

Point cloud objective quality metrics can be categorized into three classes, based on the input to the metrics: image-based, color-based and geometry-based. Image-based metrics take the rendered point cloud image or image sequences as input and assess the quality of the point clouds. Geometry-based metrics primarily rely on the geometric information stored at each point in the point cloud, without considering color attributes. Color-based metrics, on the other hand, use the color information of each point to assess point cloud quality. Some metrics, such as PCQM [START_REF] Meynet | PCQM: A fullreference quality metric for colored 3D point clouds[END_REF], have the capability to utilize both geometry and color information for quality assessment.

Furthermore, each metric can be categorized into three based on the presence of reference point cloud information as full-reference (FR), reduced-reference (RR) and no-reference (NR). FR metrics access all information from the reference point cloud in addition to the distorted point cloud. RR metrics can access only partial information (features) from the reference point clouds. NR metrics assess the quality of the point cloud without any access to the reference point cloud.

In this section, we benchmark 14 image-based, 9 colorbased and 17 geometry-based metrics from the literature. Some of these metrics were omitted from the results due to minor differences to their variants. An introduction to the selected metrics is provided in Section V-A. We employ various figures of merit and evaluation scenarios to assess the performance of these metrics, which are outlined in Section V-B. The results of the metric evaluations are presented in Section V-C, Section V-D, and Section V-E.

A. Selected Metrics

For all image-based metrics, average pooling over 30 fps video renderings has been used to predict the final quality as recommended in [START_REF] Ak | The effect of temporal sub-sampling on the accuracy of volumetric video quality assessment[END_REF]. Image-based metrics have been computed on the rendered frames used during the subjective test, and include simple measures such as MSE, PSNR, SSIM [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF], MS-SSIM [START_REF] Wang | Multiscale structural similarity for image quality assessment[END_REF], and 11 other more sophisticated metrics. Feature Similarity Index (FSIM [START_REF] Zhang | FSIM: A feature similarity index for image quality assessment[END_REF]) and its colordependent variant, FSIMc [START_REF] Zhang | FSIM: A feature similarity index for image quality assessment[END_REF], fall under the category of full-reference metrics. They rely on phase congruency and gradient magnitude to locally quantify image quality, utilizing phase congruency as a weighting function to yield a single quality score. Gradient Magnitude Similarity Deviation (GMSD [START_REF] Xue | Gradient magnitude similarity deviation: A highly efficient perceptual image quality index[END_REF]) is another full-reference metric that employs pixel-wise gradient magnitude similarity to predict image quality. D-JNDQ [START_REF] Ak | From just noticeable differences to image quality[END_REF] is a learning-based full-reference metric trained on the first Just Noticeable Difference (JND) points of JPEG compression artifacts. It combines a white-box optical and retinal pathway model with a Siamese neural network to predict image quality. MW-PSNR [START_REF] Sandić-Stanković | DIBR synthesized image quality assessment based on morphological wavelets[END_REF], [START_REF] Sandić-Stanković | DIBR-synthesized image quality assessment based on morphological multi-scale approach[END_REF] relies on morphological wavelet decomposition and the Mean Squared Error (MSE) of the wavelet sub-bands. Our evaluation includes both full-reference (MW-PSNR-FR) and reduced-reference (MW-PSNR-RR) versions of this metric. ADM2 [START_REF] Li | Image quality assessment by separately evaluating detail losses and additive impairments[END_REF] assesses image quality by separating detail losses and additive impairments. It encompasses features also used in the Video Multi-method Assessment Fusion (VMAF) metric [START_REF] Li | Toward a practical perceptual video quality metric[END_REF]. VIF [START_REF] Sheikh | Image information and visual quality[END_REF] quantifies the information present in the reference image and how much of this reference information can be extracted from the distorted image. It is another feature used in the VMAF metric [START_REF] Li | Toward a practical perceptual video quality metric[END_REF]. VMAF [START_REF] Li | Toward a practical perceptual video quality metric[END_REF], proposed by Netflix, fuses several image-based features, including ADP2 and VIF, along with a simple temporal feature to evaluate video quality. FVVDP [START_REF] Mantiuk | FovVideoVDP: A visible difference predictor for wide field-of-view video[END_REF] models the response of the human visual system to differences across the temporal domain and the visual field.

In addition to image-based metrics, several geometry-based metrics were also evaluated over the dataset. In the last decade, three fundamental approaches were proposed to evaluate PC quality focusing on point-to-point [START_REF] Mekuria | Evaluation criteria for PCC (Point Cloud Compression)[END_REF] (p2point), point-toplane [START_REF] Tian | Geometric distortion metrics for point cloud compression[END_REF] (p2plane) and plane-to-plane [START_REF] Alexiou | Point cloud quality assessment metric based on angular similarity[END_REF] (pl2plane) differences in 3D space. The p2point and p2plane metrics are computed using either mean square error (MSE) or peak peak signal-to-noise ratio (PSNR). In this context, the term "plane" refers to the surface of a point defined by its normal vector. The pl2plane metrics are computed using either MSE or root mean square (RMS). While categorized as a geometry-based metric, PCQM [START_REF] Meynet | PCQM: A fullreference quality metric for colored 3D point clouds[END_REF] uses a linear combination of several geometry-based (curvature comparison, curvature contrast, and curvature structure) and color-based features (lightness comparison, lightness contrast, lightness structure, chroma comparison, and hue comparison) to assess the visual quality of a point cloud. PointSSIM [START_REF]Towards a point cloud structural similarity metric[END_REF] offers three geometrybased variants with slight differences in both implementation and performance.

Moreover, color differences between the points in reference and distorted PC can be quantified with PSNR to estimate the visual quality of the distorted PC. We applied this metric on Y, U, and V channels separately and referred as Color-Y-PSNR, Color-U-PSNR, and Color-V-PSNR respectively. 3 color-based variants of the PointSSIM [START_REF]Towards a point cloud structural similarity metric[END_REF] metric were also included in the evaluation. Fig. 7. Ideal distributions of metric score differences for "Different vs Similar" and "Better vs Worse" analysis. A greater metric score difference is expected for different pairs in "Different vs Similar" analysis. For "Better vs Worse" analysis, metric score differences are expected to be positive and negative respectively for better and worse pairs.

B. Evaluation Criteria

We evaluate the performance of the selected metrics in three different scenarios. In Section V-C, we aim to assess the perceptual quality of the PPC covering the entire quality range. This is the most generic and traditional evaluation scenario for quality metrics. In Section V-D, we evaluate the performance of the metrics in the high-quality range. This scenario is particularly relevant for applications such as content production, high-quality streaming, and digital twins, where maintaining the highest visual fidelity is crucial. In Section V-E, we evaluate the metric performances for their sensitivity to quality differences within different versions of the same point cloud content. This evaluation scenario is especially suitable to optimization scenarios such as point cloud compression and enhancement, and as loss functions in end-to-end PC learning pipelines. For broad and high-quality range evaluation scenarios, we employ traditional correlation measures, while Krasula's method [START_REF] Krasula | On the accuracy of objective image and video quality models: New methodology for performance evaluation[END_REF] is used for the Intra-SRC evaluation scenario.

Correlation measures: Pearson's linear correlation coefficient (PLCC) measures the prediction accuracy of the objective metrics and Spearman's rank-order correlation coefficient (SROCC) measures the strength of prediction monotonicity [START_REF] Itu-R | Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models[END_REF]. Following the recommendations [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF], [START_REF] Itu-R | Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models[END_REF], a 5parameter logistic function was fitted prior to evaluation. For both PLCC and SROCC, the values are in the range [0, 1] and higher values indicate a better correlation.

Krasula's method [START_REF] Krasula | On the accuracy of objective image and video quality models: New methodology for performance evaluation[END_REF]: It involves two stages of analysis: "Different vs Similar" and "Better vs Worse". In the "Different vs Similar" analysis, pairs of PPC from the dataset are split into two categories as pairs with (i.e., different) and without (i.e., similar) statistically significant differences. For a given pair of PPC, one way ANOVA followed by Tukey's honest significance difference test [START_REF] Tukey | Comparing individual means in the analysis of variance[END_REF] is used to measure the statistical significance of the differences. The assumption is that the absolute differences in metric predictions for "different" pairs should be larger than the "similar" pairs. Receiver Operating Characteristic (ROC) analysis is used to quantify the performance, and expressed as Area Under the ROC Curve (AUC). In the "Better vs Worse" analysis, pairs identified as "different" in the first stage are used. In this stage, the aim is to measure how well the metrics identify the better PPC in pairs with statistically significant difference. Metric performance in this stage can be expressed as the correct classification percentage as well as AUC values, similar to the first stage.

An illustrative example of how metric score differences should be distributed for each stage is depicted in Figure 7. In "Different vs Similar" analysis, higher metric score differences are expected for "Different" pairs and lower differences for "Similar" pairs. In the "Better vs Worse" analysis, for pairs categorized as "Better", positive metric score differences are expected, indicating that the first point cloud in the pair is better than the second one. Conversely, for pairs categorized as "Worse", negative metric score differences are expected, indicating that the first point cloud in the pair is worse than the second one.

C. Broad Quality Range Evaluation

Broad quality range evaluation scenario is the generic and commonly used use-case in the literature and it typically involves calculating metric performances with the traditional correlation measures (e.g., PLCC, SROCC, etc.). The entire dataset is used for this evaluation, and results reported as PLCC and SROCC values between the metric predictions and the collected MOS.

Table III presents the PLCC and SROCC of each metric in this evaluation scenario. The metrics are categorized into three group based on input type that they are operating on, as previously discussed in Section V-A. The first two column show the metrics' PLCC and SROCC scores on the entire dataset. Additionally, metric performances were evaluated for individual compression algorithms and the results are presented in subsequent columns as indicated above each column.

PCQM [START_REF] Meynet | PCQM: A fullreference quality metric for colored 3D point clouds[END_REF] and p2plane-MSE [START_REF] Tian | Geometric distortion metrics for point cloud compression[END_REF] exhibit the best performance on the entire dataset among the selected metrics, despite their poorer performance in predicting GeoCNN compression distortions. Among color-based metrics, we again notice a similar pattern on the accuracy of metrics when it comes to GeoCNN compression distortions. PointSSIM [START_REF]Towards a point cloud structural similarity metric[END_REF] variants perform relatively better than other metrics in this category.

Simple image-based metrics (e.g., MSE, PSNR, SSIM [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF], MS-SSIM [START_REF] Wang | Multiscale structural similarity for image quality assessment[END_REF]) have low accuracy across all compression categories and consequently on the whole dataset. VMAF [START_REF] Li | Toward a practical perceptual video quality metric[END_REF] shows the best performance among image-based metrics in the whole dataset. We also observe a general trend among image-based metrics towards a lack of accuracy on VPCC compression distortions. On another note, we observe that D-JNDQ [START_REF] Ak | From just noticeable differences to image quality[END_REF] performs the best to predict GeoCNN distortions among the selected metrics, despite not being retrained on the dataset. [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF] 0.2901 0.2119 MS-SSIM [START_REF] Wang | Multiscale structural similarity for image quality assessment[END_REF] 0.2697 0.1600 FSIM [START_REF] Zhang | FSIM: A feature similarity index for image quality assessment[END_REF] 0.3100 0.2508 FSIMc [START_REF] Zhang | FSIM: A feature similarity index for image quality assessment[END_REF] 0.3100 0.2502 GMSD [START_REF] Xue | Gradient magnitude similarity deviation: A highly efficient perceptual image quality index[END_REF] 0.3503 0.2993 D-JNDQ [START_REF] Ak | From just noticeable differences to image quality[END_REF] 0.4120 0.3771 MW-PSNR-FR [START_REF] Sandić-Stanković | DIBR synthesized image quality assessment based on morphological wavelets[END_REF] 0.1548 0.1490 MW-PSNR-RR [START_REF] Sandić-Stanković | DIBR-synthesized image quality assessment based on morphological multi-scale approach[END_REF] 0.2795 0.2633 ADM2 [START_REF] Li | Image quality assessment by separately evaluating detail losses and additive impairments[END_REF] 0.3171 0.2750 VIF-scale3 [START_REF] Sheikh | Image information and visual quality[END_REF] 0.2697 0.1600 VMAF [START_REF] Li | Toward a practical perceptual video quality metric[END_REF] 0.3466 0.3067 FVVDP [START_REF] Mantiuk | FovVideoVDP: A visible difference predictor for wide field-of-view video[END_REF] 0 [START_REF] Mekuria | Evaluation criteria for PCC (Point Cloud Compression)[END_REF] 0.1077 0.0774 p2plane-MSE [START_REF] Tian | Geometric distortion metrics for point cloud compression[END_REF] 0.5708 0.5418 p2plane-PSNR [START_REF] Tian | Geometric distortion metrics for point cloud compression[END_REF] 0.1067 0.1512 pl2plane-Mean [START_REF] Alexiou | Point cloud quality assessment metric based on angular similarity[END_REF] 0.0292 0.0511 pl2plane-RMS [START_REF] Alexiou | Point cloud quality assessment metric based on angular similarity[END_REF] 0.0770 0.0939 pl2plane-MSE [START_REF] Alexiou | Point cloud quality assessment metric based on angular similarity[END_REF] 0.0494 0.0642 PCQM [START_REF] Meynet | PCQM: A fullreference quality metric for colored 3D point clouds[END_REF] 0.5038 0.4775 PointSSIM-GeomAB [START_REF]Towards a point cloud structural similarity metric[END_REF] 0.4418 0.4527 PointSSIM-GeomBA [START_REF]Towards a point cloud structural similarity metric[END_REF] 0.4466 0.4617 PointSSIM-GeomSym [START_REF]Towards a point cloud structural similarity metric[END_REF] 0.4455 0.4615

D. High Quality Range Evaluation

High-quality range evaluation is crucial for applications that aim to deliver top-tier content, such as high-quality streaming and digital twins. It's important to note that metrics that perform well in the general quality range may not exhibit the same level of accuracy in the high-quality range. Therefore, we conducted an analysis to assess the accuracy of quality metrics specifically on the high-quality part of the dataset, where the MOS is greater than or equal to 3.5. This evaluation helps identify which metrics excel in scenarios where maintaining exceptionally high quality is a priority.

In this evaluation scenario, metric performances are relatively low overall, as indicated in Table IV. However, the relative order of the metrics in terms of their performances remains relatively consistent with the broad quality range evaluation. p2plane-MSE [START_REF] Tian | Geometric distortion metrics for point cloud compression[END_REF] performs the best overall while D-JNDQ [START_REF] Ak | From just noticeable differences to image quality[END_REF] is the best performing image-based metric and the PointSSIM-ColorBA [START_REF]Towards a point cloud structural similarity metric[END_REF] is the best performing colorbased metric for this use-case.

Various factors contribute to the low performance of metrics on the high quality range. First of all, most metrics are developed to account for a wider range of distortion intensities. Features, parameters, and modules used in a wide quality range are not as related to a more narrow range. A narrow quality range results in lack of statistically significant difference between the data points, resulting in a more difficult problem.

E. Intra-SRC Evaluation

Intra-SRC evaluation focuses on assessing metrics' performance when comparing PPC derived from a single pristine SRC at various compression levels. It allows us to gauge how well metrics can discriminate between different compression levels originating from the same SRC, helping us optimize processes that rely on such discrimination. This evaluation scenario is valuable for applications like fine-tuning compression and enhancement algorithms, training machine learning models for end-to-end processing, and other situations where fidelity is a primary concern.

Prior to the analysis, we preprocess the subjective scores as described in Krasula's method [START_REF] Krasula | On the accuracy of objective image and video quality models: New methodology for performance evaluation[END_REF]. First, 20 PPCs were paired within each source point cloud, generating (20 × (20 -1)/2) pairs per SRC. In total, we end up with 14143 pairs. Afterwards, a one-way ANOVA test is applied to individual scores collected for each stimulus in each pair, followed by Tukey's Honest test. 5019 pairs among the total 14143 were identified as "Similar" whereas 9124 contains a statistically significant different between the two PPC and thus identified as "Different". From those "Different" pairs, we split them into two roughly equal-sized groups as "Better" and "Worse" depending on the order of the pair. There are 4075 "Better" and 5049 "Worse" pairs. Due to space limitations of the manuscript, we report the result of the analysis only on the selected 6 metrics among the initial list presented in Section V-A. The rest of the results can be acquired via the provided scripts in the GitHub repository of the dataset.

Different vs Similar Analysis: The top row in Figure 8 presents the results of the analysis as histograms of metric score differences for "Different" and "Similar" pairs. We expect better-performing metrics to provide metric score distributions similar to the ideal case as depicted in Figure 7. Additionally, performance of each metric quantified with AUC values, reported under each metric name. We observe a better performance from PCQM, providing a higher AUC value and a very similar distribution to the ideal case. Statistical significance tests on this task also reveals that PCQM performs significantly better than all other metrics in "Different vs Similar" task.

Better vs Worse Analysis: Similar to the previous stage, bottom row of the Figure 8 presents the results as histograms of metric score differences and quantifies the performance of each metric with AUC and CC values. We observe that most metrics perform relatively well on identifying "Better" and "Worse" pairs apart. PCQM shows a very similar distribution to the ideal case depicted in Figure 7 as also reflected by the AUC and CC values.

VI. CONCLUSION

We conducted a large-scale crowdsourcing study on point cloud compression quality assessment. To the best of our knowledge, this is the largest publicly available point cloud quality assessment dataset. It contains 75 source point clouds, each compressed with 4 different compression algorithms resulting in nearly 1500 processed point clouds. More than 3500 naive observers participated to the experiment. Similarly, the plots in the bottom row show the metric score differences for pairs categorized as "Better" and "Worse" for the Intra-SRC evaluation. Again, the metric score differences are individually normalized within minimum and maximum ranges. The height of the bars denotes the number of occurrences, and each bar ranges between 0 and 800. The metric names are indicated at the top left corner of each plot and the AUC and CC are reported in the top right corner of each plot.

Our study revealed several noteworthy insights regarding objective quality metrics' performances. While most point cloud objective quality metrics perform well in predicting GPCC distortions, the majority of the metrics still struggle with VPCC distortions. Furthermore, an even larger majority fall short in assessing the quality GeoCNN distortions, a learning-based compression algorithm. This highlights a pressing need for improved quality metrics capable of accurately evaluating learning-based compression distortions. Given the scarcity of learning-based compression algorithms in publicly available datasets (see Table I), the need for better quality metrics that can accurately predict learning-based compression distortions is revealed in this work.

Additionally, we observed significant room for improvement in metrics designed for high-quality content. The correlations between the MOS and the best-performing metric predictions (p2plane-MSE in this scenario) remained below 0.60. On a more positive note, our intra-SRC evaluation scenario yielded more promising results. The best-performing metrics demonstrated more promising results for distinguishing the higher-quality point cloud. Nevertheless, there remains room for improvement in predicting the statistical significance of quality differences between point clouds.

As part of our commitment to advancing point cloud quality assessment, we are making our dataset publicly available. This dataset includes mean and individual opinion scores, along with scripts for metric evaluation in various scenarios. It also comprises all point clouds and their associated video renderings, forming what we call the BASICS dataset. We believe that the release of the BASICS dataset will contribute significantly to the improvement of existing point cloud quality metrics, the development of more robust ones, and the resolution of the challenges highlighted in this study. Finally, we plan to address some of the limitations in future studies by including dynamic PCs, a wider variety of semantic categories and styles, and other learning-based compression algorithms which performs both geometry and attribute coding.

Fig. 1 .

 1 Fig. 1. Sample renderings of 7 point clouds from each of the 3 semantic categories in the dataset (i.e., 7 of the total of 25 for each category).

Fig. 2 .

 2 Fig. 2. Sample frames from the video renderings of a selected processed point cloud (PPC), showing results for each compression algorithm.

Fig. 3 .

 3 Fig. 3. Visualization of the rendering trajectory from top and front views.

Fig. 4 .

 4 Fig. 4. Sample screenshots from the experiment. Rendered point cloud videos were shown side-by-side (above), and each stimulus was followed by a voting screen (below).

Fig. 5 .

 5 Fig. 5. Selected frames from the SRC and PPC video renderings of the 2 dummies used in all playlists.

  11 https://github.com/kyillene/ZREC This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TMM.2024.3355642 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: University of Nantes. Downloaded on January 19,2024 at 11:23:45 UTC from IEEE Xplore. Restrictions apply.

Fig. 6 .

 6 Fig. 6. Heatmaps represent the MOS-BPP curve densities of each compression algorithm. Centers of the 2 dimensional Gaussian distribution fit over each compression level for each algorithm is plotted as a line.

Fig. 8 .

 8 Fig.8. The plots in the top row depict the metric score differences for pairs categorized as "Different" and "Similar" for the Intra-SRC evaluation. Each metric's score differences are individually normalized within minimum and maximum ranges. The height of the bars represents the number of occurrences, and each bar ranges between 0 and 1500. The metric names are indicated at the top of each plot, and the AUC values are reported below each metric name. Similarly, the plots in the bottom row show the metric score differences for pairs categorized as "Better" and "Worse" for the Intra-SRC evaluation. Again, the metric score differences are individually normalized within minimum and maximum ranges. The height of the bars denotes the number of occurrences, and each bar ranges between 0 and 800. The metric names are indicated at the top left corner of each plot and the AUC and CC are reported in the top right corner of each plot.

TABLE III COLUMNS

 III "ALL" PRESENT THE PEARSON AND SPEARMAN CORRELATION COEFFICIENTS BETWEEN THE LISTED METRIC PREDICTIONS AND MOS (WITH ZREC [29]) OF THE ALL PPC IN THE BROAD QUALITY RANGE. MOREOVER, METRIC PERFORMANCES FOR EACH COMPRESSION ALGORITHM ARE ALSO INDIVIDUALLY REPORTED IN THE CORRESPONDING COLUMNS.

				All	GeoCNN	GPCC Predlift	GPCC Raht	VPCC
	Category	Metric	PLCC	SROCC	PLCC	SROCC	PLCC	SROCC	PLCC	SROCC	PLCC	SROCC
		MSE	0.2387	0.2226	0.4099	0.4183	0.3349	0.2578	0.3194	0.2387	0.1306	0.0649
		PSNR	0.2488	0.2269	0.4220	0.4231	0.3395	0.2621	0.3238	0.2429	0.1149	0.0691
		SSIM [33]	0.6119	0.5431	0.5496	0.5373	0.7491	0.6145	0.7599	0.6365	0.4034	0.3675
		MS-SSIM [34]	0.5481	0.4607	0.4944	0.4813	0.6744	0.5286	0.6847	0.5473	0.3553	0.2951
		FSIM [35]	0.6335	0.5612	0.5724	0.5714	0.7610	0.6308	0.7683	0.6501	0.4042	0.3748
		FSIMc [35]	0.6334	0.5608	0.5713	0.5703	0.7607	0.6298	0.7679	0.6495	0.4040	0.3745
	Image	GMSD [36]	0.6716	0.6113	0.6331	0.6336	0.7988	0.6718	0.7944	0.6854	0.4556	0.4390
	Based	D-JNDQ [37]	0.6732	0.6313	0.7215	0.7229	0.7891	0.6762	0.8017	0.7001	0.4352	0.4402
		MW-PSNR-FR [38]	0.3479	0.3296	0.4590	0.4577	0.4471	0.3766	0.4366	0.3668	0.1956	0.1624
		MW-PSNR-RR [39]	0.5110	0.4975	0.5704	0.5647	0.6250	0.5591	0.6275	0.5600	0.3011	0.3107
		ADM2 [40]	0.7283	0.6520	0.6434	0.6216	0.8408	0.6887	0.8362	0.7073	0.5513	0.5386
		VIF-scale3 [42]	0.6492	0.5947	0.5962	0.6035	0.7705	0.6525	0.7745	0.6717	0.4311	0.4249
		VMAF [41]	0.7419	0.6686	0.6572	0.6391	0.8540	0.7091	0.8532	0.7310	0.5541	0.5412
		FVVDP [43]	0.6936	0.6417	0.6312	0.6457	0.8197	0.6983	0.8300	0.7255	0.4676	0.4631
		Color-Y-PSNR	0.5376	0.5282	0.2166	0.2448	0.7407	0.7018	0.7567	0.7208	0.4251	0.4335
		Color-U-PSNR	0.5432	0.5105	0.2585	0.1254	0.6814	0.6529	0.6602	0.6291	0.3838	0.3751
	Color	Color-V-PSNR	0.5729	0.5411	0.3034	0.2661	0.7014	0.6755	0.6893	0.6614	0.4428	0.4238
	Based	PointSSIM-ColorAB [47]	0.7291	0.6907	0.5855	0.3936	0.8021	0.7962	0.8249	0.8278	0.7378	0.7675
		PointSSIM-ColorBA [47]	0.7241	0.6928	0.6044	0.4312	0.8033	0.7984	0.8257	0.8288	0.7287	0.7611
		PointSSIM-ColorSym [47]	0.7250	0.6919	0.6021	0.4296	0.8028	0.7975	0.8253	0.8279	0.7317	0.7633
		p2point-MSE [44]	0.8427	0.7759	0.6060	0.6114	0.9718	0.8863	0.9707	0.9002	0.7221	0.7154
		p2point-PSNR [44]	0.6827	0.4850	0.2798	0.2257	0.7697	0.5876	0.7822	0.6164	0.5172	0.4294
		p2plane-MSE [45]	0.8866	0.8370	0.6865	0.6415	0.9681	0.8886	0.9678	0.9028	0.7915	0.8068
		p2plane-PSNR [45]	0.7001	0.5164	0.3216	0.3113	0.7576	0.5794	0.7708	0.6071	0.5880	0.4816
	Geometry Based	pl2plane-Mean [46] pl2plane-RMS [46] pl2plane-MSE [46]	0.1393 0.1197 0.1189	0.1272 0.1002 0.1002	0.3390 0.2205 0.3334	0.1193 0.0553 0.2193	0.1730 0.1511 0.1508	0.1368 0.1195 0.1195	0.1753 0.1565 0.1562	0.1610 0.1448 0.1448	0.1029 0.0942 0.0942	0.0994 0.0777 0.0777
		PCQM [31]	0.8878	0.8102	0.4475	0.2965	0.9510	0.8746	0.9584	0.8953	0.8507	0.8332
		PointSSIM-GeomAB [47]	0.7760	0.7196	0.5497	0.5469	0.9067	0.8510	0.9091	0.8784	0.5439	0.5527
		PointSSIM-GeomBA [47]	0.7644	0.7145	0.5572	0.5783	0.9075	0.8477	0.9107	0.8757	0.5102	0.5148
		PointSSIM-GeomSym [47]	0.7731	0.7226	0.5566	0.5767	0.9094	0.8510	0.9116	0.8782	0.5677	0.5493

TABLE IV COLUMNS

 IV PRESENT THE PEARSON AND SPEARMAN CORRELATION COEFFICIENTS BETWEEN THE LISTED METRIC PREDICTIONS AND MOS (WITH ZREC [29]) OF THE PPC IN THE HIGH-QUALITY RANGE WHEREM OS ≥ 3.5.

	Category	Metric	PLCC	SROCC
		MSE	0.1187	0.0920
		PSNR	0.1476	0.1397
		SSIM
	Image
	Based

https://v-sense.scss.tcd.ie/

https://www.xdprod.com/

https://www.sketchfab.com/

https://github.com/mauriceqch/2023_BASICS_PC_toolbox

https://www.blender.org/

https://www.meshlab.net/

https://www.cloudcompare.org/main.html

This article has been accepted for publication in IEEE Transactions on Multimedia. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TMM.2024.3355642 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information. Authorized licensed use limited to: University of Nantes. Downloaded on January 19,2024 at 11:23:45 UTC from IEEE Xplore. Restrictions apply.

ACKNOWLEDGMENTS

This work has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant Agreement No. 765911 (Re-alVision) and from Science Foundation Ireland (SFI) under the Grant Number 15/RP/27760.