Jingwen Zhu

Anne-Flore Perrin

Patrick Le Callet

Subjective test methodology optimization and prediction framework for Just Noticeable Difference and Satisfied User Ratio for compressed HD video

Keywords: Video Quality Assessment, Just Noticeable Difference, Satisfied User Ratio, HD videos

Just Noticeable Difference (JND) and Satisfied User Ratio (SUR) has been widely investigated for compressed image and video to use the least resources (e.g., storage and bandwidth) without damaging the Quality of Experience (QoE) for end users. However, the current JND subjective test methodologies are extremely time consuming due to the large range of encoding parameters. Besides, the state-of-the-arts SUR/JND prediction models get non-negligible prediction error due to the limited masking effect features. To this end, we first proposed a preprocessing method to reduce the JND subjective test time by using dynamic range of encoding parameters and collected a new Video-Wise JND (VW-JND) datasets for HD videos: HD-VJND. Afterwards, based on the collected datasets, we proposed a SUR prediction framework by extracting 3 types of features 1) masking effect features; 2) bitstreams features; 3) content features. Feature selection is applied to extracted features before regression. Besides, we also compared the direct and indirect SUR value predictions methods. Experiment results shows that our proposed optimization can reduce 7.14% of the subjective experiment time compared to the widely used Robust Binary Search (RBS). Furthermore, the proposed SUR and JND prediction frameworks outperform the SOTA model in HD-VJND datasets.

I. INTRODUCTION

Human Visual System (HVS) cannot perceive detailed differences between the pristine images/videos and their compressed versions with small distortion. In the context of image/video compression, Just Noticeable Difference (JND) is the minimum distortion level from which human eye begins to perceive difference between the reference image/video and the compressed ones when increasing the level of compression. Specifically, when the reference is the no-compression version, e.g., sources (SRCs) video, the JND is called 1st JND, and if we take the 1st JND as the new reference/anchor, the next JND is then the 2nd JND, etc. The proxy of JND for compressed image/video is usually the encoding parameters of codec, e.g., quantization parameter (QP) in H.264. Acquisition of JND has been widely investigated both subjectively [START_REF] Jin | Statistical study on perceived jpeg image quality via mcl-jci dataset construction and analysis[END_REF]- [START_REF] Wang | Videoset: A large-scale compressed video quality dataset based on jnd measurement[END_REF] and objectively [START_REF] Fan | Sur-net: Predicting the satisfied user ratio curve for image compression with deep learning[END_REF]- [START_REF] Zhang | Deep learning based just noticeable difference and perceptual quality prediction models for compressed video[END_REF] for image/video compression because it helps save resources e.g., storage and bandwidth, without damaging the quality for end-users.

It is well known that JND depends on 3 factors: (1) visualization setting,(e.g., display, environment...) [START_REF] Fan | Picture-level just noticeable difference for symmetrically and asymmetrically compressed stereoscopic images: Subjective quality assessment study and datasets[END_REF] subjects and (3) image/video contents [START_REF] Zhu | On the benefit of parameter-driven approaches for the modeling and the prediction of satisfied user ratio for compressed video[END_REF]. In this paper, the first two factors are fixed and we will focus on the impact of difference of contents on the 1st JND of High-Definition (HD) videos.

For the same content, the JND varies with observers because of their various visual sensitivity. To tackle this variation, Satisfied User Ratio (SUR) curve has been widely used [START_REF] Wang | Videoset: A large-scale compressed video quality dataset based on jnd measurement[END_REF]- [START_REF] Lin | Sur-featnet: Predicting the satisfied user ratio curve for image compression with deep feature learning[END_REF], [START_REF] Zhang | Satisfieduser-ratio modeling for compressed video[END_REF], [START_REF] Zhang | Deep learning based just noticeable difference and perceptual quality prediction models for compressed video[END_REF] to measure JNDs for a group of observers. SUR curve is defined as the Complementary Cumulative Distribution Function (CCDF) of the distribution of JNDs for a group of observers [START_REF] Zhu | On the benefit of parameter-driven approaches for the modeling and the prediction of satisfied user ratio for compressed video[END_REF]. Therefore, SUR curve is monotonic nonincreasing, it indicates the relationship between the distortion level d and the percentage of observers p% who are satisfied. Specifically, for a given distortion level d, its corresponding value p% in the SUR curve means that there are p% observers in the group who do not perceive any difference between the reference and distorted videos with distortion level smaller than d. For a given threshold p, the corresponding distortion level d in the SUR curve is defined as p%SUR. p%SUR is denoted as p%JND in some previous works [START_REF] Wang | Videoset: A large-scale compressed video quality dataset based on jnd measurement[END_REF], [START_REF] Zhang | Satisfieduser-ratio modeling for compressed video[END_REF], [START_REF] Zhang | Deep learning based just noticeable difference and perceptual quality prediction models for compressed video[END_REF]. In this work, we use p%SUR to avoid misunderstanding.

Subjective study: Subjective JND datasets are the base on JND studies. Table I is the comparison of existing VW-JND. One of the biggest challenges in JND subjective testing is that the JND search process is extremely time consuming because of the large range of encoding parameters, for example in VideoSet [START_REF] Wang | Videoset: A large-scale compressed video quality dataset based on jnd measurement[END_REF], every SRC is encoded into 51 Processed Video Sequences (PVSs) using H.264 with QP from 1 to 51. The list of these PVSs is named as JND candidate playlist (JCP). Instead of comparing the reference with each PVS from QP = 1 until JND is found, Wang et al. [START_REF] Wang | Videoset: A large-scale compressed video quality dataset based on jnd measurement[END_REF] proposed a Robust Binary Search (RBS) procedure which is inspired by the widely used binary search algorithm in computer science. During the Classic Binary Search (CBS), the observer will be asked at the first time if they can perceive the difference between the video with QP = 0 (reference) and QP = 25 (middle of the original interval of JCP). If "YES", the interval QP = [START_REF] Ferri | Comparative study of techniques for large-scale feature selection[END_REF]51] will be excluded in the next comparison; if "NO", the interval QP = [0, 24] will be excluded. However, this will bring problems when the observer makes unconfident decision for the previous comparison. The RBS is a modified version of the CBS which only eliminate one quarter of the original interval instead of removing the half of it, e.g., interval QP = [39, 51] instead of QP = [START_REF] Ferri | Comparative study of techniques for large-scale feature selection[END_REF]51] in the previous example.

However, the JND subjective experiment time is still nonnegligible even with the help of RBS. In this work, we proposed a method to reduce the subjective test time and collected a new VW-JND datasets named as HD-VJND for HEVC with dynamic range of Constant Rate Factor (CRF).

Objective study: It is not practical and extremely time consuming to conduct JND subjective tests for every new content. Therefore, it is important to develop objective JND/SUR prediction models based on the subjective JND datasets. Wang et al. [START_REF] Wang | Prediction of satisfied user ratio for compressed video[END_REF] proposed a model using Support Vector Regression (SVR) to predict SUR curve and accordingly 75%SUR based on masking effect features [START_REF] Hu | Compressed image quality metric based on perceptually weighted distortion[END_REF], [START_REF] Hu | Objective video quality assessment based on perceptually weighted mean squared error[END_REF] extracted from SRCs and quality degradation features computed from all PVSs on VideoSet. By considering the spatial and temporal information features via deep learning, Zhang et al. [START_REF] Zhang | Deep learning based just noticeable difference and perceptual quality prediction models for compressed video[END_REF] improved the SUR prediction accuracy using Video Wise Spatial-Temporal SUR. Instead of using encoding parameter QP as proxy of SUR curve, Zhang et al. [START_REF] Zhang | Satisfieduser-ratio modeling for compressed video[END_REF] used bitrates as the proxy of SUR curve and made predictions with Gaussian Processes Regression (GPR) by extracting masking effect features, recompression features and basic attribute features.

However, all the above mentioned JND/SUR prediction models are based on the assumption that the distribution of individual JND for a group of observers follows Gaussian distribution, which may not be the optimal mathematical modeling. Furthermore, when predicting SUR curve, they predict each point of the curve by using features from every PVSs, which is not practical and has high computational complexity because each video needs firstly to be compressed into many PVSs. A previous work [START_REF] Zhu | On the benefit of parameter-driven approaches for the modeling and the prediction of satisfied user ratio for compressed video[END_REF] investigated these problems by firstly comparing several different mathematical modelings of SUR curve and secondly computing the SUR curve by predicting the modeling parameters only based on the features of SRC. Nevertheless, the prediction errors of SUR curve and 75%SUR are still non-negligible due to the limitation of masking effect features. In this work, we propose a SUR prediction framework (see Fig. 2) that extract many different types of features followed by features selection and regression.

II. METHODOLOGY A. HD-VJND datasets

We collected a new VW-JND datasets named as HD-VJND datasets, as shown in the last row of Table I. 180 contents are selected from 606 video clips provided by research partner based on the content selection strategy proposed in [START_REF] Ling | Towards perceptually-optimized compression of user generated content (ugc): Prediction of ugc rate-distortion category[END_REF] such that the selected contents are with wide-range of difference. The duration of each video is 10 seconds. 20 observers with correct visual acuity involved in the subjective JND test. The displays are 55-inch calibrated 4K TVs and the viewing distance was set as 3H for HD contents as recommended in ITU-R BT.2022 [START_REF] Bt | General viewing conditions for subjective assessment of quality of sdtv and hdtv television pictures on flat panel displays[END_REF], where H is the height of the screened video. As mentioned in Section I, the JND search procedure is still time consuming with the RBS process. The JND search time depends on the length of JCP (e.g., length of JCP = 51 in VideoSet per content). Meanwhile, it is well known that from a certain level of compression (e.g., QP = 40), it is almost certain that anyone with correct visual acuity can perceive difference between the reference and the PVS. Therefore, we proposed a method to optimize the JND subjective test time by reducing the length of JCP with the help of a pre-processing using the mapping function from VMAF to JND proposed in [START_REF] Zhu | A framework to map vmaf with the probability of just noticeable difference between video encoding recipes[END_REF]. The mapping function for HD videos is shown in Fig. 1. It can be observed that the higher the VMAF difference (∆VMAF) between two videos (same content, different encoding recipes), the more likely it is for humans to perceive differences between them in terms of quality.

The idea is to remove the low quality PVSs that human eyes can perceive "for sure" differences to reduce the numbers of comparison before finding the JND. For a given threshold thr%, the corresponding value of ∆VMAF in the mapping function is denoted as V thr% . The reference for the 1st JND is SRC, therefore ∆VMAF = VMAF(SRC) -VMAF(P V S) and VMAF(SRC) = 100. The PVS whose ∆VMAF is larger than V thr% will be removed from the JCP. Eq.( 1) stipulates the condition to eliminate the PVS to save subjective test time.

VMAF(P V S) < 100 -V thr% (1)
As illustrated in Table II, we compared the maximum number With the decrease of the thr% in the mapping function, V thr% will reduce and the number of PVSs eliminated will increase according to Eq.( 1), thus the average length for JCP will decrease. However, it is possible that the JND video will be eliminated during this procedure. The last columns in Table II indicate the number of videos whose JND are excluded during the pre-processing. It can be observed that only when the threshold is close to 1, we can ensure not to remove any JND in VideoSet. For CBS, it is well known that the maximum number of comparison is

log 2 (len(JCP )) = log (1/2) (1/len(JCP )) , (2)
However, the interval to keep for RBS is 3/4 instead of 1/2 in each iteration, thus the maximum number of comparison is calculated by log (3/4) (1/len(JCP )). It can be concluded that our proposed method can reduce by 7.14% the subjective test duration without removing any mandatory information. Therefore, we used this pre-processing for HD-VJND collection. To our best knowledge, this is the first VW-JND datasets using HEVC as codec and CRF as JND proxy.

B. SUR prediction framework

There are two steps (see Fig. 2) for the entire pipeline: (1)Modeling; (2)Prediction. Modeling includes computing the empirical SUR curve (SUR emp ) from the JND distribution of the group-users and finding the best mathematics model to fit the SUR emp . The fitted SUR curve is denoted as SUR analy .

For more details about the modeling, we refer readers to [START_REF] Zhu | On the benefit of parameter-driven approaches for the modeling and the prediction of satisfied user ratio for compressed video[END_REF]. After generating ground truth from modeling, we use SRC as input to extract features, select features and make predictions. The prediction framework is detailed as follows:

1) Feature extraction: Three types of features: (1) masking effect features, (2) bitstream features and (3) content features are extracted as illustrated in Fig. 2.

Masking effect Features measures randomness/regularity temporally (temporal randomness (TR) [START_REF] Hu | Objective video quality assessment based on perceptually weighted mean squared error[END_REF]) and spatially (spatial randomness (SR) [START_REF] Hu | Compressed image quality metric based on perceptually weighted distortion[END_REF]). When the randomness is high, it will be difficult for human to perceive difference, it masks the distortion for HVS. Masking effect features were used in [START_REF] Wang | Prediction of satisfied user ratio for compressed video[END_REF], [START_REF] Zhu | On the benefit of parameter-driven approaches for the modeling and the prediction of satisfied user ratio for compressed video[END_REF] to predict JND.

As shown in Fig. 2, SRC is segmented into small video patches both spatially and temporally to extract features from the eye fixation level. The dimensions of video patches are set the same as [START_REF] Wang | Prediction of satisfied user ratio for compressed video[END_REF]. SR and TR are calculated on each small video patche to obtain feature matrices F SR and F T R . The statistic histogram (Eq. ( 3)) with number of bins equals to 20 is applied as pooling method to reduce the feature dimension.

-

→ SR = Hist 20 (F SR ), -→ T R = Hist 20 (F T R ) (3)
Bitstream Features are widely used for light-weight quality estimation [START_REF] Raake | Scalable video quality model for itu-t p. 1203 (aka p. nats) for bitstream-based monitoring of http adaptive streaming[END_REF]. Before extracting bitstream features in Table III, SRCs are first compressed into a near lossless PVS with CRF = 5. The bitstream features are extracted using videoparse [START_REF] Rao | Bitstream-based model standard for 4k/uhd: Itu-t p.1204.3 -model details, evaluation, analysis and open source implementation[END_REF] without decoding pixel information.

The temporal and spatial pooling function are defined as: 4)) and spatial (Eq. ( 5)) pooling function.

F time = {M ean, Std, M ax, Skew, Kurt} (4)
F space = {M ean, Std} (5)
Content Features include 7 types of features: Spatial Information(SI) [START_REF] Recommendation | Subjective video quality assessment methods for multimedia applications[END_REF], Temporal Information (TI) [START_REF] Recommendation | Subjective video quality assessment methods for multimedia applications[END_REF], Chrominance Information (CI) [START_REF] Wolf | Video quality measurement techniques[END_REF], Contrast Information(CTI) [START_REF] Wolf | Video quality measurement techniques[END_REF], Spatial Perceptual Information (SPI) [START_REF] Wolf | Video quality measurement techniques[END_REF], Colorfulness (CF) [START_REF] Hasler | Measuring colorfulness in natural images[END_REF] and Grey Level Co-occurrence Matrix(GLCM) [START_REF] Haralick | Textural features for image classification[END_REF]. As illustrated in Fig. 2, they are extracted directly from the pixel level of the SRC [START_REF] Ling | Towards perceptually-optimized compression of user generated content (ugc): Prediction of ugc rate-distortion category[END_REF]. The temporal pooling function for content features is the same with bitstream features (Eq. ( 4)), and the spatial pooling function is defind as:

F space = {M ean, Std, M ax, Skew, Kurt}. (6)
The co-occurrence matrix (CM) is computed based on image patches. For each small patche, we calculated 6 features as shown in Eq. ( 7):

F patch = {contrast, dissimilarity, homogeneity, ASM, energy, correlation}, (7)
where

contrast = l-1 i,j=0 CM i,j (i -j) 2 , the dissim- ilarity diss = l-1 i,j=0 CM i,j |i -j|, the homogeneigy homo = l-1 i,j=0 CMi,j 1+(i-j) 2 , Angular Second Moment: ASM = l-1 i,j=0 CM 2 i,j , energy = √ ASM and correlation = l-1 i,j=0 CM i,j (i-µi)(j-µj ) √ σ 2 i σ 2 j
, in which l is the level of luminance of original image patch (l=255 for 8 bit image), i, j are the horizontal and vertical index of CM respectively; µ, σ are the mean and variance of CM.

2) Features selection: As shown in Fig. 2, all the extracted features are concatenated into one vector. The exhibited vector has dimension of 399. We then used Forward-Sequential Feature Selection (F-SFS) [START_REF] Ferri | Comparative study of techniques for large-scale feature selection[END_REF]. It is a greedy procedure. More specifically, we initially find the feature that maximizes a cross-validated score when an estimator is trained on this single feature. Once that first feature is selected, we repeat the ,

Features dimension

SI + = F time {F space {Sobel [Y n (i, j)]}} 5*5 =25 T I + = F time {F space [M n (i, j)]} where M n (i, j) = Y n (i, j) -Y n-1 (i, j) 5*5 =25 CI Cb = F time {F space [Cb n (i, j)]} CI Cr = F time {W R × F space [Cr n (i, j)]} where W R = 1.
rg = R n (i, j) -G n (i, j), yb = 1 2 (R n (i, j) + G n (i, j)) -B n (i, j) 5*5+5 = 30 CF = F time {CF n } where CF n = σ rgyb + 0.3µ rgyb σ rgyb = σ 2 rg + σ 2 yb , µ rgyb = µ 2 rg + µ 2 yb 5 GLCM = F time F space F patch (CM
where CM is the co-occurrence matrix 1 5*(5*6)=150 1 https://scikit-image.org/docs/0.7.0/api/skimage.feature.texture.html 2 F time and Fspace are the temporal (Eq. ( 4)) and spatial (Eq. ( 6)) pooling function; F patch is the functions to compute the texture features of the spatial patch with size n = 64 * 64. 3 Y , Cr and Cb are the luminance and two chroma components; R, G and B are the red, green and blue channels.

procedure by adding the new feature that maximizes the crossvalidated score to the set of selected features. The procedure stops when the desired number N of selected features is reached. Grid search was adapted to determine N .

3) Regression: The selected features will be fed into a SVR for prediction. As shown in Fig. 2, there are two ways to predict p%SUR: (A) indirect p%SUR prediction through SUR modeling, (B) direct p%SUR prediction without modeling. For the indirect mode, analytical SUR curve (SUR analy ) and its parameters are determined by fitting the empirical SUR curve (SUR emp ). We first obtain the SUR pred curve by predicting the fitted parameters from the features. p%SUR can be computed from the SUR pred curve. If one is only interested in the value of p%SUR (e.g., the demand of a streaming service provider is to satisfy 75% clients), but not the SUR curve, the direct prediction without modeling can be adapted as illustrated in the dotted box in Fig. 2.

III. EXPERIMENTS AND RESULTS

The logistic function showed best prediction performance in [START_REF] Zhu | On the benefit of parameter-driven approaches for the modeling and the prediction of satisfied user ratio for compressed video[END_REF], hence it is employed in the indirect prediction model. Before feeding the extracted features to the SVR, all the features are normalized by applying z-score transformation. The estimator for F-SFS is the SVR and the metric of features selection is the Mean Square Error (MSE). The optimal number of selected features is 55, detailed in Table V. It can be observed that bitstream features has highest selection rate, which means the bitstream features such as motions are significant for SUR value prediction. Experiment results show that our proposed models (both direct and indirect mode) outperforms (reduced ∆SUR |P-E| by 40%) the baseline model [START_REF] Zhu | On the benefit of parameter-driven approaches for the modeling and the prediction of satisfied user ratio for compressed video[END_REF]. The direct p%SUR prediction mode without modeling has the smallest prediction error in terms of ∆75%SUR. However, the indirect model provides us more information (the SUR curve and the 75%SUR value) compared to the direct model that outputs only the 75%SUR value. Furthermore, it could be observed in Table VI that the indirect model has smaller standard deviation than direct p%SUR prediction model which indicates that the indirect model helps stabilize the variation of the prediction error.

IV. CONCLUSION

In this paper, we proposed a pre-processing method for JND subjective test by using the mapping function between VMAF and JND. This method helps us to determine the dynamic range of encoding parameters which helps to reduce 7.14% of subjective test time. Furthermore, we proposed a SUR/JND prediction framework including feature extraction/selection and regression through 3 types of features. Experiment results show that our proposed framework outperforms the SOTA both in SUR curve and 75%SUR value prediction.

Fig. 1 .

 1 Fig. 1. Mapping function between ∆VMAF and Probability of perceive difference between 2 videos with different encoding recipes for HD videos of comparison. Estimated time of one comparison is 20 seconds, it includes the time for one observer to watch the reference and distorted videos and to answer the question of whether they can perceive differences. Mean of len(JCP ) is the average length of the JCP across the entire datasets.With the decrease of the thr% in the mapping function, V thr% will reduce and the number of PVSs eliminated will increase according to Eq.(1), thus the average length for JCP will decrease. However, it is possible that the JND video will be eliminated during this procedure. The last columns in TableIIindicate the number of videos whose JND are excluded during the pre-processing. It can be observed that only when the threshold is close to 1, we can ensure not to remove any JND in VideoSet. For CBS, it is well known that the maximum

Fig. 2 .

 2 Fig. 2. Illustration of the pipeline of SUR and JND (1) modeling and (2) prediction framework

5 5 *

 5 5+5*5=50 CT I = F time {F space [Y n (i, j)]} 5*5 = 25 SP I SI13 = F time {F space [R n (i, j)]} where R n (i, j) = H n (i, j) 2 + V n (i, j) 2 , SP I HV 13 = F time mean[HV (i,j)] mean[HV (i,j)]

  where M ean is the average value, Std indicates standard deviation, M ax denotes maximum, Skew represents skewness, and Kurt is the kurtosis. The dimension of features equals to the product of the dimension of F time and F space (e.g., for motion features, we first compute the M ean and Std of motions intra frame (spatially); afterwards, M ean, Std, M ax, Skew and Kurt are calculated based on the two previouscomputed spatial value (M ean and Std for each frame) inter frame (temporally) respectively.)

TABLE IV CONTENT

 IV FEATURES SUMMARY.

  Predicted and Analytical SUR curve (∆SUR |P-A| ) is the Mean Average Error (MAE) between them. ∆SUR |P-A| indicates the error between the fitted analytical SUR curve and the predicted one. Difference between Predicted and Empirical SUR curve (∆SUR |P-E| ) is evaluated as well. ∆75%SUR is evaluated in the same way. The results are shown in Table VI. For the model who predict directly the p%SUR (Direct mode), the ∆SUR |P-A| and ∆SUR |P-E| don't exist. Similarly, we cannot compute ∆75%SUR |P-A| because the SUR analy doesn't exist without modeling.

		TABLE V
	NUMBER OF FEATURES SELECTED PER CATEGORY
	Features type	Selected/original	Ratio of selection
	Masking effect	9/40	0.2250
	Bitstream	15/49	0.3061
	Content features	31/310	0.1000
	Each model is evaluated in HD-VJND datasets with 5-
	fold cross validation with random split (fixed random state).
	Hyper parameters of SVR are determined by grid search (ker-
	nel='rbf', C=0.1, epsilon=0.0001, gamma='scale'). Difference
	between

ACKNOWLEDGMENT

This work was supported in part by Amazon Prime Video. We thank Dr. Sriram Sethuraman and Dr. Kumar Rahul for helpful discussions and insights into this topic.