Jingwen Zhu

Ali Ak

Patrick Le Callet

Sriram Sethuraman

Kumar Rahul

ZREC: ROBUST RECOVERY OF MEAN AND PERCENTILE OPINION SCORES

Keywords: Observer screening, MOS recovery, Satisfied User Ratio, Quality of Experience

Observer screening and subject opinion score recovery is essential for collecting a reliable QoE database. This paper proposes a new method, ZREC * , which uses Z-scores to estimate subject bias, inconsistency, and content ambiguity. Additionally, we propose Mean Opinion Score (MOS) recovery and Percentile Opinion Score (POS) recovery scheme based on the three estimated parameters. ZREC does not fully reject subjects, rather adjust their coefficients in the MOS/POS recovery, allowing for more efficient use of data collection. The estimated parameters of ZREC are highly correlated with more complex solver-based methods and standards. In addition, ZREC recovers MOS with smaller confidence intervals than the state of the art. Experimental results also demonstrate that using recovered p th POS as ground truth during training improves the performance of Satisfied User Ratio (SUR) prediction.

INTRODUCTION

Observer screening is essential steps of collecting a reliable Quality of Experience (QoE) database. A range of methods [START_REF] Li | A simple model for subject behavior in subjective experiments[END_REF][START_REF] Li | A probabilistic graphical model for analyzing the subjective visual quality assessment data from crowdsourcing[END_REF][START_REF] Li | Gpm: A generic probabilistic model to recover annotator's behavior and ground truth labeling[END_REF][START_REF] Ak | Rv-tmo: Large-scale dataset for subjective quality assessment of tone mapped images[END_REF][START_REF] Ak | On spammer detection in crowdsourcing pairwise comparison tasks: Case study on two multimedia qoe assessment scenarios[END_REF] with varying complexities have been proposed, and various standards [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF][START_REF] Itu-R | Subjective video quality assessment methods for multimedia applications[END_REF][START_REF] Itu-R | Methods for the subjective assessment of video quality, audio quality and audiovisual quality of internet video and distribution quality television in any environment[END_REF] include recommendations for this purpose. It has also been demonstrated that training learning-based metrics on recovered MOS can enhance their performance to a certain extent [START_REF] Perrin | When is the cleaning of subjective data relevant to train ugc video quality metrics?[END_REF].

The collected QoE measurements in subjective studies are often characterized as a combination of subject bias, inconsistency and the underlying quality of the stimuli [START_REF] Li | A simple model for subject behavior in subjective experiments[END_REF]. Subject bias refers to the systematic error of a subject towards a certain direction, e.g. a positive bias indicates the subjects overall tendency to perceive a higher quality. Subject inconsistency is associated with the random unexplained error included in the observations, such as lack of attention, malicious intentions etc. On another front, the content ambiguity defines the level of difficulty in evaluating a stimulus due to its inherent ambiguity. § Below, we provide a summary of commonly used MOS recovery methods from the literature and briefly discuss their advantages and disadvantages. In addition, Table 1 provides a quick overview of the parameters estimated by each model.

• BT500: ITU-R BT.500 Recommendation [START_REF] Itu-R | Methodology for the subjective assessment of the quality of television pictures[END_REF] defines an outlier rejection procedure. Subjects are rejected based on the number of opinion scores outside of the predefined amount of standard deviation range of the population. If a subject found to be an outlier, all of his/her opinions are removed from the dataset. MOS is calculated as the mean of remaining subjects. Due to hard-coded thresholds and removing all votes of a detected outlier, the MOS recovery may result in even larger confidence intervals • P913-12.4: ITU-R P.913 Recommendation clause 12.4 [START_REF] Itu-R | Methods for the subjective assessment of video quality, audio quality and audiovisual quality of internet video and distribution quality television in any environment[END_REF] proposes a procedure based on both bias removal and outlier rejection. For each subject, the bias is calculated as the average difference between the MOS and subjects' opinion score of each stimulus. Estimated biases are removed from subject opinion scores and then MOS can be calculated as the average of bias-removed opinion scores, optionally after rejecting outliers. It can be seen that P913-12.4 is a slight improvement over BT500. However after removing the subject bias, the observer are still rejected with hard-coded parameters or treated equally by ignoring the subject inconsistency. • P913-12.6: ITU-R P.913 Recommendation clause 12.6 [START_REF] Itu-R | Methods for the subjective assessment of video quality, audio quality and audiovisual quality of internet video and distribution quality television in any environment[END_REF] defines a procedure where MOS is recovered by bias removal and subject inconsistency weighting. Same procedure is also included in ITU-R P.910 Recommendation Annex-E [START_REF] Itu-R | Subjective video quality assessment methods for multimedia applications[END_REF]. The procedure defines the individual opinion scores of a subject as the combination of subject bias, inconsistency and the true quality of the stimuli and jointly solves these three parameters. Two solvers are proposed for the approach in [START_REF] Li | A simple model for subject behavior in subjective experiments[END_REF]. Due to minimal differences between the solvers, we only consider the Alternating Projection (AP) solver in this work. P913-12.6 can be seen as the next step of the P913-12.4 by additionally considering subject inconsistency during MOS recovery. Note that the model does not provide any estimate for content ambiguity. • MLE: Li et al. [START_REF] Li | Recover subjective quality scores from noisy measurements[END_REF] proposed a a MOS recovery approach by jointly estimating bias and inconsistency of subject and content ambiguity with Maximum Likelihood Estimation (MLE) and belief propagation. In addition to bias and inconsistency of subject as P913-12.6, MLE also provide content ambiguity. However, it is acknowledged by the authors that the MLE solver has the issue of lacking of uniqueness in its solution in certain cases, e.g., it cannot find solutions for HD-VJND dataset (see Sec.2). To address the limitations of previous work, we propose an alternative method that relies on Z-score to estimate subject bias, inconsistency, and content ambiguity. We also present a simple yet efficient MOS and POS recovery scheme. Our proposed model is more robust to different use-cases and datasets as it does not require a solver, which can sometimes result in convergence issues. The contributions of this work are:

• A simple yet robust statistical model for estimating subject bias, inconsistency, and content ambiguity from subjective opinion scores. • A MOS and POS recovery method based on the estimated subject bias and inconsistency. • Performance comparison between the proposed model and the state-of-the-art, validated by estimating CIs and quantifying the impact of p th POS recovery on the accuracy of SUR prediction models.

QOE DATASETS

The performance of ZREC and other existing models from the literature in terms of MOS and POS recovery, as well as estimation of subject bias, inconsistency and content ambiguity, have been assessed on two datasets with distinct characteristics.

HD-VJND: is a Video-Wise Just Noticeable Differences (VW-JND) dataset for HD videos [START_REF] Zhu | Subjective test methodology optimization and prediction framework for just noticeable difference and satisfied user ratio for compressed hd video[END_REF]. There are 180 source content (SRC) evaluated by 20 naive subjects with correct visual acuity. Each SRC has been compressed with HEVC with different Constant Rate Factor (CRF) and presented to each subject via Robust Binary Search [START_REF] Zhu | Subjective test methodology optimization and prediction framework for just noticeable difference and satisfied user ratio for compressed hd video[END_REF] to find the JND of each subject. Therefore the proxy of JND is represented by CRF value. Satisfied-User-Ratio (SUR) curve is the complementary cumulative distribution function of the individual JNDs of a viewer group [START_REF] Zhu | On the benefit of parameter-driven approaches for the modeling and the prediction of satisfied user ratio for compressed video[END_REF]. q%SUR is the CRF value that corresponds to a SUR value on the SUR curve equals to threshold q%. 75% is the most commonly used threshold [START_REF] Zhang | Satisfied-user-ratio modeling for compressed video[END_REF][START_REF] Wang | Prediction of satisfied user ratio for compressed video[END_REF][START_REF] Zhang | Deep learning based just noticeable difference and perceptual quality prediction models for compressed video[END_REF][START_REF] Lin | Sur-featnet: Predicting the satisfied user ratio curve for image compression with deep feature learning[END_REF][START_REF] Fan | Sur-net: Predicting the satisfied user ratio curve for image compression with deep learning[END_REF]. In this work, the individual JND annotations for each subject are considered as the opinion scores. Additionally, the q%SUR is equivalent to the p th percentile (p = 1-q, see Eq. [START_REF] Zhu | Subjective test methodology optimization and prediction framework for just noticeable difference and satisfied user ratio for compressed hd video[END_REF] in Sec.3) of opinion score. The opinion scores were used for MOS/CI validation and parameter estimation experiments as well as to measure the impact of POS recovery on the accuracy of SUR prediction models.

Netflix Public: Netflix Public Dataset [START_REF]Netflix public dataset[END_REF] is a publicly available video quality dataset with 79 Processed Video Sequences (PVS) where each evaluated by 26 subjects. We used the opinion scores for MOS/CI validation and parameter estimation experiments.

PROPOSED MODEL

Let o i,j be the opinion score annotated by subject i for stimulus j. For a subjective dataset that consist of m stimulus and have been evaluated by n subjects, the original annotation can be represented by a matrix O ∈ R n×m . For every stimulus, we first compute the mean and standard deviation of the opinion score annotated by each subject:

m(j) = 1 n n i=1 o i,j , where j = 1, 2, . . . m (1)
s(j) =   1 n n i=1 (o i,j -m(j)) 2   , where j = 1, 2, . . . m
(2) where m, s ∈ R 1×m . Afterwards, we acquire the Z-score matrix Z from the the raw opinion score matrix O as:

Z = O -Im Is (3)
where

I = [1, 1, . . . 1] T and I ∈ R n×1 .
Each element z i,j in matrix Z represents the number of standard deviations by which the opinion score o i,j is away from the m j . The following analyses are mainly based on the Z-score matrix Z.

Subject bias and inconsistency

Let B ∈ R 1×n and C ∈ R 1×n the vector of bias and inconsistency of n subjects respectively. Bias and inconsistency for subject i is calculated with the mean and standard deviation of the Z-score for subject i across all stimulus:

B(i) =   1 m m j=1 z i,j   , where i = 1, 2, . . . n (4)
C(i) =   1 m m j=1 (z i,j -B(i)) 2   , where i = 1, 2, . . . n (5)
The key distinction between the estimation of subject bias in ZREC and P913.12-4 is that ZREC describes the subject bias in the standard deviation range of each stimulus, while P913.12-4 describes it in the opinion score range. By modeling subject bias in the standard deviation range of individual stimuli, ZREC takes stimulus ambiguity into account.

Content ambiguity

It is important to clarify the difference between stimuli ambiguity and content ambiguity. In QoE datasets, multiple stimuli (i.e., PVS) can be generated from a unique source content (i.e., SRC). We define the stimulus ambiguity for j as the standard deviation of subjects' opinion score (Eq.( 2)). Consequently, content ambiguity is defined as the mean ambiguity of all stimuli that belong to a particular content l:

A(l) =   1 h j∈g s(j)   , where l = 1, 2, . . . t (6)
h is the number of stimulus for content l, g the list of stimulus index of content l, t is the total number of contents in the entire datasets, A ∈ R 1×t .

Mean opinion score recovery

We first remove the bias of each subject for each stimuli from the original annotation O. The unbiased opinion score matrix U ∈ R n×m is calculated with:

U = O-B T s (7)
u i,j is the element of U, which is the opinion score of subject i for stimuli j after the removal of the bias of subject i. In Eq.( 7), we multiply the bias of each observers B T ∈ R n×1 with the standard deviation s ∈ R 1×m of different subjects' opinion score for every stimuli in Eq.( 2) in order to re-scale the Z-score to the original opinion score range.

To calculate the recovered MOS of stimuli j, denoted R(j), we employ a weighting scheme that takes into account the inconsistency of the opinion scores provided by different subjects. Specifically, instead of simply averaging the unbiased scores u j across all subjects, we use a weighted average of u j , where the weight assigned to each score is inversely proportional to the subject's inconsistency. This means that subjects with higher inconsistency are given less weight, and their opinion scores have less influence on the final MOS calculation.

R(j) =     n i=1 C(i) -2 u i,j n i=1 C(i) -2     , where j = 1, 2, . . . m (8)
Similar with the recovered MOS, weighted standard devi-ation is calculated as:

σ w (j) =      n n -1 × n i=1 C(i) -2 (u i,j -R(j)) 2 n i=1 C(i) -2      (9)
Where j = 1, 2, ...m. The factor of n/(n -1) is intended to account for the number of degrees of freedom, thus giving us an unbiased estimation of the standard deviation of the population [START_REF] Bevington | Data reduction and error analysis for the physical sciences[END_REF]. The 95% CI is thus computed with: Require: subject inconsistencies, C n Require: percentile to be calculated, p number of subjects = n, number of stimuli = m total weight of the population, w = sum(C -2 n ) percentile weight, w p = w × p/100 initialize Q p , a zero vector (with size=m) to store p th percentile opinion score for each stimuli for each stimuli j in m do U n ← get subject opinions for stimuli j from U n,m U n-sorted ← sort U n in ascending order w n-sorted ← sort C -2 n with same indices as U n-sorted initialize current weight, w c = 0 initialize current subject index, i = 0 while w c < w p do Q p (j) ← get current subject (i) opinion from U n-sorted and set it as the p th percentile score w ← get current subject (i) weight from w n-sorted w c ← w c + w i ← i + 1 end while end for return Q p Some subjective/objective studies are not interested in mean of opinion scores but the percentile of opinion scores. For JND and SUR studies [START_REF] Zhu | Subjective test methodology optimization and prediction framework for just noticeable difference and satisfied user ratio for compressed hd video[END_REF][START_REF] Zhang | Satisfied-user-ratio modeling for compressed video[END_REF][START_REF] Zhang | Deep learning based just noticeable difference and perceptual quality prediction models for compressed video[END_REF][START_REF] Lin | Sur-featnet: Predicting the satisfied user ratio curve for image compression with deep feature learning[END_REF][START_REF] Fan | Sur-net: Predicting the satisfied user ratio curve for image compression with deep learning[END_REF][START_REF] Wang | Analysis and prediction of jnd-based video quality model[END_REF], 75%SUR is commonly used to train and evaluate objective metric. It can be easily proved that for a given stimuli j:

CI(j) = R(j) ± 1.96 σ w (j) √ n (
q%SUR(o i ) = (1 -q) -th percentile(o i ), (11)
75%SUR is in fact 25 th percentile. Therefore, we provide a weighted percentile approach where subject bias and inconsistency is taken into account. Algorithm 1 depicts the process to calculate weighted percentiles Q p of an unbiased opinion score matrix U n,m of n subjects and m stimuli for a given percentile p in range [0, 100]. To evaluate the comparative reliability of the confidence intervals generated by each method, we performed a bootstrapping analysis comprising 1000 iterations. In each iteration, we randomly selected half of the subjects and recovered the MOS with each model. Results shows that ZREC and P913.12-6 exhibit the lowest average CI values while maintaining a relatively high CI% level. Despite having a higher CI% value, P913.12-4 displays a significantly larger average CI compared to ZREC and P913.12-6.

Estimated parameters

In this section, we analyze the correlation between the subject bias, inconsistency and content ambiguity across the tested models. As summarized in the Table 1, BT500 does not estimate any of the parameters and thus excluded from the correlation analysis. Moreover, P913.12.4 cannot estimate subject inconsistency and content ambiguity while P913.12.6 cannot estimate content ambiguity. In addition, MLE fails to con- verge to a solution for HD-VJND dataset. Table 3 depicts the PLCC values between the indicated model pairs in each row. The results indicate that the tested models are well correlated in terms of subject bias. On the other hand, estimated subject inconsistencies show slight differences between models. Finally, MLE and ZREC shows relatively high correlations in terms of content ambiguity. Despite the lower correlations for subject inconsistencies, ZREC estimations are in line with the standards. It is impossible to know which model estimations are closer to the ground truth, however the analysis showcases the relative reliability of the approach.

Impact of percentile opinion score recovery on the accuracy of SUR prediction models

Previous work [START_REF] Perrin | When is the cleaning of subjective data relevant to train ugc video quality metrics?[END_REF] has shown that training objective quality models on cleaned data can improve the prediction performance. In this work, we compared the performance of the 75%SUR prediction model [START_REF] Zhu | Subjective test methodology optimization and prediction framework for just noticeable difference and satisfied user ratio for compressed hd video[END_REF] trained on 75%SUR from original datasets without recovery and 75%SUR (25 th percentile) recovered by ZREC and P913-12.6 respectively. Because P913-12.6 only provide MOS recovery but not percentile recovery, we use Algorithm 1 with the subject bias and inconsistency of P913-12.6 as input. The mean and variance of absolute error of 75%SUR for different training data are shown in Table 4. It can be observed that the 75%SUR prediction model trained both on ZREC and P913-12.6 improved the prediction, in which ZREC get a smaller prediction error than P913-12.6.

CONCLUSION

We introduced ZREC to estimate subject bias, inconsistency and content ambiguity, all of which are fundamental for QoE studies. Using these parameters, ZREC can recover the MOS and the POS whichever is more suitable for the QoE use-case in question. Our findings indicate that ZREC can produce slightly tighter CIs for MOS recovery on two datasets compared to the current state of the art models, albeit with a minor reduction in accuracy. A tighter CI allows to reduce the required number of subjects in the subjective study without sacrificing from the accuracy and resolving power. Furthermore, the results of our experiments on the SUR prediction use-case demonstrate that ZREC can improve the performance of objective quality metrics by providing a more reliable ground truth with 25 th POS recovery.

Table 1 .

 1 Summary of the estimated parameters by each mos recovery model.

		BT500 P913-12.4 P913-12.6 MLE ZREC
	Subject Inconsistency	✗	✗	✓	✓	✓
	Subject Bias	✗	✓	✓	✓	✓
	Content Ambiguity	✗	✗	✗	✓	✓

Equal contribution * available at: https://github.com/kyillene/ZREC

  10) 3.4. P th percentile opinion score recovery Algorithm 1 Calculate Q p , weighted p th percentile opinion scores Require: unbiased subject opinions matrix, U n,m

Table 2 .

 2 Analysis of the CI of the recovered MOS for four different methods on the Netflix Public and the HD-VJND datasets. Avg CI represents the length of the CI for each method on each dataset with all subjects included in MOS recovery. CI% represents the percentage of recovered MOS values that fall within the confidence interval range based on 1000 bootstrapping iterations. In each iteration, only half of the subjects in each dataset are used to recover the MOS.

		NETFLIX	HD-VJND
		Avg CI	CI%	Avg CI	CI%
	BT500	0.5153 0.5645 1.2612 0.9285
	P913-12.4 0.4986 0.9102 1.1254 0.8671
	P913-12.6 0.4420 0.8885 1.0217 0.8805
	ZREC	0.4172 0.8783 0.9813 0.8554

Table 3 .

 3 Pearson linear correlation coefficient (PLCC) between the estimated parameters of subject inconsistency, subject bias, and content ambiguity across various models. Table 2 depicts the average 95% CI of the recovered MOS on the Netflix Public and HD-VJND datasets with all subjects.

		Subject	Subject	Content
		Inconsistency	Bias	Ambiguity
	Model Pair		NETFLIX
	MLE -ZREC	0.9282	0.9952	0.9663
	MLE -P913.12-6	0.9669	0.9964	-
	P913.12-6 -ZREC	0.9372	0.9965	-
	P913.12-4 -MLE	-	0.9992	-
	P913.12-4 -P913.12-6	-	0.9999	-
	P913.12-4 -ZREC	-	0.9965	-
	Model Pair	HD-VJND
	P913.12-6 -ZREC	0.9603	0.9994	-
	P913.12-4 -P913.12-6	-	0.9999	-
	P913.12-4 -ZREC	-	0.9994	-
	4. EXPERIMENT RESULTS
	4.1. MOS recovery and confidence intervals

Table 4 .

 4 The mean and variance of absolute errors on 75%SUR prediction with SUR prediction model[START_REF] Zhu | Subjective test methodology optimization and prediction framework for just noticeable difference and satisfied user ratio for compressed hd video[END_REF] on HD-VJND dataset without any recovery and with ZREC and P913.12-6 POS recovery.

	|∆75%SUR|	Without Recovery[11]	P913-12.6 POS Recovery	ZREC POS Recovery
	Mean Error	0.7489	0.7175	0.6883
	Error Variance	0.9224	0.7198	0.6989