Shaoguo Wen

Suiyi Ling

Junle Wang

Ximing Chen

Lizhi Fang

Yanqing Jing

Patrick Le Callet

SUBJECTIVE AND OBJECTIVE QUALITY ASSESSMENT OF MOBILE GAMING VIDEO

Keywords: Subjective quality assessment, objective quality metric, gaming video, model distillation

Nowadays, with the vigorous expansion and development of gaming video streaming techniques and services, the expectation of users, especially the mobile phone users, for higher quality of experience is also growing swiftly. As most of the existing research focuses on traditional video streaming, there is a clear lack of both subjective study and objective quality models that are tailored for quality assessment of mobile gaming content. To this end, in this study, we first present a brand new Tencent Gaming Video dataset containing 1293 mobile gaming sequences encoded with three different codecs. Second, we propose an objective quality framework, namely Efficient hard-RAnk Quality Estimator (ERAQUE), that is equipped with (1) a novel hard pairwise ranking loss, which forces the model to put more emphasis on differentiating similar pairs; (2) an adapted model distillation strategy, which could be utilized to compress the proposed model efficiently without causing significant performance drop. Extensive experiments demonstrate the efficiency and robustness of our model.

INTRODUCTION

Gaming video streaming is composed of a wide-range of online services including gaming video streaming, esports broadcasting and cloud gaming services. In the past decade, the booming international popular gaming video streaming platforms including Twitch, YouTube, Smashcast, Afreeca TV, Gosu Gamers, etc., and cloud gaming platforms like Google Stadia and Nvidia Geforce Now, are gradually taking a large share of video streaming [START_REF] Barman | Gamingvideoset: a dataset for gaming video streaming applications[END_REF]. Most of these services have also mobile app to meet the growing prevalence of mobile usage in modern life. According to [START_REF] Ling | Towards better quality assessment of high-quality videos[END_REF], gaming content also makes up a significant proportion of of User Generated Contents (UGC) on social media platforms, where a certain amount of users browse/view the contents using their mobile phone by a daily base. As reported in [START_REF] Zuckerman | 13 CURRENT GAMING TRENDS: 2020/2021 DATA, STATISTICS & PREDICTIONS[END_REF], mobile gaming accounted for 58.8% of the digital games market in 2019. This percentage is growing even more dramatically due to the Suiyi Ling and Shaoguan Wen make equal contribution, Junle Wang is the corresponding author.

global COVID-19 pandemic in 2020. Users' higher requirements for better Quality of Experience (QoE) necessitate robust quality control of these gaming contents. Different from the contents on common video streaming platforms [START_REF] Ling | Towards better quality assessment of high-quality videos[END_REF], quality assessment of gaming videos raise new challenges to the community as: their temporal complexity could be significantly higher; unlike natural videos, they are graphical rendered/generated contents; most of the gaming streaming platform requires real-time quality evaluation; and cloud gaming is sensitive to delay. Notwithstanding the fact that subjective study is time-consuming and expensive to conduct, it is still the foundation for the development of rigorous objective quality metrics. As most of the existing subjective studies focus mainly on video on traditional Video on Demand streaming services [START_REF] Barman | Gamingvideoset: a dataset for gaming video streaming applications[END_REF], studies conducted for gaming contents, especially for mobile applications, are still scarce.

With the burgeon of video encoding techniques and hardware based acceleration frameworks, dedicated fast codecs have been developed to alleviate relevant stresses. The development of the video objective quality metrics should also keep pace with the one of the codecs. Among No-Reference (NR), Reduced-Reference (RR), and Full-Reference (FR) metrics, NR metrics are of greater value, and significantly more piratical for real-time gaming content quality evaluation [START_REF] Zadtootaghaj | Demi: deep video quality estimation model using perceptual video quality dimensions[END_REF] since pristine reference is not always existing or accessible. Advanced deep learning based NR quality metric enjoyed a big leap in performance in recent years. The development of Deep Learning (DL) has brought along a new wave of NR quality assessment models that perform markedly better than traditional metrics. However, very few of them target the gaming application, typically for mobile users. Recall that most of those DL based models are complex, cumbersome and thus barely could be deployed on mobile directly for real-time prediction.

Based on the discussions above, in this study, we present a novel large-scale mobile gaming video data set, and a brand new Efficient hard-RAnk Quality Estimator (ERAQUE) to remedy the lack of existing subjective and objective studies.

RELATED WORK

Subjective study: Recently, an image dataset for gaming content was released [START_REF] Ling | A subjective study of multi-dimensional aesthetic assessment for mobile game image[END_REF], where each individual image was labeled with four dimensions including the 'overall quality' dimension. However, in this study the 'overall quality' was subjectively assessed from an aesthetic perspective of view.

arXiv:2103.05099v1 [cs.CV] 27 Jan 2021

The 'GamingVideoSET' [START_REF] Barman | Gamingvideoset: a dataset for gaming video streaming applications[END_REF], consists of 576 sequences encoded using H.264 from 24 gaming content. KUGVD [START_REF] Barman | No-reference video quality estimation based on machine learning for passive gaming video streaming applications[END_REF] is another public available gaming video dataset, but only 6 source contents were collected. Cloud Gaming Video Dataset (CGVDS) [START_REF] Zadtootaghaj | Quality estimation models for gaming video streaming services using perceptual video quality dimensions[END_REF] is one of the largest existing public gaming video dataset in the domain. Although the number of considered contents in this subjective study is slightly larger than the one in GamingVideoSET, more training data is still required for mainstream deep learning models. Most importantly, none of the above subjective studies were conducted for mobile games, with mobile phones.

Objective models: The demand of No-Reference quality metrics is growing significantly, especially for real-time quality control. Several dedicated blind quality metrics were developed to meet the rapidly growing need. In [START_REF] Utke | Towards deep learning methods for quality assessment of computer-generated imagery[END_REF], Utke et al. explored different Convolutional Neural Network (CNN) architectures, e.g., DenseNet 121 , without the help of reference video, to predict VMAF score for gaming videos. A blind quality metric was presented in [START_REF] Zadtootaghaj | Nr-gvqm: A no reference gaming video quality metric[END_REF], where Support Vector Regression (SVR) was applied to regress quality scores using frame-level index. Recently, the G.1072 planning model [START_REF]Opinion model predicting gaming quality of experience for cloud gaming services[END_REF], which evalutes the quality of gaming videos based on the videos' information, including metadata, codec information, etc., was summarized in the ITU-T recommendation standard (ITU-T.G1072). Similarly, a parametric video quality metric was introudced in [START_REF] Saman Zadtootaghaj | Quality estimation models for gaming video streaming services using perceptual video quality dimensions[END_REF] by Zadtootaghaj et al. Among existing NR video metrics, DEMI [START_REF] Zadtootaghaj | Demi: deep video quality estimation model using perceptual video quality dimensions[END_REF] is one of the most recent stateof-the-art video quality gaming metric. It was developed by training a CNN to capture videos distortions guided by full reference quality metric, finetuning the obtained network on a smaller quality datasets, and finally predicting the quality scores utilizing Random Forest. Following a similar recipe, a three-step framework NDNetGaming [START_REF] Utke | NDNetGaming -Development of a No-Reference Deep CNN for Gaming Video Quality Prediction[END_REF] was designed using also CNN, with a novel temporal pooling methodology that considers the temporal masking effect. However, none of them were designed and tested on mobile gaming contents.

THE SUBJECTIVE STUDY

In this section, details of our novel Tencent Gaming Video (TGV) dataset and the subjective experiment are provided. Overall information of the dataset is summarized in Table 1. Content: Our gaming video dataset contains 1293 sequences. In total 150 source gaming videos were collected from 17 mobile games. These games are available from the Tencent internal gaming center. Thumbnails of selected mobile gaming contents are shown in Fig. 1. There different codecs, including the H264, H265, and one internal Tencent codec, were employed to encode each of the content with a variety of different setting/configurations, e.g., CRF values were selected randomly from [1, ..., 50]. Environment & Observers: The Absolute Category Rating (ACR) subjective protocol was used for the test. During the experiment, 19 participants with normal or corrected-tonormal acuity were asked to score the gaming videos. Among them, there were 13 males, and 6 females, most of them are between 23-26 years old. The subjective experiment was conducted in an regularized office room with, where illuminance, etc. was controlled according to [START_REF] Itu | Methodology for the subjective assessment of the quality of television pictures[END_REF]. During the subjective experiment, the observers was asked to launch a internally developed platform to start the test, using their own mobile phone. In another word, different models of mobile phone were utilized by different participants to conduct the subjective test, which is consistent with real application scenario. After collecting the subjective data, two outliers were removed using the screening tools recommended by [START_REF] Itu | Methodology for the subjective assessment of the quality of television pictures[END_REF].

THE PROPOSED OBJECTIVE MODEL

Network Architecture

The architecture of the proposed ERAQUE for gaming video quality assessment is depicted in Fig. 2. In quality domain, due to limited training data, deep complex models are prone to overfitting [START_REF] Ling | Re-visiting discriminator for blind free-viewpoint image quality assessment[END_REF]. Therefore, the first part of the network is a light-weight backbone network. Subsequently, the Global Averaged Pooling [START_REF] Szegedy | Going deeper with convolutions[END_REF] is plugged in to obtain more streamlined latent representation following with two Fully Connected (FC) layers to output the final quality score. When predicting quality score for gaming videos, frames are first extracted and fed into the model, and then the frame-wise scores are averaged to obtain the final quality score of the video. Based on different use case under different scenarios, network like Mobilenet [START_REF] Andrew G Howard | Mobilenets: Efficient convolutional neural networks for mobile vision applications[END_REF], Shufflenet [START_REF] Zhang | Shufflenet: An extremely efficient convolutional neural network for mobile devices[END_REF], ResNet-18 etc., could be employed accordingly. In this study, we start with the classical ResNet 18 network that was pre-trained on Imagenet as backbone, and finetune it using quality data. This model is further compressed our re-adapted model distillation strategy. Details are shown in Section 4.3.

Loss function

Let ŷ and y be the quality score predicted by the objective model, and the ground truth quality score collected from the subjective experiment respectively. n is the total number of considered videos. The loss function of our proposed model is composed of two parts as defined below, where λ is a parameter that balances the two losses:

L = L mae + λ • L hard-rank . (1)
The first part is the Mean Absolute Error (MAE) loss L mae between the ground truth and predicted scores:

L mae = n i=1 | ŷi -y i | n . (2)
The second part is a hard pair-wise ranking loss L ij hard-rank that is inspired by the metric learning based framework proposed in [START_REF] Liu | Rankiqa: Learning from rankings for no-reference image quality assessment[END_REF]. Given a pair of stimuli (y i , y j ), the proposed L ij hard-rank is designed as:

L ij hard-rank = l 2 (y i , y j )•max(0, |y i -y j |-l 1 (y i , y j )•( ŷi -ŷj )), (3)
where l 1 and l 2 are further defined in equation ( 4) and ( 5) correspondingly, and τ is a margin (similar to the ones defined in [START_REF] Liu | Rankiqa: Learning from rankings for no-reference image quality assessment[END_REF]):

l 1 = 1, y i ≥ y j -1, otherwise (4)
l 2 = 1, 0 < |y i -y j | ≤ τ 0, otherwise (5)
In the quality domain, it is easier for most of the objective models to predict the quality scores for that are of significantly different quality [START_REF] Ling | Strategy for boosting pair comparison and improving quality assessment accuracy[END_REF][START_REF] Li | Hybridmst: A hybrid active sampling strategy for pairwise preference aggregation[END_REF]. If the visual difference of two considered stimuli is high enough for observers to differentiate them, the obtained subjective data would be of less bias and errors [START_REF] Li | Gpm: A generic probabilistic model to recover annotator's behavior and ground truth labeling[END_REF], and thus cause less uncertainty for the prediction of objective quality models [START_REF] Ling | Rate-distortion video coding and uncertainties: to be blindly chasing marginal improvement or to be greener[END_REF]. Therefore, following a similar concept of hard sample defined in [START_REF] Ling | Few-shot pill recognition[END_REF], for quality assessment of gaming video, we consider the pairs that have similar quality scores as hard samples. Intuitively, l 2 is a gating function, where only hard sample pairs that have close quality scores (i.e.,0 < |y i -y j | ≤ τ ) are taken into account, and L ij hard-rank is thus named as the 'hard pair-wise ranking loss'. In this study, we first normalize the ground truth quality scores into a range of [0, 1], and set τ = 0.1 empirically.

Model Distillation

In general, when using ResNet-like models following a defacto standard, the network is commonly constructed by stacking a certain number of blocks. These models could be divided into 4 stages, where each stage contains a certain number of repeated residual blocks following by a sequence of convolution layers that downsample the input feature. For ResNet18, there are 2 residual block at each stage. Obviously, the more blocks there is, the more sophisticated the model is, i.e., more parameters. Knowledge distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF][START_REF] Zagoruyko | Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer[END_REF] can transfer knowledge from a complex model, i.e., the teacher model, to a lighter version of itself, i.e., the student model. This 'teacher-student' model compression technique, is of great potential to obtain lighter network with more stationary performance, as training a small model from the scratch often ends up encountering issues like underfitting.

The concise overview of our re-adapted teacher-student training framework for model distillation is depicted in Fig. 3. In the framework of the proposed objective model, the backbone ResNet18 is utilized as the teacher model within the knowledge distillation pipeline. In concrete words, the backbone ResNet18 was first trained until the performance no longer increases, and was freezed during the distillation procedure, as shown in the upper part of Fig. 3.

Afterwards, ResNet18-tiny was used as the student model, where there was only 1 block at each stage and the remaining architecture is the same as original ResNet18, as presented in the lower part of Fig. 3. Unlike the default setting considered in common classification task, essentially, our model is a regression model. Therefore, the knowledge distillation loss proposed in [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] is not applicable in our use case. Hence, the Knowledge distillation (KD) loss l kd is re-adapted as:

l kd = |ŷ s -ŷt |, (6)
where ŷs and ŷt are the predicted scores from student model and teacher model respectively.

It is worth noting that, throughout the knowledge distillation process, the student network does not only learn how to predict better the quality score, but also the latent representation using the intermediate convolution layers. Inspired by the attention mechanism proposed in [START_REF] Zagoruyko | Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer[END_REF], their attention loss is re-adapted to distillate intermediate feature maps between the student model and teacher model. Let {A} be the set of feature map output from the aforementioned 4 stages in the ResNet network, and A i denotes feature map output from the i th stage. Then, we have:

A i = C i i=1 |A i |, (7)
where C i is the i th channel of A i . The attention loss used for transfer attention could be then defined as:

l i at = 1 2n i • vec( i stu ) ||vec( A i stu )|| 2 - vec( A i tea ) ||vec( A i tea )|| 2 2 , (8)
where vec(•) is the vectorization operation, n i represents the total number of nodes in A i . Finally, the total loss L T -S for training the student model could be written as:

L T -S = L org + l kd + 1 4 4 i=1 l i at (9)
where L org is the distillation-free loss given in equation ( 1).

EXPERIMENT

The proposed TGV dataset was employed to benchmark the NR metrics designed for quality assessment of gaming videos as summarized in section 2. The entire dataset was divided into 80%, 20% as training and hold-out testing sets respectively. It is worth mentioning that there is no overlap in terms of gaming contents between the training and testing set to ensure the generality of the model. Apart from those NR metrics, 4 commonly used, especially in treaming industry [START_REF] Ling | Towards better quality assessment of high-quality videos[END_REF], full reference metrics were also tested, including the Peak Signal to Noise Ratio (PSNR) [START_REF] Wang | Mean squared error: Love it or leave it? a new look at signal fidelity measures[END_REF], the Structural Similarity Index (SSIM) [START_REF] Wang | Image quality assessment: from error visibility to structural similarity[END_REF], the Multi-Scale Structural Similarity (MS-SSIM) [START_REF] Wang | Multiscale structural similarity for image quality assessment[END_REF], and the Video Multi-Method Assessment Fusion (VMAF) [START_REF] Li | Vmaf: The journey continues[END_REF]. To evaluate the performances of the considered objective quality metrics, commonly used performance evacuation methodology including the Pearson correlation coefficient (PCC), the Spearman's rank order correlation coefficient (SCC), the Kendall correlation coefficient (KCC) and the Root Mean Squared Error (RMSE) are computed between the mean opinion scores and the predicted quality scores.

During fine-tuning (training), all the frames extracted from videos were first rescaled to 1080P with zero paddings. To augment the data, the input frames were further randomly cropped into 540 × 960, and flipped left to right or vice versa. The initial learning rate was set as 10 -04 , and the Cosine Annealing learning rate decay strategy was applied. We used the Adam optimizer, and the training was stopped after 50 epochs. To avoid overfitting, weight decay was set as 5 -04 . λ in equation ( 1) was set to 1.

Overall performance: The results are shown in Table 5, where the best performances are highlighted in bold. ERAQUE teacher , ERAQUE student and ResNet18 tiny, are the original teacher backbone model, the student model after distillation and the ResNet18 tiny that is trained from scratch correspondingly. In general, the proposed ResNet18 teacher achieves the best performance among all the compared metrics, including the full reference metrics. It even far surpasses the other CNN based state-of-the-art metric NDNetgaming, with PCC values of 0.9646 vs. 0.8579. The distilled version of our model also outperforms the other metrics. These observations demonstrate the effectiveness of our approach. The performances of the two ResNet18 tiny models also elucidate the advantage of training light-weight models via knowledge distillation. Advantage of knowledge distillation: Information of the teacher, student networks are summarized in Table 5, where #para indicates the number of network parameters. It is obvious that the complexity of the student network is around half of the teacher model in any respect. To further verify whether the difference of performance between the teacher and student is significant, i.e., whether the predicted scores are significantly different, the F-test analysis based on the residual difference between the predicted objective scores and the subjective Mean Opinion Score (MOS) values as described in [START_REF] Ling | Prediction of the influence of navigation scan-path on perceived quality of free-viewpoint videos[END_REF] was employed. The obtained results indicate no significant different between the performances of the proposed model and the distilled model, which demonstrates the efficiency of the teacher-student model distillation strategy.

CONCLUSION

In this study, a large-scale subjective study for the quality assessment of mobile gaming videos. According to our best knowledge, our TGV is one of the largest existing relevant dataset. To better quantify the quality of mobile gaming contents, a novel quality assessment framework has also been presented. According to the experimental results, the proposed metrics outperform state-of-the-art quality metrics, and our model distillation strategy could achieve high trade-off between the model complexity and the performance.

Fig. 1 .

 1 Fig. 1. Thumbnails of selected mobile gaming contents.Environment & Observers: The Absolute Category Rating (ACR) subjective protocol was used for the test. During the experiment, 19 participants with normal or corrected-tonormal acuity were asked to score the gaming videos. Among them, there were 13 males, and 6 females, most of them are between 23-26 years old. The subjective experiment was conducted in an regularized office room with, where illuminance, etc. was controlled according to[START_REF] Itu | Methodology for the subjective assessment of the quality of television pictures[END_REF]. During the subjective experiment, the observers was asked to launch a internally developed platform to start the test, using their own mobile phone. In another word, different models of mobile phone were utilized by different participants to conduct the subjective test, which is consistent with real application scenario. After collecting the subjective data, two outliers were removed using the screening tools recommended by[START_REF] Itu | Methodology for the subjective assessment of the quality of television pictures[END_REF].

Fig. 2 .

 2 Fig. 2. Architecture of the proposed quality metric.

Fig. 3 .

 3 Fig. 3. Teacher-student network for model distillation.

Table 1 .

 1 Summary of out TGV dataset.

	Duration	5 sec
	Frame rate	30 f/s
	Resolution 480P, 720P, 1080P
	Bitrate	100, 200, ..., 10 4

Table 2 .

 2 Performances comparison of considered Metrics.

		PCC	SCC	KCC	RMSE
	Full Reference Metric (FR)
	PSNR [26]	0.7467	0.7348	0.5445	0.7077
	SSIM [27]	0.8709	0.8654	0.6763	0.5229
	MS-SSIM [28]	0.8492	0.8457	0.6536	0.5618
	VMAF [29]	0.9130	0.9102	0.7436	0.4341
	No Reference Metric (NR)
	GamingPara [11]	0.5568	0.4402	0.3268	0.9084
	ITU-T G.1072 [10]	0.0633	0.0534	0.0412	1.0628
	DEMI [4]	0.7536	0.7455	0.5382	0.7479
	NDNetgaming [12]	0.8579	0.8477	0.6578	0.5845
	ResNet18 tiny	0.8298	0.8219	0.6398	2.6031
	ERAQUE student	0.9646	0.9641	0.8436	2.6387
	ERAQUE teacher	0.9714	0.9712	0.8635	0.2697

Table 3 .

 3 Comparison of teacher, student models.

		#para	FLOPs	storage size of the model
	ERAQUE teacher	11 M	18.91 G	45 M
	ERAQUE student	5.4 M	9.28 G	21 M