Mihai Gabriel Constantin
email: mgconstantin@imag.pub.ro

Chen Kang

Gabriela Dinu

Frédéric Dufaux

Giuseppe Valenzise

Bogdan Ionescu

Using Aesthetics and Action Recognition-based Networks for the Prediction of Media Memorability

In this working note paper we present the contribution and results of the participation of the UPB-L2S team to the MediaEval 2019 Predicting Media Memorability Task. The task requires participants to develop machine learning systems able to predict automatically whether a video will be memorable for the viewer, and for how long (e.g., hours, or days). To solve the task, we investigated several aesthetics and action recognition-based deep neural networks, either by fine-tuning models or by using them as pre-trained feature extractors. Results from different systems were aggregated in various fusion schemes. Experimental results are positive showing the potential of transfer learning for this tasks.

INTRODUCTION

Media Memorability was studied extensively in recent years, playing an important role in the analysis of human perception and understanding of media content. This domain was approached by numerous scientists from different perspectives and fields of study, including psychology [START_REF] Brady | Visual long-term memory has a massive storage capacity for object details[END_REF][START_REF] Roger | Recognition memory for words, sentences, and pictures[END_REF] and computer vision [START_REF] Cohendet | VideoMem: Constructing, Analyzing, Predicting Short-term and Long-term Video Memorability[END_REF][START_REF] Shekhar | Show and recall: Learning what makes videos memorable[END_REF], while several works analyzed the correlation between memorability and other visual perception concepts like interestingness and aesthetics [START_REF] Gabriel Constantin | Computational understanding of visual interestingness beyond semantics: literature survey and analysis of covariates[END_REF][START_REF] Isola | What makes a photograph memorable[END_REF]. In this context, the MediaEval 2019 Predicting Media Memorability task requires participants to create systems that can predict the short-term and long-term memorability of a set of soundless videos. The dataset, annotation protocol, precomputed features, and ground truth data are described in the task overview paper [START_REF] Gabriel Constantin | Predicting Media Memorability Task at MediaEval 2019[END_REF].

APPROACH

For our approach, we used several deep neural network models based on image aesthetics and action recognition. For the first category, we fine-tuned the aesthetic deep model presented in [START_REF] Kang | Predicting Subjectivity in Image Aesthetics Assessment[END_REF]. It is based on the ResNet-101 architecture [START_REF] He | Deep residual learning for image recognition[END_REF]. For the action recognition networks, we used features extracted from the I3D [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF] and TSN [START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF] networks and attempted to augment these features with the C3D features provided by the task organizers. Finally, we performed some late fusion experiments to further improve the results of these individual runs. Figure 1 summarizes and presents these approaches. The approaches are detailed in the following.

Aesthetics networks

The aesthetic-based approach modifies the ResNet-101 architecture [START_REF] He | Deep residual learning for image recognition[END_REF], trained on the AVA dataset [START_REF] Murray | AVA: A large-scale database for aesthetic visual analysis[END_REF] for the prediction of image aesthetic value, following the approach described in [START_REF] Kang | Predicting Subjectivity in Image Aesthetics Assessment[END_REF]. This approach generates a deep neural model that can process single image aesthetics and must be fine-tuned to process the short and long term memorability of videos. To generate a training dataset that will support the fine-tuning process, we extracted key-frames in two ways: (i) key frames from the 4th, 5th, and 6th second of each sample; (ii) one key frame every two seconds to test multi-frame training. In the retraining stage of the network for the memorability task, the provided devset is randomly split into three parts, with 65% of the samples representing the training set, 25% the test set and 10% the validation set. We adapted the last layer for this task by creating a fully connected layer with 2,048 inputs and 1 output.

During the fine-tuning process, we applied mean square error as loss function, using an initial learning rate of 0.0001. We ran the training process for 15 epochs, with a batch size of 32.

Action recognition networks

Apart from the precomputed C3D features, we extracted the "Mixed_5" layer from the I3D network [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF], trained on the Kinetics dataset [START_REF] Kay | The kinetics human action video dataset[END_REF] and the "Inception_5" layer of the TSN network [START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF], trained on the UCF101 dataset [START_REF] Soomro | UCF101: A dataset of 101 human actions classes from videos in the wild[END_REF]. These features were used as inputs for a Support Vector Regression algorithm that generates the final memorability scores. We conducted preliminary early fusion tests with combinations of these features in order to select the best possible combinations, testing both each feature vector individually and all possible combinations of two feature vectors. We also employed a PCA dimensionality reduction, reducing the size of each vector to 128 elements. Finally, to train the SVR system, we used a random 4-fold approach, with 75% of the data representing the training set and 25% representing the validation set. We used parameter tuning for the SVR model, via a RBF kernel and performing a grid search with two parameters: the C parameter and the gamma parameter (taking values 10 k , where k ∈ [-4, ..., 4]).

Late fusion

We employed several late fusion schemes on the best performing systems, trying to benefit from their combined strengths. We used three different strategies for combining these scores, namely: (i) LFMax, where we took the maximum score for each media sample;

(ii) LFMin, where we took the minimum score; (iii) LFWeight, where each score from different samples was multiplied with a weight w.

We assigned each weight varying values according to the formula w = 1-r /c, where the rank r had the value 0 for the best performing system, 1 for the second best and so on, and c represents a coefficient that dictates rank influence on the weights.

EXPERIMENTAL RESULTS

The development dataset consists of 8,000 videos, annotated with short and long term memory scores, while the test dataset consists of 2,000 videos. The official metric used in the task is Spearman's rank correlation (ρ). The best performing systems in the development phase are selected, retrained on the whole devset by using the optimal parameters and lastly run on the testset data.

Results on the devset

During the tests performed on the devset, several systems and combinations of parameters stood out as best performers. Table 1 shows the performances recorded by the best performing aesthetic, action-based, and late fusion systems. We used several dataset variations in retraining the aestheticbased deep network. More precisely, we found that, for the shortterm memorability, the best performing systems were the ones trained with keyframes extracted from the 5th second and the ones extracted from the multi-frame approach. The results were both similar with a Spearman's ρ of 0.45. On the other hand, in the long-term memorability subtask we found that the best performing systems were the ones trained with keyframes from the 5th frame. Although this may seem somewhat surprising, giving that bigger data sets usually account for better results, we believe that the reason behind this is that each video contains only one scene. Therefore not much additional information is given to the system when more frames are extracted because the frames are very similar. However, we would also like to point out that the results for the other frame extraction schemes were not much lower than these.

Regarding the 3D action-recognition based systems, we noticed that individual systems, based on only one feature vector (TSN, I3D or C3D) had a low performance, with a Spearman's ρ score of under 0.42. This performance further dropped when we used the original vectors, without applying PCA reduction, therefore demonstrating the positive influence that dimensionality reduction has on the final results. Therefore we decided to apply an early fusion scheme, where we tested all the possible combinations of the feature vectors, by concatenating them. The best performing combinations were TSN + I3D and C3D + I3D.

Finally, in the late fusion part of the experiment, we generally decided to test late fusion schemes between the two action-recognition based systems and between the best performing action-recognition system (TSN + I3D) and the aesthetic-based system. In general, results for the LFMin systems were underperforming, while the LFMax systems were better than their components, but without bringing a significant increase in results. The best performing late fusion schemes proved to be based on LFWeight, more precisely using a c value of 5. This was an expected result, as it confirms some of our previous work in other MediaEval tasks [START_REF] Gabriel Constantin | LAPI at MediaEval 2017-Predicting Media Interestingness[END_REF].

Results on the testset

For the final phase, we retrained all the systems on the entire set of videos from devset, using the parameters computed in the previous phases and tested them on the videos from the testset. Table 1 presents also the results for this phase.

As expected, the best performance comes from a late fusion system using both aesthetic and action-based components (shortterm ρ = 0.477 and long-term ρ = 0.232). Generally, we observe that the system ranking for the submitted systems is consistent with the one we observed during the development phase, however, the results are lower than those predicted then, with significant drops in performance for the aesthetic-based system and the action-based (C3D + I3D) approaches. In terms of single-system performance, the action-based TSN + I3D system performs best, followed by the aesthetic-based system.

CONCLUSIONS

In this paper we presented the UPB-L2S approach for predicting media memorability at MediaEval. We created a framework that uses aesthetic and action recognition based systems and some late fusion combinations of these systems, that predict short-term and long-term memorability scores for soundless video samples. The results show that these systems are able to individually predict these scores, while the best results are achieved via late fusion weighted schemes. This enforces the idea of better exploiting transfer learning to tasks where labeled data are in particular hard to obtain.

Figure 1 :

 1 Figure 1: The diagram of the proposed solution.

Table 1 :

 1 Results of the proposed runs (preliminary experiments on devset, and official results on testset).

	Devset -Spearman's ρ	Testset -Spearman's ρ

ACKNOWLEDGMENTS

This work was partially supported by the Romanian Ministry of Innovation and Research (UEFISCDI, project SPIA-VA, agreement 2SOL/2017, grant PN-III-P2-2.1-SOL-2016-02-0002).


Chen Kang
email: chen.kang@l2s.centralesupelec.fr

Giuseppe Valenzise
email: giuseppe.valenzise@l2s.centralesupelec.fr

Frédéric Dufaux
email: frederic.dufaux@l2s.centralesupelec.fr

Predicting Subjectivity in Image Aesthetics Assessment

Keywords: Aesthetic quality, subjectivity, distribution prediction

Conventional image aesthetic quality prediction aims at predicting the average score of a picture or its aesthetic class (good/bad quality). However, aesthetic prediction is intrinsically subjective, and images with similar mean aesthetic scores/class might display very different levels of consensus by human raters. Recent work has dealt with aesthetic subjectivity by predicting the distribution of human scores. However, predicting the distribution is not directly interpretable in terms of subjectivity, and might be sub-optimal compared to directly estimating subjectivity descriptors computed from ground-truth scores. In this paper, we propose several measures of subjectivity, ranging from simple statistical measures such as the standard deviation of the scores, to newly proposed descriptors inspired by information theory. We evaluate the prediction performance of these measures when they are computed from predicted score distributions or when they are directly learned from groundtruth data. We find that the latter strategy provides in general better results, though there is still a large space for improvement in aesthetic subjectivity prediction.

I. INTRODUCTION

The goal of image aesthetic quality assessment is to determine how beautiful an image looks to a human observer. The automatic prediction of image aesthetic quality has received an increasing attention in the past few years in the multimedia community. This is due, on one hand, to the potential impact that aesthetic quality prediction has on applications such as image enhancement, recommendation or retrieval [START_REF] Deng | Image aesthetic assessment: An experimental survey[END_REF]. On the other hand, the availability of large-scale datasets with human annotations [START_REF] Murray | AVA: a large-scale database for aesthetic visual analysis[END_REF], [START_REF] Kong | Photo aesthetics ranking network with attributes and content adaptation[END_REF] has enabled the use of modern machine learning tools, such as deep learning, to predict aesthetic scores for images displaying a wide variety of contents and characteristics.

Most of existing aesthetic quality prediction approaches assume that aesthetic quality can be represented by a single value, e.g., the mean aesthetic score or the aesthetic class (good/bad). However, this assumption does not take into account the intrinsic subjectivity of aesthetic assessment, which may be influenced by personal background, interests, mood, etc. Indeed, experimental psychology studies show that, while beauty is conveyed by objective visual clues, the resulting aesthetic appraisal is subjective and depends on how the visual This work is founded by China Scholarship Council. The two images, displayed in the top-left panels, have similar mean score but different distribution of aesthetic judgments given by human raters, shown in the histograms on the top-right panels. The tables report several measures that compactly describe subjectivity based on the score distribution, which are described in Section III.

clues are processed by higher-level cognitive areas in the brain [START_REF] Reber | Processing fluency and aesthetic pleasure: Is beauty in the perceiver's processing experience?[END_REF]. As a result, summarizing aesthetic quality with a single value is not in general sufficient to capture the subjectivity of aesthetic perception, which we define in this paper as the degree of consensus about the aesthetic value of a picture when the latter is judged by a panel of human raters. The top two rows of Figure 1 illustrate this with an example: two images from the AVA dataset have a similar average aesthetic score, but a different degree of subjectivity. In the image in Figure 1(a), it is evident that humans tend to agree more on the aesthetic quality of the image, while the judgments are more dispersed for the image in Figure 1(b). Intuitively, being able to predict aesthetic subjectivity can provide valuable information in order to determine to which extent aesthetic predictions can 978-1-7281-1817-8/19/$31.00 ©2019 IEEE be trusted. This in turn could be beneficial in applications such as enhancement or retrieval, in order to obtain more reliable and accurate results.

Recent work has tackled aesthetic subjectivity by predicting the distribution of subjective scores of an image [START_REF] Jin | Image aesthetic predictors based on weighted CNNs[END_REF]- [START_REF] Jin | Predicting aesthetic score distribution through cumulative jensen-shannon divergence[END_REF]. Specifically, these methods leverage the availability of groundtruth aesthetic score distributions obtained by a large number of human annotations, offered by large-scale datasets such as AVA [START_REF] Murray | AVA: a large-scale database for aesthetic visual analysis[END_REF], and employ different loss functions to measure the distance between probability distributions. However, aesthetic subjectivity is described only implicitly by the score distribution. Instead, we are interested at quantifying and predicting this subjectivity explicitly through a scalar value that summarizes the score distribution by describing the raters' consensus, and which could be used by automatic image analysis algorithms.

Therefore, in this paper we analyze several measures of subjectivity based on the distribution of the scores, including simple statistical descriptors such as standard deviation, as well as new proposed features inspired by information theory. We evaluate the prediction accuracy of the proposed subjectivity measures using state-of-the-art aesthetic score distribution prediction, and compare these results with directly learning the subjectivity scores. Our experiments show that directly learning subjectivity measures leads, in general, to better performance than first predicting the score distribution and then computing subjectivity based on it. Despite the improvement obtained by our approach using several performance indicators, our findings show that predicting aesthetic subjectivity is a much more difficult task than predicting the average aesthetic score of a picture.

The rest of the paper is organized as follows. In Section II we review related work on image aesthetics prediction, and we present several subjectivity descriptors in Section III. We evaluate different prediction schemes for these measures in Section IV. Finally, Section V concludes the paper.

II. RELATED WORK

General aesthetics prediction deals with predicting image aesthetics for any kind of image content, in contrast to taskspecific aesthetics where the class or object of the picture is known, e.g., images of faces [START_REF] Bianco | Benchmark analysis of representative deep neural network architectures[END_REF]. In this work we focus on the general image aesthetics problem. Traditional methods for predicting image aesthetics have been employing handcrafted features to describe well-known photographic rules and perceptual attributes, such as clarity, depth of field, colorfulness, dynamic range, etc. [10]- [START_REF] Hulusic | Perceived dynamic range of HDR images[END_REF]. These methods have the advantage to provide an interpretable explanation of image aesthetics, but fail in capturing accurately complex aesthetic phenomena. As a result, these approaches tend to perform poorly when tested in real-world conditions with a wide content variety.

With the advent of deep learning and the availability of large-scale datasets with thousands or hundreds of thousands pictures [START_REF] Murray | AVA: a large-scale database for aesthetic visual analysis[END_REF], [START_REF] Kong | Photo aesthetics ranking network with attributes and content adaptation[END_REF], the accuracy of aesthetics prediction has been constantly improving. In this context, the problem of aesthetic assessment has been mainly formulated as predicting the average score or the aesthetic class of an image [START_REF] Deng | Image aesthetic assessment: An experimental survey[END_REF], [START_REF] Kong | Photo aesthetics ranking network with attributes and content adaptation[END_REF], [START_REF] Lu | Rapid: Rating pictorial aesthetics using deep learning[END_REF], [START_REF] Lu | Deep multi-patch aggregation network for image style, aesthetics, and quality estimation[END_REF].

On the other hand, the problem of subjectivity in image aesthetic quality assessment has been rarely studied. Park et al. [START_REF] Park | Personalized image aesthetic quality assessment by joint regression and ranking[END_REF] consider personal taste in addition to general aesthetic score, by adapting a model to match specific user preferences obtained from user interactions. Differently from that work, we do not target personalized aesthetic prediction, but rather aim at assessing the level of consensus of a panel of humans about the general aesthetic value of a picture.

Recently, a few studies have considered aesthetic subjectivity by predicting the distribution of human scores, rather than simply the mean score. Bin Jin et al. [START_REF] Jin | Image aesthetic predictors based on weighted CNNs[END_REF] predict the distribution of aesthetic scores based on a weighted loss which accounts for the non-uniform distribution of the scores in the AVA dataset, using a modified VGG-16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] network. Their loss function uses the chi-square distance to evaluate the distribution prediction. Later, Murray et al. [START_REF] Murray | A deep architecture for unified aesthetic prediction[END_REF] employ the Huber loss and spatial pyramid pooling [START_REF] He | Spatial pyramid pooling in deep convolutional networks for visual recognition[END_REF] to predict distributions. In the NIMA system [START_REF] Talebi | NIMA: Neural image assessment[END_REF], the loss function consists of the squared Earth Mover's Distance (EMD), which is shown to lead to better mean score prediction performance from the estimated distributions. Jin et al. [START_REF] Jin | Predicting aesthetic score distribution through cumulative jensen-shannon divergence[END_REF] propose to use the cumulative Jensen-Shannon Divergence (CJS-CNN) as loss function. They also present an extended version of this loss using a function of the kurtosis of ground-truth score distribution to weigh CJS (RS-CJS). Kurtosis is used as a proxy to aesthetic "reliability", and used to penalize more those images whose distribution is considered unreliable. In this paper, instead of predicting the score distributions, we propose (for the first time to our knowledge) to define explicitly subjectivity measures and directly predict them.

The majority of the above-mentioned works employ the benchmark AVA dataset [START_REF] Murray | AVA: a large-scale database for aesthetic visual analysis[END_REF] to learn distributions or average scores. It contains over 250,000 images from photography amateurs' websites, which is much larger comparing to previous datasets like CUHK-PQ [START_REF] Luo | Content-based photo quality assessment[END_REF], [START_REF] Tang | Content-based photo quality assessment[END_REF]. The images are collected by approximately 1400 challenges from viewers who voted integer scores in the range [START_REF] Deng | Image aesthetic assessment: An experimental survey[END_REF][START_REF] Datta | Studying aesthetics in photographic images using a computational approach[END_REF]. Compared to other datasets like AADB [START_REF] Kong | Photo aesthetics ranking network with attributes and content adaptation[END_REF] which only has around 5 voters for each image, the number of votes in AVA ranges between 78 and 549, and the average is around 210, thus enabling a more reliable estimation of score distributions. In this work, we also employ the AVA dataset to train and evaluate subjectivity prediction.

III. PREDICTION OF SUBJECTIVITY

We consider a dataset of N images {I n }, n = 1 . . . N , where each image has been voted by M n human raters on a discrete scale with k levels, s = {s 1 , . . . , s k }. We model the M n aesthetic scores x n for each image I n as a realization of a categorical random variable with distribution p n (x n ), which we approximate with the normalized sample histogram p n (x n ). Given p n (x n ), we define µ n and m n as the mean and median of x n , respectively. In order to describe the level of consensus of human raters about the aesthetic quality of a given image, we propose using the following measures:

• Standard Deviation (SD) of the score distribution, which describes the dispersion of the scores around the average score, that is:

SD n = k i=1 p n (i) • (x n (i) -µ n ) 2 . (1)
A higher value of SD indicates a lower consensus around the average score, and thus higher subjectivity. • Mean Absolute Deviation around the median (MAD), defined as the sample average deviation of the scores around the median score, that is:

M AD n = 1 M n Mn i=1 |x n (i) -m n |. (2)
As for SD, higher values of MAD imply higher subjectivity.

• Distance to Uniform Distribution (DUD). The entropy of a distribution characterizes the degree of uncertainty of the associated random variable, and could be in principle used to quantify subjectivity. However, entropy does not take into account the ordinal nature of aesthetic scores, as illustrated in Figure 2. Instead of measuring entropy, we consider the distance of the score distribution p n (x n ) from the distribution having the maximum entropy over s, which is the uniform distribution. We quantify this distance using the 2-Wasserstein metric1 d W (p n , u s ), that is:

DU D n = d W (p n , u s ) = k i=1 (P n (i) -U s (i)) 2 1/2 , (3)
where u s is the discrete uniform distribution defined over the categories s, and P n and U s are the cumulative distribution functions of p n and u s , respectively.

A lower value of DUD implies that the score distribution is more similar to the uniform distribution, and thus the degree of subjectivity is higher.

• Distance from the Maximum Entropy Distribution (MED). Since the uniform distribution has always a mean value equal to the midpoint of the score scale, the DUD measure tends to penalize more skewed distributions having mean values close to the extremes of the quality scale. To overcome this bias, we compare the score distribution with the maximum entropy distribution over the quality scale having the same mean. More specifically, we look for a discrete distribution q s which solves the following optimization problem:

maximize q H(q) subject to µ [q] = µ n ,
where H denotes discrete entropy and µ[q] is the mean of q. It can be shown [START_REF] Cover | Elements of information theory[END_REF] that the solution of this problem is

q s (s i ) = e λsi k i=1 e λsi , ( 4
)
where λ is numerically found so that i s i q s (s i ) = µ n .

Then MED for image n is defined as:

M ED n = d W (p n , q s ) = k i=1 (P n (i) -Q s (i)) 2 1/2 , (5)
where Q s is the cumulative distribution of q s . As for the DUD measure, the lower MED is, the higher is the subjectivity of an image.

The table in Figure 1 shows an example of these measures computed for the two images in the top panel. We can observe that all of them capture correctly the degree of consensus of the score distributions. In the following, we will study how accurately each of these measures can be predicted, either directly or by means of predicted score distributions.

A. Subjectivity Prediction Framework

In order to predict the subjectivity measures proposed above, we consider two options: i) we predict the score distribution indirectly using an existing score prediction method as mentioned in Section II; or ii) we compute subjectivity measures on ground-truth scores, and learn to predict them directly.

Indirect subjectivity prediction: The underlying motivation of predicting score distributions lies in the possibility to derive aesthetic subjectivity [START_REF] Jin | Image aesthetic predictors based on weighted CNNs[END_REF], [START_REF] Murray | A deep architecture for unified aesthetic prediction[END_REF], [START_REF] Jin | Predicting aesthetic score distribution through cumulative jensen-shannon divergence[END_REF]. Therefore, we first consider state-of-the-art aesthetic distribution predictors (see Section II) to estimate the subjectivity measures introduced above, as illustrated in Figure 3(a). The advantage of this approach is that, once the distribution is estimated, one can compute any subjectivity measure from it. However, we will show experimentally that this approach is generally sub-optimal compared to directly estimating a subjectivity score. Direct subjectivity prediction: A limitation of the indirect subjectivity prediction is that the estimated distribution scores are generally noisy, as even the best method to predict the histogram of aesthetic scores has limited performance [START_REF] Jin | Predicting aesthetic score distribution through cumulative jensen-shannon divergence[END_REF]. In principle, assuming a deep neural network predicting aesthetic distribution approximates the maximum likelihood estimator [START_REF] Goodfellow | Deep learning[END_REF], a subjectivity estimator based on the predicted distributions is asymptotically efficient [START_REF] Kay | Fundamentals of statistical signal processing[END_REF]. However, in practice the number of samples used for training the network is finite, and prediction errors on the distribution might lead to worse prediction performance of subjectivity.

Therefore, we consider the alternative approach consisting in predicting directly the ground-truth subjectivity measures, as shown in Figure 3(b). The subjectivity measures are computed on the ground-truth aesthetic distribution. We use afterwards a deep convolutional neural network to predict these subjectivity measures.

IV. EXPERIMENTAL RESULTS

In this section we analyze the prediction performance of the aesthetic subjectivity measures introduced in Section III.

A. Experimental setup

We choose Resnet-34 as network structure to predict subjectivity. According to our experiments and the test in [START_REF] Bianco | Benchmark analysis of representative deep neural network architectures[END_REF], Resnet-34 provides similar accuracy result as VGG-16, but uses less memory. In addition, we also use Resnet-101 to study the influence of a deeper network structure in the direct aesthetic subjectivity prediction. The last (fully connected) layer of Resnet-34 is replaced by 3 fully connected layers: two 512 × 512 fully connected layers plus a 512 × 1 layer to give a 1-dimensional output. For Resnet-101, the two additional fully connected layers have size 2048 × 2048. The drop out rate is 0.5 for every fully connected layer.

We use Pytorch models pre-trained on ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF], and we fine-tune them using training images from the AVA dataset [START_REF] Murray | AVA: a large-scale database for aesthetic visual analysis[END_REF]. We use the standard test set of AVA as in previous work, which consists of 19,930 images. This leaves 260,264 images for train and validation. We randomly pick 23,553 pictures for validation, corresponding to approximately 10% of the training set size. All of the input images are resized to 224 × 224 pixels. Even though previous methods often augment data with horizontal flip, we decide not to do any kind of data augmentation, as differently from classification or recognition tasks, the ground truth in aesthetics is obtained by human raters and might be influenced by flipping. We employ Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] and a batch size of 64. The learning rate is decreased by 10 times when the loss does not change over two consecutive epochs. We fix the initial learning rate to 10 -5 and the maximum number of iterations to 40,000. We employ the L1 norm as loss function for the direct prediction of the subjectivity measures.

For the indirect subjectivity prediction, we consider the following three methods for predicting aesthetic score distributions: the work of Bin Jin et al. [START_REF] Jin | Image aesthetic predictors based on weighted CNNs[END_REF] (chi-square distance loss); NIMA [START_REF] Talebi | NIMA: Neural image assessment[END_REF] (Earth Mover's Distance loss); and the RSCJS method of Jin et al. [START_REF] Jin | Predicting aesthetic score distribution through cumulative jensen-shannon divergence[END_REF] (cumulative Jensen-Shannon divergence loss). Bin Jin et al. provide their trained model (using VGG-16), but use a different (smaller) test set than the standard AVA one. Since their test set is not provided, and for the sake of a fair comparison with other methods, We run their model on the standard AVA test set instead. For NIMA and RSCJS, the original code is not available, and we reimplemented them following the original papers. For NIMA and RSCJS, we use Resnet-34, modified as discussed above.

B. Performance Indicators

We evaluate the prediction of the subjectivity measures using 4 performance indicators:

C. Experiment Results

Tables I-IV show direct and indirect subjectivity prediction performance. We observe that direct subjectivity prediction always outperforms indirect prediction through distribution scores, for all the proposed subjectivity measures. In particular, for the same network complexity (Resnet-34), predicting directly the subjectivity is clearly better than predicting the score distribution first and computing subjectivity based on it. A possible explanation can be obtained by looking at the results of distribution prediction, as shown in Figure 4, which compares the average predicted aesthetic score distribution vs. the average ground-truth one. For the three distribution prediction methods considered here, we notice that, on average, predicted distributions are different from the original and may even be shifted. Notice that all of the proposed subjectivity measures are affected by errors in the prediction of the histogram. Although direct prediction improves all the considered performance indicators, we observe that overall the prediction performance is still not satisfactory, e.g., the SROCC is just slightly above 0.4. We might wonder whether this is due to a limited capacity of the Resnet-34 model we employed. Therefore, in order to study how subjectivity prediction performance improves with a more complex network, we tested the direct prediction scheme using Resnet-101, which is much deeper than Resnet-34. As expected, the results generally improve over the simpler Resnet-34. However, this improvement is in most case only marginal, showing that aesthetic subjectivity prediction is intrinsically a hard problem -at least a harder one than predicting the average aesthetic score, where SROCC between predicted and ground-truth values is higher than 0.6 [START_REF] Talebi | NIMA: Neural image assessment[END_REF].

Comparing the different subjectivity measures, those inspired by information theory (DUD and MED) are in general those with higher prediction performance. Among the statistical motivated descriptors, the SD is generally predicted more accurately than MAD. We can assume that, for the same neural network model complexity, a ground-truth variable which has a higher dependence on the input is easier to predict, or, in other terms, target variables which tend to be more "noisy" will be more difficult to learn. Thus, we can argue that the subjectivity measures based on information theory are somewhat more robust than statistical deviation measures. A possible rationale behind this could be that both DUD and MED are based on distances between histograms, which take into account the whole score distribution. On the other hand, SD completely captures data variability when the underlying score distribution is Gaussian, which is the case for only 62% of AVA images [START_REF] Jin | Predicting aesthetic score distribution through cumulative jensen-shannon divergence[END_REF]. MAD is supposed to be more robust to skewed distributions, but it might be affected by the sample median computation, which on a 10-dimensional distribution as for aesthetic scores can only take values over a small set, i.e., {1, 1.5, 2, . . . , 10}.

Notice that the DUD measure achieves the best correlation among the four subjectivity measures, despite the fact that it penalizes more those images with distributions having mean score far from the midpoint of the quality scale. These are also the images that which are less frequent in the AVA dataset. Therefore, DUD might implicitly act as a weighting scheme during learning, similar to [START_REF] Jin | Image aesthetic predictors based on weighted CNNs[END_REF]. However, this effect might be less evident on more balanced datasets, and should be verified by further experimental evidence. We leave this to future work.

V. CONCLUSION

In this paper we have analyzed the problem of defining and predicting aesthetic score subjectivity, intended as the degree of consensus human raters express about the aesthetic value of a picture. To this end, we have considered several measures of subjectivity, and two possible subjectivity prediction frameworks.

Among the analyzed descriptors of subjectivity, we have found that our newly proposed measures inspired by information theoretical principles are, in general, easier to learn, indicating that they might be more discriminative and robust compared to simpler statistical deviation measures.

We have shown that predicting subjectivity from predicted score distributions is, in general, sub-optimal compared to directly predicting it from ground-truth subjectivity scores. This indicates that, in practice, aesthetic score distribution predictors are not sufficiently accurate to enable assessing correctly the aesthetic subjectivity.

Despite our approach achieves state-of-the-art subjectivity prediction performance, we recognize that predicting subjectivity is a much harder task than predicting, e.g., the average aesthetic score -histogram-prediction-based methods can achieve correlations of 0.6 or higher for that task. We believe that this is partially due, in addition to the complexity of the task in itself, to the noisy nature of current aesthetic datasets. This is evident for the benchmark AVA dataset, where aesthetic scores are influenced by many factors that go beyond the pure aesthetic value of a picture. Building cleaner and more reliable aesthetic datasets is among the future directions to consider in this field.

Fig. 1 .

 1 Fig. 1. Example of aesthetic subjectivity for two images of the AVA dataset. The two images, displayed in the top-left panels, have similar mean score but different distribution of aesthetic judgments given by human raters, shown in the histograms on the top-right panels. The tables report several measures that compactly describe subjectivity based on the score distribution, which are described in Section III.

Fig. 2 .

 2 Fig. 2. The two score distributions have the same entropy, but the one on the right has a higher degree of subjectivity.

Fig. 3 .

 3 Fig.3. Subjectivity Prediction Framework. In the indirect prediction framework, an aesthetic score distribution is estimated first, and subjectivity measures are computed over it. We compare this approach with directly predicting subjectivity computed on ground-truth distributions (b).

Fig. 4 .

 4 Fig.4. Average predicted score distribution vs. ground-truth score distribution over the test set, for the three considered state-of-the-art distribution prediction methods. Notice that for all of them, the average predicted distribution is shifted compared to the original.

TABLE I PEARSON

 I 'S LINEAR CORRELATION COEFFICIENT (PLCC)

	Methods	SD	MAD MED	DUD
	Bin Jin's [5]	0.145	0.159	0.178	0.096
	NIMA [7]	0.169	0.187	0.211	0.255
	RSCJS [8]	0.187	0.199	0.227	0.281
	Direct (Resnet-34)	0.274	0.276	0.323	0.351
	Direct (Resnet-101)	0.307	0.304	0.333	0.360

Note that the

2-Wasserstein metric is sometimes confused with the Earth Mover Distance, e.g., in[START_REF] Talebi | NIMA: Neural image assessment[END_REF]. However, for the sake of precision, the Earth Mover Distance corresponds to the 1-Wasserstein metric.


Dayong Wang
email: wangdayong@cqupt.edu.cn

Yu Sun
email: yusun@uca.edu

Weisheng Li

Ce Zhu

Frédéric Dufaux

Frederic Dufaux
email: frederic.dufaux@l2s.centralesupelec.fr

FAST INTER MODE PREDICTIONS FOR SHVC

Keywords: SHVC, inter mode, square mode, nonsquare mode, early termination

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

With the rapid development of science and technology, video applications have been widely used in our daily life. Various video applications, such as digital TV broadcasting, video conferencing, wireless video streaming, and smart phone communications, have become ubiquitous. Different applications may use various devices, and users may also use various devices even for the same application. These devices may have different screen resolutions, processing capabilities, and network bandwidth requirements. In addition, different networks, such as broadband networks and wireless networks, may have different characteristics. Even in the same network, network bandwidth may also vary in different conditions. This requires that video streaming can adapt to different devices and network bandwidths. Scalable Video Coding (SVC), an extension of H.264/AVC, can efficiently solve these problems [START_REF] Schwarz | Overview of the scalable video coding extension of the h.264/avc standard[END_REF]. In order to provide scalability, SVC consists of a base layer (BL) and one or more enhancement layers (ELs). A SVC stream may support temporal, spatial and quality scalability, or a combination thereof. Scalability in temporal, spatial and quality refer to frame rate from low to high, spatial resolution from low to high and SNR from low to high, respectively. In recent years, the newest video coding standard, namely HEVC, has been developed. Since HEVC uses more advanced features and higher efficiency coding tools, it has better coding efficiency than previous video coding standards. HEVC only uses about half bit rates in comparison with H.264/AVC, while maintaining the same video quality level [START_REF] Correa | Performance and computational complexity assessment of high-efficiency video encoders[END_REF]. Since HEVC has very high coding efficiency, it attracts extensive attentions and several significant improvements have been further made on top of HEVC [START_REF] Li | Lagrangian multiplier adaptation for ratedistortion optimization with inter-frame dependency[END_REF][START_REF] Gao | Temporally dependent rate-distortion optimization for low-delay hierarchical video coding[END_REF][START_REF] Gao | Source distortion temporal propagation analysis for random-access hierarchical video coding optimization[END_REF]. In order to accommodate heterogeneous device capabilities, network conditions and client applications, Scalable High Efficiency Video Coding (SHVC), an extension of HEVC, has been proposed. Similar to SVC, SHVC also supports scalability in temporal, spatial, quality. In addition, it also provides bitdepth scalability (bit depth from low to high, e.g., 8 to 10 bit), and color gamut scalability (color gamut from narrow to wide, e.g., ITU-R Recommendation BT.709 to BT.2020) [START_REF] Boyce | Overview of shvc: Scalable extensions of the high efficiency video coding standard[END_REF].

Although HEVC has very high coding efficiency, its coding complexity is also very high, with an increase by about two to four times when compared to H.264/AVC [START_REF] Correa | Performance and computational complexity assessment of high-efficiency video encoders[END_REF]. In SHVC, multiple layers need to be encoded, using inter-layer predictions. Further, the encoding process of each layer is the same as that of HEVC. Thus, the coding process of SHVC is even more complex than that of HEVC. Therefore, it is highly desired to improve the coding speed, especially for wireless and realtime applications.

In order to improve the coding speed, we propose a new scheme for partition prediction to accelerate the coding of Inter prediction for quality SHVC (QS). The novelties and the contributions of the proposed algorithm are summarized below: (1) Inter-layer Reference (ILR) mode and merge mode are predicted based on depth correlation; (2) Square partitions, including ILR mode, merge mode and inter2Nx2N, are early terminated based on their RD-cost relationships; (3) Non-square partitions are predicted based on the distribution of the residual coefficients of square partitions.

RELATED WORK

To reduce the coding complexity of HEVC, several methods have been proposed in [START_REF] Pan | Early merge mode decision based on motion estimation and hierarchical depth correlation for hevc[END_REF][START_REF] Shen | An effective cu size decision method for hevc encoders[END_REF]. Pan et al. [START_REF] Pan | Early merge mode decision based on motion estimation and hierarchical depth correlation for hevc[END_REF] uses the mode of depth correlation, AZB, and the motion estimation information to early detect merge mode. The research in [START_REF] Shen | An effective cu size decision method for hevc encoders[END_REF] predicts candidate depths based on the previous frame and neighboring CUs, and then early terminate unnecessary CUs according to motion homogeneity checking, RD cost checking and SKIP mode checking.

Some research focus on improving the coding speed of SHVC. The research in [START_REF] Tohidypour | Probabilistic approach for predicting the size of coding units in the quadtree structure of the quality and spatial scalable hevc[END_REF] uses spatial and inter-layer correlations to propose a proba-bilistic approach to predict the quad-tree structure of coding tree units (CTUs) for quality and spatial scalable SHVC. Tohidypour et al. use coding information of relative CUs including neighboring CU in EL and the co-located CU in BL to build a probabilistic model,and then uses the proposed probabilistic model to predict the candidate modes of the current CU in the EL to improve the coding speed [START_REF] Tohidypour | Online-learning-based mode prediction method for quality scalable extension of the high efficiency video coding (hevc) standard[END_REF]. Wang et al. [START_REF] Wang | Efficient multi-strategy intra prediction for quality scalable high efficiency video coding[END_REF] predict candidate depths. In each candidate depth, if ILR mode early termination condition is satisfied, intra mode prediction can be skipped; otherwise intra mode prediction needs to be checked. Afterwards, if depth early termination is satisfied, other depths can be directly skipped to improve the coding speed.

Since both SHVC and HEVC use the same advanced coding tools, the complexity reduction algorithms developed for HEVC can also be utilized for SHVC. But these algorithms developed for HEVC do not exploit inter-layer correlation, so they cannot effectively improve SHVC's coding speed. In addition, the algorithms developed for SHVC use the same way to predict all candidate partitions and modes. However, since different partitions and modes have different features, using the same method is not always efficient. Therefore, in this research, we propose a fast mode prediction for quality SHVC.

PROPOSED FAST PARTITION AND MODE DECISION

In every coding depth of SHVC, a CU is predicted by square partition and non-square partition. Square partition includes ILR mode, merge mode and Inter2Nx2N and Intra2Nx2N mode. Non-square partition includes 2NxN, Nx2N, 2NxnU, 2NxnD, nLx2N and nRx2N. As NxN is only used when depth is 3, we have not considered this mode for the sake of simplicity. More specifically, we have developed two methods to improve the coding speed for square partition and non-square partition, respectively.

Square partition mode prediction

To improve the coding speed, for square partition, we have conducted experiments to obtain the proportions of the four modes (ILR mode, merge mode, Inter2Nx2N mode, In-tra2Nx2N mode).

ILR and merge modes prediction

Since the depth is strongly correlated, the parent CU and its children CUs may use similar modes. In order to investigate the depth correlation degree between parent CU and its children CUs in ILR and merge modes, we have conducted experiments using the same condition mentioned above. Based on these experiments, the depth correlation degree of ILR and merge modes are listed in Table 2.

In Table 2, depth correlation degree refers to the probability that children CUs select ILR and merge modes given parent CU using ILR and merge modes. From Table 2, we can find that the average depth correlation degree is 96%. This means that when a parent CU uses ILR and merge modes, its children CUs basically also use ILR and merge modes. Therefore, when a parent CU uses ILR and merge modes, its children CUs can only check ILR and merge modes.

Square partition mode early termination

When the parent CU of the current CU does not use ILR and merge modes, Inter2Nx2N needs to be checked. As mentioned above, when all modes of square partition have been checked, if their RD cost difference is not significant, we only need to check square partition modes and skip non-square partition directly. In order to determine whether they have a significant difference, we can compare their quantized residual coefficients. If the differences of their quantized residual coefficients between ILR & merge modes, and Inter2Nx2N mode are very small, that means that their RD costs are not significantly different [?]. Since every CU is composed of multiple 4x4 blocks, we calculate quantized residual coefficients in 4x4 block and derive the decision as follows.

For a DCT coefficient w , its quantized value z is:

z = w/Q step + 1/6, (1)
where Q step is quantization step. As we know, if the condition |(w 1 -w 2 )/Q step | ≤ 1 is met, no information needs to be coded. Thus, we can regard that they are no significant difference. As a result, we can derive:

|w 1 -w 2 | ≤ Q step , ( 2
)
where w 1 is the residual coefficients of ILR and merge modes, and w 2 is residual coefficients of Inter2Nx2N mode.

In SHVC, DCT coefficient w of 4x4 blocks is computed as:

w = 3 u=0 3 v=0 h iu x uv h jv (128 × 128) , (3)
Combing ( 2) and (3), we can derive:

|w 1 -w 2 | ≤ 3 u=0 3 v=0 h iu x 1 uv -x 2 uv h jv (128 × 128) = 3 u=0 3 v=0 |h iu h jv | x 1 uv -x 2 uv (128 × 128) , ( 4
) where h iu refers to the value in the integer DCT transform matrix at the (i, u) position in SHVC, and x uv refers to the residual value in the residual matrix at the (u, v) position. Since the maximum value of |h iu | is 83, combing (2) and (4), we can obtain:

3 u=0 3 v=0 x 1 uv -x 2 uv ≤ 2.5Q step (5)
From ( 5), we can derive:

3 u=0 3 v=0 x 1 uv - 3 u=0 3 v=0 x 2 uv ≤ 2.5Q step (6)
Sum of Absolute Difference (SAD) of two 4x4 blocks is obtained by [START_REF] Boyce | Overview of shvc: Scalable extensions of the high efficiency video coding standard[END_REF] and can be written as:

|SAD 1 -SAD 2 | ≤ 2.5Q step (7)
Since only depth 0, 1 and 2, namely 64x64, 32x32, 16x16 CU, have non-square partition, we can derive:

|SAD 1 -SAD 2 | ≤ 40mQ step , (8)
According to CU size, m is set 16, 4, 1, corresponding to 64x64, 32x32, 16x16 CU, respectively. Since SAD costs have strong relationship with RD costs, for convenience purposes, we have replaced SAD costs with RD costs, ( 8) can be written as:

|RD 1 -RD 2 | ≤ 40mQ step (9)
If the above condition is met, we can only select square partition modes and skip non-square partition modes directly. Otherwise, we still need to check non-square partition modes.

Non-square partition mode prediction

Non-square partition mode includes inter2NxN, interNx2N, inter2NxnU, inter2NxnD, internLx2N, internRx2N, as shown in Fig. 1.

In general, if the prediction is efficient, residual coefficients will obey a certain distribution. Therefore, we can also predict the best partition by studying the distribution of residual coefficients. Residual coefficients are typically modeled using a Gaussian distribution [START_REF] Cho | Fast cu splitting and pruning for suboptimal cu partitioning in hevc intra coding[END_REF] or a Laplacian distribution [START_REF] Lee | Modeling rates and distortions based on a mixture of laplacian distributions for interpredicted residues in quadtree coding of hevc[END_REF]. The Gaussian distribution is selected due to its superior performance in our experiments. Suppose residual coefficients of a CU obey a Gaussian distribution, we can divide a CU into two parts and decide whether they have significant difference to select the corresponding partition mode. Hypothesis testing is used to determine if the two parts for each division present a significant difference. For this purpose, t-test is used, which is the test statistics to investigate the significance of the difference between two sampled expected values. The expected values of the CU residual coefficients can be calculated by:

X 1 = M i=1 N j=1 r ij M × N , ( 10
)
where M and N respectively are the width and height of the partitioned CU,r ij is the residual coefficient value in (i, j) . CUs can take three sizes, 64x64, 32x32 and 16x16. According to the knowledge of mathematical statistics, if residual coefficients of a CU, obeying a Gaussian distribution, are divided into two parts. These two parts will also obey a Gaussian distribution. Suppose that the residual coefficients of two parts in each division can be respectively modeled as:

X 1 ∼ N μ 1 , σ 2 1 , X 2 ∼ N μ 2 , σ 2 2 , (11)
where X 1 is a Gaussian distribution of one part, X 2 is a Gaussian distribution of the other part in a division, μ and σ 2 are their expected value and variances. The test statistics t can be computed as:

t = X 1 -X 2 -(μ 1 -μ 2 ) S w 1 n1 + 1 n2 , ( 12
)
where

S 2 w = (n1-1)S 2 1 +(n2-1)S 2 2 n1+n2-2
, S 2 1 and S 2 2 are the two parts' variances respectively. Assuming μ 1 =μ 2 , the condition can be expressed as:

X 1 -X 2 -(μ 1 -μ 2 ) S w 1 n1 + 1 n2 ≤ t α/2 (n 1 + n 2 -1) , ( 13
)
where n 1 and n 2 are the numbers of residual coefficients of the two parts in each subdivision, and α is the significance level value. Given α, the corresponding threshold t α/2 (n 1 +n 2 -1) is determined by T distribution table. Since there are six non-square modes shown in Fig. 1, if each mode selects its own threshold, threshold values would be very complex. In order to simplify threshold values, we use the unified value. If the condition is satisfied, we can assume that there is no significant difference, and we do not need to check the corresponding modes. For example, in Fig. 1.(a), if top and bottom parts have a significant difference, then inter2NxN needs to be selected. Similarly, in Fig. 1.(b), if the left and right parts have a significant difference, then interNx2N need to be selected. The other modes are selected in the same way.

The optimal α and t α/2 (n 1 + n 2 -1) can be obtained through experiments. We have performed experiments using the same condition above. Commonly used values for α, 0.5, 0.4, 0.3, 0.2, 0.1, 0.05, 0.02, 0.01 and 0.005, and the corresponding threshold values, 0.674, 0.842, 1.036, 1.282, 1.645, 1.96, 2.326, 2.576 and 2.807, are tested. Table 3 shows the corresponding coding efficiencies, which are measured by BDBR under different sequences and thresholds. BDBR represents the bitrate difference at an equal PSNR [START_REF] Bjontegaard | calculation of average psnr difference between rdcurves[END_REF]. A negative BDBR represents an increase in coding efficiency when compared with the reference software. From Table 3 we can observe that as the threshold value increases, the loss of coding efficiency basically increases. However, the increase is very slow, and the loss of coding efficiencies under all thresholds is very small. Obviously, the higher the threshold is, the greater the coding speed improves. Therefore, the highest threshold value, 2.807, is selected.

EXPERIMENTAL RESULT

The proposed fast inter prediction algorithm for QS has been implemented in the SHVC reference software (SHM 11. 4 and Table 5, with Q1 and Q2, respectively.

From Table 4, we can find that the average BDBR and TS of the proposed algorithm are 1.09% and 60.56%, respectively, while the average BDBR and TS of EMD are 1.55% and 55.01%. In Table 5, the coding efficiency (BDBR) of both our algorithm and EMD are degraded when compared to BDBR in Table 4. This is because that, the QP difference between BL and EL in Table 5 CUs may use non-square modes and the accuracy of ILR and merge modes prediction may degrade to some extent. Therefore, coding efficiency is degraded in Table 5. From Table 5, we can find that our algorithm's average BDBR and TS are 1.9% and 59.41%, respectively, while EMD's average BDBR and TS are 2.03% and 52.99%. Therefore, our proposed algorithm outperforms EMD in terms of both coding efficiency and coding speed.

CONCLUSION

In this paper, we have proposed a novel and effective inter mode prediction algorithm for Quality SHVC.The proposed algorithm can be further improved by combining with fast motion estimation methods like that in [START_REF] Zhu | Enhanced hexagonal search for fast block motion estimation[END_REF]. Our future work includes depth prediction to further improve the coding speed of inter prediction for quality SHVC and video communication [START_REF] Su | Fast splitting based tag identification algorithm for anti-collision in uhf rfid system[END_REF].

Fig. 1 .

 1 Fig. 1. Non-square partition.

Table 1 .

 1 The Proportions of modes in square partition

	Sequence	ILR& Merg Inter2Nx2N Intra2Nx2N
	Sunflower	83.4%	15.4%	1.2%
	Tractor	86.1%	12.9%	1.0%
	Flowervase	79.0%	19.9%	1.1%
	PartyScene	84.5%	14.2%	1.3%
	BlowingBubbles	82.9%	16.5%	0.6%
	RaceHorses	81.1%	17.5%	1.4%
	Park	87.5%	12.1%	0.4%
	Town	88.1%	11.6%	0.3%
	Average	84.1%	15.0%	0.9%

Table 2 .

 2 The depth correlation degree of ILR and merge modes

	As suggested in common SHM test conditions (CSTC) [12],
	each test sequence is encoded under the Random Access (RA)
	main configuration. The QPs used for the BL are (26, 30,
	34, 38), and the corresponding QPs used for the EL are (22,
	26, 30, 34). Based on these experiments, the proportions of
	modes in square partition are listed in Table 1.
	From Table 1, we can observe that: (1) ILR and merge
	modes have the highest probability; (2) Inter2Nx2N has a

To meet the different resolution requirements, two sequences in each class B, C, D and E are selected in our experiments. More specifically, the following test sequences are used: Sunflower and Tractor in class B; Flowervase and PartyScene in class C; BlowingBubbles and RaceHorses in class D; and Parkruner and Town in class E.

graded. Therefore, we first check ILR and merge modes, and decide whether they are the best modes. In the affirmative, we can only select these modes and skip Inter2Nx2N mode. Otherwise, we still need to check Inter2Nx2N mode and compare the difference of RD costs between ILR and merge modes and Inter2Nx2N mode. If the difference is not significant, square partition can be regarded as the best mode and non-square partition modes can be skipped directly. Otherwise, non-square partition modes need to be further checked. Base on the above analysis, in this paper, we develop ILR and merge modes prediction and square partition mode early termination. The details are listed below.

Table 3 .

 3 Coding efficiencies under different threshold values

	Threshold	0.674 0.842 1.036 1.282 1.645 1.96 2.326 2.576 2.807
	Sunflower	0.1	-0.1	-0.1	0	-0.1	0.1	0.2	0.1	0
	Tractor	0	0	0	0	0.1	0.1	0.1	0.1	0.1
	Flowervase	0.1	0.1	0.2	0.3	0.3	0.4	0.3	0.4	0.2
	PartyScene	0	0.1	0.1	0.1	0.1	0.1	0.1	0.1	0.1
	BlowingBubbles	0.1	0.1	0	0.2	0.2	0.2	0.2	0.2	0.2
	RaceHorses	0	0	0	0	0.1	0	0	0	0
	Park	0.1	0.1	0.1	0.1	0.1	0.1	0.1	0.1	0.1
	Town	0.2	0.2	0.3	0.3	0.3	0.3	0.4	0.3	0.4
	Average	0.075 0.063 0.075 0.13	0.14 0.163 0.175 0.163 0.138
	a server with Intel (R) 2.0 GHz CPU and 30 GB memory. The
	experimental parameters are set according to the CSTC [12].
	The proposed algorithm is developed for RA coding struc-
	ture. The QPs in the BL are (26, 30, 34, 38), and and the
	corresponding QPs used for the EL are set to be (22, 26, 30,
	34) and (20, 24, 28, 32), respectively. Since the eight training
	sequences are no longer used in our overall performance test-
	ing, our proposed algorithm is generic and effective for dif-
	ferent types of video contents. Coding efficiency is measured
	by BDBR, and coding speed is evaluated by coding time in
	the EL. TS represents the percentage of coding time savings
	in the EL. In order to further demonstrate the performance of
	our proposed algorithm, we have compared it with EMD al-
	gorithm proposed in [7]. Two settings of QPs: Q1= (22, 26,
	30, 34) and Q2= (20, 24, 28, 32) are used. Their performance
	comparisons in terms of coding efficiency and coding speed
	are shown in Table

Table 4 .

 4 is 6, which is larger than the QP difference 4 used in Table4. The larger QP difference causes the weaker inter-layer correlation. In addition, since QPs of EL in Table5are smaller than those in Table4, more Performance comparison of the different methods

	Sequence	Proposed BDBR TS	EMD [7] BDBR TS
	Traffic	1.0%	61.06% 1.33% 64.65%
	PeopleOnStreet 0.90% 60.64% 0.45% 42.81%
	BasketballDrive 0.50% 60.43% 2.26% 54.23%
	BQTerrace	1.10% 61.15% 2.37% 53.98%
	Cactus	0.90% 60.14% 1.07% 55.36%
	Kimono	1.30% 59.71% 0.98% 51.88%
	ParkScene	1.90% 60.76% 2.38% 64.65%
	Average	1.09% 60.56% 1.55% 55.01%

Table 5 .

 5 Performance comparison of the different methods

	Sequence	Proposed BDBR TS	EMD [7] BDBR TS
	Traffic	2.20% 58.89% 1.82% 62.34%
	PeopleOnStreet 2.10% 58.95% 1.03% 40.16%
	BasketballDrive 2.00% 59.42% 2.84% 51.27%
	BQTerrace	1.80% 61.18% 2.93% 52.49%
	Cactus	1.80% 59.25% 1.55% 52.79%
	Kimono	1.30% 59.01% 1.53% 50.01%
	ParkScene	1.80% 59.21% 2.52% 61.87%
	Average	1.90% 59.41% 2.03% 52.99%

This work was supported in part by the Natural Science Foundation of China under Grant 61401247, U1713213, 61472055, 61571102 and 61571070, in part by Chongqing science and Technology Commission Project under Grant cstc2015jcyjBX0009, cstc2016jcyjA0543, cstc2017jcyjAX0142, cstc2018jcyjAX0525 and cstc2018jcyjAX0225, in part by the Key Project of Sichuan Provincial Department of Science and Technology under Grant 2018JY0035, in part by Ministry of Education -China Mobile Research Fund Project under Grant MCM20170203, in part by faculty sabbatical leave fund from Uni. of Central Arkansas.


Mohsen Abdoli

Félix Henry

Patrice Brault

Frédéric Dufaux

Pierre Duhamel

TRANSFORM COEFFICIENT CODING FOR SCREEN CONTENT IN VERSATILE VIDEO CODING (VVC)

Keywords: Residual transform coding, VVC, Screen content

A transform coefficient coding scheme is proposed for 4 × 4 blocks in Versatile Video Coding (VVC), targeting screen content applications. The proposed algorithm, called Unary Bitplane Coding (UBC), uses unary codes of the coefficient amplitudes and represents each block by their bitplanes. This representation allows exploiting further contextual information for source separation during the entropy coding. Experiments in the Joint Exploration test Model (JEM) show that replacing the existing transform coding with UBC only for 4 × 4 blocks brings on average 2.8% and 3.4% BD-R gain in the random access and all intra modes, respectively.

INTRODUCTION

With the growing applications of screen content in the past years, their efficient compression has become a priority for video codecs. In the context of future video coding, this priority is reflected in the "versatility" aspect of the future standard which is referred to as Versatile Video Coding (VVC) [START_REF] Wien | Preliminary joint call for evidence on video compression with capability beyond HEVC[END_REF]. While targetting natural content is a priority, this codec is also supposed to have tools dedicated to screen content coding [START_REF] Xu | Intra block copy in HEVC screen content coding extensions[END_REF][START_REF] Guo | Palette mode for screen content coding[END_REF][START_REF] Lai | AHG6: On adaptive color transform[END_REF][START_REF] Zhang | Adaptive color-space transform in HEVC screen content coding[END_REF][START_REF] Li | Adaptive motion vector resolution for screen content[END_REF][START_REF] Abdoli | AHG11: Block DPCM for screen content coding[END_REF]. This is in contrast with High Efficiency Video Coding (HEVC), where an ad-hoc extension of the standard (i.e. HEVC-SCC) was dedicated to screen content [START_REF] Xu | Overview of the emerging HEVC screen content coding extension[END_REF].

Modern video coding standards benefit from block prediction tools to decorrelate the signal and remove its redundancies. A prediction phase is often followed by a residual coding phase to reconstruct a high quality image. Although the stateof-the-art prediction tools perform significantly well, a relatively large redundancy usually remains in the residual signal. In order to further compress the residual, this redundancy is exploited by residual transformation into the frequency domain. This allows transmission of residual information in a few transform coefficients (e.g. DCT, DST etc.).

The existing transform coefficient transmission algorithms of VVC, referred to as residual coding in the rest of this paper, has remained almost unchanged since HEVC [START_REF] Sole | Transform coefficient coding in HEVC[END_REF]. For 4 × 4 blocks, that are solely considered in this paper, the following steps are performed: 1) signaling scan position of the last significant (i.e. non-zero) coefficient, 2) coding all amplitudes from the beginning until the last scan position, and finally 3) coding coefficient signs. For each amplitude, a prefix and a suffix is coded. The prefix uses a unary code of length up to three, depending to the amplitude value. If the amplitude is greater than or equal to two, the suffix for the remaining of the amplitude is coded in the bypass mode using Golomb-Rice and Exp-Golomb codes [START_REF] Sole | Transform coefficient coding in HEVC[END_REF]. Although the above algorithm is usually efficient in removing the correlations remaining in the transform blocks, it still has rooms to improve for screen content. For instance, Fig. 1 shows two histograms corresponding the last significant position in all 4×4 blocks, in natural and screen contents. As can be seen, this position tends to occur later in screen blocks than in natural blocks, leaving fewer non-significant coefficients at the end of blocks. To address this kind of problems, two major solutions have been proposed during the standardization of HEVC. The first solution is the Transform Skip Mode (TSM), which encodes residual values in the pixel domain [START_REF] Mrak | Improving screen content coding in HEVC by transform skipping[END_REF]. The second approach, called Residual DPCM (RDPCM), performs a DPCM-based prediction on the pixeldomain residual obtained from the TSM [START_REF] Tan | Residual DPCM for lossless coding in HEVC[END_REF][START_REF] Naccari | Inter prediction residual DPCM[END_REF][START_REF] Joshi | AHG8: Residual DPCM for visually lossless coding[END_REF].

Previously, it was shown that coding 4 × 4 blocks in VVC requires special attention as they are the smallest partitions and usually correspond to high detail textures [START_REF] Abdoli | Intra prediction using in-loop residual coding for the post-HEVC standard[END_REF]. Moreover, they are usually the most popular block size in all QP values [START_REF] Abdoli | Short-distance intra prediction of screen content in versatile video coding (VVC)[END_REF]. Therefore, a transform coefficient coding, called Unary Bitplane Coding (UBC) is proposed in this paper for 4 × 4 blocks of screen content. In this algorithm, amplitudes are individually binarized using unary codes and then are represented with bitplanes. This representation allows to extract contextual information from coded bins and to categorize them with respect to their statistics. One Context-Adaptive Binary Arithmetic Coding (CABAC) model is then dedicated to each category during the coding. The rest of this paper is organized as follows. In section II, the amplitude coding framework is introduced. Section III describes the proposed configuration of this framework integrated in VVC.

In section IV, experimental results are presented and finally, section V draws conclusions.

UBC FRAMEWORK

In this section, a flexible framework is described, aiming at obtaining as much contextual information as possible from the coded bins. Then, it optimizes a merging scheme to define a limited set of entropy coding contexts. From this aspect, the framework of this section shares its general idea with another residual coding algorithm proposed for the AV1 standard [START_REF] Han | A level-map approach to transform coefficient coding[END_REF] 2.1. Binarization

The binarization scheme in UBC uses unary codes to produce bins of individual amplitude. The bins of amplitudes in a block are then represented by bitplanes as shown in Fig. 2, where each unsigned decimal value C in the amplitude block is first binarized to a unary code of C + 1 bins, including C intermediate '1' bins followed by a terminal '0' bin.

The main benefit of the unary bitplanes compared to binary bitplanes is the contextual information accessibility. For instance, let us assume that the encoder is currently in the L-th bitplane and bin values of all coefficients in the lower layers are available. For encoding a bin at the level L of a certain coefficient, the encoder attempts to access contextual information from neighboring amplitudes, available up to the level L. By accessing a bin value of '1' or '0' at the (L-1)-th level of one of its neighbors, the encoder can immediately decide whether the amplitude of the neighbor is less than L-1, or at least L-1. On the contrary, with binary codes, the encoder has to access all the bin values down to layer zero, in order to obtain the same contextual information about that neighbor.

Source separation based on contextual features

Given a feature space, defined on the unary bins of the UBC, each feature describes a contextual situation of the bin associated to it. These contextual situations, simply called situations hereafter, are then used to classify bins into categories with similar statistical behavior.

Let us consider coding of the significance bin, which indicates whether a coefficient is non-zero or zero. A 2D feature space is defined for the significance bin by 1) its frequency band, and 2) its neighborhood density.

To evaluate the impact of the significance bin source separation with the above features, a dataset of coded significance bins, from 4 × 4 blocks was prepared. In this experiment, a frequency band is considered as "low" if it is located in the top-left part of the block (otherwise, "high"). Moreover, the neighborhood density of a coefficient is considered as a "low" density, if less than half of its available neighbors are significant (otherwise, "high"). Now let B, F and D be three binary random variables corresponding to the significance bin, the frequency band and the neighborhood density, respectively. By using either F or D for source separation of the significance bin B, four configurations (i.e. cfg) can be defined. Table 1 shows the efficiency of each configuration in terms of the significance bin entropy H, calculated as:

H cfg = - b=0,1 P cfg (B = b) × log 2 P cfg (B = b), (1)
As can be seen from this simple example, a source separation by using proper features can reduce the transmission rate of bins. In the proposed transform amplitude coding, we aim at using the bitplane representation of transform blocks to provide a simple contextual information access for feature extraction. A classifier is then trained on the extracted features to optimize the source separation scheme constrained by the final number of clusters.

Situation reduction by K-Means

The number of situations can easily become too large for being coded by exclusive CABAC context models. To address this, the second step of the proposed framework applies a K-Means algorithm to categorize situations based on their statistical behavior.

Let D be a dataset of actual samples from coded video streams with different QPs. Each entry of this dataset is the unary bitplane representation of one unsigned amplitude block of size 4 × 4, as shown in Fig. 2. Assuming that D contains a total of N bin bins, one can define a feature space and extract feature vectors from all bins. Given the fact that the feature vectors in this space represent situations, a space with N sit situations actually separates the significance bin information source into N sit sub-sources. The main goal of K-Means is to derive a coarser source separation scheme with N ctx coding contexts. The output of this process is supposedly a table T to map each of N sit situations into one of N ctx coding contexts. Algorithm 1 describes the proposed K-Means algorithm.

Algorithm 1 K-Means algorithm for reduction of N sit situations into N ctx coding contexts. In order to compute the rate of the signal in D, given the situation to context table T , the algorithm uses an entropybased rate of the signal. Let B be the binary random variable of bins in D and G(B) a function that extracts features of B and returns its situation index. Also define D i as the subset of D containing all the bins whose situations are mapped to context ctx i , as:

D i = {B|B ∈ D, T [G(B)] = ctx i }.
(2)

The probability distribution function of the sub-source separated by context ctx i can be expressed as:

P i (B) = P (B|B ∈ D i ). (3)
Moreover, its sub-source entropy is computed as:

Rate i = -Len(D i ) b=0,1 log 2 P i (B = b), (4)
where Len(D i ) is the number of bins in D i , such as:

Nctx-1 i=0 Len(D i ) = N bin . (5)
Therefore, the total rate of the dataset D is calculated as:

Rate = Nctx-1 i=0 Rate i = -N bin Nctx-1 i=0 b=0,1 log 2 P i (B = b). (6)
Experiments show that adequate iterations with the above K-Means algorithm can guarantee a convergence.

PROPOSED UBC CONFIGURATION

In this section, the proposed configuration of the UBC framework is explained. This algorithm replaces the coefficient coding of VVC for 4 × 4 blocks to address its inefficiency.

Feature space definition

Three bin level features are used in the proposed UBC configuration. These features are namely the neighborhood density v d , the bitplane number v l and the frequency band v f and compose a feature vector V =< v d , v l , v f > for each bin.

Neighborhood density v d : To model the spatial density at the level L of a coefficient, we define a small neighborhood of 3 × 3 around it, as depicted in Fig. 3. As can be seen, the current bin to encode (starred) has access to its causal neighbors at level L and its non-causal neighbors at level L -1. With a maximum of eight available neighbors, nb i , i = 0, 1, ..., 7, as shown with gray cells in Fig. 3, the density feature can have 2 8 = 256 values and is calculated as:

v d = 7 i=0 2 nbi . (7)
Bitplane number v l : The second proposed feature to extract from a bin is the bitplane level L it is located in. Technically, the upper bound of coefficient amplitudes is 65635. However, very large amplitudes rarely occur with conventional QPs. Therefore, to limit the value of v l , a threshold at level 15 is applied to give 16 different v l values:

v l = min(L, 15). (8)
Frequency v f : There are 16 different frequencies inside a 4 × 4 transform block. The frequency feature value in the proposed UBC algorithm uses the scan order index.

Bucketizing the feature space

In order to apply the proposed UBC algorithm, the look-up table T , explained in the previous section, needs to be stored at both encoder and decoder sides. As explained, each entry of T corresponds to one situation defined by a feature vector

V =< v d , v l , v f >
(v d × v l × v f )
, which is expensive to store, especially at the decoder side.

To address this problem, the feature space is bucketized with another round of K-Means to quantize the domain of the features. In this round, an offline brute-force search is carried out on each feature separately, in order to find the best bucketizing scheme. During the search process of each feature, other features are used with their original domain to evaluate different bucketizing schemes on the current feature. The result of each search is a bucket table to map the original domain of its corresponding feature to a bucketized domain. Finally, all three bucket tables are integrated to be used before the situation to context table T, which now has a smaller size, due to the smaller domain of each feature.

The outcome of the bucketizing step is a coarser representation of the feature space of V which would cause a performance drop compared to the full domain feature space. Therefore, a compromise has been made between the performance and the bucket tables sizes. As a result, three bucket tables have been optimized to reduce the original feature space size of 65536 to 30 × 4 × 4 = 480 situations (v d × v l × v f ).

RESULTS

For performance evaluation, the transform coding of JEM5 is replaced by the UBC for all 4 × 4 blocks [START_REF] Chen | Algorithm description of joint exploration test model 5 (JEM 5)[END_REF]. This is performed by storing a situation to context table T of size 480, as well as three bucket tables at both encoder and decoder sides. Table 2 compares the coding performance of UBC against JEM on the screen content of the JVET and the HEVC-SCC Common Test Conditions (CTCs) [START_REF] Suehring | JVET common test conditions and software reference configurations[END_REF][START_REF] Yu | Common test conditions for screen content coding[END_REF]. In order to avoid over-training of T, none of the test sequences in Table 2 was used. As can be seen, the UBC algorithm improves BD-R [START_REF] Bjontegaard | Calculation of average PSNR differences between rd-curves[END_REF] performance of JEM by 2.8% and 3.4% in the random access and all intra modes, respectively. The better performance in the all intra mode could be due to the higher density of 4 × 4 blocks in intra slices than inter slices as shown in Table 3.

CONCLUSION

A transform coefficient coding algorithm, Unary Bitplane Coding, is proposed to improve the decorrelation remaining in the quantized coefficients of screen content. For this purpose, a set of contextual situations are defined on unary bitplanes of coefficients and then are merged to a few coding contexts by CABAC. The experiments show that UBC achieves an average gain of 2.8% and 3.4% in the random access and all intra modes, respectively.

Fig. 1 .

 1 Fig. 1. Histogram of the last significant position in 4 × 4 blocks of natural and screen content.

Fig. 2 .

 2 Fig. 2. Unary bitplane binarization of 4 × 4 amplitude blocks.

Fig. 3 .

 3 Fig. 3. The 3 × 3 neighborhood around a coefficient at levels L and L -1.

Table 1 .

 1 Description of four configurations u, f, d and fd, using F and D features, along with their corresponding entropy of the significance bin B.

	cfg	Contextual features	Number of situations	P cfg	H cfg
	u	None	1	Pu(B) = P (B)	0.98
	f	F	2	P f (B) = P (B|F )	0.92
	d	D	2	P d (B) = P (B|D)	0.67
	fd	F,D	4	P f d (B) = P (B|F, D)	0.64

  As can be seen in Algorithm 1, the context of each situation sit is assigned after minimizing a rate function. This greedy approach uses the latest context assignment stored in the table T and attempts to update T [sit] with a new context ctx * with lowest rate, if possible.

	3:	Initialize: sit ← 1
	4: top:
	5:	if sit > N sit then
	6:	return T
	7:	Initialize: ctx ← 1, ctx * ← -1, Rate * ← ∞
	8: loop:
	9:	if ctx > Nctx then
	10:	T [sit] ← ctx * // assign the best context
	11:	sit ← sit + 1
	12:	goto top
	13:	else
	14:	T [sit] ← ctx
	15:	Rate ← getRate(T )
	16:	if Rate < Rate * then
	17:	Rate * ← Rate
	18:	ctx * ← ctx
	19:	ctx ← ctx + 1
	20:	goto loop

1: procedure K-MEANS 2:

Initialize randomly entries of T with 1, ..., Nctx.

  and contains the CABAC context model number associated to that situation. Therefore, with no feature space reduction, this table should have an excessive number of 256 × 16 × 16 = 65536 entries

Table 2 .

 2 Performance of the UBC against JEM, in terms of BD-rate gain (%) and coding time (%).

	Res.	Sequence	Random access BD-rate ET/DT	All intra BD-rate ET/DT
	2560	Basketball Sc	-2.74	125/114	-2.70	133/116
	×	MissionCtrlClip2	-2.82	147/120	-2.99	150/128
	1440	Average	-2.78	136/117	-2.84	141/122
		FlyingGraphics	-1.55	130/125	-2.26	131/128
		Desktop	-4.42	138/126	-6.57	146/135
		Console	-3.05	140/132	-6.48	146/140
	1920	ChineseEditing	-3.23	140/129	-3.16	147/137
	×	MissionCtrlClip3	-2.68	125/134	-3.31	148/137
	1080 Robot	+0.04	133/116	+0.06	132/124
		ChinaSpeed	-1.38	107/118	-3.58	138/129
		Average	-2.32	128/126	-3.68	141/133
		Web browsing	-4.88	132/120	-4.10	136/122
	1280	Map	-0.98	138/109	-1.06	147/121
	×	Programming	-4.51	135/104	-4.88	139/112
	720	SlideShow	-3.06	140/119	-3.42	149/130
		SlideEditing	-3.93	126/108	-5.19	134/118
		Average	-3.47	115/112	-2.84	147/121
		Total Average	-2.8	126/119	-3.4	141/127

Table 3 .

 3 Statistics of 4 × 4 blocks in intra and inter slices of screen contents.

	QP	Population Inter Intra Inter Intra Rate
	22	53%	51%	19%	36%
	27	52%	52%	15%	37%
	32	43%	56%	10%	38%
	37	3%	55%	1%	37%