Thierry Dumas

Aline Roumy

Christine Guillemot

Context-adaptive neural network based prediction for image compression

Keywords: Image compression, intra prediction, neural networks

published or not. The documents may come

I. INTRODUCTION

I NTRA prediction is a key component of image and video compression algorithms and in particular of recent coding standards such as H.265 [START_REF] Wien | High efficiency video coding: coding tools and specification[END_REF]. The goal of intra prediction is to infer a block of pixels from the previously encoded and decoded neighborhood. The predicted block is subtracted from the original block to yield a residue which is then encoded. Intra prediction modes used in practice rely on very simple models of dependencies between the block to be predicted and its neighborhood. This is the case of the H.265 standard which selects according to a rate-distortion criterion one mode among 35 fixed and simple prediction functions. The H.265 prediction functions consist in simply propagating the pixel values along specified directions [START_REF] Lainema | Intra coding of the HEVC standard[END_REF]. This approach is suitable in the presence of contours, hence in small regions containing oriented edges [START_REF] Bell | Edges are the independent components of natural scenes[END_REF], [START_REF] Van Hateren | Independent component filters of natural images compared with simple cells in primary visual cortex[END_REF], [START_REF] Lee | Sparse deep belief net model for visual area V2[END_REF]. However, it fails in large areas usually containing more complex textures [START_REF] Hosoya | A hierarchical statistical model of natural images explains tuning properties in V2[END_REF], [START_REF] Lee | Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations[END_REF], [START_REF] Ranzato | Modeling natural images using gated MRFs[END_REF]. Instead of simply propagating pixels in the causal neighborhood, the authors in [START_REF] Tan | Intra prediction by template matching[END_REF] look for the best predictor within the image by searching for the best match with the so-called template of the block to be predicted. The authors in [START_REF] Turkan | Image prediction based on neighbor embedding methods[END_REF] further exploit self-similarities within the image with more complex models defined as linear combinations of k-nearest patches in the neighborhood.

Thierry Dumas is currently with Interdigital, 35576 Cesson-Sévigné, France (e-mail: thierry.dumas@interdigital.com). Aline Roumy and Christine Guillemot are with INRIA Rennes, 35042 Rennes, France (e-mail: aline.roumy@inria.fr, christine.guillemot@inria.fr).

This work has been supported by the French Defense Procurement Agency (DGA).

In this paper, we consider the problem of designing an intra prediction function that can predict both simple textures in small image blocks, as well as complex textures in larger ones.

To create an optimal intra prediction function, the probabilistic model of natural images is needed. Let us consider a pixel, denoted by the random variable X, to be predicted from its neighboring decoded pixels. These neighboring decoded pixels are represented as a set B of observed random variables. The authors in [START_REF] Wiegand | Source coding: part I of fundamentals of source and video coding[END_REF] demonstrate that the optimal prediction X * of X, i.e. the prediction that minimizes the mean squared prediction error, is the conditional expectation E [X|B]. Yet, no existing model of natural images gives a reliable E [X|B].

However, neural networks have proved capable of learning a reliable model of the probability of image pixels for prediction. For example, in [START_REF] Theis | Generative image modeling using spatial LSTMs[END_REF], [START_REF] Van Der Oord | Pixel recurrent neural networks[END_REF], recurrent neural networks sequentially update their internal representation of the dependencies between the pixels in the known region of an image and then generate the next pixel in the unknown region of the image.

In this paper, we consider the problem of learning, with the help of neural networks, a reliable model of dependencies between a block, possibly containing a complex texture, and its neighborhood that we refer to as its context. Note that neural networks have already been considered in [START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF] for intra block prediction. However, the authors in [START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF] only take into consideration blocks of sizes 4 × 4, 8 × 8, 16 × 16, and 32 × 32 pixels and use fully-connected neural networks. Here, we consider both fully-connected and convolutional neural networks. We show that, while fully-connected neural networks give good performance for small block sizes, convolutional neural networks are more appropriate, both in terms of prediction PSNR and PSNR-rate performance gains, for large block sizes. The choice of neural network is block size dependent, hence does not need to be signalled to the decoder. This set of neural networks, called Prediction Neural Networks Set (PNNS), has been integrated into a H.265 codec, showing PSNR-rate performance gains from 1.46% to 5.20%.

In summary, the contributions of this paper are as follows:

• We propose a set of neural network architectures, including both fully-connected and convolutional neural networks, for intra image prediction. • We show that, in the case of large block sizes, convolutional neural networks yield more accurate predictions compared with fully-connected ones. • Thanks to the use of masks of random sizes during training, the neural networks of PNNS well adapt to the available context that may vary. E.g. in H.265, the available context, hence the number of known pixels in the neighborhood, depends on the position of the considered prediction unit within the coding unit and within the coding tree unit. • Unlike the H.265 intra prediction modes, which are each specialized in predicting a specific texture, the proposed PNNS, trained on a large unconstrained set of images, is able to model a large set of complex textures. • We prove experimentally a surprising property of the neural networks for intra prediction: they do not need to be trained on distorted contexts, meaning that the neural networks trained on undistorted contexts generalize well on distorted contexts, even for severe distortions. The code to reproduce our numerical results and train the neural networks is available online 1 .

II. CONDITIONS FOR EFFICIENT NEURAL NETWORK BASED INTRA PREDICTION

Prediction is a key method in rate distortion theory, when complexity is an issue. Indeed, the complexity of vector quantization is prohibitive, and scalar quantization is rather used. But, scalar quantization cannot exploit the statistical correlations between data samples. This task can be done via prediction [START_REF] Gray | Quantization[END_REF]. Prediction can only be made from data samples available at the decoder, i.e. causal and distorted data samples. By distorted causal data samples we mean previously encoded and decoded pixels above and on the left side of the image block to be predicted. This set of pixels is often referred to as the context of the block to be predicted.

Optimal prediction, i.e. conditional expectation [START_REF] Wiegand | Source coding: part I of fundamentals of source and video coding[END_REF], requires knowing the conditional distribution of the image block to be predicted given causal and distorted data samples. Estimating such a conditional distribution is difficult. The use of the predictor by the decoder would in addition require sending the distribution parameters. Classical approaches in predictive coding consist in proposing a set of predefined functions and choosing the best of them in a rate-distortion sense. Thus, the number of possible functions is limited. On the other hand, neural networks can approximate many functions, in particular complex predictive functions such as the generation of future video frames given an input sequence of frames [START_REF] Ranzato | Video (language) modeling: a baseline for generative models of natural videos[END_REF], [START_REF] Mathieu | Deep multi-scale video prediction beyond mean square error[END_REF].

But, the use of neural networks for intra prediction within an image coding scheme raises several questions that we address in this paper. What neural network architecture provides enough power of representation to map causal and distorted data samples to an accurate prediction of a given image block? What context size should be used? Section III looks for a neural network architecture and the optimal number of causal and distorted data samples for predicting a given image block. Moreover, the amount of causal and distorted data samples available at the decoder varies. It depends on the partitioning of the image and the position of the block to be predicted within the image. Section IV trains the neural networks so that they adapt to the variable context size. Finally, can neural networks compensate for the quantization noise in its input and be efficient in a rate-distortion sense? Sections V and VI answer these two questions with experimental evidence.

III. PROPOSED NEURAL NETWORK BASED INTRA

PREDICTION

Unlike standard intra prediction in which the encoder chooses the best mode in a rate-distortion sense among several pre-defined modes, only one neural network among a set of neural networks does the prediction here. Unlike [START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF], our set contains both fully-connected and convolutional neural networks. This section first presents our set of neural networks. Then, it explains how one neural network is selected for predicting a given image block and the context is defined according to the block size. Finally, the specificities of the integration of our neural networks into H.265 are detailed.

A. Fully-connected and convolutional neural networks

Let X be a context containing decoded pixels above and on the left side of a square image block Y of width m ∈ N * to be predicted (see Figure 1). The transformation of X into a prediction Ŷ of Y via either a fully-connected neural network f m , parametrized by θ m , or a convolutional neural network g m , parametrized by φ m , is described in [START_REF] Wien | High efficiency video coding: coding tools and specification[END_REF]. The corresponding architectures are depicted in Figures 2 and3.

X c = X -α Ŷc = f m (X c ; θ m ) g m (X c ; φ m ) (1)
Ŷ = max min Ŷc + α, 255 , 0

During optimization, each input variable to a neural network must be approximatively zero-centered over the training set to accelerate convergence [START_REF] Lecun | Efficient backprop[END_REF]. Besides, since the pixel space corresponds to both the input and output spaces in intra prediction, it makes sense to normalize the input and output similarly. One could subtract from each image block to be predicted and its context their respective mean during training. But, this entails sending the mean of a block to be predicted to the decoder during the test phase. Instead, the mean pixel intensity α is first computed over all the training images. Then, α is subtracted from each image block to be predicted and its context during training. During the test phase, α is subtracted from the context (see [START_REF] Wien | High efficiency video coding: coding tools and specification[END_REF] where the subscript c stands for centered).

This preprocessing implies a postprocessing of the neural network output. More precisely, the learned mean pixels intensity is added to the output and the result is clipped to [0, 255] (see [START_REF] Wien | High efficiency video coding: coding tools and specification[END_REF]).

The first operation for both architectures consists in formatting the context to ease the computations in the neural network. In the case of a fully-connected neural network, all elements in the context are connected such that there is no need to keep the 2D structure of the context [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. Therefore, the context is first vectorized, and fast vector-matrix arithmetics can be used (see Figure 2). However, in the case of a convolutional neural network, fast computation of 2D filtering requires to keep the 2D structure of the context. Moreover, again for fast computation, the shape of the input to the convolution has to be rectangular. That is why the context is split into two rectangles X 0 and X 1 (see Figures 1 and3). X 0 and X 1 are then processed by distinct convolutions.

The proposed fully-connected architecture is composed of 4 fully-connected layers. The first layer computes an overcomplete representation of the context to reach p ∈ N * output coefficients. Overcompleteness is chosen as it is observed empirically that overcomplete representations in early layers boost the performance of neural networks [START_REF] Hinton | Reducing the dimensionality of data with neural networks[END_REF], [START_REF] Krizhevsky | Using very deep autoencoders for content-based image retrieval[END_REF], [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. The next two layers keep the number of coefficients unchanged, while the last layer reduces it to provide the predicted image block. The first three layers have LeakyReLU [START_REF] Maas | Rectifier nonlinearities improve neural network acoustic models[END_REF] with slope 0.1 as non-linear activation. The last layer has no non-linear activation. This is because the postprocessing discussed earlier contains already a non-linearity, which consists in first adding the learned mean pixels intensity to the output and clipping the result to [0, 255].

The first task of the convolutional architecture is the computation of features characterizing the dependencies between the elements in X 0 . X 0 is thus fed into a stack of convolutional layers. This yields a stack Z 0 of l ∈ N * feature maps (see Figure 3). Similarly, X 1 is fed into another stack of convolutional layers. This yields a stack Z 1 of l feature maps.

All the elements in the context can be relevant for predicting any image block pixel. This implies that the information associated to all spatial locations in the context has to be merged. That is why the next layer in the convolutional architecture merges spatially Z 0 and Z 1 (see Figure 3). More precisely, for i ∈ [|1, l|], all the coefficients of the i th feature map of Z 0 and of the i th feature map of Z 1 are merged through affine combinations. Then, LeakyReLU with slope 0.1 is applied, yielding the merged stack Z of feature maps. Note that this layer bears similarities with the "channelwise fully-connected layer" [START_REF] Pathak | Context encoders: feature learning by inpainting[END_REF]. But, unlike the "channelwise fully-connected layer", it merges two stacks of feature maps of different height and width. Its advantage over a fully-connected layer is that it contains l times less weights.

The last task of the convolutional architecture is to merge the information of the different feature maps of Z. Z is thus fed into a stack of transposed convolutional layers [START_REF] Hoh | Learning deconvolution network for semantic segmentation[END_REF], [START_REF] Dumoulin | A guide to convolutional arithmetic for deep learning[END_REF]. This yields the predicted image block (see Figure 3). Note that all convolutional layers and transposed convolutional layers, apart from the last transposed convolutional layer, have LeakyReLU with slope 0.1 as non-linear activation. The last transposed convolutional layer has no non-linear activation due to the postprocessing discussed earlier.

B. Growth rate of the context size with the block size

Now that the architectures and the shape of the context are defined, the size of the context remains to be optimized. The causal neighborhood of the image block to be predicted used by the H.265 intra prediction modes is limited to one row of 2m+1 decoded pixels above the block and one column of 2m decoded pixels on the left side of the block. However, a context of such a small size is not sufficient for neural networks as a neural network relies on the spatial distribution of the decoded pixels intensity in the context to predict complex textures. Therefore, the context size has to be larger than 4m + 1.

But, an issue arises when the size of the context grows too much. Indeed, if the image block to be predicted is close to either the top edge of the decoded image D or its left edge, a large context goes out of the bounds of the decoded image. The neural network prediction is impossible. There is thus a tradeoff to find a suitable size for the context.

Let us look at decoded pixels above and on the left side of Y to develop intuitions regarding this tradeoff. When m is small, the long range spatial dependencies between these decoded pixels are not relevant for predicting Y (see Figure 4). In this case, the size of the context should be small so that the above-mentioned issue is limited. However, when m is large, such long range spatial dependencies are informative for predicting Y. The size of the context should now be large, despite the issue. Therefore, the context size should be a function of m q , q ≥ 1.

From there, we ran several preliminary experiments in which q ∈ {1, 2, 3} and the PSNR between Ŷ and Y was measured. The conclusion is that a neural network yields the best PSNRs when the size of the context grows with m 2 , i.e. the ratio between the size of the context and the size of the image block to be predicted is constant. This makes sense as, in the most common regression tasks involving neural networks, such as super-resolution [START_REF] Dong | Image super-resolution using deep convolutional networks[END_REF], [START_REF] Lim | Enhanced deep residual networks for single image super-resolution[END_REF], segmentation [START_REF] Badrinarayanan | Segnet: a deep convolutional encoder-decoder architecture for image segmentation[END_REF], [START_REF] Luc | Predicting deeper into the future of semantic segmentation[END_REF] or video interpolation [START_REF] Meyer | Phase-based frame interpolation for video[END_REF], [START_REF] Niklaus | Video frame interpolation via adaptive convolution[END_REF], [START_REF] Van Amersfoort | Frame interpolation with multi-scale deep loss functions and generative adversarial networks[END_REF], the ratio between the input image dimension and the output image dimension also remains constant while the height and width of the output image increase. Given the previous conclusions, X 0 is a rectangle of height 2m and width m. X 1 is a rectangle of height m and width 3m.

C. Criterion for selecting a proposed neural network

Now that the context is defined with respect to m, all that remains is to choose between a fully-connected neural network and a convolutional one according to m.

Convolutional neural networks allow better extracting 2D local image structures with fewer parameters compared to fully-connected neural networks [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. This is verified by the better prediction performances of the convolutional neural networks for m > 8 (see Sections IV-E and IV-F). However, this property becomes less critical for small block sizes. We indeed experimentally show that, when m ≤ 8, for the prediction problem, fully-connected architectures outperform convolutional architectures.

Therefore, when integrating the neural network based intra predictor into a codec like H.265, the criterion is to predict a block of width m > 8 via a proposed convolutional neural network, and a block of width m ≤ 8 via a fully-connected neural network.

D. Integration of the neural network based intra prediction into H.265

A specificity of H.265 is the quadtree structure partitioning, which determines the range of values for m and the number of available decoded pixels in the context.

In H.265 [START_REF] Wien | High efficiency video coding: coding tools and specification[END_REF], an image is partitioned into Coding Tree Units (CTUs). A CTU is composed of one luminance Coding Tree Block (CTB), two chrominance CTBs, and syntax elements. For simplicity, let us focus on a single CTB, e.g. the luminance CTB. The CTB size is a designed parameter but the commonly used CTB size is 64 × 64 pixels. A CTB can be directly used as Coding Block (CB) or can be split into 4 32 × 32 CBs. Then, each 32 × 32 CB can be iteratively split until the size of a resulting CB reaches a minimum size. The minimum size is a designed parameter. It can be as small as 8 × 8 pixels, and is set to this value in most configurations. A Prediction Block (PB) is a block on which the prediction is applied. If the size of a CB is not the minimum size, this CB is identical to its PB. Otherwise, in the case of intra prediction, this CB can be split into 4 4 × 4 PBs. More splittings of this CB into PBs exist for inter prediction [START_REF] Wien | High efficiency video coding: coding tools and specification[END_REF]. A recursive rate-distortion optimization finds the optimal splitting of each CTB.

Due to this partioning, m ∈ {4, 8, 16, 32, 64}. For each m ∈ {4, 8}, a fully-connected neural network is constructed with internal size p = 1200. Similarly, one convolutional neural network is constructed per block width m ∈ {16, 32, 64}. The convolutional architecture for each m ∈ {16, 32, 64} is detailed in Appendix A.

Another consequence of this partitioning is that the number of available decoded pixels in the context depends on m and the position of image block to be predicted in the current CTB. For instance, if the block is located at the bottom of the current CTB, the bottommost m 2 pixels in the context are not decoded yet. More generally, it might happen that a group of n 0 × m pixels, n 0 ∈ {0, 4, ..., m}, located at the bottom of the context, is not decoded yet. Similarly, a group of m × n 1 pixels, n 1 ∈ {0, 4, ..., m}, located furthest to the right in the context, may not have been decoded yet (see Figure 5). When pixels are not decoded yet, the solution in H.265 is to copy a decoded pixel into its neighboring undecoded pixels. But, this copy process cannot be re-used here. Indeed, it would fool the neural network and make it believe that, in an undecoded group of pixels and its surroundings, the spatial distribution of pixels intensity follows the regularity induced by the copy process. Alternatively, it is possible to indicate to a neural network that the two undecoded groups are unknown by masking them. The mask is set to the learned mean pixels intensity over the training set so that, after subtracting it from the context, the average of each input variable to a neural network over the training set is still near 0. More precisely regarding the masking, the first undecoded group in the context is fully covered by an α-mask of height n 0 and width m. The second undecoded group in the context is fully covered by an α-mask of height m and width n 1 (see Figure 5). The two α-masks together are denoted M n0,n1 . In Section IV-A, the neural networks will be trained so that they adapt to this variable number of available decoded pixels in the context.

Figure 6 summarizes the integration of the neural network based intra prediction scheme into H.265. The last issue to address is when no context is available. This occurs when the context goes out of the bounds of the decoded image, i.e. the pixel at the top-left of the context is not inside the decoded image. In this case, no neural network is used, and a zero prediction O of Y is returned. In Figure 6, e test extracts X from D with respect to the position of Y while applying M n0,n1 to X, following Figure 5.

IV. NEURAL NETWORKS TRAINING

This section explains how our neural networks are trained. Notably, an adaptation to the changing number of available decoded pixels in the input context is proposed. Then, an experiment shows the effectiveness of this approach. Moreover, this experiment compares the predictions of convolutional neural networks and those of fully-connected neural networks in terms of prediction PSNR in the case of large image blocks to be predicted.

A. Adaptation of the neural networks to the variable number of available decoded pixels via random context masking

The fact that n 0 and n 1 vary during the test phase, e.g. in H.265, has to be considered during the training phase. It would be unpractical to train one set of neural networks for each possible pair {n 0 , n 1 }. Instead, we propose to train the neural networks while feeding them with contexts containing a variable number of known pixels. More precisely, during the training phase, n 0 and n 1 are sampled uniformly from the set {0, 4, ..., m}. This way, the amount of available information in a training context is viewed as a random process the neural networks have to cope with.

B. Objective function to be minimized

The goal of the prediction is to minimize the Euclidean distance between the image block to be predicted and its estimate, or in other words to minimize the variance of the difference between the block and its prediction [START_REF] Wiegand | Source coding: part I of fundamentals of source and video coding[END_REF], also called the residue. The choice of the L2 norm is a consequence of the L2 norm chosen to measure the distortion between an image and its reconstruction. So, this Euclidean distance is minimized to learn the parameters θ m . Moreover, regularization by L2 norm of the weights (not the biases), denoted [θ m ] W , is applied [START_REF] Bengio | Practical recommendations for gradient-based training of deep architectures[END_REF] (see ( 2)).

min θm E [ Y c -f m (X c ; θ m ) 2 ] + λ [θ m ] W 2 2
(2)

Y c = Y -α
The expectation E [.] is approximated by averaging over a training set of image blocks to be predicted, each paired with its context. For learning the parameters φ m , (2) is used, replacing θ m with φ m and f m with g m .

The optimization algorithm is ADAM [START_REF] Kingma | Adam: a method for stochastic optimization[END_REF] with minibatches of size 100. The learning rate is 0.0001 for a fullyconnected neural network and 0.0004 for a convolutional one. The number of iterations is 800000. The learning is divided by 10 after 400000, 600000, and 700000 iterations. Regarding the other hyperparameters of ADAM, the recommended values [START_REF] Kingma | Adam: a method for stochastic optimization[END_REF] are used. The learning rates and λ = 0.0005 were found via an exhaustive search. For a given m, a neural network is trained for each combination of learning rate and λ value in {0.007, 0.004, 0.001, 0.0007, 0.0004, 0.0001, 0.00007, 0.00004} × {0.001, 0.0005, 0.0001}. We select the pair of parameter values giving the largest mean prediction PSNR on a validation set.

For both fully-connected and convolutional neural networks, the weights of all layers excluding the first one are initialized via Xavier's initialization [START_REF] Glorot | Understanding the difficulty of training deep feedforward neural networks[END_REF]. Xavier's initialization allows to accelerate the training of several consecutive layers by making the variance of the input to all layers equivalent. The problem concerning the first layer is that the pixel values in the input contexts, belonging to [-α, 255 -α], have a large variance, which leads to instabilities during training. That is why, in the first layer, the variance of the output is reduced with respect to that of the input by initializing the weights from a Gaussian distribution with mean 0 and standard deviation 0.01. The biases of all layers are initialized to 0.

C. Training data

The experiments in Sections IV-D and VI involve luminance images. That is why, for training, image blocks to be predicted, each paired with its context, are extracted from luminance images.

One 320 × 320 luminance crop I is, if possible, extracted from each RGB image in the ILSVRC2012 training set [START_REF] Deng | ImageNet: a large-scale hierarchical image database[END_REF]. This yields a set Γ = I (i) i=1...1048717 . The choice of the training set of pairs of contexts and image blocks to be predicted is a critical issue. Moreover, it needs to be handled differently for fully-connected and convolutional neural networks. Indeed, a convolutional neural network predicts large image blocks with complex textures, hence its need for high power of representation (see Appendix A). As a consequence, it overfits during training if no training data augmentation [START_REF] Ciresan | Deep, big, simple neural nets for handwritten digit recognition[END_REF], [START_REF] Ciresan | Multi-column deep neural networks for image classification[END_REF], [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], [START_REF] He | Deep residual learning for image recognition[END_REF] is used. In constrast, a fully-connected neural network predicts small blocks with simple textures, hence its need for relatively low power of representation. Thus, it does not overfit during training without training data augmentation. Moreover, we have noticed that a training data augmentation scheme creates a bottleneck in training time for a fully-connected neural network. Therefore, training data augmentation is used for a convolutional neural network exclusively. The dependencies between a block to be predicted and its context should not be altered during the training data augmentation. Therefore, in our training data augmentation scheme, the luminance crops in Γ are exclusively randomly rotated and flipped. Precisely, for each step of ADAM, the scheme is Algorithm 1. s rotation rotates its input image by angle ψ ∈ {0, π / 2, π, 3π / 2} radians. s flipping flips its input image horizontally with probability 0.5. e train is the same function as e test , except that e train extracts X (i) , Y (i) from potentially rotated and flipped I instead of extracting X from D and the position of the extraction is random instead of being defined by the order of decoding. For training a fully-connected neural network, X

ψ ∼ U 0, π 2 , π, 3π 2 n 0 , n 1 ∼ U {0, 4, ..., m} J = s flipping s rotation I (i) , ψ X (i) , Y (i) = e train (J, m, n 0 , n 1 , α) X (i) c = X (i) -α Y (i) c = Y (i) -α Output: X (i) c , Y (i) c i=1...100
.

during the test phase in a coding scheme, a context has quantization noise. This will be discussed during several experiments in Section VI-D.

D. Effectiveness of the random context masking

A natural question is whether the random context masking applied during training to adapt the neural networks to the variable number of known pixels in the context degrades the prediction performance. To address this question, a neural network trained with random context masking is compared to a set of neural networks, each trained with a fixed mask size. The experiments are performed using fully-connected neural networks for block sizes 4 × 4 and 8 × 8 pixels (f 4 and f 8 ), and convolutional neural networks for block sizes 16 × 16, 32 × 32, and 64 × 64 pixels (g 16 , g 32 , and g 64 ).

The experiments are carried out using the 24 RGB images in the Kodak suite [START_REF] Kodak Suite | [END_REF], converted into luminance. 960 image blocks to be predicted, each paired with its context, are extracted from these luminance images. Table I shows the average PSNR between the image block to be predicted and its prediction via PNNS over the 960 blocks, denoted PSNR PNNS,m , for each block width m and each test pair {n 0 , n 1 }. We see that a neural network trained with a fixed mask size has performance in terms of PSNR that significantly degrades when the mask size during the training phase and the test phase differ. By contrast, a neural network trained with random context masking allows to get the best (bold) or the second best (italic) performance in terms of PSNR for all the possible mask sizes during the test phase. Moreover, when the second best PSNR performance is achieved, the second best PSNR is very close to the best one.

A second set of experiments is related to the success rate µ PNNS,m of the neural network based prediction, i.e. the rate at which PNNS provides a better prediction in terms of PSNR than any other prediction of H.265. Again, the neural network trained with random context masking achieves the best (bold) or the second best rate (italic), where the second best rate is very close to the best rate (see Table II).

Therefore, we conclude that random context masking does not alter the performance of the trained neural networks. Besides, random context masking is an effective way of dealing with the changing number of known pixels in the context. Section VI will discuss rate-distortion performance. Note that, in order to fix n 0 and n 1 during the test phase, the previous experiments have been carried out outside H.265.

E. Relevance of convolutional neural networks for predicting large image blocks

Let us have a look at the overall efficiency of our convolutional neural networks for predicting large image blocks before comparing convolutional neural networks and fully-connected neural networks in this case. Figures

neural networks of PNNS yielding these predictions are trained via random context masking. Note also that n 0 = n 1 = 0 during the test phase. In Figure 11, the image block to be predicted contains the frame of a motorbike. There is no better H.265 intra prediction mode in terms of prediction PSNR than the DC mode in this case. In contrast, PNNS can predict a coarse version of the frame of the motorbike. In Figure 12, the block to be predicted contains lines of various directions. PNNS predicts a combination of diagonal lines, vertical lines and horizontal lines, which is not feasible for a H.265 intra prediction mode. However, Figures 9 network respectively in Figure 10. Therefore, the convolutional neural networks of PNNS is often able to model a large set of complex textures found in large image blocks. Table III compares our set of neural networks {f 4 , f 8 , g 16 , g 32 , g 64 } with the set of 4 fully-connected neural networks in [START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF], called Intra Prediction Fully-Connected Networks Single (IPFCN-S), in terms of prediction PSNR. "Single" refers to the single set of 4 fully-connected neural networks. This differs from the two sets of fully-connected neural networks mentioned later on in Section VI-C. The 4 fully-connected neural networks in [START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF] predict image blocks of sizes 4 × 4, 8 × 8, 16 × 16, and 32 × 32 pixels respectively. Let PSNR IPFCN-S,m be the PSNR between the image block and its prediction via IPFCN-S, averaged over the 960 blocks. We thank the authors in [START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF] for sharing the trained model of each fully-connected neural network of IPFCN-S. Table III reveals that the performance of PNNS in terms of prediction PSNR is better than that of IPFCN-S for all sizes of image blocks to be predicted. More interestingly, when looking at the success rate µ IPFCN-S,m of IPFCN-S, i.e. the rate at which IPFCN-S provides a better prediction in terms of PSNR than any other prediction of H.265, the difference µ PNNS,m -µ IPFCN-S,m increases with m (see Table IV). This shows that convolutional neural networks are more appropriate than fully-connected ones in terms of prediction PSNR for large block sizes. Note that, in Tables III andIV, there is no comparison for block width 64 pixels as this block width is not considered in [START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF].

F. Justification of fully-connected neural networks for predicting small image blocks

Now that the benefit of using convolutional neural networks rather than fully-connected ones for predicting large image blocks is shown, the choice of fully-connected neural networks for predicting small blocks must be justified. Table V displays the average PSNR between the image block to be predicted and its prediction over the 960 blocks for both a fullyconnected architecture and a convolutional one for each block width m ∈ {4, 8, 16}. The two fully-connected architectures f 4 and f 8 are already presented in Figure 2. The fullyconnected architecture f 16 is also that in Figure 2. The three convolutional architectures g 4 , g 8 , and g 16 are shown in Table XIII in Appendix A. We see that the convolutional neural network provides better prediction PSNRs on average than its fully-connected counterpart only when m > 8. This justifies why f 4 and f 8 are selected for predicting blocks of width 4 and 8 pixels respectively.

G. Different objective functions

In the objective function to be minimized over the neural network parameters, the distortion term involves the L2 norm of the difference between the image block to be predicted and its estimate (see [START_REF] Lainema | Intra coding of the HEVC standard[END_REF]). We also tried an alternative. This alternative consisted in replacing the L2 norm by the L1 norm. The choice of the L1 norm could be justified by the fact TABLE V: Comparison of the average prediction PSNR given by the fully-connected architecture and that given by the convolutional one for different pairs {n 0 , n 1 } during the test phase: (a) PSNR PNNS,4 given by f 4 and g 4 , (b) PSNR PNNS,8 given by f 8 and g 8 , and (c) PSNR PNNS, [START_REF] Ranzato | Video (language) modeling: a baseline for generative models of natural videos[END_REF] given by f 16 and g 16 . All neural networks are trained by drawing n 0 and n 1 uniformly from the set U {0, 4, 8, 12, ..., m}.

(c)
that, in H.265, for a given image block to be predicted, the criterion for selecting several intra prediction modes among all modes involves the sum of absolute differences between the image block to be predicted and its estimate 3 . But, this alternative distortion term did not yield any increase of the mean prediction PSNRs compared to those shown in Table I.

A common alternative to the regularization by L2 norm of the weights is the regularization by L1 norm of the weights [START_REF] Bengio | Practical recommendations for gradient-based training of deep architectures[END_REF]. Both approaches have been tested and it was observed that the regularization by L2 norm of the weights slightly reduces the mean prediction error on a validation set at the end of the training.

V. SIGNALLING OF THE PREDICTION MODES IN H.265

Before moving on to the experiments in Section VI where PNNS is integrated into a H.265 codec, the last issue is the signalling of the prediction modes inside H.265. Indeed, the integration of PNNS into H.265 requires to revisit the way all modes are signalled. Section V describes two ways of signalling PNNS into H.265. The purpose of setting up these two ways is to identify later on which signalling yields the largest PSNR-rate performance gains and discuss why a difference between them exists. The first signalling is the substitution of a H.265 intra prediction mode with PNNS. The second signalling is a switch between PNNS and the H.265 intra prediction modes.

A. Substitution of a H.265 intra prediction mode with PNNS

Section V-A first describes how H.265 selects the best of its 35 intra prediction modes for predicting a given image block. Based on this, a criterion for finding the mode to be replaced with PNNS is built.

To select the best of its 35 intra prediction modes for predicting a given image block, H.265 proceeds in two steps. During the first step, the 35 modes compete with one another. Each mode takes the causal neighborhood of the block to compute a different prediction of the block. The sum of absolute differences between the input block and its prediction is linearly combined with the mode signalling cost, yielding the mode "fast" cost 2 . The modes associated to a given number of lowest "fast" costs are put into a "fast" list 3 . During the second step, only the modes in the "fast" list compete with one another. The mode with the lowest rate-distortion cost is the best mode.

Knowing this, it seems natural to replace the mode that achieves the least frequently the lowest rate-distortion cost. Therefore, the frequency of interest ν m ∈ [0, 1] , m ∈ {4, 8, 16, 32, 64}, is the number of times a mode has the lowest rate-distortion cost when m = m divided by the number of times the above-mentioned selection process is run when m = m. To be generic, ν m should not be associated to luminance images of a specific type. That is why 100 380×480 luminance crops extracted from the BSDS300 dataset [START_REF] Martin | A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[END_REF] and 100 1200 × 1600 luminance crops extracted from the INRIA Holidays dataset [START_REF] Jegou | Hamming embedding and weak geometry consistency for large scale image search[END_REF] are encoded with H.265 to compute ν m . In this case, the mode of index 18 has on average the lowest ν m when m ∈ {4, 16, 32, 64} (see Figure 13). Note that this conclusion is verified with QP ∈ {22, 27, 32, 37, 42}. Note also that statistics about the frequency of selection of each H.265 intra prediction mode have already been analyzed [START_REF] Chernyak | Analysis of the intra predictions in H.265/HEVC[END_REF], [START_REF] Bougacha | Statistical analysis of intra prediction in HEVC video encoder[END_REF]. But, they are incomplete for our case as they take into account few videos and do not differentiate each value of m. Thus, PNNS replaces the H.265 mode of index 18.

As explained thereafter, the signalling cost the H.265 intra prediction mode of index 18 is variable and can be relatively large. When substituting the H.265 intra prediction mode of index 18 with PNNS, this variable cost transfers to PNNS. In contrast, the purpose of the second signalling of PNNS inside H.265 is to induce a fixed and relatively low signalling cost of PNNS.

B. Switch between PNNS and the H.265 intra prediction modes

The authors in [START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF] propose to signal the neural network mode with a single bit. This leads to the signalling of the modes summarized in Table VI. In addition, we modify the process of selecting the 3 Most Probable Modes (MPMs) [START_REF] Sullivan | Overview of the High Efficiency Video Coding (HEVC) Standard[END_REF] of the current PB to make the signalling of the modes even more efficient. More precisely, if the neural network mode is the mode selected for predicting the PB above the current PB or the PB on the left side of the current PB, then the neural network mode belongs to the MPMs of the current PB. As a result, redundancy appears as the neural network mode has not only the codeword 1 but also the codeword of a MPM. That is why, in the case where PNNS belongs to the MPMs of the current PB, we substitute each MPM being PNNS with either planar, DC or the vertical mode of index 26 such that the 3 MPMs of the current PB are different from each other. Besides, planar takes priority over DC, DC having priority over the vertical mode of index 26. See the code 1 for further details regarding this choice. 2 "fast" stresses that the cost computation is relatively low. 3 See "TEncSearch::estIntraPredLumaQT" at https://hevc.hhi.fraunhofer.de/ trac/hevc/browser/trunk/source/Lib/TLibEncoder/TEncSearch.cpp The key remark concerning Section V is that there is a tradeoff between the signalling cost of PNNS versus the signalling cost of the H.265 intra prediction modes. Indeed, the substitution (see Section V-A) keeps the signalling cost of the H.265 intra prediction modes constant but the signalling cost of PNNS can be up to 6 bits. Instead, the switch decreases the signalling cost of PNNS and raises that of each H.265 intra prediction mode.

VI. EXPERIMENTS

Now that two ways of signalling PNNS inside H.265 are specified, these two ways can be compared in terms of PSNR-rate performance gains. Moreover, PNNS integrated into H.265 can be compared to IPFCN-S integrated into H.265 in terms of PSNR-rate performance gains.

A. Experimental settings

The H.265 HM16.15 software is used in all the following experiments. The configuration is all-intra main. Note that the following settings only mention the integration of PNNS into H.265 via the substitution of the H.265 intra prediction mode of index 18 with PNNS, so called "PNNS substitution". But, the same settings apply to the integration of PNNS into H.265 via the switch between PNNS and the H.265 intra prediction modes, so called "PNNS switch". The PSNRrate performance of "PNNS substitution" with respect to H.265 is computed using the Bjontegaard metric [START_REF] Bjontegaard | Calculation of average PSNR differences between RDcurves[END_REF], which is the average saving in bitrate of the rate-distortion curve of "PNNS substitution" with respect to the rate-distortion curve of H.265. It is interesting to analyze whether there exists a range of bitrates for which "PNNS substitution" is more beneficial. That is why 3 different ranges of bitrates are presented. The first range, called "low rate", refers to QP ∈ {32, 34, 37, 39, 42}. The second range, called "high rate", corresponds to QP ∈ {17, 19, 22, 24, 27}. The third range, called "all rates", computes the Bjontegaard metric with the complete set of QP values from 17 to 42. The H.265 common test condition [START_REF] Rosewarne | Common test conditions and software reference configurations for HEVC range extensions[END_REF] recommends {22, 27, 32, 37} as QPs setting. Yet, we add several QPs to the recommended setting. This is because, to compute the Bjontegaard metric for "low rate" for instance, a polynom of degree 3 is fitted to rate-distortion points, and at least 4 rate-distortion points, i.e. 4 different QPs, are required to get a good fit. The most striking observation is that the PSNR-rate performance gains generated by "PNNS switch" are always larger than those provided by "PNNS substitution" (see Tables VII, VIII, IX, and X). This has two explanations. Firstly, "PNNS substitution" is hampered by the suppression of the original H.265 intra prediction mode of index 18. Indeed, the PSNRrate performance gain is degraded when the original H.265 intra prediction mode of index 18 is a relevant mode for encoding a luminance image. The most telling example is the luminance channel of the first frame of "BasketballDrill". When this channel is encoded via H.265, for the original H.265 intra prediction mode of index 18, ν 4 = 0.085, ν 8 = 0.100, ν 16 = 0.116, ν 32 = 0.085, and ν 64 = 0.088. This means that, compared to the average statistics in Figure 13, the original H.265 intra prediction mode of index 18 is used approximatively 10 times more frequently. This explains why the PSNR-rate performance gain provided by "PNNS substitution" is only 0.39% (see Table IX). The other way round, the luminance channel of the first frame of "BasketballDrive" is an insightful example. When this channel is encoded via H.265, for the original H.265 intra prediction mode of index 18, ν 4 = 0.004, ν 8 = 0.005, ν 16 = 0.004, ν 32 = 0.004, and ν 64 = 0.000. In this case, the original H.265 intra prediction mode of index 18 is almost never used. "PNNS substitution" thus yields 3.65% of PSNR-rate performance gain.

There is another explanation for the gap in PSNR-rate performance gain between "PNNS substitution" and "PNNS switch". As shown in Section IV-E, PNNS is able to model a large set of complex textures found in large image blocks. PNNS is also able to model a large set of simple textures found in small blocks. Following the principle of Huffman Coding, an intra prediction mode that gives on average predictions of good quality, such as PNNS, should be signalled using fewer bits. However, an intra prediction mode that seldom yields the highest prediction quality, such as the H.265 intra prediction mode of index 4, should be signalled using more bits. This corresponds exactly to the principle of the switch between PNNS and the H.265 intra prediction modes. Therefore, "PNNS switch" beats "PNNS substitution" in terms of PSNR-rate performance gains. Figures 14 and15 each compares the reconstruction of a luminance image via H.265 and its reconstruction via "PNNS switch" at similar reconstruction PSNRs. More visual comparisons are available on the website 5 .

Another interesting conclusion emerges when comparing "low rate" and "high rate". There is no specific range of bitrate for which "PNNS substitution" is more profitable (see Tables VII, VIII, IX, and X). Note that, in few cases, the PNSR-rate performance gain in "all rates" is slightly larger than those in "low rate" and "high rate". This happens when the area between the rate-distortion curve of "PNNS substitution" and the rate-distortion curve of H.265 gets relatively large in the range QP ∈ [START_REF] Kodak Suite | [END_REF][START_REF] Meyer | Phase-based frame interpolation for video[END_REF].

C. Comparison with the state-of-the-art Now, "PNNS switch" is compared to IPFCN-S integrated into H.265 in terms of PSNR-rate performance gains. It is important to note that the authors in [START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF] [START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF]. We observe that the PSNR-rate performance gains of "PNNS switch" are larger than those of IPFCN-S integrated into H.265, apart from the case of the video sequence "ParkScene" (see Table XI). Note that, for several videos sequences, the difference in PSNRrate performance gains between "PNNS switch" and IPFCN-S integrated into H.265 is significant. For instance, for the video sequence "BasketballPass", the PSNR-rate performance gain of "PNNS switch" is 3.08% whereas that of IPFCN-S integrated into H.265 is 1.1%. Therefore, the use of both fullyconnected neural networks and convolutional neural networks for intra prediction, the training with random context masking and the training data augmentation for training the convolutional neural networks of PNNS help boost the PSNR-rate performance gains. This is consistent with the conclusion in Section IV-E. Note that, even when comparing the PSNR-rate performance gains of "PNNS switch" with those of IPFCN-D integrated into H.265 which are reported in [START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF], "PNNS switch" often yields larger gains.

D. Robustness of the neural networks to quantization noise in their input context

In this section, we show that the proposed PNNS performs well with test data compressed with different QP values despite the fact that the neural networks of PNNS have been trained only once with no quantization noise in their input context. Indeed, let us consider two different "PNNS switch". In the first "PNNS switch", our 5 neural networks, one for each block size, are dedicated to all QPs. Note that the first "PNNS switch" corresponds to the "PNNS switch" that has been used so far. In the second "PNNS switch", a first set of 5 neural networks is dedicated to QP ≤ 27 whereas a second set is dedicated to QP > 27. Unlike the first set of neural networks, the second set is trained on contexts that are encoded and decoded via H.265 with QP ∼ U {32, 37, 42} for each training context. For the third test set, the difference in PSNR-rate performance gain between the first "PNNS switch" and the second "PNNS switch" ranges between 0.0% and 0.1%. This means that there is no need to train the neural networks of PNNS on contexts with quantization noise.

E. Complexity

A fully-connected neural network needs an overcomplete representation to provide predictions with high quality. That is why the number of neurons in each fully-connected layer is usually much larger than the size of the context. Likewise, the number of feature maps in each convolutional layer of a convolutional neural network is usually large. This incurs a high computational cost. Table XII gives the encoding and decoding times for "PNNS switch" and IPFCN-S and shows comparable running times for both solutions. A Bi-Xeon CPU E5-2620 is used for "PNNS switch".

F. Memory consumption

In addition to the complexity, another issue arising from the integration of PNNS into H.265 is the increase of the memory consumption. For a regular intra prediction mode in H.265, the context around the image block to be predicted consists of one row of 2m + 1 pixels above the block and one column of 2m pixels on the left side of the block. However, for a neural network of PNNS, the context around the image block consists of m rows of 3m pixels above the block and m columns of 2m pixels on the left side of the block (see Figure 5). Therefore, the hardware must be adapted to handle the large memory required by the input contexts to the neural networks of PNNS.

VII. CONCLUSION

This paper has presented a set of neural network architectures, including both fully-connected neural networks and convolutional neural networks, for intra prediction. It is shown that fully-connected neural networks are well adapted to the prediction of image blocks of small sizes whereas convolutional ones provide better predictions in large blocks. Our neural networks are trained via a random context masking of their context so that they adapt to the variable number of available decoded pixels for the prediction in a coding scheme. When integrated into a H.265 codec, the proposed neural networks are shown to give rate-distortion performance gains compared with the H.265 intra prediction. Moreover, it is shown that these neural networks can cope with the quantization noise present in the prediction context, i.e. they can be trained on undistorted contexts, and then generalize well on distorted contexts in a coding scheme. This greatly simplifies training as quantization noise does not need to be taken into account during training.

APPENDIX A CONVOLUTIONAL ARCHITECTURES FOR H.265

The architecture of the stack of convolutional layers g c m . ; φ c,0 m that is applied to the context portion X 0 located on the left side of the image block to be predicted is shown in Table XIII for m ∈ {4, 16, 64}. The architecture of the stack of convolutional layers g c m . ; φ c,1 m that is applied to the context porton X 1 located above the image block is identical to that of g c m . ; φ c,0 m . The slope of LeakyReLU is equal to 0.1. The architecture of g c ; φ c,0 64 but, in the third layer, the filter size is 3 × 3 × 128, the number of filters is 128, and the stride is 1. Moreover, in the fourth layer, the filter size is 5 × 5 × 128 and the number of filters is 256. In the fifth layer, the filter size is 3 × 3 × 256 and the number of filters is 256.

To obtain the architecture of g t m , each sequence of layers in g c m is reversed. Besides, each convolution is replaced by a transposed convolution (see Table XIII).

Fig. 1 :

 1 Fig. 1: Illustration of the relative positions of X, X 0 , X 1 , and Y.

Fig. 2 :

 2 Fig. 2: Illustration of the fully-connected architecture f m . FC:p|LeakyReLU denotes the fully-connected layer with p output neurons and LeakyReLU with slope 0.1 as non-linear activation. Unlike FC:p|LeakyReLU, FC:p has no non-linear activation. θ m gathers the weights and biases of the 4 fullyconnected layers of f m .

Fig. 3 :Fig. 4 :

 34 Fig. 3: Illustration of the convolutional architecture g m in which g c m , g m , and g t m denote respectively a stack of convolutional layers, the merger, and the stack of transposed convolutional layers. φ c m , φ m , and φ t m gather the weights and biases of respectively g c m , g m , and g t m . φ m = φ c,0 m , φ c,1 m , φ m , φ t m .

Fig. 5 :

 5 Fig. 5: Illustration of the masking of the undecoded pixels in X for H.265. The luminance channel of the first image in the Kodak suite is being encoded via H.265 with QP = 37. Here, m = 8 and n 0 = n 1 = 4.

Fig. 6 :

 6 Fig. 6: Illustration of the neural network based intra prediction scheme inside H.265.

Algorithm 1

 1 c i=1...10000000 is generated offline from Γ, i.e. before the training starts. The issue regarding this generation of training data is that the training contexts have no quantization noise whereas, Training data augmentation for the convolutional neural networks Inputs: Γ, m, α. ∀i ∈ [|1,100|],i ∼ U[START_REF] Wien | High efficiency video coding: coding tools and specification[END_REF] 1048717]

Fig. 7 :Fig. 8 :

 78 Fig. 7: Prediction of a block of size 16 × 16 pixels via the best H.265 mode in terms of PSNR and PNNS: (a) H.265 causal neighborhood, (b) PNNS context, (c) block to be predicted, (d) predicted block via the best H.265 mode (of index 27) in terms of PSNR, and (e) predicted block via PNNS.

Fig. 9 :

 9 Fig. 9: Prediction of a block of size 16 × 16 pixels via the best H.265 mode in terms of PSNR and PNNS: (a) H.265 causal neighborhood, (b) PNNS context, (c) block to be predicted, (d) predicted block via the best H.265 mode (of index 25) in terms of PSNR, and (e) predicted block via PNNS.

Fig. 10 :Fig. 11 :

 1011 Fig. 10: Prediction of a block of size 16×16 pixels via the best H.265 mode in terms of PSNR and PNNS: (a) H.265 causal neighborhood, (b) PNNS context, (c) block to be predicted, (d) predicted block via the best H.265 mode (of index 29) in terms of PSNR, and (e) predicted block via PNNS.

Fig. 12 :

 12 Fig. 12: Prediction of a block of size 64×64 pixels via the best H.265 mode in terms of PSNR and PNNS: (a) H.265 causal neighborhood, (b) PNNS context, (c) block to be predicted, (d) predicted block via the best H.265 mode (planar) in terms of PSNR, and (e) predicted block via PNNS.

Fig. 13 :

 13 Fig. 13: Analysis of (a, b) ν 4 , (c, d) ν 16 , and (e, f) ν 64 for each mode. 100 luminance crops from either (a, c, e) the BSDS300 dataset or (b, d, f) the INRIA Holidays dataset are encoded via H.265 with QP = 32.

Four

  test sets are used to cover a wide variety of images. The first test set contains the luminance channels of respectively Barbara, Lena, Mandrill and Peppers. The second test set contains the 24 RGB images in the Kodak suite, converted into luminance. The third test set gathers the 13 videos sequences of the classes B, C, and D of the H.265 CTC, converted into luminance. The fourth test set contains 6 widely used videos sequences4 , converted into luminance. Our work is dedicated to image coding. That is why only the first frame of each video sequence in the third and fourth test sets are considered. It is important to note that the training in Section IV, the extraction of the frequency of selection of each H.265 intra prediction mode in Section V, and the current experiments involve 7 distinct sets of luminance images. This way, PNNS is not tuned for any specific test luminance image.

8 . ; φ c,0 8 is similar to that of g c 4 . 32 .

 88432 ; φ c,0 4 but, in the first layer, the filter size is 5 × 5 × 1, the number of filters is 64, and the stride is 2. Moreover, in the second layer, the filter size is 3 × 3 × 64 and the number of filters is 64. The architecture of g c ; φ c,0 32 is similar to that of g c 64 .

Fig. 14 :Fig. 15 :

 1415 Fig. 14: Comparison of (a) a 100 × 100 crop of the luminance channel of the first frame of BQMall, (b) its reconstruction via H.265, and (c) its reconstruction via "PNNS switch". QP = 27. For the luminance channel of the first frame of BQMall, for H.265, {rate = 0.670 bpp, PSNR = 38.569 dB}. For "PNNS switch", {rate = 0.644 bpp, PSNR = 38.513 dB}.

TABLE I :

 I Comparison of (a) PNSR PNNS,4 , (b) PSNR PNNS,8 , (c) PSNR PNNS,16 , (d) PSNR PNNS,32 , and (e) PSNR PNNS,64 for different pairs {n 0 , n 1 } during the training and test phases. U {0, 4, ..., m} denotes the fact that, during the training phase, n 0 and n 1 are uniformly drawn from the set {0, 4, 8, 12, ..., m}. Test {n 0 , n 1 } Training g 16 with {n 0 , n 1 } {0, 0} {0, 16} {16, 0} {16, 16} U {0, 4, ..., 16} Training g 32 with {n 0 , n 1 } {0, 0} {0, 32} {32, 0} {32, 32} U {0, 4, ..., 32}

	Test {n 0 , n 1 }	Training f 4 with {n 0 , n 1 } {0, 0} {0, 4} {4, 0} {4, 4} U {0, 4}
	{0, 0}		34.63 34.39 34.44 34.23	34 .57
	{0, 4}		32.90 34 .39 32.82 34.23	34.42
	{4, 0}		32.79 32.68 34.44 34.23	34 .39
	{4, 4}		30.93 32.68 32.82 34.23	34 .20
					(a)
	Test {n 0 , n 1 }	Training f 8 with {n 0 , n 1 } {0, 0} {0, 8} {8, 0} {8, 8} U {0, 4, 8}
	{0, 0}		31.85 31.73 31.68 31.62	31 .79
	{0, 8}		30.93 31.73 30.83 31.62	31 .66
	{8, 0}		30.70 30.75 31.68 31.62	31.68
	{8, 8}		29.58 30.75 30.83 31.62	31 .56
					(b)
	{0, 0}	29 .23 22.69	24.85	20.76	29.25
	{0, 16}	28.65 29.12	24.66	23.99	29 .11
	{16, 0}	28.43	22.60	29 .06	22.99	29.12
	{16, 16}	27.87	28.37	28.35	28.98	28 .97
					(c)
	{0, 0}	25.93 22.13	22.29	18.85	25 .92
	{0, 32}	25.39 25.79	22.18	21.75	25 .76
	{32, 0}	25.41	22.02	25.82	20.38	25 .80
	{32, 32}	25.00	25.31	25.32	25.63	25 .62
					(d)
	Test {n 0 , n 1 }	Training g 64 with {n 0 , n 1 } {0, 0} {0, 64} {64, 0} {64, 64} U {0, 4, ..., 64}
	{0, 0}	21 .46 19.41	19.78	18.17	21.47
	{0, 64}	21.27 21 .35	19.68	19.95	21.38
	{64, 0}	21.11	19.18	21.34	19.06	21.34
	{64, 64}	20.94	21.08	21.16	21.27	21.27

Test {n 0 , n 1 }

  7, 8, 9, 10, 11, and 12 each compares the prediction of an image block provided by the best H.265 intra prediction mode in terms of prediction PSNR and the prediction provided by PNNS. Note that, the

TABLE II :

 II Comparison of success rates in percentage (a) µ PNNS,4 , (b) µ PNNS,8 , (c) µ PNNS,16 , (d) µ PNNS,32 , and (e) µ PNNS,64 for different pairs {n 0 , n 1 } during the training and test phases. U {0, 4, ..., m} denotes the fact that, during the training phase, n 0 and n 1 are uniformly drawn from the set {0, 4, 8, 12, ..., m}.

	Test {n 0 , n 1 }		Training f 4 with {n 0 , n 1 } {0, 0} {0, 4} {4, 0} {4, 4} U {0, 4}
	{0, 0}			22%	17%	19%	16%	19 %
	{0, 4}			15%	18%	13%	15%	17 %
	{4, 0}			11%	11%	20%	17%	19 %
	{4, 4}			11%	12%	13%	16%	15 %
					(a)
	Test {n 0 , n 1 }	Training f 8 with {n 0 , n 1 } {0, 0} {0, 8} {8, 0} {8, 8} U {0, 4, 8}
	{0, 0}		33%	30%		30%	30%	32 %
	{0, 8}		21%	31%		20%	31%	29%
	{8, 0}		21%	20%		34%	31%	32 %
	{8, 8}		16%	20%		20%	31%	30 %
					(b)
	Test {n 0 , n 1 }	Training g 16 with {n 0 , n 1 } {0, 0} {0, 16} {16, 0} {16, 16} U {0, 4, ..., 16}
	{0, 0}	55%	18%	20%	9%	54 %
	{0, 16}	42%	51%	18%	18%	50 %
	{16, 0}	40%	15%	51 %	17%	53%
	{16, 16}	33%	36%	40%	52%	49 %
					(c)
	Test {n 0 , n 1 }	Training g 32 with {n 0 , n 1 } {0, 0} {0, 32} {32, 0} {32, 32} U {0, 4, ..., 32}
	{0, 0}	63%	30%	30%	14%	63%
	{0, 32}	54%	61%	27%	27%	61%
	{32, 0}	54%	28%	63%	22%	62 %
	{32, 32}	46%	53%	52%	60%	60%
					(d)
	Test {n 0 , n 1 }	Training g 64 with {n 0 , n 1 } {0, 0} {0, 64} {64, 0} {64, 64} U {0, 4, ..., 64}
	{0, 0}	68%	39%	43%	27%	67 %
	{0, 64}	63%	64 %	41%	44%	65%
	{64, 0}	62%	36%	66 %	38%	68%
	{64, 64}	57%	61%	63%	64 %	66%

TABLE III :

 III Comparison of PSNR PNNS,m and PSNR IPFCN-S,m[START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF] for m ∈ {4, 8, 16, 32, 64}. f 4 , f 8 , g 16 , g 32 , and g 64 are trained via random context masking. During the test phase, n 0 = n 1 = 0.

	m PSNR PNNS,m PSNR IPFCN-S,m [14]
	4	34.57	33.70
	8	32.01	31.44
	16	29.25	28.71
	32	25.52	24.96
	64	21.47	-

TABLE IV :

 IV Comparison of success rates in percentage µ PNNS,m and µ IPFCN-S,m[START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF], for m ∈ {4, 8, 16, 32, 64}. f 4 , f 8 , g 16 , g 32 , and g 64 are trained via random context masking. During the test phase, n 0 = n 1 = 0.

	m µ PNNS,m µ IPFCN-S,m [14]
	4	19%	14%
	8	31%	26%
	16	54%	35%
	32	60%	41%
	64	67%	-

TABLE VI :

 VI Signalling of the modes described in[START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF].

	Mode	codeword
	Neural network mode	1
	First MPM	010
	Second MPM	0110
	Third MPM	0111
	Non-MPM and non-neural network mode 00 {5bits}

TABLE VII :

 VII PSNR-rate performance gains compared with H.265 of "PNNS substitution" and "PNNS switch" for the first test set.

			PSNR-rate performance gain
	Image name	"PNNS substitution"	"PNNS switch"
		Low rate High rate All rates	All rates
	Barbara	2.47%	1.31%	1.79%	2.77%
	Lena	1.68%	2.11%	2.05%	3.78%
	Mandrill	0.77%	0.58%	0.67%	1.46%
	Peppers	1.61%	1.50%	1.71%	3.63%

TABLE VIII :

 VIII PSNR-rate performance gains compared with H.265 of "PNNS substitution" and "PNNS switch" for the second test set.

			PSNR-rate performance gain
	Kodak image index	"PNNS substitution"	"PNNS switch"
		Low rate High rate All rates	All rates
	1	1.20%	0.74%	0.95%	2.06%
	2	0.59%	0.91%	0.88%	2.16%
	3	0.91%	2.04%	1.59%	3.22%
	4	1.78%	1.75%	1.80%	3.23%
	5	1.40%	2.45%	2.08%	4.01%
	6	1.43%	0.81%	1.12%	2.11%
	7	1.12%	2.36%	1.76%	3.86%
	8	1.01%	0.83%	0.98%	1.79%
	9	1.54%	1.43%	1.46%	3.05%
	10	2.20%	2.37%	2.42%	3.84%
	11	0.93%	0.91%	1.00%	2.41%
	12	0.96%	1.02%	1.07%	2.33%
	13	0.83%	0.5%	0.64%	1.76%
	14	0.96%	1.28%	1.20%	2.76%
	15	1.53%	1.19%	1.37%	2.62%
	16	0.66%	0.62%	0.70%	1.69%
	17	1.35%	2.03%	1.80%	3.69%
	18	0.68%	0.96%	0.88%	1.98%
	19	1.44%	0.86%	1.05%	2.06%
	20	0.92%	1.61%	1.38%	2.71%
	21	0.99%	0.83%	0.94%	2.28%
	22	0.56%	0.88%	0.78%	2.22%
	23	1.20%	2.45%	2.03%	4.20%
	24	0.68%	0.87%	0.80%	1.73%

TABLE IX

 IX

		: PSNR-rate performance gains compared with
	H.265 of "PNNS substitution" and "PNNS switch" for the
	third test set.
				PSNR-rate performance gain
		Video sequence	"PNNS substitution"	"PNNS switch"
			Low rate High rate All rates	All rates
		BQTerrace	1.66%	0.95%	1.29%	2.44%
		BasketballDrive	4.80%	2.87%	3.65%	5.20%
	B	Cactus	1.48%	1.51%	1.58%	3.05%
		ParkScene	0.64%	1.16%	0.97%	2.58%
		Kimono	1.28%	1.55%	1.65%	2.92%
		BQMall	1.20%	1.30%	1.30%	3.14%
	C	BasketballDrill RaceHorsesC	-1.18% 1.34%	1.34% 1.58%	0.39% 1.53%	3.50% 3.29%
		PartyScene	1.02%	0.91%	0.96%	2.42%
		BQSquare	0.79%	0.86%	0.86%	2.21%
	D	BasketballPass BlowingBubbles 0.66% 1.61%	1.80% 1.22%	1.48% 1.02%	3.08% 2.65%
		RaceHorses	1.32%	1.63%	1.54%	3.28%

TABLE X

 X

	B. Analysis of the two ways of signalling the PNNS mode
	inside H.265
	: PSNR-rate performance gains compared with
	H.265 of "PNNS substitution" and "PNNS switch" for the
	fourth test set.
			PSNR-rate performance gain
	Video sequence	"PNNS substitution"	"PNNS switch"
		Low rate High rate All rates	All rates
	Bus	1.67%	1.17%	1.45%	2.74%
	City	1.55%	1.19%	1.35%	2.51%
	Crew	1.56%	1.24%	1.38%	3.10%
	Football	1.44%	1.78%	1.78%	3.52%
	Harbour	1.80%	0.73%	1.25%	2.51%
	Soccer	0.96%	0.95%	1.03%	1.90%

  develop two versions of their set of 4 fully-connected neural networks for intra prediction. The first version, called IPFCN-S, is the one used in Section IV-E. The 4 fully-connected neural networks are trained on an unconstrained training set of image blocks to be predicted, each paired with its context. The second version is called Intra Prediction Fully-Connected Networks Double (IPFCN-D). The training data are dissociated into two groups. One group gathers image blocks exhibiting textures with angular directions, each paired with its context. The other group gathers image blocks exhibiting textures with nonangular directions, each paired with its context. In IPFCN-D, there are two sets of 4 fully-connected neural networks, each set being trained on a different group of training data. Then, the two sets are integrated into H.265. IPFCN-D gives slightly larger PSNR-rate performance gains than IPFCN-S. The comparison below involves IPFCN-S as our training set is not dissociated. But, this dissociation could also be applied to the training set of the neural networks of PNNS. "PNNS switch" and IPFCN-S integrated into H.265 are compared on the third test set. The PSNR-rate performance gains of IPFCN-S are reported from

TABLE XI :

 XI PSNR-rate performance gains of our proposed "PNNS switch" and IPFCN-S[START_REF] Li | Fully-connected networkbased intra prediction for image coding[END_REF] inside H.265 for the third test set. The reference is H.265.

		Video sequence	PSNR-rate performance gain our "PNNS switch" IPFCN-S [14] inside H.265
		BQTerrace	2.44%	1.8%
		BasketballDrive	5.20%	3.4%
	B	Cactus	3.05%	2.7%
		ParkScene	2.58%	2.8%
		Kimono	2.92%	2.7%
		BQMall	3.14%	2.0%
	C	BasketballDrill RaceHorsesC	3.50% 3.29%	1.1% 2.9%
		PartyScene	2.42%	1.3%
		BQSquare	2.21%	0.6%
	D	BasketballPass BlowingBubbles	3.08% 2.65%	1.1% 1.6%
		RaceHorses	3.28%	2.8%

TABLE XII :

 XII Average computation time ratio with respect to H.265.

		our "PNNS switch" IPFCN-S [14] inside H.265
	Encoding	51	46
	Decoding	191	190

ftp://ftp.tnt.uni-hannover.de/pub/svc/testsequences/

https://www.irisa.fr/temics/demos/prediction neural network/ PredictionNeuralNetwork.htm