Navaneeth Kamballur Kottayil

Giuseppe Valenzise

Frederic Dufaux

Irene Cheng

LEARNING LOCAL DISTORTION VISIBILITY FROM IMAGE QUALITY

Accurate prediction of local distortion visibility thresholds is critical in many image and video processing applications. Existing methods require an accurate modeling of the human visual system, and are derived through pshycophysical experiments with simple, artificial stimuli. These approaches, however, are difficult to generalize to natural images with complex types of distortion. In this paper, we explore a different perspective, and we investigate whether it is possible to learn local distortion visibility from image quality scores. We propose a convolutional neural network based optimization framework to infer local detection thresholds in a distorted image. Our model is trained on multiple quality datasets, and the results are correlated with empirical visibility thresholds collected on complex stimuli in a recent study. Our results are comparable to state-of-the-art mathematical models that were trained on phsycovisual data directly. This suggests that it is possible to predict psychophysical phenomena from visibility information embedded in image quality scores.

INTRODUCTION

Predicting the visibility of visual distortion is of paramount importance in a number of image processing applications, such as image compression [START_REF] Chandler | A Patch-Based Structural Masking Model with an Application to Compression[END_REF][START_REF] Watson | DCTune: A technique for visual optimization of DCT quantization matrices for individual images[END_REF], watermarking [START_REF] Koz | Oblivious Spatio-Temporal Watermarking of Digital Video by Exploiting the Human Visual System[END_REF] and quality assessment [START_REF] Daly | Visible differences predictor: an algorithm for the assessment of image fidelity[END_REF][START_REF] Chandler | VSNR: A wavelet-based visual signal-to-noise ratio for natural images[END_REF]. Models of contrast detection and masking have been widely studied in the literature, e.g., [START_REF] Daly | Visible differences predictor: an algorithm for the assessment of image fidelity[END_REF][START_REF] Watson | Model of visual contrast gain control and pattern masking[END_REF][START_REF] Carney | Modelfest: year one results and plans for future years[END_REF]. These studies are based on an accurate modeling of the human visual system (HVS), including intra-ocular scattering, contrast sensitivity at different frequencies, luminance masking, intra-and inter-channel masking, etc.

Traditionally, measurements of distortion visibility are obtained through psychophysical experiments employing simple, artificial stimuli, such as Gabor patterns, sine-wave gratings, or wide-band noise [START_REF] Legge | Contrast masking in human vision[END_REF][START_REF] Watson | Model of visual contrast gain control and pattern masking[END_REF]. The simplicity and nonnaturalness of the stimuli enable to describe them through well-defined features (e.g., frequency, brightness, etc.). These features are then used to derive mathematical models of distortion visibility. In order to understand how these observations generalize to the case of natural image masks, Alam et al. [START_REF] Alam | Local masking in natural images: A database and analysis[END_REF] have recently presented a dataset of local masking for natural scenes. Specifically, they use Gabor patterns as stimuli and natural images as masks to measure detection thresholds -the minimum magnitude of the stimulus making it distinguishable from the background mask -for small image patches. Later, this data has been used to train a convolutional neural network in order to predict the detection threshold of a patch from its input pixel values [START_REF] Alam | A computational model for predicting local distortion visibility via convolutional neural network trainedon natural scenes[END_REF].

Conventional approaches to model distortion visibility strongly rely upon psychophysical experiments that are, in their nature, based on a simplification of real-world conditions. For example, models that describe visibility of sine-wave gratings might be enough to predict visibility in DCT-based image compression; however, they are probably failing in modeling different or multiple concurrent artifacts. Furthermore, local visibility is influenced by surrounding regions, and is ultimately linked to image semantics. It is evident that modeling all these complex factors only through psychophysical experiments is unfeasible.

In this work, we explore a different perspective. Instead of learning distortion visibility directly from psychophysical data, we propose to learn it indirectly, leveraging the large availability of alternative, yet related, data: subjectively annotated image quality assessment (IQA) datasets. Image quality scores provide higher level information about the visual appearance of a picture, compared to psychophysical measurements. At the same time, they bring information about the visibility of distortion. Indeed, a common assumption in image quality assessment is that the perceived quality is directly related to the visibility of the error signal [START_REF] Zhu | A no-reference objective image quality metric based on perceptually weighted local noise[END_REF][START_REF] Wang | Image Quality Assessment: From Error Visibility to Structural Similarity[END_REF]. In other words, the per pixel error is weighted locally by the ensemble of perceptual phenomena, such as contrast sensitivity and several forms of masking, which discount its visibility to the human visual system.

Visual quality scores thus implicitly embed latent information on error visibility. In our recent work [START_REF] Kottayil | Blind Quality Estimation by Disentangling Perceptual and Noisy Features in High Dynamic Range Images[END_REF], we have proposed a deep convolutional neural network (CNN) architecture to disentangle the per pixel distortion and what we called the "perceptual resistance", in the context of noreference quality estimation of high dyanamic range compressed pictures. Our results demonstrated that it is possible to effectively estimate these two terms over a broad range of qualities, starting from supra-threshold quality scores. In this paper, we investigate whether a similar approach can also be used to predict near-threshold visibility. To this end, we train our proposed system in a full-reference fashion, i.e., we assume that the error signal is known, as it is the case in most IQA datasets. However, the inference step does not require the knowledge of the error, and can produce an estimate of local masking for any input image. Interestingly (and perhaps surprisingly), we find that perceptual scaling learned from image quality scores can predict the detection thresholds in [START_REF] Alam | Local masking in natural images: A database and analysis[END_REF] with similar accuracy as the CNN-based regressor in [START_REF] Alam | A computational model for predicting local distortion visibility via convolutional neural network trainedon natural scenes[END_REF], although our model is learned on other datasets with different contents and several kinds of visual impairements. This makes the proposed approach potentially more general than previous work.

The rest of the paper is organized as follows. In Section 2, we describe our model to derive visibility threshold from image quality scores. In Section 3, we compare our predictions to state-of-the-art models on a local masking visibility dataset. Finally, we draw conclusions in Section 4.

PROPOSED MODEL

In this section we present our proposed model to estimate local distortion visibility thresholds. Specifically, we first discuss the assumptions of our model and provide a mathematical framework to split (supra-threshold) quality scores into per pixel error and a perceptual scaling term, which we show in Section 3 to be a good predictor of visibility thresholds in local masking. Afterwards, we describe how to implement this model using a deep convolutional neural network architecture.

Mathematical framework and assumptions

Let I R and I D be an original reference image and its distorted version, respectively, and Q ∈ [0, 1] the quality score for I D . Without loss of generality, we assume that Q is given as a differential mean opinion score (DMOS). We assume that we have access to the local quality q(i, j) of an image patch I D (i, j) of size N × N pixels, centered at location (i, j). The pooling process linking the quality of individual patches to the overall quality Q may depend on many factors, e.g., saliency [START_REF] Moorthy | Visual Importance Pooling for Image Quality Assessment[END_REF]. Here, we assume that the local quality of an image block is the same as the global image quality score, similarly to the setting in [START_REF] Kang | Convolutional neural networks for no-reference image quality assessment[END_REF]. While this is a strong assumption, which is often not met in practice, it has been proved to be accurate enough to predict image quality [START_REF] Kang | Convolutional neural networks for no-reference image quality assessment[END_REF][START_REF] Kottayil | Blind Quality Estimation by Disentangling Perceptual and Noisy Features in High Dynamic Range Images[END_REF].

In order to model local quality, we further assume that per pixel distortion in a patch is discounted by a perceptual weight, T (i, j), that accounts for typical masking and visibility effects. Specifically, we measure pixel distortion through the average absolute error E(i, j) of a patch, defined as:

E(i, j) = 1 N 2 N 2 k=1 |I D (i, j) k -I R (i, j) k | , ( 1
)
where k is the pixel index in the patch. We then approximate local quality as a function of the error and the perceptual weight, that is:

q(i, j) ≈ 1 -exp α • E(i, j) T (i, j) β , (2)
where α and β are two model parameters.

Notice that this formulation has been previously used to model the probability of detecting localized noise distortion, e.g., in [START_REF] Zhu | A no-reference objective image quality metric based on perceptually weighted local noise[END_REF], but we introduce an additional scaling factor α. Eq. ( 2) is inspired by the common practice in vision science of expressing the magnitude of the error in multiples of the just-noticeable difference [START_REF] Watson | DCTune: A technique for visual optimization of DCT quantization matrices for individual images[END_REF], although the relationship in this case is nonlinear. We therefore refer to T (i, j) as the local visibility threshold, even if this is technically correct only for near-threshold distortion. We show in Section 3 that, for this latter case, T does indeed model local visibility. Moreover, the thresholds we estimate are not in the same scale as those obtained through psychophysical experiments. We compensate for this by means of the parameter α, which provides an additional degree of freedom in the optimization to push the predicted local quality as close as possible to ground truth. The value of the parameter β is generally found by matching the results of psychophysical experiments. In practice, we found that the choice of β does not significantly affect the performance of our model, and α alone provides already enough flexibility to minimize the loss function. Hence, we simply set β = 1 in Eq. (2).

Implementation

A scheme of the proposed model is shown in Figure 1. The input of our systems are overlapping patches of 32 × 32 pixels, similar to [START_REF] Kang | Convolutional neural networks for no-reference image quality assessment[END_REF][START_REF] Kottayil | Blind Quality Estimation by Disentangling Perceptual and Noisy Features in High Dynamic Range Images[END_REF]. The distortion, E(i, j), is computed directly as in Eq. (1) during the training, where both I D and I R are available. The local visibility thresholds T (i, j) are computed in a module that we name "P-net", while an estimate of local quality, q(i, j) is obtained by implementing Eq. (2) in the "Mixing function" block. Notice that this structure is required for training the P-net, as T is considered a latent variable which depends implicitly on the observations of the input content and perceived quality. Instead, for inference the P-net is employed as a standalone block. Furthermore, we are generally interested in applying the learned P-net on the original pictures, rather than on the distorted ones. However, we found that training the P-net with noisy versions of the image was more effective, as this increases the variability of input data, leading to improved generalization capabilities.

Our model is trained to predict local quality q(i, j) using the ground-truth quality q(i, j) as targets, by minimizing the following cost function: J(i, j) = |q(i, j) -q(i, j)| .

(

) 3
Notice that q(i, j) depends implicitly on the latent variables T (i, j) through Eq. ( 2). Thus, when optimizing J, the visibility thresholds are adjusted in such a way to weigh the error coherently with the observed ground-truth quality.

For the architecture of P-net, we make use of a handcrafted layer that we named as augmented input layer [START_REF] Kottayil | Blind Quality Estimation by Disentangling Perceptual and Noisy Features in High Dynamic Range Images[END_REF]. In this layer, in addition to the luminance values of the N × N block, we compute the mean, variance and Mean Subtracted Contrast Normalized (MSCN) image [START_REF] Moorthy | Blind image quality assessment: From natural scene statistics to perceptual quality[END_REF]. The latter is defined as:

M SCN (x, y) = I(x, y) -µ M [I(x, y)] σ M [I(x, y)] + , (4)
where (x, y) denotes the location of a pixel in the patch, µ M [I(x, y)] is the mean and σ M [I(x, y)] the variance of the patch, computed by replacing every pixel (x, y) with the mean and variance, respectively, over a local Gaussian window of size M ≤ N (M = 7 in our case) around (x, y). The regularization term is set to 0.01. This is followed by one convolutional layer with 32 5 × 5 filters and a fully connected layer with 100 nodes. We use relu activation in all neurons. Dropout layers are used to prevent overfitting. The implementation of the network can be found at https://github.com/kamballu/Deep-Visibility.

RESULTS AND ANALYSIS

In order to assess how well the estimated distortion visibility predicts ground-truth data from psychophysical experiments, we test the proposed model on the dataset of local masking thresholds in [START_REF] Alam | Local masking in natural images: A database and analysis[END_REF]. This dataset collects measured threshold values for 1080 image patches of size 85 × 85 pixels, extracted from the CSIQ dataset [START_REF] Larson | Most apparent distortion: a dual strategy for full-reference image quality assessment[END_REF]. The detection thresholds are reported in terms of root-mean-squared (RMS) contrast and expressed in decibels (dB).

To test the proposed model, we train it on three different datasets: the CSIQ dataset [START_REF] Larson | Most apparent distortion: a dual strategy for full-reference image quality assessment[END_REF], containing 855 images, 6 types of distortions; the TID 2013 dataset [START_REF] Ponomarenko | Image database TID2013: Peculiarities, results and perspectives[END_REF], with 3000 images and 25 types of distortion; and the LIVE dataset [START_REF] Sheikh | A Statistical Evaluation of Recent Full Reference Quality Assessment Algorithms[END_REF], featuring 779 images and 5 types of distortion. In general, the thresholds T (i, j) found with our model do not lie on the same scale as those in [START_REF] Alam | Local masking in natural images: A database and analysis[END_REF]. In addition, the P-net can be trained on different IQA datasets, and the interpretation of DMOS in each dataset depends on the experiment carried out to collect the data [START_REF] Zerman | The relation between MOS and pairwise comparisons and the importance of cross-content comparisons[END_REF]. Following a typical protocol in the evaluation of quality metrics [START_REF] Itu-T | Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models[END_REF], we compensate for this mismatch by linearizing the predictions with respect to psychophysical groundtruth through a monotonic third-order polynomial fitting before evaluating their statistical accuracy.

Performance

A comparison of the predicted and ground-truth local visibility thresholds is illustrated in the scatter plots of Figure 2, where our P-net has been trained on the LIVE and TID 2013 datasets, respectively. Each point in the scatter plot represents a 85 × 85 patch of [START_REF] Alam | Local masking in natural images: A database and analysis[END_REF]. Since our predictor can produce per pixel estimates of distortion visibility (using overlapping patches), we decimate the maps produced by the Pnet to match the resolution of the ground truth, using a simple averaging filter (see Figures 4 and5).

We can observe from Figure 2 that the predicted thresholds capture relatively well the overall trends of the measures obtained from psychophysical experiments, even if they have been obtained by training on very different data (IQA scores) and using different source contents. This indicates that learning visibility thresholds from generic IQA datasets is feasible and can generalize sufficiently well. Nevertheless, we notice in Figure 2 that in some cases the predicted thresholds deviate significantly from the measured ones. Especially for the LIVE dataset, this degrades the performance, measured by the Pearson linear correlation coefficient (PLCC) r = 0.56. To further investigate this, we analyze the distribution of the average luminance intensity of patches in the TID and LIVE datasets compared to CSIQ in Figure 3. Notice that there is only a negligible amount of patches for the LIVE dataset in the intensity range [0, 10] and [250,255]. This lack of data can affect the performance of prediction in this specific intensity range. We verify this by highlighting in red those patches of the test set having luminance outside the interval [START_REF] Alam | A computational model for predicting local distortion visibility via convolutional neural network trainedon natural scenes[END_REF]250] in Figure 2. We observe that these points correspond indeed to the outliers in the scatter plot. By removing these few very 6.521 Teo & Heeger [START_REF] Teo | Perceptual image distortion[END_REF] 6.861 Chandler et al. [START_REF] Chandler | A Patch-Based Structural Masking Model with an Application to Compression[END_REF] 6.879 Optimized GC [START_REF] Alam | A computational model for predicting local distortion visibility via convolutional neural network trainedon natural scenes[END_REF] 5.192 Alam et al. CNN [START_REF] Alam | A computational model for predicting local distortion visibility via convolutional neural network trainedon natural scenes[END_REF] 5.475

Proposed CSIQ quality scores [START_REF] Larson | Most apparent distortion: a dual strategy for full-reference image quality assessment[END_REF] 5.691 LIVE quality scores [START_REF] Sheikh | A Statistical Evaluation of Recent Full Reference Quality Assessment Algorithms[END_REF] 5.991 TID 2013 quality scores [START_REF] Ponomarenko | Image database TID2013: Peculiarities, results and perspectives[END_REF] 5.626

Table 1: Performance comparison between different algorithms. Highlighted are the best state-of-the-art methods (handcrafted and CNN-based), as well as our results.

dark or very bright patches (28 patches out of 1080), we observe that the PLCC increases to r = 0.68. We compare in Table 1 the performance of our method (trained with three different IQA datasets) with other predictors of local visibility thresholds proposed in the literature. The majority of these methods use handcrafted models directly derived from psychophysical experiment, while [START_REF] Alam | A computational model for predicting local distortion visibility via convolutional neural network trainedon natural scenes[END_REF] employs a convolutional neural network. The performance criterion is root-mean-squared error (RMSE) between the ground-truth and predicted thresholds. We observe that our approach provides comparable or better results to stateof-the-art methods. It should be noticed that the two methods in [START_REF] Alam | A computational model for predicting local distortion visibility via convolutional neural network trainedon natural scenes[END_REF] are trained on the same dataset [START_REF] Alam | Local masking in natural images: A database and analysis[END_REF] used for test, and thus their performance represents a sort of accuracy upper bound. Conversely, the proposed method achieve competitive results even if it is trained on the TID 2013 or LIVE datasets.

A qualitative illustration of the predicted thresholds is reported in Figure 4, where we also show per picture PLCC. We observe that the perceptual thresholds produced by our approach are intuitive, e.g., thresholds are lower for relatively smooth regions (sky) and higher for more complex regions (grass, leaves). There are also cases in which the prediction fails (examples in Figure 5). This happens in dark patches, as discussed above, where the training datasets do not offer sufficient samples to learn robustly the visibility thresholds.

CONCLUSION

We present a method to derive local visibility thresholds from image quality scores using a neural-network-based approach. We do not use the result of psycovisual experiments. Our experiments demonstrate that the latent information about distortion visibility carried by supra-threshold quality scores can be recovered and used to predict near-threshold local masking. One advantage of our approach, compared to models based on psychophysical data, is that it can leverage the larger availability of subjectively annotated image quality datasets. We plan to formalize further this approach in the future and apply it to objective image quality assessment.

Fig. 1 :

 1 Fig. 1: Neural network architecture for extracting contrast detection thresholds from IQA databases.

Fig. 2 :

 2 Fig. 2: Scatter plots of contrast detection thresholds derived using our method vs experimentally measured values. The system is trained on (a) LIVE dataset (b) TID 2013 dataset. The polynomial fitting line is show in red. Red points correspond to patches whose luminance is outside the range of the training datasets, see Figure 3. r is the PLCC (after fitting) on the whole test set; r is the PLCC excluding the red points.

Fig. 3 :

 3 Fig. 3: Distribution of average intensity of 32 × 32 patches in LIVE, CSIQ and TID datasets. The distributions of the three datasets overlap only in the intensity range [10, 250].

Fig. 4 :

 4 Fig. 4: Examples of local visibility thresholds produced by our method trained on TID 2013. The estimated thresholds are decimated to match the resolution of the ground truth. Both sets of values are scaled to fit in the range [0, 1] for visualization, with 0 (black) being the lowest threshold.

Fig. 5 :

 5 Fig. 5: Some failure cases of the proposed method trained on TID 2013. Often, these cases correspond to patches with low luminance, which are underrepresented in the training data.


Yuyang Liu
email: yuyangliu@std.uestc.edu.cn

Ce Zhu

Min Mao
email: minmao@std.uestc.edu.cn

Fangliang Song
email: songfangliang@std.uestc.edu.cn

Frederic Dufaux
email: frederic.dufaux@l2s.centralesupelec.fr

Xiang Zhang

Analytical Distortion Aware Video Coding for Computer Based Video Analysis

Keywords: Video coding, video analysis, rate-analyticaldistortion optimization, HEVC

With the development of artificial intelligence, more and more multimedia applications for various tasks have emerged in our daily life. Meanwhile, as one of the main information sources of the applications, a huge amount of video data has been being generated by portable or mounted cameras in daily basis for varying purposes including surveillance, in which case we may need computers to "watch" videos to save labor cost. However, most video coding standards are designed for the highest human perceptual quality given a bit rate by minimizing a fidelity cost function (e.g., mean squared error, MSE), assuming the content will be consumed by human beings. In view of the above considerations, this paper proposes a new rate-analytical-distortion optimization method (RADO) for video analysis. Specifically, we consider moving object detection as the analysis task. Accordingly, we develop a novel rate analytical distortion (RAD) model for video coding, where the analytical distortion is related to the object detection performance expressed in terms of F-measure. As shown in the experimental results, the performance of the video analysis task can be significantly improved (up to 40% reduction of analytical distortion) with a slight bit rate increase.

I. INTRODUCTION

Recent advances in computer vision and the increasing availability of portable or installed cameras have enabled new emerging multimedia applications. In many application scenarios, especially for surveillance, computers have to deal with various analysis tasks, such as object detection, tracking and recognition. Consequently, computer is the first or maybe the only "audience" in these scenarios. At the same time, the considerable amount of generated videos also need to be compressed due to cost-effective storage and bandwidth limitation.

Video coding has been studied for decades, mainly concentrating on issues related to high coding efficiency. Most of the widely deployed video coding standards, such as H.264/AVC and HEVC (High Efficiency Video Coding) are designed for human visual properties by maximizing a fidelity function (e.g. peak signal to noise ratio, PSNR) at a given bit rate, with human beings as the target audience. Besides, there are many developed rate-distortion optimization (RDO) schemes for HEVC standard [START_REF] Li | QP Refinement according to Lagrange multiplier for high efficiency video coding[END_REF]- [START_REF] Li | Lagrangian multiplier adaptation for rate-distortion optimization with inter-frame dependency[END_REF]. However, applying the traditional RDO scheme during video compression may be suboptimal when the video is intended for machine analysis. The critical issue is that compression distortions may bring a negative impact on the video analysis performance.

In this context, recently research work focuses on featurepreserving coding, where only feature descriptors are transmitted to the server [START_REF] Redondi | Compress-then-analysis vs. analysis-then-compress: Two paradigms for image analysis in visual sensor networks[END_REF]. As only feature descriptors are transmitted, the video cannot be watched at the server side. Consequently, it is inapplicable in some scenarios (e.g. video surveillance) where it is necessary to visualize the video content. In contrast, some research work focuses on feature-preserving video coding. Baroffio et al [START_REF] Baroffio | Coding visual features extracted from video sequences[END_REF] proposed a coding architecture designed for local features (e.g. SIFT, SURF) extracted from video sequences. The local feature descriptors were encoded by intra and inter-frame coding modes respectively. The final coding mode was determined by comparing the costs of intra-frame versus inter-frame coding. In their later work [START_REF] Baroffio | Coding binary local features extracted from video sequences[END_REF], the coding architecture was applied to code binary local features. In order to decrease the adverse effect of video compression on feature-matching performance, Chao and Steinbach [START_REF] Chao | Keypoint encoding for improved feature extraction from compressed video at low bitrates[END_REF] proposed a novel framework, in which keypoints extracted from a video were encoded and transmitted along with the compressed video to the server. However, this framework needs more bits since the keypoints are transmitted as side information.

Few studies have addressed video coding for video analysis tasks. Korshunov et al. [START_REF] Korshunov | Critical video quality for distributed automated video surveillance[END_REF] proposed a formal rateaccuracy optimization framework, where the encoding parameters in distributed video surveillance systems could be determined given a target bit rate or accuracy. Effrosyni and Pascal [START_REF] Kokiopoulou | Semantic coding by supervised dimensionality reduction[END_REF] proposed a supervised dimensionality reduction scheme which provides a tradeoff between compression and discriminant feature extraction. Liao et al. [START_REF] Liao | An analysisoriented ROI based coding approach on surveillance video data[END_REF] proposed an analysis-oriented ROI based coding approach to reduce the effect of video compression on the performance of video analysis. However, the approach in [START_REF] Liao | An analysisoriented ROI based coding approach on surveillance video data[END_REF][START_REF] Xiong | Face region based conversational video coding[END_REF] needs the prior knowledge of interest objects to detect ROI, which cannot always be obtained in practice.

In this paper, we develop a new rate-analytical-distortion optimization (RADO) method for video analysis, where the term "analytical distortion" represents the difference of the video analysis algorithms' performance when the video quality degrades. Meanwhile, we introduce an analytical distortion prediction (ADP) model which uses the compression distortion to estimate the analytical distortion. Specifically, we choose to focus on one fundamental video analysis task, moving object detection and accordingly develop a novel rate analytical distortion (RAD) model. It is worthy to note that our proposed coding method is fully standard compatible and the encoded bit-stream can be decoded by any HEVC decoder.

The remainder of this paper is organized as follows. Section II presents our proposed RADO method and the corresponding coding framework. Section III presents the ADP model and parameter acquisition. Section IV experimentally evaluates the proposed method. Section V concludes the paper.

II. PROPOSED CODING METHOD FOR VIDEO ANALYSIS

In this section, we introduce our proposed coding method for video analysis. Specifically, we build up an analytical inconsistency model to represent the difference of video analysis performance when the video quality degrades, partly inspired by the work in [START_REF] Korshunov | Critical video quality for distributed automated video surveillance[END_REF]. We use the term "analytical distortion" hereafter to denote the difference of video analysis performance.

A. Analytical Inconsistency Model

Inspired by the fact that the video analysis algorithm's performance may decline after a lossy video compression, we design an analytical inconsistency model to represent how the analysis algorithm behaves when the video quality degrades. Clearly, many evaluation criterions exist for different kinds of video analysis tasks. Even for a specific task, several evaluation metrics may be used to assess the performance. However, it is unmanageable to simply adopt all the evaluation metrics in the analytical inconsistency model, since the dimension of the model will be extremely high. In order to avoid this drawback, only one metric is chosen for each kind of video analysis task. Therefore, given a video analysis task T and the corresponding evaluation metric M, the analytical inconsistency model can be defined as ( , )

A D D T M ∆
, where D ∆ can be obtained as

o c D D D ∆ = - (1)
In ( 1), o D and c D denote the analysis results of the original video and the compressed video respectively. Video Analysis (e.g. Tracking)

Video Analysis (e.g. Tracking)

Fig. 1 Illustration for the analytical distortion acquisition procedure.

Fig. 1 illustrates how the analytical distortion can be obtained by [START_REF] Li | QP Refinement according to Lagrange multiplier for high efficiency video coding[END_REF]. In this paper, we do not aim at studying the performance of a video analysis algorithm itself. Rather, we are more specifically concerned with how the analysis algorithm behaves when the video quality is degraded.

B. Rate Analytical Distortion Optimization

Video compression is a well-studied problem and is commonly formulated and optimized by the trade-off between the entropy of the discretized representation (rate) and the error arising from the quantization (distortion). In this context, many RDO techniques [START_REF] Ortega | Rate-distortion methods for image and video compression: An overview[END_REF][START_REF] Sullivan | Rate-distortion optimization for video compression[END_REF] have been developed for modern video compression schemes. Typically, the objective of RDO is to minimize distortion D at a given rate R, under the available bit rate budget constraint T R , which can be formulated as min { } . .

T D s t R R ≤ . (2)
To solve this constrained optimization problem, the Lagrangian multiplier method [START_REF] Everett | Generalized Lagrange multiplier method for solving problems of optimum allocation of resources[END_REF] is widely employed by converting it to an unconstrained form as min { }, where

J J D R λ = + , ( 3
)
where J is the Lagrangian cost function, and λ is the Lagrangian multiplier. According to the work in [START_REF] Gish | Asymptotically efficient quantizing[END_REF], λ in (3) can be calculated by

2 dD c q dR λ = - = ⋅ , ( 4
)
where c is a constant and q denotes the quantization stepsize. Notably, distortion D in ( 2) is designed as a human perceptual related cost function P D in pixel domain, such as MSE, PSNR or Sum of Absolute Difference (SAD).

Meanwhile, in some typical application scenarios such as face detection and recognition, the compressed video will not only be consumed by computers running video analysis algorithms, but also watched by human beings for some application-related purposes. Therefore, it is necessary to reduce the negative effect introduced by compression while maintaining video coding efficiency. For this purpose, we formulate the RADO problem by jointly minimizing the compression distortion P D in pixel domain and the analytical distortion A D at a given rate. It can be written as min .

.

P A T D D s t R R α + ≤ , ( 5
)
where α is a weighting parameter. It is obvious that ( 5) is reduced to the traditional RDO formulation in (2) when α = 0. To solve the constrained RADO problem in ( 5), the Lagrangian multiplier method is also employed by converting it to an unconstrained form, which can be written as min { }, where

new new P A new J J D D R α λ = + + . (6)
It is worthy to note that the Lagrangian multiplier new λ in (6) is quite different that the λ defined in [START_REF] Li | Lagrangian multiplier adaptation for rate-distortion optimization with inter-frame dependency[END_REF]. In order to distinguish the two different Lagrangian multipliers new λ and λ , the Lagrangian λ in (4) will be denoted as HM λ in the remaining of this paper. Besides, equation ( 4) is applied in the HEVC test model to calculate the Lagrangian multiplier HM λ .

It is difficult to get the optimal solution in (6) without modeling the interaction between A D and R. Therefore, we empirically design a derivable RAD model, which can be expressed as

2 1 e A C D R C = , (7)
where 1 C and 2 C are constant parameters (the derivation of this model will be further discussed in section III). Besides, according to (4), R is also differentiable with respect to P D .

Consequently, the minimal cost new J is obtained when

0 new P A new J D D R R R α λ ∂ ∂ ∂ = + + = ∂ ∂ ∂ , (8)
P A new D D R R λ α ∂ ∂ = - - ∂ ∂ . ( 9
)
Combining (4), equation ( 9) can be rewritten as

A new HM D R λ λ α ∂ = - ∂ . ( 10
)
In conclusion, the new Lagrangian multiplier new λ can be calculated by [START_REF] Kokiopoulou | Semantic coding by supervised dimensionality reduction[END_REF] for a given weighting parameter α .

C. Proposed Coding Framework

The analytical distortion A D is needed during the coding process in our proposed RADO approach. In general, the accurate A D can be obtained by two approaches: (i) integrating the analysis algorithm into video codec and (ii) running the analysis algorithm twice on the original video and the compressed version to compare the difference between the analysis results, as shown in Fig. 1. However, both approaches are very time consuming. To overcome this difficulty, we introduce an ADP model. Namely, the proposed coding framework consists of two steps: a training procedure and a coding procedure.

 Training procedure: The purpose of the training procedure is to obtain the parameters of the RAD model and the ADP model. During the training procedure, a set of reference video sequences are selected and compressed by HEVC codec with different quantization parameter (QP) values. Next, we run the analysis algorithm on the reference video and the compressed versions. In this way, both the compression distortion P D (i.e. SAD) and the analytical distortion A D can be calculated. Finally, from the statistics, the ADP model can be constructed and the parameters of the RAD model can be estimated.

 Coding procedure: In the coding procedure, each frame is encoded twice. Firstly, the frame is compressed using the traditional RDO method and then SAD can be calculated by comparison between the original and reconstructed frames. Secondly, given a video analysis task and the corresponding evaluation distortion A D can be predicted from SAD using the designed ADP model. Furthermore, according to the RAD model in [START_REF] Baroffio | Coding binary local features extracted from video sequences[END_REF], we can express the derivation of A D with respect to R. Then the new Lagrangian multiplier new λ can be calculated by [START_REF] Kokiopoulou | Semantic coding by supervised dimensionality reduction[END_REF] for a given α . Finally, the same frame is compressed a second time using our proposed RADO method.

Note that, each frame needs to be compressed twice in our proposed framework, which will definitely increase the complexity of the encoder. However, if necessary, approached designed for estimating SAD before encoding [START_REF] Li | Lagrangian multiplier adaptation for rate-distortion optimization with inter-frame dependency[END_REF] can be used to avoid the two pass encoding process.

III. VIDEO CODING FOR MOVING OBJECT DETECTION

To show the effectiveness of the proposed coding scheme, we consider a task of moving object detection. Indeed, moving object detection is a fundamental component in many application scenarios. For this purpose, we select the moving object detection algorithm in [START_REF] Lei | Robust object detection with interleaved categorization and segmentation[END_REF], and use Fmeasure [START_REF] Dhome | A benchmark for background subtraction algorithms in monocular vision: a comparative study[END_REF] as the evaluation metric.

A. Analytical Distortion Prediction

In order to build the ADP model, we run some experiments on four reference video sequences (Video 1-4), including three indoor videos (Video 1-3) from PETS2006 [START_REF] Thirde | Overview of the PETS2006 challenge[END_REF] and one outdoor video (Video 4) from CAVIAR [20]. The four reference video sequences are compressed using the HEVC reference software (HM 16.7) [21] under Low-Delay P (LDP) configuration with QPs from 4 to 41. Thus, for each of the 4 original sequences, we get 38 compressed versions with different quality levels. The analytical distortion of one frame can be obtained by comparing the analysis algorithm performance between the reference video and its compressed version. Using F-measure, it can be expressed as

1 i A i D F = -, (11)
where i A D represents the analytical distortion of the i-the frame in video sequence. Finally, the average analytical distortion S A D over the video sequence S can be denoted as

1 1 N S i A A i D D N = = ∑ , ( 12
)
where N is the total number of frames. In the ADP model and the RAD model, the analytical distortion is then represented as S A D .

) 14
where 1 P and 2 P are constant parameters that are obtained by linear regression in the training procedure.

B. Rate Analytical Distortion Model

D

Note that we choose bpp (bit per pixel) to denote the bit rate to reduce the influence of different frame rates and resolutions. From Fig. 3, it can be seen that the bit rate decreases exponentially with the increase of S A D . This observation gives a proof of the validity of RAD model expressed in [START_REF] Baroffio | Coding binary local features extracted from video sequences[END_REF]. Additionally, the parameters of RAD model are obtained by a regression process according to the statistical results in the training procedure.

C. Weighting Parameter α Acquisition

In order to explore the impact of the weighting parameter α on the proposed framework, a set of α values ranging from 0.1 to 0.9 are tested. Fig. 4(a) shows the experimental results when QP = 27. It can be seen that the average analytical distortion S A D tends to increase smoothly along with α . Therefore we set α to be 0.1 as an initial value in this paper. Meanwhile, Fig. 4(b) implies that the object area of one frame may have a strong impact on the frame level analytical distortion denoted as i A D , since smaller object has less contribution to SAD of one frame. Fig. 5 illustrates the detection results. From Fig. 5, it can be seen that the total detected object area in Video 3 is smaller than that in Video 1. Inspired by the above observations, we calculate the variance of i A D under different QPs and collect the total detected object area of each original frame. The results are shown in Fig. 6. In Fig. 6, the analytical distortion variance is much bigger when the object area is smaller, which implies that smaller object area should be compressed with higher quality to avoid a high fluctuation of analytical distortion. As a consequence, the weighting parameter α is updated according to the detected object area of each frame, which can be expressed as 0.1 exp( )

Area W H α * = * , ( 15
)
where W and H denote the frame width and frame height and Area represents the total detected object area of each frame.

In the coding procedure, the total object area can be derived by a frame subtraction method [START_REF] Singla | Motion detection based on frame difference method[END_REF].

IV. EXPERIMENTAL RESULTS

In this section, simulations are conducted to evaluate the performance of the proposed method in the HEVC reference software HM 16.7. Low-Delay P (LDP) configuration is tested. Four QP values [START_REF] Singla | Motion detection based on frame difference method[END_REF]27,32,37) are used and BD-rate saving (BDBR) is employed to measure the RD performance of the proposed method. The RAD curves of our proposed approach and HEVC on each video sequence are shown in Fig. 7. In order to show the performance of our proposed method, another 4 test video sequences Video 5-8 are chosen, including 3 indoor videos from PETS2006, and 1 outdoor video from CAVIAR. Detailed properties of the four sequences are given in Table 1. As a first observation in Fig. 7, it can be seen that our proposed approach can reduce the analytical distortion effectively under low bit rate (QP = 37) while the coding efficiency decreases slightly in terms of BD-rate (BDBR). In the best scenario, up to 40% reduction in terms of average analytical distortion is achieved for QP 27 and 22 in Fig. 7(c), since the object area in Video 7 is smaller than that in other video sequences. In contrast, the analytical distortion increases at low bit rate (QP = 32, 37) in Fig. 7(c), since the background has a great impact on SAD which results in a lot of false positives. Table . 2 shows the performance in terms of BDBR between our proposed approach and HEVC. We can see that the coding performance of our approach increases about 0.9% in average. The reason is that our proposed RADO method may change the mode decision for coding units and the best chosen mode is no longer optimal in terms of RD. The complexity comparison between HM 16.7 and the proposed method are shown in Table 3 in terms of the encoding time. It can be seen that the encoding time of the proposed method is about 1.8 times compared with that of HEVC. V. CONCLUSION

In this paper, we attempt to design a new video coding mechanism for video analysis and propose a new ratedistortion method. To illustrate the validation of our proposed method, we consider moving object detection as an example and propose a rate-analytical-distortion model for video coding. More specifically, the compression distortion is used to predict the analytical distortion. The experimental results show that our proposed model can effectively reduce the analytical distortion. However, our proposed method is a two-pass encoding method which increases the complexity. In our future work, we will aim at reducing the complexity by applying some SAD estimation approaches to avoid the two-pass encoding process. Besides, investigating a more adaption model to derive the weighting parameter α between the compression distortion and the analytical distortion is also an important aspect which will be addressed in the future.

  Fig. 2 Statistical results of S A D as a function of

4 Fig. 3

 43 Fig. 3 Statistical results of the bit rate (bpp) as a function of S A

Fig. 4

 4 Fig. 4 Impact of α on the proposed framework. (a) In order to show the results, α is enlarged 10 times in the figure and 4 sequences are compressed using same QP (27). (b) Compressed with the same α (0.1), the analytical distortion of 599-th frame i A D in Video 1 and Video 3 are shown under different QPs.

Fig. 5

 5 Fig. 5 The first column shows the 599-th original frames in Video 1 and Video 3 respectively. The second column represents the detection results of the original frames and the third column denotes the detection results of their compressed versions (QP = 37).

Fig. 6

 6 Fig. 6 Analytical distortion variance vs. object area

TABLE 1 .

 1 PROPERTIES OF THE FOUR VIDEO SEQUENCES

	Sequence Resolution	Frame Number	Frame Rate
	Video 5	720x576	600	25
	Video 6	720x576	600	25
	Video 7	720x576	600	25
	Video 8	800x600	200	30

TABLE 2 .

 2 LUMA BDBR UNDER LDP CONFIGURATION

	Video Sequence	BDBR
	Video 5	0.5%
	Video 6	0.6%
	Video 7	1.1%
	Video 8	2.4%
	Average	0.9%

TABLE 3 .

 3 ENCODING TIME RATIO

	Sequence	Encoding Time (sec.) HM Proposed	Encoding Time Ratio
	Video 5	7919.043	14286.955	180.413%
	Video 6	7898.877	13849.762	175.338%
	Video 7	8141.746	14894.478	182.940%
	Video 8	1617.044	2865.203	177.188%

ACKNOWLEDGMENT

This work was suppoted in part by the National Natural Science Foundation of China under Grant 61571102 and in part by the Other Funds under Grant 2015AA015903 and Grant ZYGX2014Z003.


Mohsen Abdoli
email: mohsen.abdoli@orange.com

Félix Henry
email: felix.henry@orange.com

Patrice Brault
email: patrice.brault@l2s.centralesupelec.fr

Pierre Duhamel
email: pierre.duhamel@l2s.centralesupelec.fr

Frédéric Dufaux
email: frederic.dufaux@l2s.centralesupelec.fr

Intra Prediction Using In-Loop Residual Coding for the post-HEVC Standard

A few years after standardization of the High Efficiency Video Coding (HEVC), now the Joint Video Exploration Team (JVET) group is exploring post-HEVC video compression technologies. In the intra prediction domain, this effort has resulted in an algorithm with 67 internal modes, new filters and tools which significantly improve HEVC. However, the improved algorithm still suffers from the long distance prediction inaccuracy problem. In this paper, we propose an In-Loop Residual coding Intra Prediction (ILR-IP) algorithm which utilizes innerblock reconstructed pixels as references to reduce the distance from predicted pixels. This is done by using the ILR signal for partially reconstructing each pixel, right after its prediction and before its block-level out-loop residual calculation. The ILR signal is decided in the rate-distortion sense, by a brute-force search on a QP-dependent finite codebook that is known to the decoder. Experiments show that the proposed ILR-IP algorithm improves the existing method in the Joint Exploration Model (JEM) up to 0.45% in terms of bit rate saving, without complexity overhead at the decoder side.

I. INTRODUCTION

After two successful collaboration of ITU-T Video Coding Expert Group (VCEG) and ISO/IEC Motion Picture Expert Group (MPEG) in the standardization of the Advanced Video Coding (AVC) and the High Efficiency Video Coding (HEVC), a new collaboration team was formed in 2015 to investigate the potential need for the next generation video coding [START_REF] Chen | Coding tools investigation for next generation video coding[END_REF]. The agenda of this team that is called the Joint Video Exploration Team (JVET) is to study new video compression technologies for a future video coding standard. This effort has resulted in a state-of-the-art video codec called Joint Exploration Model (JEM). As of May 2017, JEM5.1 provides about 30% bitrate saving compared to HEVC. Some of the most significant changes compared to HEVC are as follows:

• The Quad Tree Binary Tree (QTBT) block partitioning instead of the Quad Tree, • Block size up to 256x256 instead of 64x64,

• 67 intra prediction modes (IPM) instead of 35, • Multiple transforms, • Adaptive motion vector resolution. In the intra prediction, the new set of IPMs includes the traditional DC and planar plus 65 angular modes covering the same angle range as HEVC with double precision. Although the new intra prediction has a better performance than HEVC, it is still incapable to accurately predict when reference pixels from previous blocks are not well-correlated to the distant pixels inside the current block. This problem occurs when the content changes inside a block, which results in high energy concentration in the bottom-right corner of the residual block.

The long distance prediction inaccuracy problem in lossy intra prediction has been sparsely studied in the literature. The Short Distance Intra Prediction (SDIP) [START_REF] Cao | Short distance intra coding scheme for HEVC[END_REF] was first proposed to Joint Collaborating Team on Video Coding (JCT-VC) in 2010 [START_REF] Lai | New intra prediction using the correlation between pixels and lines[END_REF]. In SDIP, the prediction remains at the Prediction Unit (PU) level by forcing a split into thin rectangular horizontal/vertical PUs, expecting that the small height/width of the PU solves the long distance prediction inaccuracy problem. In Combined Intra Prediction (CIP) [START_REF] André | Improved combined intra prediction for higher video compression efficiency[END_REF], the algorithm performs an additional inner-block pixel-level prediction and combines it with the traditional prediction using a weighted average. The fact that the pixel-level prediction of CIP uses an estimation of reconstructed inner-block references limits its performance, due to propagation of the estimation error. In [START_REF] Gao | Differential pulse code modulation intra prediction for high efficiency video coding[END_REF], the algorithm uses a DPCM-based pixel-level prediction with an out-loop residual signal. Like CIP, this method suffers from the estimation error propagation through the block.

In this paper, the proposed In-Loop Residual coding Intra Prediction (ILR-IP) tries to solve the long distance prediction inaccuracy problem by performing a pixel-level prediction using inner-block references. These inner-block references are reconstructed by the In-Loop Residual (ILR) signal. The ILR signal that has the same size as the block, is decided by a search loop in the rate-distortion sense and is transmitted besides the regular out-loop residual. Unlike the regular residual, the ILR transmission is done by vector quantization using a codebook that is known at the decoder side. As will be shown, using the ILR signal improves the prediction performance in case of in-block content change.

The rest of this paper is organized as follows. In section II, the philosophy and general structure of ILR-IP is discussed. In section III, the key elements of ILR-IP and different design choices are explained in detail. In section IV, the performance of the proposed algorithm is discussed. Finally, section V concludes the paper.

II. PRINCIPLES OF IN-LOOP RESIDUAL INTRA PREDICTION (ILR-IP)

The idea of inner-block pixel prediction for solving the long distance prediction inaccuracy problem can be better perceived 978-1-5090-3649-3/17/$31.00 ©2017 IEEE at the decoder side, where reference pixels and residual signal are available. Figure 1 shows an example 8 × 8 block with its regular references, at the decoder side. According to the conventional intra prediction, a vertical angular IPM is most likely to be selected for this block, which copies reference pixels from the top reference line to all positions inside the block. As can be seen, the content in the bottom-right corner of the block is completely uncorrelated to its corresponding reference pixels. Normally, this would probably result in further splitting the block, thus requiring more data to be sent. However, it seems that the decoder might have the chance to use correlated inner-block local references in that corner without having to split the block.

For instance, at the time of pixel prediction at X, all surrounding pixels [A, B, C] are already predicted by the regular intra prediction. Besides, the residual signal is available at the stage of decompression process, which makes [A, B, C] fully reconstructible. Therefore, it seems straightforward to utilize these inner-block local references for the prediction of X at the decoder side. However the main obstacle is that, basically, the prediction decisions are made by an encoder which is unable to perform the same short distant prediction. The reason is that, unlike the decoder, the residual signal is not available at the stage of prediction at the encoder side. Therefore, it is impossible at the encoder side to reconstruct [A, B, C]. In other words, if the encoder aims at using [A, B, C] during the prediction step, it needs their residual values, which are calculated after the prediction step of the entire block is finished. Therefore there is a chicken-and-egg data dependency at the encoder side due to the block-based nature of the codec.

The ILR-IP algorithm removes the above chicken-and-egg data dependency, by providing the ILR signal, during the prediction at the encoder side. This signal is progressively added to predicted pixels to correct them and make them usable as inner-block references for the prediction of the following pixels within the current block. Since the main purpose of the ILR signal is the "correction" of predicted pixels, then the performance of the proposed algorithm directly depends on the pixel correction accuracy of the ILR signal. Therefore, the choice of the ILR signal is crucial at the encoder side.

In order to achieve the highest pixel correction accuracy, one might simply calculate a residual signal in the spatial domain that perfectly corrects pixels as if it were used as the ILR signal. This signal that is called the "minimum residual (MinRes)" in the remaining of this paper, can be easily calculated by accessing the original pixels at the encoder. In order to perform the same perfect pixel correction at the decoder, the MinRes signal needs to be transmitted in a lossless manner. However, this is not feasible due to its high rate. Therefore, a practical ILR signal must be decided in the rate-distortion sense. In the proposed algorithm, a vector quantization approach is used to train a codebook based on the MinRes values. This codebook is then fully explored in an encoder loop to select the ILR candidate with minimum ratedistortion cost (RDCost). The corresponding codebook index is then transmitted to the decoder.

III. PROPOSED ILR-IP ALGORITHM

The existing intra prediction with 67 internal IPMs works very well for content with directional texture or plain structure, but has problems with more complex types of content. Therefore, the proposed ILR-IP is designed to coexist alongside the regular prediction, in order to improve the performance for problematic types of content. More specifically, in the new design, the intra prediction of each block has the choice between the regular intra prediction and the proposed ILR-IP algorithm. This choice is made in the rate-distortion sense and imposes an overhead of one flag per block to signal the selected algorithm. Figure 2 schematically shows the choice for the intra prediction algorithm at the block level.

In the ILR branch of Figure 2, first the 'ILR-Loop' box decides about the 'ILR' signal. This decision can be made by any algorithm at the encoder side and needs to be transmitted to the decoder side. The ILR signal is then used as the input to the ILR-IP algorithm to produce the prediction signal ('Pred'). In the ILR-IP, the pixel prediction process starts from top-left corner of the block and progresses toward the bottom-right corner in a lexicographical scanning order. At each position of the scan, a short distance pixel prediction function takes three neighboring pixels (i.e. top, left, top-left) as input and predict the pixel at that position. The immediate step after the prediction is the pixel correction by adding the ILR value to the predicted value at that position. This step that will be expanded in the subsection III-B, prepares the pixel that was just predicted, for future use as an inner-block reference.

Although the ILR signal is responsible for correcting the pixels after prediction, it is not yet able to completely replace the regular out-loop residual. Hence, the regular out-loop residual signal 'Res' is also calculated by comparing the 'Pred' signal to the original pixels values 'ORG'.

As soon as all three signals of 'ILR', 'Pred' and 'Res' are available, the RDCost of the block, after compression by ILR-IP, can be calculated. The rate and distortion calculations are performed in the 'CABAC' and the 'PSNR', respectively. For the rate calculation, the bits for transmission of the all syntax elements are counted. This includes the syntax elements of the ILR signal, the algorithm flag (to indicate use of the ILR-IP) and the regular out-loop residual. Since the ILR-IP does not The current platform of ILR-IP is flexible in the sense that different algorithms can be implemented in the following component independently:

• ILR signal decision: what values should be given to the algorithm as the ILR signal? • ILR signal transmission: how should the ILR signal be transmitted to the decoder? • Pixel prediction function: given the ILR signal, how pixels should be predicted using inner-block references? The existing intra prediction has been through a comprehensive optimization process which includes the adaptation of other codec tools that have an impact on/from the intra prediction. Hence, the performance of any new algorithm with major changes cannot be fairly evaluated, unless either all the tools are re-optimized, or the comparison is performed in an environment which is free from the impacts of other tools. In the rest of this section, first a fair simplified 1D experiment for validating the idea of ILR-IP is explained. Then a 2D implementation in the latest version of the JEM is provided.

A. 1D experiment

The main goal in this 1D experiment is to keep all the conditions as fair as possible for both regular and proposed intra prediction algorithms. The common specifications of this simplified 1D testbed include:

• The picture partitioning of one block size of 1×4,

• The pixel prediction that takes two reference pixels and performs linear extrapolation (planar), • Entropy coding of the regular quantized coefficients of the [out-loop] residual, • The transform blocks by the 1D-DCT on 1×4 blocks, Fig. 3. The regular and the ILR intra prediction with planar pixel prediction function in the 1D testbed

• One flag per block that is transmitted to signal the selected intra prediction algorithm. Figure 3 shows a line of image in the above 1D testbed, with a content change that coincides with the second 1×4 block (i.e. current block). Here, the first two values in the diagram (i.e. the bold lines) are the regular reference pixels from the previous block. As can be seen, these references are not able to properly predict the last two pixels of the current block. However, the ILR-IP algorithm (which is given a proper ILR signal) is able to adapt to the change and predict the pixels by using corrected inner-block references.

According to the above specifications, the regular intra prediction algorithm is straightforward. However for the ILR-IP, two design choices have to be made:

1) ILR signal decision by brute force search in the transform domain: As explained, the MinRes in the spatial domain can be defined for each block, as a signal that provides zero distortion if used as the ILR signal. This signal can simply be calculated for each pixel, by 1) using original pixels as the inner-block prediction references, 2) predicting the pixel, and then 3) subtracting the predicted pixel from the original pixel.

Despite the fact that the MinRes signal is too costly to be transmitted in a lossless manner, the ILR-IP uses its quantized coefficients as the starting point for a brute-force search to obtain the 1D ILR signal. Given the 1×4 block of quantized coefficients of the MinRes signal in the transformed domain as

T M = [t (1) M , t (2)
M , t (3)
M , t (4)
M ], the ILR-IP finds the optimal ILR signal by exploring all candidates

T C = [t (1) C , t (2)
C , t (3)
C , t (4)
C ] where:

t (i) C = t (i) M + d (i) , i=1,2,3,4. (1)
The displacement d (i) of each coefficient in Eq. 1 is limited to integer values in an arbitrary interval [-b, b]. In the experiments presented in this section, b = 10 is selected.

2) ILR signal transmission by entropy coding of the coefficient: Since the ILR signal is determined in the quantized transform domain, it is decided that the encoding method is the same as that of the regular out-loop residual. This includes using entropy coding for encoding the transform coefficients.

Experiments show that the compression performance of the ILR-IP within the 1D testbed is better than the regular intra prediction by 6.2% on average. These experiments are performed on the sequences of the classes B, C and D. One should note that the simplifications of the 1D testbed have highly impacted both performances. Hence, this compression gain is simply not achievable in the actual 2D JEM framework. However, it shows the potential of the proposed algorithm.

B. 2D implementation in the state-of-the-art JEM video codec

The ILR-IP algorithm in 2D is implemented inside the reference software JEM5.0.1. Here we introduce three main design choices made for the 2D ILR-IP model:

1) Pixel prediction by LOCO-I function : The 2D pixel prediction function takes three neighbor pixels as references and implements the LOCO-I function [START_REF] Weinberger | The LOCO-I lossless image compression algorithm: Principles and standardization into JPEG-LS[END_REF]. This function, which is mostly used for lossless image compression (e.g. JPEG-LS), checks the existence of a vertical or horizontal edge. Eq. 2 formulates the LOCO-I in Figure . 4 at prediction position (p), given the three references at positions (a),(b),(c).

x(p) = f (x (a) , x(b) , x(c) ) =      min(x (a) , x(b) ), if x(c) ≤ max(x (a) , x(b) ). max(x (a) , x(b) ), if x(c) ≥ min(x (a) , x(b) ). x(a) + x(b) -x(c) , Otherwise. (2)
Generally, x(s) at scanning position (s) can either be a "corrected" pixel inside current block or a "decoded" pixel from previous blocks. In the case that x(s) belongs to current block (i.e. inner-block reference), the correction is done by:

x(s) = x(s) + y (s) , (3)
where y (s) is the ILR value at (s) and will be discussed later.

2) ILR signal decision by vector quantization: Unlike the 1D experiment, the brute-force search approach is not practical in 2D, due to the very large search space. Therefore, the ILR signal in 2D is determined by a vector quantization approach with a finite QP-dependent codebook [START_REF] Huang | Mode dependent vector quantization with a rate-distortion optimized codebook for residue coding in video compression[END_REF].

Ideally, the codebook should provide proper ILR signals for all block, close enough to their MinRes. Therefore, we decide to adopt the MinRes idea to train the codebook by a modified version of the iterative Linde-Buzo-Gray (LBG) algorithm [START_REF] Linde | An algorithm for vector quantizer design[END_REF]. To do so, a training sequence consisting of many H×W video blocks is prepared. These blocks contain actual content to be compressed by ILR-IP and are taken from a large and diverse set of video sequences with different resolutions and contexts (i.e. natural, synthetic etc.). In addition, an initial codebook is randomly set. This choice is not critical as the goal is to gradually converge to a proper codebook by iterating with the LBG. Each iteration of the LBG-based codebook construction consists of two steps, namely "classification" and "update": Classification: Given a codebook at iteration t, represented by N centroids C t = {Y t i ; i = 1, 2, ..., N }, as well as a large set of training sequences S, the classification step decides about the centroid of each sample X ∈ S and categorizes it in one of the N classes:

R t i = {X : d(X, Y t i ) < d(X, Y t j ); all j ∈ [1, N ] and j = i}, (4
) where d(X, Y) is an arbitrary function for calculating the distance between the sample X and the centroid candidate Y.

Here, categorizing block X into class i means selecting the centroid Y i as the ILR signal of X.

The standard LBG is normally used for representing a large set of points by a limited set of centroids and the goal is to minimize the approximation error from the centroids. Hence, a ρ-norm distance measure (i.e. Euclidean distance) is used as the distance function d. However, in the context of video compression, we aim at minimizing the RDCost as d, rather than a ρ-norm measure.

Update: After classification of all samples, each centroid of the codebook is updated by a centroid function that basically minimizes a global distance measure based on d:

C t+1 = {cent(R t i ); i = 1, ..., N}. (5)
In the standard LBG, the arithmetic mean as the centroid function, along with the Euclidean distance as d, guarantee a minimization of the global distance measure. Moreover, enough iterations on Eq. 4 and Eq. 5 guarantee the convergence of the codebook [START_REF] Chamie | Design and analysis of distributed averaging with quantized communication[END_REF]. However, the codebook construction problem in the ILR-IP is slightly more complicated, as all decisions are made through a non-linear prediction process. More precisely, the spatially progressive prediction scheme of ILR-IP makes it theoretically possible to decrease prediction error, while increasing Euclidean distance from the MinRes. As a consequence, the arithmetic mean as the centroid function in Eq. 5 no longer guarantees the global distance minimization.

Due to the above limitation, a slight, yet important modification is made on the centroid function of the LBG algorithm. In the proposed method, each centroid is updated pixel-bypixel. This is comparable to the block-level updating if the standard LBG was adopted. The difference is that in the pixellevel updating, when one value of a centroid is updated, it will immediately be used for updating the next values of that centroid. However, in the block-level updating, all values at all scanning positions are updated at once by using the old centroid from the previous iteration. In the rest of this section, all discussions for the centroid updating are only focused on the i-th class of the codebook (i = 1, 2, ..., N ).

Let R t i = {X l ; l = 1, ..., L} be the set of samples that were classified in the i-th class at the t-th iteration by Eq. 4. Each sample in this set is an H×W block. Moreover, a scanning order specifies the order of predicting pixels as well as updating centroid values. Therefore, given the scanning order and the training samples of the i-th class as R t i , the l-th sample can be vectorized following the scanning order as:

X l = [x (1) l , ..., x (H×W) l ]. (6)
The corresponding centroid for the above samples at iteration t, can accordingly be vectorized in the same order:

Y t i = [y t, (1) i

, ..., y t,(H×W) i

].

Moreover, the prediction pixels may be calculated by applying Eq. 2 given Y t i :

Xt l = [x t, (1) l , ..., xt,(H×W) l
].

Note that the superscript t on each predicted pixel x indicates that its corresponding inner-block references at its top, left and top-left, were corrected by the centroid at iteration t. By referring to Figure 4, Eq. 2 and Eq. 3, this can be written: ).

The adopted centroid function performs a pixel-level update with respect to the scanning order: 1) it starts from the first scanning position of the centroid at iteration t; 2) it updates the centroid value at that position; 3) it replaces the updated value in the ongoing centroid of iteration t. After these steps are done for all positions of the scanning order, the entire centroid is updated and ready for the next iteration t + 1. Eq. 10 formulates the pixel-level update at position (p) that requires performing the pixel prediction for all L samples of the class at that position:

y t+1,(p) i = 1 L L l=1 (o (p) - xt+1,(p) l ), (10)
where o (p) is the original pixel value at the scanning position (p). As shown, xt+1,(p) l belongs to the iteration t + 1 instead of t. According to Eq. 9, this indicates that its inner-block references were corrected by the centroid of iteration t+1 that was just updated in the ongoing update phase. This is possible since the scanning positions at top, left and top-left of the (p) are precedent to (p), thus, their corresponding centroid values are already updated in the current update phase.

After Eq. 10 is performed on all scanning positions (p), the i-th centroid for the iteration t + 1 is given as:

Y t+1 i = [y t+1,(1) i , ..., y t+1,(H×W) i ]. (11)
The above steps explain the centroid update step in one iteration of the codebook construction process for one centroid of the codebook. The goal is to repeat both the classification and update steps until convergence.

Figure 5 visually summarizes the above steps for a 2×2 centroid Y, with L samples in R and the given scanning order. As can be seen, the update process starts with the Y t in the first row and first column. At the end of each row, the corresponding centroid value at that position is updated and replaced in the Y t . This process repeats until the last position is updated and the whole centroid is updated to Y t+1 . 3) ILR signal transmission by fixed-length coding: Unlike the 1D testbed where entropy coding was used, here the fixedlength coding of codebook index is used for IRL transmission. This is due to the probability distribution function of the centroids indexes that roughly follows uniform probability. Therefore, the ILR transmission simply includes equiprobable encoding of n = log(N ) bits indicating one of 2 n centroids in the codebook of size N.

IV. COMPRESSION PERFORMANCE OF THE 2D ILR-IP

In this section, the performance of the ILR-IP algorithm is compared to the state-of-the-art reference software of JEM5.0.1 in All-Intra coding mode and conforming the Common Test Conditions (CTC). Without loss of generality, here the algorithm is limited to 4×4 blocks. Experiments are run with codebooks of size 16, 32, 64, 128 and 256. These codebooks are trained by the modified LBG algorithm introduced in section III-B2 and on an independent training set.

To reach the maximum performance of JEM, all the new tools except non-separable secondary transforms (NSST) [START_REF] Zhao | Nsst: Non-separable secondary transforms for next generation video coding[END_REF] and explicit multiple core transforms (EMT) [START_REF] Zhao | Enhanced multiple transform for video coding[END_REF] are activated. In fact these two tools are perfectly optimized for encoding the regular out-loop residual signal, while the shape of the out-loop residual of the ILR-IP blocks is essentially different due to the existence of the ILR. Consequently, the existing transform choices do not necessarily improve out-loop residual coding of the ILR-IP as efficiently as they improve the regular out-loop residual coding. Hence, these two transform-related tools are switched off in both algorithms for a fair comparison.

Table I shows the Bjøntegaard delta (BD) rate of the ILR-IP against JEM5.0.1, where it consistently outperforms the JEM. The peak performance of the ILR-IP is for the artificial videos in the Class F. This can be justified by the fact that in artificial videos, in-block content changes usually appear on sharp edges that separate two pure plain areas. Therefore, changes can be compensated more easily by ILR signals that have non-zero values only on the edges. Table II shows the run-time complexity comparison between the ILR-IP and the JEM5.0.1. As shown, the proposed ILR-IP algorithm brings compression gain with no complexity overhead at the decoder side, which is crucial for adoption of a new technology. From Table I, it can be seen that the performance of the ILR-IP linearly improves as the size of the codebook increases. Therefore, one might aim at further increasing the codebook size to achieve better performance. However, the main obstacle in the current design is the encoder complexity of codebook exploration. Currently, the encoder has to calculate RDCost for all centroids in the codebook to pick the best one.

It is important to note that the current codebook exploration can be accelerated without major impact on the performance. For instance, one might explore only a small subset of the codebook to pick the suboptimal ILR signal for each block. It was observed during implementations that the second-best and third-best ILR candidates in the codebooks are not much worse than the optimal ILR. This is left as future work.

V. CONCLUSION

In this paper, the long distance prediction inaccuracy problem was targeted. The main philosophy was that in case of in-block content change, the references from previous blocks are uncorrelated and therefore, nearer in-block pixels should be used as reference. The proposed ILR-IP introduces an additional signal to progressively correct the inner-block pixels during the prediction. The corrected pixels are then used as reference instead of the references from previous blocks. The validity of the ILR-IP algorithm was first tested in a 1D testbed where it outperformed the regular prediction. In the 2D, the proposed algorithm was implemented in the software JEM5.01. The encoder was given the possibility pick the best algorithm according to the RDCost. This imposed an overhead of one extra flag per block plus the encoder side complexity. Experiments show that the ILR-IP is able to compensate the flag overhead and bring up to 0.45% BD-rate saving, without complexity overhead at the decoder side.

The current version of the ILR-IP is limited to the 4×4 blocks and codebook size up to 256. In future works, we will consider extending it to larger blocks and codebooks. This will give ILR-IP more opportunity to improve the RDCost for high QPs where 4×4 blocks are less likely to be selected.

Fig. 1 .

 1 Fig. 1. Long distance inaccuracy problem: regular references at top row and left column are not correlated to bottom-right area, due to the content change.

Fig. 2 .

 2 Fig. 2. The block level decision based on the rate-distortion cost (RDCost) between the regular intra prediction and the ILR-IP.

Fig. 4 .

 4 Fig. 4. Three references (a),(b),(c) around the pixel (p) to be predicted.

Fig. 5 .

 5 Fig. 5. Four stages of pixel-level centroid update for a 2×2 block according to the given scanning order. At each stage, the gray pixel indicates the scanning position of the centroid that is being updated. The regular references of the 2×2 blocks are shown by dotted squares.

TABLE II RUN

 II -TIME COMPLEXITY OF THE ILR-IP ALGORITHM COMPARED TO THE JEM5.0.1 AT THE ENCODER AND THE DECODER SIDE

			Codebook size
		16	32	64	128 256
	ET (%)	117 126 137	165 216
	DT (%) 100 100 100	100 100


G Radtke

D Taverna

N Menguy

S Pandolfi

A Courac

Y Le Godec

O L Krivanek

T C Lovejoy

Polarization selectivity in vibrational electron-energy-loss spectroscopy

Keywords: numbers: 79.20.Uv, 68.37.Ma, 78.20.Bh

come

Ultra-high resolution electron-energy-loss spectroscopy (EELS) in the (scanning) transmission electron microscope ((S)TEM) has recently attracted a considerable attention as an emerging technique to investigate the vibrational properties of condensed matter with an unprecedented spatial resolution [1][2][3]. When implemented in a non-penetrating geometry, i.e. using an electron beam positioned at a finite distance away from the edge of the sample, energy loss processes are dominated by dipole scattering arising from long-range electric fields generated through lattice or molecular vibrations of polar specimens. This aloof geometry has recently proved very effective for imaging confined surface phonon modes in nanostructures [2,4] and highly beneficial for studying vibrational properties of beam-sensitive specimens [5][6][7] due to the drastic reduction of damage caused by the aloof electron beam when compared to more standard transmission geometries [8].

In this Letter, we add a new dimension to this approach by demonstrating that it can also be employed to assess the symmetry of normal modes occuring at the Brillouin zone center of anisotropic materials. Indeed, although the electron probe lacks the intrinsic angular selectivity of polarized photons used in optical spectroscopies, the scattering geometry in EELS provides a similar insight into the symmetry of vibrational excitations with the further advantage of a high spatial resolution. This is illustrated through the study of uniaxial icosahedral boron subphosphide B 12 P 2 where phonon mode symmetry is unambiguously retrieved from spectra acquired on crystals with distinct orientations. B 12 P 2 is a member of the α-rhombohedral boron structure group that includes compounds such as α-B 12 , B 12 As 2 , B 12 O 2 or B 12 C 3 and, as most of them, displays a number of technologically interesting properties. B 12 P 2 is a wide band gap semiconductor (3.35 eV) [9], a thermoelectric with a high positive Seebeck coefficient [10] and displays both high thermal and chemical stabilities as well as an extreme resistance to damage induced by high energy particle radiations [11,12]. This peculiar behaviour is attributed to a self-healing mechanism whose origin has recently been proposed to lie in the very low activation energy for the diffusion and recombination of interstitial-vacancy pairs [13] and makes it a potential candidate for betavoltaics [12]. This compound crystallizes in the R 3m rhombohedral space group. Its atomic structure is represented in Fig. 1 within the hexagonal triple cell. It consists in B 12 icosahedra linked together either through B-B bonds along the c-axis of the crystal or through B-P-B bonds in the perpendicular plane. Two crystallographically inequivalent boron sites are therefore present in the structure. The polar site, represented with dark green atoms in Fig. 1, are located on six vertices of the 12-atom cluster to form the top and bottom triangular faces and are exclusively bonded to boron atoms, belonging either to the same or to an adjacent icosahedron. The equatorial sites, represented by light green atoms, form a puckered hexagon in the plane perpendicular to the c-axis. As shown in Fig. 1, phosphorus cations are fourfold coordinated, forming bonds with an other cation and with three adjacent equatorial boron atoms.

As for its symmetry, B 12 P 2 is an uniaxial material, leading to EELS active modes with well-defined characteristics at the Γ point. The decomposition of the vibrational degrees of freedom on the irreducible representations of the crystal point group (D 3d ) gives [START_REF] Sherwood | Vibrational spectroscopy of solids[END_REF]

Γ vib. = 5A 1g ⊕ 2A 1u ⊕ 2A 2g ⊕ 5A 2u ⊕ 7E u ⊕ 7E g ,
from which the A 2u ⊕ E u acoustic modes should be removed to obtain the symmetry properties of the 39 remaining optical modes

Γ op. = 5A 1g ⊕ 2A 1u ⊕ 2A 2g ⊕ 4A 2u ⊕ 6E u ⊕ 7E g .
As EELS active modes in aloof geometry are necessarily those generating a non-zero electric polarization, we can conclude that a maximum of ten distinct peaks can be observed on the experimental spectra, four of A 2u and six of E u symmetry, the former giving rise to a macroscopic polarization parallel to the c-axis and the latter, to a polarization in the perpendicular plane.

The structure and vibrational properties of B 12 P 2 have been determined in the framework of density functional theory (DFT) using the Quantum ESPRESSO [START_REF] Giannozzi | [END_REF] package. Calculations were performed in the 14-atom trigonal unit cell using ultrasoft pseudopotentials [18] with plane-wave and charge-density cutoffs of 80 Ry and 640 Ry, respectively. The Brillouin zone of the trigonal crystal was sampled using a 8×8×8 k-point Monkhorst-Pack grid [19] and the generalized gradient approximation of Perdew, Burke, and Ernzerhof [20] (GGA-PBE) employed for exchange and correlation. This choice has been guided by the fairly accurate results usually provided by GGAs on the structural properties of strongly bound solids [14] and confirmed a posteriori by the excellent agreement obtained between optimized lattice parameters of a = 5.990 and c = 11.845 Å (hexagonal triple cell) and the average experimental values of 5.986 and 11.848 Å reported in Ref. 15 for stoichiometric B 12 P 2 . The dynamical matrix at the Γ point, electronic dielectric and Born effective charge tensors obtained using density functional perturbation theory (DFPT) were employed to calculate the principal components of the low-frequency local (q = 0) dielectric tensor ↔ (ω) shown in Fig. 2, entirely from first-principles [21,22]. Throughout this study, an average atomic mass of 10.811 a.m.u. was employed for boron, accounting for the relative natural abundance of 10 B and 11 B isotopes [24]. Consistently with our previous group-theoretical analysis, || (ω) and ⊥ (ω), where the parallel and perpendicular subscripts refer to the c-axis of the hexagonal triple cell, are respectively built from four resonances associated with the four modes of A 2u symmetry and six resonances associated with the six modes of E u symmetry occuring at the Brillouin zone center of B 12 P 2 . A schematic representation of these modes is provided in the Supplemental Material [23]. Note that these modes could be distinguished in polarized infrared absorption (IR) experiments carried out on single crystals as the former are active when light is polarized parallel to the c-axis and the latter when light is polarized in the perpendicular plane.

The local dielectric tensor is the key component entering the classical description of electron scattering by sur- face phonons occurring in aloof geometry in the context of the local continuum dielectric model [4,25]. In this classical, quasi-static approach [25][26][27][28], the energy-loss probability per unit angular frequency and path length, for an electron travelling along a rectilinear trajectory parallel to, and at a finite distance b from, the surface of a specimen occupying a half-space, is given by

d 2 P (ω, b) dω dx = e 2 (2π) 2 v 2 0 × ∞ -∞ Im [α (ω, q x , q y )] e -2b √ q 2 x +q 2 y q 2 x + q 2 y dq y , (1)
where v the electron velocity and (q x , q y ) the components of the scattering vector in the surface plane (see the scattering geometry in the insets of Fig. 3 (d) and (h) for the orientation of the cartesian coordinates). This expression involves the imaginary part of the polarizability of the specimen, which maxima correspond to the energy of the surface vibrational excitations. When both the surface and the electron beam are parallel to the c-axis, the polarizability can be expressed as

α(ω) = ⊥ (ω)(q 2 x || (ω) + q 2 y ⊥ (ω))/Q 2 -1 ⊥ (ω)(q 2 x || (ω) + q 2 y ⊥ (ω))/Q 2 + 1 . ( 2
)
In this expression, Q = q 2 x + q 2 y is the norm of the scattering vector parallel to the surface [25]. It should be stressed here that Eq. 2 depends explicitly on the scattering vector, leading to an imbalance in the role played by the two dielectric components. On the contrary, for a surface perpendicular to the c-axis of the crystal, the expression of the polarizability is particularly simple as it becomes independent of the scattering vector

α(ω) = || (ω) ⊥ (ω) -1 || (ω) ⊥ (ω) + 1 , (3)
and only involves the geometrical mean of the two dielectric components [29].

In expressions ( 1) and ( 2), the scattering vector along the electron trajectory is given by q x = ω/v which, for a typical loss of 70 meV and a primary electron energy of 30 keV, is of the order of 10 -4 Å-1 . The amplitude of the perpendicular component (q r in insets of Fig. 3 (d) and (h)) is more difficult to assess as it is usually not limited by the relatively large collection angle used in the experiments, but most likely by the low-q filter term occurring in the integral of Eq. 1. A typical average value up to an order of magnitude larger than the parallel component can however be estimated based on Eq. 1, resulting in a scattering vector oriented predominantly in the plane perpendicular to the beam trajectory.

Anisotropy resulting from the differences between Eq. 2 and Eq. 3 was mesured experimentally on B 12 P 2 microcrystals with different orientations.

Polycrystalline B 12 P 2 , synthetized by ultra-fast mechanochemical method through a reaction between BPO 4 , Mg and MgB 2 [31], was crushed into powder and dispersed on a lacey carbon grid. A first TEM inspection of the specimen led to the identification of micron or submicron crystals with sharp edges located in large carbon film holes and distinct crystallographic orientations, as shown in Fig. 3(a) and (e). The specimen was then analyzed in a Nion U-HERMES 200 system operated at 30kV. Diffraction patterns recorded prior to the EELS acquisition and shown in Fig. 3(b) and (f) illustrate the geometry of these experiments: for the crystal shown in (a), the c-axis is parallel to the electron beam and parallel to the surface whereas for the crystal shown in (e), the c-axis is approximately perpendicular to the electron beam and to the surface. Fig. 3(c) and (g) summarize these results by showing the hexagonal triple cell viewed along the beam direction.

Vibrational EELS spectra were acquired in aloof geometry at a distance of 10 nm from the edge of the specimen. The final spectra shown in Fig. 3(d) and(h) result from the acquisition of 5000 spectra of 50 ms each, subsequently realigned and summed, leading to an overall energy resolution of 6 meV. Convergence and collection semi-angles were set to α = β = 10 mrad. In both cases, a total number of eight peaks [30], including the low intensity shoulder of the peak located at 62 meV, is observed in the 40-125 meV energy range. The essential difference between the two spectra arises from the dramatic variations of the peak intensities induced by the crystal orientation.

In order to gain insight into the origin of these variations, theoretical spectra were calculated using Eq. 1, the dielectric components shown in Fig. 2 and the polarizabilities given in Eq. 2 and (3). These last two expressions correspond to the simplified geometries shown in inset of Fig. 3(d) and (h). Although significant deviations from these idealized geometries are observed in actual experiments, the predictions provided by the local continuum dielectric model reproduce very well the experimental results for both orientations. These results allow for precise symmetry assignment of each visible peak and lead to an important observation: whereas the intensity of the peaks associated with phonon modes of E u symmetry is largely enhanced in the first orientation (see Fig. 3(d)), it is strongly reduced in the second (see Fig. 3(h)). Conversely, the intensity of the peaks corresponding to phonons of A 2u symmetry is relatively weak in the first orientation but reinforced in the second. This behaviour is a direct consequence of Eq. 2 where the larger perpendicular (q y ) component of the scattering vector strengthens the intensity of modes generating an in-plane polarization compared to the geometrical average of Eq. 3. The high sensitivity of the aloof-beam EELS signal to the crystal orientation therefore appears as a very efficient means to discriminate phonon modes according to the direction of the macroscopic polarization they are building, either parallel or perpendicular to the crystal c-axis. This crystal orientation dependence can thus be employed to determine the symmetry of EELS active normal modes. Note that although similar information can be retrieved from polarized IR absorption experiments, the lack of sufficiently large single crystals, however, often hinders such analysis [32,33].

In conclusion, the present work demonstrates the capability of aloof-beam vibrational EELS to access the symmetry of phonon modes observed in anisotropic crystals. Distinct intensity variations are observed for peaks associated with phonons modes building macroscopic polarizations with different spatial orientations when changing the geometry of the experiment, i.e. changing the relative orientation of the electron beam and the surface with respect to the crystal axes. A great advantage of this approach resides in the high spatial resolution of EELS when implemented in a (S)TEM, therefore allowing to carry out this type of analysis at the nanoscale. This work was granted access to the HPC resources of IDRIS under the allocation 2019-A0060910820 attributed by GENCI (Grand Equipement National de Calcul Intensif). The authors thank V.A. Mukhanov for the sample synthesis as well as K. March, E. Balan and P. Giura for very fruitful discussions. This work was supported by Innovation Program Satt IDF Innov. (grant agreement N • L14S495).

FIG. 1 :

 1 FIG. 1: (color online) Structure of B12P2: (a) side-view and (b) top view of the hexagonal triple cell. For clarity, parts of the lattice have been omitted in (b). Phosphorus atom are in light purple, polar boron in deep green and equatorial boron in light green.

FIG. 2 :

 2 FIG.2:(color online) Principal components of the local dielectric tensor ↔ (ω) of B12P2 in the infrared domain of frequencies calculated with a damping parameter Γm = 4 cm -1 . Component perpendicular ( ⊥ ) and parallel ( || ) to the c-axis of the crystal are shown in the lower and upper panels, respectively. Real and imaginary parts of the dielectric tensor components are represented, respectively, with dashed and solid lines.

FIG. 3 :

 3 FIG. 3: (color online) Comparison between the experimental and theoretical vibrational EELS spectra of B12P2 for two microcrystals with different crystallographic orientations. (a) Conventional bright-field TEM image of the crystal acquired at 200 keV prior to the STEM experiment. The approximate beam position used for the 30 keV STEM-EELS experiment is indicated with a red point. (b) Indexed 30 keV diffraction pattern acquired in the STEM just before the EELS experiment showing that the electron beam propagates parallel to the c-axis of the crystal. (c) Corresponding orientation of the hexagonal triple crystal cell. (d) Comparison between the experimental EELS spectrum acquired in aloof geometry at a distance b ≈ 10 nm from the specimen edge and the theoretical spectrum calculated using Eq. 1 and the principal components of the dielectric tensor shown in Fig. 2. The symmetry of the modes corresponding to each peak is indicated. The idealized geometry employed in the calculations is shown in inset. (e)-(h) Same as (a)-(d) with the c-axis of the microcrystal close to perpendicular to the electron beam.


C Pasianot

Luca Messina
email: messina@kth.se

Nicolas Castin

Pär Olsson

Christophe Domain

Roberto C Pasianot

Göran Gustafsson

Ab initio-trained neural-network driven Monte Carlo simulations of microstructure evolution in irradiated iron alloys SMINS-4 Structural Materials for Innovative Nuclear Systems Manchester, UK

Ab initio-trained neural network driven monte carlo simulations of microstructure evolution in irradiated iron alloys

• Migration barriers depend on local atomic environment.

• Too many combinations!

• Poor thermodyanmic and kinetic descriptions with inaccurate predictions.

ATOMISTIC KMC OBJECT KMC

Atomic transition rates Object event rates, obtained by AKMC

y j = φ j (x i ) ( w 0 , w ji ) j x i y i y j = φ j (x i ) ( w 0 , w ji ) j y j = φ j (x i ) ( w 0 , w ji ) j y j = φ j (x i ) ( w 0 , w ji ) j y j = φ j (x i ) ( w 0 , w ji ) j y j = φ j (x i ) ( w 0 ,

PREDICTIONS OF UNKNOWN CASES

EAM-ANN simulations of FeCu thermal aging 6 6

• Hybdrid AKMC/OKMC simulations of Cu precipitation in α-Fe (thermal ageing).

• Clusters N ≥ 15 considered as objects.

barriers!

Interatomic potentials Density Functional Theory

• Fitted on experimental/ab initio data.

• Highly costumizable to desired specific properties.

• Computationally cheap.

• System-specific.

• Fitting is a time-consuming and non-trivial/non-linear task.

• Difficult to be accurate on many alloy properties at once.

• Provides detailed description of alloy thermodynamic and kinetic properties (no need of compromises).

• Can be applied to complex multicomponent alloys with little complexity addition.

• Computationally expensive (for large amount of configurations).

• Limited simulation-cell size and amount of computations.

Neural network training

• Test case: thermal aging of FeCu alloys (only 1 vacancy).

• 2000 configurations (training + validation).

• Maximize variety of selected atomic environments. Thermal ageing of FeCu alloys 20 atoms, 750 K 10 -4 10 -3 10 -2 10 -1 10 0 10 1 10 2 10 -10 10 -9 10 -8 10 -4 10 -3 10 -2 10 -1 10 0 10 1 10 2 10 -4 10 -3 10 -2 10 -1 10 0 10 1 10 2 10 -10 10 -9 10 -8 10 -4 10 -3 10 -2 10 -1 10 0 10 1 10 2 10 22 10 23 10 24

Average cluster radius [m]

Cluster density [m -3 ] Fe-0.6% Cu Fe-0.6% Cu

Annealing time [h] Annealing time [h]

773 K

• General satisfactory agreement.

• Improved time-scale match with respect to IAP-based study.

• Overestimation of cluster density in dilute alloy.

• Incorrect DFT prediction of solubility limit (0.76 eV vs 0.5 eV).

11

Confidential -Not to be disseminated!

FeCu test case -conclusions

• It is nowadays possible to obtain large DFT database and apply advanced regression schemes.

• Improved thermal ageing evolution (well-described time scales).

• Thermodynamic description is not accurate because of inexact DFT prediction. Need to perform ANN regression on equilibrium energies and migration barriers separately.

• Technical limitations due to computational costs (box size and amount of calculations).

• Next: FeCr alloys and ANN-based cohesive models.

Advanced modeling of FeCu and FeCr 20 atoms, 750 K Objective: train two ANN's to perform thermodynamic and kinetic modeling separately.

THERMODYNAMIC ANN KINETIC ANN

Predicts total energy (cohesive model) of a given atomic configuration, by interpolation on a database of DFT-computed energies.

Predicting the saddle-point energy of a given defect migration event, by interpolation on a database of DFT-computed migration barriers OR by using the TD cohesive model.

APPLICATIONS

RIGID-LATTICE

• 3 •

 3 Introduction: Transition rates in KMC simulations • Neural networks • Thermal ageing of FeCu alloys • Advanced modelling of FeCu and FeCr alloys Microstructure evolution driven by point-defect migration.

N.

  Castin et al., J. Chem. Phys. 135 (2011), 064502. • 3 stages a) Barriers predicted with artificial neural network, based on 10 4 NEB calculations with EAM interatomic potential. b) Calculation of stability and mobility of N ≥ 15 clusters. Clusters can emitt the vacancy, or a V-Cu pair. c) AKMC/OKMC simulations of thermal ageing. Coalescence of mobile precipitates plays an important role.

•

  , 300 eV, 33 k-pointsFull-kp relaxations of end states NEB relaxations with 2 3 kp grid Full-kp energy calculation of saddle point Much larger stability, higher dissociation energy• Explained by low vacancy formation energy in BCC copper (0.85 eV in Cu vs 2.18 eV in Fe).• Lower activation energy• Much longer, temperaturedependent mean free paths.Precipitation driven by coalescence of medium-sized clusters.

•

  Frequent loss of Cu atoms before vacancy emission. • Independent of cluster size and temperature until 1100 K. • Related to vacancy-copper correlation (drag) active below 1100 K (*) . * L. Messina et al., Phys. Rev. B 90 (2014).

  FeCu AND FeCr POTENTIAL -Phase diagram calculation by means of Metropolis Monte Carlo. -Interpolation of DFT energies neglecting atomic forces. AKMC THERMAL AGEING OF FeCr ALLOYS -Vacancy migration barriers calculated by two separate ANN's using the rigid-lattice potential and DFT-computed saddle-point energies. LATTICE-FREE FeCu AND FeCr POTENTIALS -Atomic forces are included indirectly. -Saddle-point energies are obtained and compared with the DFT values. -Allows to extend the set of DFT migration barriers.

TOTAL ENERGY:

Atomic energy functions (estimation of average energy assigned to each atom of species x). Outputs of the neural network trained on DFT energies of 4000 atomic configurations.

Local atomic density around atom a (describes univoquely the local atomic environment).

PREDICTED PHASE DIAGRAMS (MMC)

Very satisfactory agreement Underestimated Cu solubility

FeCu FeCr

Cu concentration [at%] Cr concentration [at%]

Confidential -Not to be disseminated! Conclusions & Perspectives

THANKS FOR YOUR ATTENTION!

• First-of-a-kind parameterization of KMC simulations with fully DFT-based advanced regression scheme, applied to FeCu and FeCr alloys.

• Neural networks are widely employed to ensure full transferability of DFT properties to the KMC simulation.

• Portability to different and more complex systems with little additional complexity.

• Limitations due to the high computational cost of massive amounts of DFT calculations are overcome by developing ANN cohesive models mimicking DFT ("DFT potentials").

• Preliminary applications to FeCu and FeCr alloys shows satisfactory agreement with thermal-ageing experiments.

• Future application to electron-irradiated FeCr alloys and FeMnNi alloys is foreseen, with the final goal of approaching real RPV steel compositions.