Marc Decombas

Francois Capman
email: francois.capman@fr.thalesgroup.comfrederic.dufaux

Erwann Renan

Frederic Dufaux

Beatrice Pesquet-Popescu
email: beatrice.pesquet@telecom-paristech.fr

Seam

M Décombas Ab

Seam Carving for semantic video coding

Keywords: video coding, seam carving, image resizing, saliency map, content-aware semantic compression

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Video compression aims at efficiently representing image data in order to allow easy storage and transmission. Common solutions include standards such as H.264/AVC [START_REF] Wiegand | Overview of the H.264/AVC Video Coding Standard[END_REF] with the goal of maximizing fidelity between the original and compressed images at a given bitrate. In this context, fidelity is generally assessed using well-known quality metrics such as Mean Square Error (MSE) and Peak Signal-to-Noise-Ratio (PSNR), or perceptually-based metrics such as Structural SIMilarity Index (SSIM) [START_REF] Wang | Image quality assessment: From error visibility to structural similarity[END_REF].

In this paper, we consider the problem of video compression under a different angle. More specifically, we aim at preserving the visual quality of the important parts and the semantic content of a video. We target security applications, such as video monitoring using drones, where it is very important to preserve the salient regions but the bitrate available is very low. As the background carries a very low significance, it can be shrunk. The bitrate is then greatly reduced while the quality and position of the important objects in the video are preserved.

In order to achieve this objective and to improve video compression efficiency, we propose to use seam carving. Avidan and Shamir [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF] introduced seam carving for content-aware image resizing that preserves semantic content. In particular, this technique has been shown to efficiently adapt the size of an image in order to match a display size. Seam carving is a method based on an energy function and a cumulative energy function. Most past works try to improve the energy function [START_REF] Hwang | Content-aware image resizing using perceptual seam carving with human attention model[END_REF] [START_REF] Chen | Content-Aware Video Seam Carving Based on Bag of Visual Cubes[END_REF][6] [START_REF] Achanta | Saliency detection for content-aware image resizing[END_REF] and the cumulative energy function [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF][8] [START_REF] Frankovich | Enhanced Seam Carving via Integration of Energy Gradient Functionals[END_REF] to avoid that important parts of the image be deformed and to keep temporal coherence for video resizing. A few more recent works use seam carving to compress images and video sequences [START_REF] Anh | Seam carving extension: a compression perspective[END_REF] [START_REF] Tanaka | Image coding using concentration and dilution based on seam carving with hierarchical search[END_REF][12] [START_REF] Tanaka | Improved image concentration for artifact-free image dilution and its application to image coding[END_REF][14] [START_REF] Tanaka | Seam carving with rate-dependent seam path information[END_REF].

In this paper, we define a new method where seam carving is applied in conjunction with a conventional video coding scheme such as H.264/AVC. As our method is a pre and post processing, the generated bitstream is compatible with all types of video and image standards.

More specifically, the video is split into Groups Of Pictures (GOP) with a predefined size. Next, we use seam carving to spatially reduce the original GOP dimensions in order to discard insignificant information while safeguarding semantically important regions. For each GOP, the number of seams suppressed is defined according to its content and is the same for all frames in the GOP. The motivation is to take into account temporal coherence and to have a more robust approach. Video compression is then applied on the downscaled video. Given that the quantity of data has been greatly reduced, compression is more effective than when applied on the original GOP. In order to be able to recover the original spatial dimensions without disturbing the objects semantic and the scene geometry, we analyze all the seams positions and encode some key points. With this approach, the amount of data to transmit the seams information is greatly reduced. At the decoder side, these key points control the seam carving up-sampling by modifying the energy function. In this way, the new seams positions are adapted with the content.

We carry out extensive performance evaluation experiments to show that the bitrate is greatly decreased when using the proposed seam carving based video coding. This gain is obtained while maintaining high visual quality on the semantically interesting regions. This paper is structured as follows. An overview of seam carving is presented in Sec. 2. The proposed semantic video compression method based on seam carving is introduced in Sec. 3. Experimental results are presented in Sec. 4 to assess the performance of the proposed approach. Finally, we draw conclusions and discuss future work directions in Sec. 5.

SEAM CARVING REVIEW

Seam carving is a new approach to resize images or video sequences while preserving semantically important content [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF]. A seam is an optimal 8-connected path of pixels on a single image from top to bottom or left to right. Formally, let I be an n x m image, the term vertical seam is defined to be the set of points

1 2 3 41 5 6 7 589 A 3 4BCDEF D7 589 A F 1 DF BCDE BCD E F (1)
with x the horizontal coordinate of the point.

Note that similar to the removal of a row or column from an image, removing the pixels of a seam from an image has only a local effect: all the pixels of the image are shifted left (or up) to compensate for the missing path. The cost of the seam is defined by an energy function and an energy cumulative function. Most of the works have focused on these two functions in order to improve the seam path such that important regions are preserved.

Energy functions

The first energy function proposed by Avidan and Shamir [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF] is the gradient magnitude on the luminance component. In [START_REF] Anh | Seam carving extension: a compression perspective[END_REF], Anh et al. use a combination of a saliency map and the magnitude of gradient. Other works propose to use different kinds of saliency maps [START_REF] Hwang | Content-aware image resizing using perceptual seam carving with human attention model[END_REF][5][6] [START_REF] Achanta | Saliency detection for content-aware image resizing[END_REF] with or without taking the temporal aspect into account.

Cumulative energy functions

These cumulative energy functions are dynamic programming algorithms and are used to find the minimum cumulative energy path. In the case of a vertical seam, the first step is to traverse the image from top to bottom and compute the cumulative minimum energy M for all possible connected seams for each entry (i,j). Avidan and Shamir [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF] propose to use backward energy to find the optimal seam path. The drawback of this method is that it does not measure the consequence of the suppression of a seam and can create some artifacts. Therefore, Rubinstein et al. propose in [START_REF] Rubinstein | Improved seam carving for video retargeting[END_REF] to use forward energy defined as:

CDF E 3 CDF E CD F E CDF E CD F E CDF E CD F E CDF E ! C"E with CDF E 3 #CDF E #CDF E #CD F E #CDF E (3) CDF E 3 #CDF E #CDF E (4) CDF E 3 #CDF E #CDF E #CD F E #CDF E (5)
And P(i,j) is an additional pixel based energy measure, for instance an energy function as defined in Sec. 2.1.

The advantage of this function is that it measures the energy due to newly inserted spurious edges. Indeed, artifacts may appear when a seam is removed and previously non adjacent pixels become neighbors. Recently, Frankovich and Wong propose an absolute energy which is a combination of forward energy and energy gradient [START_REF] Frankovich | Enhanced Seam Carving via Integration of Energy Gradient Functionals[END_REF]. To take the temporal aspect into account, Rubinstein et al. [START_REF] Rubinstein | Improved seam carving for video retargeting[END_REF] use graphcuts to compute temporal seams.

Seam carving for image/video compression

Seam carving has been recently applied to image/video compression [10][11][12][13][14] [START_REF] Tanaka | Seam carving with rate-dependent seam path information[END_REF].

In [START_REF] Anh | Seam carving extension: a compression perspective[END_REF], Anh et al. introduce a content-aware multi-size image compression based on seam carving. More precisely, the proposed codec encodes an image into a content-aware progressive bitstream where the seams position and value are encoded. As for the seams position, the absolute coordinate x of the first pixel of the first column is encoded and then the difference between the first pixel of the previous column and the first pixel of the current column. For the other points, the difference between the current pixel and the previous pixel of the same seam is encoded. For the color, the same concept is used by replacing the x coordinate by the absolute R, G, B values. For all the encoded information, an arithmetic coding is used. To compute the seam carving, a saliency map multiplied with the gradient is used as energy function. A Region Of Interest (ROI) bounding box is calculated with the dimensions w 0 x h 0 on the saliency map to control the reduction process and stop it when the reduced image reaches 1.5w 0 x 1.5h 0.

Tanaka et al. propose an image coding scheme which incorporates seam carving [START_REF] Tanaka | Image coding using concentration and dilution based on seam carving with hierarchical search[END_REF]. An image reduced by seam carving is transmitted, along with information about the seams position. As this information is very costly, the seams positions are approximated and the seams R, G, B values are not transmitted. More precisely, the seams positions are estimated as "pillars" of length N, such that the cost of a seam is (log 2 3)/N bpp. To define the pillar length, a top down approach is applied on a modified cumulative energy function (combination of the forward energy and seams bitrate with a Lagrangian multiplier). The stopping criterion is based on the cumulative energy: if a seam has a cumulative cost superior to a threshold, the reduction process is stopped. Results are evaluated with a SSIM metric [START_REF] Wang | Image quality assessment: From error visibility to structural similarity[END_REF].

In [START_REF] Wang | Cartesian resizing of image and video for data compression[END_REF], Wang et al. simplify seam carving principle by requiring suppression of straight lines (vertically and horizontally). They combine reduction of the number of frames with the reduction of the spatial dimension by using this method. The energy function is a gradient computed in the same direction as seam carving. A diffusion function is used to avoid seams which are too close to one another, as it may create visual distortions. The evaluation is done with PSNR.

The method proposed in [START_REF] Tanaka | Improved image concentration for artifact-free image dilution and its application to image coding[END_REF] improves upon [START_REF] Tanaka | Image coding using concentration and dilution based on seam carving with hierarchical search[END_REF] to limit artifacts created during interpolation. The idea is that the seams should avoid textured areas and the border of objects. If seams cross these areas or are too close to one another, artifacts may appear during reconstruction by linear interpolation. Therefore seams are required to pass through uniform areas even though these may include important regions of the image. For this purpose, a bottom-up approach is used instead of a top-down approach to define the length of an optimal pillar and to update the Lagrangian multiplier.

In [START_REF] Tanaka | Image concentration and dilution for video coding[END_REF], the authors apply the method in [START_REF] Tanaka | Image coding using concentration and dilution based on seam carving with hierarchical search[END_REF] for video reduction based on the graphcut approach of [START_REF] Rubinstein | Improved seam carving for video retargeting[END_REF]. The same seam is deleted for the current GOP and an 8-connexity is allowed with the seam from the previous and the next GOP in order to avoid artifacts at the transition between GOPs. To compute the seam for the current GOP, all its frames are used but the intra-frame is given more importance. During the evaluation, only one Quantization Parameter (QP) is used for all the frames of the two test sequences. In [START_REF] Tanaka | Seam carving with rate-dependent seam path information[END_REF], a piecewise linear approximation is proposed to find the optimal seam. The novelty is that the pieces of seams can have different directions and lengths.

In [START_REF] Tanaka | Image coding using concentration and dilution based on seam carving with hierarchical search[END_REF][13][14] [START_REF] Tanaka | Seam carving with rate-dependent seam path information[END_REF], the seam computation is modified by combining the forward energy with the seams encoding cost.

Limitations of current seam carving approaches for image/video compression

When seam carving is applied to image and video compression, the decoder has to synthesize seams data discarded at the encoder in order to recover the original dimensions. The traditional approach for up-sampling by seam carving is to apply the same process as in down-sampling by seam carving, but instead of deleting seams, to rather insert them for instance by duplication or interpolation. Two major difficulties arise. First, this process does not ensure that the inserted seams are in the same position as the deleted seams. Second, pixel duplication or interpolation may introduce artifacts in the synthesized image or video sequence.

The approaches discussed in Sec. 2.3 do not use seam carving up-sampling to synthesize the seams. Instead, the position of each seam is encoded. However, the cost to transmit this information may be prohibitive compared to the saving resulting from the size reduction. Consequently, these approaches only suppress few seams in order to limit seams data overhead and to avoid synthesis problems.

Our main contribution is a new way to represent seams for compression and synthesis. More specifically, a first step is to identify groups of seams and then to select the most relevant ones which are encoded and transmitted. This approach results in very low overhead to transmit seams position. We have also defined a new way to find seams which avoids important areas by modifying the cumulative energy maps. If a seam cross an important region, instead of stopping the process of reduction, the last value of the seam in the cumulative energy map is saturated and the seam carving is restarted. For the synthesis, a modified cumulative energy map allows good reconstruction of the scene geometry and preservation of the salient objects.

PROPOSED SEMANTIC VIDEO CODING USING SEAM CARVING

Given the limitations of current seam carving approaches for image/video compression, our objective is to significantly reduce the bitrate while preserving the semantic content. For this purpose, we propose a new approach for semantic video compression with seam carving.

General approach

A block diagram of the proposed architecture is shown in Figure 1. The different modules will be described in details hereafter. The main idea is to use seam carving to reduce the dimension of the video sequence. Then, the reduced video sequence is encoded with a standard H.264/AVC codec and the seams are encoded with our proposed scheme. After transmission, the videos sequence is reconstructed at the decoder side to recover the original dimension and to preserve scene geometry.

Figure 1. Architecture of proposed semantic video coding using seam carving.

The input video sequence is first divided into GOPs with a predefined size.

Seam carving is next applied to each frame of the GOP as illustrated in Figure 2. As discussed in the state-of-the-art, several methods exist to compute the energy function and the cumulative energy function. In this paper, we choose the saliency map of [START_REF] Rahtu | Simple and efficient saliency detector for background subtraction[END_REF] combined with a gradient as energy function. The proposed seam carving algorithm is detailed in Figure 2 (a). The saliency map of [START_REF] Rahtu | Simple and efficient saliency detector for background subtraction[END_REF] has the advantage to be multiscale, needs no training and is computed in the perceptual color space CIE Lab. To take into account temporal aspects, the current saliency map is linearly weighted with the previous one. To smooth the block effect from the multiscale approach while preserving the important parts of the frame, we then multiply the saliency map with a gradient to form our energy function.

As the forward energy in [START_REF] Rubinstein | Improved seam carving for video retargeting[END_REF] has proved to be more efficient than the backward energy [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF], it is chosen as cumulative energy function.

In parallel, we apply a threshold Binarisation_Threshold on the saliency map to separate the important objects from the background and to obtain a saliency binary mask. Next, a median filter is first applied to suppress small artifacts, followed by a dilatation to avoid that seams pass too close to salient regions.

To control the number of seams to be removed, the following process is carried out. Seam carving is first executed until the stopping point when a seam passes inside the saliency binary mask is reached. At this stage, the last position of the seam is saturated. Seam carving can then continue while avoiding salient regions. This saturation process is repeated Nb_Seam_Jump times. This approach allows to avoid important areas even though the cumulative energy is low.

As shown in Figure 2 (b), seam carving is applied vertically and horizontally on each frame, using the algorithm of Figure 2 (a). As the standard video coding H.264/AVC uses a 16x16 macro block structure, we suppress seams by multiples of 16.

(a) (b) The GOP-based seam carving process is illustrated in Figure 3. It is desirable to keep the spatial dimension for the GOP constant and in multiple of 16. In this way, no pixel padding is needed and the effectiveness of H.264/AVC video coding is improved. Moreover, it helps guaranteeing temporally coherent processing. For this purpose, seam carving is applied on each frame of the GOP as described in Figure 2. The number of seams that can be suppressed for each frame of the GOP is obtained. Next, we choose as selection criterion the smallest number of seams removable among the number of seams suppressible for each frame of the GOP. Finally, the reduced GOP frames are encoded using H.264/AVC video coding. In parallel, seams position is encoded as described in Sec. 3.2 and 3.3. Figure 3. GOP-based seam carving process.

123456789ABCA D2EFA5E179 F4672A8A 7946A1EF2 186E6789A A5857D2 1234523A1EF2 123456789ABCA D2EFA5E179 F4672A8A 186E6789AA 5849621A5857D2 7D6A8A 21675EAD2EFD 7D6A8A 817896EAD2EFD

Seams analysis to define group of seams

Our objective is to encode seams in a lossy way with few bits. In this section, we present a method to identify key points, or group of seams, in order to avoid a large data overhead. It is motivated by the observation that seams avoid important objects in the image and are therefore concentrated in some areas. More specifically, key points are automatically detected and their positions are transmitted. For the sake of simplicity, we describe the method in the case of a vertical seam carving, but it is straightforward to extend it to the horizontal case. Table 1 summarizes this approach as pseudocode.

Table 1. Pseudocode for group of seams detection. First, for each line perpendicular to the seam carving path, we scan the position of the seams and regroup together the positions. More precisely, given a current seam position, if the distance to the next seam position is under a threshold T_Gpe, we add the position to the group, otherwise we create a new group. The process is iterated until the end of the line. The next step is the selection of key groups on a predetermined number of lines Nb_Key_Lines. To initialize the process, we select equidistant reference lines. Then, each line y oscillates in the interval

Define Group of seams

$% & '()*++,-. / 0 % & '()*++,-. / 1 (6)
and the line having the largest group is selected. Afterwards, we save the position of the key line and all the group of seams position on this line. We underline that it is not judicious to keep only the position of the largest group of seams on the key lines as it may results in shifting and distorting the frame during seams synthesis.

Figure 4 illustrates our parameters in the case of a vertical seam carving for a 10x5 pixels image. Each color represents a seam. If T_Gpe=0, the first line (y=1) is composed of one group of 4 seams and a second group of 2 seams. If T_Gpe=2, the first line is composed of one group of 6 seams. We continue our illustration by fixing T_Gpe=1 and a number of key line Nb_Key_Lines=1. The key line is initialized at the line y=3. The next step is to find the largest group of seams. If K oscillate =0, the key line selected is y=3 and the largest group of seam is the first one with 4 seams. If K oscillate =2, the key line selected is the line y=2 and the largest group of seams have 6 seams. In this case, we save the position of the line y=2, the position of the group of seam, x=3 and the number of seams inside which is 6.

Figure 4. Example of seams analysis to define a group of seams.

Group of seams encoding

To encode the seams position while limiting the overhead cost, we exploit the characteristics of our data representation.

For the sake of simplicity, we describe the encoding for vertical seams, but it is straightforward to extend it to the horizontal case. Table 2 summarizes the procedure as pseudocode.

For each GOP, we first encode the number of vertical seams suppressed, Nb_Seams_Suppressed_V, with (ceil(log 2 (Width))) bits and the number of key lines, Nb_Key_Lines_V, with (ceil(log 2 (Height))) bits.

For each frame of the GOP, we then encode the maximum number of groups of seams, noted Nb_Groups_Max, with (ceil(log 2 (Nb_Seams_Suppressed_V))) bits , and the maximum number of seams in a group, noted Nb_Seams_Max, with (ceil(log 2 (Nb_Seams_Suppressed_V))) bits.

For each key line, we first encode the number of groups for this key line, Nb_Groups, with (ceil (log 2 (Nb_Groups_Max))) bits. Next, we encode the coordinate of the line, Position_Key_Line, with (ceil (log 2 (Height_Reduce -Coordinate_Previous_Line))) bits. For the first line Coordinate_Previous_Line = 0 and for the other lines, Coordinate_Previous_Line is the position of the previous line.

Finally, for each group in the key line, we encode the coordinate of the group with (ceil(log 2 (Width_Reduce -Coordinate_Previous_Group))). For the first group, Coordinate_Previous_Group=0 and for the subsequent groups, Coordinate_Previous_Group is the position of the previous group. We encode also the number of seams in the groups with (ceil (log 2 (Nb_Seams_Max -Nb_Seams_Accumul))) bits. For the first group, Nb_Seams_Accumul=0, and for the others groups Nb_Seams_Accumul is the number of seams previously accumulated in the key line.

Table 2. Pseudocode for group of seams encoding for vertical seams; the process is similar for horizontal seams.

Semantic

Number of bits used for coding

For

Seam synthesis controlled by group of seams

In this section, we present the modified energy map to require seams to pass through the key points defined during the seams analysis. This approach prevents the seams to pass through important objects and allows the significant regions to find their original position.

To proceed, we follow these different steps. While the frame has not reach its original size, we compute the energy map of the frame at the current dimension. For each key line, we retrieve the positions of the group of seams. Afterwards, we compute the energy cumulative map. Every time we reach a key line, the energy cumulative map is modified by adding a value Value_Add to the line excepted at the position of group of seams. We continue by finding the seam that minimizes the cumulative energy map and we add the seam in the frame and compute the pixel value. For the seams pixel value, we can use inpainting as in [START_REF] Domingues | Stream carving: An adaptive seam carving algorithm[END_REF] or a simple linear interpolation as in [3][11][13] [START_REF] Tanaka | Image concentration and dilution for video coding[END_REF]. The linear interpolation sometimes creates synthesis artifacts. Figure 5 shows an example of the approach for the sequence Soccer with the seams and key lines visible. In Figure 5 (a), the key lines correspond to the main objects. In Figure 5

RESULTS

To illustrate our results, we make the choice to use 3 different CIF sequences: Coastguard, Container, Soccer and 1 SIF sequence: Tennis. The sequences are temporally subsampled to 5 fps and we make our tests on 5 GOPs of 3 frames.

The proposed seam carving method adapts the amount of reduction to the content. For instance, we reach average spatial reduction of 50 % for Coastguard, 19.4% for Soccer, 26.55% for Tennis and 27.12% for Container. It is explained by the observation that salient objects are relatively small in Coastguard, whereas Soccer includes larger salient objects with fast motion. Tennis and Container fall in-between.

The different parameters have been heuristically fixed to T_Gpe=1, Nb_Key_Lines=4, K Oscillate =20 pixels, Value_Add=10000, Binarisation_Threshold=0.75, and Nb_Seam_Jump=16. We compare our approach with a standard H.264/AVC coding scheme, an approach using seam carving but without transmission of seams and one with a lossless coding of all the seams positions. H.264/AVC is used in intra coding mode. For the approach without transmission of seams, we make the following improvement. Instead of inserting N seams by finding the N first minimum of energy in the cumulative energy map, we use a binary map based on the saliency map to control the insertion. More precisely, the seam carving process finds the minimum of energy until an important area on the binary map is reached. At this moment, X seams are inserted. This process continues until N seams have been inserted and the image reaches its original size.

Overhead information has to be send to synthesis the seams at the decoder side and avoid object distortions and geometry scene errors. The corresponding data rate is independent of the video coding bitrate. Therefore, the cost to transmit the seams becomes more important at high compression ratios.

The reinsertion of the seams creates two kinds of distortion: the first one due to the position of the reinsertion which may create local translation, and the second one due to the padding of the seam. We underline that we are interested to preserve only the important content, whereas the background can be more distorted. As we apply the same quantization for all the methods, we assume that the quality of the important regions remains the same. However, full-reference metrics such as PSNR and SSIM [START_REF] Wang | Image quality assessment: From error visibility to structural similarity[END_REF] are not appropriate to assess the quality of our proposed method.

Figure 6 illustrates the coded bitrate for different video sequences and for different Quantization Parameters Intra (QPI). These results are also summarized in Table 3. Bitrate saving obtained by seam carving depends strongly on the amount of spatial reduction, but also on the background content. For instance, a suppressed moving texture leads to more significant bitrate saving than the suppression of uniform areas. The proposed technique leads to significant bitrate savings at high bitrates (low QPI), when compared to H.264/AVC. However, these gains decrease along with a decreasing bitrate (high QPI). At mid-range (QPI=24), our approach results in bitrate savings of 31.55%, 10.69%, 7.92% and 8.68% for Coastguard, Soccer, Tennis and Container respectively. For all sequences, the bitrate obtained with our proposed approach is very close to the one with the method without seams transmission. In other words, the overhead to transmit seams position with the proposed technique is very small. Finally, the transmission of all the seams position results in a very large overhead. Hence, this variant is only viable at very low QPI and performs poorly at high QPI. 8, Figure 9 and Figure 10 show some coded images examples at QPI=24 for the same four approaches under consideration. The synthesis of the seams allows keeping the context such that the visual quality of the important regions is preserved. In Container and Soccer, the synthesis of the seams in the background is very hard to noticed with our approach and the results are very close to the approach with the transmission of all the seams positions. In Coastguard and Tennis, as we use a linear interpolation and as seams pass through texture areas, artifacts are visible. However, the context remains preserved. Nevertheless, the synthesized sequences give visual results near the approach that transmit all the seams positions. Conversely, results without transmission of seams are rather poor. The player is severely distorted in Tennis. In Coastguard, the bottom of the coastguard boat is disturbed and its position is shifted. In Soccer, the three soccer players are disturbed. The two on the left are bigger and the one on the right is taller.

CONCLUSION

The main contribution of this paper is the development of a new semantic content-adaptive video coding scheme. We apply seam carving to reduce the spatial resolution of the video and compress the resulting reduced video sequences with a standard video codec. A new low cost representation of seams is introduced. This information controls the seam carving up-sampling during seams synthesis at the decoder side, in order to preserve the semantic of the video sequences. The salient objects are preserved, and the geometry scene is well reconstructed. Compare to conventional H.264/AVC coding, the proposed technique leads to significant bitrate savings.

Future works will address new metrics to assess seam carving performance, a better handling of temporal aspects, and an improved approach to synthesis the background.

Figure 2 .

 2 Figure 2. Proposed technique: (a) detailed seam carving algorithm, (b) application to vertical and horizontal seam carving reduction.

1 End

 1 , Nb_Groups[Idx_Line]]++ Else Nb_Groups[Idx_Line] ++ Position_Group_Seam[Idx_Line, Nb_Groups[Idx_Line]] = seam[Idx_Seam_In_Line] Nb_Seams_In_Group[Idx_Line, Nb_Groups[Idx_Line]] =

If Idx_Key_Line == 1

 1 groups of seams (ceil(log2(Nb_Seams_Suppressed_V))) Nb_Seams_Max maximum number of seams in a group (ceil(log2(Nb_Seams_Suppressed_V))) For Idx_Key_Line = 1 : Nb_Key_Lines Nb_ Groups[Idx_Key_Line] number of groups for this key line (ceil(log2(Nb_Groups_Max))) Position_Key_Line[Idx_Key_Line] coordinate of the key line (ceil(log2(Height_Reduce))) Else Position_Key_Line[Idx_Key_Line] coordinate of the key line (ceil(log2(Height_Reduce-Position_Key_Line[Idx_Key_Line-1]))) of seams (ceil(log2(Width_Reduce-Position_Group_Seam[Position_Key_Line[Idx_Key_Line], Idx_Group-1]))) Nb_seams_Accumul += Nb_Seams_In_Group [Position_Key_Line [Idx_Key_Line], Idx_Group-1] Number of seam previously encoded Nb_Seams_In_Group [Position_Key_Line [Idx_Key_Line], Idx_Group] Number of seams in the group (ceil(log2(Nb_Seams_Suppressed_V -Nb_seams_Accumul)))

Figure 5 .

 5 Figure 5 shows an example of the approach for the sequence Soccer with the seams and key lines visible. In Figure5(a), the key lines correspond to the main objects. In Figure5(b), the reconstruction seams are well positioned on the key lines. The objects are not deformed and the geometry of the image is preserved. (a) (b)

Figure 6 .

 6 Figure 6. Bitrate (kbps) as a function of quantification parameter (QPI): (a) Coastguard with 50% spatial reduction for the total sequence, (b) Soccer with 19.4% spatial reduction for the total sequence, (c) Tennis with 26.55% spatial reduction for the total sequence, (d) Container with 27.12% spatial reduction for the total sequence.

Figure 7 .Figure 8 .Figure 9 .Figure 10 .

 78910 Figure 7. Coastguard: (a) conventional H.264/AVC, (b) no seam coding, (c) our proposed approach, (d) total seam coding. (a) (b) (c) (d)

Table 3 .

 3 Percentage of bitrate saved compared to H.264/AVC as a function of QPI:(1) no seam coding, (2) our proposed approach, (3) total seam coding.

	Coastguard

: 50% spatial reduction for the total sequence Soccer : 19.4 % spatial reduction for the total sequence Tennis : 26.55% spatial reduction for the total sequence Container: 27.12% spatial reduction for the total sequence QPI

		(1)	(2)	(3)	(1)	(2)	(3)	(1)	(2)	(3)	(1)	(2)	(3)
	6	39.88 39.47 23.03 14.95 14.69	7.35	18.74 18.50	9.60	15.78 15.50	4.93
	12	40.35 39.71 14.27 14.62 14.24	3.44	18.90 18.54	5.44	15.46 15.03	-1.20
	18	37.41 36.47	-1.40	12.90 12.32	-4.24	14.90 14.33	-6.53	12.28 11.61 -13.63
	24	33.11 31.55 -31.22 11.70 10.69 -18.00	8.96	7.92	-29.85	9.81	8.68	-33.77
	30	30.34 27.40	-90.8	10.91	8.87	-49.49	6.79	4.79	-67.79	6.93	4.88	-72.20
	36	28.75 22.37 -234.3	8.89	3.94	-137.3	7.54	3.34	-149.4	5.24	1.25	-149.0
	Figure 7, Figure


Marc Decombas

Erwann Renan

Francois Capman

Frederic Dufaux

Beatrice Pesquet-Popescu

M Décombas

Closed loop Seams Approximation for Video Compression

Keywords: Resizing, Seam carving, Content aware compression, I

published or not. The documents may come   L'archive ouverte pluridisciplinaire

INTRODUCTION

Traditional video coding approaches like H.264/AVC [START_REF] Wiegand | Overview of the H.264/AVC Video Coding Standard[END_REF] are not optimal from a psychovisual point of view as they do not explicitly consider the semantic content of the video and in particular the salient objects.

In this paper, we target defense and security application for which video transmission at low bitrate is needed due to limited infrastructures. In this context, the overall image quality is not a crucial criterion. It is preferable to maintain the semantic meaning of the scene, in such a way that users can correctly interpret the content and take decisions in critical conditions.

For this purpose, we propose a semantic content-aware video coding scheme based on seam carving, which concentrates salient information in a reduced resolution sequence. More specifically, the background is suppressed and the bitrate is greatly reduced. Some extra information is transmitted to rebuild seams at the decoder side. Therefore, the quality and position of the salient objects in the video are preserved.

Our previous work [START_REF] Décombas | Seam carving for semantic video coding[END_REF][3] is based on the content-aware image resizing seam carving algorithm in [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF] which efficiently suppress non semantic content. The resulting coding scheme is shown to achieve significant bitrate savings compared to conventional H.264/AVC, while the salient objects are preserved and the scene geometry is well reconstructed. To avoid expensive extra information to rebuild the seams, we have proposed to approximate the seams by regrouping them together and encoding areas where the seams are concentrated.

These areas are typically between salient objects. During the regrouping process, seams are associated to a representative one. However, in the presence of spatially scattered seams, some of them are not well represented. This can leads to geometric distortions as the resynthesize seams are not inserted at the same position as the original seams. Moreover, it leads to higher encoding cost of the reduced frame due to the high frequencies create during seams suppression.

In this paper, we improve upon [START_REF] Décombas | Seam carving for semantic video coding[END_REF][3] on a specific point. After having defined the seams to suppress, a resized image is obtained and the seams are approximated by the approach from [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF]. Instead of transmitting this resized image, a new one is defined from the original image and the approximated seams. More precisely, the concept is to include the seams decoder inside the encoder. In this way, the seams at the encoder side are more similar to the seams at the decoder side than in the original approach.

The proposed semantic video coding scheme based on seam carving has been evaluated with the object-based quality metric in [START_REF] Décombas | A new object based quality metric based on SIFT and SSIM[END_REF] on Container. The results show a significant performance gain for GEOMETRIC_SIFT, as well as a ratedistortion improvement for SSIM_SIFT.

II. SEAM CARVING REVIEW

Seam carving is an approach to resize images or video sequences while preserving the semantic content [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF]. A seam is defined as an optimal -8 connected path of pixels on a single image from top to bottom or left to right. Formally, let I be an n x m image, the term vertical seam is defined to be the set of points

{ } { ( ) } | ( ) ( )|
with x the horizontal coordinate of the point. To define the seam, an energy function and a cumulative energy function are needed.

A. Energy function

The energy function defines the salient parts of an image to obtain an energy map. The first one based on a gradient on the luminance has been proposed by Avidan and Shamir [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF]. The gradient is efficient to highlight the textured areas and the border of the objects. But the textured areas are not necessarily salient. To address this problem, improvement has been done by using more complex saliency algorithm like in [START_REF] Anh | Seam carving extension: a compression perspective[END_REF] and [START_REF] Domingues | Stream carving: An adaptive seam carving algorithm[END_REF].

B. Cumulative energy function

To define the optimal seam to remove it is necessary to use dynamic programming in the cumulative energy function applied on the energy map. Avidan and Shamir first proposed backward energy [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF]. However, this function is done design to measure the consequence of seams suppression and leads to some visual artifacts. Therefore, Rubinstein et al. proposed in [START_REF] Rubinstein | Improved seam carving for video retargeting[END_REF] to use forward energy, defined as:

( ) ( ) { ( ) ( ) ( ) ( ) ( ) ( ) where
( ) | ( ) ( )| | ( ) ( )| ( ) | ( ) ( )| ( ) | ( ) ( )| | ( ) ( )
| and e(i,j) is an additional pixel based energy measure, for instance an energy function, | |is the absolute value and I the image.

C. Compression application

Seam carving has been applied to image/video compression

[2] [3][6][11][12][13][14][15].
Anh et al. note in [START_REF] Anh | Seam carving extension: a compression perspective[END_REF] that existing spatial scalable coders only support dyadic resolutions and are not content-aware. So they introduce a content-aware multi-size image compression based on seam carving. The image is spatially reduced until touching the Region Of Interest (ROI). The reduced image is encoded and the position and the content of the seams are also encoded. This leads to an important cost to represent the seam information. Moreover, severe block-artifacts occur on the boundaries of the ROI and non-ROI regions.

To solve the problem of block artifacts in [START_REF] Anh | Seam carving extension: a compression perspective[END_REF], Deng combine in [START_REF] Deng | Content-Based Image Compression for Arbitrary-Resolution Display Devices[END_REF] the advantages of seam carving and waveletbased coding to obtain a novel content-based spatial-scalable compression scheme.

At the opposite, to avoid as much as possible the overhead information, a seam can be simplified by a straight line. This approach, called data pruning, has been used by Võ in [START_REF] Võ | Selective Data Pruning-Based Compression Using High-Order Edge-Directed Interpolation[END_REF] to spatially reduce the frames and encode them with H.264/AVC and has been used by Wang et al. in [START_REF] Wang | Cartesian resizing of image and video for data compression[END_REF] to reduce the temporal dimension. The main problem of the data pruning is the strong constraint on the seams which may lead to visual distortions.

Tanaka et al. propose a compromise between the two approaches. The seam positions are encoded using piecewise straight lines [START_REF] Tanaka | Image coding using concentration and dilution based on seam carving with hierarchical search[END_REF] and the temporal aspect is added in [START_REF] Tanaka | Generalized selective data pruning for video sequence[END_REF].

All these techniques perform well while the bitrate is sufficiently high. But at very low bitrates, the overhead information for the seam positions becomes too significant and it is preferable to use a traditional coder.

To solve this problem that appears at very low bitrates, the works in [START_REF] Décombas | Seam carving for semantic video coding[END_REF] and [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF] approximates the seams by regrouping them and only encoding some key points where the seams are concentrated. This idea is based on the observation that avoiding the salient objects, the seams are concentrated between them. In [START_REF] Décombas | Seam carving for semantic video coding[END_REF], the key points are detected using a concentration criterion and the seams on the lines having the highest concentration points are regrouped and encoded. This information is used during the reconstruction of the frame to control the position of the seams. This is done by modifying the cumulative energy map at the decoder. However, the method in [START_REF] Décombas | Seam carving for semantic video coding[END_REF] has some limitations. The energy map has a simple design concerning the temporal aspect, the synthesis needs enhancement and the seams encoding approach is limited in the case of a high concentration points and lots of isolated seams on the same line.

The works in [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF] addresses the shortcomings of [START_REF] Décombas | Seam carving for semantic video coding[END_REF] by introducing several improved modules leading to better performances. A new energy map with superior temporal robustness, including a better combination of the saliency and gradient maps and an improved stopping criterion has been proposed. A new way to define groups of seams based on kmedian clustering has been then defined. In this approach, the frames are reduced and the seams are then approximated. During the reinsertion, some geometric artifacts can appear.

The closed loop approach consists in including the seams decoder inside the coder to obtain the approximated seams. This will lead to less difference between the seams at the decoder and the seams at the coders. These approximated seams are used to redefine a new resized image, leading to better performances. Instead of transmitting the resized image defined at the beginning, a new one is defined from the original image and the approximated seams.

III. PROPOSED SEAM CARVING FOR SEMANTIC CODING

The proposed seam carving scheme for semantic video coding is based on our earlier work in [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF].

Seam carving is used to reduce the dimension of the video sequence, while still preserving the semantic relevant objects. Then, the reduced video is encoded with H.264/AVC and the seams are encoded with our proposed scheme. After transmission, the video sequence is reconstructed at the decoder side. The information about the seams is used to recover the original dimension and to preserve the scene geometry. Fig. 1 shows the global approach. The process to define the list of seams to suppress is illustrated in Fig. 2. Taking a frame as input, an energy function is applied to define an energy map. Post-processing is applied to suppress small region and inflate the objects in the energy map. To define the seams, a cumulative energy function is then applied on the energy map. In parallel, the energy map is binarized to identify salient objects. The resulting control map is then used to govern the reduction process. More specifically, seam carving is iteratively carried out, as long as seams do not cross the salient objects in the control map. In this paper, we introduced a closed loop approach consisting in defining a new resized image from approximated seams by reproducing the insertion process used at the decoder side. Fig. 3 illustrates the global process. A list of initial seams is first obtained by applying the process in Fig. 2. The list of seams is expressed on the original frame and reordered. In this way, K representative seams are defined from the initial list of seams. As the idea is to detect and encode the areas where the seams are concentrated, group of seams are defined by k-median clustering [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF].

The next step is to simulate the seams reinsertion process taking place at the decoder side. The idea is that the seams suppressed at the encoder should closely match the seams reinserted at the decoder. For this purpose, the reduced frame is used to compute an energy map on which the groups of seams are inserted to control the position of the seams as in [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF]. In this way, a list of approximated seams similar to the one obtained at the decoder is available to compute a new reduced frame. This new reduced frame is then encoded with H.264/AVC encoder and transmitted.

IV. RESULTS

Container CIF sequence with available binary segmentation mask [21] is used to evaluate the performance of the closed loop approach. The sequence is temporally subsampled to 5 fps and we make our tests on 5 GOPs of 3 frames. 6 median seams with 12 groups of seams are defined horizontally and vertically. H.264/AVC encoding is in full Intra mode. A spatial size reduction of 40 % is reached for Container.

Assessing the performance of a semantic object-based video coding scheme is challenging. Common quality metrics fail when geometric deformations occur. In [START_REF] Décombas | A new object based quality metric based on SIFT and SSIM[END_REF], a metric especially designed to measure the compression artifacts and the geometric deformation has been proposed. Compression artifacts due to encoders like H.264/AVC are measured by SSIM windows around SIFT points. This metric is called SSIM_SIFT. The geometric artifacts are measured by the standard deviation between the matching of the SIFT points and is called GEOMETRIC_SIFT. The Table I illustrate the influence of the closed loop approach on the GEOMETRIC_SIFT metric, compared to [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF]. [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF] AND THE CLOSED LOOP APPROACH Geometric_Sift score for the Approach from [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF] Geometric_Sift score for the closed loop approach Container 0,52 0,17

GEOMETRIC_SIFT has been divided by 3. The initial seams are defined in the sky and in the water. During the seams approximation, all the seams are approximated in the water creating a small translation. With the closed loop approach, the translation is avoided as the seams before and after the approximation are in the water.

The Fig. 4 illustrates the rate distortion curves for the Container ship considered as the salient objects. The visual distortion is now expressed using SSIM_SIFT. We compare conventional H.264/AVC coding, the seam carving approach from [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF] and the closed loop approach proposed in this paper. In Fig. 4, the closed loop approach leads to improved performance compared to the approach in [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF]. Moreover, the proposed scheme also outperforms H.264/AVC. For example, at a SSIM_SIFT of 0.97, the bitrate of the H.264/AVC approach is about 352 bps. With the same SSIM_SIFT, the seam carving approach of [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF] is about 405 bps and the closed loop approach reaches 314 bps. This represents a bitrate saving of 12% compared to the H.264/AVC approach and of 28% compared to the seam carving approach from [START_REF] Décombas | Improved seam carving for semantic video coding[END_REF].

Visually, the process is illustrated in Fig. 5. On the left, the original seams are visible in white. In the image in the center, the seams are approximated and reinserted in the original reduce frame. This is the result obtained at the decoder side with the approach from [START_REF] Décombas | Seam carving for semantic video coding[END_REF][3] and also at the encoder when the seams decoder is included. In the image on the right, the seams are reinserted in the new reduce frame and are visible at the decoder side. We remind that the decoder is included in the encoder and so the closed loop process is applied. The approximation between the first and second image is higher than the approximation between the second and the third frame. This leads to less geometric distortion. V. CONCLUSION This paper describes a new approach to limit geometric distortions in a seam carving based semantic video coding. More specifically, initial seams are approximated by simulating the seam reinsertion process. A new resized image is then determined from the approximated seams. With this approach, during synthesis at the decoder side, the reinserted seams are more similar to the seams suppressed at the encoder. Experimental results show that the proposed approach lead to better coding performance, using two object-based quality metrics, GEOMETRIC_SIFT and SSIM_SIFT.

Fig. 1 .

 1 Fig. 1. Architecture of the proposed semantic video coding using seam carving

Fig. 2 .

 2 Fig. 2. List of seams process

Fig. 3 .

 3 Fig. 3. Closed loop approach to redefine new resized image

Fig. 4 .

 4 Fig. 4. RD curves for container

Fig. 5 .

 5 Fig. 5. Seam visible in container. From left to right: Original seams at the encoder, Seams at the decoder with the approach from [2][3] and at the encoder with the closed loop approach, Seams at the decoder with the closed loop approach

TABLE I GEOMETRIC_SIFT

 I SCORE FOR THE SEAM CARVING APPROACH FROM


Marc Decombas
email: marc.decombas@thalesgroup.com

Frederic Dufaux
email: frederic.dufaux@telecom-paristech.fr

Erwann Renan
email: erwann.renan@thalesgroup.com

Beatrice Pesquet-Popescu
email: beatrice.pesquet@telecom-paristech.fr

Francois Capman
email: francois.capman@thalesgroup.com

M Décombas

E Renan #

Improved Seam Carving for Semantic Video Coding

come   L'archive ouverte pluridisciplinaire

I. INTRODUCTION

Traditional video coding approaches like H.264/AVC [START_REF] Wiegand | Overview of the H.264/AVC Video Coding Standard[END_REF] or HEVC aim at minimizing the Mean Squared Error (MSE). Moreover, they do not explicitly consider visually salient regions. As a result, they may not be optimal from a psychovisual point of view.

We target defense and security applications with limited infrastructures, constraining video transmission at low data rates. In this context, the overall image quality is not a crucial criterion. Instead, the objective is to maintain the semantic meaning in such a way that users can correctly interpret the content and take decisions in critical conditions.

For this purpose, we propose a semantic content-aware video coding scheme based on seam carving, which concentrates salient information in a reduced resolution sequence. In this way, the background is suppressed and the bitrate is greatly reduced. Some extra information is transmitted to rebuild seams at the decoder. Therefore, the quality and position of the salient objects in the video are preserved.

Our previous work [START_REF] Décombas | Seam carving for semantic video coding[END_REF] is based on the content-aware image resizing seam carving algorithm in [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF] which efficiently preserves semantic content. The coding scheme in [START_REF] Décombas | Seam carving for semantic video coding[END_REF] is shown to achieve significant bitrate savings compared to conventional H.264/AVC, while the salient objects are preserved and the geometry scene is well reconstructed.

In this paper, we improve upon [START_REF] Décombas | Seam carving for semantic video coding[END_REF] on several points. Firstly, we introduce a new energy map with superior temporal robustness, including a better combination of the saliency and gradient maps, and an improved stopping criterion. A new way to define groups of seams based on kmedian clustering is then proposed. Finally, an improved background synthesis is used based on Shift-map [START_REF] Svarm | Shift-map Image Registration[END_REF], which gives better visual results.

The proposed semantic video coding scheme based on seam carving has been evaluated with the object-based quality metric in [START_REF] Décombas | A new object based quality metric based on SIFT and SSIM[END_REF] on three objects from the sequences coastguard and container. In the case of coastguard, our approach gives notably better results at all bitrates for the two objects. In the case of container, smaller gains are obtained.

II. SEAM CARVING REVIEW

Seam carving is an approach to resize images or video sequences while preserving the semantic content [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF]. A seam is defined as an optimal -8 connected path of pixels on a single image from top to bottom or left to right. Formally, let I be an n x m image, the term vertical seam is defined to be the set of points 1 2 3 41 5 6 7 589 A 3 4BCDEF D7 589 A F 1 DF BCDE BCD E With x the horizontal coordinate of the point. To define the seam, an energy function and a cumulative energy function are needed.

A. Energy function

The energy function allows defining the salient parts of an image. The first one has been proposed by Avidan and Shamir [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF]and is based on a gradient on the luminance. The gradient is efficient to highlight the textured areas, although these areas are not necessarily salient. To address this problem, saliency map information has been integrated in [START_REF] Anh | Seam carving extension: a compression perspective[END_REF] and [START_REF] Domingues | Stream carving: An adaptive seam carving algorithm[END_REF].

B. Cumulative energy function

After defining the salient part in the energy function, it is necessary to define the optimal seam to remove. This is done by dynamic programming in the cumulative energy function. Avidan and Shamir first proposed backward energy [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF] to suppress paths with minimum energy. However, this function fails to measure the consequence of seams suppression and leads to some visual artifacts. Therefore, Rubinstein et al. proposed in [START_REF] Rubinstein | Improved seam carving for video retargeting[END_REF] to use forward energy, which takes into account seam suppression. It is defined as:

CDF E 3 CDF E CD F E CDF E CD F E CDF E CD F E CDF EF !
where CDF E 3 "CDF E "CDF E "CD F E "CDF E CDF E 3 "CDF E "CDF E1 CDF E 3 "CDF E "CDF E "CD F E "CDF E and e(i,j) is an additional pixel based energy measure, for instance an energy function, and I the image.

C. Content synthesis

When seam carving is used to enlarge an image, content synthesis is needed. Linear interpolation, one of the simplest methods to synthesize missing areas, is often used in seam carving [START_REF] Avidan | Seam Carving for Content-Aware Image Resizing[END_REF] and [START_REF] Rubinstein | Improved seam carving for video retargeting[END_REF]. It performs well as long as seams do not cross textured areas or get too close from each other. In [START_REF] Võ | Selective Data Pruning-Based Compression Using High-Order Edge-Directed Interpolation[END_REF], Võ proposes a multiframe interpolation using neighbor frames. However, these interpolation techniques tend to fail when the missing area to be recovered is too large. Domingues et al. propose in [START_REF] Domingues | Stream carving: An adaptive seam carving algorithm[END_REF] to use the inpainting approach from Bertalmio et al. [START_REF] Bertalmio | Navier-stokes, fluid dynamics, and image and video inpainting[END_REF]. This inpainting is quite efficient for rebuilding structures, but often fails to reconstruct textured areas.

D. Compression application

Seam carving has been applied to image/video compression [START_REF] Décombas | Seam carving for semantic video coding[END_REF], [START_REF] Anh | Seam carving extension: a compression perspective[END_REF], [START_REF] Deng | Content-Based Image Compression for Arbitrary-Resolution Display Devices[END_REF], [START_REF] Võ | Selective Data Pruning-Based Compression Using High-Order Edge-Directed Interpolation[END_REF], [START_REF] Wang | Cartesian resizing of image and video for data compression[END_REF], [START_REF] Tanaka | Image coding using concentration and dilution based on seam carving with hierarchical search[END_REF] and [START_REF] Tanaka | Generalized selective data pruning for video sequence[END_REF].

In [START_REF] Anh | Seam carving extension: a compression perspective[END_REF], Anh et al. introduce a content-aware multi-size image compression based on seam carving, because existing spatial scalable coders only support dyadic resolutions and are not content-aware. They reduce the spatial dimension until touching the Region Of Interest (ROI) and encode the reduced image and all the information of the seams (position and content).This leads to an important cost to represent the seam information. Moreover, severe block-artifacts occur on the boundaries of the ROI and non-ROI regions.

In [START_REF] Deng | Content-Based Image Compression for Arbitrary-Resolution Display Devices[END_REF], Deng combine the advantages of seam carving and wavelet-based coding to obtain a novel content-based spatialscalable compression scheme that solve the problem of block artifacts in [START_REF] Anh | Seam carving extension: a compression perspective[END_REF].

To address the problem of overhead information, a seam can be simplified by a straight line. Data pruning has been used by Võ in [START_REF] Võ | Selective Data Pruning-Based Compression Using High-Order Edge-Directed Interpolation[END_REF]to spatially reduce the frames and encode them with H.264/AVC. Based on the same principle, Wang et al. propose to also reduce the temporal dimension [START_REF] Wang | Cartesian resizing of image and video for data compression[END_REF]. The problem of these approaches is the strong constraint on the seams, which may lead to visual distortions.

Tanaka et al. propose a compromise between the two approaches. The seam positions are encoded using piecewise straight lines [START_REF] Tanaka | Image coding using concentration and dilution based on seam carving with hierarchical search[END_REF]. The temporal aspect is considered in [START_REF] Tanaka | Generalized selective data pruning for video sequence[END_REF].

All these techniques perform well when the bitrate is sufficiently high. But at very low bitrates, the overhead information for the seam positions becomes too significant and it is preferable to use a traditional coder.

Based on the above considerations, the scheme in [START_REF] Décombas | Seam carving for semantic video coding[END_REF] approximates the seams by regrouping them and only encoding some key points. Given that the seams avoid salient objects, seams are regrouped with a concentration criterion and each group is encoded. This information is used during the reconstruction of the frame to control the position of the seams. This is done by modifying the cumulative energy map at the decoder. However, the method in [START_REF] Décombas | Seam carving for semantic video coding[END_REF] has some limitations. The energy map has a simple design concerning the temporal aspect and the synthesis needs enhancement.

The new scheme proposed in this paper addresses the shortcomings of [START_REF] Décombas | Seam carving for semantic video coding[END_REF] by introducing several improved modules leading to better performances.

III. QUALITY METRICS

Assessing the performance of a semantic content-aware video coding scheme remains a challenge.

Metrics such as Peak-Signal-to-Noise-Ratio (PSNR) or Structural SIMilarity (SSIM) [START_REF] Wang | Image quality assessment: From error visibility to structural similarity[END_REF] compare corresponding pixels or blocks in the reference and processed images. However, these metrics fail when geometric deformations occur.

Some metrics have been defined for image retargeting. Azuma et al. have defined in [START_REF] Azuma | SSIM based image quality assessment applicable to resized images[END_REF] a full reference metric based on Scale-Invariant Feature Transform (SIFT) [START_REF] Lowe | Distinctive image features from scale invariant key points[END_REF] and SSIM to evaluate images resized by different retargeting algorithm. [START_REF] Liu | Image Retargeting Quality Assessment[END_REF]. The objective of [START_REF] Azuma | SSIM based image quality assessment applicable to resized images[END_REF], [START_REF] Liu | Image Retargeting Quality Assessment[END_REF] is to evaluate resized (smaller) images. These metrics are not designed to evaluate compression artifacts and do not take into account the geometric deformations that can occur in some content-based video coding schemes.

Liu et al. presented an objective metric simulating the Human Vision System (HVS) based on global geometric structures and local pixel correspondence based on SIFT in

Therefore, in [START_REF] Décombas | A new object based quality metric based on SIFT and SSIM[END_REF] we have presented a metric especially designed for this problem. This metric measures two kinds of artifacts, geometric artifacts and compression artifacts. Compression artifacts due to encoders like H.264/AVC are measured by SSIM windows around SIFT points. These windows are totally included in the object, so the result is not disturbed by the synthesis of the content of the seams. To measure the geometric artifacts, the standard deviation between the matching of the SIFT points is computed. Subjective tests in [START_REF] Décombas | A new object based quality metric based on SIFT and SSIM[END_REF] show that the metric achieves high correlation with Mean Opinion Scores (MOS).

IV. PROPOSED SEAM CARVING FOR SEMANTIC CODING

A. General approach

The proposed seam carving scheme for semantic video coding is based on our earlier work in [START_REF] Décombas | Seam carving for semantic video coding[END_REF]. The main idea is to use seam carving to reduce the dimension of the video sequence, while still preserving the semantic relevant objects. Then, the reduced video is encoded with H.264/AVC High Profile and the seams are encoded with our proposed scheme.

After transmission, the video sequence is reconstructed at the decoder side. Missing areas are synthesized by seam synthesis in order to recover the original dimension and to preserve scene geometry. Fig. 1 shows the global approach. In this paper, we introduce several improvements compared to [START_REF] Décombas | Seam carving for semantic video coding[END_REF]. First, we propose a new energy map, achieving superior temporal robustness and providing a better stopping criterion. The stopping criterion is automatically defined to preserve the salient content. Second, we develop a new way to define and encode the seams, resulting in an improved representation. Thirdly, we propose to use the shift-map method [START_REF] Svarm | Shift-map Image Registration[END_REF] to synthesize the content at the decoder. These contributions are discussed in more details hereafter. Finally, our proposed approach is evaluated with the object-based quality metric in [START_REF] Décombas | A new object based quality metric based on SIFT and SSIM[END_REF] to assess rate-distortion performance.

B. Proposed energy map

The input video sequence is first divided into GOPs with a predefined size. Seam carving is next applied to each frame of the GOP as illustrated in Fig. 2. For each frame, an energy function is defined based on gradient, saliency and temporal information. Post processing is applied on this energy map to better preserve salient objects. The forward cumulative energy map is then computed to define the seam to be suppressed. In parallel, the energy map is binarized to obtain a control map. This control map is used to decide when to stop seams suppression. More precisely, seam carving is iterated until touching an object in the control map. Fig. 2. Overall seam carving scheme 1) Saliency map: In the proposed scheme, we use the saliency map from [START_REF] Rahtu | Simple and efficient saliency detector for background subtraction[END_REF]. It has the advantage to be multiscale, needs no training and is computed in the perceptual color space CIE Lab.

In order to improve temporal robustness, the motion intensity between the current frame and the previous or the next frame is integrated in the saliency map. The saliency map is defined based on 4 features: 3 from the CIE Lab color space and the motion intensity. In the case of a moving camera, the background has a global motion, whereas other objects follow local motion. In order to emphasize salient moving objects, global motion is computed and substracted from the motion intensity obtained by optical flow. Finally, a temporal weighting of the saliency map is applied. The temporal processing to compute the saliency map is illustrated in the Fig. 3. 1 has been empirically fixed to 0.3. 2) Energy function: The energy function is then obtained by combining the saliency map and the gradient information. The gradient has a high value on the objects borders only. Therefore, a simple multiplication of the saliency map and gradient, as proposed in [START_REF] Décombas | Seam carving for semantic video coding[END_REF], may lead to the elimination of salient regions within the objects where the gradient is small.

To alleviate this weakness, a new combination is proposed here, based on a linear combination of the saliency map and the gradient 8 9:; 3 < 8 =>?@ ADBC8 =>?@ E ACBC8 =>?@ E ADBC8 =>?@ E D C <E 8 E?F5GAHI ADBC8 E?F5GAHI E ACBJ8 E?F5GAHI K ADBC8 E?F5GAHI E where E Grad is the energy obtained from the gradient, E Saliency the energy obtained by the process defined in Fig. 3 and 2 is a weighting parameter empirically fixed to 0.3. This combination allows a better preservation of the information obtained from the saliency map (in particular inside the objects) and enhances the importance of the objects borders.

3) Post-processing: In [START_REF] Décombas | Seam carving for semantic video coding[END_REF], median filter and dilatation are applied on the control map in order to improve salient objects detection. This approach has the advantage to well preserve salient objects but stops the seam suppression too early. To alleviate this shortcoming, the same post-processing is now applied to the energy map.

In this way, the reduction process can go further with a better preservation of the salient objects, because the seams will avoid the area highlighted in the control map, as the control map is only a binarisation of the energy map.

4) Isolated seams discarding:

It is desirable to keep the spatial dimension for the GOP constant and in multiple of 16. In this way, no pixel padding is needed and the effectiveness of the subsequent H.264/AVC video coding is improved.

For this purpose and as the number of seams to suppress is not constant from frame to frame, we propose in the present scheme to discard isolated seams. More specifically, For each seam, the distance between the current seam and the seam at his left, respectively at his right, is computed. Next, the lowest value among the two distances is kept and associated to the current seam. Finally, the seam having the largest distance is suppressed. The process is iterated till the frame size reaches a multiple of 16 to avoid pixel padding.

C. Proposed seam carving clustering

Seam carving is first applied in the vertical direction and then in the horizontal direction. In each direction, seam carving suppresses seams one by one in an iterative process.

The coordinate of the seams are defined relative to the original image dimension. Next, all the seams are rearranged by ordering the horizontal, respectively vertical, coordinates in an increasing order. In this way, the seams are defined from left to right, respectively top to bottom, and they do not intersect. This is illustrated with 3 seams in Fig. 4. In order to constraint the amount of overhead information needed to represent seams, they are clustered into groups of seams whose coordinates will be transmitted. For this purpose, a k-median algorithm is used to find median seams. The number of median seams is defined by the user. Hereafter, the number of median seams has been empirically chosen in a range between 3 and 6. Each seam is then associated to a cluster by minimizing its Euclidian distance.

The median seams are used to define the coordinates of each cluster. Given that the seam carving reinsertion process is performed from left to right and therefore pushes the right part of the image, it is necessary to translate the median seams coordinates accordingly. By this way, after the reinsertion, the groups of seams are properly centered.

We now identify concentration areas in the perpendicular direction in order to define group of seams. We have noted that the seams are concentrated between salient objects. The coordinates of these concentration areas have to be encoded to correctly reposition the salient objects at the decoder. To find these areas, for each median seam, the cumulative Euclidean distance between the median seam and its associated group of seams is calculated. This is done for each point of the median seam. The concentration areas are then defined as the points with shortest distances.

To avoid keeping many points from the same concentration area, the seam is beforehand divided into subparts and the minimum cumulative Euclidean distance is searched for each subpart. The number of subparts is defined by the user. Clearly, defining more subparts results in more precise seams at the synthesis. An example is shown in Fig. 5 where 6 seams are represented. In this example, 2 median seams are defined but not represented here and the seams are divided into 2 subparts. The red rectangle represents the group of seams. The position of the group of seams, and the number of seams for each group is encoded.

All the process is resumed in the Fig. 6.

D. Proposed synthesis

In seam carving, synthesis commonly uses linear interpolation. This interpolation performs reasonably well when the seams do not cross textured areas or are not close to each other. The use of saliency map to create the energy map allows the seams to cross non salient textured areas and preserve the salient parts. Moreover, our cluster-based seams positions encoding approach ensures that the seams are grouped together. Therefore, linear interpolation is not suited for our seam carving approach.

To solve this problem, the inpainting algorithm from Svarm and Strandmark based on Shift-map has been chosen [START_REF] Svarm | Shift-map Image Registration[END_REF].This algorithm is based on energy minimization over a large space of labels and is optimized using Gaussian pyramid.

More specifically, the area to be synthesized is filled after the reinsertion of one group of seams. The synthesis is performed over a search area. On the one hand, this search area should not be too large to avoid using weakly correlated data. On the other hand, it should be sufficiently large to preserve enough information to rebuild the texture. Therefore, the search area is defined as the smallest rectangular box including the group of seams, as illustrated in blue in Fig. 7. The search area is then extended to twice the number of seams in the group, in order to have enough texture information, as represented in red in the Fig. 7. To assess the performance of the proposed coding scheme, we use 2 CIF sequences with available binary segmentation masks [21]: coastguard and stefan. The sequences are temporally subsampled to 5 fps and we make our tests on 5 GOPs of 3 frames. 6 median seams with 3 key points are defined horizontally and vertically. The threshold of binarisation has been fixed at 0.75 for coastguard and 0.9 for stefan. These parameters have been empirically defined. The encoding is in full-Intra mode.

With the proposed seam carving scheme, the process is iterated until touching an object in the control map. A size reduction of 41.6 % is reached for coastguard and 18.9% for stefan. In comparison, a size reduction of 17.8% was reached for coastguard and 10 % for stefan with the seam carving scheme in [START_REF] Décombas | Seam carving for semantic video coding[END_REF]. The percentage of bitrate saved compared to H.264/AVC High Profile for different methods is illustrated in Table I. Compared to an approach without seam encoding, where the objects are deformed, our approach represents a small overhead. At the opposite, an approach where all the seams are encoded cannot be envisaged for low bitrate applications. Compared to the approach in [START_REF] Décombas | Seam carving for semantic video coding[END_REF], we reach a higher reduction due to a better seams selection. By this way, for QP=42, the bitrate saved reaches 19.3% and 10.6% compared to H.264/AVC for coastguard and stefan respectively. The gain is due to the proposed seam definition and representation.

Fig. 8, Fig. 9 and Fig. 10 illustrate the rate distortion curves for the two boats of coastguard and stefan considered as the salient objects. To measure the distortion, we use the objectbased quality metric developed in [START_REF] Décombas | A new object based quality metric based on SIFT and SSIM[END_REF]. It is clear that for coastguard, our approach gives notably better results at all bitrates. For stefan, seams have suppressed the crowd leading to good performance gains. The Fig. 11 shows the visual quality of the synthesized frame obtained using the traditional linear interpolation and the proposed approach based on shift-map. The water is better synthesized. However, some artifacts remain visible due to the lack of information in this large removed zone. [START_REF] Décombas | Seam carving for semantic video coding[END_REF], we have introduced several new contributions leading to improved coding performance. More specifically, we have introduced a new energy map with a better temporal aspect, as well as a better combination of the saliency and gradient maps, which allows a better preservation of the salient objects. Groups of seams are now defined with a k-median algorithm. The utilization of the Shift-map [START_REF] Svarm | Shift-map Image Registration[END_REF] to synthesize the seams content at the decoder side gives better visual results on the background. Performance assessment shows notable coding gains on the coastguard and stefan test sequence, using an object-based quality metric [START_REF] Décombas | A new object based quality metric based on SIFT and SSIM[END_REF]. At a quality of SSIM=0.9, we reach a bitrate saving of 24% for the left object of coastguard and 10.7% for Stefan.

In future work, we will consider improving the Shift-map synthesis by exploiting the temporal aspects. Moreover, we will also investigate seam carving along the time axis.

Fig. 1 .

 1 Fig. 1. Architecture of the proposed semantic video coding using seam carving

Fig. 3 .

 3 Fig. 3. Temporal processing for saliency map

Fig. 4 .

 4 Fig. 4. Seams reordering : On the left, seams before reordering, on the right, seams reordered

Fig. 5 .

 5 Fig. 5. Definition of the group of seams

2 Fig. 6 .

 26 Fig. 6. Seams encoding by regrouping approach

Fig. 7 .

 7 Fig. 7. Example of seam synthesis V. RESULTSTo assess the performance of the proposed coding scheme, we use 2 CIF sequences with available binary segmentation masks [21]: coastguard and stefan. The sequences are temporally subsampled to 5 fps and we make our tests on 5 GOPs of 3 frames. 6 median seams with 3 key points are defined horizontally and vertically. The threshold of binarisation has been fixed at 0.75 for coastguard and 0.9 for stefan. These parameters have been empirically defined. The encoding is in full-Intra mode.With the proposed seam carving scheme, the process is iterated until touching an object in the control map. A size reduction of 41.6 % is reached for coastguard and 18.9% for stefan. In comparison, a size reduction of 17.8% was reached for coastguard and 10 % for stefan with the seam carving

Fig. 8 .Fig. 9 . 2 Fig. 10 .

 89210 Fig. 8. RD curves for coastguard -Object 1

Fig. 11 .

 11 Fig. 11. Synthesis results for coastguard. Left: Shift-map, Right: linear interpolation VI. CONCLUSION This paper describes a new semantic content-aware video coding scheme based on seam carving. Compared to our earlier work [2], we have introduced several new contributions leading to improved coding performance. More specifically, we have introduced a new energy map with a better temporal aspect, as well as a better combination of the saliency and gradient maps, which allows a better preservation of the salient objects. Groups of seams are now defined with a

TABLE I PERCENTAGE

 I OF BITRATE SAVED COMPARED TO H.264/AVC AS A FUNCTION

	OF QPI: (1) NO SEAM CODING, (2) OUR PROPOSED APPROACH, (3) TOTAL
			SEAM CODING, (4) APPROACH FROM [2]
			coastguard			stefan
	R		41.6%		17.8%		18.9%		10%
		(1)	(2)	(3)	(4)	(1)	(2)	(3)	(4)
	24	12314	12354	678394	A3214	773B4	77314 6A35B4	737C4
	27	11314	1D384	61A3C4	A3514	77374	75384 671324	53A84
	30	173A4	17374	6C53A4	C3594	77354	753C4 6D53B4	53A24
	33	153B4	D8394 67793A4	932C4	77354	75394 6153B4	53954
	36	DA394	DC3D4 6789394	B39A4	77314	753C4 62B374	53954
	39	D9314	D2374 617C354	13A94	773B4	753A4 6923A4	53274
	42	DD3A4	78314 6B1A3C4	73D94	773C4	75394 6883D4	53D84


Marc Decombas
email: marc.decombas@thalesgroup.com

Frederic Dufaux
email: dufaux@telecom-paristech.fr

Erwann Renan
email: erwann.renan@thalesgroup.com

Beatrice Pesquet-Popescu

Francois Capman
email: francois.capman@thalesgroup.com

Frederic Dufaux

Beatrice Pesquet-Popescu

Marc Décombas

Frédéric Dufaux

Béatrice Pesquet-Popescu

François Capman

A NEW OBJECT BASED QUALITY METRIC BASED ON SIFT AND SSIM

Keywords: Object-based metric, SSIM, SIFT, subjective evaluation

We propose a full reference visual quality metric to evaluate a semantic coding system which may not preserve exactly the position and/or the shape of objects. The metric is based on Scale-Invariant Feature Transform (SIFT) points. More specifically, Structural SIMilarity (SSIM) on windows around the SIFT points measures the compression artifacts (SSIM_SIFT). Conversely, the standard deviation of the matching distance between the SIFT points measures the geometric distortion (GEOMETRIC_SIFT). We validate our metric with subjective evaluation and reach a Spearman correlation of 0.86 for SSIM_SIFT and 0.74 for GEOMETRIC_SIFT.

INTRODUCTION

We consider semantic video coding techniques aiming at preserving the visual quality of salient objects which may undergo small displacements and geometric deformations. Such schemes are especially suited for security and monitoring applications. Examples include content-based coding methods based on seam carving [START_REF] Décombas | Seam carving for semantic video coding[END_REF] [START_REF] Tanaka | Improved image concentration for artifact-free image dilution and its application to image coding[END_REF].

In this paper, our goal is to develop a full reference object-based visual quality metric to evaluate a semantic coding system under the assumption that the position and the shape of objects may have been considerably modified.

Traditional fidelity metrics such as Peak-Signal-to-Noise-Ratio (PSNR) or Structural SIMilarity (SSIM) [START_REF] Wang | Image quality assessment: From error visibility to structural similarity[END_REF] compare corresponding pixels or blocks in the reference and processed images. Therefore, these approaches fail whenever geometric displacements or deformations occur.

In [START_REF] Wang | Translation Insensitive Image Similarity in Complex Wavelet Domain[END_REF], Wang and Simoncelli propose a complex wavelet domain image similarity measure that is insensitive to luminance change, contrast change and spatial translation. This metric is robust for small geometric distortions relative to the size of the wavelet filter. However, It does not handle large displacements, nor assesses geometric deformations.

Rubinstein et al. presents a subjective evaluation for image retargeting and intents to create an objective metric [START_REF] Rubinstein | A Comparative Study of Image Retargeting[END_REF]. Specific features are identified in retargeted media that are more important for viewers. They conclude that the resizing method having the best subjective score is also the one having the worst score with their objective metric. Therefore, a reliable metric remains a challenge.

A full reference metric based on Scale-Invariant Feature Transform (SIFT) [START_REF] Lowe | Distinctive image features from scale invariant keypoints[END_REF] and SSIM has been developed by Azuma et al. [START_REF] Azuma | SSIM based image quality assessment applicable to resized images[END_REF] in order to evaluate images resized by different retargeting algorithms. In [START_REF] Liu | Image Retargeting Quality Assessment[END_REF], Liu et al. present an objective metric simulating the Human Vision System (HVS) based on global geometric structures and local pixel correspondence based on SIFT.

The common objective of [7][8] is to evaluate resized (smaller) images. However, both methods are not designed to measure compression artifacts and do not take into account geometric deformations possibly occurring in content-based coding schemes (e.g. [1][2]). Another limitation is that both metrics compare two entire images. Therefore, they fail to assess the quality of a specific (salient) object. Finally, the metric in [START_REF] Azuma | SSIM based image quality assessment applicable to resized images[END_REF] has not been validated by subjective tests.

In this paper, we introduce an object-based visual quality metric by selecting and matching SIFT points in the object to evaluate. SIFT points allow to put in correspondence the same object in the reference and processed images even though it has been geometrically distorted. Our proposed metric gives two scores. The first one applies SSIM in the neighborhood of matching SIFT points and is referred to as SSIM_SIFT. Thus, it measures traditional compression artifacts such as those resulting from H.264/AVC (Advanced Video Coding). The second score measures the geometric deformation of the object. It is based on the standard deviation of matching SIFT points coordinates and is referred to as GEOMETRIC_SIFT. These two measures have been validated by subjective evaluation following the Double Stimulus Impairment Scale (DSIS) protocol [START_REF]Methodology for the subjective of the quality of television pictures[END_REF].

In summary, our main contribution is a metric that : (i) is object-based and not disturbed by distortions in the background, (ii) measures compression artifacts, (iii) measures geometric artifacts in the object, and (iv) is validated by subjective tests.

PROBLEM DEFINITION

For the purpose of object-based semantic coding, approaches based on seam carving have been proposed [START_REF] Décombas | Seam carving for semantic video coding[END_REF] [START_REF] Tanaka | Improved image concentration for artifact-free image dilution and its application to image coding[END_REF]. Seam carving is a method of resizing that suppresses or adds lines in non-salient parts of an image. Hereafter, we more specifically consider the method proposed in [START_REF] Décombas | Seam carving for semantic video coding[END_REF] without loss of generality. In this method, seam carving is applied as pre-and post-processing in conjunction with a conventional H.264/AVC video coding scheme, as illustrated in Fig. 1. Consequently, the method may introduce both traditional compression artifacts of H.264/AVC and geometric artifacts from the seam carving and synthesis. In this context, where salient objects are preserved but may undergo small displacements and deformations, traditional quality metrics such as PSNR or SSIM [START_REF] Wang | Image quality assessment: From error visibility to structural similarity[END_REF] fail.

PROPOSED OBJECT-BASED METRIC

In this paper, we propose a full reference metric to assess an object based coding system which has possibly modified the position and/or the shape of objects.

The metric relies on the combination of SIFT and SSIM to evaluate both compression artifacts and object deformations. SIFT [START_REF] Lowe | Distinctive image features from scale invariant keypoints[END_REF] is an approach for detecting and extracting local feature descriptors invariant to different changes, in particular rotation, scaling and, in general, geometric deformations. In the proposed metric, SIFT allows to match an object from the original image with a potentially deformed object in the processed image.

Figure 2 represents the proposed full-reference metric. It takes three images as input: the original image, the processed image #1 which has been altered by seam carving but without compression, and the processed image #2 which has been modified both by seam carving and compression.

In a first step, we extract SIFT points from the original image, as well as the processed image #1. This is done in order to avoid the sensitivity of SIFT to coding artifacts. For the reference image, we only select SIFT points inside the considered object.

Next, SIFT points matching is performed from the original image towards the processed image as well as from the processed image towards the original one to increase robustness. Finally, a statistical analysis is performed on the matching distances in order to eliminate outliers. This step is useful to indentify erroneously matched SIFT pairs. Formally, a pair of points is considered an outlier if the following equation holds σ µ 3

) ( > - i D with 1 = = N i i D N 1 ) ( 1 µ and 2 3 4 5 6 7 1 - - = = N i i D N 1 2 ) ) ( ( 1 1 µ σ
where N is the number of SIFT points, i =1,.., N denotes the index of the current SIFT point, D is the distance between a pair of matching SIFT points in the original and processed images respectively, 1 is the mean distance between the matching SIFT pairs, and 2 is the standard deviation. For GEOMETRIC_SIFT, we simply measure the standard deviation 2 between matched SIFT points. This component of the metric captures the non-rigid deformation of the object.

In turn, the SSIM_SIFT component of the metric assesses the visual content of the object. First, nonoverlapping W x W pixel windows are defined centered at each SIFT point and wholly contained inside the object. For this purpose, SIFT points with an associated window laying partly outside the object or with a spatial distance inferior to W pixels are discarded. The window dimension W=11 is chosen to cover enough of the surroundings.

Since SIFT points coordinates are not integer values, it can cause a mismatch of ±1 pixel, horizontally and/or vertically, when the window from the original image is compared with the window in the processed image. Thus, nine positions, representing all {-1, 0, 1} pixel shifts horizontally and vertically, are tested and the one with the minimal Mean Square Error (MSE) is kept. Finally, SSIM is applied on all the windows defined by the above process, leading to the SSIM_SIFT measure.

SUBJECTIVE EVALUATION PROTOCOL

Following the ITU-R BT.500-12 recommendation [START_REF]Methodology for the subjective of the quality of television pictures[END_REF], the protocol DSIS is chosen for subjective evaluation.

During the session, as a variation to standard DSIS, the assessor is first presented with a binary mask defining the object, then an unimpaired reference, and finally with the same picture impaired. At the beginning of each session, a training is given to the observers about the subjective assessment. In particular, assessors are specifically instructed to concentrate on the corresponding object. Afterwards, the assessor is asked to vote using the fivegrade impairment scale: 5 imperceptible, 4 perceptible, but not annoying, 3 slightly annoying, 2 annoying, 1 very annoying.

Each assessor evaluates 30 images altered with different levels of artifacts spanning a large range of visual quality. The five first images are used for training and corresponding scores are discarded. Subjective scores are then processed and analyzed according to [START_REF]Methodology for the subjective of the quality of television pictures[END_REF].

RESULTS

To validate our metric, we use the object-based compression method described in [START_REF] Décombas | Seam carving for semantic video coding[END_REF] to generate sequences presenting H.264/AVC compression artifacts, geometrical deformations and repositioning artifacts of the salient objects. Experiments are carried out using the test sequences Container and Coastguard in CIF format.

Performance of SSIM_SIFT

In a first set of experiments, we evaluate the performance of the proposed SSIM_SIFT, and in particular its ability to assess object-based visual quality in the presence of small displacements and deformations of the salient object.

Seam carving usually stops when it reaches objects defined by a saliency map. In this first experiment, seam carving parameters in [START_REF] Décombas | Seam carving for semantic video coding[END_REF] have been selected in order to achieve minor geometric distortions of the salient object. Nevertheless, a few seams may go through a salient object leading to artifacts. In addition, small deformations may also be introduced when reinserting seams during synthesis.

Figure 3 illustrates the proposed SSIM_SIFT metric. The Fig. 3 (a) and (b) show the SIFT windows (black squares) used to compute SSIM_SIFT in the original image and the processed image #1 (i.e. altered by seam carving but without H.264/AVC compression). In Fig. 3 (b), it can be observed that the container ship is well preserved, although the background is noticeably distorted. Moreover, the position and shape of the ship have been slightly altered. As a reference, SSIM_Mask is a straightforward extension of SSIM computed on a salient object as defined by its binary mask. More precisely, SSIM is only calculated on the 11 x 11 windows which are wholly contained in the binary mask.

To validate the proposed SSIM_SIFT metric, a subjective evaluation was done with 14 non-expert assessors following the procedure described in Sec. 4. Six images quantized with QP={18, 36, 39, 42, 48}, for a total of 30 images, were shown in a random order to each assessor. We have found no outliers among assessors when following the procedure defined in ITU-R BT.500-12 [START_REF]Methodology for the subjective of the quality of television pictures[END_REF].

Figure 4 shows the proposed SIFT_SSIM as a function of the Mean Opinion Score (MOS). The Spearman correlation is 0.86 and the Pearson correlation is 0.86 for the proposed SIFT_SSIM, showing a strong correlation. In comparison, the Spearman correlation is 0.20 and the Pearson correlation is 0.14 for SSIM_Mask. Clearly, SSIM_Mask fails as it cannot handle small geometric displacement or deformation.

Performance of GEOMETRIC_SIFT

We now evaluate the performance of the proposed GEOMETRIC_SIFT to measure object deformation. For this purpose, images with different levels of geometric deformation resulting from seam carving, but without compression artifacts (QP=0), are considered. A new evaluation with 11 assessors has been performed. During this evaluation, no assessor has been detected as an outlier. The result of the experiment is given in Fig. 5. The Spearman correlation is -0.74 and the Pearson correlation is -0.67.

Correlations are lowered due two images with poor performances.

The image corresponding to GEOMETRIC_SIFT=5.24 (right most point in Fig. 5) is shown in Fig. 6 (a). GEOMETRIC_SIFT is high, as the ship is elongated and has slightly moved as shown in Fig. 6 (b). However, as the artifact of translation and deformation is hard to notice, the MOS remains high. The image corresponding to GEOMETRIC_SIFT=0.46 and MOS=1.72 (lower left in Fig. 5) is shown in Fig. 7. The object of interest (the container ship) itself is wellpreserved, however the borders of the object have been strongly distorted. In such a case, the assessor may have evaluated the border region instead of the ship alone. This underlines one of the limitations of this evaluation. The assessor can be influenced by the background.

CONCLUSION

In this paper we present an object-based full reference visual quality metric based on SIFT and SSIM. It can be used for images where the objects have their position and/or shape modified. The two proposed components have been validated by a subjective evaluation following DSIS. SSIM_SIFT gives a Spearman and a Pearson correlation of 0.86. Evaluation of deformation artifacts with GEOMETRIC_SIFT gives a Spearman correlation of -0.74 and a Pearson Correlation of -0.67.

In future work, we will aim at including additional attributes and combining different components into a single overall quality score.

Fig. 1 :

 1 Fig. 1: Architecture of seam carving based video coding.

Fig. 2 :

 2 Fig. 2: Proposed object-based metric.

Fig. 3 :

 3 SIFT windows used to compute SSIM_SIFT (black squares), (a) original image, (b) processed image #1.

Fig. 4 :

 4 Fig. 4: SIFT_SSIM as a function of MOS.

Fig. 5 :

 5 Fig. 5: GEOMETRIC_SIFT as a function of MOS.

Fig. 6 :

 6 (a) Container, frame 18, container object, (b) difference between the original and processed images.

Fig. 7 :

 7 Fig. 7: Container, frame 23, container object.