Xiaoqin Wang
email: xiaoqin.wang@monash.edu

Y Ahmet

Member, IEEE Tom Drummond
email: tom.drummond@monash.edu

Member, IEEE Enrico Natalizio
email: enrico.natalizio@hds.utc.fr

Member, IEEE Isabelle Fantoni
email: isabelle.fantoni@hds.utc.fr

Member, IEEE Vincent Frémont
email: cent.fremont@hds.utc.fr

Y A S Wang

T ¸ekercioglu
email: ahmet.sekercioglu@monash.edu

Drummond

ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY (UNDER REVIEW) 1 Fast Depth Video Compression for Mobile RGB-D Sensors

Keywords: Depth map, motion compensation, depth video coding, inter-frame correlation

We propose a new method, called 3D Image Warping Based Depth Video Compression (IW-DVC), for fast and efficient compression of depth images captured by mobile RGB-D sensors.

The emergence of low-cost RGB-D sensors has created opportunities to find new solutions for a number of computer vision and networked robotics problems, such as 3D map building, immersive telepresence or remote sensing. However, efficient transmission and storage of depth data still presents a challenging task to the research community in these applications. Image/video compression has been comprehensively studied, and several methods have already been developed. But, these methods result in unacceptably suboptimal outcomes when applied to the depth images.

We have designed the IW-DVC method to exploit the special properties of the depth data to achieve a high compression ratio while preserving the quality of the captured depth images. Our solution combines the egomotion estimation and 3D image warping techniques and includes a lossless coding scheme which is capable of adapting to depth data with a high dynamic range. IW-DVC operates in high-speed, suitable for real-time applications, and is able to attain an enhanced motion compensation accuracy compared with the conventional approaches. Also, it removes the existing redundant information between the depth frames to further increase compression efficiency. Our experiments show that IW-DVC attains a very high performance yielding significant compression ratios without sacrificing image quality.

I. INTRODUCTION

R ECENTLY invented low-cost RGB-D sensors, such as Kinect [START_REF] Freedman | Depth Mapping Using Projected Patterns[END_REF] and PrimeSense Carmine [START_REF]PrimeSense: 3D Sensing Technology Solutions[END_REF], have attracted immense interest from the computer vision, multimedia, and robotics communities [START_REF] Hitomi | 3D Scanning Using RGB-D Imaging Devices: A Survey[END_REF]. By integrating RGB cameras and infrared sensing, RGB-D sensors provide synchronized color and depth images at high frame rates. The complementary nature of the depth and visual information makes them suitable for applications in various research fields, including three-dimensional (3D) mapping and localization [START_REF] Henry | RGB-D Mapping: Using Kinect-Style Depth Cameras for Dense 3D Modeling of Indoor Environments[END_REF], [START_REF] Endres | 3-D Mapping with an RGB-D Camera[END_REF], or scene/object reconstruction [START_REF] Newcombe | KinectFusion: Real-Time Dense Surface Mapping and Tracking[END_REF], [START_REF] Kim | Occupancy Mapping and Surface Reconstruction Using Local Gaussian Processes with Kinect Sensors[END_REF]. In these applications, a mobile RGB-D sensor is used to observe the texture and geometrical structure information of a static scene/object from different viewpoints. Since such applications generate large amount of visual and depth information, which usually needs to be remotely accessed, an efficient compression algorithm is necessary to minimize the memory utilization and communication bit rate as much as possible.

Data compression for visual information is now a wellestablished technology. There are numerous lossless and lossy compression algorithms for image and video applications, JPEG 2000 [START_REF] Taubman | JPEG2000: Image Compression Fundamentals, Standards and Practice[END_REF] and H.264/AVC [START_REF] Wiegand | Overview of the H. 264/AVC Video Coding Standard[END_REF] being the most prominent ones. The coding of depth images, however, is a recent research topic. A depth image, representing the relative distance from the recording camera to an object in the 3D space, usually consists of smooth regions and sharp edges at boundaries between the object and background. A typical way to compress a depth image sequence is by processing each frame as a standard gray scale image for viewing purposes and applying the standard video coding schemes. Redundancy between successive frames can be removed by estimating the motion between frames and then generating the motion vectors (MVs), which describe the motion of the pixel information repeatedly shown in successive frames. In standard video coding techniques, 2D block matching algorithms [START_REF] Puri | An efficient Block-Matching Algorithm for Motion-Compensated Coding[END_REF] are used for motion estimation, in which frames are divided into blocks of M ⇥ N pixels, such as 16 ⇥ 16 and 8 ⇥ 8. A search is performed to find a matching block from a frame i in some other frame j. However, this approach is unfortunately very suboptimal for depth image sequences. Large, homogeneous areas on the surface of an object can be divided into small blocks and the sharp discontinuities at object boundaries can be placed into the same block. Because of this, these schemes result in significant coding artifacts along the depth discontinuities in the reconstructed depth images, especially when the compression ratio is high [START_REF] Krishnamurthy | Compression and Transmission of Depth Maps for Image-Based Rendering[END_REF], [START_REF] Chai | Depth Map Compression for Real-Time View-Based Rendering[END_REF].

The above mentioned shortcomings of the standard image and video coding algorithms are the driving force for concentrating the research efforts in developing new data compression schemes by considering the specific properties of depth images. The schemes can be broadly classified under two categories: Approaches developed to remove the (i) temporal redundancies [START_REF] Grewatsch | Sharing of Motion Vectors in 3D Video Coding[END_REF], [START_REF] Hewage | A Novel Frame Concealment Method for Depth Maps Using Corresponding Colour Motion Vectors[END_REF], [START_REF] Daribo | Motion Vector Sharing and Bitrate Allocation for 3D Video-Plus-Depth Coding[END_REF], [START_REF] Shahriyar | Inherently Edge-Preserving Depth-Map Coding Without Explicit Edge Detection and Approximation[END_REF], [START_REF] Kamolrat | 3D Motion Estimation for Depth Image Coding in 3D Video Coding[END_REF], [START_REF] Fan | Three-Dimensional Depth Map Motion Estimation and Compensation for 3D Video Compression[END_REF], [START_REF] Nguyen | Efficient Techniques for Depth Video Compression Using Weighted Mode Filtering[END_REF], or (ii) interview redundancies [START_REF] Zhang | Joint Multiview Video Plus Depth Coding[END_REF], [START_REF] Schwarz | Overview of the Scalable Video Coding Extension of the H.264/AVC Standard[END_REF], [START_REF] Bilen | A Multi-View Video Codec Based on H.264[END_REF], [START_REF] Ekmekc ¸ioglu | A Temporal Subsampling Approach for Multiview Depth Map Compression[END_REF], [START_REF] Lee | A Fast and Efficient Multi-View Depth Image Coding Method Based on Temporal and Inter-View Correlations of Texture Images[END_REF], [START_REF] Fehn | Depth-Image-Based rendering (DIBR), Compression, and Transmission for a New Approach on 3D-TV[END_REF], [START_REF] Lee | Efficient Multiview Depth Video Coding Using Depth Synthesis Prediction[END_REF], [START_REF] Mark | Post-Rendering 3D Image Warping: Visibility, Reconstruction and Performance for Depth-Image Warping[END_REF], [START_REF] Wang | Block-Based Depth Maps Interpolation for Efficient Multiview Content Generation[END_REF] in depth video images. The starting point of all these schemes is 2D block matching algorithms. In the following paragraphs we provide a brief overview of these methods.

The method proposed in [START_REF] Grewatsch | Sharing of Motion Vectors in 3D Video Coding[END_REF] exploits the correspondences between the depth images and the corresponding color frames captured by a texture camera. It adopts a conventional block matching approach to determine the MVs according to the texture information. The MVs from the texture information are considered to be encoding both the texture and depth image sequences. A number of techniques [START_REF] Daribo | Motion Vector Sharing and Bitrate Allocation for 3D Video-Plus-Depth Coding[END_REF], [START_REF] Kamolrat | 3D Motion Estimation for Depth Image Coding in 3D Video Coding[END_REF], [START_REF] Fan | Three-Dimensional Depth Map Motion Estimation and Compensation for 3D Video Compression[END_REF], [START_REF] Nguyen | Efficient Techniques for Depth Video Compression Using Weighted Mode Filtering[END_REF] are developed based on this concept and provide enhanced performances. Daribo et al. [START_REF] Daribo | Motion Vector Sharing and Bitrate Allocation for 3D Video-Plus-Depth Coding[END_REF] proposed a MV sharing algorithm based on the correlation between the motion of the texture and of the depth. This algorithm considers the motion of a block at the same coordinates in both video texture and depth images and uses a joint distortion criterion to generate common MVs for both texture and depth. Shahriyar et al. [START_REF] Shahriyar | Inherently Edge-Preserving Depth-Map Coding Without Explicit Edge Detection and Approximation[END_REF] proposed an inherently edge-preserving depth-map coding scheme. Their scheme uses the texture motion vectors, avoids distortion on edges, and accurately preserves the depth information on edges. Fan et al. [START_REF] Fan | Three-Dimensional Depth Map Motion Estimation and Compensation for 3D Video Compression[END_REF] proposed a motion estimation method with various block sizes. It first determines the block size and its corresponding MV using the color information. Then, a z-direction motion estimation is used to correct the depth values in each block. A similar algorithm, proposed in [START_REF] Kamolrat | 3D Motion Estimation for Depth Image Coding in 3D Video Coding[END_REF], replaces the 2D block matching algorithm with a 3D one operating in horizontal, vertical, and depth dimensions. The method by Nguyen et al. [START_REF] Nguyen | Efficient Techniques for Depth Video Compression Using Weighted Mode Filtering[END_REF] compresses the depth video by using a weighted mode filter to suppress the coding artifacts.

In addition to the temporal redundancy in the frames captured by a single camera at different times, inter-view redundancy also exists in the frames captured from various viewpoints in a multi-camera system. Many research studies have been proposed to remove these inter-view correlations and achieve efficient compression for depth video in multiview scenarios. The study by Zhang et al. [START_REF] Zhang | Joint Multiview Video Plus Depth Coding[END_REF] combines multiview video coding and the prediction of MVs in the depth image sequence from those of the texture image sequence. Ekmekc ¸ioglu et al. [START_REF] Ekmekc ¸ioglu | A Temporal Subsampling Approach for Multiview Depth Map Compression[END_REF] proposed a coding scheme in which the bit rate used for multi-view depth image coding is further reduced by skipping one or more temporal layers of selected depth image views. The method proposed by Lee et al. [START_REF] Lee | A Fast and Efficient Multi-View Depth Image Coding Method Based on Temporal and Inter-View Correlations of Texture Images[END_REF] reduces the bit rate by skipping depth blocks under consideration of temporal and inter-view correlations of texture images. The temporal and inter-view correlations are measured from the temporally successive pictures and the neighboring views synthesized through pixel-by-pixel mapping respectively. Lee et al. [START_REF] Lee | Efficient Multiview Depth Video Coding Using Depth Synthesis Prediction[END_REF] describe a multi-view depth video coding scheme that incorporates depth view synthesis, H.264/MVC, and additional prediction modes. A depth image-based rendering technique [START_REF] Mark | Post-Rendering 3D Image Warping: Visibility, Reconstruction and Performance for Depth-Image Warping[END_REF] is adopted in this approach to generate an additional reference depth image for the current viewpoint. The interview correlation is exploited by using a texture and depth view synthesis approach. A block-based depth image interpolation approach was proposed by Wang et al. [START_REF] Wang | Block-Based Depth Maps Interpolation for Efficient Multiview Content Generation[END_REF]. In this scheme, the first and last frames in a texture video are treated as the key frames with known depth images. The remaining depth images corresponding to other texture frames can be recovered by the proposed bidirectional prediction algorithm.

To the best of our knowledge, these represent the most relevant works related to the depth video coding in which the characteristics of depth images are preserved and the temporal and inter-view correlations are exploited. However, all of these . . algorithms focus on compressing the depth images captured by cameras that remain in a fixed position. The situation becomes inevitably much more complicated when moving cameras involved. As the distance between a moving camera and the objects in a scene changes across time, depth values of the same objects change in successive depth frames. Therefore, the coding schemes with motion compensation methods for static cameras become very inaccurate or even become useless with mobile cameras.

In this paper, we consider typical situations in visual-SLAM (simultaneous localization and mapping) or surface reconstruction applications in which a mobile RGB-D sensor collects depth and visual information of a static environment. As the RGB information can be compressed easily by conventional image/video coding schemes, we only focus on developing a fast and efficient coding method for the depth video. An overview of the new scheme we propose, called 3D Image Warping Based Depth Video Compression (IW-DVC), is shown in Fig. 1.

We first eliminate the redundant depth data at the encoder side (Fig. 1(a)). Then the bitstream is further compressed to maximize its storage and transmission efficiency. For the elimination of redundant depth data, instead of using conventional motion compensation algorithms with 2D block-based matching, we propose a novel method which combines egomotion estimation of a mobile RGB-D sensor and 3D image warping technique [START_REF] Fehn | Depth-Image-Based Rendering (DIBR), Compression, and Transmission for a New Approach on 3D-TV[END_REF]. This approach enables better prediction of frames, and so leading to an enhanced compression performance by taking advantage of the characteristic properties of depth images. Additionally, this motion compensation method is computationally efficient and can operate in real-time, since it does not need to derive the mean squared error (MSE) block by block, and it further can be used as a front-end for other depth image coding schemes to remove the redundancy between consecutive frames prior to further coding.

We have also included our lossless compression scheme, designed particularly for depth images captured by RGB-D sensors, called Differential Huffman Coding with Multiple Lookup Tables (DHC-M). In IW-DVC framework, each Iframe is encoded using DHC-M directly. DHC-M is also applied on each P-frame after the motion compensation and performs entropy coding allowing for higher compression ratios.

At the decoder side, crack-filling algorithms are adopted to deal with the under-sampling problem [START_REF] Zhu | 3D-TV System with Depth-Image-Based Rendering: Architectures, Techniques and Challenges[END_REF] in the reconstructed depth images.

The proposed framework can be implemented directly on a mobile RGB-D sensor system to achieve efficient depth video storage and transmission. The received depth information can be used for many applications, such as visual mapping and exploration, 3D scene reconstruction, or free viewpoint video rendering. The main accomplishments of the study presented in this paper can be summarized as:

• Theoretical development and practical implementation of the first motion compensation algorithm for depth video captured by a mobile sensor/camera, • Thorough performance evaluation of the proposed framework under the various parameter changes to the system, and • Extensive experiments using multiple datasets to evaluate the coding performance under a variety of scenes. The rest of the paper is organized as follows. In Section II, the mechanism of the RGB-D sensor is presented and the 3D image warping based depth video compression (IW-DVC) framework is explained. Experimental results are presented and analyzed in Section III, followed by our concluding remarks.

II. 3D IMAGE WARPING BASED DEPTH VIDEO COMPRESSION FRAMEWORK

Given a mobile RGB-D sensor which uses depth sensing to explore and map an environment, significant redundancy exists in the captured depth video, as the same surface geometrical information repeatedly appears in the consecutive depth frames, especially at high frame rates. Our objective is to prevent the same depth information from being unnecessarily transmitted and stored by removing this redundancy as much as possible. A general description of the framework we proposed is shown in Fig. 1. The list of symbols used through the paper is given in Table I.

A brief overview of the system is as follows. Let Z I and Z P denote an I-frame and P-frame in a group of pictures (GoP) in captured depth video (Table I can be referred for the descriptions of the mathematical symbols used throughout the paper). The system consists of two main components: motion compensation algorithm and a lossless coding scheme. Before encoding Z P into a bitstream, the differences between Z P and Z I should be determined in motion compensation first. The first step of the encoding procedure (Fig. 1(a)) is the interframe motion estimation. We estimate the motion of a moving sensor in the time interval of capturing depth frames Z I and Z P . In the second step, ẐP , a prediction of Z P is generated by using the interframe motion information, forward estimation/reverse check, and block-based update. This procedure estimates the newly observed depth information in Z P but not in the reference frame Z I . Then, only the newly observed information in Z P is encoded using an entropy coding scheme. Current pixel/reference pixel with the invalid depth value (2047 for the sensor we use).

V i , V i 1
Current pixel/reference pixel with a valid depth value. q h () Conditional probability distribution when the reference pixel has the invalid depth value.

qv()

Conditional probability distribution when the reference pixel has a valid depth value. As a result, the redundancy in the depth video is taken out during the encoding process.

In the decoding process, the received bitstream is decoded with the same codec used at the encoder side. I-frames can be directly decoded from bitstream. Each P-frame has to be reconstructed using the decoded newly observed information (intra blocks) in each P-frame and its corresponding Iframe. Then, a crack-filling approach is used to deal with the under-sampling issue [START_REF] Mark | Post-Rendering 3D Warping[END_REF] and enhance the quality of the reconstructed P-frames. Detailed explanation of each step is described in the next section preceding with a very brief overview of the RGB-D sensor.

A. RGB-D Sensor Mechanism

Microsoft Kinect (shown installed on WSRNLab's [START_REF]Wireless Sensor and Robot Networks Laboratory (WSRNLab)[END_REF] experimental robot eyeBug in Fig. 2), is one of the most popular RGB-D sensors. It contains an RGB camera and an IR projector-camera pair, which can produce color and depth images at a rate of 30 fps. As described by its inventors [START_REF] Freedman | Depth Mapping Using Projected Patterns[END_REF], depth is measured via a triangulation process that is based on the detection of transverse shifts of local dot patterns in the IR speckle with respect to its reference patterns at a known distance to the device. This process is repeated for all local regions in the IR speckle and produces a disparity-based depth image. The default RGB video stream provided by Kinect is in 8-bit VGA resolution (640 ⇥ 480 pixels) and the monochrome depth video stream is also in VGA resolution with 11-bit depth, which provides 2,048 levels of sensitivity. If the Kinect cannot determine the distance to a point, it denotes this with an invalid depth value.

Note that all the depth images presented in this paper are scaled down to 8 bits per pixel for the visual inspection purposes.

B. Interframe Motion Estimation

Conventional 2D block-based motion estimation algorithms use 2D block matching approach to estimate the MVs which can map the pixels from the reference frame to the current frame. However, this kind of approaches relies on the assumption that the pixels representing the visual information on the same object's surface stays unchanged even if the object or the camera is in motion. Therefore, it is clear that these 2D blockbased motion estimation approaches are not optimal for depth videos as the depth frames represent the distance of objects within a scene relating to the camera position and the depth values change when the camera or object is moving.

Taking advantage of the depth image characteristics, the depth pixels in the reference frame can be mapped to the current frame. Consider the vector p e = [x y z 1] T which represents a real world point in Euclidean space by using homogeneous coordinates. Given the intrinsic parameters of an RGB-D sensor: principal point coordinates (i c , j c ) and focal length of the camera (f x , f y ), p e can be estimated from the corresponding pixel in depth image by using the pinhole camera model as

p e ⌘ 1 z ⇥ x y z 1 ⇤ T ⌘ h i ic fx j jc fy 1 1 z i T , (1)
where (i, j) denotes the pixel coordinates of this real world point projection in the depth image, and z is the corresponding depth value reported by the camera.

In the discussion that follows, we assume that p e can be observed in Z I and Z P captured by the mobile RGB-D sensor, and the projections of p e are located at pixel coordinates (i I , j I ) and (i P , j P ) on the depth frames Z I and Z P , respectively. Also, under the assumption that the world coordinates system is equivalent to the mobile sensor coordinate system, the depth pixel (projection) at (i I , j I ) in Z I can establish a relationship between the depth pixel at (i P , j P ) in Z P as follows, h i P ic fx j P jc fy

1 1 z P i T = M h i I ic fx j I jc fy 1 1 z I i T
(2) and, to simplify the equation, by doing some rudimentary algebraic substitutions we obtain following equation in inverse depth coordinate,

[u P v P 1 q P ] T = M [u I v I 1 q I ] T . (3)
The transformation matrix M represents the 6DoF motion model, which describes the motion of the sensor and the transformation of the structure between a pair of depth images. It has the form

M =  R t 0 0 0 1 , ( 4
)
where R is a 3 ⇥ 3 rotation matrix and t is a 3 ⇥ 1 translation vector. Therefore, if we have the accurate information about the transformation matrix M, each pixel in the reference depth frame can find its corresponding pixel in the current depth frame.

For RGB-D sensors, several approaches for estimation of M can be found in the research literature. Their comparative evaluation, and our proposed method for M estimation can be found in [START_REF] Wang | A Real-Time Distributed Relative Pose Estimation Algorithm for RGB-D Camera Equipped Visual Sensor Networks[END_REF], [START_REF] Wang | Vision-Based Cooperative Pose Estimation for Localization in Multi-Robot Systems Equipped with RGB-D Cameras[END_REF]. Our approach is a variant of the Iterative Closest Point (ICP) algorithm [START_REF] Besl | A Method for Registration of 3-D Shapes[END_REF] which has enhanced performance in circumstances with heavy occlusion. Even though this algorithm is initially developed for estimating the relative pose between two RGB-D sensors, it can also estimate a single sensor's motion during the time period for capturing a pair of depth frames. It estimates M through explicit registration of surface geometries extracted from two depth frames. The registration problem is approached by iteratively minimizing a cost function whose error metrics are defined based on the bidirectional point-to-plane geometrical relationship. Suppose that correspondences between N = N I + N P 1 pairs of points from two depth images Z I and Z P , are established, we can then estimate the transformation matrix M by minimizing the bidirectional point-to-plane error metric, C, expressed in normal least squares form as

C = N I X l=1 h w l,I (Mp l I p l ⇤ P ) • ñl ⇤ ,P i 2 + N P X l=1 h w l,P (M 1 p l P p l ⇤ I ) • ñl ⇤ ,I i 2 , (5)
where p l I and p l P are the sampled points in depth frames Z I and Z P , p l⇤ P and p l⇤ I are their corresponding points respectively. The variables w l,I and w l,P are weight parameters for correspondences established in opposite directions between pairs. Also, ñl ⇤ ,P and ñl ⇤ ,I are the surface normals at the corresponding points p l ⇤ P and p l ⇤ I in real world coordinates, and

ñl ⇤ ,P = ⇥ ↵ l ⇤ ,P l ⇤ ,P l ⇤ ,P 0 ⇤ T , (6)
ñl ⇤ ,I = ⇥ ↵ l ⇤ ,I l ⇤ ,I l ⇤ ,I 0 ⇤ T . (7)
The cost function presented in Eq. 5 consists of two parts: 1) the sum of squared distances in the forward direction from depth images Z I to Z P , and

2) the sum of square distances in the backward direction from Z P to Z I . However, to estimate the 6DoF motion that minimizes Eq. 5 would require partial derivatives of M to be obtained. In order to make this problem solvable, we simplify Eq. 5 into the following form

C = N I X l=1 ⇥ w l,I (Mp l I p l⇤ P ) • ñl ⇤ ,P ⇤ 2 + N P X l=1 h w l,P (Mp l ⇤ I p l P ) • ñl,P i 2 , (8)
We can then estimate M by re-weighting the least squares operation in an ICP framework. Details of the criteria for selecting the weight function can be found in [START_REF] Wang | A Real-Time Distributed Relative Pose Estimation Algorithm for RGB-D Camera Equipped Visual Sensor Networks[END_REF]. The weight function chosen for forward direction is

w l,I = ( c/ ⇥ c + (z l⇤ P z l I ) ⇤ if z l I  z l⇤ P c/ ⇥ c + (z l⇤ P z l I ) 2 ⇤ otherwise, (9)
where z l I is the depth value of the sampled point p l I , z l⇤ k is the depth value of the corresponding point p l⇤ P , c is the mean of deviation between depth values of sampled points and depth values of corresponding points. Likewise, the weight function used in the backward direction is

w l,P = ( c/ ⇥ c + (z l⇤ I z l P ) ⇤ if z l P  z l⇤ I c/ ⇥ c + (z l⇤ I z l P ) 2 ⇤ otherwise. ( 10
)
Algorithm 1 Inter-frame motion estimation procedure An overview of the entire process is presented in Algorithm 1. In this coarse-to-fine algorithm, each iteration generates an update E to the sensor's pose which modifies the transformation matrix M. E takes the same form as M which may be parameterized by a 6-dimensional motion vector having the elements ↵ 1 , ↵ 2 , . . . , ↵ 6 via the exponential map and their corresponding group generator matrices G 1 , G 1 , . . . , G 6 as Here G 1 , G 2 and G 3 are the generators of translations in x, y and z directions, while G 4 , G 5 and G 6 are rotations about x, y and z axes respectively. Afterwards, the task becomes finding the ↵ 1 , . . . , ↵ 6 that describe the inter-frame transformation. Based on Eq. 3, through determining the partial derivatives of u P , v P and q P with respect to the elements of the motion vector ↵ 1 , . . . , ↵ 6 , the Jacobian matrix for each established corresponding point pair can be obtained from

E = exp 0 @ 6 X j=1 ↵ j G j 1 A , (11)
J = 2 4 q I 0 u I q I u I v I 1 + u 2 I v I 0 q I v I q I 1 v 2 I v I u I u I 0 0 q 2 I v I q I u I q I 0 3 5 . ( 12
)
The six-dimensional motion vector, which minimizes Eq. 5, is then determined iteratively by the least squares solution

B = (K T WK) 1 K T WY (13)
in which W is a diagonal matrix weighting the bidirectional point-to-plane correspondences based on Eq. 8. B, Y, and K are matrices

B = 2
K = ⇥ ñ0 1 ⇤ ,P J1 ... ñ0 N ⇤ I ,P JN I ñ0 1,P J1⇤ ... ñ0 N P ,P J N ⇤ P ⇤ T (14)
Here, ñ0 l,P = ⇥ ↵ l,P l,P l,P ⇤ T is the surface normal expressed in a slightly different form than the one shown in Eqs. 6 and 7. To detect the convergence of our algorithm, we use the thresholds for the ICP framework presented in [START_REF] Lui | Robust Egomotion Estimation Using ICP in Inverse Depth Coordinates[END_REF]. The required number of iterations mostly depends on the motion between two frames. If the motion is small, only a few iterations is required. Otherwise, a large number of iterations is needed. As the speed of the camera is not high, algorithm usually converges in a few iterations. Once the algorithm converges, the registration is considered as completed and the M is determined.

C. Forward Estimation/Reverse Check and Block-based Update

After determining M, the correspondences between the depth pixels in Z I and depth pixels in Z P are established by forward estimation/reverse check and block-based update procedures. They are explained in this section. 1) Forward Estimation: In the forward estimation, we try to generate a depth frame ẐP which is the prediction of the depth frame Z P . We first initialize the pixels in ẐP to the invalid depth value. Then, according to Eq. 2, each pixel in depth frame Z I is warped to a coordinate in ẐP . In this process, it can happen that two or more different depth pixels are warped to the same pixel coordinate in ẐP . This oversampling issue happens because some 3D world points are occluded by the other ones at the new viewpoint. We adopt Z-buffer [START_REF] Mori | View Generation with 3D Warping Using Depth Information for FTV[END_REF] to solve this problem. By contrast, some regions in ẐP may have no valid depth information as none of the pixels in Z I can be warped to these regions. We name these as hole regions and pixels in them have invalid depth values. They indicate that some depth information in Z P cannot be predicted from Z I Therefore, the regions in Z P with the same locations of hole regions in ẐP are assumed to be containing the newly observed depth information at time P . An example of this process is shown in Fig. 3. In this example, the regions containing the depth information that can only be observed by Z P are outlined in yellow lines.

2) Reverse Check: Though the forward estimation can detect the newly observed depth information in the current frame Z P in most cases, it may fail to operate correctly in the situations when some points are occluded by the objects that can only be seen in Z P . A typical scenario is shown in Fig. 4. In this example, as object A cannot be observed in I-frame, the forward estimation will falsely treat the background in red as the surface that can be observed in Z P . However, the surface of object A in green is observed in Z P instead of the red background surface. Therefore, the forward estimation cannot accurately determine the newly observed depth information of Z P in this case.

In order to solve this problem, we introduce a reverse check mechanism. Similar to the warping process in the forward in the reverse check process the pixels on depth frame Z P can be warped to generate a depth frame, ẐI , which is captured I-frame's viewpoint virtually. The warping process in the reverse check can be described as h i I ic fx j I jc fy

1 1 z I i T = M 1 h i P ic fx j P jc fy 1 1 z P i T . ( 16
)
Pixel at (i P , j P ) in Z P can be mapped to (i I , j I ) in ẐI . In this process, the pixels representing the range information of the green surface on object A will move out of the image coordinate range and will not be shown in ẐI . Thus we need to find the pixels in Z P that move out of the image coordinate range in the reverse check process to complete the newly observed information determined by forward estimation.

3) Block-Based Update: In order to identify the newly observed depth information in the current depth frame, forward estimation and reverse check mechanisms are used in combination.

First, based on the transformation matrix M, the forward estimation (Eq. 2) generates a virtual depth frame ẐP . Second, with the reverse check mechanism, ẐI is generated from Z P (Eq. 16). We record the coordinates of the pixels in Z P which are warped out of the image coordinate range of ẐI . Third, we use these recorded coordinates to set the values of the corresponding pixels in ẐP to holes.

Then the virtual depth frame ẐP and captured frame Z P are uniformly subdivided into 8⇥8 pixels macro blocks. We search for the blocks in ẐP with the number of pixels with invalid depth values which are above a pre-determined threshold 2 . Blocks in ẐP can then be classified into two groups as follows if (number of pixels with invalid depth values in the block) < threshold then "Skip block": ZI has sufficient data to predict the depth information in the block with the same coordinate in ZP . else "Intra block": The block with the same coordinate in frame ZP contains the depth information that ẐP does not have, and consequently cannot be predicted from ZI . The depth information in the block with the same coordinate in ZP should be included in the encoding process. end if

D. Differential Huffman Coding with Multiple Lookup Tables (DHC-M)

By using the process explained in Section II-C, we can extract the newly observed depth information from every Pframe in a depth video. However, there is still further room for improvement. As the depth frames usually contain a large number of smooth regions, there is generally a high degree of correlation between adjacent pixels. So, we can assert that a high degree of pixel-to-pixel correlation will result in a high compression ratio in differential coding. As the accurate information in keyframes [START_REF] Klein | Improving the Agility of Keyframe-Based SLAM[END_REF] is the prerequisite of successful operating many applications [START_REF] Henry | RGB-D Mapping: Using Kinect-Style Depth Cameras for Dense 3D Modeling of Indoor Environments[END_REF], such as SLAM and 3D reconstruction, we need to keep the full fidelity of these frames which are usually the I-frames in a GoP. Therefore, we propose a lossless coding scheme to encode complete I-frames and only the newly observed information in P-frames. We designed our coding scheme, called Differential Huffman Coding with Multiple Lookup Tables (DHC-M), to be fast and capable of compressing the depth images in a lossless manner without introducing any artificial refinements. DHC-M scheme can be applied on complete depth frames or individual blocks.

The DHC-M scheme operates by comparing the current depth pixel with its reference pixel (which is the neighboring pixel of the pixel under consideration). If a pixel is at the end of a row, then it is treated as the reference pixel of the pixel at the same column under it (see Fig. 5). difference between the values of a pixel and its reference pixel is used in the encoding process. However, the depth image captured by RGB-D sensors has a portion of pixels that have invalid depth value (holes). invalid value is set at 2047 which is the largest number can be represented by 11bit data. This characteristic make the difference between the current and reference pixels have a high dynamic range. If we directly apply the standard entropy coding on the difference values, the compression ratio will be very limited. It is because the probabilities of difference values are distributed in a very wide range. In order to efficiently encode these difference values, we centralize the probability distribution by classifying the difference values into two groups according to the value of the reference pixel and using two different lookup tables to encode the differences.

Let X i represent the current pixel. If it has a valid depth value, we use the symbol V i , or H i , if it is a hole. Similarly for its reference pixel, we use the symbols either V i 1 or H i 1 depending on whether it is a hole or not.

The first lookup table is used for the cases when a reference pixel is a hole, and its values are generated according to the conditional probabilities as follows

p(X i |H i 1 ) = ( p(V i |H i 1 ) if the current pixel is valid p(H i |H i 1 ) otherwise, (17)
where p(V i |H i 1 ) = q h (V i ), and p(H i |H i 1 ) = q h (H i H i 1 ). The second lookup table is for the cases that where the reference pixel has a valid depth value, and is established based on the conditional probabilities as follows

p(X i |V i 1 ) = ( p(V i |V i 1 ) if the current pixel is valid p(H i |V i 1 ) otherwise, (18)
where p(V i |V i 1 ) = q v (V i V i 1 ), and p(H i |V i 1 ) = q v (H i ). We use the standard technique of Huffman coding [START_REF] Huffman | A method for the construction of minimum-redundancy codes[END_REF] to generate two lookup tables based on the probability distributions of q h () and q v (). Probability distributions of q h () and q v () are determined empirically by collecting information over a number of representative depth images which are captured in scenarios with varying geometrical structures.

In the encoding stage, the scheme first determines whether a reference pixel is a hole. If it is, the first lookup table (generated by using Eq. 17) is used to determine the codeword of the difference value. Otherwise, the second lookup table (Eq. 18) is used instead.

E. Decoding Process and Under-Sampling Problem

At the decoder side, the received bitstream is decoded with the same lookup tables used at the encoder side. In the decoding process, if the reference pixel is a hole, the first lookup table is used to decode the current pixel, otherwise the second table is used. As no motion compensation is applied on I-frame, every I-frame can decoded losslessly. While after motion compensation, only some blocks of each P-frame are transmitted with the transformation matrix. Therefore, each P-frame is lossy encoded and need to be decoded using its corresponding I-frame and transformation matrix besides transmitted blocks. We first apply the transformation matrix on each pixel in I-frame (use Eq. 2) to generate the prediction of the P-frame, ẐP . Then, we paste the transmitted blocks on ẐP . Eventually, P-frame is reconstructed at decoder side.

Directly applying warping equations may cause some visual artifacts in the synthesized view, like disocclusions and cracks. Some remarkable works [START_REF] Zhang | Stereoscopic image generation based on depth images for 3D TV[END_REF], [START_REF] Oh | Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-D video[END_REF], [START_REF] Zhu | A New Perspective on Hole Generation and Filling in DIBR Based View Synthesis[END_REF], [START_REF] Mori | View Generation with 3D Warping Using Depth Information for FTV[END_REF] have been proposed to reduce these adverse effects, especially the effects of disocclusions. In our system, the disocclusions effects are fixed using information of original P-frames in block-based update process. Here, we only need to deal with the cracks which generated due to the under-sampling problem. Undersampling occurs when the warping process "rotates" a surface in such a way that the viewing angle gets closer to the normal or "reduces" the distance between the sensor and the surface (Fig. 6). In such situations, the original image does not have enough information to predict the same surface from the viewpoint of the generated image.

A distinguishing feature of the pixels in cracks is that they have invalid depth value and their neighboring pixels usually have valid depth values. There are two general approaches to fill the cracks. The first approach is to use the value of the nearest non-hole pixel in the left/right side to fill the pixels in the cracks [START_REF] Mark | Post-Rendering 3D Warping[END_REF], [START_REF] Lee | A Fast and Efficient Multi-View Depth Image Coding Method Based on Temporal and Inter-View Correlations of Texture Images[END_REF]. This method is inaccurate and not robust for scenes that contain objects with complex geometries and variable camera motions. The second approach uses a median filter to smooth the whole images [START_REF] Mori | View Generation with 3D Warping Using Depth Information for FTV[END_REF], [START_REF] Oh | Hole filling method using depth based in-painting for view synthesis in free viewpoint television and 3-D video[END_REF]. This approach exhibits good performance on filling the cracks, but it smoothes the complete image and introduces noise in regions with correct depth values, especially on the object boundaries. In order to avoid this adverse effect, we have modified it by using an adaptive median filter. The filter is only applied on the pixels with invalid depth value instead of the whole image. Detailed performance evaluation of these three crack-filling algorithms is presented in Section III-B2.

III. PERFORMANCE EVALUATION

We have implemented the IW-DVC framework in C++ using libCVD [START_REF]libCVD -Computer Vision Library[END_REF], OpenCV [START_REF]OpenCV: Open Source Computer Vision Library[END_REF], and OpenKinect [START_REF]OpenKinect Library[END_REF] libraries on a personal computer with an Intel i7-M620 2.66 GHz processor and 4GB memory. In order to evaluate its performance we have conducted a series of experiments by using datasets selected archive provided by the Computer Vision Group in Technical University of Munich [START_REF] Sturm | A Benchmark for the Evaluation of RGB-D SLAM Systems[END_REF]. All datasets in this archive consist of color and depth images captured by a mobile RGB-D sensor. The datasets also provide the intrinsic parameters of the RGB-D sensor used. All depth images were captured at a rate of 30 fps (full frame rate) with a resolution of 640 ⇥ 480 pixels. The pixels are trained to represent the distance in meters. For our experiments, we have selected the data sets containing images captured over static scenes with sufficiently rich geometrical features. The archival names of the selected datasets are included in Table II. Before conducting the experiments, we have converted the depth images in these datasets back into their raw values so that each pixel has 2048 levels of sensitivity.

The two major components of the IW-DVC framework, motion compensation and the lossless coding scheme DHC-M have been tested separately. DHC-M has been tested compared against the JPEG2000 lossless mode and context-adaptive binary arithmetic coding (CABAC) [START_REF] Marpe | Context-Based Adaptive Binary Arithmetic Coding in the H.264/AVC Video Compression Standard[END_REF]. In the second set of experiments, the performance of the proposed motion compensation approach has been compared to the first motion compensation algorithm for depth video that uses 2D blockbased motion vector sharing (2D-BMS) [START_REF] Grewatsch | Sharing of Motion Vectors in 3D Video Coding[END_REF] and recently developed 3D block-based motion vector sharing approach (3D-BMS) [START_REF] Fan | Three-Dimensional Depth Map Motion Estimation and Compensation for 3D Video Compression[END_REF].

There are two main kinds of approaches to evaluate the quality of the reconstructed depth images: (1) subjective, or (2) objective methods, such as peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM). Subjective methods [START_REF] Umar | Subjective quality assessment of 3d videos[END_REF], [START_REF] Bosc | A Quality Assessment Protocol for Free-Viewpoint Video Sequences Synthesized from Decompressed Depth Data[END_REF] require human viewers to train the assessment program and judge the quality of the reconstructed images. However, different from the conventional 8-bit depth images, the pixels in the depth images captured by RGB-D sensors contain 11bit data. Original depth information in these images cannot be seen directly. If we downscale them to 8 bits per pixel before using subjective evaluation methods, a certain amount of information is inevitably lost. Therefore, the subjective methods are not applicable in this project, and consequently we have adopted objective methods to evaluate performances.

In the second set of experiments, we have used both PSNR and SSIM to evaluate the quality of the reconstructed depth images. We also have adopted the entropy approach to measure the differences between the reconstructed image and the uncompressed original image. Then, we have analyzed the performance of different crack-filling algorithms. In the third set of the experiments, we have analyzed the encoding complexity based on the processing time for each step and presented time complexity of each step in big O notation. For the latter three sets of experiments, the GoP size was set to 10 and the block size was 8 ⇥ 8.

A. Performance Evaluation of DHC-M Scheme

In the first set of experiments, in order to evaluate its effectiveness, we compared the DHC-M scheme against the standard JPEG2000 lossless mode and CABAC. Here, we have tested it alone without running the motion compensation block (Fig. 1(a)).

DHC-M achieves its best performance when the individualized probability distributions are used to build lookup tables in different scenes. However, DHC-M cannot be universally used in this case. In this experiment, we randomly sampled 30 images in each dataset and generated lookup tables of DHC-M accordingly. So the same lookup tables can be used for all datasets, and DHC-M is able to achieve its near-best performance. The results are presented in Table II.

The size of a single uncompressed depth image is 614.4 KB. It is obvious that the DHC-M scheme outperforms JPEG2000 lossless mode and CABAC in all datasets. The average compression ratio of DHC-M is 18.7% higher than the average compression ratio of JPEG2000 and 127.9% higher than the average compression ratio of CABAC. It is because the images captured by RGB-D sensors have a much higher dynamic range than conventional gray-scale images and the pixels' values switch between the valid values and the invalid value (the largest value) frequently. DHC-M, which takes this feature into account and uses two lookup tables alternatively, centralizes the probability distribution of the difference values and achieves better context modeling. While JPEG2000 and CABAC, which are not specially designed for RGB-D sensor, do not consider this property. Furthermore, in lossless mode, JPEG2000 use a reversible wavelet transform and no quantization is performed. Thus, all bits planes have to be encoded, JPEG2000 can hardly achieve the best performance.

B. Objective Evaluations of Reconstructed Depth Images

PSNR and SSIM can only indicate the degree of closeness between the reconstructed and original depth images. As the depth images captured by RGB-D sensors almost always have a number of pixels with invalid depth value, we need a quality evaluation method which can describe the noises of different kinds of pixels separately. Therefore, we use the entropy of the difference between the reconstructed depth image and the original depth image to evaluate the performance of the IW-DVC framework. As GOP size was 10, the system encoded one complete depth frame (I-frame) losslessly in every 10 frames. The redundancies in the other depth frames (P-frames) of the same group were removed by our motion compensation approach presented in Section II-C. Then, only the newly observed depth information in these frames were coded by DHC-M. 1) System without Crack-Filling Algorithm: The objective of this experiment is to evaluate the effects of different block update thresholds on compression ratio and quality of the reconstructed depth image without implementing the crackfilling algorithm at the decoder side. Three thresholds were tested, which are 1/2, 1/3, and 1/6 of the overall number of pixels in one block. Two entropies for holes and pixels with valid values in the reconstructed depth images were derived. The entropies indicate the theoretical number of bits required to describe the difference between one pixel in the reconstructed depth frame and its corresponding pixel in the original uncompressed depth frame. We also derived the average number of bits required to correct the errors in the holes and pixels with valid values in one complete reconstructed depth frame to the original one (ANBR). The smaller ANBR, the higher reconstructed quality is achieved.

The average entropies and ANBRs of seven datasets are presented in Table III besides PSNR and SSIM.

According to Table III, the compression ratio decreases when the threshold drops. It is because more intra blocks need to be updated and encoded if the threshold is low. In the meantime, when more blocks in the original depth frame are updated, fewer prediction errors exist in the reconstructed depth image. Thus, both entropies decline along with the threshold, and ANBR decreases simultaneously.

To verify the superiority of our system, we have implemented 2D-BMS [START_REF] Grewatsch | Sharing of Motion Vectors in 3D Video Coding[END_REF] and 3D-BMS [START_REF] Fan | Three-Dimensional Depth Map Motion Estimation and Compensation for 3D Video Compression[END_REF] on DHC-M and tested them on the same datasets. 2D-BMS algorithm uses the motion vectors derived from the texture information to encode both color and depth video. 3D-BMS, developed recently, uses variable block sizes to perform motion estimation. In addition, it estimates motion vector in z direction and reported good reconstruction quality on the depth video captured by a static camera. The results are presented in Tables IV andV.

By comparing the results in Tables III, V, and IV, it is clear that the 3D-BMS has enhanced accuracy than 2D-BMS, and our proposed motion compensation scheme with update threshold at 1/3 and 1/6 outperforms 2D-BMS and 3D-BMS in reconstruction accuracy. Since in each dataset, our system has much smaller entropies and ANBR. PSNR and SSIM results of our proposed methods are also higher than the other two algorithms. Moreover, our proposed method has higher compression ratios than 2D-BMS and 3D-BMS. One reason is that our approach only needs one 6D motion vector to describe the inter-frame motion. However, the conventional block-based motion compensation methods require a motion vector for each block.

We have selected three representative examples of results with noticeable differences/artifacts (Fig. 7). The scenarios in these three examples contain both smooth surfaces and abruptly changing object boundaries to allow us to check the performance of algorithms in a wide range of conditions. The smooth surfaces in the original images were reconstructed as coarse surfaces with block-artifacts by 2D-BMS (see the surface of the cabinet in the middle and left images of Fig. 7(b)). It is because the depth information of the same surface changes in the consecutive frames captured by a mobile RGB-D sensor, while 2D-BMS does not consider this issue and does not change the values of the pixels representing the same surface in consecutive frames. 3D-BMS, which has an extra motion vector on z direction and uses variable block sizes, achieves relatively good performance in dealing with this problem. However, it still generates block artifacts on object boundaries. It is because different surfaces on the object boundaries can be placed in the same block. The distance changes on different surfaces are different. Then, the same motion vector on z direction is used for all the pixels in the same block, which leads to incorrect prediction on object boundaries. Therefore, we can see the block artifacts on the smooth surfaces are removed generally in the middle image of Fig. 7(c). But a large number of artifacts and blurs appear on the object boundaries in the left and right images of Fig. 7(c), when the scenes have complex geometrical structures. IW-DVC framework, which uses a 6D motion vector, not only maps each pixel from the reference frame to the predicted frame, but also changes its value individually. Therefore, the IW-DVC scheme does not introduce these block-artifacts and nicely preserves the object boundary information (see Fig. 7(d)). The results of this experiment accord our analysis in the beginning of this paper that the block-based motion compensation schemes are suboptimal for depth video captured by a mobile sensor. However, in Fig. 7(d), it is obvious that some cracks regularly appear in the reconstructed depth images. This drawback of our motion compensation algorithm can be overcome easily by a crack-filling algorithm which is addressed in the next section.

2) System with Crack-Filling Algorithm: In this experiment, we focus on determining the best performing crack- filling algorithm in recovering depth information of the cracks created due to the under-sampling problem. Based on the results of the experiments presented in Section III-B1, we have chosen 1/3 as the update threshold in this experiment by considering both compression ratio and the quality of the reconstructed depth images. We have tested the three crack-filling algorithms described in Section II-E. Comparison results are presented in Table VI. Percent accuracy of each method has been calculated as follows accuracy = correctly filled pixels max(pixels need filling, filled pixels) ⇥ 100%.

Tables III and VI show that the third method has the highest percent accuracy, PSNR, and SSIM. The first method, which considers the horizontal neighboring pixels, is only suitable for cracks generated by a horizontally rotating and translating camera. However, in these datasets, the mobile camera moves and rotates in all directions. The second method has a similar performance with the third one. But, as it is applied on complete images, it smoothes the overall image and introduces noise in areas without cracks. Because of this, the entropy of holes decreases while the entropy of pixels with valid values increases. An important observation to make here is that the application of third method leads to a significant reduction of prediction errors while maintaining a high compression ratio.

The third method has higher ANBR, PSNR, and SSIM values, evan though it uses an update threshold of 1/3 (Table VI) compared to 1/6 (Table III). Thus, we adopt the third crackfilling algorithm at the decoder side to enhance the quality of the reconstructed images and preserve high compression ratios. The corresponding enhanced results are shown in Fig. 7(e).

C. Encoding Complexity Analysis

In this set of experiments, we present time complexity of each step in big O notation and further analyze encoding complexity based on the processing time of each step. We implemented the encoder with the update threshold at 1/3 on our computer. We measured the average processing time of each step required for encoding one frame in milliseconds. The results are summarized in Table VII.

In Fig. 8, we present seven histograms to illustrate probability distributions of the numbers of iterations in different datasets. We conducted extra experiments and found there is not any direct relation between the number of sample points and the required number of iterations. We also list the average number of iterations required for convergence in the process of inter-frame motion estimation in Table VII.

In Table VII, n w and n s represent the number of pixels in the searching window and the number of sampled points in inter-frame motion estimation process. N f represents the number of pixels in one complete frame. As N f n w n s , the time complexity of our motion compensation algorithm is O(N f ). The time complexity of 2D-BMS is O(N w N f ). N w indicates the number of different block positions in the matching window (N w n w ). The time complexity of 3D-BMS is also O(N w N f ). Without estimating MVs in a block by block manner, our proposed motion compensation approach with linear time complexity is more efficient than 2D-BMS and 3D-BMS. Moreover, according to Table VII, it is clear that the depth video can be encoded up to around 25 fps in real time on a standard computer.

IV. CONCLUDING REMARKS

We have presented a novel coding framework, called 3D image warping based depth video compression (IW-DVC), for efficiently removing the existing redundancy in the depth video frames captured by a mobile RGB-D sensor. In particular, the motion compensation scheme included in the framework, designed to exploit the unique characteristics of depth images, and works cooperatively with the egomotion estimation and 3D image warping.

Experimental results show that our motion compensation method reduces the prediction errors of 2D-BMS and 3D-BMS for depth video captured by a mobile sensor to 55.9% and 72.8% on average, respectively. Also, they demonstrate that IW-DVC framework is capable of keeping the quality of the reconstructed depth image at a high level and can accurately determine the newly observed depth information in each frame. This significantly enhances the compression ratio. Furthermore, the results show that the IW-DVC framework is capable of operating in real time. As a final note, the losslessly encoded I-frames, IW-DVC is suitable for many applications that have high requirements on the accuracy of keyframes.

Fig. 1 .

 1 Fig. 1. Operational overview of the 3D Image Warping Based Depth Video Compression (IW-DVC) framework.

Fig. 2 .

 2 Fig. 2. A Microsoft Kinect RGB-D sensor, shown as installed vertically on Monash University's experimental mobile robot "eyeBug". Samples of captured color and depth images are shown as well.

Fig. 3 .

 3 Fig.3. An intuitive example of forward estimation. The depth frame ẐP is predicted from Z I as the frame captured at P-frame's viewpoint virtually. After comparing ẐP and Z P , the newly observed information in Z P is outlined in yellow.

Fig. 4 .

 4 Fig.4. An example of situations that may lead to forward estimation failures: Object A is not in the field of view of the sensor at I-frame's location. Object enters into the view of the sensor at P-frame's location, and consequently a section of the background surface becomes occluded.

Fig. 5 .

 5 Fig. 5. Depth pixels and their reference pixels in a frame.

Fig. 6 .

 6 Fig. 6. Cracks can be introduced during the image warping process due to the under-sampling problem.

Fig. 7 .

 7 Fig. 7. Examples of the original depth images and their reconstructed counterparts in datasets 1, 3 and 6: (a) Originals; (b) Reconstructed depth images generated by 2D-BMS; (c) Reconstructed depth images generated by 3D-BMS; (d) Reconstructed depth images generated by the IW-DVC framework with an update threshold of 1/3; (e) Enhancements of depth images in (d) by 3 rd crack-filling algorithm.

Fig. 8 .

 8 Fig. 8. Probability distributions of the number of iterations required for each of the 7 datasets.

TABLE II COMPRESSION

 II PERFORMANCE OF JPEG2000, CABAC, AND DHC-M.

	DATASET		JPEG2000	CABAC	DHC-M
	ARCHIVE NAME	ID	AVERAGE	COMP.	AVERAGE	COMP.	AVERAGE	COMP.
			SIZE (KB)	RATIO	SIZE (KB)	RATIO	SIZE (KB)	RATIO
	freiburg1 plant	1	72.43	8.48	148.48	4.14	58.42	10.52
	freiburg2 dishes	2	68.34	9.00	140.84	4.36	54.91	11.19
	freiburg3 cabinet	3	60.02	10.24	118.72	5.18	51.40	11.95
	freiburg3 large cabinet	4	57.47	10.69	105.62	5.82	50.74	12.11
	freiburg3 structure texture far	5	61.60	9.97	123.91	4.96	52.31	11.75
	freiburg3 long office household	6	68.51	8.97	107.93	5.69	56.23	10.93
	freiburg1 xyz	7	62.74	9.79	127.88	4.80	54.76	11.22

TABLE III QUALITY

 III EVALUATION OF THE RECONSTRUCTED DEPTH IMAGES GENERATED BY IW-DVC FRAMEWORK WITH USING DIFFERENT BLOCK UPDATE

						THRESHOLDS.
	DATA SET	THRES HOLD	COMP. RATIO	PSNR	SSIM	ENTROPY PIXELS WITH VALID VALUE	HOLES	AVG. NUMBER OF BITS REQUIRED TO CORRECT THE ERRORS IN A RECONSTRUCTED DEPTH IMAGE (ANBR) PIXELS WITH VALID VALUE HOLES OVERALL
		1/2	594	33.53	0.8356	1.49	1.07	3.46⇥10 5	8.03⇥10 4	4.36⇥10 5
	1	1/3	285	34.79	0.8585	1.45	0.81	3.38⇥10 5	6.28⇥10 4	4.00⇥10 5
		1/6	139	35.91	0.8818	1.44	0.65	3.34⇥10 5	4.78⇥10 4	3.82⇥10 5
		1/2	590	35.66	0.8780	1.28	1.19	3.29⇥10 5	5.93⇥10 4	3.89⇥10 5
	2	1/3	374	37.88	0.8942	1.25	0.82	3.22⇥10 5	4.01⇥10 4	3.62⇥10 5
		1/6	256	38.71	0.9010	1.21	0.62	3.12⇥10 5	3.02⇥10 4	3.42⇥10 5
		1/2	869	27.15	0.7254	1.47	1.92	3.79⇥10 5	9.53⇥10 4	4.74⇥10 5
	3	1/3	352	29.18	0.8144	1.45	1.75	3.73⇥10 5	8.73⇥10 4	4.60⇥10 5
		1/6	269	31.05	0.8321	1.40	1.53	3.60⇥10 5	7.73⇥10 4	4.38⇥10 5
		1/2	395	42.87	0.9256	1.16	0.61	2.62⇥10 5	4.93⇥10 4	3.12⇥10 5
	4	1/3	268	43.26	0.9296	1.12	0.45	2.54⇥10 5	3.61⇥10 4	2.90⇥10 5
		1/6	211	43.88	0.9347	1.07	0.31	2.41⇥10 5	2.54⇥10 4	2.67⇥10 5
		1/2	829	41.77	0.8770	1.13	0.61	2.99⇥10 5	2.59⇥10 4	3.25⇥10 5
	5	1/3	415	42.49	0.8883	1.11	0.55	2.93⇥10 5	2.34⇥10 4	3.16⇥10 5
		1/6	291	44.05	0.9040	1.09	0.48	2.88⇥10 5	2.26⇥10 4	3.09⇥10 5
		1/2	625	46.51	0.9256	1.00	0.93	2.54⇥10 5	4.89⇥10 4	3.03⇥10 5
	6	1/3	342	48.06	0.9399	0.98	0.75	2.51⇥10 5	3.91⇥10 4	2.90⇥10 5
		1/6	220	49.26	0.9479	0.97	0.64	2.48⇥10 5	3.33⇥10 4	2.81⇥10 5
		1/2	357	35.49	0.8117	1.25	1.45	2.86⇥10 5	1.14⇥10 4	4.01⇥10 5
	7	1/3	250	36.09	0.8303	1.20	1.26	2.75⇥10 5	9.96⇥10 4	3.74⇥10 5
		1/6	180	37.63	0.8592	1.13	0.96	2.58⇥10 5	7.49⇥10 4	3.33⇥10 5

TABLE IV QUALITY

 IV EVALUATION OF THE RECONSTRUCTED DEPTH IMAGES GENERATED BY 2D-BMS.

	DATA SET	COMP. RATIO	PSNR	SSIM	ENTROPY PIXELS WITH VALID VALUE	HOLES	AVG. NUMBER OF BITS REQUIRED TO CORRECT THE ERRORS IN A RECONSTRUCTED DEPTH IMAGE (ANBR) PIXELS WITH VALID VALUE HOLES OVERALL
	1	156	23.92	0.7321	2.82	1.84	6.54⇥10 5	1.39⇥10 5	7.93⇥10 5
	2	265	24.27	0.8256	1.78	1.66	4.60⇥10 5	8.27⇥10 4	5.42⇥10 5
	3	214	21.65	0.5471	1.95	1.46	5.04⇥10 5	7.13⇥10 4	5.75⇥10 5
	4	220	35.76	0.8562	1.19	1.10	2.68⇥10 5	8.99⇥10 4	3.58⇥10 5
	5	239	24.62	0.7707	2.73	0.77	7.33⇥10 5	2.95⇥10 4	7.62⇥10 5
	6	189	42.21	0.9068	1.06	1.06	2.70⇥10 5	5.65⇥10 4	3.27⇥10 5
	7	166	26.44	0.7759	2.38	1.36	5.53⇥10 5	1.02⇥10 5	6.55⇥10 5

TABLE V QUALITY

 V EVALUATION OF THE RECONSTRUCTED DEPTH GENERATED BY 3D-BMS.

	DATA SET	COMP. RATIO	PSNR	SSIM	ENTROPY PIXELS WITH VALID VALUE	HOLES	AVG. NUMBER OF BITS REQUIRED TO CORRECT THE ERRORS IN A RECONSTRUCTED DEPTH IMAGE (ANBR) PIXELS WITH VALID VALUE HOLES OVERALL
	1	118	28.68	0.7894	1.73	1.30	4.04⇥10 5	9.59⇥10 4	5.00⇥10 5
	2	136	29.65	0.8730	1.40	0.95	3.64⇥10 5	4.67⇥10 4	4.11⇥10 5
	3	149	27.06	0.7125	1.63	1.38	4.22⇥10 5	6.76⇥10 4	4.89⇥10 5
	4	124	39.29	0.8723	1.05	1.04	2.37⇥10 5	8.54⇥10 4	3.22⇥10 5
	5	125	33.62	0.8107	2.09	0.73	5.60⇥10 5	2.79⇥10 4	5.88⇥10 5
	6	134	47.19	0.9184	0.99	0.93	2.51⇥10 5	5.06⇥10 4	3.01⇥10 5
	7	115	32.18	0.8360	1.63	1.18	3.80⇥10 5	8.86⇥10 4	4.68⇥10 5

TABLE VI QUALITY

 VI EVALUATION OF THE RECONSTRUCTED DEPTH IMAGES ENHANCED BY DIFFERENT CRACK-FILLING ALGORITHMS, UPDATE THRESHOLD = 1/3.

	DATA SET	METHOD	ACCURACY PERCENT	PSNR	SSIM	ENTROPY PIXELS WITH VALID VALUE	HOLES	AVG. NUMBER OF BITS REQUIRED TO CORRECT THE ERRORS IN A RECONSTRUCTED DEPTH IMAGE (ANBR) PIXELS WITH VALID VALUE HOLES OVERALL
		1	48.7%	33.21	0.8645	1.62	0.04	3.95⇥10 5	2.48⇥10 4	3.97⇥10 5
	1	2	81.5%	35.77	0.8956	1.46	0.30	3.45⇥10 5	2.20⇥10 4	3.67⇥10 5
		3	81.5%	36.05	0.8985	1.45	0.23	3.44⇥10 5	1.67⇥10 4	3.61⇥10 5
		1	20.2%	34.75	0.9286	1.30	0.05	3.43⇥10 5	2.10⇥10 3	3.45⇥10 5
	2	2	83.0%	39.41	0.9665	1.24	0.26	3.22⇥10 5	1.23⇥10 4	3.34⇥10 5
		3	83.0%	39.87	0.9676	1.25	0.19	3.25⇥10 5	9.08⇥10 3	3.33⇥10 5
		1	26.6%	29.72	0.7167	1.65	0.09	4.41⇥10 5	3.64⇥10 4	4.44⇥10 5
	3	2	91.2%	32.67	0.8478	1.51	0.84	3.90⇥10 5	4.06⇥10 4	4.31⇥10 5
			91.2%	34.03	0.8574	1.46	0.34	5	1.66⇥10 4	3.96⇥10 5
		1	18.8%	42.01	0.9520	1.17	0.06	2.77⇥10 5	4.07⇥10 3	2.82⇥10 5
	4	2	88.6%	46.27	0.9744	1.13	0.11	2.60⇥10 5	8.58⇥10 3	2.69⇥10 5
		3	88.6%	48.55	0.9776	1.13	0.07	2.59⇥10 5	5.47⇥10 3	2.65⇥10 5
		1	32.5%	42.69	0.9179	1.17	0.05	3.18⇥10 5	1.92⇥10 3	3.20⇥10 5
	5	2	86.4%	43.77	0.9483	1.12	0.18	3.00⇥10 5	7.38⇥10 3	3.07⇥10 5
		3	86.4%	44.92	0.9506	1.12	0.14	2.99⇥10 5	5.72⇥10 3	3.05⇥10 5
		1	42.4%	44.21	0.9184	1.05	0.06	2.75⇥10 5	2.60⇥10 3	2.78⇥10 5
	6	2	88.9%	46.99	0.9643	0.99	0.41	2.54⇥10 5	2.10⇥10 4	2.75⇥10 5
		3	88.9%	49.67	0.9679	0.99	0.26	2.54⇥10 5	1.33⇥10 4	2.67⇥10 5
		1	31.9%	36.98	0.8611	1.32	0.16	3.22⇥10 5	1.07⇥10 4	3.33⇥10 5
	7	2	85.5%	37.50	0.8970	1.22	0.46	2.88⇥10 5	3.37⇥10 4	3.21⇥10 5
		3	85.5%	38.16	0.8993	1.21	0.41	2.84⇥10 5	3.00⇥10 4	3.14⇥10 5

TABLE VII PROCESSING

 VII TIME OF THE PROPOSED FRAMEWORK IN EACH STEP FOR VARIOUS DATASETS.

	DATASET	NUMBER OF ITERATIONS	INTERFRAME MOTION ESTIMATION	FORWARD ESTIMATION/ REVERSE CHECK	BLOCK UPDATE	DHC-M	OVERALL
	1	7.54	14.45 (31.12%)	26.95 (58.04%)	1.15 (2.48%)	3.88 (8.36%)	46.43
	2	4.01	12.72 (30.22%)	24.10 (57.26%)	1.14 (2.71%)	4.13 (9.81%)	42.09
	3	6.21	11.87 (27.29%)	25.51 (58.66%)	1.58 (3.63%)	4.53 (10.42%)	43.49
	4	5.95	11.64 (28.16%)	24.82 (60.05%)	1.20 (2.90%)	3.67 (8.88%)	41.33
	5	6.58	10.55 (24.11%)	27.98 (63.94%)	1.12 (2.56%)	4.11 (9.39%)	43.76
	6	6.63	9.31 (23.32%)	25.26 (63.28%)	1.18 (2.96%)	4.17 (10.45%)	39.92
	7	6.57	14.36 (33.43%)	23.38 (54.20%)	1.32 (3.07%)	3.99 (9.29%)	42.95
	Average	6.21	12.13 (28.31%)	25.41 (59.30%)	1.24 (2.89%)	4.07 (9.50%)	42.85
	Time complexity	-	O(nwns)	O(N f )	O(N f )	O(N f )	O(N f )

A larger number of sampled points leads to more accurate results. However, the processing time increases at the same time. By considering both time consumption and accuracy, we have chosen N I = N P =

in this paper.

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY (UNDER REVIEW)

Based on different compression requirements, the value of the threshold can be set as a certain portion of the total number of pixels in one block, such as 1/2 or 1/6. A lower threshold leads to lower compression ratios.

This work was supported by the Australian Research Council Centre of Excellence for Robotic Vision (project number CE140100016).

X.


Jana Brunner
email: brunner@zas.gwz-berlin.de

Phil Hoole

Pascal Perrier

The use of sensory feedback in the adaptation of perturbed /s

) with the prosthesis and auditory feedback masked, and (3) with the prosthesis and auditory feedback available. The results show that the initial adaptation attempts without auditory feedback are dependent on the prosthesis type and directed towards reaching the original tongue palate contact pattern. In a second adaptation step with auditory feedback available speakers reorganised tongue and jaw movements in order to produce more subtle acoustic characteristics of the sound such as the high amplitude noise which is typical for sibilants.

Introduction

Previous work on perturbation has shown that sensory (auditory and tactile) feedback is essential in order to adapt for a perturbation of the articulation. Jones & Munhall [START_REF] Jones | Learning to produce speech with an altered vocal tract: The role of auditory feedback[END_REF], for example, show that speakers need auditory feedback in order to adapt the acoustic characteristics in /s/ produced with extended upper incisors. Honda & Murano [START_REF] Honda | Effects of tactile and auditory feedback on compensatory articulatory response to an unexpected palatal perturbation[END_REF] tested the influence of both auditory and tactile feedback during adaptation to an inflatable palate. They report the best results in adaptation when both kinds of feedback were available, less good results when auditory feedback was masked, even worse results when tactile feedback was masked and the worst results when both kinds of feedback were unavailable.

The present study investigates the influence of auditory feedback on adaptation when speech is perturbed with different palatal prostheses. Our expectation was that when the auditory feedback is masked and speakers have only tactile feedback available, adaptation will depend on the prosthesis type, since speakers will try to achieve the tongue palate contact pattern they used to have in the unperturbed condition. When both kinds of feedback are available, speakers will change the adaptive strategy and try to produce the acoustic output (spectrally characterised by the same acoustic centre of gravity) they used to have, even if this involves a change of the tongue palate contact pattern.

Methods

A perturbation experiment was carried out where speakers' palate shape was modified with a palatal prosthesis. Two types of palatal prostheses were used, one which lowered the palate and moved the alveolar ridge posteriorily ("alveolar prosthesis"), and one which lowered the palatal surface by filling out the palatal vault ("central prosthesis"). Five German subjects took part in the study. Three of them, TP, AP and DS had an alveolar prosthesis, the other two, BP and SK, had a central prosthesis. The articulatory movements of the speakers were recorded via electromagnetic articulography (three speakers with the AG 100 and two with the AG 500). Sensors were placed midsagittally on the tongue tip, tongue dorsum and tongue back, jaw and both lips. For the present purpose the data of the tongue tip and the jaw sensor were analysed.

In a first session speakers were recorded without the perturbation (to be called unperturbed condition). Then the artificial palate was inserted and speakers' auditory feedback was masked with white noise (to be called white noise condition). Afterwards, the masking noise was removed and speakers were recorded while adapting with auditory feedback available (to be called auditory feedback condition). /s/ was recorded in the nonsense word /'zasa/ which was spoken in a carrier phrase: Ich sah sassa an ("I looked at zassa."). There were 20 repetitions (randomised with other material) in each session. Acoustic recordings were carried out as well.

The fricative was acoustically segmented (friction onset to friction offset) in each utterance. The acoustic centre of gravity (COG) for a band pass filtered signal (700 Hz to 12 kHz) was measured. Furthermore, the horizontal position of the tongue tip sensor and the vertical position of the jaw sensor in the middle of the acoustically measured consonant interval were estimated. In order to judge whether the measured parameters differed significantly across sessions repeated measures ANOVAs were calculated for data split by speaker.

Expectations

Speakers with an alveolar prosthesis were expected to retract the tongue in the white noise condition, in order to keep the alveolar place of articulation. This would result in lower COG values due to a lengthening of the front vocal tract cavity. In the session thereafter (with auditory feedback available) these speakers were expected to adapt the tongue position so that the original COG would be reached.

In contrast to that, speakers with a central prosthesis were expected not to retract the tongue, since their alveolar ridge was not moved posteriorily by the prosthesis. The centre of gravity should therefore stay the same in all sessions for these speakers.

Both groups of speakers were expected to lower the jaw when speaking with the prosthesis in order to avoid a complete vocal tract closure. Acoustic centre of gravity. The results for the measurements of the centre of gravity are shown in figure 1. Each bar shows the mean over one session. Bars belonging to the same triple refer to one speaker and they are in the order of the recordings: the first (black) bars of a triple correspond to the unperturbed session, the second (grey) ones to the white noise condition and the third (white) ones to the auditory feedback condition. The two leftmost speakers (SK and BP) had central prostheses, the other three had alveolar prostheses.

Results

SK

For four speakers (SK and BP with a central prosthesis and AP and DS with an alveolar prosthesis) the COG is lower in the white noise condition than in the unperturbed condition. For three speakers (SK with a central prosthesis and TP and DS with an alveolar prosthesis) the COG is lower when auditory feedback is available than when it is not. The difference between the unperturbed and the auditory feedback condition are significant for three speakers. Horizontal tongue position. Figure 2 shows the results for the horizontal tongue position. Again the abscissa gives the speaker, the first bar of a triple refers to the unperturbed session, the second to the white noise condition and the third to the auditory feedback condition. The ordinate gives the horizontal position of the tongue tip sensor. Higher values represent a more retracted position.

The two speakers with a central prosthesis (SK and BP) protrude the tongue in the white noise condition but correct for this change afterwards. For the three speakers with an alveolar prosthesis (TP, AP, DS) a consistent retraction of the tongue tip over the three sessions was found (but only one of those speakers, DS, also has a consistent decrease in COG across the three sessions.

Vertical jaw position. Figure 3 shows the results for the vertical jaw position. Higher values signify a higher jaw position, lower values a lower one. For all speakers the jaw position is lowered when the prosthesis is inserted, but raised again when auditory feedback becomes available. This difference between the white noise and the auditory feedback condition is significant for all speakers. Speakers with a central prosthesis seem to raise the jaw a little less than speakers with an alveolar prosthesis. When the difference between the white noise condition and the auditory feedback condition is significant this is signaled by a bracket above the bars.

Summary of the results

As expected, we found a difference in the adaptive behaviour of speakers with different prosthesis types: Speakers with an alveolar prosthesis retracted the tongue in the white noise condition whereas speakers with a central prosthesis protruded it. However, a correction of this positional change in the auditory feedback condition could only be found for speakers with a central prosthesis.

Furthermore, the expected relation between COG and horizontal position of the tongue could not be found. For a retracted tongue the COG did not always decrease, nor did it increase consistently for a protruded tongue. This suggests that the COG is affected by a further mechanism apart from the constriction position.

As expected, independent of the prosthesis type the jaw was lowered for all speakers when the prosthesis was inserted. Contrary to our expectations, however, it was raised in the third session.

Discussion

Even if the results thus match our expectations only partly, they can be interpreted in the following way. When speakers insert the prosthesis (white noise condition) they might try to keep the same global articulatory position in reference to the palate: For the speakers with an alveolar prosthesis the tongue is retracted because the alveolar ridge is retracted; for the other speakers the tongue is protruded probably as a mechanical effect of the prosthesis.

When auditory feedback becomes available speakers' first aim should be to check the acoustic output they produce and to correct it. However, no adaptation of the COG can be found. Hence COG could be an inappropriate parameter to characterize the perception of the fricatives, which was already suggested by Jones & Munhall [START_REF] Jones | Learning to produce speech with an altered vocal tract: The role of auditory feedback[END_REF].

However, the adaptation efforts could be directed towards another, more prominent acoustic characteristic of the sound. /s/ is a sibilant, and an important acoustic characteristic of sibilants is the high amplitude noise produced by directing the air jet against an obstacle (the incisors) in front of the constriction (Shadle [4], Shadle [START_REF] Shadle | The effect of geometry on source mechanisms of fricative consonants[END_REF]). This, however, necessitates a high jaw position, the distance between the incisors is too large and no turbulences are created (Mooshammer et al. [START_REF] Mooshammer | Interarticulator cohesion within coronal consonant production[END_REF]). This higher jaw position can in fact be found in the auditory feedback condition. When auditory feedback becomes available speakers can thus be assumed to notice that they are lacking the high amplitude noise typical for /s/, and this might lead them to raise the jaw. The retraction of the tongue which can be found in the auditory feedback condition could be seen as a consequence of the higher jaw position: When the jaw is high the space available for the tongue in the alveolar region can be assumed to be no longer sufficient so that the tongue is moved to a more retracted position where there is more space.

To conclude, our results show that certain properties of the sound, such as the constriction size, can be adapted by using tactile feedback only. More subtle acoustic characteristics of /s/, however, require a reorganisation of the articulatory strategy, i.e. the positional relation of tongue and jaw has to be changed. For these articulatory changes auditory feedback becomes essential, as shown by the changes observed in the third session.

Figure 1 :

 1 Figure 1: Mean COG values in the different sessions. Error bars show standard error. When the difference between the unperturbed and the auditory feedback session is significant this is signalled by a bracket above the bars. Significance levels: *p < 0.05, **p < 0.01, ***p < 0.001.

Figure 2 :

 2 Figure 2: As figure 1 but showing mean horizontal position of the tongue tip. Higher values denote more retracted positions.

Figure 3 :

 3 Figure 3: As figure 1 but showing mean vertical position of the jaw sensor at the consonantal target position in /s/. Higher values denote higher positions.When the difference between the white noise condition and the auditory feedback condition is significant this is signaled by a bracket above the bars.