Derek Chan

Hylke Buisman

Christian Theobalt

Sebastian Thrun

A Noise-Aware Filter for Real-Time Depth Upsampling

A new generation of active 3D range sensors, such as time-of-flight cameras, enables recording of full-frame depth maps at video frame rate. Unfortunately, the captured data are typically starkly contaminated by noise and the sensors feature only a rather limited image resolution. We therefore present a pipeline to enhance the quality and increase the spatial resolution of range data in real-time by upsampling the range information with the data from a high resolution video camera. Our algorithm is an adaptive multi-lateral upsampling filter that takes into account the inherent noisy nature of real-time depth data. Thus, we can greatly improve reconstruction quality, boost the resolution of the data to that of the video sensor, and prevent unwanted artifacts like texture copy into geometry. Our technique has been crafted to achieve improvement in depth map quality while maintaining high computational efficiency for a real-time application. By implementing our approach on the GPU, the creation of a real-time 3D camera with video camera resolution is feasible.

Introduction

In recent years, a new type of camera has been developed that measures the timeof-flight (TOF) of infrared light to reconstruct 3D scene geometry at real-time frame rates, (e.g. [START_REF] Oggier | 3D-Imaging in Real-Time with Miniaturized Optical Range Camera[END_REF]) largely independently of scene texture. Unfortunately, being a relatively young technology, time-of-flight imaging has not enjoyed the same advances with respect to image resolution, speed, and image quality, that have been made in traditional 2D intensity imaging. In consequence, current TOF sensors provide depth readings of comparably low image resolution (e.g. 176 × 144 pixels for the MESA Swissranger TM SR3100) that are heavily contaminated with noise in the distance readings.

The ability to rapidly capture good quality, high-resolution depth images of dynamic scenes even in environments that are not suitable for image-based stereo (e.g. due to lack of texture), enables a variety of intriguing applications. Examples are environment perception in robotics, dynamic scene reconstruction for 3D video, marker-less motion capture and human computer interaction, to name only a few.

In this paper, we therefore propose a new method to capture depth data at highresolution, at a minimal noise level, and at near real-time frame rates. To achieve this goal, we simultaneously record a dynamic scene with a high-resolution video camera and a frame-synchronized low-resolution depth sensor. Our method upsamples the lowresolution data of the depth sensor to the high resolution of the video camera. To serve this purpose, it enforces statistical relations between the high-resolution video images and the low-resolution depth maps, such as the collocation of intensity and depth edges, as well as the assumption of smooth geometry in uniformly colored regions. In order to fulfill the real-time processing requirements, we pose our algorithm as a real-time filtering method. Our approach capitalizes on and extends the concept of the imagebased bilateral filter, an edge-preserving image filter [START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF]. The bilateral filter adaptively shapes its spatial smoothing kernel by weighting it with a local color similarity term, thereby preventing blurring across intensity gradients.

In previous work, Kopf et al. [START_REF] Kopf | Joint bilateral upsampling[END_REF] proposed a method called joint bilateral upsampling (JBU) to lift a low resolution tone-mapping result to the high resolution of a mega-pixel intensity image. This idea of splitting the filter domain into one intensity part and one depth part is also a core concept of our depth data upsampling. Unfortunately, the use of a color image to guide the up-sampling of a depth map admittedly runs the risk of inadvertently copying texture from the color image into actually smooth areas of the captured depth, Sect. 3. This effect is particularly noticeable if depth sensor readings contain a high amount of random noise in the first place.

We therefore extend the original bilateral upsampling approach and propose a new noise-aware filter, termed NAFDU (Noise-Aware Filter for Depth Upsampling), that adaptively dampens the influence of color similarity in depth regions which are likely to be geometrically smooth but heavily noise affected, Sect. 4. This way, the unwanted effect of texture copying is almost entirely eliminated, which makes our method particularly suitable to be applied with currently available TOF imaging devices. Our algorithm can be effectively and efficiently implemented on state-of-the art graphics processing units (GPUs) to yield high-quality high-resolution depth readings in real-time. We have tested our approach using a prototype system comprising of one SR3100 time-of-flight depth camera and one high-resolution video camera, Sect. 5. Our results show that our noise-aware filtering concept enables faithful reconstruction of detailed 3D scene geometry at a high level of detail and with less unwanted artifacts than previous methods from the literature would deliver, Sect. 6.

Related Work

Depending on the application in mind, different strategies can be used to enhance the image resolution of noisy range sensors, such as time-of-flight cameras [START_REF] Oggier | 3D-Imaging in Real-Time with Miniaturized Optical Range Camera[END_REF]. In case the recorded scene is static, several measurements of the scene can be averaged or the integration time of the sensor can be increased to reduce the uncertainty in distance readings. For static scenes it is also feasible to combine depth readings from several slightly displaced viewpoints, and apply a superresolution scheme to the depth maps, which yields 3D data of higher fidelity [START_REF] Schuon | High-quality scanning using time-offlight depth superresolution[END_REF][START_REF] Rosenbush | Super-resolution enhancement of flash LADAR range data[END_REF].

If real-time acquisition of dynamic scenes is the goal, however, all previously mentioned methods are not applicable. One way to meet the demanding performance requirements of real-time depth upsampling is to fuse the data of a high resolution video camera and a low-resolution depth sensor. A common upsampling recipe enforces heuristics between depth and intensity images, such as the collocation of depth and intensity gradients, and the smoothness of geometry in areas of similar color.

An approach that put this idea into practice was proposed by Diebel et al. [START_REF] Diebel | An application of markov random fields to range sensing[END_REF]. They fuse depth maps of a laser range finder with intensity maps of a color camera by defining the posterior probability of the high-resolution reconstruction as a Markov Random Field (MRF) and optimizing for the maximum-a-posteriori (MAP) solution.

Following similar heuristics, Kopf et al. proposed an alternative fusion method called joint bilateral upsampling [START_REF] Kopf | Joint bilateral upsampling[END_REF]. Their algorithm uses a modification of the bilateral filter, an edge-preserving smoothing filter for intensity images [START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF]. The bilateral filter locally shapes the spatial smoothing kernel by multiplying with a color similarity term, a.k.a the range term, which yields an edge-preserving smoothing filter. Kopf et al. capitalize on this adaptive smoothing capability and bilaterally upsample a low-resolution tone mapping result such that it matches the resolution of a multi-megapixel intensity image. Recently, Crabb et al. applied bilateral upsampling to range data captured with a time-of-flight camera in order to do real-time matting [START_REF] Crabb | Real-time foreground segmentation via range and color imaging[END_REF].

At this point we would like to mention that researchers also investigated new bilateral filter decompositions and approximations to improve runtime performance [START_REF] Weiss | Fast median and bilateral filtering[END_REF][START_REF] Paris | A fast approximation of the bilateral filter using a signal processing approach[END_REF]. Although it can be assumed that variants of such techniques would also allow for further speedups of our new filter, we have currently not explored this direction further, since already with the current design a GPU implementation allows for real-time performance.

Yang et al. [START_REF] Yang | Spatial-Depth Super Resolution for Range Images[END_REF] also propose an upsampling technique based upon a bilateral filter. However, they do not use a joint bilateral technique to link the two images. Rather they use the high resolution image to create a cost volume to which they apply a standard bilateral filter. This requires them to run a 2D bilateral kernel over multiple slices of a volume. Even with GPU optimizations that we propose in this paper, their effective runtime would be our runtime multiplied by the number of cost slices, and thus orders of magnitude higher than for our approach.

All methods mentioned so far fail to address the problem that in the presence of severe noise in the depth readings, the fundamental heuristic assumptions about the relation between depth and intensity may lead to erroneous copying of texture into actually smooth geometry. In this paper, we therefore propose a new adaptive filter that prevents these artifacts, yet enables real-time high-quality upsampling when implemented on the GPU.

The Bilateral Filter and Standard Bilateral Upsampling

As background for the concepts developed later, we begin with a description of the bilateral filter, an edge-preserving smoothing filter for intensity images which was first described in [START_REF] Tomasi | Bilateral filtering for gray and color images[END_REF]. The bilateral filter adaptively shapes a spatial smoothing kernel by multiplying it with a second term, typically referred to as the range term. The range term measures the photometric similarity of the filtered pixel to each pixel from the local neighborhood. The effect of this two-domain analysis is that in smoothly colored regions standard spatial smoothing is applied, while in the vicinity of color gradients unwanted smoothing across the edge is prevented. A bilateral filter can be expressed as follows:

J p = 1 k p q∈Ω I q f (||p -q||)g(||I p -I q ||) ,
where I p is the intensity of pixel p in input image I, Ω is a spatial neighborhood around p, J p is the value of pixel p in the filtered image J, and k p is a normalizing factor. In this description we recognize first a data term f (||p-q||) and then a range term g(||I p -I q ||).

Generally, f and g are taken to be Gaussian functions with standard deviations σ s and σ r .

In [START_REF] Kopf | Joint bilateral upsampling[END_REF], Kopf et al. propose joint bilateral upsampling, a variation of this filter which can be used to upsample a low resolution image I to the resolution of a high-resolution reference intensity image Ĩ. Instead of evaluating both components of the filter on I, the high resolution reference image is used in the range term, which enables edgepreserving upsampling of I to the resolution of Ĩ. To achieve this goal, the upsampled image Sp is computed as follows:

Sp = 1 k p q ↓ ∈Ω I q ↓ f (||p ↓ -q ↓ ||)g(|| Ĩp -Ĩq ||) .
Here p ↓ and q ↓ are pixels in the low-resolution image and p and q the corresponding pixels in the high-resolution image.

In their original work Kopf et al. used this method to upsample low-resolution tonemapping results to mega-pixel resolution. The technique can also apply the JBU concept to upsample depth maps captured with a range sensor, as further investigated by [START_REF] Crabb | Real-time foreground segmentation via range and color imaging[END_REF]. In this case the reference image Ĩ is a high resolution color image, and I is the lowresolution depth map.

Unfortunately, our experiments showed that direct application of JBU for depth upsampling frequently exhibits artifacts in reconstructed geometry that can be attributed to erroneous assumptions about the correlation of color and depth data, Figs. 1 and4. It is important to note that when JBU is applied to depth data up-sampling, the assumption is made that when neighboring pixels in the reference image have similar color, they also have a similar depth. Clearly, this heuristic assumption is not always valid: two objects with similar color can be at different depths in the scene. In many cases, though, this assumption is actually valid. However, in situations where it does not hold, we can observe two different types of artifacts: One type of artifact that may arise is edge blurring in the depth map. This generally occurs when the two objects on either side of a depth discontinuity have similar color. A second type of artifact is texture copying. Here, textures from the reference image are transferred into geometric patterns that appear in the upsampled depth maps, and thus the reconstructed geometry, Figs. 1 and4. Texture copying either occurs near depth edges where some of the texture of one object has a similar color to (the texture of) the second object, or on actually smooth surfaces where the depth map is noisy, but the reference image has a distinct texture. Note that these assumptions and their related artifacts are also common with other aforementioned techniques, such as [START_REF] Yang | Spatial-Depth Super Resolution for Range Images[END_REF] and [START_REF] Diebel | An application of markov random fields to range sensing[END_REF], which also use a color image to up-sample depth maps, see for instance Figs. 3 and5 for examples of texture copy in MRF upsampling.

Especially, texture copying frequently occurs when bilaterally upsampling noisy time-of-flight depth maps. We therefore propose in the next section a new noise-aware filter that prevents these artifacts and still allows for real-time upsampling.

NAFDU: A New Filter for Noise Aware Depth Upsampling

To address the specific requirements of depth superresolution and denoising for noisy real-time 3D sensors, like time-of-flight cameras, we propose a new multi-lateral noiseaware filter for depth upsampling (NAFDU). It is our goal to preserve the beneficial properties of bilateral upsampling in those areas where the data match our assumptions, and to prevent artifacts in those areas where standard bilateral upsampling is likely to cause erroneous texture copy. The NAFDU filter takes the following form:

Sp = 1 k ′ p q ↓ ∈Ω I q ↓ f (||p ↓ -q ↓ ||)[α(∆ Ω )g(|| Ĩp -Ĩq ||) + (1 -α(∆ Ω ))h(||I p ↓ -I q ↓ ||)]
where f, g and h are all Gaussian functions, and α(∆ Ω ) is a blending function. A high weight α makes our filter behave like a standard bilateral upsampling filter, henceforth referred to as case I of the filter. A low weight α gives higher influence to the modified range term h(||I p ↓ -I q ↓ ||), henceforth named case II of the filter. The latter range term makes the filter behave like a bilateral filter whose both domains are the same gray-level depth maps, i.e. it behaves like an edge preserving filter that smooths the 3D geometry independently of information from the color intensity images.

The crux is to decide when the filter ought to resort to case I and case II respectively. In a nutshell we want the filter to switch to case II only if areas are processed that are actually geometrically smooth but heavily contaminated with random noise in the distance measurement which would lead to texture copy. To serve this purpose, we define our blending function α(∆ Ω ) in the interval [0, 1] as follows:

α(∆ Ω ) = 1 1 + e -ǫ•(∆ Ω -τ ) .
Here, ∆ Ω is the difference between the maximum and minimum gray-scale value (i.e. measured depth value) in pixel neighborhood Ω. This difference is evaluated on a lowpass filtered version of the raw depth map (3 × 3 Gaussian filter kernel). We reason that if this difference is below a certain range, the local surface patch is most likely to be smooth and only noise-affected -thus case II should be triggered, Fig. 2(a). If the min-max difference lies beyond that range, however, we are likely to look at a true geometric detail and case I is the appropriate choice, Fig. 2(b). Fig. 2(c) shows a plot of the weighting function α. The parameter ǫ controls how wide the transition area (in terms of the min-max difference) between case I and case II is. Parameter τ controls at what min-max difference the blending interval shall be centered. Appropriate values for ǫ and τ depend on the characteristics of the employed depth sensor and can be found through experiments. The design of α allows for a gradual blending between case I and case II which prevents artifacts in transition areas.

Please note that the computation of ∆ Ω on a low-pass filtered depth map is important as it enables us to reliably disambiguate between isolated random noise peaks and actual depth edges, as shown in Fig. 2(a),(b). We would also like to note that it is important that the range term h in case II takes the described form (a Gaussian) and cannot simply be set to a box filter in the range domain. With this design we achieve much better preservation of depth discontinuities if α lies in the transition zone. We would also like to mention that by choosing a small spatial support for our NAFDU filter (3 × 3 or 5 × 5), any form of texture copy around true depth edges can be minimized in practice while high-quality denoising is still feasible.

Implementation

The method described requires both 3D range maps and high resolution color images of the same scene that are pixel aligned. To obtain raw data, we built a test-system comprising of a Mesa Swissranger TM SR3100 time-of-flight camera and a Point Grey TM Flea2 video camera. The two cameras are placed side-by-side (as closely as possible) and are frame-synchronized. The Flea2 video camera provides 1024 × 768 pixels of image resolution, and the Swissranger produces depth maps at a granularity of 176 × 144 pixels. Jointly, the test system can record scenes at up to 25 fps. To ensure that capturing is occurring as fast as possible, the camera devices are accessed in a multi-threaded setup.

Temporal synchronization is implemented in a software solution based upon the clock capture mechanism of the Flea2 camera.

To align the depth and video images we resort to the gray-scale IR intensity images that the depth camera provides in addition to depth maps. Given this, we can apply any of a number of standard multi-view camera calibration techniques or simple homographic warping [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF]. In our experiments, we resort to the latter alternative.

The actual upsampling filter is implemented on a state-of-the-art graphics card. Since filter operations on individual pixels can be carried out independently, we can capitalize on the massive stream processing power of modern GPUs which by now feature up to 256 individual stream processors. In particular, we employ Nvidia's CUDA programming framework [START_REF]Nvidia[END_REF] to port our algorithm onto graphics hardware. Overall, we thereby achieve real-time up-sampling and denoising performance, see Sect. 6.

Results

As shown in Figs. 3, 4 and5, our method successfully upsamples the low resolution depth maps to higher resolution. True geometry detail in the data, such as sharp depth discontinuities, are preserved and enhanced, the random noise level is greatly reduced, and corroborating artifacts typical for previous fusion methods from the literature, in particular texture copying, are prevented. By exploiting the GPU as a fast stream processor, real-time noise aware upsampling is feasible. Overall, our NAFDU filter design successfully handles the data produced by state-of-the-art time-of-flight cameras which exhibit significantly higher random noise levels than most active scanning devices for static scenes.

To demonstrate the effectiveness of our approach we applied our technique to various scenes including our own recorded sequences as well as scenes from the Middlebury stereo benchmark data set (http://vision.middlebury.edu/stereo/data/). In the following, we analyze three main aspects of our method more closely. We first demonstrate the visual superiority of our results to results obtained with previous fusion-based upsampling methods, Sect. 6.1. Thereafter, we discuss the advantageous run time properties of our algorithm, and discuss practical performance gains in comparison to optimizationbased upsampling methods, Sect. 6.2. Finally, we provide a quantitative comparison of the accuracy of our outputs to those from MRF-based upsampling [START_REF] Diebel | An application of markov random fields to range sensing[END_REF] using data from the Middlebury data set, Sect. 6.3 We end the section by noting some overall findings from our results and by discussing some possible future directions of investigation for our work, Sect. 6.4.

Upsampling Quality and Effectiveness of Noise-aware Filtering

At the low integration times required for scene capture at 25 fps, the depth data provided by the SR3100 are severely noise-affected, as can be seen in Figs. 3 and4. We have tested our method on several different scenes recorded with our test rig, ranging from scenes with moving people to static scenes with objects of different textural appearance. In either case, the scenes have been recorded at a frame rate of 25 fps and upsampling was performed on each frame individually.

Fig. 3 shows that with our upsampling approach, the resolution of the depth data can be effectively raised to the level of the video camera. Overall, the noise level is greatly reduced and sharp details in the original scene, such as edges, are preserved and enhanced, Figs. 3(d) and 4(f),(g) show that all previous fusion-based upsampling methods, in this case exemplified by the MRF and JBU methods, fail to appropriately address the high noise level in time-of-flight imagers. Consequently, intensity texture patterns are embossed as 3D structure into the upsampled geometry, as it can be seen on the book cover in Fig. 4(g) when using JBU, the checkerboard in Fig. 4(f),(g) when using either MRF or JBU, or the shopping bag in Fig. 1(d) when using JBU. In contrast, our noise-aware scheme effectively prevents this texture copy, and thus our achieved reconstruction quality exceeds the one obtained with previous approaches.

Runtime Analysis

On the one hand, our technique has been designed to prevent specific noise-related upsampling artifacts, but is has also been designed with computational efficiency in mind. Our method's basis as a filter allows for effective parallelization when implemented on a GPU. For the following tests, we ran on an Athlon 5600+ with an Nvidia GeForce When rendering 3D geometry, the significantly higher quality of our result (e) as opposed to the raw depth (d) is very obvious. Our algorithm also does not erroneously copy intensity texture from the checkerboard or the book cover into actually planar geometry as it happens with MRF (f) and JBU (g) upsampling (zoomed to red area in (d) for better visibility).

8600 GTS upsampling from 176 × 144 depth data to 800 × 600 resolution. The method scales well with the number of stream processors and even higher frame rates would be feasible with more recent GPUs. Although a similar performance range could be reached with normal JBU, our method produces better quality, as shown before.

As shown in the previous section, the output quality of MRF-based upsampling [START_REF] Diebel | An application of markov random fields to range sensing[END_REF] is comparable to the JBU result and therefore inferior to our algorithm on highly noisy data. In addition to this quality difference, we in practice also see a performance advantage for our filter. In their paper, Diebel et al. use an iterative solver to find the MAP upsampled depth values based on a MRF. To obtain an as fair as possible comparison, we implemented their error metric and its gradient computation on the GPU to perform a runtime analysis. We also used an implementation of the error metric and gradient computation on a cpu solved with an iterative L-BFGS-B solver [START_REF] Byrd | A limited memory algorithm for bound constrained optimization[END_REF] to derive depth results for the following quantitative comparison section.

To find an optimal set of MRF parameters, we regularly sampled the space of each parameter, optimized for the upsampled depth maps, and kept the parameter combination which yielded the minimal root mean square error with respect to a ground truth result. With the so-found optimal set of MRF parameters, it consistently took over 150 iterations to converge to a solution when ran upon upsampling several scenes from the Middlebury dataset. When performing the error metric and gradient computation on the GPU along with appropriate loading of data to and from the GPU, it took on average 755 ms to compute 150 iterations. This assumes connection of the GPU computation to our CPU solver, but the time is absent of the time actually required by the solver to compute new gradient directions. In the event of a full GPU solution, data would not have to transfer from the GPU, at which point the iterative runtime of the gradient computation is 279 ms. This value is also absent of the time required for a GPU based solver to find a new gradient direction. In contrast, the GPU implementation of our NAFDU method averages 49 ms, which includes transfering data to and from the GPU. The required number of iterations shows that even with all possible GPU optimizations the runtimes of our algorithm, that produces better results anyway, will be hard to match.

Finally, we would like to note that our approach performs much more efficiently than the multi-plane bilateral filtering and upsampling method of Yang et al. [START_REF] Yang | Spatial-Depth Super Resolution for Range Images[END_REF]. Although their method reportedly produces higher quality results than MRF or JBU methods, it will also fall short to prevent texture copying. Furthermore, since it requires many iterations of a bilateral filter at each time step, it is infeasible for real-time applications.

Quantitative Accuracy Comparison

To illustrate the working range of our NAFDU filter, we evaluate its performance also in the case of upsampling of depth data with little noise. To this end we quantitatively compare the output of our algorithm to the results obtained with MRF-upsampling. As test data we employ the Cones, Teddy and Venus scenes from the Middlebury stereo data set which also provides ground truth depth maps in addition to the intensity images, Fig. 5. The ground truth depth maps were downsampled by two, four, and eight times to use as input into both methods.

The comparison of root-mean-squared errors in Table 1 shows that under global error measure our method performs at least as good as other related methods from the literature when upsampling low noise depth maps. However, an inspection of individual parts of the upsampled images shows that the MRF approach still exhibits areas with texture copy (red arrows in Fig. 5(d)), whereas our upsampled result, Fig. 5(c), does not show this. It is thus fair to conclude that our method enables real-time performance, and at the same time delivers as good or even better results as previous methods from the literature.

Discussion and Future Work

In previous sections we showed that our noise-aware upsampling filter allows us to produce high-resolution noise-reduced depth data in real-time even with a highly noisy TOF camera. The real-time requirement, however, makes some simplifying assumptions necessary for which we would like to develop improved alternatives in the future. First, our filter switches between its two operating modes using a fundamentally heuristic model that requires manual parameter setting. In future, we plan to investigate ways to learn the correct blending function from real data. Also, our 2D filter is defined in the image plane of the sensor, and we plan to research if it is feasible to define it in a local 3D domain which may enable further improvement of our results. Additional testing would be useful to determine the limits of effective spatial resolution increase; we have focused on demonstrating the techniques ability to improve range data to video resolutions.

Furthermore, some stepping artifacts in the results are due to 8-bit quantization which we will resolve in future by moving to a full floating point pipeline. Finally, we would like to note that our current warping-based alignment may lead to non-exact depth and video registration in some areas, which may explain remaining blur in our results around some actually sharp depth edges. An ideal hardware solution would have the video and depth sensors record through the same optics which would greatly facilitate alignment. 1. RMSE quantitative comparison on depth map pixel values against the MRF method using the Middlebury data set. Raw denotes the RMSE of the downsampled raw depth map. Note that our method produces results similar to MRF on these less challenging data, and in two cases (bolded) performs clearly better.

Conclusion

In this paper we presented a new noise-aware upsampling filter that enables us to fuse in real-time the data from a low-resolution noisy depth sensor and a high resolution video camera. Real-time fusion enables us to upsample the low resolution 3D data to the image resolution of the video sensor while preserving features, reducing random noise, and eliminating erroneous texture embossing artifacts. We have shown that the results of our method exceed the reconstruction quality obtainable with related methods from the literature. Using the parallel stream processing power of a modern graphics processor, the construction of a high-resolution real-time 3D video camera is feasible.

Fig. 1 .

 1 Fig.1. A high resolution color image (a) and low resolution depth map (b) upsampled using conventional JBU (c). Note the copying of the intensity texture into geometry. The lettering of the words 'reusable' and 'recyclable' can be visually seen in the depth map upsampled when noise is not taken into account. The visible text in the depth is even more apparent when the range information is rendered as 3D geometry (d) (zoomed in for clarity).

Fig. 2 .

 2 Fig. 2. (a),(b) Cross-sections of depth maps (conceptual illustration). (a) In regions of low depth gradient, noise contamination of the raw data (blue) in the pink highlighted region can cause the difference between the max and min depth values to be high; however, the difference in the blurred depth input (red) remains small. (b) When an actual depth discontinuity occurs such as in the pink highlighted region, the difference between the max and min depth values is noticeable in the both the noisy raw data (blue) and the blurred depth (red). (c) We blend case I and case II of our filter using a sigmoid function. The width of the transition region can be increased by descreasing epsilon as shown from the red line to the blue line.

Fig. 3 .

 3 Fig.3. Our technique is applied to depth maps (top) which can be reconstructed as 3D geometry (bottom). Taking a low resolution depth image (a), we are able to upsample to a high resolution depth map (b). We can then apply the texture from the color image (c) to the reconstructed 3D geometry (f). The resolution, quality and detail level of the high-res geometry (e) is much higher than of the raw low-res depth (d).

Fig. 4 .

 4 Fig.4. From a video image (a) and low resolution depth map (b) we create a depth map at video resolution (c). When rendering 3D geometry, the significantly higher quality of our result (e) as opposed to the raw depth (d) is very obvious. Our algorithm also does not erroneously copy intensity texture from the checkerboard or the book cover into actually planar geometry as it happens with MRF (f) and JBU (g) upsampling (zoomed to red area in (d) for better visibility).

Fig. 5 .

 5 Fig. 5. Venus scene from the Middlebury data set. We applied our method (c) and MRF upsampling (d) to the four-times downsampled ground truth depth (b). While both methods reconstruct high-frequency content like edges faithfully, in contrast to our result, the MRF result exhibits slight texture copy in the areas marked with red arrows (best visible in electronic version).


O Deniz

M Castrillon

J Lorenzo

M Hernandez

L Anton

G Bueno

Eyewear Selector

The widespread availability of portable computing power and inexpensive digital cameras is opening up new possibilities for retailers. One example is in optical shops, where a number of systems exist that facilitate eyeglasses selection. These systems are now more necessary as the market is saturated with an increasingly complex array of lenses, frames, coatings, tints, photochromic and polarizing treatments, etc. Research challenges encompass Computer Vision, Multimedia and Human-Computer Interaction. Cost factors are also of importance for widespread product acceptance. This paper describes a low-cost system that allows the user to visualize different spectacle models in live video. The user can also move the spectacles to adjust its position on the face. Experiments show the potential of the system.

Introduction

The widespread availability of portable computing power and inexpensive digital cameras are opening up new possibilities for retailers in some markets. One example is in optical shops, where a number of systems exist that facilitate eyeglasses selection. These systems are now more necessary as the market is saturated with an increasingly complex array of lenses, frames, coatings, tints, photochromic and polarizing treatments, etc. [START_REF] Roberts | Modern dispensing tools. options for customised spectacle wear[END_REF]. The number of clients can grow only if the selection process is shortened or automated. A number of deployed systems have already demonstrated that eyeglass selectors can increase sales and customer satisfaction.

From a research viewpoint, such systems represent an interesting application of Computer Vision, Multimedia and Human-Computer Interaction. Computerbased systems for eyeglasses selection can be roughly classified according to a) use of live video or still image, and b) 3D or 2D-based rendering.

With still images, two options are possible. Some systems use a photo of the user without spectacles and then superimpose models on the image. Other systems simply take photos of the users wearing the different spectacles, allowing them to select the frame they like by direct comparison of the captured images. The use of still images is particularly convenient for web-based software. A number of sites are currently available that allow the user to upload his/her photo and see the spectacles superimposed on it. Some systems can automatically extract facial features from the picture. In most of them, however, the user has to mark the pupils in the photo.

3D systems model the user's head and have the advantage that a depiction can be rendered from different viewpoints, see for example [2,[START_REF]Visionix: 3DiView 3D virtual try-on[END_REF]. 2D-based rendering does not work well for large out-of-plane rotations. 3D systems can also be of great help to opticians, as they can take measurements needed to manufacture the frames. However, 3D systems use special hardware and computing power, which can make them too expensive for most optical shops.

Most commercial 2D systems use still images, see for example [START_REF]ABS: Smart Look[END_REF][START_REF] Zeiss | Lens Frame Assistant[END_REF][START_REF] Co | Camirror[END_REF]. This paper describes a 2D live-video eyeglasses selection system. Live video has an advantage over the use of static photos. Even if the user remains practically still, the experience is more realistic: other people near the user appear on the image, spectacles can be placed on the face by the user, etc. Live video effectively creates the illusion of a mirror. The paper is organized as follows. Section 2 gives an overview of the system. Sections 3 to 5 describe the modules in detail. Experiments are shown in Section 6. Finally, the main conclusions and ideas for future work are outlined.

System overview

The hardware system has the following components: a Windows box and two Sony FCB cameras with motorized zoom, focus, white balance and shutter speed. The cameras are placed together on top of the screen (either a computer monitor or a projector can be used). Camera 1 has a resolution of 384x288 pixels and makes no use of the zoom, while Camera 2 (352x288 pixels) uses a (fixed) zoom such that only the user's face appears in the image, see Fig. 1. Camera 2 only captures gray scale frames. The monitor displays the full-screen live video of Camera 1 with overlaid spectacles and on-screen buttons for showing the Previous/Next spectacles. The software system has the following modules: homeostatic image stabilization, face and eye detection and spectacle management. The first module tries to keep image characteristics stable by using the motorized parameters of the cameras. The face and eye detection module localize the position of the user's eyes. The spectacle management module is in charge of overlaying spectacles and controlling spectacle fitting and on-screen buttons. The following sections describe the modules in detail.

Homeostatic Image Stabilization

Homeostasis is defined in the Merriam Webster dictionary as "a relatively stable state of equilibrium or a tendency toward such a state between the different but interdependent elements or groups of elements of an organism, population, or group". The state of equilibrium is normally related to the survival of the organism in an environment. Organisms are endowed with regulation mechanisms, generally referred to as homeostatic adaptation, in order to maintain this state of equilibrium. This idea has been used by some authors for building systems that carry out their activity in a complex environment [START_REF] Breazeal | A motivational system for regulating human-robot interaction[END_REF][START_REF] Gadanho | Robot learning driven by emotions[END_REF]. Arkin and Balch in their AuRA architecture [START_REF] Arkin | AuRA: Principles and practice in review[END_REF] propose a homeostatic adaptation system which modifies the performance of the overall motor response according to the level of internal parameters such as battery or temperature. Another work which includes a homeostatic adaptation mechanism is the proposal of Hsiang [START_REF] Hsiang | Integrated planning and control of mobile robot with self-organizing neural network[END_REF] who introduces it to regulate the dynamic behavior of the robot during task execution.

In most computer vision systems, performance heavily depends on the quality of the images supplied by the acquisition subsystem. Face detection systems that make use of the skin color depend on the white balance, tracking systems based on edge detection depend on image contrast and so on. On the other hand, image quality is affected by environmental conditions, namely lighting conditions or distance from the object of interest to the camera. The typical scenario for an eyewear selection system is an optical shop, where environmental conditions change significantly throughout the day. Homeostatic adaptation will try to compensate for these effects on the image by making use of the adjustable parameters of the cameras. Since the proposed system does not have a body, we simulate the physiological changes that influence the homeostasis mechanism. Cañamero [START_REF] Cañamero | Modeling motivations and emotions as a basis for intelligent behavior[END_REF] proposes synthetic hormones to imitate physiological changes. This approach was adopted in our system by implementing synthetic hormones that reflect the internal state of the vision system (Fig. 2).

The internal state of the vision system is represented by four hormones associated to luminance (h luminance), contrast (h contrast), color constancy (h whitebalance) and size of the object (h size), see Figure 2. The homeostatic mechanism will be in charge of keeping this internal state into a regime which will allow the system to operate with acceptable performance. An important element in a homeostatic mechanism is its adaptive aspect. When the internal state of the body is too far away from the desired regime, the homeostatic mechanism must recover it as soon as possible. The adaptive response of the homeostatic mechanism is governed by the hormone levels which are computed from the controlled variables by means of a sigmoid mapping. In this way, we can implement adaptive strategies more easily in the drives since the hormone levels that define the normal and urgent recovery zones are always the same independently of the values of the controlled variables. Some methods that can be used to compute these variables are described in [START_REF] Lorenzo | Introduction of homeostatic regulation in face detection[END_REF].

Face and Eye Detection

Several approaches have recently appeared tackling reliable face detection in real time [START_REF] Li | Statistical learning of multi-view face detection[END_REF][START_REF] Schneiderman | A statistical method for 3d object detection applied to faces and cars[END_REF][START_REF] Viola | Robust real-time face detection[END_REF], and making face detection less environment dependent. Cue combination usually provides greater robustness and higher processing speeds, particularly for live video stream processing. The face detection system used in the selector (see [START_REF] Castrillon | On Real-Time Face Detection in Video Streams[END_REF]) integrates, among other cues, different classifiers based on the general object detection framework by Viola and Jones [START_REF] Viola | Robust real-time face detection[END_REF], skin color, multilevel tracking, etc. The detection system provides not only face detection but also eye location in many situations. This additional feature reduces the number of false alarms, due to the fact that it is less probable that both detectors, i.e., face and eyes, are activated simultaneously with a false alarm.

In order to further minimize the influence of false alarms, we extended the facial feature detector capabilities, locating not only eyes but also the nose and the mouth. For that reason, several Viola-Jones' framework based detectors have been computed for the chosen inner facial elements. Positive samples were obtained by annotating manually the eye, nose and the mouth location in 7000 facial images taken randomly from the Internet. The images were later normalized by means of eyes information to 59 × 65 pixels, see Figure 3 The facial elements detection procedure is only applied in those areas which bear evidence of containing a face. This is true for regions in the current frame, where a face has been detected, or in areas with detected faces in the previous frame. For video stream processing, given the estimated area for each feature, candidates are searched in those areas not only by means of Viola-Jones' based facial features detectors, but also by SSD(sum of squared differences)-tracking previous facial elements. Once all the candidates have been obtained, the combination with the highest probability is selected and a likelihood based on the normalized positions for nose and mouth is computed for this combination, see Fig. 3.

The face and eye localization system described above works with images provided by Camera 1. The zoom camera (Camera 2) is used to capture the user's face with larger resolution than Camera 1. This can potentially provide a more precise and stable localization. Both eyes are searched for in the images taken by the zoom camera. A Viola-Jones detector is used along with tracking of eye patterns. Complex eye localization methods were discarded in order to keep an acceptable frame rate of the whole system. As the spectacles will have to be superimposed in the images taken from Camera 1, the localizations found in each Camera-2 frame have to be mapped onto the Camera-1 frame. Whenever an eye pair localization is obtained, the eye patterns in those localizations are scaled down. The scale factor is the ratio of intereye distances found in frames of the two cameras. The scaled eye patterns are then searched for in the images captured by Camera 1. This search is carried out in the vicinity of the last eye pair localization obtained for Camera 1. Fig. 4 shows the process.

Spectacle Management

Spectacles are superimposed on Camera 1 images through alpha blending. This process is basically a mixing of two images, with the mixing weights given by a third image. The models are made up of two images: the spectacles and the alpha channel. The alpha channel defines the zones of the spectacles that are translucent (i.e. the mixing weights). The spectacle models were obtained by taking frontal photographs of real spectacles of a local optical shop. The pictures were cropped and the alpha channels extracted using image editing software.

Spectacle models are scaled according to the intereye distance, rotated, and finally placed on screen according to the eye midpoint. Blending is performed only in the affected image region. Note that eye localization has an inherent error, which is also present in the midpoint. The eye midpoint has to be obtained robustly. The spetacles should move with the face, otherwise the rendition will appear unrealistic. Thus, a Lucas-Kanade pyramidal tracker [START_REF] Bouguet | Pyramidal implementation of the Lucas Kanade feature tracker[END_REF] tracks strong corners within the face region. The average displacement vector of the tracking points is used in each frame to correct the displacement of the eye midpoint. Spectacle models are stored as images. The center of the spectacle image is placed on the eye midpoint. This may lead to undesired results if the spectacle image is not well centered. Horizontal centering is not difficult to achieve, though the vertical center is subjective. Besides, each user's facial characteristics may require different placements over his/her nose. In order to tackle this, spectacle placement gesture detection was added to the system.

Considering the unrestricted context of this application, in terms of hands gestures the use of multiple or complex detectors would produce an approach not suitable for real time processing. In consequence, we have chosen the skin color approach for faster processing. However, instead of using a predefined color space definition, the information obtained from the face blob (see the previous section) is used to estimate the an histogram-based skin color model for that individual. The skin color model is used to locate other skin-like blobs in the image, and in consequence to find the hands for a given face.

The spectacle placement algorithm is based on detecting the placement gesture on the skin-color image. Along the vertical sides of the face rectangle a series of small lateral rectangles are considered, see Fig. 5. Their size is proportional to the size of the detected face, considering anthropomorphic relations. The skin-color image is softened using a Gaussian filter with an aperture that equals the detected face width. Thus, isolated pixels and small blobs are removed, while the face and hand blobs create consistent high-valued regions. The hand vertical position is given by the position of the rectangle R containing the highest sum of skin-color pixels (the integral image is used to speed up sums of skin pixels). However, the hand must be in contact with the head. In order to check the "touching-the-head" condition, pixel values are analysed in R. Skincolor continuity is checked from the face side through half the width of R. Every column should contain at least one pixel with a high enough skin-color value (32 on normalized conditions). Otherwise, the hand may be still approaching the face or leaving it. Once the hand is detected as "touching the head", its relative displacement is used to move the spectacles upward and downward. When the hand no longer touches the head the final relative displacement is stored with the current spectacle model. Fig. 6 shows a sequence in which the user is fitting the spectacles.

Experiments

The face detection system was tested with video sequences. Seventy-four sequences corresponding to different individuals, cameras and environments with a resolution of 320x240 were recorded. They represent a single individual sat and speaking in front of the camera or moderating a TV news program. The face pose is mainly frontal, but it is not controlled, i.e. lateral views and occlusions due to arm movements are possible. The eyes are not always visible. The total set contains 26338 images.

In order to test the detectors performance, the sequences were manually annotated, therefore the face containers are available for the whole set of images. However, eye locations are available only for a subset of 4059 images. The eyes location allows us to compute the actual distance between them, which will be referred below as EyeDist. This value will be used to estimate the goodness of eye detection.

Two different criteria have been defined to establish whether a detection is correct: a) Correct face criterium: A face is considered correctly detected if the detected face overlaps at least 80% of the annotated area and the area difference is not doubled, and b) Correct eye criterium: The eyes of a face detected are considered correctly detected if for both eyes the distance to manually marked eyes is lower than a threshold that depends on the actual distance between the eyes, EyeDist.

Table 1 shows the results obtained after processing the whole set of sequences with different detectors. The correct detection ratios (TD) are given considering the whole sequence, and the false detection ratios (FD) are related to the total number of detections. Rowley's detector is notably slower than the others, but it provides eye detection for the 78% of detected faces, feature which is not considered by Viola-Jones' detector. As for the face detector, it is observed that it performs more than twice faster than Viola-Jones' detector, and almost ten times faster than Rowley's. Speed was the main goal in our application, the face detector is critical for the live-video selector.

Rowley [START_REF] Rowley | Neural network-based face detection[END_REF] Viola-Jones [START_REF] Viola | Robust real-time face detection[END_REF] face detector used here [ Face detection depends heavily on skin color and pattern matching to detect faces. Therefore, experiments were carried our to study the influence of luminance and white balance on face detection performance. Fig. 7 shows the values of the h luminance and h whitebalance hormones along with the face detection rate for an individual moving in front of the camera. The dashed lines represent the changes in the environmental condition (lighting). When the system starts the detection rate is high and it decreases slightly when more lights are switched on (30-57 secs.). When the lights are switched on, both the h luminance and h whitebalance hormones go out of their desired states but the homeostatic mechanism recovers them after a delay, larger for the h whitebalance hormone than for the h luminance one. In the experiment the homeostatic mechanism is deactivated after 70 seconds, so when the conditions change again the state of the hormones is not recovered and the performance of the system decreases.

In order to measure the localization capabilities of the system, seven video sequences were recorded in which a subject moves his head, from almost no motion to extreme movements. The eyes of the subject were manually located so as to have ground truth data. Table 2 shows the number of frames and amount of motion of each video sequence. In sequences 6 and 7 the head movements were exaggerated for testing purposes and they do not represent a typical situation.

The effect of the zoom camera is shown in Table 3. The first thing to note is that localization errors using information from the zoom camera (Camera 2) are larger than those of Camera 1. Despite the higher resolution available in the zoom camera, this finding is explained by two reasons: a) The eye localizer of the second camera is much simpler than the face detector (though 80% faster) and, b) Head motion often make the eyes go out of the zoomed image. As Camera 1 data are better for localization, they have priority in the combination. That is, whenever localization data are available from the face detector they are used to place the spectacles on screen. Data obtained with Camera 2 are only used when the face detector can not provide an eye pair localization.

The combined use of information of the two cameras does not improve either one of them alone, except in the cases marked with bold in the table. However, the use of the second camera is advantageous in terms of number of frames with available eye localization, see Table 4. This allows the spectacles to remain longer on screen, even if the individual is moving. In another experiment, facial tracking (see Section 4) as a localization aid was tested. Table 5 shows that its use is specially advantageous when there is large head motion. The tracker was is reinitialized every 60 frames in order to avoid excessive drifting.

Conclusions

The affordability of cameras and portable computing power is facilitating the introduction of computer vision in optical shops. Retailers are particularly interested in automating or accelerating the selection process. Most commercially available systems for eyewear selection use static pictures. This paper describes a patent pending live-video eyeglasses selection system based on computer vision techniques. The system, which runs at 9.5 frames per second on general purpose hardware, has a homeostatic module that keeps image parameters controlled. This is achieved using cameras with motorized zoom, iris, white balance, etc. This feature can be specially useful in environments with changing illumination and shadows, like in an optical shop. The system also has a face and eye detection module and a spectacle management module. Further improvements are possible like adding zoom, mirror-like spectacles, etc. Also, it may possible to eliminate the second camera, which would allow the system to be adapted to commodity hardware. Modern laptops, for example, include an integrated webcam and sufficient computing power to be used as a low-cost, portable, eyewear selector. This would also open the possibility of using the system via Internet.

Fig. 1 .

 1 Fig. 1. Left: image of Camera 1. Right: image of Camera 2.

Fig. 2 .

 2 Fig. 2. Elements of the homeostatic adaptation mechanism.

  (left). Five different detectors were computed: 1-2) Left and right eye (18 × 12 pixels), 3) eye pair (22 × 5), 4) nose (22 × 15), and 5) mouth (22 × 15).

Fig. 3 .

 3 Fig. 3. Normalized face sample and likely locations for nose and mouth positions after normalization.

Fig. 4 .

 4 Fig. 4. Eye localization with the zoom camera.

Fig. 5 .

 5 Fig. 5. Spectacle placement gesture detection. Left: the biggest white rectangle represents the face area.

Fig. 6 .

 6 Fig. 6. Spectacle placement sequence.

Fig. 7 .

 7 Fig. 7. Face detection rate with and without homeostatic mechanism.

Table 1 .

 1 Results (%) for face and eye detection processing using a PIV 2.2Ghz.

	16]

Table 2 .

 2 Video sequences used in the experiments. N=number of frames, V=variance of eye position.

		Video sequence
	1 2	3	4	5	6	7
	N 126 175 176 148 119 129 208
	V 8.2 11.1 11.3 27.1 37.7 120.8 164.4

Table 3 .

 3 Eye localization errors. REE=right eye error, LEE=left eye error. Standard deviations between parentheses.

	Video	Camera 1	Camera 2	Combination
	sequence REE	LEE	REE	LEE	REE	LEE
	1	1.68(1.01) 1.53(0.87) 3.08(2.41) 2.85(1.78) 1.61(1.01) 1.53(0.87)
	2	2.78(2.91) 2.81(1.21) 5.44(5.17) 4.27(3.25) 2.71(2.92) 2.73(1.25)
	3	2.39(1.00) 2.03(0.80) 1.38(0.93) 2.37(1.23) 2.36(0.98) 2.03(0.78)
	4	1.86(1.21) 2.69(1.41) 2.96(2.94) 2.22(1.43) 1.99(1.19) 2.40(1.27)
	5	2.63(1.39) 2.37(1.16) 2.48(1.57) 2.69(1.78) 2.54(1.34) 2.33(1.51)
	6	3.03(2.75) 2.64(1.76) 6.82(7.86) 9.81(10.03) 6.14(7.22) 7.79(9.27)
	7	2.29(1.24) 2.22(1.55) 5.36(4.82) 7.91(11.48) 2.82(2.13) 4.81(9.93)

Table 4 .

 4 Number of frames with available eye pair localization.

		Video Camera 1 Camera 2 Combination
		1	124/126 109/126	124/126
		2	173/175 57/175	173/175
		3	174/176 126/176	174/176
		4	76/148	81/148	111/148
		5	100/119 89/119	113/119
		6	51/129	71/129	83/129
		7	165/208 157/208	203/208
	Video No tracking With tracking Improvement
	1	1.451(0.677) 1.307(0.648)	9.91%
	2	2.286(1.515) 2.189(1.467)	4.25%
	3	1.505(0.626) 1.413(0.732)	6.13%
	4	2.112(1.147) 1.775(0.951)	15.99%
	5	2.079(1.057) 2.037(1.062)	2.00%
	6 6.835(12.112) 7.026(11.346)	-2.80%
	7	3.349(6.230) 3.334(5.788)	0.43%

Table 5 .

 5 Squared errors of the eye midpoint position. Here the tracking of facial points was given a weight equal to the no-tracking eye midpoint localization.

Acknowledgments

This work was partially supported by Project ULPGC07-008 from Universidad de Las Palmas de Gran Canaria.