Keywords: Computer Vision, Multiple View Geometry, Transparent Surfaces, Photometry, refraction, Reflection, Mirrors, Light-path Triangulation, Reconstruction, Fresnel Equations. 1

We then focus our attention on using both geometric and photometric cues for reconstructing transparent 3D surfaces. We show that in the presence of known illumination, we can recover the shape of such objects from single or multiple views. The cornerstone of our approach are the Fresnel equations, and we both derive and analyze their use for 3D reconstruction. Finally, we show our approach could be used to produce high quality reconstructions, and discuss other potential future applications.

refractive sphere. Note that the refractive sphere represents two refractions. Our goal is to reconstruct the shape of these specular objects (without assuming a model like a sphere), using information from the surrounding environment and images. . . . . . . . [START_REF] Blake | Geometry from specularity[END_REF] 1.6 Depiction of the specular surface reconstruction problem from [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF]. Window", a conic that represents the horizon of the outside world. Photo courtesy gerb's photostream, http://www.flickr.com/photos/gerb/196296131/ . . . . . . . . . . [START_REF] Ding | Recovering shape characteristics of near-flat specular surfaces[END_REF] 3.1 (a) Description of the general theory behind our approach. While the acquisition is similar to that of Kutulakos et al [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF], we also include radiance in our measurements (depicted by the changing color of rays while they travel from the illuminant to the 3.8 Simulation results for photometric error and refractive index mis. We simulated a curved object, and captured radiometric information from a camera according to Fresnel theories. We then added noise to this data (left), or used a slightly different refractive index (right) to reconstruct the surface. Notice that for noise, although the noise was added to the radiometric information that was recorded, the radiance ratio is considerable stable w.r.t the noise (since the denominator in the radiance ratio, which is the illuminant, has a high value). This results in the reconstructions being reasonably stable. However, when the refractive index is changed, depending on the angle of incidence and the depth-normal relationship, the reconstruction accuracy changes. While here we list the worst possible scenario, when the camera is close to the object, moving the camera further away like in the case of our datasets considerably strengthens the robustness of our results. Notice also, that in case we underestimate the refractive index the depth variation of the surface remains more or less intact, while its mean distance from camera increases. Thus, even with an underestimate it is possible to get reasonable results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 vi 3.9 Comparison of simulations between our approach and [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF]. In LP-1, corresponding 3D points are normally close to the object, which results in increased error in triangulation and normal estimation. Note that in the same scenario, we have much better normal information because of photometric information. While LP-2 is robust because correspondences are far away, its highly impractical since use of LCD's for correspondence is problematic (because of light fall-off, scattering etc.). Details in text. . . . . . . . . . . What makes the problem so hard is the complex interaction between the various physical elements of the scene (materials, light etc.) and the fundamental limitations of the camera (perspective projection, dynamic range etc.). One of the main components of image understanding is 3D reconstruction, and a lot of significant progress has been made in it in the last few decades. We focus on 3D reconstruction of specular surfaces in this thesis. Surfaces with material properties that help them reflect or refract an incoming ray of light in a very narrow solid angle of directions are called specular surfaces (Figure 1.1). There are man-made and naturally occuring specular sufaces;

A
they are ubiquitous and also widely used in industry. In this chapter, we first give a brief overview of 3D reconstruction with a bias towards specular surfaces (transparent and reflective). We argue that while a lot of progress has been achieved in "generic" 3D reconstruction, specular objects have been largely left out because their peculiar appearance does not sit well with most reconstruction algorithms. However, considering their ubiquitous nature (in fact, the camera lens itself is one example) any 3D reconstruction system will have to eventually deal with them, before being deployable on a large scale in the real world.

We look at physical properties of specular surfaces and devise algorithms that utilize these properties to reconstruct them from multiple images in the forthcoming chapters. While many existing works employ a variety of other sensors for this reconstruction task, we consciously choose to avoid them in our works. This is to facilitate a deeper understanding of the physics behind specular image formation, combined with an intention to have a wider set of scenarios where our algorithms and understanding might prove useful/insightful.

3D Reconstruction

3D reconstruction is the task of creating 3 dimensional models that faithfully comply with a set of 2D images of the object(s)/scene involved. Several approaches like structure-from-motion (sfm), stereo, photometric stereo are popular in the computer vision literature. While sfm and stereo are widely used for outdoor scene reconstruction, photometric stereo generally involves indoor scenes captured in controlled lighting environments. Current state-of-the-art 3D reconstruction algorithms perform remarkably well on both small and large scales. In fact, while reconstruction of entire cities has been recently attempted on the large scale side, reconstruction of mesostructures and extremely challenging objects like hair have also been attempted successfully.

(a) (b) (c) (d) (e) (f) (g)
One of the main obstacles that 3D reconstruction techniques have come across over the years has been the interaction of light and various materials, especially when it produces appearance varying results across images. In general, 3D reconstruction approaches rely on the Lambertian assumption of surface reconstruction. While in sfm and stereo algorithms, this results in the appearance constancy assumption (which is the basis of feature detection and matching algorithms), photometric stereo directly uses the Lambertian property to determine the relationship between pixel intensities and incident angle of the light rays from the light source. When the Lambertian surface assumption about material property fails, it introduces correspondence issues in sfm and stereo algorithms. Even in photometric stereo algorithms where correspondence is a non-issue, the failure of the Lambertian assumption causes reconstruction problems. To address this issue, various schemes have been proposed

that take into account increasing generalizations of the Lambertian assumption.

In the case of specular objects, the Lambertian assumption is not at all valid. In fact, specularities have been argued to be characteristic of a specular object, and have been used for object recognition.

As we show in the next chapter and as is known previously also, the multiple view geometry of specular surfaces too is very different from non-specular surfaces. This results in traditional sfm and stereo algorithms ignoring specular objects altogether, while traditional photometric stereo approaches have to undergo considerable modification before application to specular surfaces. Even 3D sensors like the ones used in Kinect perform poorly in the presence of specular surfaces.

One other challenge for specular objects is the modeling of their appearance. The appearance of specular objects not only depends on the material, but also on their shape, the incident illumination and the appearance of the scene in which they are placed. This makes it extremely difficult to use the appearance model of one specular object for the reconstruction of another. In fact, several different BRDF (Bidirectional Reflectance Distribution Function) models have been proposed to reconstruct different kinds of specular and near-specular surfaces; there are hardly any non-parametric approaches.

To summarize, two main obstacles prevent 3D reconstruction algorithms from being applied to specular surfaces and especially transparent surfaces.

• The Lambertian reflectance assumption is violated. This affects correspondence estimation in sfm/stereo approaches, and affects pixel intensity measurements in photometric stereo.

• Specular surface appearance depends on a variety of factors like object shape, incident illumination and surrounding environment. This results in the inability to produce a specific 3D reconstruction algorithm that applies to all specular surfaces.

In the presence of these problems, it makes sense to look at algorithms that are specifically designed for specular surfaces. Our belief is that better image understanding could be obtained by investigating approaches that rely on a minimal amount of hardware, and thus we focus on a combination of geometry and photometry for specular surface reconstruction in this thesis. In fact, already some preivous works sharing our perspective of image understanding can be seen in [START_REF] Aswin | Image invariants for smooth reflective surfaces[END_REF] where the authors propose photometric invariants useful for detection and recognition of transparent surfaces that are based on geometric properties of smooth surfaces. In the next section, we give a brief history of the various approaches employed for reconstruction of specular surfaces to set the context for our works. Additionally, we list and briefly outline the various contributions in this thesis.

History with a Specularity Bias

The history of specular object reconstruction is divided in two parts. On one hand, the reconstruction (calibration) of catadioptric cameras has been important for image acquisitions of a specific kind used in robotics, panorama creation etc. The reconstruction of transparent specular surfaces was first initiated in photogrammetry for use in underwater photography [START_REF] Maas | New developments in multimedia photogrammetry[END_REF]. More recently, in computer vision transparent object reconstruction has been attempted using physical properties of transparent surfaces like their specular property and polarization.

The Case of Mirrors

.2 shows some catadioptric systems, the first class of mirrors that were used in computer vision. Their primary usage was to enhance the field of view of conventional cameras, to obtain wide fields of view (FOV) [START_REF] Baker | A Theory of single-viewpoint catadioptric image formation[END_REF]. Using a mirror with a camera required "calibration" of the mirror, which in turn meant knowing the shape parameters and position of the mirror w.r.t the camera. Thus calibration implicitly involved 3D reconstruction of the surface, although in a restricted sense since the shape model (hyperbolic, parabolic etc.) was already known in these cases. The primary purposes of using catadioptric systems was for 1) Panoramic imaging [START_REF] Kar-Han Tan | Multiview panoramic cameras using mirror pyramids[END_REF] and 2) SLAM applications [START_REF] Kim | Visual SLAM by Single-Camera Catadioptric Stereo[END_REF]. Generic specular surface reconstruction was first proposed in [START_REF] Blake | Specular stereo. International Joint Conference on Artificial Intelligence[END_REF], where the authors considered reconstructing the surface by observing highlights in stereo images. In general, the first set of reconstruction algorithms noted that appearance of specularities in different images could contain clues to local shape information [START_REF] Blake | Geometry from specularity[END_REF]. These works were predated by the seminal work of [START_REF] Koenderink | Photometric invariants related to solid shape[END_REF], which outlined the different photometric invariants related to reflective surfaces. Since specularities are produced by illumination reflected off the specular surface, photometric stereo methods [START_REF] Ikeuchi | Determining surface orientations of specular surfaces by using photometric stereo method[END_REF] were also extended to specular surface reconstruction. Finally, since it was realized that motion of specularities contain information about local shape [START_REF] Blake | Geometry from specularity[END_REF], a natural extension was to use not just specularities but motion of all reflected points on the surface of a mirror for its reconstruction. This lead to a "shape from distortion motion" area of approaches [1; 115; 118].

Catadioptrics One of the earliest uses of spherical mirrors was demonstrated in [START_REF] Nayar | Sphereo: determining depth using two specular spheres and a single camera[END_REF], where the authors used two spheres of known radii, placed at known positions w.r.t the camera in order to compute the depth of the world using a single image. This was one of the first uses of a non-single viewpoint camera for depth estimation, made easy because of the simplicity of a spherical setup and manually measured position estimates. Subsequently, the semilar work of single viewpoint catadioptric cameras [START_REF] Baker | A Theory of single-viewpoint catadioptric image formation[END_REF] generalized the kind of mirrors that could be used in a catadioptric setup while ensuring that the entire system had a single effective viewpoint. Since using such mirrors with cam-eras requires knowledge of their shape and position w.r.t the camera, several "calibration" algorithms were proposed to acquire these parameters [38; 39]. These calibration methods determined the shape parameters, given known shape models and constituted one of the first reconstruction algorithms for mirrors. The single-viewpoint assumption proved useful because it allowed extending traditional multiple view geometry [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF] to the case of catadioptric setups.

This approach was later extended to include non-central viewpoints i.e. camera-mirror setups that were not single viewpoint systems. The advantages of such systems included increased control over FOVs (Field of View) and general better designing flexibility [START_REF] Swaminathn | Non-single viewpoint catadioptric cameras: geometry and analysis[END_REF]. Calibrating such cameras could now involve specific algorithms that took into account shape models [START_REF] Geyer | Paracatadioptric camera calibration[END_REF] or could take the form of general calibration of a system of rays [START_REF] Sturm | A generic concept for camera calibration[END_REF]. While the former approaches performed reconstruction, the latter just reconstructed rays and the eventual shape of the mirror could be obtained by triangulation of captured ray directions and incident ray directions in 3D. A recent survey and taxonomy of non-central camera calibration can be found in [START_REF] El | Non-central catadioptric sensors auto-calibration[END_REF]. While many of the previous approaches could be restructured to get 3D reconstruction of arbitrarily-shaped mirrors [82; 112; 131], reducing the number of 2D-3D correspondences gives an unstable solution [START_REF] Bonfort | Reconstruction de surfaces réfléchissantes à partir d'images[END_REF]. Since it might not be practically feasible to get many images using for the same mirror-camera configuration in many general settings, it might be worthwhile looking at how different mirrors might be reconstructed using piecewise modeling. We investigate this idea in the fourth chapter.

Specular Flow Methods

When a moving camera observes a specular surface, or when a mirrorcamera setup moves in the world, the appearance of the specular surface captured by the camera changes over time [START_REF] Zisserman | The information available to a moving observer from specularities[END_REF]. The optical flow obtained from the multiple images captured in such a setup is called specular flow [START_REF] Roth | Specular flow and the recovery of surface structure[END_REF]. A part of this problem has been known from a long time as Alhazen's Billiard's problem [START_REF] Smith | The remarkable Ibn al-Haytham[END_REF]. The deviation of specular flow approaches from previous attempts at this area was the relaxation of the knowledge of the illumination source [13; 14; 154]. Thus it directly retrieved camera motion and mirror shape by just observing reflections off the surface. While progress has been made in this field, specular shape from flow algorithms still provide results that are below the state-of-the-art when compared to other approaches that assume known illumination sources (or known 2d-3d correspondences, because they serve the same functionality in the case of mirrors) [16; 17; 82]. Some notable approaches in this direction are [1; 118; 143]. In fact, it is useful to note that some of these algorithms are related to non single viewpoint cameras [START_REF] Vasilyev | Dense specular shape from multiple specular flows[END_REF]. It might also be worthwhile investigating whether the difficulty in specular flow based approaches is in any way connected to the instability of pose estimation of a checkerboard pattern in the presence of a generic mirror, when a direct view of the checkerboard pattern is not available [START_REF] Bonfort | Reconstruction de surfaces réfléchissantes à partir d'images[END_REF]. Finally, while specular flow approaches exploit the motion information produced in images and their relationship to mirror shape and world motion, we restrict the focus in this thesis to static instances of specular surface reconstruction approaches. The underlying motivation for us is two-fold

• We believe that reconstruction of surfaces from static setups has not been given much attention.

In fact, many of our approaches just require a single view (not to be confused with single image)

and in some cases, we actually are able to obtain more information about the object shape than traditional 3D reconstruction scenarios (with opaque objects).

• We believe that the understanding obtained from solving and thinking about such setups gives us more insights into the general cases of shape from specular flow. More importantly, the relationship between specular flow and non-single viewpoint cameras is well worth exploring, although in the current thesis this direction is not explored. We however explore related areas that might give insights into the nature of this connection.

The Case of Transparency

Transparent objects are one of the few categories in which reconstruction methods are still in their nascent stage in terms of accuracy, ease of use etc. Traditional shape estimation techniques make assumptions that are rendered invalid when the image formation process of transparent objects is considered. A recent survey [START_REF] Ihrke | Transparent and reflective scene reconstruction[END_REF] indicates the various approaches used to acquire the shape of transparent objects. Figure 1.4 lists the various categories which are considered in this survey. In this section, we give a brief description of some of these approaches, as well as others used in photogrammetry [START_REF] Maas | New developments in multimedia photogrammetry[END_REF].

Active Methods One of the most accurate object shape acquisition techniques in computer vision is structured light projection. Active methods like these use an external stimulus (like projector light)

to modify the image acquired by the camera, in a way that makes it easier to compute the shape of the object of interest. While it is a highly accurate technique in the case of opaque objects, there The different classes of transparent surfaces as described in [START_REF] Ihrke | Transparent and reflective scene reconstruction[END_REF] have only been recent attempts at active "scanning" of transparent surfaces. This is because of the presence of optical phenomenon like sub-surface scatterring, interreflections, scattering (in the case of participating media) etc. Recent approaches like [START_REF] Morris | Reconstructing the surface of inhomogeneous transparent scenes by scattertrace photography[END_REF] have extended active techniques to transparent surfaces by modeling their surface properties so as to isolate their reflective capabilites from other optical phenomenon, and use it for reconstruction.

Polarization Polarization is a physical property of surfaces that has been used for reconstruction purposes. When an unpolarized beam of light is incident on a reflective/refractive surface (not necessarily a mirror), its reflected and refracted components are partially polarized, if the object is dielectric. Since the angle of polarization is directly related to the ratio of incident and reflected/refracted light through Fresnel equations [START_REF]Fresnel equations[END_REF], determining the polarization angle gives useful information about the shape of the object. In fact, earlier approaches have used a camera with a linear polarizer to determine this angle, and subsequently used it for reconstruction [START_REF] Huynh | Shape and refractive index recovery from single-view polarisation images[END_REF]. Recently, many papers have used this approach to reconstruct transparent surfaces [88; 89]. This extension normally requires a relatively sophisticated setup, and both forward and inverse (predicting the shape and comparing it with the image obtained in order to refine the shape) methods have been proposed in this area [7; 116]. Other approaches have also been reported in the literature that have used polarization to reconstruct metallic mirror-like surfaces [START_REF] Morel | Active lighting applied to 3D reconstruction of specular metallic surfaces by polarization imaging[END_REF].

Shape from X While several of the approaches mentioned earlier (like specular flow) could come under the category of shape from X, we choose to put methods that are not directly relevant to this thesis, but still related, in this category. Figure 1.3 (middle) gives one such setup [START_REF] Matusik | Acquisition and rendering of transparent and refractive objects[END_REF]. The authors propose an entire system which segments and starts with a coarse reconstruction of a transparent surface irrespective of its transluscency (several categories in Figure 1.4). This is then refined using the several images obtained with various light sources, as shown in the setup (Figure 1.3, middle image). Some other approaches "acquire" the shape of transluscent surfaces, like the ones depicted in Figure 1.7. In [START_REF] Gu | Compressive Structured Light for Recovering Inhomogeneous Participating Media[END_REF], the authors recover the structure of inhomogeneous media like milk by using compressed sensing based algorithms to analyze the images obtained after structured light is projected onto a setup consisting of the object of interest immersed in media like water.

Photometric Methods

Traditional photometric methods employ a static viewpoint (relative position of camera and object is fixed), and move the light source to generate varying appearance of the object [START_REF] Woodman | Photometric method for determining surface orientation from multiple images[END_REF]. While several assumptions about photometric stereo have been relaxed subsequently, the first application to specular surfaces was done by [START_REF] Ikeuchi | Determining surface orientations of specular surfaces by using photometric stereo method[END_REF]. The approach used in this method considered a distance light source and a single-view approach in which multiple images under changing illumination were obtained. Surface orientations were then recovered using this data. Further extensions to this approach were done either using exemplars [START_REF] Hertzmann | Shape and materials by example: a photometric stereo approach[END_REF], color [START_REF] Mallick | Beyond lambert: Reconstructing specular surfaces using color[END_REF] using Helmholtz theories [START_REF] Zickler | Helmholtz Stereopsis: exploiting reciprocity for surface reconstruction[END_REF] etc. The general idea in photometric stereo approaches has been to consider specular surfaces as surfaces with unique BRDF's, which means that moving away from the Lambertian assumption is necessary [START_REF] Mallick | Beyond lambert: Reconstructing specular surfaces using color[END_REF]. While the previous approaches extend the photometric approach to specular surfaces, they are restricted to mirror like BRDF's. In the presence of transparent surfaces, additional optical phenomena like interreflections, scattering, sub-surface scattering, chromatic aberration are involved [START_REF] Morris | Reconstructing the surface of inhomogeneous transparent scenes by scattertrace photography[END_REF], because of which the photometric data obtained is noisy (with bias). Unfortunately, the general approach of photometric stereo does not provide the scope for removing such errors. In such a case, several approaches like polarization based approaches [7; 88], or robust specularity estimation based approaches [START_REF] Yeung | Adequate reconstruction of transparent objects on a shoestring budget[END_REF] or physical model based approaches that can detect and remove such errors [START_REF] Morris | Reconstructing the surface of inhomogeneous transparent scenes by scattertrace photography[END_REF] have to be used. Note that while these approaches use light sources in order to estimate the shape, some like [START_REF] Morris | Reconstructing the surface of inhomogeneous transparent scenes by scattertrace photography[END_REF] do not use a BRDF based approach to extract surface normals based on the intensity of light image by the camera at a point, and so they can only be related to photometric stereo methods by a distance.

Light-Path Approaches

A light-path is defined as the path taken by a ray of light from the point of emergence till the time when it reaches the camera [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF]. When the scene consists of specular surfaces, this path is piecewise-linear 3D, although it may not be planar (this distinction becomes important in the fourth chapter). Thus specular surface reconstruction could be alternately defined as reconstruction of all the light-paths that pass through the scene and are captured by the camera. While not explicitly stating so, several approaches have attempted this in the past, or have light-path reconstruction as a reasonably simple extension [3; 17; 82; 112; 131]. One of the major problems with this type of approach is that it is inherently local in nature, in the sense that so far, most approaches in the literature focus on reconstructing individual light-paths separately and a global algorithm for reconstructing all light-paths simultaneously with the shape of the object(s) is missing [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF]. The sphere. Note that the refractive sphere represents two refractions. Our goal is to reconstruct the shape of these specular objects (without assuming a model like a sphere), using information from the surrounding environment and images.

authors in [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF] also impose a limit on the number of bounces (number of piecewise linear segments) a light-path can have if its reconstruction should be possible. However, we believe that light-path reconstruction is closely related to non-single viewpoint geometry, and reasoning about both can lead to solutions to cases where there are more than two bounces inolved per light-path. In fact, we extend this limit in some cases in the third and fourth chapter.

Summary

To summarize, there is an extensive literature in specular object reconstruction. While many approaches have focused on extending traditional reconstruction methods to the case of specular surfaces, the most successful methods currently focus on using algorithms tailor made to the characteristics of specular surfaces in order to produce reasonable reconstructions [START_REF] Francken | High quality mesostructure acquisition using specularities[END_REF]. In the next section, we outline the specular reconstruction problem, and give a brief overview of our line of reasoning, while presenting our contributions.

The case of specularity

In this section, we define the quintessential specular surface reconstruction problem considered in this thesis. Figure 1.6 depicts the scenario that we consider, partially, while 1.5 gives two examples of the kind of images we deal with. A known / unknown 3D point p is reflected/refracted off several surfaces, at points v i before being finally imaged as point q at the camera c. This light-path may or may not be accompanied by a radiance measurement which in the case of the transparent surface, would involve its refractive index relative to the outside medium, apart from the angle of incidence of the incoming light ray and the shape (normal) of the object itself. Finally, we can consider a single light-path in isolation or multiple light-paths together. Our objectives are many-fold.

• To derive basic understanding of the interplay between transparency and light-paths. To further relate it to the image formation process and the multiple view geometry involved.

• To use this understanding for the reconstruction of transparent surfaces from multiple images.

To extend this understanding to general specular surfaces (i.e. mirrors too) and further relate them to existing understanding on the specular image formation process.

• To use additional input (like radiance) readily available without additional equipment (polarizers) in order to investigate the extent to which light-path triangulation [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF] may be solved in several cases.

• To identify new problems and scopes of extension of works established in this thesis.

Reconstruction vs Acquisition

Figure 1.7 shows several systems aimed at "acquiring" properties of transparent surfaces. In many cases, like the work of [START_REF] Matusik | Acquisition and rendering of transparent and refractive objects[END_REF], the acquisition involves reconstructing the object involved, while in many others like [START_REF] Srinivasa | Acquiring scattering properties of participating media[END_REF] only the optical properties of the object are desired. Most methods of acquisition are for purposes of using the material in graphics applications [64; 85; 97]. While acquiring properties accurately is a hard process, approaches with graphics applications in mind tend to ignore the image understanding perspective and employ several additional equipment like a fluorescent dye in the case of [START_REF] Hullin | Fluorescent immersion range scanning[END_REF]. However, our argument is that reconstruction based approaches [73; 91] etc. are "minimalist" in their usage of additional equipment because of the need for application to outdoor scenarios where a robot with a camera or a mobile vision platform might not have access to such equipment. We believe that our approaches help us better serve these goals because of the following reasons.

• We model the physical nature of reflection/refraction in order to obtain the 3D reconstruction of scenes. This is exactly the extension of general 3D reconstruction algorithms (since appearance of specular surfaces is reflective/refractive in nature) to the case of specularity.

• 3D reconstruction of objects considered in this thesis results from an understanding of the interactions between light and object material, and their resulting effects on image formation.

Thus, data acquired from our setups, for example, could be used to understanding the nature of interreflections within transparent surfaces.

• As opposed to acquisition techniques, we restrict our study to specular surfaces. This allows us to focus on a particular set of optical phenomenon, while ignoring other complex ones like scattering. This isolation enables better understanding.

Physical Modeling vs Light-Path Modeling

Figure 1.8 shows results from various reconstruction approaches in the literature [73; 88; 91]. These reconstruction approaches can also be divided into two parts, based on the ideas and models they use.

Physical modeling based approaches [68; 88] try to extend non-triangulation based methods (in many cases, variational approaches like [START_REF] Solem | A variational analysis of shape from specularities using sparse data[END_REF]) to the case of specular surfaces. While such approaches also produce encouraging results, we stick to the light-path triangulation perspective for a few reasons.

• Acquisition Ease As we show in the later chapters, our acquisition setups are generally much simpler and data easier to acquire (without additional equipment like polarizers and diffusers) than current physical modeling based approaches.

• Usage of Inputs While being easier on the image acquisition side, we also end up utilizing all properties of specular surfaces (like polarization, radiance ratios etc.) that most of the physical modeling approaches use. In fact, our models might also be categorized as physical models within a light-path framework.

Our Contributions

In the previous sections, we have outlined our reasoning for choosing this perspective for studying specular object reconstruction. In this thesis, we make the following contributions to this line of study.

Chapter 2 In the first part of this thesis, we analyze the multiple view geometry of a refractive planar surface. We consider the case when one or multiple cameras in a medium (for example air) are looking at a scene in another medium (for example water), with the interface between the two media being flat. The case of underwater photography fits this description. Since a perspective projection model no longer fits this scenario, at first we derive the forward projection model and related camera matrix. We show that 3D lines in a scene map to quartic curves in the image in such a scene. An interesting observation about this scenario is that there is a unique image curve for every 3D line in the world, assuming a homogenous refractive index. We then derive multiple view quantities like the fundamental and homography matrices related to this scene and count arguments for tasks like pose estimation of cameras from multiple images. We also show that when the camera is in a heavier medium, the horizon maps to a conic ("Snell's window") which can be decomposed to obtain the parameters of the separating interface.

Chapter 3 Transparent specular objects also modify and re-distribute the light energy incident on their surfaces. The governing model of this re-distribution is represented by Fresnel's equations.

Using this as our cornerstone, we describe a method that combines both geometric and photometric information to do reconstruction of arbitrarily shaped transparent specular surfaces. We show that our approach leads to a very uncomplicated acquisition process, while keeping our approach fairly simple. First, we analyze several minimal cases for shape reconstruction, and derive novel constraints for reconstruction that combine geometric and Fresnel theories about transparent surfaces. Secondly, we illustrate the complementary nature of these cues which helps us gain additional information about the object, which is otherwise impossible to obtain. Finally, we discuss practical aspects of our reconstruction algorithm while presenting reconstruction results on challenging datasets. Our results

show that high quality reconstructions can be achieved in challenging scenarios.

Chapter 4

We then extend our understanding of planar refractive geometry to derive algorithms for recovering the geometry of multiple planar refractive surfaces from a single image. A typical example of such a scenario would be looking through a fishtank, or underwater photography. We propose a simple technique to compute the normal of such surfaces given in various scenarios, by equating our setup to an axial camera. This allows us to fit RANSAC based approaches like the 8-point algorithm for fundamental matrix computation to our case, in a manner similar to axial distortion estimation techniques in the computer vision literature. Interestingly, this approach naturally extends to planar light-paths within axial systems. To this extent we then show that the same model could be used to reconstruct reflective surfaces consisting of multiple parallel planes or axially aligned spheres. We

show encouraging 3D reconstruction results, and analyze the accuracy of results obtained using this approach.

Conclusion

In conclusion, we analyze several different geometric and photometric properties of specular transparent surfaces and explore two ends of a reconstruction spectrum: shape specific reconstruction and shape independent reconstruction. In both cases, we present several novel theoretical and algorithmic contributions with encouraging results. We show how the perspective of light-path triangulation could be extended for better image understanding of specular surfaces. Potential applications include robotics [START_REF] Kim | Visual SLAM by Single-Camera Catadioptric Stereo[END_REF] and graphics [START_REF] Matusik | Acquisition and rendering of transparent and refractive objects[END_REF], applications like specular object detection and recognition [105; 117].

2

Multi-View Geometry of the Refractive Plane

Introduction

One of the ways in which we can group and study light-paths is to consider grouping light-paths that are the projections of a 3D point or line on a camera, after undergoing several reflections, refractions or both. In this chapter, we consider the simplest case of such a scenario where the imaging characteristics of a camera in one medium looking at a scene in a different medium with a planar, transparent interface between the two is studied.

The multi-view geometry resulting from opaque scenes is now well understood, for the case of perspective projection. To some extent, even the insertion of reflective elements has been studied in the area of catadioptrics [130; 132]. The phenomenon of refraction, however, has largely been left un-addressed in the vision community. The introduction of refractive elements into a scene changes the multi-view geometry that results from the imaging phenomenon. Until recently, the study of this change had been restricted to multimedia photogrammetry [START_REF] Maas | New developments in multimedia photogrammetry[END_REF] and oceanic engineering [START_REF] Negahdaripour | Opitcacoustic stereo imaging, system calibration and 3-d reconstruction[END_REF], where the major perspectives were to either neglect the refraction [50; 102], view refraction as an error or aberration to perspective imaging [107; 126], or to look at its correction as an iterative optimization problem [START_REF] Maas | New developments in multimedia photogrammetry[END_REF]. In computer vision, some of the first attempts have come in the recent past [9; 73; 74; 92; 141]. Kutulakos et al. [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF] investigate the geometry of light-path triangulation, which aims to find conditions and algorithms where reconstruction of individual tracks of light is possible. The work of Morris and Kutulakos [START_REF] Nigel | Dynamic refraction stereo[END_REF] looks at refractive stereo. The main idea here is to estimate the normal of the refractive surface, given 2D-3D correspondences, irrespective of the refractive index. They argue that under the assumption that light is refracted only once, two views are sufficient to reconstruct an arbitrary refractive surface. The work of Nayar et al. [START_REF] Ben | What does motion reveal about transparency[END_REF] concentrates on the estimation of the shape of the object by optimizing a function that minimizes the difference between observed and predicted images, based on a suitable parametric model of the object shape, as well as assuming an affine setting. The works of Singh et al. [START_REF] Treibitz | Flat refractive geometry[END_REF] and Lavest et al. [START_REF] Lavest | Underwater camera calibration[END_REF] concentrate on the internal calibration of cameras underwater, when a planar refractive surface separates the two media (air and water).

However, to the best of our knowledge, the epipolar geometry resulting from refraction has not been analyzed till now. In the presence of refraction, this geometry not only encodes the relative position of the cameras, but also information about the relative refractive index between the two media in question, and the geometry of the surface separating the concerned media. In this paper, we concentrate on the multi-view relationships induced by a refractive planar surface. We choose to deviate from the generic scene assumption of Kutulakos et al [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF], which enables us to completely describe the epipolar geometry in the presence of refraction. In this respect, our work is closer to Barreto & Sturm [START_REF] Sturm | General imaging geometry for central catadioptric cameras[END_REF] and Glaeser & Schröcker [START_REF] Glaeser | Reflections on refractions[END_REF]. In [START_REF] Glaeser | Reflections on refractions[END_REF], the authors define the equation that governs the projection of a 3D point onto an image, while being refracted across a planar surface. We follow an approach that is along the lines of [START_REF] Sturm | General imaging geometry for central catadioptric cameras[END_REF] in deriving the various multi-view relationships in the presence of refractions by a planar surface whereas did so for [START_REF] Sturm | General imaging geometry for central catadioptric cameras[END_REF] did so for reflection off quadrics (central catadioptric cameras).

In the following sections, we derive the various multi-view relationships between two views of a scene, when a planar transparent surface separates the scene from the camera. The two media are assumed to have different refractive indices. First, we define the projection matrix for such cameras for lines, where we show that a 3D line is mapped to a quartic curve. Next we derive the fundamental matrix between two views, which turns out to be a function of the camera poses and the plane normal and refractive index. We then derive an expression for the homography between a scene plane and its image on the camera. Finally, we give algorithms for obtaining the relative pose between two cameras in specialized cases, by deriving the expression of what is commonly referred to as Snell's

Window.

Background & Notation Consider the scenario where the scene contains a planar transparent refractive surface, with cameras on one side of the plane and the actual scene, composed of opaque objects, on the other side. Let the relative refractive index between the two media on either side of the plane be denoted by λ (=

n 1 n 2
), where n 1 is in the medium with cameras). Let us remember that the incident and refracted angles are related by

sin(θ 2 ) = λ sin(θ 1 ) (2.1)
where θ 1 is the incident angle, and θ 2 is the refracted angle.

Images of objects on the opposite side of the cameras are formed after refraction by the surface in consideration. We wish to deduce external calibration information about the cameras and the geometry of the refracting plane from such images alone. Further along this chapter, we make certain assumptions about the geometry of the scene, refractive index etc. to simplify matters. Such assumptions will be detailed when necessary.

In the rest of the chapter, we use the following coordinate system. Let the 3D refractive plane,

denoted by π = v ⊤ d ⊤
take the values 0 0 1 0 ⊤ , which aligns it with the X-Y plane of the world coordinate system. Let two cameras P 1 and P 2 be situated in the following manner

P 1 = R 1 I -t 1 (2.2) P 2 = R 2 I -t 2 (2.3) t 1 = 0 0 -1 (2.4)
where R 1 , R 2 are rotation matrices. Here, the position t 1 of the first camera is fixed without loss of generality. The position t 2 of the other camera, is not restricted. Now, the only undecided component about the world coordinate system is the location of the X-and Y-axes, which can vary upto rotation about the Z-axis. Although this can easily be fixed by assuming the first camera's coordinate axes to be aligned with that of the world, we ignore this unnecessary assumption for now. Finally, for the purposes of this chapter, the internal parameters of all cameras are assumed to be I 3×3 ,i.e., the cameras are calibrated. Now, let us define notations for points. We denote an image point by the variable x, which is a 3 × 1 homogeneous vector defined as x = x y 1 ⊤ . The direction vector for an image point is a 4 × 1 homogeneous coordinate and is represented by q = q3×1 0

⊤ = (R ⊤ x) ⊤ 0 ⊤
, and its j th coordinate for the i th camera is represented by q i, j , or by q j when there is only one camera. The vector x represents the unit vector corresponding to image point x. Thus q = 1. A vector formed by collecting elements from another is represented in braces. For example, L 1,(6,1,2) represents a vector formed by elements 6, 1 and 2 in L 1 . The Plücker coordinates of a 3D line is denoted by L. Matrices are represented using the font M, and vectors are represented as v. M denotes lifted coordinates ( q = q 2 1 q 1 q 2 q 2 2 q 1 q 3 q 2 q 3 q 2 3 ⊤ [63; 130]). Finally, if two vectors are related by a linear transformation (q = R ⊤ x), their lifted coordinates are related by [63; 130]

q = D -1 S(R ⊗ R)S ⊤ x (2.5)
Henceforth, all derivations are done with q, i.e. the back-projection direction for image point x, while noting that the corresponding equations for x can be obtained (if desired) by using Equation 2.5.

The Camera Projection Matrix

The first step in deriving the epipolar geometry needed to describe multi-view relationships is the camera matrix. We proceed to derive the camera matrix by first defining the back-projection of a point, and then focusing on the condition that it intersects a 3D line.

Back-projecting a Point

It has been previously shown that the image of a 3D point after refraction from a plane is the solution of a quartic equation that lies within a specified interval [START_REF] Glaeser | Reflections on refractions[END_REF]. We proceed to show that the image of a line after refraction is a quartic curve, and from this projection we derive a description of the camera matrix for refraction.

Consider a 3D point with inhomogeneous coordinates X. An image point x with back-projection direction q (both in homogeneous coordinates) represents an image corresponding to X, if the line from the camera center in the direction of q passes through X after refraction. After developing the refraction equation 2.1 for vectors, we can get the point Q on the refraction plane π, and the direction of the refracted ray R as

Q = t 3 q -q 3 t -q 3 , R =        q 1 λ q 2 λ (1 -λ 2 ) + λ 2 q 2 3 0        (2.6)
where Q 3 = 0. The corresponding Plücker coordinates of the refracted ray can be written as

L = Q 4 R 1...3 -R 4 Q 1...3 Q 1...3 × R 1...3 =           -q 1 q 3 λ -q 2 q 3 λ -q 3 (1 -λ 2 ) + λ 2 q 2 3 (q 2 t 3 -q 3 t 2 ) (1 -λ 2 ) + λ 2 q 2 3 (q 3 t 1 -t 3 q 1 ) (1 -λ 2 ) + λ 2 q 2 3 (q 1 t 2 -q 2 t 1 )q 3 λ           (2.7)
Now, if L intersects a 3D line, it can be concluded that this 3D line projects onto the points given by the locus of q, and hence x. Using this principle, we derive the projection matrix.

The Refractive Projection Matrix

Let A, B be two homogenous points on some 3D line L. The Plücker coordinates of this line are given as

L 1 = A 4 B (1,2,3) -B 4 A (1,2,3) A (1,2,3) × B (1,2 ,3)

⊤

. The back-projection line L in Equation 2.7 intersects this line iff these two lines satisfy the Klein quadric constraint [START_REF] Pottmann | Computational line geometry[END_REF], defined by the matrix

W = 0 I I 0
. This can be developed as follows

L ⊤ 1 WL = 0 (2.8) L ⊤ 1,(4,5,6) L ⊤ 1,(1,2,3) L = 0 (2.9) L 1,(6,1,2) ⊤     -q 3 q 2 t 3 -q 3 t 2 q 3 t 1 -t 3 q 1     (1 -λ 2 ) + λ 2 q 2 3 = q 3 λ L 1,(4,5,3) ⊤     q 1 q 2 q 2 t 1 -q 1 t 2     (2.10)
Squaring both sides removes the square root. Noting that for two vectors a and b, (a

• b) 2 = D a • b or a • D b where D = diag 1 2 1 2 2 1 , we get S(1 -λ 2 + λ 2 q 2 3 ) -Tλ 2 q 2 3 D s q = 0 (2.11) S(1 -λ 2 ) λ 2 ( S -T) D s 0 0 D s Ds q q 2 3 q = 0 (2.12) S = L 1,(6,1,2) ⊤     0 0 -1 0 t 3 -t 2 -t 3 0 t 1     t s , T = L 1,(4,5,3) ⊤     1 0 0 0 1 0 -t 2 t 1 0     t t (2.13)
S and T are defined as functions of t s and t t respectively, using Kronecker products and symmetric vectorization as defined in Equation 2.5 (See also [63; 130]).

Observe that equation 2.12. defines a quartic curve in the image coordinates (since the lifted coordinates of q and x in Equation 2.5 are related by linear transformations). Note that the camera position t 1 , refractive index λ and R are already absorbed in q. We can thus conclude that a 3D line is imaged to a quartic curve in a perspective camera, if seen through a refractive planar surface. Notice that the coefficents of the quartic curve in q can be defined as a linear function of L 1 . Let

c = D⊤ s S ⊤ (1 -λ 2 ) λ 2 ( S -T) ⊤ (2.14) = D⊤ s (1 -λ 2 )D s -1 S s t s ⊗ t s S s ⊤ 0 λ 2 D s -1 S s t s ⊗ t s S s ⊤ -λ 2 D t -1 S t t t ⊗ t t S ⊤ t P L 1,(6,1,2) L 1, (4,5,3) (2.15)
Then, equation 2.12 can be written as c ⊤ q q 2 3 q = 0. c contains all coefficients of the quartic curve that is the image of L 1 . Equation 2.15 shows that this curve can be computed by applying a projection matrix P to the lifted coordinates of the 3D line.

We have thus derived the projection matrix for a perspective camera viewing a scene separated by

The Refractive Fundamental Matrix

In the previous section, we defined the projection matrix for 3D lines. Continuing the same argument, we can see that the epipolar curve in the first image is nothing but the projection of a line. This line, in turn, is the back-projection of an image point from the second camera. Using the back-projection equation 2.7, we get the back-projected ray from the second image as

L 2 =            -q 2,1 q 2,3 λ -q 2,2 q 2,3 λ -q 2,3 1 -λ 2 (1 -q 2 2,3 ) (q 2,2 t 2,3 -q 2,3 t 2,2 ) 1 -λ 2 (1 -q 2 2,3 ) (q 2,3 t 2,1 + t 2,3 q 2,1 ) 1 -λ 2 (1 -q 2 2,3 ) (q 2,1 q 2,3 t 2,2 -q 2,2 q 2,3 t 2,1 )λ            (2.16)
where q 2 = q 2,1 q 2,2 q 2,3 ⊤ and t 2 = t 2,1 t 2,2 t 2,3 are the image point and the position of the second camera. Substituting L 2 in the place of L 1 in Equation 2.15 will give us the desired result, i.e. the image of the back-projection line L 2 , which is nothing else than the epipolar curve associated with

q 2 .
We now express the epipolar constraint, i.e. the constraint that a point q in the first image lies on the epipolar curve of q 2 . To do so, we first observe that

L 2,(6,1,2) = D -1 u S u t u ⊗ t u S ⊤ u q 2 q 2 2,3 λ 2 , t u =     -1 0 0 0 -1 0 t 2,2 -t 2,1 0     (2.17) L 2,(4,5,3) = D -1 v S v t v ⊗ t v S ⊤ v q 2 (1 -λ 2 + q 2 2,3 λ 2 ), t v =     0 t 2,3 -t 2,2 t 2,3 0 t 2,1 0 0 -1     (2.18)
Now, equation 2.15 can be modified to give the following

q 2 q 2 q 2 2,3 ⊤ 0 (1 -λ 2 )Λ v λ 2 Λ u λ 2 Λ v (1 -λ 2 )Λ s 0 λ 2 Λ s -λ 2 Λ t Ds q qq 2 3 = 0 (2.19) q 2 q 2 q 2 2,3 ⊤ (1 -λ 2 )λ 2 Λ v Λ s -(1 -λ 2 )λ 2 Λ v Λ t (1 -λ 2 )λ 2 Λ u Λ s + λ 4 Λ v Λ s -λ 4 Λ v Λ t Ds F q qq 2 3 = 0 (2.20)
where

Λ i = D -1 i S i t i ⊗ t i S ⊤ i .
Equation 2.20 defines the Fundamental matrix between two perspective cameras, when looking across a planar refractive surface. F is a matrix of dimensions 12×12 that relates the lifted coordinates in one image to a quartic curve in the other image. In a coordinate system where the refractive plane

The Refractive Homography Matrix

In this section, we derive the relationship between a scene plane in 3D, and its image after refraction.

Unlike the traditional case, we show that in the current scenario, this transformation is represented by a family of homographies that map one set of conics to another set.

Consider a cone of rays emerging from the camera, centered at the camera center, with an axis that is aligned with the normal of the refractive plane and an aperture 2θ 1 . Let us call this the incident cone (Figure 2.2a). Since any ray which is part of this cone surface, makes the same incident angle with the normal of the refractive plane, all refracted rays from this cone make the same angle with the normal of the refractive plane. Thus the refracted rays form a cone centered at the line joining the camera center and the plane, collinear with the normal of the refractive plane, with an aperture of θ 2 = 2 sin -1 λ sin θ 1 . Let us call this the refracted cone.

Since the image is formed by cutting the incident cone with the image plane, and the actual scene plane cuts the refracted cone, the 3D homography can be defined as a family of transformations, that transform one conic (formed by the scene plane and refracted cone) to another (formed by the image plane and the incident cone). This family is a function of θ 1 .

Given a ray q, the point Q and the refracted ray direction R are given by

Q = ( t 1 q ⊤ -q t ⊤ 1 )π (2.21) = q × t × -q q⊤ 0 π (2.22) = (q × t) × v -qd cos(θ 1 ) = (v) × (t) × q -qd cos(θ 1 ) (2.23) R = q 1 λ q 2 λ cos(θ 2 ) 0 ⊤ (2.24)
where θ 1 is the incident angle and θ 2 is the refracted angle, π = v ⊤ d ⊤ (general position of the refractive plane) and (•) × represents a skew symmetric matrix that corresponds to a cross product with the vector inside parenthesis. The point S where the above line will strike a plane

π 1 = v ⊤ 1 d 1 is now given by S = RQ ⊤ -QR ⊤ π 1 (2.25) = (Q (1,2,3) × R (1,2,3) ) × cos(θ 1 )R (1,2,3) -cos(θ 1 )R (1,2,3) 0 π 1 (2.26) = -(v 1 ) × (Q (1,2,3) × R (1,2,3) ) + cos(θ 1 )R (1,2,3) d 1 -cos(θ 1 )R (1,2,3) • v 1 (2.27) = -(v 1 ) × (v) × (t) × q × R (1,2,3) + (v 1 ) × (q × R (1,2,3) )d + cos(θ 1 )R (1,2,3) d 1 -cos(θ 1 )R (1,2,3) • v 1 (2.28) =    -(v 1 ) × (v) × (t) × + d v 1 × d 1 I 3×3 0 0 0 -v ⊤ 1    q × R (1,2,3) cos(θ 1 )R (1,2,3) (2.29)
The vector q × R (1,2,3) expands to give q 2 (cos(θ 2 )cos(θ 1 )) q 1 (cos(θ 1 )cos(θ 2 )) 0 ⊤ . We can isolate the cosines into a separate matrix to get

S =    -(v 1 ) × (v) × (t) × + d v 1 × d 1 I 3×3 0 0 0 -v ⊤ 1               q 2 (cos(θ 2 ) -cos(θ 1 )) q 1 (cos(θ 1 ) -cos(θ 2 )) 0 cos(θ 1 )q 1 λ cos(θ 1 )q 2 λ cos(θ 1 ) cos(θ 2 )            (2.30) =    -(v 1 ) × (v) × (t) × + d v 1 × d 1 I 3×3 0 0 0 -v ⊤ 1              0 a 0 -a 0 0 0 0 0 λ cos(θ 1 ) 0 0 0 λ cos(θ 1 ) 0 0 0 cos(θ 2 )           q (2.31) =        -(v 1 ) × (v) × (t) × a × + d v 1 × a × + d 1 λ cos(θ 1 )I 2×2 0 0 ⊤ cos(θ 2 ) -v ⊤ 1 λ cos(θ 1 )I 2×2 0 0 ⊤ cos(θ 2 )        q (2.32) (2.33) ∼ H θ 1 x (2.34)
with

H θ 1 =       -(v 1 ) × (v) × (t) × a × + d v 1 × a × + d 1 λ cos(θ 1 )I 2×2 0 0 ⊤ cos(θ 2 ) -v ⊤ 1 λ cos(θ 1 )I 2×2 0 0 ⊤ cos(θ 2 )       R ⊤ (2.35) a = cos(θ 2 ) -cos(θ 1 ) (2.36) a = 0 0 a ⊤ (2.37)
Equation 2.35 shows how the homography matrix can be expressed as a function of the incident and refracted angles. From the above expression, it is clear that there exists a family of homographies that are linear in the cosines of these angles, which fully represent the projection of a plane onto the image. Although the above expression can be further developed to obtain an expression in lifted coordinates that is independent of these angles, i.e., a homography that applies to all points of the image and scene plane, we omit the derivation here because of space constraints.

Exploiting Snell's Window

Till now, we have discussed the general case of planar refraction, which is applicable to any scene irrespective of the relative refractive index. Of particular importance, is a special case, when the camera is in a denser medium and looking outward into a lighter medium. This amounts to a relative refractive index that is > 1 (as per our current convention). In this case, it is possible to image the peripheral rays: rays that make an angle of π/2 with the refractive surface normal after refraction, and hence are parallel to the refracting surface. The importance of these rays is that any ray with an incident angle greater than the incident angle of these rays is reflected back into the denser medium.

This set of rays thus captures the periphery of the world on the other side of the surface, which in the case of underwater imagery is the horizon (Figure 2.2b).

The Image of Snell's Window

Consider a point x that is on the image of this periphery. Since after refraction the ray from this point is parallel to the refractive surface, the refracted angle is π 2 , which means

1 -λ 2 + λ 2 (π ⊤ q) 2 = 0 (2.38)
where q = (R ⊤ x) ⊤ 0 ⊤ , with x being the unit vector of image point x in the camera coordinate system. Let us develop these equations further.

q ⊤ ππ ⊤ q = 1 - 1 λ 2 (2.39) x⊤ R ⊤ vv ⊤ Rx = 1 - 1 λ 2 (2.40) 1 x 2 x ⊤ R ⊤ vv ⊤ Rx = 1 - 1 λ 2 (2.41) x ⊤ R ⊤ vv ⊤ Rx = x ⊤ I 3×3 x(1 - 1 λ 2 ) (2.42) x ⊤ R ⊤ vv ⊤ R -(1 -1 λ 2 )I 3×3 x = 0 (2.43)
which shows that the periphery is a conic in image coordinates. The term R ⊤ v represents the refractive plane normal in the camera coordinate system. One of the main advantages of Equation 2.43 is that one of the conic's eigenvectors is the normal of the refractive plane.

R ⊤ vv ⊤ R -(1 -1 λ 2 )I 3×3 R ⊤ v = 1 λ 2 R ⊤ v (2.44) since v ⊤ v = 1.
In fact, it is easy to show that the eigenvalues of the above matrix are 1

λ 2 , -1+ 1 λ 2 , -1+ 1 λ 2 .
Thus the important property of the eigenvalues and eigenvectors of the above matrix is that the only positive eigenvalue is the square inverse of the relative refractive index, while the corresponding eigenvector is the normal of the refractive plane in the camera coordinate system. Thus we have shown that the image of the horizon after refraction from a planar scene is a conic, and that using a simple SVD based algorithm it is possible to recover both the plane normal and the relative refractive index from a single image.

Relative Pose Computation

Given that we can estimate the plane normal (in the camera coordinate system), and the refractive index, we now show how to compute the relative pose of a second camera with respect to the first.

Let us first note that since both the cameras see the same refractive surface, estimating the normal of the plane in the camera coordinate systems of both cameras can help estimating the rotation between them upto a rotation about the normal itself. Additionally, we can set the perpendicular distance from the camera center of one of the cameras to the refractive plane to 1, so as to set the scale of the scene. Since we have chosen a world coordinate system where the plane normal is aligned with the Z-axis, there are 4 unknowns that remain to be computed, namely the rotation about the Z-axis and the translation parameters.

In order to solve for these parameters, let us start by observing the fundamental matrix equation (2.20). Observe that the image points (rays) are a function of the unknown rotation matrix of the second camera, which is of the form

R =     cos(α) sin(α) 0 -sin(α) cos(α) 0 0 0 1   
 . Thus, the vector q 2 q 2 q 2 2,3 ⊤ is quadratic in the elements of the rotation matrix R. Equation (2.20) thus gives constraints on α and t 2 . Thus, 4 point-correspondences should be sufficient for a solution, although we have not been able

to obtain a solution using either minimal solution packages or empirically.

Discussion and Conclusion

In this chapter, we have defined the epipolar geometry for scenes where a single plane separates two media with different refractive indices. The camera(s) are in one media, while the object(s) being observed are on the other side. Such scenarios occur often in underwater vision. We have shown that for the case when the camera is in a denser medium, it is even possible to estimate the refractive index and the refractive surface geometry from a single image, by observing the so-called "Snell's

Window". Further, we have shown a method to compute the relative pose between two cameras in such a scenario.

The contributions of this work are theoretical, and of conceptual value. Our main motivation for this work has been to explicitly model the geometry behind imaging in such scenarios. Apart from the theoretical value of the results we have shown thus far, what is also important is that using the results of this chapter, it is actually possible now to measure the deviation of the standard perspective model in areas like underwater imagery. This is of importance while modeling errors in observation using a perspective model.

3

Photogeometric Reconstruction of Transparent Objects

Introduction

In the previous chapter we looked at the multiple view geometry associated with a plane. While we presented approaches that recover the geometry of the scene and camera using a purely geometric fashion, a practical reconstruction algorithm has to contend with scene illumination effects also in dealing with transparent refractive surfaces. Also while we derived multiview constraints relating to a plane modeling transparent surfaces in a piecewise planar approach using the constraints we derived is not easily achieved. To this extent, in this chapter we try to derive an image based approach to reconstruct general reflective and transparent surfaces that combines both geometric and photometric information. and approaches that try to extend traditional shape acquisition approaches to the case of transparent objects. Among the approaches relying on material properties, geometric and photometric cues are the most prominent inputs to reconstruction algorithms.

Transparent objects referred to in the recent computer vision literature could be categorized into two kinds: specular and transluscent. Specular transparent objects are those whose surfaces exhibit specular reflection and refraction: for every ray of light incident on the surface, one refracted ray and one reflected ray is emitted. Transluscent surfaces generally do not observe this property, they emit multiple reflected/refracted rays for every incident ray. In this chapter, we focus on specular transparent objects. For such objects, geometric cues are an important source of information. A popular type of approach is based on the so-called light-path triangulation principle [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF]. Roughly speaking, one acquires images of a reference object with known shape, seen through the transparent object or reflected by it. Using camera calibration information, known relative position(s) of reference object and camera, and matches between the reference object and the camera images, one may recover the "light paths" associated with such matches by techniques akin to structure-from-motion and multi-view geometry, provided sufficiently many images in different positions are acquired. A light path refers here to a sequence of straight line segments that connect an object point and its image, where successive segments are the result of reflections/refractions in the transparent object's surface(s). Let us consider an object point emitting light in "all" directions. Underlying the light path triangulation approach is the assumption that only light emitted in a single direction, eventually hits the image plane of the camera, after undergoing reflections/refractions in the transparent object. This in turn relies on assuming an infinitesimal camera aperture. Also, in some cases there may be multiple light paths associated with a single object point, but it is assumed that a single one is observable by the camera.

Similar works use specular highlights produced by transparent/specular objects and the geometric cues these provide on the object surface. While specular highlights are sometimes referred to as photometric cues, we rather consider them as geometric ones -photometric analysis is usually restricted to identifying highlights, after which only geometric information is used. This is similar to light path triangulation and similar methods, where photometric information is used to determine matches but is then discarded from further processing.

In our work, we fully combine photometric and geometric information for the reconstruction of transparent objects. By photometric information, we mean the intensity values of image pixels and the irradiance values of points of the reference object whose reflections/refractions are imaged by the camera (in practice, we use screens as reference objects). Let us come back to light paths, as defined just above. In addition to modeling the geometric path light follows from an object point until its image point, we also model the photometric changes occurring along this path. In particular, real objects are neither perfect mirrors nor perfectly transparent: hence, each reflection/refraction causes a loss of irradiance. More precisely, light hitting a refractice surface gets partly reflected, partly refracted. For specific surface materials, theoretic models exist for how much light gets reflected and how much refracted. This depends on the refractive indices of the media in which the light

(a) (b)
Figure 3.1: (a) Description of the general theory behind our approach. While the acquisition is similar to that of Kutulakos et al [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF], we also include radiance in our measurements (depicted by the changing color of rays while they travel from the illuminant to the camera pixel). (b) Our setup to "acquire" the shape of transparent objects consists simply of a CRT monitor as light source and a camera looking at light reflected / refracted off the object.

travels as well as on the incidence angle between light path and surface normal. Hence, modeling and measuring this phenomenon brings about information on the shape of the transparent object;

importantly, this information is complementary to the information given by purely geometric input to light path triangulation. In essence: consider a scenario where the available information (number of images) is insufficient to provide a unique solution for the surface shape using light path triangulation.

In other words, there are multiple surface shapes that reflect/refract light rays emitted from object points in a way that they hit the matching image points. However, when tracing light paths and in addition, tracing the loss of irradiance along the paths due to reflection/refraction and comparing this against measured image intensities, then one gets one additional constraint per match that may allow to uncover a unique solution for our reconstruction problem.

We present an approach that combines the merits of utilizing both geometric and photometric cues.

Our approach is along the same lines as [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF], with one important difference. For every "light-path" that we capture, we record both geometric information (position and direction of light rays captured by and originating from light source, depending on requirement) and radiometric information (radiance of light at the beginning and end of each light-path). We show that our approach benefits from the following advantages because it uses a combination of cues: 1. It is able to extend the minimal case scenarios of reconstructions beyond those proposed in Kutulakos et al. [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF]. Thus we reduce the number of views required for reconstruction.

2. We show that it is possible to reconstruct objects in the presence of significant amounts of interreflections, caustics and other phenomena that would normally prove a hindrance to geometry based approaches.

The rest of the chapter is structured as follows. Section 3.2 places previous works in perspective.

Section 3.3 outlines the underlying theory and minimal solutions for our approach, and discuss some practical problems to be countered. Section 3.6 shows results for various cases. Finally, Section 3.7

presents an overall summary of our approach, points to future directions of research and presents potential applications.

Notation: In the rest of the chapter, we use the following convention. Bold symbols like E denote vectors with direction and magnitude, while Ê denotes its corresponding (unit) direction and E, its magnitude. Subscripts E s are either used to denote the components of a vector in a corresponding direction (in which case E s is the magnitude of the component), or in case of scalars, used to differentiate variables referring to the same physical phenomenon or variable.

Related Work

In the past, several approaches have used either geometric or photometric cues to reconstruct transparent surfaces. Geometric approaches typically measure the deviation from perspective imaging produced by a refractive transparent object, and recover the shape as a solution that explains the observation. In [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF], the authors present a minimal case study of the conditions in which refractive surfaces can be reconstructed. They re-cast transparent object reconstruction as the task of reconstructing the path of each individual ray of light that is captured by a camera after refraction through a transparent surface. They show that individually, it is impossible to reconstruct the entire trajectory of a ray of light after it has undergone more than 2 refractions. Earlier approaches have focussed on parameterizing the object to be reconstructed and then obtaining its parameters by explaining the distortion it produces in several cameras [START_REF] Ben | What does motion reveal about transparency[END_REF]. Other examples of shape recovery from distortion analysis include the more recent work by [START_REF] Morris | Reconstructing the surface of inhomogeneous transparent scenes by scattertrace photography[END_REF], which analyzes the specific case of a single dynamic transparent surface that distorts a known background and is observed by multiple cameras. The object surface is reconstructed using a modified stereo reconstruction approach that uses a new distance function. Finally, some recent approaches have also looked at learning based solutions that minimize a certain criterion of optimality [32; 140].

Apart from geometry, photometric information also turns out to be very important in the case of transparent objects since they simultaneously reflect and refract light. In fact, since they reflect light like a specular surface, in principle, any method for specular object reconstruction (like traditional photometric stereo) may be applied. Many recent photometric approaches have tried to reconstruct transparent surfaces by studying their specularities. While [START_REF] Yeung | Adequate reconstruction of transparent objects on a shoestring budget[END_REF] provides a low cost approach to reconstruction by studying specular highlights, [START_REF] Morris | Reconstructing the surface of inhomogeneous transparent scenes by scattertrace photography[END_REF] shows how to reconstruct transparent surfaces with inhomogeneous refractive interiors, by measuring highlights multiple times to remove extraneous effects like scattering, interreflections etc. One important aspect of photometry is polarization.

When unpolarized light is reflected or transmitted across a refractive surface, it gets partially polarized. This degree of polarization is connected to the shape of the object, and several authors have attempted to explore this connection. In [START_REF] Huynh | Shape and refractive index recovery from single-view polarisation images[END_REF] the authors measure the polarization angle using multiple images from a single view taken with different orientations of a polarizer. They then use Fresnel theory to derive a relationship between measured polarization state and the angle of reflection. Another approach involving polarization is presented in [START_REF] Miyazaki | Shape estimation of transparent objects by using inverse polarization ray tracing[END_REF]. Here, the authors start from an initial guess for the shape of the object involved, and using a technique called polarization ray tracing, they predict the polarization information for that shape, and arrive at the correct shape by minimizing the difference between observed and predicted states.

Since refractive objects present a challenging reconstruction problem, many authors have resorted to using active approaches for reconstructions. Methods like fluorescent immersion range scanning [START_REF] Hullin | Fluorescent immersion range scanning[END_REF] and tomographic reconstructions present alternate approaches that are of practical value when objects are of manageable size.

Physical Modeling

Consider a ray of light traveling in space in a direction represented by the vector î. Suppose this ray strikes a refractive medium as shown in Figure 3.2a (inset). This ray is accompanied by an electric field, whose oscillation direction we denote by the vector Ê. The radiant energy of this ray is related to the magnitude of its Poynting vector [60], which is also proportional to the square of the maximum amplitude of its electric field

I in ∝ E 2 (3.1)
where E denotes the maximum amplitude of the electric field [60]. When such a ray falls on the boundary of a refractive medium, part of this energy is transmitted, and the rest is reflected (assuming no absorption by the media). Our contention is that the ratio of irradiance of the transmitted or reflected rays with respect to the incident ray contains information about the 3D structure of the surface. The ratio of these energies can be computed using Fresnel equations relative to the plane of refraction, the plane that contains the incident ray, the normal and the refracted ray.

Let us denote the normal of this plane as π, and subsequently divide the electric field into two components (directions): one that is perpendicular to this plane ( Ês ) and the other that is parallel to this plane ( Êp ). (∴ E 2 = E 2 s + E 2 p ). Thus given incident angle θ 1 and refracted angle θ 2 (Figure 3.2a (inset)), and refractive indices (n 1 , n 2 ), the following equations describe the phenomenon Polarization A single light ray's electric field could be oriented in any direction that is perpendicular to the direction of ray propagation. The above equations show that transmitted and reflected energies depend on this direction. However, since a camera is a light integration device, it records information from several such rays in a range of directions at each pixel. This helps us integrate out the bias of a single ray towards a particular direction. However, we assume that the illuminant that we use generates light rays that are not biased towards any electric field orientation (unpolarized light).

n 1 sin(θ 1 ) = n 2 sin(θ 2 ) (3.2) R s = sin 2 (θ 1 -θ 2 ) sin 2 (θ 1 + θ 2 ) (3.3) R p = tan 2 (θ 1 -θ 2 ) tan 2 (θ 1 + θ 2 ) (3.4) T s = 4 sin 2 (θ 2 ) cos 2 (θ 1 ) sin 2 (θ 1 + θ 2 ) (3.5) T p = 4 sin 2 (θ 2 ) cos 2 (θ 1 ) sin 2 (θ 1 + θ 2 ) cos 2 (θ 1 -θ 2 ) (3.6) I r l ∝ E 2 s R s + E 2 p R p (3.7) I r r ∝ E 2 s T s + E 2 p T p ( 3
For unpolarized light, the values of E s and E p can be written as E s = E cos(φ) and

E p = E sin(φ),
where φ is the angle made by the E vector with π. We use the term polarization angle to refer to φ in this chapter. In the next section, we analyze the change in irradiance captured by a camera when light bounces (reflects or refracts) off a surface once and twice, since it is known that geometrically it is impossible to reconstruct more than two bounces, when reconstructing each light-path individually [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF]. With photometry too, reconstruction of light-paths with two bounces is not possible without using Fresnel equations.

I Image Intensity. I l

Intensity after reflection.

I r

Intensity after refraction.

Theory of Bounces

In this section, we develop relations between reflected or transmitted light energy and its relation to the 3D structure of the transparent object. For convenience, we hereafter denote an ensemble of rays traveling in the same direction in the same position in space but with different polarization angles, as a single ray. We further use the symbol I to represent the irradiance of such an ensemble as measured by a camera. All notations used in this chapter, are summarized in tables 3.1 and 3.2.

Case 1: Single Bounce

Consider a ray î back-projected from a CCD pixel of a camera. Assume that this back-projected ray reflects off a transparent surface once, before hitting any number of illuminant points X m (one illuminant per image) in space. Figure (reference) depicts this scenario. In such a case, the average irradiance of reflected components can be expressed as

I r l = R s E 2 1 π π 2 -π 2 cos 2 (φ)dφ + R p E 2 1 π π 2 -π 2 sin 2 (φ) (3.9) = 1 2 (R s E 2 + R p E 2 ) (3.10) = R s + R p 2 (E 2 ) (3.11) = R s + R p 2 I in (3.12)
where I in represents the radiance of the ray emitted from one point X m . R s and R p are defined in equations 3.3 and 3.4. If the light is partially polarized, then φ varies within a range (l, u) depending on the degree of polarization. Also, since all directions are not present in equal probability, the irradiance equation has to be modified to

I r l = R s E 2 u l cos 2 (φ)ρ(φ)dφ + R p E 2 u l sin 2 (φ)ρ(φ) (3. 13
)
where φ is the probability distribution function with u l ρ(φ) = 1. The transmitted energy I r r also has a similar formulation, following from equation (3.8). Notice that equations (3.3)-(3.4) can be used to show that light rays after reflection / refraction are partially polarized [60]. We now list the two main triangulation results of the single bounce case. The problem of triangulation is to estimate the point(s) at which every light-path intersects the transparent surface along with the normal at that point, given 2D-3D correspondences. We use the notation of [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF] to represent the triangulation problems (< 1, 2, 3 > means 1 view, 2 bounces and 3 3D correspondences per light-path). Both the problems mentioned below have been earlier shown to be tractable with lesser information [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF].

Result 1: < 1, 1, 1 > Triangulation Consider any point at depth d k from the camera center along î. This point is associated with a unique normal nk such that the reflected ray passes through the point X m . In the case of refraction, this point is associated with a set of normal-refractive index pairs that allow the refracted ray to pass through X m . However, it is also associated with a radiance ratio

I α = I r l /I in = 2

Case 2: Double Bounce

When light bounces off a transparent dielectric surface, it is partially polarized. If the outgoing light's electric field is parametrized by the angle ψ then we have

ρ(ψ) l = R s R p |(R p -R s ) cos 2 (ψ) + R s | 1 π (3.14) ρ(ψ) r = T s T p |(T p -T s ) cos 2 (ψ) + T s | 1 π (3.15)
where the limits are (l = -π 2 , u = π 2 ) (See Appendix (Section .2). Thus, in the case of a second bounce, the incident light is already partially polarized. In this case, we use equation 3.13 to derive the irradiance measured as a function of the radiance emitted by the illuminant. This irradiance is given as 2

I ll = E 2 1 2 ( 2 R s 1 R p + 1 R s 2 R p ) sin 2 (α) + ( 1 R s 2 R s + 1 R p 2 R p ) cos 2 (α) (3.16)
where α is the angle between the normals to the two planes of refraction/reflection. (Please refer to the Appendix (Section .2) for a proof).

Result 3: < 2, 2, 2 > Triangulation Consider two cameras looking at a transparent object, which refracts light from a known illuminant twice. Further assume that for each camera, two measurements were made per light ray. This corresponds to the case < 2, 2, 2 > as per the convention of [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF]. Since equations (3.14, 3.15) give us one extra constraint per light ray, it is now possible to solve for the 3D structure of the transparent object given known refractive index.

Result 4: < 3, 2, 2 > Triangulation It is known that this problem is tractable in the case of known refractive index. However, it is now possible to solve for this case even with unknown refractive index, since all the radiance ratios obtained have to be consistent with both the light-path geometry and photometric observations.

A table containing the minimal scenarios for transparent objects that can now be solved with the help of irradiance measurements is shown in Table 3.3. It is interesting to note that transparent objects have lesser minimal requirements for reconstruction than mirror like objects. While in this section we have presented a counting argument to solving single and double bounce cases, we will detail the characteristics of the solutions that can be obtained in the next section. Specifically, we analyze the solution space for both cases and present some practical aspects of data acquisition.

Methods of Solutions

Understanding the physics behind the radiance of a single ray of light captured by a camera gives us the radiometric tools needed to reconstruct a light-path. In this section, we use this understanding

One reference point. (M

= 1) K = 1 K = 2 K ≥ 3 N = 1 N = 2 × *
Two or more reference points. (M ≥ 1)

K = 1 K = 2 K ≥ 3 N = 1 × * N = 2 × * N = 3 × * × N ≥ 4 × * × *
Table 3.3: Tractable triangulation problems. Updated from [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF]. The asterik symbol represents the fact that even in the case of only reflection, the relative refracitve index can be computed.

to derive the solution space of depths and normals that could explain the geometric and photometric characteristics of a light-ray observed by a camera.

Single Reflection or Refraction

The category of single bounce cases pertains to the scenario where a light-path is composed of two line segments. This could mean that the light ray associated with the light-path underwent refraction or reflection. In this section, we deal with the case of reflections, i.e. the camera acquires images of the monitor, reflected in the object to be reconstructed. The case of refraction is analogous.

Let us now consider a single pixel and the acquired intensity I l . The pixel's line of sight is known by calibration. Let d be the depth of the object along this line of sight, P be the intersection point of the object surface and the line of sight, and n the surface normal at that point. Further, given the matching (cf. above), we know the point X on the monitor whose reflection is seen in the pixel.

Our goal is to compute the depth d. We do so by first computing the incident angle θ 1 between the surface normal and the incident light ray, from which it is trivial to compute d.

Since our setup is radiometrically calibrated, we have, from (equations (3.3,3.4)):

I l = sin 2 (θ 1 -θ 2 ) 2 sin 2 (θ 1 + θ 2 ) 1 + cos 2 (θ 1 + θ 2 ) cos 2 (θ 1 -θ 2 ) I Let r = I l I in
; we then get the equation

r = sin 2 (θ 1 -θ 2 ) 2 sin 2 (θ 1 + θ 2 ) 1 + cos 2 (θ 1 + θ 2 ) cos 2 (θ 1 -θ 2 )
Note that although we are considering the case of reflection here, the refracted angle θ 2 nevertheless appears in the equation, due to the "light fall-off" caused by the object's refractive property.

In the following let us denote s 1 = sin(θ 1 ) and c 1 = cos(θ 1 ). Further, we use λ =

n 1 n 2 as the relative refractive index, D = 1 -λ 2 s 2 1 and c 2 = ± 1 -s 2 2 = ± 1 -λ 2 s 2 1 = ± D.
Thus, from the above equation, we get

r = R s + R p 2 = sin 2 (θ 1 -θ 2 ) + cos 2 (θ 1 + θ 2 ) tan 2 (θ 1 -θ 2 ) 2 sin 2 (θ 1 + θ 2 ) = s 1 c 2 -c 1 s 2 2 + c 1 c 2 -s 1 s 2 2 { t 1 -t 2} 2 { 1+t 1 t 2} 2 2 s 1 c 2 + c 1 s 2 2 = ±s 1 D -λc 1 s 1 2 + ±c 1 D -λs 2 1 2 t 1 ∓λ s 1 D 2 1±t 1 n s 1 D 2 2 ±s 1 D + λc 1 s 1 2 = ±s 1 D -λc 1 s 1 2 + s 2 1 ±c 1 D -λs 2 1 2 D∓λc 1 c 1 D 2 c 1 D±ns 2 1 c 1 D 2 2 ±s 1 D + λc 1 s 1 2 = ±s 1 D -λc 1 s 1 2 + s 2 1 ±c 1 D -λs 2 1 2 D∓λc 1 c 1 D±λs 2 1 2 2 ±s 1 D + λc 1 s 1 2 = ±s 1 D -λc 1 s 1 2 c 1 D ± λs 2 1 2 + s 2 1 ±c 1 D -λs 2 1 2 D ∓ λc 1 2 2 ±s 1 D + λc 1 s 1 2 c 1 D ± λs 2 1 2
We observe that this is a quartic polynomial in sin 2 θ 1 , i.e. the computation of θ 1 can be considered as being (close to) a closed-form solution.

Given an estimate for θ 1 , the depth d can be computed by triangulating the incident and reflected rays (since camera centre and 3D correspondence are known, and knowledge of θ 1 gives normal and reflected ray directions) Among the up to 8 possible real solutions for θ 1 , at most 4 will correspond to a positive depth, i.e. a surface point in front of the camera. Finding a unique θ 1 and thus depth, is made possible in most practical circumstances, as follows. First, the absolute value of θ 1 must be below 90 Let us summarize the above findings. From images acquired with a completely static setup, we are able to compute the depth of the transparent object, for each pixel in which a reflection is visible.

To do so, we need to know the refractive index of the object's material. A unique solution for depth is possible in a large range of practical conditions. In case these are not fulfilled, one may still use bounds on the object depth to get a unique solution.

Double Bounce

In this section we consider the problem of solving for the depth of surfaces in the presence of two bounces. We first analyse the simplest case, when a rectangular slab separates the camera and the reference. To further simplify the case, we consider the camera plane, the reference planes and the planes of the slab to be parallel. Hence the problem reduces to the estimation of the two "depths" of the slab's planes. Figure 3.3 illustrates this case and the associated coordinate system in which we operate. Note that like opaque objects, each point on the surface of the slab produces one image point. However, unlike opaque objects, the correspondence between image points of a particular surface point in C 1 and C 2 cannot be computed since the surface point does not have an "appearance" but simply transfers a fraction of the appearance of its background. In fact, every back-projected image point from C 1 or C 2 intersects the slab surface at two points and hence has two "depths" where

m 1 k = m1 k m1 k
. Now, the first segment of the light-path of C 2 , and its angle with normal n k , 1 θ 2 k can be defined as

i 2 k = C 1 + d k i 1 1 -C 2 C 1 + d 1 k i 1 1 -C 2 = ī2 k ī2 k (3.23) 1 θ 2 k = cos -1 (n k • i 2 k ) (3.24)
In a similar manner to the approach described for m 1 k , we can derive the equation for m 2 k as

m2 k = n k cos( 1 θ 2 ′ k ) + λ(i 2 k -n k cos( 1 θ 2 k )) (3.25) = n k (1 -λ 2 sin 2 ( 1 θ 2 k )) + λ(i 2 k -n k (n k • i 2 k )) (3.26) = n k (1 -λ 2 ) + λ 2 (n k • i 2 k ) 2 + λ(i 2 k -n k (n k • i 2 k )) (3.27)
Finally, we can define the geometric constraint as the condition that the two rays

X 2 + α ī2 k and C 2 + γi 2 k + β m2 k intersect (where γ is d 1 k i 1 1 + C 1 -C 2 )
. In other words

C 2 + γi 2 k + β m2 k = X 2 + α ī2 k (3.28) (C 2 -X 2 ) × ī2 k + 0 + β m2 k × ī2 k = 0 (3.29) (C 2 -X 2 ) × ī2 k = -β m2 k × ī2 k (3.30)
The above constraint really specifies that for a valid set of parameters the planes given by (X 2 -C 2 ), i 2 k and n k , i 2 k are one and the same. By using equation (3.25), we get

(C 2 -X 2 ) × ī2 k = -β n k × ī2 k cos( 1 θ 2 ′ k ) + λ(0 -n k × ī2 k cos( 1 θ 2 k )) (3.31) = -β(n k × ī2 k )(cos( 1 θ 2 ′ k ) -cos( 1 θ 2 k )) (3.32) ((C 2 -X 2 ) × ī2 k ) × (n k × ī2 k ) = 0 (3.33) (C 2 -X 2 ) × (C 1 -C 2 ) a +d 1 k (C 2 -X 2 ) × i 1 1 b × cos( 1 θ 1 k ) i 1 1 × (C 1 -C 2 ) x + sin( 1 θ 1 k ) j 1 1 × (C 1 -C 2 ) y +d 1 k sin( 1 θ 1 k ) ( j 1 1 × i 1 1 ) z = 0 (using equations (3.23 & 3.18)) (3.34)
Note that vectors a, b, x, y, z are all known quantities. We can further simplify this to remove the effect of 1 θ 1 k using the following steps

a × x cos( 1 θ 1 k ) + b × x d 1 k cos( 1 θ 1 k ) + a × y sin( 1 θ 1 k ) + b × y d 1 k sin( 1 θ 1 k ) + a × z d 1 k sin( 1 θ 1 k ) + b × z(d 1 k ) 2 sin( 1 θ 1 k ) = 0 (3.35) b × x a × x a × y + b × y d 1 k + a × z d 1 k + b × z(d 1 k ) 2 sin( 1 θ 1 k ) = 0 (3.36) b × x a × x a × y + b × y d 1 k + a × z d 1 k + b × z(d 1 k ) 2 = 0 (3.37)
This is a 2nd degree equation in d 1 k with two solutions. The two solutions can easily be disambiguated by computing the value for 1 θ 1 k using equation (3.35) (by solving for tan( 1 θ 1 k )). Further, observe that since y 1 1 = i 1 1 and y 2 k = i 2 k , the angles of the first bounce for the second light-path ( 1 θ 2 k , 1 θ 2 ′ k ) are the same as the angles for the second bounce (in reverse because of reciprocal relative refractive indices).

The photometric constraint computed using Fresnel equations can be given as

I r r = 1 2 ( 1 T s 2 T s + 1 T p 2 T p )I in (3.38) r = I r r I in = 1 2 ( 1 T s 2 T s + 1 T p 2 T p ) (3.39) Denoting s 1 = sin( 1 θ 2 k ), s 2 = sin( 1 θ 2 ′ k ) = λs 1 (where λ = n 1 n 2
), and

c 1 = cos( 1 θ 2 k ) and c 2 = 1 -s 2 2 = 928 1 -λ 2 s 2 1 = D (where D = 1 -λ 2 s 2 1 ), we get 929 1 T s = 4 sin 2 ( 1 θ 2 ′ k ) cos 2 ( 1 θ 2 k ) sin 2 ( 1 θ 2 k + 1 θ 2 ′ k ) (3.40) = 4λ 2 s 2 1 c 2 1 (s 1 c 2 + s 2 c 1 ) 2 (3.41) = 4λ 2 s 2 1 c 2 1 (λs 1 c 1 ± s 1 D) 2 = 4λ 2 s 2 1 c 2 1 λ 2 s 2 1 c 2 1 + s 2 1 D ± 2λs 2 1 c 1 D (3.42) = 4λ 2 c 2 1 λ 2 c 2 1 + D ± 2λc 1 D = 4λ 2 c 2 1 (λc 1 ± D) 2 (3.43) 2 T s = 4 sin 2 ( 1 θ 2 k ) cos 2 ( 1 θ 2 ′ k ) sin 2 ( 1 θ 2 k + 1 θ 2 ′ k ) (3.44) = 4(1 -λ 2 s 2 1 ) λ 2 c 2 1 + D ± 2λc 1 D = 4D (λc 1 ± D) 2 (3.45) 930 1 T p = 4 sin 2 ( 1 θ 2 ′ k ) cos 2 ( 1 θ 2 k ) sin 2 ( 1 θ 2 k + 1 θ 2 ′ k ) cos 2 ( 1 θ 2 k -1 θ 2 ′ k ) (3.46) = 4λ 2 c 2 1 (λc 1 ± D) 2 (c 1 c 2 + s 1 s 2 ) 2 (3.47) = 4λ 2 c 2 1 (λc 1 ± D) 2 (c 2 1 c 2 2 + s 2 1 s 2 2 + 2s 1 s 2 c 1 c 2 ) (3.48) = 4λ 2 (λc 1 ± D) 2 (c 2 2 + t 2 1 s 2 2 + 2t 1 s 2 c 2 ) (3.49) = 4λ 2 (λc 1 ± D) 2 (D + λ 2 t 2 1 s 2 1 ± 2λt 1 s 1 D) (3.50) = 4λ 2 (λc 1 ± D) 2 (λt 1 s 1 ± D) 2 (3.51) = 4λ 2 c 2 1 (λc 1 ± D) 2 (λs 2 1 ± c 1 D) 2 (3.52) 57
T p = 4 sin 2 ( 1 θ 2 k ) cos 2 ( 1 θ 2 ′ k ) sin 2 ( 1 θ 2 k + 1 θ 2 ′ k ) cos 2 ( 1 θ 2 k -1 θ 2 ′ k ) (3.53) = 4c 2 2 (λc 1 ± D) 2 (c 2 1 c 2 2 + s 2 1 s 2 2 + 2s 1 s 2 c 1 c 2 ) (3.54) = 4 (λc 1 ± D) 2 (c 2 1 + s 2 1 t 2 2 + 2s 1 t 2 c 2 ) (3.55) = 4 (λc 1 ± D) 2 (c 1 + s 1 t 2 ) 2 = 4 (λc 1 ± D) 2 (c 1 ± λs 2 1 D ) 2 (3.56) = 4D (λc 1 ± D) 2 (c 1 D ± λs 2 1 ) 2 (3.57)
Putting all these terms in equation (3.39) we get

r = 8λ 2 c 2 1 D (λc 1 ± D) 4 + 8λ 2 c 2 1 D (λc 1 ± D) 4 (c 1 D ± λs 2 1 ) 4 (3.58)
Let us develop each of these terms before combining them into one set of equations

(λc 1 ± D) 4 = (λ 2 c 2 1 + D ± 2 Dλc 1 ) 2 (3.59) = λ 4 c 4 1 + D 2 + 4λ 2 Dc 2 1 + 2λ 2 c 2 1 D ± 4λc 1 D D ± 4λ 3 c 3 1 D (3.60) = λ 4 c 4 1 + D 2 + 6λ 2 Dc 2 1 A ± D 4λc 1 (D + λ 2 c 2 1 ) B (3.61) (c 1 D ± λs 2 1 ) 4 = (c 2 1 D + λ 2 s 4 1 ± 2λs 2 1 c 1 D) 2 (3.62) = c 4 1 D 2 + λ 4 s 8 1 + 4λ 2 s 4 1 c 2 1 D + 2λ 2 c 2 1 s 4 1 D ± 4λs 2 1 c 3 1 D D ± 4λ 3 s 6 1 c 1 D (3.63) = c 4 1 D 2 + λ 4 s 8 1 + 6λ 2 s 4 1 c 2 1 D C ± D 4λs 2 1 c 1 (c 2 1 D + λ 2 s 4 1 ) E (3.64) (λc 1 ± D) 4 (c 1 D ± λs 2 1 ) 4 = AC + BE D ± D(AE + BC) (3.65)
Substituting equation (3.65, 3.63) in equation 3.58, and cross-multiplying, we get

r(AC + BE D ± D(AE + BC)) = 8λ 2 c 2 1 D(1 + C ± DE) (3.66)
We now isolate terms with D on one side and then square both sides to get

± D(rAE + r BC -8λ 2 c 2 1 DE) 2 = 8λ 2 c 2 1 D + 8λ 2 c 2 1 DC -rAC -r BE D 2 (3.67) D(rAE + r BC -8λ 2 c 2 1 DE) 2 = (8λ 2 c 2 1 D + 8λ 2 c 2 1 DC -rAC -r BE D) 2 (3.68)
In the above equation, the only troublesome term is c 1 since its definition in terms of the variable s 1 involves a square root. However, if all the terms of c 1 are of even powers, this problem does not arise.

In order to check this fact, we first expand the different terms in R.H.S and L.H.S. Note also, that many of the variables involved in the general case of double bounce are predetermined here because of the simplicity of the case. For example, the condition y 1 1 = i 1 1 is not true in general. Also, we assume that the entire light-path lies on a single plane, which is not true in the general case. Thus, it is easier to show that a unique solution may be obtained empirically. Isocontours for various values of radiance ratio are plotted. Note that they are all curves. Note also that their intersection with the curve curresponding to θ 2 = θ ′ 1 (case of parallel planes) is a single point.

AE = λ 4 c 4 1 + D 2 + 6λ 2 Dc 2 1 4λs 2 1 c 1 (c 2 1 D + λ 2 s 4 1 ) (3.69) = 4λs 2 1 c 1 λ 4 c 4 1 + D 2 + 6λ 2 Dc 2 1 c 2 1 D + λ 2 s 4 1 (3.70) BC = 4λc 1 (D + λ 2 c 2 1 ) c 4 1 D 2 + λ 4 s 8 1 + 6λ 2 s 4 1 c 2 1 D (3.71) = 4λc 1 D + λ 2 c 2 1 c 4 1 D 2 + λ 4 s 8 1 + 6λ 2 s 4 1 c 2 1 D (3.72) 8λ 2 c 2 1 DE = 32λ 3 s 2 1 c 3 1 c 2 1 D + λ 2 s 4 1 (3.73) AC = λ 4 c 4 1 + D 2 + 6λ 2 Dc 2 1 c 4 1 D 2 + λ 4 s 8 1 + 6λ 2 s 4 1 c 2 1 D (3.74) BE = 4λc 1 (D + λ 2 c 2 1 ) 4λs 2 1 c 1 (c 2 1 D + λ 2 s 4 1 ) (3.75)
= 16λ 2 s 2 1 c 2 1 D + λ 2 c 2 1 c 2 1 D + λ 2 s 4 1 (3.76) 8λ 2 c 2 1 DC = 8λ 2 c 2 1 D c 4 1 D 2 + λ 4 s 8 1 + 6λ 2 s 4 1 c 2 1 D (3.
along a light path. It shows that given refractive index, the values of incident angles that correspond to a particular radiance ratio lie on a curve in this space. A similar case occurs for refractive double bounce.

Practical considerations

The above theory shows that the irradiance of a final light ray in a light-path contains information that could be used to reconstruct the entire light-path. In this section we describe important elements of our experiments to collect radiance measurements for reconstruction. Our experiments consist of three parts. 1) We use an illuminant with known geometry to emit unpolarized light in a desired set of directions. 2) Light from the illuminant interacts with the transparent object, and reflects / refracts off its surface towards the camera after one or two bounces.

3) The camera then captures both the direction and radiance of some reflected / refracted light rays, which is used for reconstruction.

We use Canon and Nikon DSLRs for our experiments. Since we need to capture the position and radiance of an individual light ray, we adopt the pin-hole model for the camera (smallest aperture).

Although using the smallest aperture does not guarantee a single ray for every direction, the cone of rays captured by the camera at each CCD pixel can be approximated by a thin cylinder if the focal length is large, or the objects are far away. As for the focal length, there is a tradeoff between large depth-of-field and imaging the pixels of the monitor. We would like to minimize blur because it confuses radiance measurements, but on the other hand we would like to capture each pixel of the monitor on 1 or more pixels on the camera CCD (this leads to a simpler illumination model for the monitor pixel). We arrived on an acceptable range of focal lengths by trial and error. Finally, for each captured ray, we compute the corresponding pixel on the monitor from which the ray originated using standard methods [START_REF] Aliaga | A self-calibrating method for photogeometric acquisition of 3D objects[END_REF].

Unpolarized illuminant

In our experiments, we use a flat CRT monitor (LCD montiors emit polarized light), whose pose is computed with respect to an internally calibrated camera [START_REF] Sturm | Algorithms for plane based pose estimation[END_REF]. This is motivated by our need to measure the illumination and the geometry of our illuminant. We capture the light emitted by each pixel of the monitor in several directions in 3D, and fit a smooth model to this data in order to accurately measure radiance of the illumination incident on the transparent objects.

Interreflections A common problem with measuring illumination reflected / refracted off specular transparent objects is interreflections. They not only corrupt the radiance measurement, but also pose a problem to correspondence estimation. In order to remove the effect of interreflections, we use the algorithm of Nayar et al. [START_REF] Nayar | Fast separation of direct and global comefractive index recovery from single-view polarisation images[END_REF]. Instead of using a projector to light the scene, we use the CRT monitor.

We project low frequency checkerboard patterns that are shifted cyclically, and use it to compute the direct and global components of the scene. Figure 3.10 shows a result for one typical scenario.

Calibration

We internally calibrate the camera in order to compute the direction of the captured rays. We also externally calibrate the monitor w.r.t the camera. We use the monitor as the reference frame of the coordinate system, which is beneficial in the case of multiple cameras. For irradiance measurement, we first extract an unprocessed image from the RAW files using dcraw 1 . We then remove the global component of this image, and then extract direct irradiance measurements. Note that all images shown in the results section are extracted from RAW files.

Correspondence Acquiring correspondence between pixels on the monitor and pixels on the camera that correspond to the same light-path becomes slightly cumbersome when transparent objects are involved [4]. In order to overcome this, we first remove global components from images used for correspondence by applying the theory of [START_REF] Nayar | Fast separation of direct and global comefractive index recovery from single-view polarisation images[END_REF] to these binary images, and then use a graph based approach to enforce spatial smoothness in the image while extracting correspondences. We use gray codes for robustness to intensity measurement errors, and in our experience the global component removed images are easier to process for correspondences than images which contain both direct and global components.

Estimating CRT Illumination Model

In this section, we give a slightly more detailed description of how we calibrate the illumination of the CRT monitor that we use in our experiments. This is essential since without an accurate calibration, we will not obtain accurate ratios of radiances. Throughout this section we assume a flat CRT monitor screen. Our experiments use a flat CRT monitor screen with a resolution of 768 × 1024 pixels.

Consider a camera viewing the monitor screen, with no object in between (Figure 3.5). Also consider that the pose of the camera w.r.t the monitor plane is known, and that several such images have been captured with the camera at different positions w.r.t the monitor (with pose known in each case, in a common coordinate system).

Consider the j th pixel on the monitor screen. Let it have a maximum radiance of I j , captured when the camera looks directly at the screen (camera plane parallel to monitor plane). In other poses, when the camera captures its image, this pixel contributes to 1 or several pixels on the CCD. Also, its contribution to each CCD pixel might be partial (with other monitor pixels also contributing to the same CCD pixel) or whole (no other contributions to that CCD pixel). Thus, a CCD pixel receives a percentage H( j) of a monitor pixel's radiance. Note that in this aspect of the model, we deviate from the pin-hole model towards a model where each ray is represented as a cylinder, as shown in Also, when a camera CCD pixel imaging part of the monitor is back-projected into world coordinates, the back-projected ray lands on the monitor screen at an angle w.r.t to the normal of the monitor plane. Let this angle be called θ j for each monitor pixel (actually more than one CCD pixels might back-project onto the j th pixel, but since all these pixels will be in a small neighborhood, it is reasonable to assume they will make the same angle with the monitor normal).

Finally, if the pixel j has a finite area A j , then the radiance captured by the back-projected ray and hence the camera is proportional to cos(θ j ) because of the change in the solid angle subtended by the pixel.

Putting all this together, we can explain the radiance observed at camera CCD pixel l in pose k as

I l = N j m=1 I m cos(θ k m )H(m k )ρ(θ k m ) (3.78)
where N j is the bunch of pixels that project onto pixel l. Finally, if we assume I m and θ m to be constant over a small neighborhood and we assume the camera zooms in on the monitor pixels so that 1 monitor pixel projects onto several CCD pixels, we get.

I l = H( j k )I j cos(θ k j )ρ(θ k j ) (3.79)
In the above equations, all the variables are known except ρ(θ k j ) which explains the varying amount of radiance emitted by the j th monitor pixel at angle θ k j w.r.t to the monitor plane normal. Note that by construction, this function is assumed to be symmetric about the normal.

The value of H( j k ) is computed using the homography between the image of the monitor and the monitor pixels themselves (which is known by calibration). For each CCD pixel, we generate several equally spaced points within the unit area of the pixel, and transfer them to monitor pixel coordinates using the homography. We determine the value of H( j k ) based on how many of the generated points fall within the boundaries of the monitor pixel j. θ k j is computed trivially using the camera pose w.r.t monitor plane.

Experiments and Results

In the previous sections, we showed that radiance ratios could be used to reconstruct transparent surfaces, which can help in reducing the number of measurements required for reconstruction. We also list some practical aspects that are relevant to acquiring image data. In this section, we show results of three experiments. The first two experiments demonstrate the accuracy of 3D reconstruction using our method. The third and fourth experiment show our results on extremely complex scenarios, and we show that just the application of what we theorize above can give surprisingly good results.

Experiment 1: Synthetic dataset Figure 3.9 shows various results for single bounce reconstruction of a sample sinusoidal object when the 3D correspondence is noisy. We compare with two light-path triangulation approaches, one in which the 3D correspondences along a light-path are close to each other and the object (LP-1), and one in which they are far (LP-2). In both cases, noise is added to the farthest 3D correspondence. While one case (LP-1) is sensitive to noise, the second case (LP-2)

is robust but impractical. Our approach however, gives a reliable normal map even if the depth is slightly perturbed (compared to LP-2). Note that noise percentage is calculated as ratio of distance between noisy and ground truth data and distance of ground truth and object. Other results w.r.t camera noise and refractive index mismatch are present in the supplementary materials.

Experiment 2: "Water Sequence" Figure 3.10 (Left column) shows some images acquired in order to reconstruct the surface of water in a plastic bowl. The bowl is around 10 cms in diameter, and is placed about 2.5 meters from the camera. This is a scene with a very simple 3D structure (a plane) and given the smooth surface of the bowl, it also has minimal (but not negligible) interreflections and caustics.

Because of the planar nature of the scene, we compute correspondence by simply computing a homography between the reflected image and the direct image of a photograph displayed on the monitor. This homography, adjusted for the internal calibration of the camera and the aspect ratio of the pixels on the monitor, can now be used to compute the normal of the plane [START_REF] Sturm | Algorithms for plane based pose estimation[END_REF].

After computing the direct image, we use the homography to compute the ratio of directly observed and reflected radiances. Finally, we hypothesize and test individual pixels for various values of depth, and record the value that best fits the radiance ratio. Figure 3.10 shows our result (the ripples in Figure 3.8: Simulation results for photometric error and refractive index mis. We simulated a curved object, and captured radiometric information from a camera according to Fresnel theories. We then added noise to this data (left), or used a slightly different refractive index (right) to reconstruct the surface. Notice that for noise, although the noise was added to the radiometric information that was recorded, the radiance ratio is considerable stable w.r.t the noise (since the denominator in the radiance ratio, which is the illuminant, has a high value). This results in the reconstructions being reasonably stable. However, when the refractive index is changed, depending on the angle of incidence and the depth-normal relationship, the reconstruction accuracy changes. While here we list the worst possible scenario, when the camera is close to the object, moving the camera further away like in the case of our datasets considerably strengthens the robustness of our results. Notice also, that in case we underestimate the refractive index the depth variation of the surface remains more or less intact, while its mean distance from camera increases. Thus, even with an underestimate it is possible to get reasonable results.

Figure 3.9: Comparison of simulations between our approach and [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF]. In LP-1, corresponding 3D points are normally close to the object, which results in increased error in triangulation and normal estimation. Note that in the same scenario, we have much better normal information because of photometric information. While LP-2 is robust because correspondences are far away, its highly impractical since use of LCD's for correspondence is problematic (because of light fall-off, scattering etc.). Details in text.

the reconstruction are just quantized depth values). An alternate result was one obtained by using the algorithm of [START_REF] Sturm | Algorithms for plane based pose estimation[END_REF], which uses the knowledge that the object is planar, while our approach estimates per pixel depth. Comparison of the results gives us a mean squared error of around 0.1 cms (we omit correspondence errors while computing this measure), which shows the accuracy of our reconstruction. the subsurface scattering might throw off correspondence measurements. This can heavily influence reconstruction accuracy. On the other hand, robust measurement of the position and direction of light incident on the glasses from the monitor requires a large set of images to be captured while moving the monitor over, say, an optical bench. In our approach, however, both these errors are avoided because we use a single measurement per pixel for reconstruction, and use direct components of the images we capture. Again, like in the case of the "Water Sequence" we hypothesize various depth values along each back-projected pixel, and test their validity using computed radiance ratios.

Figure 3.11 shows depth and normal maps computed using our approach. Notice again how we obtain smooth maps even though no smoothness constraints are imposed on the results.

Potential Applications

In the previous section, we showed how using radiance measurements we can extract the shape of transparent objects with minimal measurements even in challenging scenarios. Now we apply the same theory inversely to the problem of calibrating the radiometric response of the camera.

Radiometric calibration is the process of extracting the function that maps image intensities to physical responses of the CCD pixels to incoming light. In short, it maps intensities to irradiances.

Earlier methods on radiometric calibration [START_REF] Mitsunaga | Radiometric Self Calibration[END_REF] focused on obtaining multiple images at varying exposures in order to obtain ratios of intensities that are a function of the ratio of exposure times and the radiometric calibration. Given ratio of exposure times and intensities, one could fit a non-linear model to intensities that would make their ratios converge to exposure ratios [START_REF] Mitsunaga | Radiometric Self Calibration[END_REF]. However, this requires several images to compute the ratios, and is limited since camera exposure times only have discrete levels of increments.

However, we could use an image from a scene like the "Water Sequence" where the object is known to be planar, in order to obtain radiance ratios that are related to the depth of the water surface. Such a scene consists of minimal interreflections and scattering, and so intensity ratios from radiometrically uncalibrated images are a function of the nonlinear radiometric responce of the camera only. Since we can compute the depth of the scene using purely geometric approaches [START_REF] Sturm | Algorithms for plane based pose estimation[END_REF], we can also obtain a set of "desired" intensity ratios. Moreover, since depth varies continuously, so do the intensity ratios and thus we get more data from a single image than from multiple images with different exposure.

This could potentially be used to radiometrically calibrate the camera.

Discussion and Conclusion

Reconstruction of transparent objects remains a challenging problem because of the lack of cues that are normally available for other objects. In this paper, we showed the existence of an approach that combines two of the more widely available cues, namely geometric and photometric cues. We showed how this leads to simplified acquisition, a decently robust algorithm, new minimal solutions, and presented challenging scenarios where our approach yielded accurate reconstructions. Applications of our approach lie in valdiating outputs of light manipulation/inference tasks like light-transport matrix estimation and interreflection removal apart from reconstruction and radiometric calibration of cameras.

4

Reconstructing Planar Light-Paths

Introduction

In the second chapter, we considered the case of a camera looking at a scene in a different medium through a planar interface. A generalization of this scenario would be a camera observing a scene This results in considerable simplification of the calibration problem via a two-step process, where the axis is computed first. Without such a simplification, calibration is difficult to achieve. Secondly, we show that axis estimation can be mapped to the classical relative orientation problem (essential matrix estimation) for which excellent solutions (e.g. 5-point algorithm [START_REF] Nistér | An efficient solution to the five-point relative pose problem[END_REF]) already exist. In fact, calibration can be done using a single plane similar to [START_REF] Tardif | Calibration of cameras with radially symmetric distortion[END_REF]. We then discuss several cases in which the computed axis estimate might be used to recover the other uknown parameters of each light-path.

Our primary contributions are as follows.

• We show that the geometry of rays in flat refraction systems corresponds to an axial camera, leading to a unified theory for calibrating such systems with multiple layers.

• By demonstrating the equivalence with classical essential matrix estimation, we propose efficient and robust algorithms for calibration using planar as well as nonplanar objects.

• We extend the same theory for flat refraction to the case of flat and spherical reflection. In fact we show that some of these cases are analogous.

• We derive theoretical upper bounds of the approximation involved when these models do not fit the actual data, in the case of normal estimation.

Related Work

Maas [START_REF] Maas | New developments in multimedia photogrammetry[END_REF] considered a three layer system assuming that the image plane is parallel to the refractive interfaces. His approach corrects for the radial shift of the projected 3D points using optimization.

Treibitz et al. [START_REF] Treibitz | Flat refractive geometry[END_REF] consider a single refraction with known refractive index in an underwater imaging scenario. They assume the distance of the interface as the single unknown parameter (when the camera is internally calibrated) and perform calibration using known depth of a planar checkerboard.

The image plane is parallel to the interface in their setup as well. In contrast, we (a) do not assume that the refractive interfaces are frontoparallel, (b) handle multiple layers with unknown layer distances, (c) consider known/unknown refractive indices, and (d) do not assume known pose of the calibrating object. We only assume that the camera is internally calibrated.

3D reconstruction under reflections/refractions has been explored in [22; 26; 73; 92; 124] either for reconstructing the scene or the medium itself. Chen et al. [START_REF] Chang | Multi-view 3D reconstruction for scenes under the refractive plane with known vertical direction[END_REF] captured two images, with and without a thick glass slab for 3D reconstruction. Both images are required to estimate the orientation of the slab and an additional image to obtain the refractive index. We show that a single set of 2D-3D correspondences from a single photo allows estimating medium thickness as well as refractive index.

Other works assume known vertical direction [START_REF] Chen | Self-calibrating depth from refraction[END_REF] or require several images for calibration [START_REF] Nigel | Dynamic refraction stereo[END_REF].

Steger and Kutulakos [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF] showed that light-path triangulation becomes degenerate when the entire light-path lies on a plane, which is the case here. Their goal is to compute the shape of the refractive medium, and they consider each light-path independently. In contrast, we have partial knowledge of shape, and light-paths can be parameterized. Thus, we can use information from multiple lightpaths to obtain the refractive index. For two refractions (air-medium-air), our analysis is consistent with [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF] in that the distance to the medium cannot be estimated. However, we show theoretically that if all refractive indices are different, light paths are not degenerate for any number of layers.

Non-Central/Axial Cameras Pless [START_REF] Pless | Using many cameras as one[END_REF] proposed algorithms for relative motion estimation for calibrated noncentral cameras. Chari and Sturm [START_REF] Chari | Multiple-view geometry of the refractive plane[END_REF] showed the existence of geometric entities such as fundamental matrix considering refraction. Generic camera calibration algorithms [START_REF] Sturm | A generic concept for camera calibration[END_REF] have been proposed to calibrate non-central cameras. Li et al. [START_REF] Li | A linear approach to motion estimation using generalized camera models[END_REF] analyzed the degeneracies in axial cameras for motion estimation. Ramalingam et al. [START_REF] Ramalingam | Theory and calibration algorithms for axial cameras[END_REF] proposed a general framework for calibrating axial cameras using three checkerboards. Their parameterization considers a general axial system and does not consider the explicit case of a pin-hole camera observing refractive/reflective surfaces, while also involving two rotations/translations. In contrast, we consider the specific case of an axial system passing through the camera center, allowing calibration from a single plane. Models based on radial distortion for calibration assume known center of distortion [START_REF] Thirthala | Multi-view geometry of 1D radial cameras and its applications to omnidirectional camera calibration[END_REF] or model each distortion circle separately [START_REF] Tardif | Calibration of cameras with radially symmetric distortion[END_REF]. We use a global model with fewer parameters for flat refractive systems.

Plane of Reflection / Refraction Constraint

Axial Camera: An axial camera is defined as a camera that captures an axial system of rays with the axis passing through the camera center. While the axial nature of many catadioptric systems is already well known [START_REF] Srikumar Ramalingam | Theory and Calibration for Axial Cameras[END_REF], we show that an N -layer flat refraction system, with n being the normal of the flat layer(s), corresponds to an axial camera. The axis is defined as the line parallel to n passing through the camera center (origin) and let â be its direction vector. Let Π be the plane of refraction (POR) containing the axis and a given camera ray. The normal n lies on Π. From Snell's law, the incoming ray, the normal and the refracted ray lie on the same plane at the point of refraction. Since the refracted ray from one layer is the incident ray for the next layer (with the same normal), hence, by induction, the entire light-path should lie on Π and all the refracted rays should intersect the axis.

Thus, all outgoing rays intersect the axis and the system is axial. Suppose we consider one such light-path, with its first ray direction being denoted by l. Let this light-path eventually pass through the 3D point RX + t (where X is known to us R and t are not). This 3D point RX + t should also lie on Π. Thus, the plane of refraction constraint for each 3D point can be written as POR Constraint :

(RX + t) ⊤ (â × l) = 0, (4.1)
where (â × l) is the normal to POR. Note that the POR constraint is independent of the number of layers N , their thicknesses d k , and the refractive indices µ k . It only depends on the axis and pose parameters. It is also independent of the nature of specularity, i.e. it applies to both reflection and refraction. Also, note that this independence means that light-paths with different numbers of bounces could be clubbed together in the POR constraint in order to estimate the axis and pose parameters.

This fact is illustrated in Figure 4.2.

Object Points Camera

Axis Computation

Let M(:) be the vector formed by stacking the colums of a matrix M and let ⊗ denote the kronecker product. Let [â] × be the 3 × 3 skew-symmetric matrix obtained from 3-vector â. The POR constraint can be re-written as

l ⊤ (â × (RX + t)) = l ⊤ EX + l ⊤ s = 0, (4.2)
where E = [â] × R and s = â × t. Note that s ⊤ â = 0 and thus the full translation t cannot be estimated using POR constraints. The component of t in the direction of the axis, t A , vanishes in s. Thus, we have 7 degrees of freedom that can be recovered: 2 for the axis, 3 for rotation and 2 for translation.

11-point Linear Algorithm : Stacking equations for 11 (X i , l i ) correspondences, we get a linear system

     (X 1⊤ ⊗ l 1⊤ ) l 1⊤ . . . . . . (X 11⊤ ⊗ l 11⊤ ) l 11⊤      B   E(:) s   = 0, (4.3)
where B is an 11 × 12 matrix whose rank in general is 11. Let B = UΣV ⊤ be the SVD of B. The solution is given by the right null singular vector of B (last column of V ). The scale factor is obtained by setting the norm of E to one.

8-point Algorithm : Notice the striking similarity between our E matrix ([â] × R) and the essential matrix [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF] for relative motion between two perspective cameras ([t] × R). This implies that we can map the axis estimation to the 5-point algorithm for essential matrix computation [START_REF] Nistér | An efficient solution to the five-point relative pose problem[END_REF]. Given 8 correspondences, we obtain an 8 × 12 matrix B as above. Let V 4 i=1 be the right null singular vectors of B. The solution lies in a four dimensional subspace   E(:)

s   = λ 1 V 1 + λ 2 V 2 + λ 3 V 3 + λ 4 V 4 , (4.4)
where the λ i 's are unknown scalars. λ 4 can be set to 1 since the solution can be recovered only upto a scale factor anyhow. The 'E' part of the solution is

E(:) = λ 1 V 1 (1 : 9) + λ 2 V 2 (1 : 9) + λ 3 V 3 (1 : 9) + V 4 (1 : 9), (4.5)
where V i (1 : 9) denote the first 9 elements of V i . Now the λ i 's can be computed using the solution in [103] by providing the above subspace vectors for E.

After recovering E and s, the axis is computed as the left null singular vector of E (since â⊤ E = 0).

The sign ambiguity in the axis is resolved by letting it point away from the camera. The translation orthogonal to the axis, t A ⊥ , can be obtained as s × â. Four solutions for R are recovered from E

as in [START_REF] Nistér | An efficient solution to the five-point relative pose problem[END_REF]. The correct solution can be recovered along with the other unknown parameters like layer thickness by analyzing the feasibility of different light-path setups as shown in Sections 4.6

Interestingly, the axis estimation is similar to the center of distortion estimation for central cameras in [START_REF] Hartley | Parameter-free radial distortion correction with center of distortion estimation[END_REF] and our 8pt algorithm can be applied.

The axis computation approach mentioned thus far is really a partial calibration of the axial camera setup mentioned earlier. While the 8pt algorithm requires generic 3D points in order to give unique solutions, it would be useful to find an alternate approach that could work with planar 3D correspondences for a variety of reasons like ease of capture, fewer images etc..

Planar Grid Calibration

We now show that calibration can also be done using a single planar grid, which is useful in practice. We describe an 8pt algorithm as follows. Without loss of generality, assume the plane is aligned with the x y plane (X i z = 0). Substituting in the coplanarity equation 4.2, the columns 7, 8, 9 of B matrix reduce to zero. Let B ′ be the reduced 8 × 9 matrix, whose rank is 8.

Thus, there is a single right null singular vector, contrary to the above case for the original full 8 × 12 matrix B. This singular vector directly gives the first two columns of E, along with s. The last column of E is recovered using Demazure constraints [START_REF] Demazure | Sur deux problemes de reconstruction[END_REF] and det(E) = 0 constraint, as follows. Note that since the above equation has only z 4 and z 2 terms, we can substitute γ = z 2 and get a quadratic equation in γ. In our experiments, we see that there are two real solutions and two imaginary solutions for z, where the real solutions differ in sign. Thus, we obtain a pair of E matrices which differ in the sign of their last column. Each pair of obtained rotation matrices also have the same property. The correct rotation matrix is chosen by checking for the determinant value of one (The determinant of incorrect rotation matrices equals -1, corresponding to a reflection).

Summary

We showed that multiple flat refractive layers correspond to an axial system, and that in such a system all light-paths are planar in nature. We presented two approaches to compute the axis and some parameters of the transformation between object and camera coordinate systems. The major advantage in axial systems is that the resulting light-paths can be analyzed in 2D rather than 3D, and this reduces the complexity of the problems considerably. This is done by projecting all 3D quantities of a single light-path onto its POR. In the next sections, we will use this property to analyze 3 kinds of axial systems: planar refractive, planar reflective and spherical reflective systems.

Before we proceed, however, we analyze the error bounds of our approach. This is done in two steps. In the first step we show simulation results that measure the noise resilience of our 8pt and 11pt algorithms, since a theoretical analysis turns out to be highly complex. We show that the 8pt algorithm gives encouraging bounded estimates of the axis and pose parameters. Given a bounded axis estimate, we then analyze the accuracy of computing the POR normal. It requires propagating an error estimate through a cross product, and we derive the error bounds on the propagated quantity.

While we will see that propagating this error through the remaining solutions to be presented in this chapter is difficult, knowledge of this error could provide useful information to produce more robust algorithms in the future.

Simulations

We present simulations for estimating the axis with Gaussian noise (variance σ 2 pixels) in feature points for Case 1 and Case 2, shown in Figure 4.3. We assume a camera with FOV of 45 • and resolution 1000 2 pixels. The scene is set by choosing d 0 = 300 units, d 1 = 450 units, µ 1 = 1.5

and µ 2 = 1, where the variables d 0 , d 1 represent the perpendicular distance between the camera center and the two layers of refraction, while the variables µ 1 , µ 2 represent relative refractive index.

We perform 100 trials for each noise setting, and plot the average error in axis, rotation and t A ⊥ in Figure 4.3. For each trial, the axis is randomly generated in a cone of half-angle 45 • around the camera's optical axis. 100 3D points are randomly generated along with R and t so that they lie within [d 0 , 2d 0 ] after the last layer. For each trial, a RANSAC based framework is employed for both 11pt and 8pt algorithms using 200 iterations to choose the best solution using the POR error. Rotation error is defined as the minimum angle of rotation required to go from the estimated rotation to the true rotation. Similarly, axis error is defined as the angle between the estimated and the true axis.

The translation error is computed as the norm of the translation error vector and is normalized using the corresponding layer thickness.

Degenerate case

Notice that when l n, the POR constraint cannot be used. This is an inherent degenerate case in all axial systems. It also induces a computational bottleneck since light-paths that are close to the line parallel to the axis and passing through the camera center are unreliable for axis computation. As expected, the 8pt algorithm performs significantly better than the 11pt algorithm as well as using all points in a least squares fashion.

Error Bound: Pairwise computation

We showed earlier how the POR constraint could be used to obtain axis and partial pose parameters.

Once the axis is computed, the POR normal corresponding to a particular image ray l can be computed as l × â. Note that because of errors, the computed value of the axis âδ might be different from the ground truth â. Assuming no error in l, we would like to know the error in the computation of the normal of POR Π.

To start the derivation, let us observe that both the quantities l × â and l × âδ lie in the plane that is perpendicular to l. Let us call this plane Θ, while noting that this is not the plane of reflection or refraction. Now let us list the other vectors that lie in this plane.

l × (â × l) → l • (l × (â × l)) = 0 (4.25) âδ -(l • âδ )l → âδ -(l • âδ )l • l = 0, ( l = 1) (4.26)
Let us also note that the second vector in the equations above is perpendicular to l × âδ while the first vector is clearly perpendicular to l × â. Proving the first claim is trivial.

âδ -(l • âδ )l • (l × âδ ) = (â δ ) • (l × âδ ) = 0 -(l • âδ ) (l • (l × âδ )) = 0 (4.27) = 0 (4.28)
Thus, we finally get the means to measure the angle between the two cross products.

∠(l × â, l × âδ ) = ∠(l × (l × â), âδ -(l • âδ )l) (4.29)
A measure of this angle is given by the dot product of the two vectors.

(l × â) × l = â -l(l • â) (4.30) (l × (â × l)) • âδ -(l • âδ )l = (â -l(l • â)) • âδ -(l • âδ )l (4.31) = â • âδ -(l • âδ )(l • â) -(l • âδ )(l • â) + (l • âδ )(l • â) (4.32) = â • âδ -(l • âδ )(l • â) (4.33) = cos(δ) -(l • âδ )(l • â) (4.34) ≈ 1 -(l • âδ )(l • â) (4.35)
where ≈ denotes the fact that cos(δ) is approximately equal to 1 for small values of δ. In the above equation, vectors have not been normalized, and so the actual value of the cosine will be given as

cos(∠(l × â, l × âδ )) = cos(δ) -(l • âδ )(l • â) 1 -(l • â) 2 1 -(l • âδ ) 2 (4.36) ≈ 1 -(l • âδ )(l • â) 1 -(l • â) 2 1 -(l • âδ ) 2 (4.37)
So far we have derived an equation to find the cosine of the angle between l × â and l × âδ . The next step is to bound this angle by indirectly bounding the cosine value. To do this let us observe that cosine and sine are monotonic functions in the domain 0 < θ < π/2. Thus assuming that the angle between l and â is θ , we need to bound the value of l × âδ . Observe that this assumption can be used to re-write equation 4.36 as

cos(∠(l × â, l × âδ )) = cos(δ) -cos(θ ) cos(θ δ ) sin(θ ) sin(θ δ ) (4.38)
Now, observe that θ δ can lie in the range, θ -δ < θ δ < θ + δ. Also observe that in the extreme cases the above equation takes the value of 1, which means that the minimum angle between the two vectors is 0. In order, to find the maximum value of the angle, we differentiate the above equation ) , θ = 4 5

• Random Samples Figure 4
.4: This figure presents the simulation results that confirm the bounds of our approximation theory. Two random 3D unit vectors were taken, and one of them was perturbed with several varying degrees. The resulting cross product of the two vectors was compared with the ground truth cross product, and the angle between them was measured. In the above figure, each colored dot represents the varying result plotted versus the ground truth angle between the two vectors. The curve that represents the theoretical bound is plotted to show the validity of our theory.

FRC :

(RX + t -p n ) × l n = 0, (4.52)
Our goal is to estimate the unknown calibration parameters n,

[d k ] N -1 k=0 , [µ k ] N k=0 as well as the un- known pose [R, t] given K 2D-3D correspondences [l i 0 , X i ] K i=1 .
From Snell's law, µ i sin(θ i ) = µ i+1 sin(θ i+1 ), where θ i is the angle between l i and n. This can be written in vector form as

l i+1 = a i+1 l i + b i+1 n, (4.53)
where a i+1 = µ i /µ i+1 and

b i+1 = -µ i l ⊤ i n -µ 2 i (l ⊤ i n) 2 -(µ 2 i -µ 2 i+1 )l ⊤ i l i µ i+1 (4.54)
Since Snell's law only depends on the ratio of the refractive indices, we assume µ 0 = 1 without loss of generality. We first derive the FRC for a single layer and a single 3D point X. The refraction point p 1 equals -d 0 l 0 /(l ⊤ 0 n). Substituting in 4.52, and using equation 4.53

(RX + t) × (a 1 l 0 + b 1 n) + b 1 d 0 (l 0 × n)/(l ⊤ 0 n) = 0, (4.55)
After substituting for a 1 and b 1 , using equation 4.54 and removing the square root term in b 1 , we get an equation with second order terms of R, t and µ i and sixth order terms of n. Thus, directly solving the FRC is quite difficult. More importantly, the complexity of the FRC equation increases with each additional layer and due to the square root term in each b i . Thus, in order to solve this problem efficiently, we need to analyze the geometry of underlying rays to derive simpler constraints. While the POR constraint (Section 4.2) allows us to estimate the normal n, we use the framework outlined above to show how the geometry of the rays could be used to estimate 5 out of 6 pose parameters independently of d i 's and µ i 's (Section 4.6).

General Planar Reflective Geometry

We now derive a constraint similar to the one proposed in Section 4.3. Consider the general setup for flat reflective geometry as shown in Figure 4.1, where a perspective camera observes a known calibration object after N reflections off parallel reflective surfaces. We work in the camera coordinate system with the camera center at the origin. Let n denote the common surface normal to all layers and [d k ] N -1 k=0 be the perpendicular distance between each successive layer and the next. d 0 represents the distance between the camera and the first layer. Let [X] K i=1 denote K 3D points on the object which are known in the object coordinate system and let [R, t] be the rigid transformation of these points.

Plane Reflective Constraint

As in Section 4.3.1, we denote light-path direction vectors by l i k . The last reflected ray l should be parallel to the line joining the transformed 3D point RX + t and the reflection point p on the last layer.

Thus, the equivalent to equation 4.52 is the Plane Reflection Constraint (PRC)

PRC : (RX + t -p) × l = 0, (4.56)
We already outlined our approach to estimate the axis and partial translation parametsr in Section 4.2

In this case, our goal is to estimate the parameters n, [d k ] N -1 k=0 and the unknown pose parameter t

A given K 2D-3D correspondences [l i 0 , X i ] K i=1 .
Using the laws of reflection, we can write the relationship between two consecutive segments of a light-path as

l k+1 = a k+1 l k + b k+1 n, (4.57)
where a k+1 = -1 and b k+1 = 2(l ⊤ k n). Substituting this in equation 4.56, we get the following constraint for PRC of one layer (with p 1 = -d 0 l 0 /(l ⊤ 0 n))

(RX + t -p 1 ) × l 1 = 0, (4.58)
(RX + t -p 1 ) × (2(l ⊤ 0 n)n -l 0 ) = 0, (4.59)
(RX + t) × (2(l ⊤ 0 n)n -l 0 ) + d 0 /(l ⊤ 0 n)2(l ⊤ 0 n)(l 0 × n) = 0, (4.60)
(RX + t) × (2(l ⊤ 0 n)n -l 0 ) + 2d 0 Π = 0 (4.61)
where Π is the direction vector of the POR. When R, n and t A ⊥ are known, notice that the above equation is linear in unknown quantities d 0 and α. Also note that recursively expanding the definition of l k+1 would give us the following

l k+1 = -l k + 2(l ⊤ k n)n (4.62) = -(-l k-1 + 2(l ⊤ k-1 n)n) + 2(-l ⊤ k-1 n + 2(l ⊤ k-1 n))n, ∵ n ⊤ n = 1 (4.63) = l k-1 (4.64)
This means that there are only 2 independent PRC equations in this case. This is understandable since reflection is a symmetric property (the incident ray is a reflection of the reflected ray) and in the absence of additional parameters like refractive index in the case of refraction, the degrees of freedom are reduced. Now, consider the case of two bounces. Since

p 2 = -d 1 l 1 /(l ⊤ 1 n) -d 0 l 0 /(l ⊤ 0 n)
and l 2 = l 0 , the PRC equation reduces to

(RX + t + 1 k=0 d k l k /(l ⊤ k n)) × l 0 = 0 (4.65)
Notice that the term d 0 vanishes in the above equation and so we are left with 2 unknowns d 1 and α.

However, once we solve for d 1 and α, we can solve for d 0 by using the following equation Degeneracy Note that in the case of planar reflections, there is a unique case of degeneracy. If all the light-paths are produced by reflection in each mirror of the system, then the entire system becomes single viewpoint. Such a system has infinitely many axes passing through it, and hence the POR constraint would not apply. Thus, for planar reflection, it is necessary that different light-paths have different numbers of bounces or intercept sets of different mirrors.

(RX + t -p 2 ) = γl 0 ( 4

General Spherical Reflective Geometry

In Section 4.4, we looked at the case of general plane reflective geometry. In this case, the equations turn out to be linear in unknowns like the translation component t A and the perpendicular distances

[d k ] N -1 k=0
. We now consider the case of general spherical reflective geometry, when a camera looks at many axially-aligned spheres with different radii [r k ] N -1 k=0 with centres with distances [d k ] N -1 k=0 between consecutive spheres. d 0 represents the centre of the first sphere in the order chosen.

Spherical Reflective Constraint

As in earlier sections, the constraint we impose for any light-path is that the last segment be in the same direction as the vector from the last point p and the transformed 3D points RX + t.

SRC :

(RX + t -p) × l = 0, (4.67)
The main difference between the above equation and PRC, however, lies in the fact that the term l N is no longer a known quantity given the axis n. Thus, this case is similar to FRC in the event that refractive indices were unknown. As earlier, we can express the relationship between direction vectors of two consecutive segments of a light-path as (Section 4. Given l k , p k , s k and r k+1 , the following equations could be used to derive the values of a k+1 and b k+1 .

l k+1 = 2(l ⊤ k n k+1 )n k+1 -l k (4.69) n k+1 = p k+1 -s k r k+1 ( 4
β k+1 = l ⊤ k s k -p ⊤ k l k -(l ⊤ k s k -p ⊤ k l k ) 2 + 2p ⊤ k s k -s ⊤ k s k + r 2 k+1 -p ⊤ k p k (4.72) s k = - k+1 j=0 d j n (4.73)
where we get β k+1 by solving the quadratic equation p k + β k+1 l k -s k 2 = r 2 k+1 . Note that β k+1 itself contains both bilinear terms and square root terms of l k and n, while l k+1 is quadratic in β k+1 . Since we found this set of equations highly difficult to untangle, we now present the minimal case of a single bounce off one sphere. In this case p 1 reduces to 0 since it is the camera centre, s 0 = -d 0 n, These three cases correspond to the most practical situations that arise, for example, when a camera looks through a fishtank or in underwater photography. We then look at the case of SRC for single and double bounces. Later we show that while both PRC and SRC might represent restricted scenarios of reconstruction, their applicability extends to more general cases since many smooth reflective surfaces could be approximately modeled as sets of planes or spheres. Before proceeding with the derivations, we first transform the coordinate system that we work in, for the sake of convenience.

Coordinate Transformations : We first apply the computed R and t A ⊥ to the 3D points X. Let X c = RX + t A ⊥ . With known axis, the analysis can be done in 2D on the plane of refraction (POR) itself as shown in Figure 4.7. Let t A = αA, where α is the unknown translation magnitude along the axis.

Let [z 2 , z 1 ] denote an orthogonal coordinate system on the POR. We choose z 1 along the axis. For a given camera ray l 0 , let z 2 = z 1 × (z 1 × l 0 ) be the orthogonal direction. The projection of X c on POR is given by u = [u x , u y ], where u x = z ⊤ 2 X c and u y = z ⊤ 1 X c . Similarly, the direction vector l i of each ray on the light-path of l 0 can be represented by a 2D vector vp i on POR, whose components are given by z ⊤ 2 v i and z ⊤ 1 v i . Let c i = vp ⊤ i z 1 and z p = [0; 1] be a unit 2D vector.

Layer Thickness

We first assume known refractive indices. Section 4.2.1 showed how to compute the axis A, rotation R and translation t A ⊥ orthogonal to the axis. When µ i 's are known, the ray directions for the entire lightpath v 0 (i), . . . , v n (i) can be pre-computed using the estimated A. The remaining unknowns are the layer thickness d i 's and the translation t A along the axis, which can be computed linearly as described below.

Linear System for N Layers

For each correspondence, the FRC for N layer system on its plane of refraction is given by

vp n × (u + αz p -q n ) = 0. (4.79)
This is because the last refracted ray vp n should coincide with the line joining the transformed 3D point u + αz p and the refraction point q n on the last layer.

q n = n-1 i=0 -d i vp i /c i . Substituting, we get vp n × vp 0 c 0 . . . vp n-1 c n-1 z p        d 0 . . . d n-1 α        = -vp n × u. (4.80)
Thus, each correspondence gives one linear equation in d i 's and α. By stacking K > n correspondences, the resulting linear system can be solved to obtain d i 's and α for n layers. After estimating α, the translation t is given by t A ⊥ + αA. However, if µ i = µ n for any i, vp i vp n and d i cannot be estimated. In addition, if µ i = µ j , only the combined layer thickness d i + d j can be estimated, since the corresponding constraints in the linear system become equal. Now we analyze some special cases.

Case 1 (Single Refraction) For a single layer, we have two unknowns d 0 and α and the FRC is given by

vp 1 × vp 0 /c 0 z p   d 0 α   = -vp 1 × u, vp 1 = a 1 vp 0 + b 1 n (4.81)
where a 1 = 1/µ 1 . Using K ≥ 2 correspondences, a least squares solution can be obtained.

Case 2 (Two Refractions) µ 0 = µ 2 This is a common scenario when looking through a refractive medium such as a thick glass slab. Here d 0 , d 1 and α are unknowns. Since µ 0 = µ 2 , vp 2 vp 0 and the FRC reduces to

vp 0 × vp 1 /c 1 z p   d 1 α   = -vp 0 × u (4.82)
Thus, we can only estimate the thickness d 1 of the medium, but not its distance d 0 . This is consistent with the analysis shown in [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF].

Case 3 (Two Refractions) µ 0 = µ 2 Now v 2 and v 0 are not parallel and the FRC is given by

vp 2 × vp 0 /c 0 vp 1 /c 1 z p     d 0 d 1 α     = -vp 2 × u (4.83)
Thus, we can estimate the distance d 0 as well.

Unknown Refractive Indices

In this section, we describe in detail the analytical solutions to compute the layer thickness and translation along the axis when the refractive indices are unknown. We apply the same coordinate transformations as described earlier. Note that on the plane of refraction, the normal n of the refracting layers is given by n = [0; -1].

Case 1: Single Refraction We have three unknowns d 0 , µ 1 and α. When µ ′ i s are unknown, ray directions cannot be pre-computed and FRC needs to be written in terms of camera rays as follows

(α 1 vp 0 + b 1 z 1 ) × (u + αz p + d 0 vp 0 /c 0 ) = 0 (4.84)
Since the camera ray vp 0 is known, we can normalize it. Let vp 0

= [v x ; v y ]. From 4.54, b 1 = v y -µ 2 1 + (v y ) 2 -1 µ 1 (4.85)
Using a 1 and b 1 , vp 1 can be obtained. Substituting vp 1 and vp 0 in the FRC equation 4.81

(d 0 v x -v y u x ) µ 2 1 + (v y ) 2 -1 + v x v y (α -d 0 -v y ) = 0 (4.86)
Removing the square root term, we get

(d 0 v x -v y u x ) 2 (γ + (v y ) 2 -1) = (v x v y (α -d 0 -u y )) 2 (4.87)
where γ = µ 2 1 . γ can be obtained as a function of d 0 and α.

γ = (v x v y (α -d 0 -u y )) 2 (d 0 v x -v y u x ) 2 -(v y ) 2 + 1 (4.88)
Let [EQ i ] 3 i=1 be the 3 equations for 3 correspondences. Using EQ 1 , γ can be obtained as a function of d 0 and α as above. Substituting γ in EQ 2 and EQ 3 makes them cubic in d 0 and quadratic in α. We get the following form for EQ 2 and EQ 3 EQ 2 : k 11 α where k i j depends on known quantities. α 2 can be eliminated between EQ 2 and EQ 3 by

EQ 4 = k 21 k 22 EQ 2 -k 11 k 12 EQ 3 . (4.91)
The resulting EQ 4 is linear in α and cubic in d 0 , using which α can be obtained as a cubic function of d 0 . Substituting α in EQ 3 and simplifying, results in a 6 th degree equation in the single unknown d 0 . Solving it results in 6 solutions. The correct solution in practice is always found by enforcing d 0 > 0, α > d 0 and µ 1 > 0. However, we have been unable to prove theoretically that this solution is unique.

Case 2: Two Refractions, µ 2 = µ 0 In this case, we have four unknowns d 0 , d 1 , µ 1 and α. However, d 0 cannot be estimated as shown in a similar case with known refractive index, earlier. The resulting FRC turns out to be independent of d 0 . For this case, the FRC is given by

vp 0 × (u + αz p -q 2 ) = 0 (4.92)
since vp 2 is parallel to vp 0 . The refraction point q 2 is given by

q 2 = q 1 -d 1 vp 1 /(vp ⊤ 1 n) (4.93)
vp 1 is given by

vp 1 = a 1 vp 0 + b 1 n, (4.94)
where a 1 = 1/µ 1 . Since the camera ray vp 0 is known, we can normalize it. Let

vp 0 = [v x ; v y ]. From 4.92, b 1 = v y -µ 2 1 + (v y ) 2 -1 µ 1 (4.95)
Using a 1 and b 1 , vp 1 and q 2 can be obtained. Substituting vp 1 and vp 0 in the FRC equation 4.92

(d 1 v x -αv x + v y u x -v x u y ) µ 2 1 + (v y ) 2 -1 + d 1 v x v y = 0 (4.96)
Removing the square root term, we get

(d 1 v x -αv x + v y u x -v x u y ) 2 (γ + (v y ) 2 -1) = (d 1 v x v y ) 2 (4.97)
where γ = µ 2 1 . Once again, γ can be obtained as a function of d 1 and α.

γ = (d 1 v x v y ) 2 (d 1 v x -αv x + v y u x -v x u y ) 2 -(v y ) 2 + 1 (4.98)
Similar to Case 1, let [EQ i ] 3 i=1 be the 3 equations for 3 correspondences. Using EQ 1 , γ can be obtained as a function of d 1 and α as above. Substituting γ in EQ 2 and EQ 3 makes them cubic in d 1 and fourth degree in α. We found it difficult to solve in Matlab, due to the large number of terms. Therefore, we used an automatic generator of Gröbner basis solver [START_REF] Kúkelová | Automatic Generator of Minimal Problem Solvers[END_REF] to obtain the final equation. It results in a 6 th degree equation.

Case 3: Two Refractions, µ 2 = µ 0 In this case, we have five unknowns d 0 , d 1 , µ 1 , µ 2 and α. However, this case is extremely difficult to solve and we were unable to get an analytical solution. As shown, in this case the FRC will result in an equation in the above five unknowns, with fourth degree terms of each unknown. Thus, it is clear that more than two layers or multi-layer systems are quite difficult to solve for analytically and require a good initial guess for non-linear refinement, when refractive indices are unknown.

For this case, the FRC is given by vp 2 × (u + αz pq 2 ) (4.99) since vp 2 is not parallel to vp 0 . vp 2 is given by

vp 2 = a 2 vp 1 + b 2 n = a 2 a 1 vp 0 + (a 2 b 1 + b 2 )n (4.100)
where a 2 = µ 1 /µ 2 and

b 2 = µ 2 1 v y µ 1 - v y -µ 2 1 +(v y ) 2 -1 µ 1 2 -µ 2 1 + µ 2 2 -µ 1 v y µ 1 - v y -µ 2 1 +(v y ) 2 -1 µ 1 µ 2 (4.101)
Using a 1 , b 1 , a 2 , b 2 , we can obtain vp 2 and q 2 . Substituting in FRC equation 4.99, we get

k 1 D 1 + k 2 D 1 D 2 + k 3 D 2 = 0, (4.102)
where Removing the square root terms, we get

k 1 = v x v
(k 2 1 D 1 + k 2 3 D 2 -k 2 2 D 1 D 2 ) 2 -4k 2 1 k 2 3 D 1 D 2 = 0 (4.108)
which is a 4 th degree equation in four unknowns d 0 , d 1 , µ 1 and α. The above equation has up to 4 th degree terms of each of the unknowns. We were not able to get a polynomial equation in a single unknown using 5 correspondences. Thus, multi-layer systems require a good initial guess when µ i 's are unknown. Figure 6 shows the pose and calibration estimates for real data (Figure 5) assuming unknown µ 1 for water, which was recovered as 1.296 (relative error 2.55%).

Single Bounce SRC

In this section, we look at solving the SRC equation 4.67 for the case of a single sphere. Note that in this case n = [0; -1] and thus vp ⊤ 0 n = -c 0 , where, as noted earlier c i = vp ⊤ 0 z 1 . To reiterate, the SRC equation for a single sphere can be written as vp 1 × (u + αz p -q 1 ) = 0 The above equation is linear in α and so we obtain α in terms of the other unknowns d 0 and r 1 as (note that β 1 is expressed in terms of d 0 and r 1 )

α = (β 1 D + 1/2r 2 1 )(v y u x -v x u y ) -( Dd 0 )(u x -β 1 v x ) v x (-β 1 D -1/2r 2 1 ) (4.119)
Using another correspondence, we get another equation similar to equation 4.118 but with different coefficients. Unfortunately, substituting the value for α in the new equation results in binomial terms of higher degree in the unkowns d 0 , r 1 . We found it difficult to solve it in both Matlab and using Gröbner Bases. We also tried other representations (using polar coordinaes, for example) but were unable to deduce a solution for this case. Note that a higher number of spheres or shapes with more complexity will contain more unknowns, and hence will be more complicated than the equations The correct solution is found by enforcing α > d i and d i > 0∀i. We also compute the pose obtained using a central (perspective) approximation from the given 2D-3D correspondences (referred by CA). parameters can be obtained using our algorithm. Notice the large translation and reprojection error, and smaller estimated t z when using a central approximation (CA) in Figure 4.8. This is expected because when looking through a denser refractive medium, objects appear closer to the camera. Note that the error due to noise in CA is insignificant compared to the error due to incorrect modeling.

Real Results

We show real results using a water tank of dimensions 508 × 260 × 300mm 3 . We use a Canon Rebel XT camera having resolution of 3456 × 2304 pixels with a 18 -55 mm zoom lens. The camera was internally calibrated offline. Figure 4.10 shows a photo of a scene consisting of three checkerboards, captured by looking through the water tank (facing 260mm side of tank). In order to obtain ground truth, we took another photo in air, using which the poses of the of checkerboards were computed. The resulting 3D points in the coordinate system of the left checkerboard are shown in 4.2), also evident from projected points in Figure 4.10. Interestingly, the central approximation can recover the rotation well enough.

Conclusion and Future Work

In this chapter we conclude this thesis on shape estimation for specular surfaces. We first summarize the main contributions of our work and then discuss possible future directions.

Summary and Discussion

Shape estimation for specular surfaces can be considered one of the last frontiers of the shape estimation problem, since for most other objects reasonably accurate algorithms exist under appropriate conditions. Because of this fact, in this thesis, we have looked at different approaches to the specular shape estimation problem. While centering around the light-path triangulation framework, we try to use different conditions and inputs to derive the following algorithms for shape estimation

• In Chapter 2, we derive the multiple view geometry of flat refraction. We consider the case where the camera and scene are in different medium, separated by a flat transparent refrac-tive surface. We analyze the multiple view geometry setup arising from such a scenario, and present useful representations for the camera projection matrix, the fundamental matrix and the homography matrix. We also show some conditions under which the normal to this surface might be easily estimated. The underlying philosophy of this approach was to gain a geometric understanding of image formation under flat refraction, and our work shows that under such circumstances it is beneficial to group the light-paths corresponding to the image of a 3D line into one entity in order to analyze the situation.

• In Chapter 3, we explore the idea of adding photometric information to the light-path triangulation framework. While the resulting geometric analysis framework remains similar to that of [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF], we show that addition of this information results in reducing the minimal requirements of reconstruction. We divide our analysis into single bounce and double bounce cases, and show encouraging results and comparisons on the single bounce case. We also theoretically analyze the conditions under which double bounce reconstruction might be possible. However at present practical problems limit our ability to produce accurate results.

• In Chapter 4, we analyze a different grouping of light-paths, when the underlying system of rays captured by the camera is axial in nature. We show that under many circumstances like refraction across parallel flat layers, such a system produces planar light-paths. We analyze the properties arising from this planarity and demonstrate the possibility of going beyond the limitations propounded in [START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF] in some cases. While encouraging results are presented on real scenarios like viewing across parallel refractive layers, we also derive unique insights into some specular reflective setups. Comparisons with a central approximation approach show the benefits of our methods.

To summarize, there is a lot of potential in light-path analysis for shape estimation of specular surfaces. While we presented 3 approaches with encouraging results, we believe there is a lot of scope for future work in this area.

Future Work

As discussed in each of the chapters, there is large room for improvements. Based on this, we list a few potential future directions for shape estimation and discuss some potential applications.

• One of the potential drawbacks of the phenomenon of specular refraction / reflection is that given all the parameters like knowledge of 3D points, specular surface geometry and camera poses and intrinsics, it is difficult to analytically derive the projection of a selected 3D point onto the image. Primarily, this problem occurs due to the piece-wise linear nature of the light-path associated with that 3D point. While a few approaches have been presented in the past [START_REF] Agrawal | A theory of flat refractive geometry[END_REF],

applications like shape estimation or ray tracing would benefit from more research in this area.

While analytical formulae generally tend to be of high order, methods like Gröbner basis could be used for arriving at hitherto unknown solutions. Of course, presence of such analytical solutions would also help in designing new optimization based reconstruction approaches for specular surfaces.

• While a lot of research has been devoted to the geometry related to specular surfaces, little has been done in other areas like appearance based analysis of transparency [START_REF] Levin | Separating reflections from a single image using local features[END_REF]. Especially in problems like robot navigation, such appearance based approaches are essential to isolate transparent objects for further geometric analysis if required. In fact, exploration of the relationship between appearance and geometry of a specular surface has found early interest [START_REF] Koenderink | Photometric invariants related to solid shape[END_REF],

and might be of immense use for shape estimation problems.

• One of the major bottlenecks to multiview reconstruction of specular surfaces is the correspondence problem. While it is impossible to estimate the correspondence between the location of the image of a 3D point on the surface of a specular object in two camera views, an analysis of simple circumstances (like points on a sphere or quadric) might provide better information for both reconstruction and specular flow problems. The fact that many everyday specular objects' shape can be describe in terms of such simple primitives highlights the potential applications of such a theory.

In summary, in this thesis we have presented multiple approaches to shape estimation of specular surfaces. In the future, we hope that our work will find useful applications in the fields of reconstruction, ray tracing and general image understanding of specular surfaces.

.

Partial Polarization

When unpolarized light with electric field magnitude || falls on a dielectric transparent object at an angle θ 1 and is refracted at an angle θ 2 , the reflected and transmitted portions of light are partially polarized. Given refractive indices n 1 and n 2 , let us compute the perpendicular and parallel components of the reflected ray. A similar derivation can be obtained for the transmitted ray.

E 1 cos(ψ) = k R s E cos(φ) (1)
E 1 sin(ψ) = k R p E sin(φ) (2) tan(ψ) = R p R s tan(φ) (3)
ψ = tan -1 ( R p R s tan(φ)) = g(φ) ( 4
)
where k is the normalizing factor to make sure both sides are components of unit vectors, φ denotes the angle between the electric field and its parallel component, and ψ denotes the same angle for the reflected ray. Since the incoming light is unpolarized, the probability of any φ occuring is ρ(φ) = 1 π .

129

Thus denoting λ = R p R s
, the probability of ψ can be computed using a variable transformation.

ρ(ψ) = 1 |g ′ (φ)| ρ(φ) (5) g ′ (φ) = 1 1 + λ 2 tan 2 (φ) (λ sec 2 (φ)) (6)
= 1

1 + λ 2 tan 2 (φ) (λ(1 + tan 2 (φ)))

= 1

1 + tan 2 (ψ) Notice that R p = 0 at Brewster's angle, which makes ρ(ψ) = 0 for all angles except ψ = 0 (where denominator reduces to R p , and so ρ(0) is undefined). This agrees with the theory that reflected light is linearly polarized perpendicular to the plane of refraction. When this reflected or transmitted light reaches a second refractive surface, the Fresnel equations apply on the basis of a new plane of refraction which is different from the previous one. Thus the new electric field E 2 is a rotated version of the electric field associated with the reflected or transmitted light E 1 . Fortunately, this just results in a shift in angle space, so ψ ′ = ψ -α where α is the angle between the two planes of refractions.

λ 2 + tan 2 (ψ) λ (see 3) (8) = λ 2 + tan 2 (ψ) λ(1 + tan 2 (ψ)) = λ 2 -1 + sec 2 (ψ) λ sec 2 (ψ) (9
This follows immediately from the fact that the two planes of refraction contain the transmitted or reflected ray, and hence thier normals must lie in a plane perpendicular to the direction of propagation of the ray.

.

Second Bounce Radiance Ratio

Following equation 3.13, we now determine the radiance ratio for the case when two reflections occur. The other cases for any combination of two bounces follows the same pattern. In order to compute the radiance ratio after the first bounce, first we need to compute the intensity of a single ray after the first bounce.

1 E = 1 R s E 2 cos(φ) + 1 R p E 2 sin(φ) (14)
Since tan(φ) =

Thus, we get

1 E = 1 E = E 1 R s 1 R p 1 R p cos 2 (ψ) + 1 R s sin 2 (ψ) (17)
1

E 2 = E 2 1 R p k 2 cos 2 (ψ) + sin 2 (ψ) (18)

4 1. 2 7 1. 3 ( 10 1. 4 10 1. 5

 4273104105 natural (b) specular surfaces. Commonly occuring (c) refractive and (d) reflective surfaces. Specular surfaces used in (e) cameras (f) catadioptrics (g) underwater photography. Figure (a) courtesy Andre Gunther. (http://www.aguntherphotography.com/greatlakes/chicago/millenium-park-bean.html). Figure (f) courtesy Maxime Lhuiller (http://wwwlasmea. bpclermont.fr/Personnel/Maxime.Lhuillier/Omni3D.html). . . . . . . . . . . . . . . . . . Catadioptric systems are widely used for panoramic image construction and visual SLAM. Two designs of catadioptric systems are presented in (a) [136] and (b) [69]. (c) A typical image captured by a catadioptric camera (rectified image shown on the top) [76] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Left) Transparent surfaces with known 3D structure may be used for 3D reconstruction of other objects [26] (object reconstructed depthmap in inset). (Middle), (Right) Acquisition of transparent surfaces is important for realistic rendering [85] . . . . . . . The different classes of transparent surfaces as described in [67] . . . . . . . . . . . . . Examples considered in this thesis. Depiction of a (a) reflective sphere, and a (b)

15 iv LIST OF FIGURES 1 . 7 16 1. 8

 1517168 known/unknown 3D point p undergoes 1 or more reflections/refractions before being imaged by a camera c. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Acquisition of transparent surfaces [52; 97](Top row) Setups to acquire participating media (wine) and inhomogeneous media (milk in water). (c) Rendering using acquired material properties. (d) One of the acquired images, and reconstruction shown in 3 views. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reconstruction of transparent surfaces. (Top row) Setups to reconstruct transparent objects ([73; 88; 91] in order). (Bottom row). 3D point cloud, mesh and normal map of the reconstructed objects produced using the various methods. . . . . . . . . . . . . . 18 2.1 (a) shows the projection of a 3D line onto the camera after refraction. (b) shows the back-projection of a line in the image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 2.2 (a) shows an illustration of the refraction principle. (b) shows an image of "Snell's

53 vOF FIGURES 3 . 4

 5334 camera pixel). (b) Our setup to "acquire" the shape of transparent objects consists simply of a CRT monitor as light source and a camera looking at light reflected / refracted off the object. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.2 (a) Depiction of a the phenomenon of specular reflection and refraction. A single ray of light incident on the surface of a transparent object is partly reflected and partly refracted. Both rays contain a fraction of energy of the incident ray, but different polarizations. (b) Reflection coefficients computed using Fresnel equations. Courtesy [36] (c) Probabilities of orientation of electric fields on incidence (blue, unpolarized), after reflection (black) and refraction (red) off a surface. Unpolarized light becomes partially polarized after a single bounce. Angles are computed in a plane perpendicular to direction of ray propagation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.3 Figure representing the simplest double bounce case. Two cameras look at a rectangular glass slab at the CRT monitor placed at two positions, producing overall 4 images. Note that in 3D the intersection point N k might not actually exist for all values of d k . . LIST Radiance ratio values plotted for various pairs of incident angles. There is no appreciable difference because of a non-zero angle between the two planes of refraction in the double bounce case. Isocontours for various values of radiance ratio are plotted. Note that they are all curves. Note also that their intersection with the curve curresponding to θ 2 = θ ′ 1 (case of parallel planes) is a single point. . . . . . . . . . . . . . . . . . . . . . . 60 3.5 Illustration depicting various elements of the illumination model. Two camera pixels are back-projected, while a pin-hole model (used for geometric calibration) assumes a back-projected ray, the illumination model assumes a back-projected cylinder. This cylinder strikes pixel j on the monitor and captures H( j) percent of its illumination in the direction θ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.6 Illumination calibration experiment (a) 8 Camera poses w.r.t. CRT monitor plane, used to measure ρ (b) 6 Pixels on the monitor for which ρ values vs θ are plotted in (c). Observe the quadratic nature of ρ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.7 Images of datasets included in this supplementary material, and some images of their acquisition setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

69 3. 10 ( 72 vii 4 . 1 ( 1 Introduction

 691072411 Left Column) Two of 25 images used to compute the direct and global images [99] to remove the effect of interreflections and caustics on radiance measurement. (Middle Left Column) Direct and Global (scaled) components. (Middle Right Column) Difference between the "direct" component and an image taken with a white pattern shown on the monitor. Bottom image is the difference. (Right Column) Correspondence map, Depth map and Reconstructed mesh of "Water" sequence. . . . . . . . . . . . . . . . . . . 70 3.11 (Left top) Normal map of "Fanta bottle" sequence. (insets) Note the fine details captured as a result of radiance ratios. (Centre top) Depth map. Blacker colors are closer to camera. (Right top) Two views of the 3D reconstruction, with lighting to highlight shape variations. Phenomenon like scratches on the bottle, inhomegenous refractive index, violation of single bounce through occlusion are some bad effects, but still reliable reconstructions are achieved. Note that since camera is placed far from the object and monitor, large changes in depth cause small changes in angle. This explains some of the "rough"-ness of the reconstruction. Note also that no smoothing or. optimization is applied for this reconstruction. (Left bottom) Normal map of "Wine glass" sequence. (Left-centre bottom) Depth map. (Right-centre bottom) Depth map produced if no interreflection removal is performed. Notice the lack of depth variation in one of the glasses (compared using insets with blue borders). Some frequency artifacts can be seen (red inset) due to interreflection removal. (Right bottom) Depth difference between the two cases. Best viewed in color. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 3.12 Normal map and Depth maps for the cokebottle and fanta sequences. . . . . . . . . . . . Top Left) Flat and Spherical reflective geometry with 1 layer. (Top Middle) The entire light-path for each pixel lies on a plane and all planes intersect in a common axis passing through the camera center. (Top Right) Once this axis is computed, analysis can be done on the plane of reflection to estimate plane distance and centre / radius of sphere. (Bottom Left) Flat refractive geometry with n layers. (Bottom Middle) The entire light-path for each pixel lies on a plane and all planes intersect in a common axis passing through the camera center. (Bottom Right) After computing the axis, analysis can be done on the plane of refraction to estimate layer thickness and refractive indices. 78 4.2 Figure showing two light-paths for the plane reflective, the spherical reflective and the flat refractive case. The red and blue light-paths bounce off different surfaces and different numbers of times. However, they can all be clubbed into the same POR constraint to estimate R and t A ⊥ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.3 Comparison of 11pt algorithm, 8pt algorithm and using all points in a least squares fashion for estimation of axis, rotation and t A ⊥ using POR constraints. . . . . . . . . . . . 86 4.4 This figure presents the simulation results that confirm the bounds of our approximation theory. Two random 3D unit vectors were taken, and one of them was perturbed with several varying degrees. The resulting cross product of the two vectors was compared with the ground truth cross product, and the angle between them was measured. In the above figure, each colored dot represents the varying result plotted versus the ground truth angle between the two vectors. The curve that represents the theoretical bound is plotted to show the validity of our theory. . . . . . . . . . . . . . . . . . . . . . . 90 4.5 Flat refractive geometry with N layers, reproduced here for clarity. . . . . . . . . . . . . 92 4.6 Figure depicting the case of the k + 1 th bounce off a sphere. Considering the values of r k+1 , s k , l k are known, the point p k+1 is still one of two points that can only be obtained by solving a quadratic equation (line-circle intersection). Note that n k+1 and eventually l k+1 depend directly on p k and thus indirectly on n and unknowns r k+1 , d 0...k+1 . . . . . 97 4.7 Three cases of refraction considered in this chapter. In each case we consider 4 variations: known/unknown refractive indices with known pose parameters, and known/unknown refractive indices with unknown pose parameters . . . . . . . . . . . . . . . . . . . . . . . 99 viii LIST OF FIGURES 4.8 Error in axis, rotation, translation and layer thickness using a planar calibration grid for different noise values, averaged over 100 trials. Rotation and translation errors using a central approximation (CA) are also shown. . . . . . . . . . . . . . . . . . . . . . . 4.9 Rotation, translation and reprojection error using our algorithm versus using a central approximation (CA) for Case 1 and Case 2. The right most plot shows the estimated t z for Case 2 over all 100 trials for σ = 1 pixel. CA estimates the object to be closer to the camera than in reality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.10 (Left) Setup. (Middle) Photo captured by looking through a water tank. Projected 3D points are overlayed by applying pose estimated using CA (green) and our algorithm (red).(Right) Reconstructed 3D points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Image understanding is a long standing fundamental goal of computer vision. The need to make sense of images has tremendous potential for use in various industries from robotics to surveillance.

Contents 1 . 1 3D 3 1. 2 6 1. 2 . 1 6 1. 2 . 2 9 2 1 . 3 14 1. 3 . 1 14 1. 3 . 2 1 . 4

 11326216229131431143214 Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . History with a Specularity Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Case of Mirrors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Case of Transparency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The case of specularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reconstruction vs Acquisition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Physical Modeling vs Light-Path Modeling . . . . . . . . . . . . . . . . . . . . . . . . . 17 Our Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Figure 1 . 1 :

 11 Figure 1.1: Different types of specular surfaces, and common applications. Man-made (a) and natural (b) specular surfaces. Commonly occuring (c) refractive and (d) reflective surfaces. Specular surfaces used in (e) cameras (f) catadioptrics (g) underwater photography. Figure (a) courtesy Andre Gunther. (http://www.aguntherphotography.com/great-lakes/chicago/millenium-park-bean.html). Figure (f) courtesy Maxime Lhuiller (http://wwwlasmea.univ-bpclermont.fr/Personnel/Maxime.Lhuillier/Omni3D.html).

Figure 1 . 2 :

 12 Figure 1.2: Catadioptric systems are widely used for panoramic image construction and visual SLAM.Two designs of catadioptric systems are presented in (a)[START_REF] Kar-Han Tan | Multiview panoramic cameras using mirror pyramids[END_REF] and (b)[START_REF] Kim | Visual SLAM by Single-Camera Catadioptric Stereo[END_REF]. (c) A typical image captured by a catadioptric camera (rectified image shown on the top)[START_REF] Lhuillier | Automatic scene structure and camera motion using a catadioptric system[END_REF]

Figure 1 . 3 :Figure 1 . 4 :

 1314 Figure 1.3: (Left) Transparent surfaces with known 3D structure may be used for 3D reconstruction of other objects[START_REF] Chen | Self-calibrating depth from refraction[END_REF] (object reconstructed depthmap in inset). (Middle), (Right) Acquisition of transparent surfaces is important for realistic rendering[START_REF] Matusik | Acquisition and rendering of transparent and refractive objects[END_REF]

Figure 1 . 5 :

 15 Figure 1.5: Examples considered in this thesis. Depiction of a (a) reflective sphere, and a (b) refractivesphere. Note that the refractive sphere represents two refractions. Our goal is to reconstruct the shape of these specular objects (without assuming a model like a sphere), using information from the surrounding environment and images.

Figure 1 . 6 :

 16 Figure 1.6: Depiction of the specular surface reconstruction problem from [73]. A known/unknown 3D point p undergoes 1 or more reflections/refractions before being imaged by a camera c.

Figure 1 . 7 :

 17 Figure 1.7: Acquisition of transparent surfaces [52; 97](Top row) Setups to acquire participating media (wine) and inhomogeneous media (milk in water). (c) Rendering using acquired material properties. (d) One of the acquired images, and reconstruction shown in 3 views.

Figure 1 . 8 :

 18 Figure 1.8: Reconstruction of transparent surfaces. (Top row) Setups to reconstruct transparent objects ([73; 88; 91] in order). (Bottom row). 3D point cloud, mesh and normal map of the reconstructed objects produced using the various methods.

Contents 2 . 1 21 2. 2 25 2. 2 . 1 25 2. 2 . 2 26 2. 3 28 2. 4 30 2. 5 33 2. 5 . 1 33 2. 5 . 2 34 2. 6

 212122521252226328430533513352346 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Camera Projection Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Back-projecting a Point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Refractive Projection Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Refractive Fundamental Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Refractive Homography Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Exploiting Snell's Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Image of Snell's Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Relative Pose Computation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Figure 2 . 1 :

 21 Figure 2.1: (a) shows the projection of a 3D line onto the camera after refraction. (b) shows the backprojection of a line in the image.

Figure 2 . 2 :

 22 Figure 2.2: (a) shows an illustration of the refraction principle. (b) shows an image of "Snell's Window", a conic that represents the horizon of the outside world. Photo courtesy gerb's photostream, http://www.flickr.com/photos/gerb/196296131/

Contents 3 . 1 36 3. 2 41 3. 3

 31362413 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Physical Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.4 Theory of Bounces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.4.

Figure 3 . 2 :

 32 Figure 3.2: (a) Depiction of a the phenomenon of specular reflection and refraction. A single ray of light incident on the surface of a transparent object is partly reflected and partly refracted. Both rays contain a fraction of energy of the incident ray, but different polarizations. (b) Reflection coefficients computed using Fresnel equations. Courtesy [36] (c) Probabilities of orientation of electric fields on incidence (blue, unpolarized), after reflection (black) and refraction (red) off a surface. Unpolarized light becomes partially polarized after a single bounce. Angles are computed in a plane perpendicular to direction of ray propagation.

. 8 )

 8 Equations (3.3)-(3.4) are called Fresnel equations [36; 60].

refractive index n 1 n 2 .Result 2 : < 1 , 1 , 2 > 1 n 2 .

 2211212 R s +R p (or I α = I r r /I in ) which is a function of the reflection (refraction) angle and the relative Thus when the refractive index is known we have one variable d k and one constraint I α . Thus, < 1, 1, 1 > Triangulation is tractable. Triangulation In this case, we need two observations (X 1 , X 2 ) to compute the value of the incident ray direction, similar to[START_REF] Kutulakos | A theory of refractive and specular 3d shape by light-path triangulation[END_REF]. Unlike them however, even when light is only reflected off a transparent object surface, equation (3.12) can be used to solve for relative refractive index n

Figure 3 . 3 :

 33 Figure 3.3: Figure representing the simplest double bounce case. Two cameras look at a rectangular glass slab at the CRT monitor placed at two positions, producing overall 4 images. Note that in 3D the intersection point N k might not actually exist for all values of d k .

77 )

 77 Observe that the R.H.S terms (equations(3.70, 3.72, 3.73)) all have odd powers of c 1 and the L.H.S terms (equations (3.74, 3.76, 3.77)) all have even powers of c 1 . Thus squaring each side would produce terms that are all even powers of c 1 . Also, observe that the term A is of degree 4 in s 1 , B is of degree 3, C of 8, D of 2 and E of 7. So both the R. H. S and L. H. S in the equation (3.68) are of degree 24.Thus we get a 24 degree equation in s 1 and degree 12 in s2 1 . Thus we get 12 solutions for s 1 since we only consider positive roots.

Figure 3 .

 3 Figure 3.4 simulates the value of radiance ratio for the space spanned by the two incident angles

Figure 3

 3 Figure 3.5.

Figure 3 . 5 :

 35 Figure 3.5: Illustration depicting various elements of the illumination model. Two camera pixels are back-projected, while a pin-hole model (used for geometric calibration) assumes a back-projected ray, the illumination model assumes a back-projected cylinder. This cylinder strikes pixel j on the monitor and captures H( j) percent of its illumination in the direction θ .

Figure 3 .Figure 3 . 6 :

 336 Figure 3.6 shows the variation of ρ with θ for 6 different pixels on the monitor. Notice how the function has a quadratic fall-off with angle. We fit a quadratic curve to this function and use its parameters in our depth estimation algorithm.

Figure 3 . 7 :

 37 Figure 3.7: Images of datasets included in this supplementary material, and some images of their acquisition setup

Figure 3 .

 3 Figure 3.10: (Left Column) Two of 25 images used to compute the direct and global images [99] to remove the effect of interreflections and caustics on radiance measurement. (Middle Left Column) Direct and Global (scaled) components. (Middle Right Column) Difference between the "direct" component and an image taken with a white pattern shown on the monitor. Bottom image is the difference. (Right Column) Correspondence map, Depth map and Reconstructed mesh of "Water" sequence.

Figure 3 . 11 :

 311 Figure 3.11: (Left top) Normal map of "Fanta bottle" sequence. (insets) Note the fine details captured as a result of radiance ratios. (Centre top) Depth map. Blacker colors are closer to camera. (Right top) Two views of the 3D reconstruction, with lighting to highlight shape variations. Phenomenon like scratches on the bottle, inhomegenous refractive index, violation of single bounce through occlusion are some bad effects, but still reliable reconstructions are achieved. Note that since camera is placed far from the object and monitor, large changes in depth cause small changes in angle. This explains some of the "rough"-ness of the reconstruction. Note also that no smoothing or. optimization is applied for this reconstruction. (Left bottom) Normal map of "Wine glass" sequence. (Left-centre bottom) Depth map. (Right-centre bottom) Depth map produced if no interreflection removal is performed. Notice the lack of depth variation in one of the glasses (compared using insets with blue borders). Some frequency artifacts can be seen (red inset) due to interreflection removal. (Right bottom) Depth difference between the two cases. Best viewed in color. 71

Figure 3 . 12 :Figure 3 . 11 .

 312311 Figure 3.12: Normal map and Depth maps for the cokebottle and fanta sequences.

  through multiple refractive planes (e.g. underwater imaging), which results in distortions and gives the illusion of the scene being closer and magnified. While 3D reconstruction in such scenarios has been analyzed in multi-media photogrammetry [62; 114; 125], such imaging setups have been relatively unaddressed in the computer vision community until recently. Calibrating such a system with multiple layers with unknown layer orientation, distances and refractive indices remains an open and challenging problem.

4. 6 . 4 107 4. 6 . 6 107 4. 7 108 4. 7 . 1 108 4. 7 . 2 109 4. 8

 6410766107710871108721098 Single Bounce SRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 4.6.5 Single SRC With Known Pose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Solving Planar Reflection With Known Pose . . . . . . . . . . . . . . . . . . . . . . . . Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Real Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Figure 4 . 1 :

 41 Figure 4.1: (Top Left) Flat and Spherical reflective geometry with 1 layer. (Top Middle) The entire light-path for each pixel lies on a plane and all planes intersect in a common axis passing through the camera center. (Top Right) Once this axis is computed, analysis can be done on the plane of reflection to estimate plane distance and centre / radius of sphere. (Bottom Left) Flat refractive geometry with n layers. (Bottom Middle) The entire light-path for each pixel lies on a plane and all planes intersect in a common axis passing through the camera center. (Bottom Right) After computing the axis, analysis can be done on the plane of refraction to estimate layer thickness and refractive indices.

[

Figure 4 . 2 :

 42 Figure 4.2: Figure showing two light-paths for the plane reflective, the spherical reflective and the flat refractive case. The red and blue light-paths bounce off different surfaces and different numbers of times. However, they can all be clubbed into the same POR constraint to estimate R and t A ⊥ .

6 -e 3 e 5 )

 65 the e i 's are estimated as above and x, y, z are unknown. Setting det(E) = 0 gives a linear equation in x, y, z which can be written as x e 2 e 6x e 3 e 5y e 1 e 6 + y e 3 e 4 + ze 1 e 5ze 2 e 4 = 0 (4.6) using which x can be obtained in terms of y and z as x = ((e 1 e 6e 3 e 4 ) y + (e 2 e 4e 1 e 5 )z)/(e 2 e

Figure 4 . 3 :

 43 Figure 4.3: Comparison of 11pt algorithm, 8pt algorithm and using all points in a least squares fashion for estimation of axis, rotation and t A ⊥ using POR constraints.

. 66 )

 66 Finally, for N bounces the PRC equation can be solved to obtain the perpendicular distance between every alternate pair of mirrors (1 & 2, 3 & 4, . . ., where mirror 1 records the first bounce of a backprojected ray from the camera). For the other pairs, we could use a generalized version of the above equation. Thus, we need K >= N /2 correspondences to get the required number of independent equations to solve in the linear least squares sense.

Figure 4 . 2 illustrates

 42 one example of such a scenario.

3 . 1 )Figure 4 . 6 :

 3146 Figure 4.6: Figure depicting the case of the k + 1 th bounce off a sphere. Considering the values of r k+1 , s k , l k are known, the point p k+1 is still one of two points that can only be obtained by solving a quadratic equation (line-circle intersection). Note that n k+1 and eventually l k+1 depend directly on p k and thus indirectly on n and unknowns r k+1 , d 0...k+1

. 70 )

 70 p k+1 = p k + β k+1 l k (4.71)

Figure 4 . 7 :

 47 Figure 4.7: Three cases of refraction considered in this chapter. In each case we consider 4 variations: known/unknown refractive indices with known pose parameters, and known/unknown refractive indices with unknown pose parameters

D 1 = µ 2 1 + (v y ) 2

 112 y (d 0 -α + d 1u y ) (4.103) k 2 = u x v yd 0 v x (4.104) k 3 = -d 1 v x v y (4.105)

(4. 109 ) 1 = d 0 c 0 -d 2 0 c 2 0 -d 2 0 + r 2 1 =q 1 = β 1 vp 0 ( 4 . 111 )n 1 = (β 1 vp 0 + d 0 n)/r 2 1 ( 4 . 112 )

 1091221104111114112 Re-writing equations 4.74-4.78 using the 2D convention presented earlier, we getβ d 0 c 0 -D (4.110) vp 1 = 2(β 1d 0 c 0 )n 1 /r 2 1vp 0 (4.113) = 2(-D)(β 1 vp 0 + d 0 n 0 )/r2 1vp 0 (from equation 4.110) (4.114) = 2(-Dβ 1 vp 0d 0 Dn)/r 2 1expanding for vp 0 and n. Expanding the second term, we getu + αz p -q 1 =   u x -β 1 v x u y + α -β 1 v y  (4.117)Finally, substituting in the SRC equation we get(-β 1 D -1/2r 2 1 )(v x u y + v x αv y u x ) -Dd 0 (u x -β 1 v y ) = 0 (4.118)

Figure 4 . 9 :

 49 Figure 4.9: Rotation, translation and reprojection error using our algorithm versus using a central approximation (CA) for Case 1 and Case 2. The right most plot shows the estimated t z for Case 2 over all 100 trials for σ = 1 pixel. CA estimates the object to be closer to the camera than in reality.

Figures 4 .

 4 Figures 4.8 shows error plots for pose and calibration parameters and the final reprojection error for different noise levels, averaged over 100 trials. These plots show that correct calibration and pose

Figure 4 . 10 :

 410 Figure 4.10: (Left) Setup. (Middle) Photo captured by looking through a water tank. Projected 3D points are overlayed by applying pose estimated using CA (green) and our algorithm (red).(Right) Reconstructed 3D points.

Figure 4 . 10 (

 410 Figure 4.10(Right). We detect corners in the captured photo and run our algorithm (Case 2) to estimate the calibration and pose parameters. The estimated thickness of the tank using our algorithm was 255.69mm, resulting in a relative error of 1.66%. Notice the large error in t z in the central approximation (Table4.2), also evident from projected points in Figure4.10. Interestingly, the central

) 1 |g 1 π ( 12 ) 1 π ( 13 )

 1112113 ′ (ψ)| = λ sec 2 (ψ) |λ 2 -1 + sec 2 (ψ)| (10) = λ |(λ 2 -1) cos 2 (ψ) + 1|(11)This Finally, this gives ρ(ψ) in the case of both reflection and transmission, asρ(ψ) l = R s R p |(R p -R s ) cos 2 (ψ) + R s | ρ(ψ) r = T s T p |(T p -T s ) cos 2 (ψ) + T s |

1 R s 1 1 R p cos 2 (ψ) 1 R 1 R s sin 2 (ψ) 1 R p cos 2

 1111112 R p tan(ψ), we can modify the above equation by notingcos 2 (φ) = p cos 2 (ψ) + 1 R s sin 2 (ψ) (ψ) + 1 R s sin 2 (ψ)

  1 Case 1: Single Bounce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.4.2 Case 2: Double Bounce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

3.5 Methods of Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.5.1 Single Reflection or Refraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.5.2 Double Bounce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.5.3 Practical considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 3.5.4 Estimating CRT Illumination Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.6 Experiments and Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 3.6.1 Potential Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.7 Discussion and Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Image-based reconstruction of transparent objects has gathered interest in the last few years

[25; 64; 73; 88; 89; 91; 93]

. Several cues and approaches have been proposed for this task. They could be broadly classified as approaches that rely on physical (material) properties of transparent objects,

  I ll r...r Intensity after k bounces of reflections and/or refractions. ρ(φ) Polarization angle probability.

	3 I l r l	Image intensity after 3 bounces. (reflection, refractin, re-
		flection).
		Single Bounce Notations
	π	Plane of refraction.
	θ 1	Angle of incidence.
	θ 2	Angle of reflection/refraction, depending on case.
	E	Electric field vector accompanying light ray.
	E	Electric field magnitude accompanying light ray. E
	E s , E p Magnitude of components of E perpendicular and parallel
		to plane.
	φ	Angle made by electric field with π.
	R s , T s Reflection/Transmission coefficient for E along the vector
		π.
	R p , T

k p Reflection/Transmission coefficient for E perpendicular to π. Note that E s = E cos(φ), E p = E sin(φ)

Table 3 . 1 :

 31 Single Bounce Notations

		Double Bounce Notations (First Bounce)
	1 π	Plane of refraction.
	1 θ 1 1 θ 2	Angle of incidence. Angle of reflection/refraction, depending on case.

1

E

Electric field vector accompanying light ray. 1 E Electric field magnitude accompanying light ray. 1 φ Angle made by electric field with π 1 . 1 R s , 1 T s Reflection/Transmission coefficient for E 1 along vector π 1 1 R p , 1 T p Reflection/Transmission coefficient for E 1 perpendicular to π 1 Double Bounce Notations (Second Bounce) 2 π Plane of refraction. 2 θ 1 Angle of incidence. 2 θ 2 Angle of reflection/refraction, depending on case. 2 E Electric field vector accompanying light ray. 2 E Electric field magnitude accompanying light ray. 2 φ Angle made by electric field with π 2 . 2 R s , 2 T s Reflection/Transmission coefficient for E 2 along vector π 2 2 R p , 2 T p Reflection/Transmission coefficient for E 2 perpendicular to π 2 Table 3.2: Double Bounce Notations

  • . Second, θ 1 is typically (much) larger than 30 • , due to the practical setup which requires that the camera have both a reflected and a direct view of the monitor. Consider the graph of r as a function of θ 1 for the refractive index of water (n 2 = 1.33), in figure (ref figure here) (here, both the camera and the monitor, are in air, i.e. n 1 = 1). One observes that for the values of r associated with θ 1 ∈ (30 • , 90 • ), there is a unique θ 1 producing these values. To be precise, θ 1 is unique up to sign, but only the positive solution corresponds to a depth/point in front of the camera.

  Radiance ratio values plotted for various pairs of incident angles. There is no appreciable difference because of a non-zero angle between the two planes of refraction in the double bounce case.

			Double Reflection Solutions (α = 0 • )					Double Reflection Solutions (α = 60 • )
				(Ref Index n1 = 1.0, n2 = 1.33)							(Ref Index n1 = 1.0, n2 = 1.33)
								Ref Ceoffs										Ref Ceoffs
									0.01											0.01
									0.005											0.005
									0.002											0.002
									0.0012											0.0012
	Radiance Ratio	0 0.05 0.1 0.15 0.2 0.25 0.3 0.35							0.0007 0.0005		0 0.05 0.1 0.15 0.2 0.25 0.3 0.35	Radiance Ratio	0 0.05 0.1 0.15 0.2 0.25 0.35 0.3							0.0007 0.0005	0 0.05 0.1 0.15 0.2 0.25 0.3 0.35
	0 0.2 0.4 0.6 0.8 1 1.2 1.4 (Radians) θ1	0.8	0.7	0.6	0.5	0.4	0.3	0.2	0.1	0	0 0.2 0.6 1 1.2 1.4 0.8 (Radians) θ1 0.4	0.8	0.7	0.6	0.5	0.4	0.3	0.2	0.1	0
							θ2											θ2
						(Radians)									(Radians)
						(a)											(b)
	Figure 3.4:

  + x 2 )e 1 K + e 2 (e 1 e 2 + e 4 e 5 + x y) + e 3 (e 1 e 3 + e 4 e 6 + x y) = 0 (4.12) e 4 (e2 1 + e 2 4 + x 2 )e 4 K + e 5 (e 1 e 2 + e 4 e 5 + x y) + e 6 (e 1 e 3 + e 4 e 6 + x y) = 0 (4.13) e 2 (e 2 2 + e 2 5 + y 2 )e 5 K + e 1 (e 1 e 2 + e 4 e 5 + x y) + e 3 (e 2 e 3 + e 5 e 6 + yz) = 0 (4.14) e 4 (e 2 2 + e 2 5 + y 2 )e 5 K + e 4 (e 1 e 2 + e 4 e 5 + x y) + e 6 (e 2 e 3 + e 5 e 6 + yz) = 0 (4.15) e 4 (e 2 3 + e 2 6 + z 2 )e 6 K + e 1 (e 1 e 3 + e 4 e 6 + xz) + e 2 (e 2 e 3 + e 5 e 6 + yz) = 0 (4.16) e 4 (e 2 3 + e 2 6 + z 2 )e 6 K + e 4 (e 1 e 3 + e 4 e 6 + xz) + e 5 (e 2 e 3 + e 5 e 6 + yz) = 0 (4.17) Note that the first three equations have cubic terms of x, y, z while the next six equations have quadratic terms. We can choose any two of these six quadratic equations. Let us choose the first two of the six quadratic equations and denote them as EQ 2 and EQ 3 . Substituting x equation (4.7) we get two equations of the following form EQ 2 : k 11 y 2 + k 12 yz + k 13 z 2 + k 14 = 0, (4.18) EQ 3 : k 21 y 2 + k 22 yz + k 23 z 2 + k 24 = 0 (4.19) where the k i j depend on the e i 's and are known coefficients. We can eliminate y 2 from the above two equations to get y in terms of z y = k 21 (k 13 z 2 + k 14 )k 11 (k 23 z 2 + k 24 ) k 11 k 22 zk 12 k 21 z (4.20) Substituting y back into EQ 3 gives a fourth degree equation in z g 1 z 4 + g 2 z 2 + g 3 = 0 (4.21) where g 1 = k 11 (k 2 11 k 2 23k 11 k 12 k 22 k 23 -2k 11 k 13 k 21 k 23 + k 11 k 13 k 2 22 + k 2 12 k 21 k 23k 12 k 13 k 21 k 22 + k 2 = k 11 (k 11 k 14 k 2 22 + 2k 13 k 14 k 2 21 + k 2 12 k 21 k 24 + 2k 2 11 k 23 k 24k 11 k 12 k 22 k 24 -2k 11 k 13 k 21 k 24 -2k 11 k 14 k 21 k 23k 12 k 14 k 21 k 22 ) (4.23) g 3 = k 11 (k 11 k 24k 14 k 21 ) 2 (4.24)

	The Demazure constraints arise from the form of E (= [.] × R) [29], and give the following nine equations
	x(2e 2 1 + 2e 2 4 + 2x 2 ) -x K + y(2e 1 e 2 + 2e 4 e 5 + 2x y)
	+z(2e 1 e 3 + 2e 4 e 6 + 2xz) = 0	(4.9)
	y(2e 2 2 + 2e 2 5 + 2 y 2 ) -y K + x(2e 1 e 2 + 2e 4 e 5 + 2x y)
	+z(2e 2 e 3 + 2e 5 e 6 + 2 yz) = 0 6 + 2z 2 ) -zK + x(2e 1 e 3 + 2e 4 e 6 + 2xz) 3 + 2e 2 z(2e 2 + y(2e 2 e 3 + 2e 6 e 6 + 2 yz) = 0 21 ) 4 13 k 2 1 + e 2 e 1 (e 2 g 2	(4.10) (4.11) (4.22)
	2 2 + e 2 3 + e 2 4 + e 2 5 + e 2 6 + x 2 + y 2 + z 2	(4.8)
	83

  w.r.t θ δ and equate it to 0.

							∂ ∂ θ δ	cos(∠(l × â, l × âδ )) = 0	(4.39)
	⇒	cos(θ ) sin(θ δ ) sin(θ ) sin(θ δ )	-	(cos(δ) -cos(θ ) cos(θ δ )) sin(θ ) cos(θ δ ) sin 2 (θ ) sin 2 (θ δ )	= 0	(4.40)
		⇒ Angular Error Between Cross Products	⇒ cos(θ ) sin(θ ) -cos(θ ) sin(θ ) (cos(δ) cos(θ δ ) -cos(θ ) + cos(θ ) sin 2 (θ δ )) -Emperical Verification of Approximation (cos(δ) cos(θ δ ) -cos(θ ) cos 2 (θ δ )) sin(θ ) sin 2 (θ δ ) sin(θ ) sin 2 (θ δ ) ⇒ cos(θ ) sin(θ ) -(cos(δ) cos(θ δ ) -cos(θ )) sin(θ ) sin 2 (θ δ ) -cos(θ ) sin(θ ) ⇒ cos(δ) cos(θ δ ) = cos(θ ) ⇒ cos(θ δ ) = cos(θ ) cos(δ) 11.46 • 22.92 • 34.38 • 45.84 • 57.30 • 68.75 • 80.21 • y = s i n -1 (	= 0 = 0 = 0	(4.41) (4.42) (4.43) (4.44) (4.45)
	0.57 • 6.25 • 11.80 • 17.19 • Substituting this value back in equation 4.36 we get the maximum value of the sine of this angle as 24.69 • 31.34 • 35.35 • 42.28 • Angular Error Between Vectors â and âδ (δ )
		min θ δ	cos(∠(l × â, l × âδ )) =	cos 2 (δ) -cos 2 (θ ) sin(θ )	(4.46)
							=	sin 2 (θ ) -sin 2 (δ) sin(θ )	(4.47)
		max θ δ	sin(∠(l × â, l × âδ ) =	1 -	(sin 2 (θ ) -sin 2 (δ)) sin 2 (θ )	(4.48)
							=	sin(δ) sin(θ )	(4.49)
							≈	δ sin(θ )	(4.50)

δ s in ( θ )

  EQ 3 : k 21 α 2 (k 22 d 2 0 + k 23 d 0 + k 24 ) + k 25 α(k 26 d 3 0 + k 27 d 2 0 + k 28 d 0 + k 29 ) + (k 41 d 3 0 + k 42 d 2 0 + k 43 d 0 + k 44 ) = 0 (4.90)

	3 0 + k 17 d 2 0 + k 18 d 0 + k 19 ) +
	(k 31 d 3 0 + k 32 d 2 0 + k 33 d 0 + k 34 ) = 0	(4.89)

2 (k 12 d 2 0 + k 13 d 0 + k 14 ) + k 15 α(k 16 d

Table 4 . 2 :

 42 ).(Right) Reconstructed 3D points. N θ x , θ y , θ z (deg) t x , t y , t z (mm) d 1 (mm) GT 131.38, 1.22, 84.07 -237.58, -128.85, 455.80 260 CA 144 130.24, 1.42, 83.84 -217.71, -120.73, 372.14 -Ours 1,2 144 131.38, 1.26, 84.12 -237.11, -128.16, 453.12 255.69 Ours 1,3 48 131.40, 1.36, 84.03 -239.76, -129.26, 456.34 272.81 Ours 4 144 131.37, 1.26, 84.12 -236.46, -127.86, 449.70 262.39 Estimates of pose and water-tank thickness d 1 for real data shown in Figure4.10 using central approximation (CA) and our algorithm. GT denotes ground truth and N denotes number of 2D-3D correspondences. Conditions: 1 Assuming µ 1 = 1.33.2 All Planes.3 Left Plane Only.4 All Planes, unknown µ 1 .

represents the epipolar quartic curve in the second image.

Command :dcraw -r 1 1 1 1 -D 0 -H 1 -q 3 -4 -v for Canon cameras. We drop the -D option for Nikon cameras.

Abstract

The task of understanding, 3D reconstruction and analysis of the multiple view geometry related to transparent objects is one of the long standing challenging problems in computer vision. In this thesis, we look at novel approaches to analyze images of transparent surfaces to deduce their geometric and photometric properties.

At first, we analyze the multiview geometry of the simple case of planar refraction. We show how the image of a 3D line is a quartic curve in an image, and thus derive the first imaging model that accounts for planar refraction. We use this approach to then derive other properties that involve multiple cameras, like fundamental and homography matrices. Finally, we propose approaches to estimate the refractive surface parameters and camera poses, given images.

We then extend our approach to derive algorithms for recovering the geometry of multiple planar refractive surfaces from a single image. We propose a simple technique to compute the normal of such surfaces given in various scenarios, by equating our setup to an axial camera. We then show that the same model could be used to reconstruct reflective surfaces using a piecewise planar assumption.

We show encouraging 3D reconstruction results, and analyse the accuracy of results obtained using this approach.

We put the terms including D on one side of the equation and the rest on the other:

After squaring both sides of the equations and bringing them back together, we get:

Finally, rearranging the terms with respect to s 1 , one obtains the following constraint in θ 1

and s 1 = sin θ 1 . associated with it. Since there are two depth values associated with every ray, and no correspondence information is available between cameras, we need two views to achieve minimal requirements for a solution. We also need two monitor positions because we need to know the final refracted ray's position and direction.

In Figure 3.3, consider a ray with direction i 1 1 from camera C 1 . This ray intersects the slab twice before ending up on monitor pixel X 1 , with final refracted ray y 1 1 . Because of the presence of parallel slabs, we have i 1 1 = y 1 1 . This light-path also contains two intersection points with the rectangular slab, 1 T 1 1 and 2 T 1 1 . In order to reconstruct this light-path, we hypothesize the positions of these intersection points. We then verify the hypothesis using camera C 2 . The verification is done in two stages. In the first stage we verify the validity of the light-path produced in C 2 because of the normal and depth values obtained using the first light-path. In the second stage, we verify whether the light-paths satisfy their photometric constraints (using Fresnel equations). Note that although there is a 2D set of values for

, not all values satisfy the Fresnel equations for the first light-path itself.

We model the different light-path hypotheses using two parameters

the normal for the first bounce, n k . Note that using an angle formulation is possible in this case since we know the plane of refraction Π 1 , as it contains C 1 , X 1 and i 1 1 . Also, generating normal hypotheses in this plane ensures that the point 2 T 1 1 exists. The remaining parameters used in our derivation are expressed in Figure 3.3.

Firstly, we look at the geometric constraint. This involves identifying the conditions for the refracted ray from C 2 to intersect its corresponding ray from X k along i 2 k . To do this, we need to derive the values of n k , d k and the direction of the middle segment of both light-paths. We define n k and the middle segment m 1 k as

Finally, we get the angle bounds as

To summarize, we have bounded the angle between the vectors l× â and l× âδ where δ is the angular error in computation of the axis and consequently, the angle between these two vectors represents the angle between the ground truth normal of the POR and the estimated normal. Figure 4.4 shows simulation results that confirm our theory. Note that the above derivation is useful in another sense.

When the checkerboard pose is known in the camera coordinate system, a closed form solution to the problem of normal estimation in the case of planar reflection or refraction can be obtained using cross products. In such a case, the error in the estimated normal is bounded in the same way as mentioned here. We know look at several individual cases of reflection/refraction to solve for the remaining light-path parameters.

General Flat Refractive Geometry

Consider the general setup for flat refractive geometry as shown in Figure 4.5, where a perspective camera observes a known calibration object via N flat refraction layers. We work in the camera coordinate system with the camera center at the origin. Let n denote the common surface normal to all layers and [d i , µ i ] be the thickness and refractive index of the i th medium respectively. d 0 represents the distance between the camera and the first layer. Let [X i ] K i=1 denote K 3D points on the object which are known in the object coordinate system and let [R, t] be the unknown rigid transformation of these points.

Flat Refraction Constraint

Let [l i 0 , l i 1 , . . . , l i n ] denote the direction vectors of each segment of the corresponding light-path. (For simplicity, we drop the superscript i for now) We assume that the internal camera calibration has been done offline and hence we know the camera ray l i 0 for each 3D point X i . The last refracted ray direction l n should coincide with the line joining the transformed 3D point RX + t and the refraction point p n on the last layer. Thus, the following Flat Refraction Constraint (FRC) should be satisfied. and the rest of the unknowns are given as

)

Note that after computing the axes parameters as described earlier, the unknowns in the above equation are d 0 and r 1 .

Summary

We have shown 3 possible scenarios in the previous section where planar axial lightpaths could be solved simultaneously, and the framework in which we will solve them. Our approach essentially consists of two steps. In the first step, we use the axial nature of this configuration in order to derive most of the pose parameters relating the observed 3D points to the camera coordinate system. In the second step, we then establish the remaining parameters of the pose and light-path configuration by noting that the vector in the direction from the last bounce on the light-path towards the corresponding 3D point should be the same as the direction of the last segment of the light-path (equations FRC, PRC and SRC). Notice that while PRC is readily solvable for any number of bounces, solutions of FRC and SRC are restricted by the knowledge of refractive indices (FRC) and the sphere location, radius and general inter-connectedness of the light-path parameters (SRC). It is worthy noting here that the sphere is the simplest case of a quadric surface, and any other surface like a paraboloid or a higher order surface would only increase the amount of unknowns and render SRC even more complicated and less solvable.

Solving for Light-Paths

In the previous section, we set the framework for recovering the light-path configuration from individual observations. In this section, we solve for the remaining unknown parameters using FRC and SRC. Note that PRC has already been solved (equations 4.65,4.66). We first consider solving FRC presented here.

Single SRC With Known Pose

Notice that in equation 4.118 there is only one term α that is related to the relative position between camera and 3D correspondences. Also, note that it is easy to isolate terms with r since mostly they are only contained in the term D. We used 2 correspondences to solve for r in terms of d, which results in four 4th degree equations involving only the sphere center. Thus we get 16 possible sphere centres, and for each solution of d, 4 values for r for a total of 64 solutions. Disambiguating could be done by observing some general rules like d > 0, r > 0, d > r and using a 3rd or more correspondences.

Solving Planar Reflection With Known Pose

Similar to the spherical case, we can also solve the planar reflection scenario in the presence of known relative poses between camera and 3D correspondences. In such a case, the POR constraint has only 1 unknown in the plane normal, and hence can be solved using 2 correspondences. As for the PRC, there is a reduction of only one variable α, and so the solution is more or less the same as described earlier.

Summary

We have shown that, for planar light-paths, it is possible to exploit the axial property to compute the relative pose of 3D correspondences and the optical system and hence reconstruct the light-paths in various scenarios. We divide the problem into two parts. In the first part, we find the axis direction and part of the pose parameters using the POR constraints. In the second part, for individual cases we use the refraction or reflection constraints to compute the remaining pose and light-path parameters. In certain cases, we also showed that reconstruction was not possible. Finally, we showed that when the relative pose is known, we can reconstruct single planar and spherical reflection using 2 correspondences.

Results

In this section, we present results for various simulations that confirm our theories, as well as real results and comparisons for flat refraction.

Simulations

Now we present simulations for the complete calibration and pose estimation process, using the earlier settings. The 8pt algorithm is used since it works better. In the RANSAC framework, after

Discussion

We have analyzed the geometry of a perspective camera imaging through multiple flat refractive layers. We developed a theory for calibration and extended it to the case of planar and spherical reflections, which can be directly used in applications such as 3D reconstruction [START_REF] Ben | What does motion reveal about transparency[END_REF]. We presented a comprehensive analysis under unknown layer distances and orientation, and known/unknown refractive indices. Since calibration can be done using a single planar grid, the proposed algorithms are useful in practical scenarios such as underwater imaging. Multiple planar grids can be used to increase the calibration accuracy similar to calibration of perspective cameras.

where k

Coordinate transformation After a light ray passes through the first bounce, the angle its electric field makes with the 1 π is ψ. However, when it strikes the second surface in plane of reflection/refraction 2 π, the angle made with 2 π is no longer ψ. Since 1 π and 2 π are normal vectors perpendicular to the middle segment of the light path, they lie in the plane that contains the electric field 2 E. Thus, this ray makes an angle ψ -α with 2 π, where α is the angle between 1 π and 2 π.