Boris Bukh

John Canny

Yasutaka Furukawa

Xavier Goaoc

Alfredo Hubard

Kathlén Kohn

Brian Osserman

Bernd I Sturmfels

Rekha Thomas

Max Lieblich

Sameer Agarwal

Keywords: Geometric models, projective geometry, cameras, silhouettes, image contours

This thesis studies mathematical models for describing the geometry of imaging processes in computer vision. In a broad sense, our contributions are focused on developing frameworks that are very general and at the same time require minimal assumptions. Our approach is in fact rooted in the language of projective geometry, which provides the most general setting for studying properties of lines and incidences that are at the heart of geometric vision. We also apply some tools from algebraic geometry, since many of the objects that we encounter are described by polynomial equations. For example, the multi-view geometry of n pinhole cameras (as well as other cameras) can be encoded in the "joint image", that is an algebraic variety in (P 2 ) n formed by all point correspondences. The Grassmannian of lines Gr(1, P 3 ) also plays a central role in our study. In particular, surfaces in the Grassmannian (or "line congruences") can be used to represent abstract cameras, that are mappings from points to viewing lines. This description is also convenient for studying in a unified manner the multi-view geometry of families of general imaging systems. In addition to modeling cameras, we also investigate the relationship between 3D shapes and their images. For arbitrary sets projecting onto opaque silhouettes, the image is determined by the set of viewing lines that meet the observed object. This leads to the study of "projective visual hulls" and "multi-view consistency". For smooth surfaces, the "image contour" is determined by the set of viewing lines that are tangent to the surface. From this perspective, the evolution of the contour from a moving viewpoint, and the associated "visual events", can be described by studying sets of lines in Gr(1, P 3 ) that have special incidence properties with the surface. I first wish to thank my advisors Jean Ponce and Martial Hebert. They introduced me to the fascinating world of geometry and vision, and provided invaluable guidance. In particular, I

Introduction

In order to develop algorithms for artificial visual systems, computer vision relies on models for representing and manipulating empirical data. In this sense, a model is an abstraction that enables one to study a certain class of physical processes using formal tools. Multi-view reconstruction, for example, can be framed as an inference task, where the set of admissible camera configurations is associated with points in some parameter space, and observations (multi-view correspondences) are used to identify an element within this parameterized set. Models are of course necessary for a theoretical investigation of empirical systems: simplifying the physical reality is a prerequisite for developing theories that can be applied in different contexts. This thesis is devoted to models for the geometry of vision. The mathematical study of the visual world dates back to ancient times, but the theoretical foundations of computational algorithms in 3D vision were developed mainly in the 80s and 90s. The theory, known as multi-view geometry [START_REF] Faugeras | The geometry of multiple images: the laws that govern the formation of multiple images of a scene and some of their applications[END_REF][START_REF] Hartley | Multiple view geometry in computer vision[END_REF], has been essential in the development of many remarkable applications, from 3D reconstruction software to today's technologies for augmented and virtual reality. Our goal in this thesis is to take the classical theory of multi-view geometry further, by developing more general models for cameras and 3D shapes, and by studying these models using tools from projective and algebraic geometry [START_REF] Harris | Algebraic geometry: a first course[END_REF].

The importance of projective geometry as a framework for modeling visual phenomena has been long known. Projective transformations and ambiguities arise naturally in reconstruction tasks, and the use of homogeneous coordinates simplifies many geometric computations (in particular, it linearizes the pinhole camera model). For these reasons, multi-view geometry is usually based on the perspective that affine and euclidean geometry are special cases of projective geometry. This idea dates back to Felix Klein's Erlangen program, and is convenient for dealing with a hierarchy of geometric models in a unified setting. In this presentation, however, we will usually disregard affine and euclidean structures, and focus for the most part on actual projective geometry. This is because many basic concepts in vision only require simple contact properties (collinearity, incidence or tangency); hence, a purely projective perspective is simpler and at the same time more general. We will also point out how some models that have traditionally been studied in a euclidean setting (e.g., visual hulls [START_REF] Guenther | Geometric modeling for computer vision[END_REF] and visual events [START_REF] Koenderink | The singularities of the visual mapping[END_REF]), can actually be defined more naturally using only projective geometry.

Our use of tools from algebraic geometry is due to the fact that many of our models will be based on polynomial representations. For example, it is well known that multi-view point correspondences are characterized by algebraic constraints in image coordinates [START_REF] Faugeras | On the geometry and algebra of the point and line correspondences between n images[END_REF], and it is actually convenient to view these conditions as the defining equations of an an algebraic variety (known as the joint image [START_REF] Triggs | Matching constraints and the joint image[END_REF], or the multi-view variety [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF]). An algebraic framework is also useful for modeling non-central imaging systems [START_REF] Sturm | Camera models and fundamental concepts used in geometric computer vision[END_REF]. We will in fact represent general cameras either geometrically as algebraic surfaces in line space (corresponding to families of viewing rays), or analytically as rational (polynomial) mappings P 3 P 2 . Furthermore, we will study projections of algebraic surfaces, describing the visual events that occur in the image contour curve as the viewpoint changes.

As mentioned above, the topics addressed in this thesis are intended to broaden the scope of the classical theory of geometric vision. For example, while traditional multi-view geometry is largely focused on systems of 2, 3, 4 cameras and the associated multifocal tensors, we believe that it is important (and interesting) to study configurations involving any number of cameras, investigating concepts such as the "joint image" or "viewing graphs". Moreover, other topics, such as non-central imaging systems or the geometry of shape reconstruction, have arguably not been studied much in a general and systematic fashion, but deserve to be part of a mathematical theory of vision. We will also often insist on the importance of line geometry [START_REF] Pottmann | Computational line geometry[END_REF] for describing visual processes. Essentially all geometric aspects in vision are in fact determined by how lines (i.e., light rays) intersect objects in space. Although the role of viewing lines is usually hidden in practical structure-from-motion algorithms by the use of image coordinates [START_REF] Ozyesil | A survey of structure from motion[END_REF], in many situations it can be convenient to make the geometry explicit. For the study of non-central imaging systems, in particular, researchers have proposed a geometric representation of cameras as mappings from points to viewing rays [START_REF] Ponce | What is a camera?[END_REF][START_REF] Pajdla | Stereo with oblique cameras[END_REF]. This idea will be developed further in the thesis, and we will investigate in detail the geometry of non-linear cameras inside the Grassmannian of lines.

The mathematical models studied in this thesis are quite general, but are still meant to be close to practical applications. Our presentation avoids unnecessary technicalities, and only parts of our discussion will require some background in algebraic geometry. On the other hand, we believe that a minimal amount of rigorous language can be useful, since it allows us to use more powerful tools, and because it helps us identify more clearly what is known and the problems that still need to be addressed. We also hope that precise terminology can facilitate interactions between researchers across different fields. There has been in fact a recent interest in geometric vision from mathematicians, with several journal papers (including some of the works presented in this thesis) and a few workshops on the topic of algebraic vision. We hope that this thesis can help bridge the gap between theoretical and practical approaches to vision.

Introduction

Pinhole-based imaging systems have a very long history. The optical phenomenon of the camera obscura was accurately described by the 11th-century Arab physicist Ibn al-Haytham (965-1039 CE) and was mentioned already in writings by Aristotle (384-322 BCE) [START_REF] Josef | History of Photography[END_REF]. Throughout the Renaissance and until the 18th century, devices that Pinhole Cameras and their Multi-View Geometry formed images by gathering light through a small aperture were used as drawing aids or for entertainment. The photographic camera, developed in the early 19th century, is an adaptation of the box-type camera obscura that was popular at that time. Today's digital cameras use complicated optics to gather and focus light, but can still be roughly modeled using pinhole geometry.

In mathematics, projective geometry emerged from the study of perspectivity, initiated by architects Filippo Brunelleschi and Leon Battista Alberti around 1425. The theory flourished in the 17th, 18th, and 19th centuries, with fundamental contributions from Desargues, Poncelet, and Klein, among many others. Problems in projective reconstruction were already studied by these classical geometers. For example, epipolar geometry was known to Hauck in 1883, and image-based 3D reconstruction was described by Finsterwalder in 1889 [START_REF] Sturm | A historical survey of geometric computer vision[END_REF]. These results were largely re-developed by computer vision scientists in the eighties and nineties (even though they were known to photogrammeters [START_REF]Manual of Photogrammetry[END_REF]). Today, the part of computer vision that studies the theoretical foundations 3D reconstruction algorithms is known as multi-view geometry. General overviews on this subject can be found in the textbooks by Faugeras, Luong and Papadopoulo [START_REF] Faugeras | The geometry of multiple images: the laws that govern the formation of multiple images of a scene and some of their applications[END_REF], and by Hartley and Zisserman [START_REF] Hartley | Multiple view geometry in computer vision[END_REF], or in the older book by Maybank [START_REF] Maybank | Theory of reconstruction from image motion[END_REF].

Despite this long history, many aspects of the theory of pinhole cameras are not fully settled. The most well-known results in multi-view geometry are concerned with multifocal tensors for 2, 3, or 4 views [START_REF] Luong | The Fundamental matrix: theory, algorithms, and stability analysis[END_REF][START_REF] Shashua | Algebraic functions for recognition[END_REF][START_REF] Richard | Lines and points in three views and the trifocal tensor[END_REF][START_REF] Hartley | Computation of the quadrifocal tensor[END_REF]. Point correspondences for arbitrary numbers of views have also been characterized [START_REF] Faugeras | On the geometry and algebra of the point and line correspondences between n images[END_REF][START_REF] Triggs | Matching constraints and the joint image[END_REF][START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF], however results are often scattered in the literature, and they sometimes actually contradict each other [START_REF] Faugeras | On the geometry and algebra of the point and line correspondences between n images[END_REF][START_REF] Hartley | Multiple view geometry in computer vision[END_REF][START_REF] Trager | Trinocular Geometry Revisited[END_REF]. Moreover, basic theoretical facts are not always common knowledge among specialists. For example, many practitioners today would probably be hard pressed to answer simple questions such as how many multilinear relations (and which ones) are necessary to characterize correspondences, or to determine the corresponding camera parameters. It is also not difficult to stumble upon open problems that have not been addressed in the literature (see for example Section 1.4). Partly because of the supply of such problems, there has been a recent interest in computer vision among mathematicians, with the emerging field of algebraic vision [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF][START_REF] Aholt | The ideal of the trifocal variety[END_REF]. As explained in the introduction to the thesis, our work often lies at the interface between vision and mathematics: our goal is to investigate problems in vision using mathematical language, but without losing focus of the underlying practical motivations.

One of our main objects of study is the joint image, introduced by Triggs [START_REF] Triggs | Matching constraints and the joint image[END_REF] and studied independently by Heyden and Åström [START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF], and by Aholt, Sturmfels and Thomas [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF]. The joint image for n cameras is a subvariety in (P 2 ) n defined as the closure of the set of all n-tuples of point correspondences. Thus, algebraic characterizations of the joint image are equivalent to the constraints that describe multi-view correspondences. In addition, the joint image is a useful conceptual tool, since it can be seen as an (almost) exact replica of 3D-space "distributed" across multiple images. A joint image also uniquely represents a configuration of cameras so, for example, we can interpret the task Introduction of 3D reconstruction as the problem of interpolating a joint image from image data (this is similar to the use of multi-view tensors, but for any number of views). We present in Section 1.3 a detailed description of the joint image, based on our paper [START_REF] Trager | The joint image handbook[END_REF]. In particular, we give a series of results that explain which sets of multilinear constraints can be used for characterizing point correspondences, or for recovering a camera configuration.

Among the results that we discuss in Section 1.3 is the (previously known) fact that fundamental matrices are sufficient for recovering a camera configuration, assuming that the pinholes of the cameras are not all aligned. On the other hand, it is not necessary to use the fundamental matrices among all pairs of cameras, and this leads to the natural question of describing how many fundamental matrices, and which ones, are actually needed. To study this problem, we use the notion of a viewing graph, introduced by Levi and Werman in [START_REF] Levi | The viewing graph[END_REF]. This is a graph in which edges represent fundamental matrices among pairs of cameras. We say that a graph is "solvable" when the associated fundamental matrices are (generically) sufficient to determine a global camera configuration. Despite its clear significance, the problem of characterizing which viewing graphs are solvable has received very little attention (mainly in [START_REF] Levi | The viewing graph[END_REF][START_REF] Rudi | Linear solvability in the viewing graph[END_REF]). In Section 1.4 we present results from our paper [START_REF] Trager | On the Solvability of Viewing Graphs[END_REF], providing several new criteria that can be used to verify whether a set of fundamental matrices determines a camera configuration.

At the end this chapter we take a closer look at the geometry of three views (Section 1.5). For example, following our paper [START_REF] Trager | Trinocular Geometry Revisited[END_REF], we describe a special class of trilinearities (previously used by Ponce, Papadopoulo, Teillaud and Triggs in [START_REF] Ponce | The absolute quadratic complex and its application to camera self calibration[END_REF]) that have an interpretation in terms of transversals of visual rays, and provide a more geometric approach for characterizing point correspondences among three views. We then turn to trifocal tensors, and show that their relationship with the joint image can be explained using a theory of "multi-graded Chow forms" that we proposed in [START_REF] Osserman | Multigraded Cayley-Chow forms[END_REF]. In brief, trifocal tensors (and similarly quadrifocal tensors for four views) can be seen as an example of a general strategy for encoding complicated algebraic varieties (in this case, the joint image) using a single polynomial form. Finally, we present some experiments from [START_REF] Trager | Trinocular Geometry Revisited[END_REF], where we used trilinear constraints to define a "trinocular-epipolar error" that can be minimized to recover camera parameters for three views.

Previous Work

Multiple-view geometry has been studied in computer vision since the seminal work of Longuet-Higgins, who proposed in 1981 the essential matrix for pairs of calibrated cameras [START_REF] Christopher | A computer algorithm for reconstructing a scene from two projections[END_REF]. Its uncalibrated counterpart, the fundamental matrix, was introduced by Luong and Faugeras [START_REF] Luong | The Fundamental matrix: theory, algorithms, and stability analysis[END_REF]. The trilinear constraints associated with three views of a straight line were discovered by Spetsakis and Aloimonos [START_REF] Spetsakis | Structure from motion using line correspondences[END_REF] and by Weng, Huang and Ahuja [START_REF] Weng | Motion and structure from line correspondences: closed-form solution, uniqueness, and optimization[END_REF]. The uncalibrated case was tackled by Shashua [START_REF] Shashua | Algebraic functions for recognition[END_REF] and by Hartley [START_REF] Richard | Lines and points in three views and the trifocal tensor[END_REF], who coined the term "trifocal tensor". The quadrifocal tensor was introduced by Pinhole Cameras and their Multi-View Geometry Triggs [START_REF] Triggs | Matching constraints and the joint image[END_REF], and Faugeras and Mourrain gave a simple characterization of all multilinear constraints associated with multiple perspective images of a point [START_REF] Faugeras | On the geometry and algebra of the point and line correspondences between n images[END_REF]. A different formulation for trilinear constraints based on line geometry was introduced in [START_REF] Ponce | The absolute quadratic complex and its application to camera self calibration[END_REF].

Closely related to our work is that of Heyden and Åström in [START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF], who also study multiview constraints from the point of view of the joint image (which they refer to as the "natural descriptor"). The authors also point out an interesting property of the epipolar constraints for three cameras in general position: these conditions uniquely determine a camera configuration however the trilinear conditions do not follow algebraically from the bilinear ones and, in fact, bilinear constraints are not sufficient for characterizing point correspondences in general. We discuss this somewhat paradoxical behavior in a more general setting in Section 1.3 of this chapter.

Aholt, Sturmfels and Thomas describe many algebraic and combinatorial properties of the joint image (in their terminology, the "multi-view variety") [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF]. They show for example that traditional bilinear, trilinear and quadrilinear constraints are generators, and form in fact a "universal Gröbner basis", for the polynomial ideal associated with the joint image (see Theorem 1.3.10). They also prove other technical results, such as the fact that the set of all joint images forms a distinguished component in the "Hilbert scheme" parameterizing all varieties with assigned Hilbert function.

The first investigation of viewing graphs and their solvability can be found in [START_REF] Levi | The viewing graph[END_REF]. In that work, Levi and Werman characterize all solvable viewing graphs with at most six vertices, and discuss a few larger solvable examples. Although they provide some useful necessary conditions (see our Proposition 1.4.6 and Example 1.4.13), they do not address the problem of solvability in general. In [START_REF] Rudi | Linear solvability in the viewing graph[END_REF], Rudi, Pizzoli and Pirri also consider viewing graphs, studying mainly whether a configuration can be recovered from a set of fundamental matrices using a linear system. They also present some "composition rules" for merging solvable graphs into larger ones. In [START_REF] Trager | The joint image handbook[END_REF], we provided a sufficient condition for solvability using 2n -3 fundamental matrices, and pointed to a possible connection with "Laman graphs" and graph rigidity theory. Indeed, Özyesil and Singer [START_REF] Ozyesil | Robust camera location estimation by convex programming[END_REF] show that if one uses essential matrices instead of fundamental ones then solvability can be characterized in terms of so-called "parallel-rigidity" for graphs. Their analysis however does not carry over to the more general setting of uncalibrated cameras. Finally, the viewing graph has also been considered in more practical work, particularly to enforce triple-wise consistency among fundamental matrices before estimating camera parameters: this is done for example by Sinha and Pollefeys [START_REF] Sudipta | Camera network calibration and synchronization from silhouettes in archived video[END_REF] and by Sweeney, Sattler, Hollerer, Turk and Pollefeys [START_REF] Sweeney | Optimizing the viewing graph for structure-from-motion[END_REF].

Main contributions

Our main contributions can be summarized as follows.

• The joint image. We make the distinction between ideal-theoretic, set-theoretic, Introduction and "weak" characterizations (our terminology) of the joint image, and use it to explain the difference between multi-view constraints that determine camera geometry, and those that directly describe point correspondences. In particular, we clarify whether bilinear, trilinear or quadrilinear constraints are necessary or sufficient for these tasks, under different assumptions on the configurations of pinholes (Proposition 1. 3.19). We also give a simple proof of the important result that a joint image characterizes a configuration of cameras (Theorem 1.3.16).

• The viewing graph. We show that the minimum number of fundamental matrices that can be used to recover a configuration of n cameras is always (11n-15)/7 (Theorem 1.4.9). We also present several new criteria for deciding whether or not a viewing graph is solvable (Theorems 1.4.12 and 1.4.14), and we describe a simple linear test that can be used to verify whether a viewing graph identifies a finite number of camera configurations (Section 1.4.5).

• Three-View geometry and Multifocal Tensors. We apply our analysis on the different types of characterizations of the joint image to the important case of three views. For example, we show that the nine trilinearities encoded in a trifocal tensor are not sufficient to completely ensure correspondence among three views (Proposition 1.5.7), although they can be used to recover the corresponding projection matrices. We also propose a new framework for understanding trifocal and quadrifocal tensors, based on the joint image and the theory of multi-graded Chow forms [START_REF] Osserman | Multigraded Cayley-Chow forms[END_REF].

Before addressing our main topics, we present in Section 1.2 a general introduction to the pinhole camera model, and to linear projections in general. We review for example the action of "world" and "image" coordinate changes, orbits under euclidean and affine motions, and projections and inverse projections of linear subspaces. These discussions will be useful later in the thesis.

Conventions. Throughout the chapter, we write P n = P(R n+1 ) for the for the ndimensional projective space over R. It is worth noting that most of our definitions from Section 1.2 can be given more generally for projective spaces P(K n+1 ) over an arbitrary field K. In the next chapters, for example, we will sometimes work over the field of complex numbers K = C. Since physical three-space is usually identified with R 3 , we focus on real spaces here.

We use bold font for vectors and matrices, and normal font for projective objects. For example, a point in P 3 will be written as p = [p] where p is a vector in R 4 and p is the equivalence class associated with p. Similarly, a projective transformation represented by a matrix M will be written as M = [M].

Pinhole Cameras and their Multi-View Geometry

Preliminaries

This section summarizes some basic aspects of the theory of pinhole cameras. The material we discuss is well-known, and can be found in textbooks such as [START_REF] Faugeras | The geometry of multiple images: the laws that govern the formation of multiple images of a scene and some of their applications[END_REF][START_REF] Hartley | Multiple view geometry in computer vision[END_REF], although our approach is sometimes different.

The organization of the section is as follows. Section 1.2 is an overview of the pinhole camera model; Section 1.2.2 introduces more formally linear projections in arbitrary spaces; in Section 1.2.3 we describe the group actions of "world" and "image" coordinate transformations; we classify orbits of cameras under euclidean and affine motions in Section 1.2.4; finally, in Section 1.2.5, we use Plücker coordinates to represent projections and inverse projections of linear spaces in arbitrary dimensions.

The pinhole model

The pinhole camera model used in computer vision is a mathematization of the geometry of the camera obscura. This is illustrated in Figure 1.1. The "imaging device" consists of a pinhole c that collects light rays and a retinal plane H where the image is formed.

A scene point x is projected onto the image point y, obtained as the intersection of H with unique line through through x and c. This map is easily described in the language of projective geometry. If we identify the three-dimensional space with P 3 , and fix a point c = P(K) for the pinhole and a plane H = P(W ) not containing c for the retinal plane (so that K, W are vector subspaces in R 4 of dimensions 1 and 3), we consider the unique vector projection R 4 = W ⊕ K → W onto W with null space K. Then the induced map on projective spaces P 3 \ {c} → H is a mathematical model for the camera obscura shown in Figure 1.1.

A more concrete description can be given in terms of projective coordinates. If we fix a projective reference frame in three-space and a projective reference frame in the retinal plane, we can describe the action of a pinhole camera using a 3 × 4 matrix of full rank, uniquely determined up to scaling. This is usually known as a projection matrix. This analytic representation is very convenient, and widely used in computer vision. However

Preliminaries

it is important to emphasize some differences between the projection matrix and the geometric model described above.

• There is no unique projection matrix associated with a fixed "camera obscura" with pinhole c and retinal plane H, since the analytic description depends on the choice of coordinates in the image and in space. In general, it is customary to assume a fixed reference frame in three-space, despite the fact that there is no distinguished coordinate system the physical 3D world. For the retinal plane, on the other hand, there is often a natural choice of coordinates, defined by the pixel grid in a digital camera.

• Conversely, given a projection matrix, it is not possible to recover the retinal plane (while this is possible for the pinhole, which corresponds to the null-space of the matrix). Indeed, the image P 2 of the projection is now an abstract plane that does not correspond to any particular embedded plane in space. Moreover, for any plane in P 3 not containing the pinhole, there exists a unique choice of coordinates so that the corresponding projection is described by the original matrix. The equivalence between all retinal planes follows from the "perspectivity" homography induced by the pinhole.

In light of these ambiguities, it is tempting to exclude the role of retinal plane from the whole imaging process, and to view a pinhole projection simply as a mapping from points into the bundle of lines that pass through the pinhole. In this setting, the projection matrix is equivalent to specifying the pinhole together with a reference frame on the corresponding line bundle. This abstract representation of a camera as a mapping from points to lines will be studied in Chapter 2. In the remaining part of this chapter, we focus entirely on the more traditional and "concrete" coordinate-based model that uses projection matrices.

Projections in projective spaces

We next define linear projections in general projective spaces. Although we are mainly interested in projections from P 3 to P 2 , it is actually natural to extend many properties of systems of cameras to the case of projections in spaces of arbitrary dimensions. It is not our purpose to develop such a general theory, and in the rest of the chapter we focus mainly on pinhole projections. Nevertheless, it is useful to give definitions in a more general setting, since projections in different dimensions will be sometimes used in later chapters.

By linear projection we always mean the projectivization of a surjective linear map on vector spaces. More precisely, if a linear map R n+1 → R m+1 (n > m) is described

Pinhole Cameras and their Multi-View Geometry by an (m + 1) × (n + 1) matrix A of full rank, the corresponding linear projection is given by

P n \ P(KerA) → P m [v] → [Av].
(1.1)

The subspace K = P(KerA) inside P n is the center of the projection, and has projective dimension n -m -1. In the case of cameras (n = 3, m = 2), the center of projection is simply the pinhole. It is clear that two matrices induce the same linear projection if and only if they are related by a non-zero scalar factor. Just as pinhole cameras, any linear projection can be realized geometrically by fixing a center K of dimension n -m -1 and another subspace H in P n of dimension m disjoint from K. These spaces determine a mapping from P 3 to H defined by p → (K ∨ p) ∧ H that can be described analytically by (1.1). See Figure 1.2.

Figure 1.2: Linear projections P 2 P 1 (left), P 3 P 1 (center), and P 3 P 2 (right). Note that the "retinal space" is a line in the first two cases (shown in dark).

It will often be convenient to write a linear projection as a map P n P m . Here the dashed arrow indicates a "rational map", i.e., an algebraic map defined on a dense open set of P n (cf. Appendix B). This simplifies notation, since we are not required to exclude the center of projection from the domain. We will also write P = [A], when the linear projection P : P n P m is induced by the matrix A.

Changes of coordinates

We briefly describe of the action of "changing coordinates" on projection mappings. This analysis will be useful for dealing with camera configurations. A similar study can be carried out for projections in arbitrary projective spaces, but we only consider pinhole cameras here. We write P for the family of all pinhole cameras, that we represent using projection matrices:

P = {P = [A], A ∈ R 3×4 of full rank} ⊂ P 11 .
There are two natural group actions on P: the group GL(4, R) acts by multiplication on the right (changes of "world" coordinates), while the group GL(3, R) acts by multiplication on the left (changes of "image" coordinates).1

Changes of world coordinates. Given pinhole camera P = [A] where A is a 3 × 4 matrix, we consider the action of multiplying A on the right by matrices in GL(4, R).

Clearly, this can be interpreted as a change of coordinates of P 3 , but also as applying a projective "camera motion" to P .

We first observe that the action of GL(4, R) on P is transitive. This means that, given any two cameras P 1 = [A 1 ] and P 2 = [A 2 ], there always exist a change of coordinates (or a camera motion) that maps P 1 to P 2 . Indeed, if c 1 = [c 1 ] is the pinhole of P 1 , we have that any invertible matrix of the form

T α,v = αA † 1 A 2 + c 1 v T ,
where

A † 1 = A T 1 (A 1 A T 1 ) -1 (a pseudo-inverse for A 1 ), α ∈ R, and v ∈ R 4 , is such that [A 1 T α,v ] = [A 2 ].
The action of GL(4, R) on P however is not free. In other words, for any camera P = [A], there exist projective transformations of P 3 that do not affect P . These transformations form the stabilizer group:

Stab GL(4,R) (P ) = {αI 4 + cv T | α ∈ R \ {0}, v ∈ R 4 } ∩ GL(4, R), (1.2)
where c = [c] is the pinhole of P , and I 4 denotes the 4 × 4 identity matrix. Indeed, all the solutions for T in AT = αA are described by (1.2). As projective transformations, elements in Stab GL(4,R) (P ) "shift" points along viewing lines through c. In fact, it is easy to see that the stabilizer only depends on the pinhole c and not on the actual projection matrix.

We also mention that there is another natural group associated with a camera P with center c = [c], namely

Stab GL(4,R) (c) = {T | Tc = αc} ⊂ GL(4, R).
This corresponds to projective transformations of P 3 that fix the pinhole c. It is clear that Stab GL(4,R) (P ) is a subgroup of Stab GL(4,R) (c). Moreover, elements in Stab GL(4,R) (c) can actually be viewed as changes of image coordinates, since they induce projective transformations on the bundle of lines through c. We now address changes of image coordinates directly.

Pinhole Cameras and their Multi-View Geometry

Changes of image coordinates. Given pinhole camera P = [A], we consider the action of multiplying A on the left by matrices in GL (3, R). This action is not transitive: if c = [c] is the pinhole of P , the orbit of P under GL(3, R) (i.e., the set of cameras that can be obtained from P by changes of image coordinates) is

Orb GL(3,R) (P ) = {Q = [B] | Bc = 0} ⊂ P.
This is simply saying that two cameras are related by a change of image coordinates if and only if they have the same pinhole. On the other hand, the stabilizer group of this action, only contains trivial projective transformations:

Stab GL(3,R) (P ) = {αI 3 , | α ∈ R \ {0}} ⊂ GL(3, R).
In more concrete terms, non-trivial projective changes of coordinates in the image will always affect a camera projection.

Geometries in projective space and orbits of cameras

This chapter (and this thesis) focuses mainly on projective camera models. In what follows, however, we give an account of euclidean and affine cameras, viewed as orbits of special subgroups of projective transformations. This serves mostly as reference for when we will generalize these concepts for non-central imaging systems (in Chapter 3).

We first briefly recall the hierarchy of different geometries that can be studied within the projective framework. This viewpoint was set forth by Felix Klein in his famous Erlangen program (1872), and is of great practical importance in computer vision. In this setting, a "geometry" can be seen as the study of some space and its "invariant properties" under the action of a group of transformations. Projective geometry, in particular, studies projective space P n up to projective transformations, that we identify with the group of (n + 1) × (n + 1) invertible matrices defined up to scale. Among these transformations, we consider subgroups of matrices of the following form (defined up to scale):

Iso(n) = T = R t 0 T 1 with R ∈ SO(n, R) , Sim(n) = T = kR t 0 T 1 with k ∈ R, R ∈ SO(n, R) , Aff(n) = T = M t 0 T 1 with M ∈ GL(n, R) , so that Iso(n) ⊂ Sim(n) ⊂ Aff(n) ⊂ GL(n + 1, R).

Preliminaries

Here SO(n, R) denotes the special orthogonal group (matrices with unit determinant). The groups of transformations Iso(n), Sim(n) Aff(n) are respectively associated with euclidean, similarity and affine geometries. Indeed, if we identify R n with the chart in P n of points of the form [x 1 , . . . , x n , 1], then these groups coincide with the classical definitions of euclidean, similarity, and affine transformations in R n (with respect to the standard euclidean metric). Each these geometries has its associated invariants, and smaller transformation groups correspond to more properties that are preserved.

Example 1.2.1. An important invariant for computer vision is the absolute quadric. This is a complex degenerate quadric in P n defined by Ω = {x n+1 = x 2 1 + . . . + x 2 n = 0}. This quadric is invariant under all similarity transformations. In fact, a projective transformation is a similarity if and only if it preserves (not necessarily pointwise) the absolute quadric.

♦

We now describe the action of these nested subgroups of transformations on projection mappings. We consider only pinhole camera projections P 3 P 2 , although a similar analysis could be carried out more generally.

We have already noted that the action of GL(4, R) on the space of cameras can be interpreted either as applying "camera motions", or as applying "change of coordinates" in P 3 . In particular, the orbit of a camera P under the action of one of the projective subgroups is the set of cameras that are equivalent to P for the corresponding geometry (in fact, one might say that cameras in the same orbit are geometrically indistinguishable, since a world reference frame is not physically defined). The equivalence of cameras for different geometries leads to a natural "taxonomy" of cameras. Although a complete classification would probably not be very difficult, we describe here only generic orbits that correspond to camera models actually used in computer vision.

Projective geometry. We have seen in Section 1.2.3 that all pinhole cameras are equivalent up to projective transformations, so they form a unique projective orbit. We can thus choose any camera as a representative for the class, for example the camera P associated with the 3 × 4 matrix    1 0 0 0 0 1 0 0 0 0 1 0

   .
(

We sometimes refer to this as the standard pinhole projection.

Affine geometry. Let P = [A] be camera with

A = [Q |b]
where N is a 3 × 3 matrix. Applying to P an affine transformation described by a matrix of the form T = M t 0 T 1 , we obtain the camera [QM | Qt+b]. This easily implies that cameras of the form

[Q |b],
where Q is an invertible 3 × 3 matrix, are all equivalent under affine transformations, so they form a single affine orbit. This is the set of finite cameras, i.e., cameras whose pinhole does not lie on the plane at infinity. Note that finite cameras are dense among all pinhole cameras, since for a generic 3 × 4 matrix, the left 3 × 3 submatrix will be invertible. The standard pinhole projection (1.3) can be taken as a representative for the affine model of finite cameras.

Among non-finite cameras whose pinhole lies at infinity, a distinguished family is associated with 3 × 4 matrices whose last row can be scaled to [0, 0, 0, 1]. These are known as affine cameras, since they can be described as affine projections using non-homogeneous coordinates. It is easy to see that affine cameras are all equivalent under affine transformations. We can choose as representative for this orbit the camera defined by

  
1 0 0 0 0 1 0 0 0 0 0 1

   .
(

1.4)

There exist other affine orbits among non-finite cameras, but their use in computer vision is very limited.

Similarity geometry. The orbit under similarities of the standard pinhole camera (1.3) is the set of cameras associated with matrices of the form [R | t] where R is an orthogonal matrix. These are sometimes called normalized or calibrated cameras. More generally, it is easy to characterize similarity orbits among all finite cameras. If [M | b] is a finite camera, then using QR-decomposition of matrices we can uniquely write

[M | b] = K[R | t], (1.5)
where R is an orthogonal matrix and K is upper triangular with positive diagonal elements. The matrix K in this decomposition is invariant to similarity transformations (up to scaling, but projection matrices are independent of scale). Moreover, any two matrices whose decomposition yields the same matrix K up to scale are related by a similarity transformation. In other words, the entries of K up to scale identify the similarity orbit of a finite camera. The matrix K is known as the calibration matrix and its six entries defined up to scale are the cameras intrinsic parameters. These have geometric interpretations (x-scale, y-scale, skew factor, coordinates of the principal point) that describe a projective transformation in P 2 that needs to be applied to standard model (1.3) to recover the similarity orbit of the given camera.

A similar analysis can be carried out for orbits among affine cameras. Any projection matrix of an affine camera can be uniquely decomposed as

M 2×3 b 0 T 1 =    α s x s s 0 0 1 0 0 0 1       kr T 1 t 1 kr T 2 t 2 0 T 1    , ( 1.6)
where r 1 , r 2 are orthonormal 3-vectors, and α x is positive. The entries α s x , s s are invariant to similarity transformations (not up to scaling) and in fact they identify the similarity orbit. When α s x = 1 and s s = 0, the camera belongs to the similarity orbit of (1.4). This type of camera is called a scaled orthographic projection (with factor k).

Euclidean geometry. The euclidean orbits among finite cameras coincide with similarity orbits. Indeed, applying to a finite camera [M|b] a similarity transformation This is of course a way of explaining the well-known scale ambiguity in pictures taken with perspective pinhole cameras: a scaling a 3D scene by T has the same effect of applying a rigid camera motion T. The fact that finite cameras are similarity-equivalent if and only if they are euclidean-equivalent means that the calibration matrix K from (1.5) also identifies euclidean orbits for finite cameras.

T = kR t 0 T 1 ,
Interestingly, the situation is different for affine cameras: in particular, it is easy to see that the affine and euclidean orbits of (1.4) are not the same. A slight variation of the decomposition (1.6) yields

M 2×3 b 0 T 1 =    α x s 0 0 α y 0 0 0 1       r T 1 t 1 r T 2 t 2 0 T 1    , (1.7)
where r 1 , r 2 are orthonormal 3-vectors, and α x , α y are positive. The entries α x , α y , s in this decomposition are invariant under euclidean transformations (but not under similarities). When α x = α y = 1 and s = 0, the camera belongs to the euclidean orbit of (1.4), and is called an orthographic projection.

Projections and inverse projections of subspaces

Using pinhole cameras, the projection of a general line in P 3 is a line in P 2 . Moreover, the pre-image of a line in P 2 is a plane in P 3 , and the pre-image of a point in P 2 is a line in P 3 . This is a particular example of a general feature of linear projections in arbitrary projective spaces, namely that projections and inverse projections of linear Pinhole Cameras and their Multi-View Geometry subspaces are again linear subspaces. All of these associations can be conveniently described using Plücker coordinates (see Appendix A). Although we are mostly interested in cameras P 3 P 2 , it is useful to introduce these mappings first for general projections P n P m .

We consider a linear projection P : P n P m with center K. If L is a k-dimensional linear subspace of P n disjoint from K (which implies k ≤ m), then the projection M = P (L) is a k-dimensional linear space in P m . In particular, writing Gr(k, P r ) for the Grassmannian of k-linear subspaces in P r , there is a map Gr(k, P n ) \ Z(k, K) → Gr(k, P m ) L → P (L), (1.8) where Z(k, K) is the set of k-spaces in P n which meet K. We can give a concrete description of (1.8) using Plücker coordinates. Assuming that P = [A], this description is based on the compound matrix C k+1 (A), whose elements are the (k + 1) × (k + 1) minors of A ordered lexicographically (this assumes Plücker coordinates are ordered lexicographically).

w H = C k+1 (A)v L .
In particular, the map (1.8) can be seen as a linear projection P k : P ( n+1 k+1 ) P ( m+1 k+1 ) where P k = [C k+1 (A)], restricted to the Grassmannian varieties. The center of P k is the linear space generated by Z(k, K) inside P ( n+1 k+1 )-1 .

Proof. Let V L be a matrix of size (n + 1) × (k + 1) whose columns are coordinate vectors spanning L. Since H = P (L), we have that W H = AV L is an (m + 1) × (k + 1) matrix whose column vectors span H. By definition, Plücker vectors v L , w H for L and H are given by C k+1 (V L ) and C k+1 (W H ). The first claim now follows from the multiplicativity property of compound matrices: if M 1 , M 2 are arbitrary matrices, then [START_REF] Roger | Matrix analysis[END_REF]. To argue that Z(k, K) generates the center of P k , we note that Z(k, K) is contained in the center, and moreover dim Span(Z(k, K)) = n+1 k+1 -m+1 k+1 -1 inside P ( n+1 k+1 )-1 .

C i (M 1 M 2 ) = C i (M 1 )C i (M 2 ) holds
To describe pre-images of linear spaces, it is convenient introduce a "dual" version of a linear projection, which we refer to as a linear embedding. This is a map J : P m → P n , with m < n, that is the projectivization of an injective linear mapping R m → R n , described by an (n + 1) × (m + 1) matrix B of full rank (and we will write J = [B]). An embedding may be viewed as an identification of (an abstract) P m with an m-dimensional

Preliminaries

subspace N inside P m , which is the image of J. In other words, specifying an embedding J is equivalent to fixing a linear space N in P m together with a projective reference frame on N .

The duality between projections and embeddings can be seen concretely by noting that a linear projection P : P n P m with P = [A] has an associated embedding J : P m → P n where J = [A T ]. More geometrically, the map J can be interpreted as a map on dual projective spaces: the projection P induces in fact a mapping (P m ) * → (P n ) * associating every hyperplane H in P m with its pre-image P -1 (H) inside P n . It is immediate to verify that using dual homogeneous coordinates this association is described by A T . In terms of this dual representation, the image of J corresponds to K ∨ , i.e., the system of all hyperplanes in P n containing the center K of the projection P . Note that since K

has dimension n -m -1, the dimension of K ∨ is indeed m (cf. Appendix A).
Embeddings are similar to projections in the way they act on linear subspaces. If M is an r-dimensional linear subspace of P m , its image J(M ) under the embedding J is an r-dimensional linear space in P n . This defines a map

Gr(r, P m ) → Gr(r, P n ) M → J(M ), (1.9)
whose image is Y (r, N ), the set of of r-linear spaces in P n contained in N (the image of J). If J = [B], the map (1.9) can be described in terms of Plücker coordinates using C r+1 (B).

v U = C r+1 (B)w M .
In particular, the map (1.9) can be thought of as a linear embedding J r : P ( m+1 r+1 ) → P ( n+1 r+1 ) where J r = [C r+1 (B)], restricted to the Grassmannian varieties. The image of J r is the linear space Y (r, N ) inside P ( m+1 r+1 )-1 .

Proof. This is completely analogous to Proposition 1.2.2.

When an embedding J = [B] is viewed as the dual of a projection P = [A] (so B = A T ), the embedding in Proposition 1.2.3 describes inverse mappings of linear spaces. More precisely, the projection P induces a map Gr(m-r-1, P m ) → Gr(n-r-1, P n ) associating an (m -r -1)-dimensional space T in P m with its (n -r -1)-dimensional pre-image P -1 (T ) inside P n . Proposition 1.2.3 describes this map for dual Plücker coordinates (in both P m and P n ). The image of J is the set of of (n -r -1)-linear spaces containing the center K of the projection P .

Pinhole Cameras and their Multi-View Geometry

Since C k+1 (A) T = C k+1 (A T ) holds, the analytical expressions of projections and inverse projections of linear spaces are related. However, while C k+1 (A) acts on linear spaces of dimension k (in P n ), its transpose C k+1 (A T ) is applied to linear spaces of codimension k (in P m ), because of the use of dual coordinates. For example, a general k-dimensional space L in P n can be projected using C k+1 (A) to a k-dimensional space M in P m , but the inverse image S = P -1 (M ) of M is obtained by applying C m-k (A T ) to the dual representation of M . In fact, S = P -1 (P (L)) has dimension n -m + k (codimension m -k), and is spanned by L and the center of projection K.

The case of cameras. We now describe projections and inverse projections of linear spaces more concretely for a camera P :

P 3 P 2 with P = [A].
The first interesting case is the mapping from lines in P 3 to lines in P 2 . This is sometimes known as the line projection map. According to Proposition 1.2.2, it is described by the 3 × 6 matrix

C 2 (A) =    A [12][12] A [12][13] A [12][14] A [12][23] A [12][24] A [12][34] A [13][12] A [13][13] A [13][14] A [13][23] A [13][24] A [13][34] A [23][12] A [23][13] A [23][14] A [23][23] A [23][24] A [23][34]    , (1.10)
where

A [ij][kl]
denotes the 2×2 minor of A corresponding to the rows i, j and columns k, l. Thus, a line in P

[v] = [v 12 , v 13 , v 23 ] where v = C 2 (A)p.
Note however that the usual representation of lines in P 2 is in terms of dual (Plücker) coordinates: this means that rather than the vector v it is often more natural to use v * = (v 23 , -v 13 , v 12 ), since the line with Plücker coordinates [v] is defined by the equation v 23 x 1 -v 13 x 2 + v 12 x 3 = 0. In particular, for this choice the line projection map becomes associated with the matrix

   A [23][12] A [23][13] A [23][14] A [23][23] A [23][24]
A [START_REF] Boyd | Convex optimization[END_REF][34] -A [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF][12] -A [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF][13] -A [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF][14] -A [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF][23] -A [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF][24] -A [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF] [START_REF] Chien | Volume/surface octrees for the representation of three-dimensional objects[END_REF] A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][12] A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][13] A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][14] A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][23] A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][24] A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][34]    .

(1.11)

We now turn to inverse projection mappings. In the notation of Proposition 1.2.3, the only interesting cases are r = 0 and r = 1. The case r = 1 is simply the dual embedding (P 2 ) * → (P 3 ) * , associating a line in P 2 with its pre-image in P 3 , that is a plane containing the pinhole. In normal dual coordinates, it is described by the 4 × 3 matrix A T . The case r = 0 is a map P 2 → Gr(1, P 3 ) associating a point in P 2 with the corresponding "viewing line" through the pinhole. According to Proposition 1.2.3, it is described by the 6 × 3 matrix C 2 (A T ), which is the transpose of (1.10). This description is however based on dual coordinates, in both P 2 = Gr(0, P 2 ) and Gr(1, P 3 ): this means that the pre-image of a point [u] = [u 1 , u 2 , u 3 ] in P 2 is a line with dual Plücker coordinates [q] = [q 34 , -q 24 , q 23 , q 14 , -q 13 , q 12 ] given by q = C 2 (A) T u * , where u * = (u 3 , -u 2 , u 1 ). To use standard (primal) coordinates in P 2 and Gr(1, P 3 ) one needs to apply the following

The Joint Image Variety 6 × 3 matrix, obtained from C 2 (A) T by appropriately fixing signs and permuting rows and columns:

         A [23][34] -A [13][34]
A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][34] -A [START_REF] Boyd | Convex optimization[END_REF] [START_REF] Boyer | On using silhouettes for camera calibration[END_REF] A [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF][24] -A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][24] A [START_REF] Boyd | Convex optimization[END_REF][23] -A [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF] [START_REF] Boyd | Convex optimization[END_REF] A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][23] A [START_REF] Boyd | Convex optimization[END_REF][14] -A [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF] [START_REF] Bernardini | The ball-pivoting algorithm for surface reconstruction[END_REF] A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][14] -A [START_REF] Boyd | Convex optimization[END_REF] [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF] A [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF][13] -A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][13] A [START_REF] Boyd | Convex optimization[END_REF][12] -A [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF] [START_REF] Guenther | Geometric modeling for computer vision[END_REF] A [START_REF] Guenther | Geometric modeling for computer vision[END_REF][12]

        
.

(1.12)

The Joint Image Variety

In this section we study systems of multiple cameras using the joint image. We begin by presenting some basic definitions and geometric properties of the joint image (Sections 1.3.1 and 1.3.2). We discuss in Section 1.3.3 the differences between families of algebraic constraints that can be used to describe point correspondences, making the distinction between "ideal-theoretic", "set-theoretic" and "weak" characterizations of the joint image. In Section 1.3.4 we introduce the space of camera configurations, and prove that a joint image uniquely characterizes a configuration (Theorem 1.3.16). Finally, in Section 1.3.5 we clarify the role of bilinear and trilinear conditions for determining point configurations and camera geometry (Theorem 1.3.19).

Basic definitions

In the following, we consider a family P 1 , . The set of all point correspondences has been previously considered, with various names. To our knowledge, it was first introduced by Triggs, who coined the term "joint image" in [START_REF] Triggs | Matching constraints and the joint image[END_REF]. Heyden and Åstrom refer to it as "natural descriptor" set, and study some of its properties in [START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF]. It was then reintroduced in mathematics as the multi-view variety by Aholt, Sturmfels and Thomas [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF]. We use Triggs' original term the "joint image", which we find to be most descriptive.

It was noted by Heyden and Åström [START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF] that the joint image is not an algebraic set, in other words it cannot be described as the zero-set of a family of polynomial equations.

Pinhole Cameras and their Multi-View Geometry In the Zariski topology, closed sets coincide with algebraic sets. This means that M n is the smallest set containing M o n that can be described by polynomial equations. The discrepancy between M n and M o n is a consequence of the fact that the 3D point x from Definition 1.3.1 is not allowed to be one of the pinholes c 1 , . . . , c n . This is a necessary requirement because at the pinholes the camera mappings are not all defined. To describe the difference between M n and M o n more concretely, we recall the notion of an epipole, that is well known in multi-view geometry. For two cameras P i and P j with distinct pinholes c i and c j respectively, the epipoles e ij , e ji are image points (P 2 ) defined by e ij = P i (c j ) and e ji = P j (c i ). Note that for n cameras P 1 , . . . , P n we expect n -1 epipoles in each image, however if any three pinholes (say c i , c j , c k ) are aligned, then the corresponding epipoles in each image will coincide (e.g., e ij = e ik ). In particular, if all pinholes are aligned there is only one epipole per image.

Proposition 1.3.5. Given n ≥ 3 cameras with non-collinear distinct pinholes, one has

M o n = M n \ E n
, where

E n = n i=1 {e 1i } × . . . × P 2 (i) × . . . × {e ni }.
(1.13)

Here P 2 (i) indicates P 2 at position i in the product, and e ij denotes the j-th epipole in the i-th image. However, if n = 2, or more generally if the cameras have collinear pinholes, then

E n = n i=1 {e 1 } × . . . × P 2 (i) × . . . × {e n } \ {(e 1 , . . . , e n )}, (1.14)
where e i now denotes the unique epipole in image i (so e i = P i (c j ) for all j = i).

A proof of this result is given at the end of the chapter. However, the intuition behind it is quite clear: an n-tuple (u 1 , . . . , u n ) in (P 2 ) n belongs to M n if and only if the corresponding viewing lines 1 , . . . , n defined by i = P -1 i (u i ) all meet, and the set E n corresponds to the n-tuples for which these lines meet (only) at one of the pinholes c i .

Example 1.3.6. For two cameras P 1 , P 2 , any point correspondence (u 1 , u 2 ) in P 2 × P 2 with u i = [u i ], i = 1, 2, satisfies an algebraic relation u T 1 Fu 2 = 0, where F is the 3 × 3 fundamental matrix associated with P 1 and P 2 [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. On the other hand, because the two epipoles e 1 , e 2 correspond to the left and right null-spaces of F, the same algebraic relation is also satisfied by all pairs in P 2 ×P 2 of the form (e 1 , u 2 ) or (u 1 , e 2 ), for arbitrary

The Joint Image Variety u 1 and u 2 . Indeed, we have in this case that

M 2 = { ([u 1 ], [u 2 ]) | u T 1 Fu 2 = 0 } ⊂ P 2 × P 2 , M o 2 = M 2 \ E 2 ,
where

E 2 = ({e 1 } × P 2 ) ∪ (P 2 × {e 2 }) \ {(e 1 , e 2 )}.
This agrees with the result of Proposition (1.3.5). ♦

In practice, we do not lose any actual information by replacing M o n by its closure M n since, as we will see shortly, the epipoles are always "distinguishable" in the joint image. In the following, we will talk about equations that "characterize point correspondences", referring to polynomial constraints that actually describe the joint image variety M n .

First properties of the joint image

By definition, the joint image M n is the closure of the image of the "joint-projection" map

P 3 \ {c 1 , . . . , c n } -→ P 2 × . . . × P 2 x -→ (P 1 (x), . . . , P n (x)). (1.15)
This map is usually injective, the only exception being when all of the pinholes are aligned (in particular, for n = 2 cameras): in this case, all points lying on the "baseline" spanned by the pinholes will have the same image. The inverse of (1.15), where it is well-defined, is the triangulation map M n P 3 . This map returns a space point given corresponding image points. Note that this is a rational map (i.e., it can be described using polynomial expressions) because it amounts to computing the intersection of visual rays. This implies that M n and P 3 are birationally equivalent (cf. Appendix B). Intuitively, this says that M n is "almost" a copy of P 3 , embedded in P 2 × . . . × P 2 . This immediately implies that the joint image is irreducible (it is not the union of proper subvarieties) and has dimension 3.

The following proposition deals with the singularities of M n . The proof is technical, and deferred to the end of the chapter. Proposition 1.3.7 (Singularities of the joint image variety). When the camera pinholes are not collinear, M n is smooth. When they are collinear (in particular for n = 2 views), then M n has a unique singular point given by the n-tuple of epipoles (e 1 , . . . , e n ).

This result shows how epipoles represent distinguished points in M n . Indeed, for any choice of k ≥ 2 cameras, there exists a natural surjective projection map M n → M k (P i 1 , . . . , P i k ), that selects subsets of the image points (note that this is also a birational equivalence). Hence, a pair of epipoles can be seen as the singular points of the projection of M n to the joint image defined by the corresponding pair of cameras.

Algebraic characterizations of the joint image

The joint image variety is a projective variety in (P 2 ) n . This means that it can be characterized by a set of "multi-homogeneous" polynomials in R[x 1 , y 1 , z 1 , . . . , x n , y n , z n ], where each triple of variables x i , y i , z i (i = 1, . . . , n) are coordinates associated with an image. A complete description of the ideal of all polynomials that vanish on the joint image is given by Aholt et al. in [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF] (see Theorem 1.3.10). Their derivation of the generators for this ideal is based on a general strategy for recovering multi-view constraints that was discussed by Heyden and Åstrom in [START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF][START_REF] Heyden | Tensorial properties of multiple view constraints[END_REF], and is also presented in [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]Chapter 17]. We will now review this approach, emphasizing the important differences between the following types of algebraic characterizations:

• An ideal-theoretic characterization of M n provides a set of polynomials that generate the ideal

I(M n ) in the ring R[x 1 , y 1 , z 1 , . . . , x n , y n , z n ].
This means that all polynomials that vanish on M n can be obtained as algebraic combinations of the given generators. A further requirement is that the polynomials form a Gröbner basis, which is not true for an arbitrary set of generators.

• A set-theoretic characterization of M n provides a set of polynomials p 1 , . . . , p n such that

M n = {(u 1 , . . . , u n ) ∈ (P 2 ) n | p i (u 1 , . . . , u n ) = 0, i = 1, . . . , n} ⊂ (P 2 ) n . (1.16)
In other words, the variety M n is cut out by the conditions {p i = 0} for i = 1, . . . , n. By definition, the polynomials p 1 , . . . , p n will belong to the ideal I(M n ), so a settheoretic characterization may require fewer conditions than those necessary to generate the whole ideal. Over C, the relationship between set-theoretic and idealtheoretic descriptions is completely governed by Hilbert's Nullstallensatz (for our purposes, in its multi-homogeneous form). See Appendix B.

• An "indirect" or "weak" characterization of M n is for us a family of polynomials p 1 , . . . , p k such that the set

M n = {(u 1 , . . . , u n ) ∈ (P 2 ) n | p i (u 1 , . . . , u n ) = 0, i = 1, . . . , k} ⊂ (P 2 ) n (1.17)
contains M n , possibly strictly, but that allows us nonetheless to recover M n in some indirect way. In other words, there must be only "feasible" M n inside the larger set M n defined by (1.17). This type of description is in general the most compact, since fewer conditions are usually necessary to characterize a larger set.

While from a purely algebraic standpoint the description at the level of ideals is most useful, for applications it is natural to focus on set-theoretic characterizations, or even more compact descriptions that encode same information in a more efficient (albeit indirect) way. We will see that these characterizations can be used to recover camera geometry from image data.

The Joint Image Variety

We now begin our derivation of multi-view constraints. Let (u 1 , . . . , u n ) in (P 2 ) n be a point correspondence for the cameras P 1 , . . . , P n . After fixing coordinate representatives u i = [u i ] and P i = [P i ], we define the 3n × (n + 4) matrix

U(u 1 , . . . , u n ) =       P 1 u 1 0 . . . 0 P 2 0 u 2 . . . 0 . . . . . . . . . . . . . . . P n 0 0 . . . u n       . (1.18)
It is clear that a necessary condition for the n-tuple (u 1 , . . . , u n ) to form a correspondence is that U(u 1 , . . . , u n ) be rank deficient. Indeed, if

P i (x) = u i with x = [x],
then there exists a non-zero vector [x; λ 1 ; . . . ; λ n ] in the null space of the matrix. Moreover, the maximal minors of U(u 1 , . . . , u n ) provide a set-theoretic characterization of the joint image:

M n = {(u 1 , . . . , u n ) ∈ (P 2 ) n | U(u 1 , . . . , u n ) is not full rank}. (1.19)
In the following it will be useful to say that a polynomial in R[x 1 , y 1 , z 1 , . . . , x n , y n , z n ] is k-linear (for k ≤ n) when it involves only k triples of variables x i , y i , z i and is linear in each triple that appears. In particular, an n-linear polynomial will be simply referred to as multilinear. For k = 2, 3, 4 we will also use "bilinear", "trilinear", and "quadrilinear", respectively.

The maximal minors U(u 1 , . . . , u n ) provide a characterization of the joint image based on multilinear (n-linear) polynomials. On the other hand, conditions of lower degree can be obtained from the maximal minors of a matrix as in (1.18) for subsets of the original matrix. Indeed, if (u 1 , . . . , u n ) is a correspondence for P 1 , . . . , P n then any subset (u α 1 , . . . , u α k ) will be a correspondence for P α 1 , . . . , P α k .

Definition 1.3.8. We indicate with S k the set of k-linear polynomials given by the maximal minors of

U α =       P α 1 u α 1 0 . . . 0 P α 2 0 u α 2 . . . 0 . . . . . . . . . . . . . . . P α k 0 0 . . . u α k      
, with α = {α 1 , . . . , α k } ranging among all subsets of size k of {1, . . . , n}.

For k = 2, the matrix U α from Definition 1.3.8 is square (of size 6 × 6), so there is only one bilinear constraint b α 1 α 2 for every pair of cameras P α 1 , P α 2 . The polynomial b α 1 α 2 in fact the well-known epipolar constraint, which is usually identified with the fundamental matrix that encodes its coefficients [START_REF] Hartley | Multiple view geometry in computer vision[END_REF].

For any k, the polynomials in S k will vanish on M n . However, it is easy to see that we can restrict ourselves to sets with k ≤ 4.

Pinhole Cameras and their Multi-View Geometry Proposition 1.3.9. Every polynomial in S n is of the form m • p where m is a monomial factor and p belongs to S k with k ≤ 4. From this we deduce that the bilinear, trilinear, and quadrilinar polynomials in S 2 , S 3 , S 4 are sufficient to characterize M n set-theoretically.

Proof. The result follows from the fact that a non-vanishing maximal minor of U requires choosing n+4 rows, with at least one row associated with each camera: this distinguishes k cameras with 2 ≤ k ≤ 4 for which more than one row is chosen. The monomial factors can be removed from the constraints since each k-linearity is multiplied by sets of monomials that cannot vanish simultaneously.

One of the main results in [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF] shows that in fact the polynomials in S 2 , S 3 , S 4 describe the joint image in a very strong sense.

Theorem 1.3.10 (Aholt et al. [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF]). The polynomials in S 2 , S 3 , S 4 form a universal Gröbner basis for the ideal I(M n ) (this means that they constitute a Gröbner basis of I(M n ) under all possible monomial orderings).

We will see shortly that this characterization based on bilinear, trilinear and quadrilinear constraints is actually very redundant. In fact, we will argue that the quadrilinear constraints are always completely unnecessary (a well known fact [START_REF] Faugeras | On the geometry and algebra of the point and line correspondences between n images[END_REF]), and, more importantly, much fewer bilinear and trilinear conditions can actually be used (Section 1.4).

We conclude this discussion with a result that provides the dimension of the vector space of all multilinear (n-linear) relations that vanish on M n . The quantity can also also be deduced from [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF]Theorem 3.6] that gives the "multigraded Hilbert function" for the ideal I(M n ).

Proposition 1.3.11. The multilinear polynomials that vanish on M n form a vector space of dimension

d n = 3 n -n+3 3 + n.
Proof sketch. It is sufficient to compute the dimension of the vector space generated by the initial terms associated to the multilinear constraints. Since the maximal minors of U(u 1 , . . . , u n ) defined in (1.18) form a Gröbner basis, the result follows from a counting argument involving the associated initial monomials.

For example, let us point out that d 2 = 1 (the epipolar constraint is the only bilinear relation for two views), d 3 = 10 (for three views there are always 10 linearly independent trilinearities), and d 4 = 50; these facts can also be verified computationally using Gröbner bases.

Remark 1.3.12. Proposition 1.3.11 is useful to explain the geometry of M n under the Segre embedding. We recall first that the Segre embedding is a map P 2 × . . . × P 2 → P 3 n -1 that identifies an n-tuple of projective points with an n-dimensional rank-one tensor defined up to scale (see also Appendix B): for example, we can identify a pair

The Joint Image Variety

(u 1 , u 2 ) = ([u 1 ], [u 2 ]
) with the rank-one 3 × 3 matrix u T 1 u 2 , and we can do the same for more factors. The image of (P 2 ) n under the Segre map is the Segre variety Seg n (P 2 ). This is a 2n-dimensional variety inside P 3 n -1 , and represents the set of rank-one tensors defined up to scale. We can now view M n as a projective sub-variety of Seg n (P 2 ). Since multilinear relations in (P 2 ) n correspond to linear ones in P 3 n -1 , we see from (1.19) that M n can be identified with Seg n (P 2 ) intersected with a linear space. The co-dimension of this linear space is the number d n provided by Proposition 1.3.11.

Camera configurations and the joint image

We interrupt for a moment our study of the joint image to introduce the fundamental notion of camera configuration. In the following, we will say that two sets of n cameras P 1 , . . . , P n and Q 1 , . . . , Q n are projectively equivalent if there exists a single projective transformation T such that

P i = Q i T (so if T = [T] with T in GL(4, R), then P i = α i Q i T for non-zero constants α i ).
Definition 1.3.13. The space of camera configurations C n is the set of n-tuples of cameras up to projective equivalence.

In other words, if we write P as the space of all pinhole cameras (which we may view as an open subset of P 11 ), then C n = P n /GL(4, R), where GL(4, R) acts on P n by ([P 1 ], . . . , [P n ])T = ([P 1 T], . . . , [P n T]). It will also be useful to say that a camera configuration of P 1 , . . . , P n is non-degenerate if the pinholes of P 1 , . . . , P n are all distinct.

The following important lemma refers to the definitions from Section 1.2.3. Lemma 1.3.14. Given two cameras P 1 , P 2 with distinct pinholes c 1 , c 2 , we consider the groups K c 1 = Stab GL(4,R) (P 1 ) and K c 2 = Stab GL(4,R) (P 2 ) inside GL(4, R) (recall that these only depend on the pinholes c 1 , c 2 ). Then we have that

K c 1 ∩ K c 2 = {kI 4 | k ∈ R \ {0}}.
In particular, projective transformations act transitively on non-degenerate camera configurations (or, more generally, whenever at least two pinholes are distinct).

Proof. Without loss of generality we may assume that c 1 = [0, 0, 0, 1] and c 2 = [0, 0, 1, 0]. Given (1.2), the groups K c 1 and K c 2 can be described as

K c 1 =               a 0 0 0 0 a 0 0 0 0 a 0 b c d e      ∈ GL(4, R)          , K c 2 =               a 0 0 0 0 a 0 0 b c d e 0 0 0 a      ∈ GL(4, R)          . The fact that K c 1 ∩ K c 2 = {kI 4 , k ∈ R} can now be verified directly.

Pinhole Cameras and their Multi-View Geometry

A consequence of this lemma is that the space of camera configurations C n can be viewed as a a homogeneous space (a manifold) of dimension 11n -15. We now discuss the relationship between camera configurations and the joint image.

) = M (Q 1 , . . . , Q n ) in (P 2 ) n .
In particular, we can define a map J :

C n → {varieties in (P 2 ) n }, (1.20)
associating a camera configuration with a joint image.

Proof. This is obvious because the joint image is independent of the choice of coordinates in P 3 : if T is a projective transformation of P 3 , then P 1 , . . . , P n and P 1 T, . . . , P n T have the same set of point correspondences.

The following important result can be viewed as the theoretical basis for multi-view reconstruction: it states that point correspondences are sufficient for determining a unique camera configuration. This fact may not seem surprising, since it is well known that multifocal tensors can be estimated from point correspondences and they determine a unique configuration. However, it is interesting that this result would not be true if we had considered projections P 2 P 1 rather than cameras P 3 P 2 (see Theorem 3.3.7 and [START_REF] Richard | Reconstruction from projections using grassmann tensors[END_REF]). We also also mention [START_REF] Ito | Projective reconstruction in algebraic vision[END_REF], that gives a proof of this fact in a more a general setting using algebraic geometry. In our proof of this result, we make use of the following lemma. Lemma 1.3.17. Let x 1 , . . . , x 6 and u 1 , . . . , u 6 be six points in P 3 and P 2 . Assuming that all these points are in general position, then there exists at most one camera P such that P (x i ) = u i for i = 1, . . . , 6.

Proof of Lemma. This is the standard set-up for "camera resectioning": every match from 3D to 2D provides two new independent conditions on the camera parameters (the open set P ⊂ P 11 ). Indeed, writing P = [P],

x i = [x i ], u i = [u i ], then we have that P (x i ) = u i is equivalent to rank[Px i |u i ] = 1,
yielding two linear conditions on the coefficients of P.

Proof of Theorem. We first consider the case of non-degenerate configurations, which is much more important in practice. We will then point out that the result holds for degenerate configurations as well.

The statement of the theorem is well known for n = 2 cameras with distinct pinholes, since a 3 × 3 fundamental matrix can be used to identify correspondences as well as a

The Joint Image Variety camera configuration [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. Let us now consider two sets of n ≥ 3 cameras P 1 , . . . , P n and Q 1 , . . . , Q n with distinct pinholes such that M (P 1 , . . . ,

P n ) = M (Q 1 , . . . , Q n ). Since this clearly implies M (P 1 , P 2 ) = M (Q 1 , Q 2 )
, we can use the statement for n = 2 to say that P 1 , P 2 and Q 1 , Q 2 are projectively equivalent. Lemma 1.3.14 now guarantees that there is only one projective transformation that maps the pair P 1 , P 2 to Q 1 , Q 2 : thus, if we assume that P 1 = Q 1 and P 2 = Q 2 , we need to show that P j = Q j for all j = 3, . . . , n. Since we can prove this by considering one pair P j , Q j at the time, we will assume n = 3.

We now fix six points x 1 , . . . , x 6 in P 3 in general position, and write u 11 , . . . , u 16 and u 21 , . . . , u 26 for points in P 2 such that P i (x j ) = u ij for i = 1, 2 and j = 1, . . . , 6. Since M (P 1 , P 2 ) is birationally equivalent to M 3 = M (P 1 , P 2 , P 3 ) = M (P 1 , P 2 , Q 3 ), for each for each i = 1, . . . , 6, we expect to find only one triple (u 1j , u 2j , v j ) in M 3 . Now we observe that necessarily P 3 (x j ) = Q 3 (x j ) = v j for j = 1, . . . , 6. Using Lemma 1.3.17, we conclude that

P 3 = Q 3 .
The idea of the previous argument is that we can use two cameras P 1 , P 2 to "reproject" image points from M (P 1 , P 2 ) to M (P 1 , P 2 , P 3 , . . . , P n ), and this determines all the cameras P 3 , . . . , P n . This strategy applies whenever there are at least two cameras with distinct pinholes. On the other hand, the result also holds when all cameras P 1 , . . . , P n have the same pinhole: in fact, in this case there is a unique set of projective transformations T 1i , i = 3, . . . , n, of P 2 such that P 1 = T 1i P i (see Section 1.2.3). It is easy to see that these transformations uniquely determine the joint image as well as the camera configuration.

We conclude this section by noting that Theorem 1.3.16 implies that the triangulation map T : M (P 1 , . . . , P n ) P 3 defined in Section 1.3.2 is unique up to composing with projective transformations in P 3 . In particular, there is a unique "projective structure" on M (P 1 , . . . , P n ), and we may talk about projective properties (linear subspaces, linear independence, etc.) inside the image space, based on the geometry of the corresponding objects in P 3 .

Bilinearities and trilinearities

We now resume our study of the different sets of multilinear constraints that can be used to describe the joint image (and thus, as we have just seen, a camera configuration). In particular, we revisit the characterization of M n from Theorem 1.3.10 by arguing that, in practice, the constraints from S 2 , S 3 , S 4 are not all necessary.

We first note that since P 2 × . . . × P 2 has dimension 2n and M n has dimension 3, one might hope to find a characterization of M n based on 2n -3 constraints. However, typically one cannot represent an algebraic set of codimension m as the intersection of m hypersurfaces (when this is possible, the set is called a "complete intersection").

Pinhole Cameras and their Multi-View Geometry

Interestingly, however, this number of constraints is often sufficient to recover M n as a distinguished component.

W = {b 12 = b 13 = b 23 = 0} = M 3 ∪ V t , (1.21)
where V t is the product of the three "trifocal lines" spanned by the epipoles [START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF]. Note that the joint image M 3 appears as an irreducible component, which means that M 3 (and all its defining equations) can be recovered indirectly. In fact, it is well known that in this case the three epipolar constraints (or the fundamental matrices) are sufficient to recover the camera configuration [START_REF] Faugeras | On the geometry and algebra of the point and line correspondences between n images[END_REF][START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. ♦

Keeping this example in mind, and coming back to a point raised in Section 1.3.3, we will say that a weak or indirect characterization of the joint image is provided by a set of polynomials that define a set W in (P2 ) n such that there is a unique "feasible" joint image variety M n inside W. 2 In light of Theorem 1.3.16, this is equivalently a set of multi-view constraints that uniquely determine a camera configuration. In contrast, a set of polynomials that cut out the joint image in the usual (set-theoretic) sense will be sometimes called a strong characterization.

The simplest way for a set of constraints to provide a weak characterization of the joint image is if the associated algebraic set W contains M n as an irreducible component of maximal dimension, as in Example 1.3.18. Note that in many practical settings, this kind of weak description can be used in place of a strong one, since it can characterize correspondences away from the spurious components (possibly using many fewer equations). In fact, we believe that some of the previously considered "necessary and sufficient conditions" for multi-view correspondence were unknowingly weak characterizations, that were satisfied by some special extraneous n-tuples of image points. For example, we will point out in Section 1.5.2 that the nine "point-point-point" conditions encoded in a trifocal tensor [START_REF] Hartley | Multiple view geometry in computer vision[END_REF] are not a complete (strong) characterization of the joint image. It is also possible to have weak characterizations of the joint image in which the associated algebraic set W is higher dimensional: we will discuss many examples in Section 1.4.

In [START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF], Heyden and Åström point out that for n ≥ 4, assuming all the pinholes to be in general position, the bilinear constraints are already sufficient to generate the trilinear and quadrilinear ones. We now extend this result, clarifying the role of the different families of constraints for all possible camera configurations. The proof of the following proposition amounts to reducing to the case n = 4 and verifying all the relations computationally. Details are given at the end of the chapter. We should observe that the first point of Theorem 1.3.19 is well known (see for example [START_REF] Faugeras | On the geometry and algebra of the point and line correspondences between n images[END_REF]), and indeed reconstruction methods are generally only based on epipolar and trifocal constraints. The second point can be deduced from the analysis for four points in general position in [START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF], although the authors do not point out this general fact. The last point is, to the best of our knowledge, new, at least in such a general formulation: it shows that trilinear relations are essential only if all pinholes are collinear, since otherwise the epipolar relations are sufficient to determine the camera configuration and, indirectly, point correspondences. See Table 1.1.

Epipolar Constraints and Viewing Graphs

Epipolar Constraints and Viewing Graphs

In Theorem 1.3.19, we considered complete families of bilinear and trilinear constraints. These sets of conditions are redundant in general: for example, it has been observed in [START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF] that for five cameras in general position, correspondences can be characterized using 9 bilinear constraints instead of the complete set of 10. In [START_REF] Trager | The joint image handbook[END_REF], we observed the following more general fact (see Figure 1.3). In general, it is convenient to represent sets of fundamental matrices between pairs of views as edges of a graph, as shown in Figure 1.3. This type of graph was called the viewing graph by Levi and Werman [START_REF] Levi | The viewing graph[END_REF]. In [START_REF] Trager | The joint image handbook[END_REF], we pointed out that there are generally many viewing graphs with 2n -3 edges that represent weak characterizations of the joint image (although we did not use the term "viewing graph", since we were not aware of [START_REF] Levi | The viewing graph[END_REF] at the time). For example, if a graph is obtained from 3-cycle by adding vertices of degree two one at the time, then the fundamental matrices associated with its edges uniquely determine a camera configuration. We investigated similar questions in more detail in our follow-up work [START_REF] Trager | On the Solvability of Viewing Graphs[END_REF]. There we used the terminology of Levi and Werman, and studied characterizations of "solvable" viewing graphs, which represent sets of fundamental matrices that determine a unique camera configuration.

This section presents the results of [START_REF] Trager | On the Solvability of Viewing Graphs[END_REF]. After introducing viewing graphs and the notion of "solvability" in Section 1.4.1, we recall some basic results from [START_REF] Levi | The viewing graph[END_REF][START_REF] Rudi | Linear solvability in the viewing graph[END_REF] in Section 1.4.2. We observe in Section 1.4.3 that any solvable viewing graphs has at least e(n) = (11n -15)/7 edges, and prove that using this minimum number is always possible (Theorem 1.4.9). Note that e(n) < 2n -3 for n ≥ 5. We then present several new criteria for deciding whether or not a viewing graph is solvable. In particular, we describe a new necessary condition for solvability that is based on the number of edges and vertices of subgraphs (Theorem 1.4.12), as well as a sufficient condition based on

Epipolar Constraints and Viewing Graphs

"moves" for adding new edges to a graph (Theorem 1.4.14). In Section 1.4.6 we describe an algebraic formulation for solvability that in principle can always be used to determine whether any viewing graph is solvable. This method is computationally challenging for larger graphs, but we also introduce a much more practical linear test, that can be used to verify whether a viewing graph identifies a finite number of camera configurations. Finally, using an implementation of all the proposed methods, we analyze in Section 1.4.7 the solvability of all minimal viewing graphs with at most 9 vertices, and discuss some interesting examples.

The viewing graph

The viewing graph is a graph in which vertices correspond to cameras, and edges represent fundamental matrices between them. More precisely, if G = (V G , E G ) is an undirected graph with n vertices, and P 1 , . . . , P n are projective cameras, we write

F G (P 1 , . . . , P n ) = {F ij = F (P i , P j ) | (i, j) ∈ E G }, (1.22)
for the set of fundamental matrices defined by the edges of G. We say that the the set

F G (P 1 , . . . , P n ) is solvable if F G (P 1 , . . . , P n ) = F G (P
) | (i, j) ∈ E G }.
This viewpoint also suggests the idea that, given any graph G, we can define a "solvable closure" G, as the graph obtained from G by adding edges corresponding to fundamental matrices that can be deduced generically from F G (P 1 , . . . , P n ). Hence, a graph is solvable if and only if its closure is a complete graph. We will return to this point in Section 1.4.5.

Simple criteria

We begin by recalling two necessary conditions for solvability that were shown in [START_REF] Levi | The viewing graph[END_REF]. These provide simple criteria to show that a viewing graph is not solvable.

Proposition 1.4.6. [START_REF] Levi | The viewing graph[END_REF] If a viewing graph with n > 3 vertices is solvable, then:

1. All vertices have degree at least 2.

No two adjacent vertices have degree 2.

We extend this result with the following necessary condition (which implies the first point in the previous statement).

c i v T 1 ] and T 2 = [I 4 + α 2 c i v T 2 ]
. These transformations fix the camera P i . If we apply T 1 to all cameras in G 1 and T 2 to all cameras in G 2 , while leaving P i fixed, we obtain a different camera configuration that gives rise to the same set of fundamental matrices as P 1 , . . . , P n for all edges in G.

We also recall a result from [START_REF] Rudi | Linear solvability in the viewing graph[END_REF] which will be used in the next section. Proposition 1.4.8. [START_REF] Rudi | Linear solvability in the viewing graph[END_REF] If G 1 and G 2 are solvable viewing graphs, then the graph G obtained by identifying two vertices from G 1 and with two from G 2 is solvable.

Note that if both pairs of vertices in the previous statement are connected by edges in G 1 and G 2 , then these two edges will automatically be identified in G.

How many fundamental matrices?

We now ask ourselves what is the minimal number of edges that a graph must have to be solvable (or, equivalently, how many fundamental matrices are required to recover a camera configuration). Since a single epipolar relation provides at most 7 constraints in the (11n -15)-dimensional space camera configuration, we deduce that any solvable graph must have at least e(n) = (11n-15)/7) edges. This fact was previously observed in [START_REF] Rudi | Linear solvability in the viewing graph[END_REF]Theorem 2]. However, compared to [START_REF] Rudi | Linear solvability in the viewing graph[END_REF], we show here that this bound is tight, i.e., that there always exists a solvable graph with e = e(n) edges. Concretely, this means that, for n generic views, there is always a way of recovering the corresponding camera configuration using e(n) fundamental matrices. Theorem 1.4.9. The minimum number of edges of a solvable viewing graph with n ≥ 2 views is e(n) = 11n -15 7 .

Proof. For n ≤ 9, examples of solvable viewing graphs with e(n) edges are illustrated Figure 1.4. The solvability of these graphs will be shown in Section 1.4.5 (all but one of these also appear in [START_REF] Levi | The viewing graph[END_REF]). In particular, let G 0 be a solvable viewing graph with 9 vertices and 12 edges. Using Proposition 1.4.

Pinhole Cameras and their Multi-View Geometry

To obtain a solvable viewing graph with n vertices and e(n) edges, we start from a solvable graph with r vertices and e(r) edges, and repeat the gluing construction described above q times. The resulting graph is solvable and has the desired number of vertices and edges.

Remark 1.4.10. In order to recover projection matrices for n views, it is common to use 2n -3 fundamental matrices (see for example [START_REF] Heyden | Tensorial properties of multiple view constraints[END_REF]Sec.4.4]). In fact, as noted at the beginning of this section, a large class of solvable viewing graphs can be defined, starting for example from a 3-cycle, by adding vertices of degree two, one at the time: this always gives a total of 2n -3 edges. For this type of viewing graphs it is possible to recover projection matrices incrementally, using a pair of fundamental matrices for each camera.

It is probably often erroneously believed that 2n -3 is actually the minimal number of epipolar constraints required: at least, we thought this to be true at first. For us, the confusion was due to the fact that the joint image has dimension three (or codimension 2n -3) in (P 2 ) n . This means means that we expect 2n -3 conditions to be necessary to cut out generically the set of image correspondences among n views. The existence of solvable viewing graphs with less than 2n -3 epipolar constraints means that, for these graphs, the associated epipolar constraints cut out a set W in (P

Constraints on fundamental matrices

Closely related to the solvability of viewing graphs is the problem of describing compatibility of fundamental matrices. Indeed, given a solvable graph G, it is not true in general that any set of fundamental matrices can be assigned to the edges of G, since fundamental matrices must satisfy some feasibility constraints in order to correspond to an actual camera configuration. For example, it is well known that the fundamental matrices F In most practical situations fundamental matrices are estimated separately, so these constraints need to be taken into account [START_REF] Sweeney | Optimizing the viewing graph for structure-from-motion[END_REF]. However, it is sometimes incorrectly stated that compatibility for any set of fundamental matrices only arises from triples and equations of the form (1.23) [157, Theorem 1] [177, Definition 1]. While it is true that for a complete set of n 2 fundamental matrices triple-wise compatibility is sufficient to guarantee global compatibility, for smaller sets of fundamental matrices other types of constraints will be necessary. For example, there are many solvable viewing graphs with no three-cycles (e.g., the graph in Figure 1.4 with n = 5), however the fundamental matrices cannot be unconstrained if 7e(n) > 11n -15, which always true unless e = 2 modulo 9 (cf. Table 1.2).

More formally, we can consider the set X of compatible fundamental matrices between all pairs of n views, so that X ⊂ (P 8 ) N where N = n 2 . Since each compatible N -tuple is associated with a camera configuration, we see that X has dimension 11n -15. Given a viewing graph G with n views, we write X G ⊂ (P 8 ) e for the projection of X onto the factors in (P 8 ) N corresponding to the edges of G. The set X G thus represents compatible fundamental matrices for pairs of views associated with the edges of G. The following result follows from dimensionality arguments.

Proposition 1.4.11. If G is solvable with n vertices, X G has dimension 11n -15.
If X G has dimension 11n -15, then the fundamental matrices assigned to the edges of G must satisfy d(n, e) = 7e -11n + 15 constraints3 . This also means that the minimum number of constraints on the fundamental matrices associated with a solvable graph is Proof. Using the same notation as above, we note that that X G is a projection of X G onto e factors of (P 8 ) e : this implies dim

d(n) = d(n, e(n)) (see
X G + 7(e -e ) ≥ dim X G , or 7e -dim X G ≤ 7e -dim X G . Since dim X G ≤ 11n -15 and dim X G = 11n -15 (because G is solvable), we obtain 7e -11n + 15 ≤ 7e -dim X G ≤ 7e -dim X G = 7e -11n + 15.
For the second statement, we consider the graph

G = ( i V G i , i E G i ). Since the edges of G i are disjoint, we have dim X G ≤ k i=1 dim X G i ≤ k i=1 (11n i -15),
and e = k i e i . The result follows again from 7e -dim X G ≤ 7e -dim X G . Example 1.4.13. In [START_REF] Levi | The viewing graph[END_REF], Levi and Werman observed that all viewing graphs of the form shown in Figure 1.5 are not solvable. This can be easily deduced from Theorem 1.4.12. Indeed, for a graph G of this form, the subgraphs G 1 , G 2 , G 3 , G 4 have disjoint edges, however we have (using the same notation as in the proof of Theorem 1.4.12)

4 i=1 d(n i , e i ) = d(n, e) -4 × 11 + 3 × 15 > d(n, e).
According to Theorem 1.4.12 this means that G is not solvable.

♦

G 1 G 2 G 3 G 4

Constructive approach for verifying solvability

Until now we have mainly discussed necessary conditions for solvability, which can be used to show that a given graph is not solvable. We next introduce a general strategy for proving that a graph is solvable. This method is not always guaranteed to work, but in practice it gives sufficient conditions for many of the graphs we tested (cf. Section 1.4.7).

Recall from the beginning of this section that we introduced the "viewing closure" G of G as the graph obtained by adding to G all edges corresponding to fundamental matrices that can be deduced from F G (P 1 , . . . , P n ). Our approach consists of a series of "moves" which describe valid ways to add new edges to a viewing graph. For this it is convenient to introduce a new type of edge in the graph, which keeps track of the fact that partial information about a fundamental matrix is available. More precisely:

• A solid (undirected) edge between vertices i and j means that the fundamental matrix between the views i and j is fixed (as before).

• A directed dashed edge (for short, a dashed arrow) between vertices i and j means that the i-th epipole in the image j is fixed.

As these definitions suggest, a solid edge also counts as a dashed double-arrow, but the converse is not true. We next introduce three basic "moves" (see Figure 1 In practice, the three moves can be applied cyclically until no new edges can be added (it is also easy to argue the order is irrelevant, because we are simply annotating information that is always deducible from the graph). Finally, we note that all three moves are constructive and linear, meaning they actually provide a practical strategy for computing all fundamental matrices: it is enough to transfer epipoles appropriately, and use them to impose linear conditions on the unknown fundamental matrices.

Algebraic tests for solvability and finite solvability

Given a viewing graph G, it is possible to write down a set of algebraic conditions that will in principle always determine whether G is solvable. One way to do this is by characterizing the set of projective transformations of P 3 that can be applied to all cameras without affecting any of the fundamental matrices represented by the edges of the viewing graph. More precisely, since every pair of vertices connected by an edge represents a projectively rigid pair of cameras, we assign a matrix g λ in GL(4, R) to each edge λ of the graph (so g λ describes a projective transformation applied to a pair of cameras). We then impose that matrices on adjacent edges act compatibly on the shared vertex/camera. If the edges λ and λ share a vertex i, then this compatibility can be written as

g λ g -1 λ = αI 4 + c i v T , (1.26)
where α is an arbitrary (nonzero) constant and v is an arbitrary vector: indeed, we are requiring that g λ g -1 λ belongs to the stabilizer group, that was given in (1.2). Thus, if G is a viewing graph with e edges and c 1 , . . . , c n are a set of pinholes, we consider the set of all compatible assignments of matrices:

T G (c 1 , . . . , c n ) = {(g λ , λ ∈ E G ) | (1.26) holds for all adjacent edges in G} ⊂ GL(4, R) e .
If G is solvable, then for general c 1 , . . . , c n the set T G (c 1 , . . . , c n ) will consist of e-tuples of matrices that are all scalar multiples of each other. This in fact means that the only way to act on all cameras without affecting the fixed fundamental matrices is to apply a single projective transformation.

By substituting random pinholes in (1.26), we can use these equations for T G (c 1 , . . . , c n ) as an algebraic test for verifying whether a viewing graph is solvable. This approach however is computationally very challenging, since it requires solving a non-linear algebraic system with a large number of variables. On the other hand, if we are only interested in the dimension of T G (c 1 , . . . , c n ), then we can use a much simpler strategy: noting that T G (c 1 , . . . , c n ) may be viewed as an algebraic group (it is a subgroup of GL(4, R) e ), it is sufficient to compute the dimension of its tangent space at any point, and in particular at the identity (i.e., the product of identity matrices). 4 An explicit representation of the tangent space of T G (c 1 , . . . , c n ) is provided by the following result (see the end of the chapter for a proof). Proposition 1.4.16. The tangent space of T G (c 1 , . . . , c n ) at the identity can be represented as the space of e-tuple of matrices (h λ , λ ∈ E G ) where each h λ is in R 4×4 (not necessary invertible), and with compatibility conditions of the form

h λ -h λ = αI 4 + c i v T , (1.27)
where α ∈ R \ {0} and v ∈ R 4 are arbitrary, and λ and λ share the vertex i.

When the pinholes have been fixed, the compatibility constraints (1.27) can be expressed as linear equations in the entries of the matrices h λ . These equations are obtained by eliminating the variables α and v from (1.27). The resulting conditions in terms of h λ , h λ , c i are rather simple, and listed explicitly at the end of the chapter. Using this approach, the dimension of T G (c 1 , . . . , c n ) is easy to determine: it is enough to fix the pinholes randomly, and compute the dimension of the induced linear system.

When T G (c 1 , . . . , c n ) has dimension d = 15 + e (which accounts for the group of projective transformations, and scale factors for each matrix g λ ), we deduce that there are at most a finite number of projectively inequivalent ways in which we can act on all the cameras without affecting the fixed fundamental matrices. In other words, the fundamental matrices associated with the edges of G determine at most a finite set of camera configurations (rather than a single configuration, which is our definition for solvability).

When this happens, we say that G is finite solvable. On the other hand, we were not able to find an example of a finite solvable graph that is provably not solvable, nor to find a proof that "finite solvability" implies "solvability". To our knowledge, whether a set of fundamental matrices can characterize a finite number of configurations, but more than a single one, is a question that has never been addressed.

Open Question: Is it possible for a viewing graph to be finite solvable without being solvable?

Our experiments show that this behavior does not occur for a small number of vertices, but we see no reason why this should be true for larger graphs. This is certainly an important issue that we hope to investigate in the future.

Experiments and examples

We have implemented and tested all of the discussed criteria and methods using the free mathematical software Sage [START_REF]SageMath, the Sage Mathematics Software System[END_REF], which includes symbolic algebra tools as well as built-in functions for manipulating graphs. We then analyzed solvability for all minimal viewing graphs with n ≤ 9 vertices and e(n) = (11n -15)/7 edges. The results are summarized in Table 1.3. For every pair (n, e(n)), we list the number of all nonisomorphic connected graphs of that size ("connected"), the number of graphs that satisfy the necessary condition from Theorem 1.4.12 ("candidates"), the number of those that satisfy the sufficient condition from Theorem 1.4.14 ("solvable with moves"), and the number of graphs that are finite solvable ("finite solvable"), using the linear method from 1.4.6. We see that Theorems 1.4.12 and 1.4.14 allow us to recover all minimal solvable graphs for n ≤ 7, since candidate graphs are always solvable with moves. On the other hand, for n = 8, and particularly for the unconstrained case n = 9, there are some graphs that we could not classify with those methods (although finite solvability was easy to verify in all cases). For the undecided graphs, we were sometimes able to prove solvability with the general algebraic method from Section 1.4. To prove this fact, one needs to keep track of more information, and record also when an epipole is constrained to a line (rather than only when an epipole is fixed, which is the purpose of dashed edges). 5Indeed, we note that:

1. The image of the pinhole 1 in the view 5 lies on the line l 51 defined by reprojection of the epipole e 41 from the view 4 to the view 5.

2. Similarly, the image of the pinhole 1 in the view 6 lies on the line l 61 defined by the reprojection of the epipole e 31 from the view 3 to the view 6.

3. Using the previous two observations, we deduce that the image of the pinhole 1 in the view 7 lies on the line l 71 obtained by transferring l 51 and l 61 to the view 7. This line is the projection from 7 of the intersection of the planes spanned by c 1 , c 4 , c 5 and c 1 , c 3 , c 6 .

4. The image of the pinhole 1 in the view 7 belongs to another line m 71 that is the reprojection of the epipole e 81 from the view 8 to the view 7.

5. For generic pinholes the lines m 71 and l 71 will be different, so their intersection determines the image of the pinhole 1 in the view 7.

This argument shows that we can draw a dashed arrow from the vertex 1 to the vertex 7. We can now prove that G is solvable using the following sequence of moves (which starts from the situation illustrated at the right of Figure 1.8). • Draw double dashed arrows (4,7) and (3,7) (move II).

• Draw a dashed arrow from 1 to 5 (move II for the four-cycle (1,4,5,7)).

• Make (3,5) solid (move III), and then also [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF][START_REF] Arrondo | Introduction to projective varieties[END_REF], and (4,7) and (3,7) (move I).

• Make (1,7) double dashed (move II for [START_REF] Abdeljaoued | Minors of Bezout matrices, subresultants and the parameterization of the degree of the polynomial greatest common divisor[END_REF][START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF][START_REF] Arrondo | A focus on focal surfaces[END_REF][START_REF] Åström | Generalised epipolar constraints[END_REF]).

• Make (1,3) solid (move III) and complete the graph using move I. ♦ Example 1.4.18. The graph shown in Figure 1.9 is the only viewing graph with n = 9 and e = 12 that is "candidate" but is not "finite solvable". The fact that it is not finite solvable can also deduced without computations. Indeed, any finite solvable viewing graph of this size cannot impose any constraints on the fundamental matrices associated with its edges (this is because d(9, 12) = 7 × 12 -11 × 9 + 15 = 0). However, the image of the pinhole 7 in the view 2 is over-constrained, because we can draw a dashed arrow 7 → 2 using move II for two distinct four-cycles ((7, 1, 2, 5) and [START_REF] Arrondo | A focus on focal surfaces[END_REF][START_REF] Aholt | The ideal of the trifocal variety[END_REF][START_REF] Edward | The plenoptic function and the elements of early vision[END_REF][START_REF] Vladimir | Singularities of smooth mappings[END_REF]). This implies that the fundamental matrices associated with the edges of the graph cannot be arbitrary. ♦ Example 1.4.19. The graph shown in Figure 1.10 is not "solvable with moves", however we can show that it is solvable: indeed, the general algebraic compatibility equations from Section 1.4.6 are in this case simple enough that they can be solved explicitly (see the end of the chapter for the computations). The fundamental matrices associated with Three-View Geometry and Multifocal Tensors the edges of this graph are unconstrained, which means that 12 arbitrary fundamental matrices can be used to recover a unique configuration of 9 cameras. ♦

Three-View Geometry and Multifocal Tensors

We now return to multi-view constraints, focusing on some aspects that we did not address in our general discussion in Section 1.3. In particular, we describe in Section 1.5.1 weak and strong characterizations of the joint image for n = 3 views. We also briefly summarize a more geometric viewpoint on trilinearities, which we used in [START_REF] Trager | Trinocular Geometry Revisited[END_REF]. In Section 1.5.2, we discuss the connection between multifocal tensors and the joint image.

Our approach is based on a more general theory of "multi-graded Chow forms" that we introduced in [START_REF] Osserman | Multigraded Cayley-Chow forms[END_REF], and that we briefly sketch here. We also make the observation that the nine "point-point-point" trilinearities encoded in the trifocal tensor provide a weak and not a strong characterization of point correspondences (Proposition 1.5.7). We conclude in Section 1.5.3 with some experiments from [START_REF] Trager | Trinocular Geometry Revisited[END_REF], where we used bilinearities and trilinearities to define a simple "trifocal-epipolar error" that can be minimized to estimate camera parameters for three views. It is interesting to note that a similar conclusion can be reached using a purely geometric arguments. In the following, a transversal to some family of lines is a line intersecting every element of this family.

Trilinearities revisited

Proposition 1.5.2. A necessary and sufficient condition for three lines to converge is that they be pairwise coplanar, and that they admit a transversal not contained in the planes defined by any two of them.

Proposition 1.5.2 follows from the observation that if three lines are pairwise coplanar then exactly one of the following is true: 1) the lines are not all coplanar and intersect in one point; 2) they are all coplanar and intersect in one point; 3) they are coplanar, and intersect pairwise in three different points. Since only in cases 1) and 2) do the three lines admit transversals not contained in the planes defined by any two of them, Proposition 1.5.2 follows.

Algebraically, the condition for viewing lines to be pairwise coplanar is expressed (in terms of image coordinates) by the epipolar constraints. The existence of a transversal through a given point, on the other hand, can be imposed using a special class of trilinear polynomials. To show this, let us consider three lines 1 , 2 , 3 associated with Plücker vectors p = (p ij ), q = (q ij ), r = (r ij ) respectively. For any i = 1, 2, 3, 4, we write p î for the subvector of p that does not contain Plücker coordinates with index i (for example, p 1 = (p 23 , p 24 , p 34 )) and similarly for q and r. We now define

t i = det[ p î | q î | r î ], i = 1, 2 , 3, 4.
(1.28)

In the following, e 1 , e 2 , e 3 , e 4 indicate the standard fundamental points in P 3 .

Lemma 1.5.3. Given some integer j in {1, 2, 3, 4}, a necessary and sufficient condition for 1 , 2 , and 3 to admit a transversal passing through e j is that t j = 0.

Proof. A necessary and sufficient condition for a line with Plücker coordinates s = (s ij ) to pass through e j is that s ĵ = 0. Writing s j for the subvector of s that contains only Plücker coordinates with an index i (i.e., the complement of s ĵ ), then the line through e j given by s = (s ij ) will intersect the lines 1 , 2 , 3 if and only if s j • p ĵ = 0, s j • q ĵ = 0 and s j • r ĵ = 0. This is equivalent to t j = 0.

Three-View Geometry and Multifocal Tensors

If we consider now a triple of image points u 1 , u 2 , u 3 for three cameras P 1 , P 2 , P 3 , then the corresponding viewing lines 1 , 2 , 3 defined by i = P -1 i (u i ) have Plücker coordinates that are linear expressions in the coordinates of the points (cf. Section 1.2.5). We conclude that the existence of a line transversal to the viewing rays and passing through one of the fundamental points can be expressed as trilinear relation t i (u 1 , u 2 , u 3 ) in image coordinates.

We next explain the relationship between these special trilinearities and the trilinear polynomials in the set S 3 from Definition 1.3.8. Given three cameras P 1 , P 2 , P 3 , we fix a point p = [p] in P 3 , different from the pinholes. If u 1 , u 2 , u 3 are three image points with u i = [u i ], we define the constraint

t p (u 1 , u 2 , u 3 ) = det    P 1 u 1 0 0 0 0 P 2 0 u 2 0 P 2 p 0 P 3 0 0 u 3 0 P 3 p    = det    P 1 u 1 0 0 0 P 1 p P 2 0 u 2 0 0 0 P 3 0 0 u 3 P 3 p 0    = det    P 1 u 1 0 0 P 1 p 0 P 2 0 u 2 0 0 P 2 p P 3 0 0 u 3 0 0    .
(

The equalities between the different expressions follow from the multilinearity of the determinant. The following result shows that this constraint corresponds to the trilinearities from (1.28), where the role of the basis points e i is replaced by the point p.

Proposition 1.5.4. A necessary and sufficient condition for three image points u 1 , u 2 , u 3 to satisfy t p (u 1 , u 2 , u 3 ) = 0 as defined in (1.29), is that the associated visual rays admit a transversal passing through p.

Proof. An element [q, λ 1 , λ 2 , λ 3 , µ 1 , µ 2 ] in the null space of

   P 1 u 1 0 0 0 0 P 2 0 u 2 0 P 2 p 0 P 3 0 0 u 3 0 P 3 p    (1.30)
determines a point q = [q] in P 3 which projects onto u 1 and on the lines joining u 2 and p 2 = P 2 (p) in the second image and u 3 and p 3 = P 3 (p) in the third image. This implies that the line = p ∨ q is a transversal to the three visual rays. Conversely, it is easy to see that a transversal through p guarantees that (1.30) has a non-trivial null space.

Pinhole Cameras and their Multi-View Geometry

The formulation in (1.29) shows that t p (u 1 , u 2 , u 3 ) = 0 is a linear combination of the trilinearities in S 3 .

Multifocal tensors

We now turn to multifocal tensors. We recall that the study of three-view and four-view geometry is traditionally based on trifocal and quadrifocal tensors. These are respectively tensors of size 3 × 3 × 3 and 3 × 3 × 3 × 3 which:

1. describe matching image features for 3 or 4 views, and 2. uniquely characterize a camera configuration.

For these characteristics, trifocal and quadrifocal tensors are similar to the fundamental matrix for two views. In particular, these tensors can be estimated linearly from image data in order to recover camera parameters. On the other hand, while the fundamental matrix characterizes point-point correspondences in two images, trifocal and quadrifocal tensors describe point-line-line and line-line-line-line matches, respectively. It is also known that analogous tensors do not exist for more than four views. We refer to [START_REF] Hartley | Multiple view geometry in computer vision[END_REF] for more details.

We now propose a new framework for understanding multifocal tensors. This viewpoint helps illustrate the basic properties of these tensors, and emphasizes their relationship with the joint image variety. We will argue that multifocal tensors can be seen as an instance of a generalized version of the Chow form for an algebraic variety. We first recall some relevant definitions (see also Appendix B for basics on projective varieties).

Chow forms. Let X be a projective variety of dimension r in P n . To characterize X, we need in general at least n -r conditions in the coordinates of P n (over C, this is strictly necessary). On the other hand, when the codim X = n -r is greater than one, we can use a Chow form to uniquely determine X using only one equation for appropriate coordinates. To see this, recall first that that from the geometric definition of dimension, we know that a generic linear space of codimension α = r + 1 will not intersect X. The set of all linear spaces of codimension α which do meet X has the following properties:

1. It forms a hypersurface Z X in the Grassmannian Gr(n -α, P n ), and can be characterized by a single equation in Plücker coordinates F X = 0 (in addition to the Plücker relations).

2. The degree of F X is the same as the degree of X 3. Given the F X , it is possible to to recover the variety X. The idea here is that a point x in P n will belong to X if and only if all (n -α)-spaces passing through x meet X.

Three-View Geometry and Multifocal Tensors

The polynomial form F X associated with X is called the Chow form of X. For proofs, we refer the reader to [START_REF] Israel M Gelfand | Discriminants, resultants, and multidimensional determinants[END_REF].

As a simple example, we consider the standard twisted cubic in P 3 . This is a curve of degree three defined by {x 1 x 3 -x 2 2 = x 2 x 4 -x 3 3 = x 1 x 4 -x 2 x 3 = 0}. The set of lines in P 3 that meet C form a three-fold inside Gr ( This polynomial represents Chow form of C.

Multi-graded Chow forms

In [START_REF] Osserman | Multigraded Cayley-Chow forms[END_REF], we proposed a natural generalization of the Chow construction for varieties inside a product of projective spaces. Our motivation was to provide an explanation to multi-view tensors, and to define analogous "multiview forms" for non-central cameras (see Section 3.4.3). The following is a very brief summary of the main results in the paper.

Let X be a projective variety in

P n 1 × . . . × P n k of dimension r. Let β = (β 1 , . . . , β k ) be a vector of indices with 0 ≤ β i ≤ n k such that k i=1 β i = r+1.
As before, by dimensionality constraints we expect that a general product of linear spaces L 1 × . . . × L k where each L i has dimension β i will not intersect X. We thus consider the set

Z X,β = {(L 1 , . . . , L k ) : X ∩ (L 1 ∩ . . . ∩ L k ) = ∅} ⊂ Gr(n -β 1 , P n 1 ) × . . . × Gr(n -β k , P n k ).
(1.31) As we show in [START_REF] Osserman | Multigraded Cayley-Chow forms[END_REF], in order for this set to actually provide a generalization of the Chow construction, we must assume some inequality conditions on the dimensions of projections of X. This is essentially because describing the expected dimensions of the intersection of varieties in a product of projective spaces is more complicated than in traditional projective spaces. In the following, the multidegree of the variety X in P n 1 × . . . × P n k is a polynomial in Z[t 1 , . . . , t k ] that generalizes the degree of a projective variety.

Theorem 1.5.5. [START_REF] Osserman | Multigraded Cayley-Chow forms[END_REF] The set Z X,β in (1.31) is a hypersurface determining X if and only if for every non-empty I {1, . . . , k} we have

i∈I β i ≤ dim p I (X),
where p I (X) is the projection of X on Π i∈I P n i . In this case, Z X,β is the zero-set of a single multi-homogeneous polynomial F X,β in Plücker coordinates. This defines the multi-graded Chow form associated with X, for the indices β. where α i = n i -β i (i.e., the multidegree of the multi-graded Chow form is given by a subset of the coefficients of the multidegree polynomial of X).

Moreover, if X has multidegree

γ=(γ 1 ,...,γ k ) a γ t γ 1 1 . . . t γ k k then F X,
Multi-graded Chow forms are also a generalization of "Grassmann tensors", introduced by Hartley and Schaffalitzky in [START_REF] Richard | Reconstruction from projections using grassmann tensors[END_REF].

Multifocal tensors and the joint image. Multifocal tensors are the result of applying the multi-graded Chow construction to the joint image variety M n inside (P 2 ) n . This can be seen as follows.

Since dim M n = 3, we need to choose β = (β 1 , . . . , β n ) such that 0 ≤ β i ≤ 2 and n i=1 β i = 4. For any such β, we can consider the set Z Mn,β as in (1.31). By definition, this set includes n-tuples (L 1 , . . . , L n ), where each L i is a linear space of codimension

β i in P 2 , such that L 1 × . . . × L n meets M n .
In other words the space Z Mn,β contains products of spaces (L 1 , . . . , L n ) for which there exists a correspondence (u 1 , . . . , u n ) in M n with u i in L i . For example, if n = 3 and β = (2, 1, 1), the set Z M 3 ,β contains triples point-line-line which "correspond" in the traditional sense of multi-view geometry.

According to Theorem 1.5.5, the set Z Mn,β will:

1. be described by a single equation F Mn,β in appropriate coordinates (note that Plücker coordinates for Gr(1, P 2 ) are simply dual projective coordinates).

2. uniquely determine M n (and thus the camera configuration, according to Theorem 1.3.16), if and only if i∈I β i ≤ 3 for all I {1, . . . , k}, since we have that dim p I (M n ) = 3 whenever |I| ≥ 2. This is equivalent to β i > 0 for all i, which together with n i=1 β i = 4 clearly requires n ≤ 4. The only possibilities are:

• For n = 2, β = (2, 2). In this case, the condition F M 2 ,β = 0 describes corresponding points in P 2 × P 2 , i.e., it is simply the epipolar constraint.

• For n = 3, β = (2, 1, 1) (or any such permutation). In this case, the condition

F M 3 ,β = 0 describes "corresponding point-line-line" triples in P 2 × (P 2 ) * × (P 2 ) * . • For n = 4, β = (1, 1, 1, 1). The condition F M 4 ,β = 0 describes "corresponding line-line-line-line" quadruplets in (P 2 ) * × (P 2 ) * × (P 2 ) * × (P 2 ) * .
As a last point, we observe that the conditions F Mn,β are multilinear in the the corresponding coordinates. Indeed, the joint image variety is known to have multidegree

t 2 1 t 2 2 . . . t 2 n   1≤i<j<k≤n 1 t i t j t k + 1≤i,j≤n 1 t 2 i t j   .
(1.32)

Three-View Geometry and Multifocal Tensors

See Corollary 3.5 in [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF]. Since all the coefficients in this polynomial are equal to one, Theorem 1.5.5 says that F Mn,β be multilinear. In particular, can identify the forms F M 3 ,β and F M 4 ,β defined above with tensors. We have thus recovered all the fundamental properties of multi-view tensors from the viewpoint of multi-graded Chow forms.

Example 1.5.6. We consider the cameras P 1 , P 2 , P 3 associated with the matrices

P 1 =    1 0 0 1 0 1 0 0 0 0 1 0    , P 2 =    1 0 0 0 0 1 0 1 0 0 1 0    , P 3 =    1 0 0 0 0 1 0 0 0 0 1 1    .
A simple computation shows that if M 3 is the joint image variety associated with these cameras, the corresponding multi-graded Chow form for β = (2, 1, 1) is defined by

u 1 l 2 k 1 + u 2 l 2 k 1 + u 3 l 3 k 1 -u 2 l 1 k 2 + u 2 l 2 k 2 -u 1 l 1 k 3 -u 3 l 1 k 3 -u 2 l 2 k 3 + u 3 l 2 k 3 -u 3 l 3 k 3 in Q[u 1 , u 2 , u 3 , k 1 , k 2 , k 3 , l 1 , l 2 , l 3 ],
where k, l are variables for dual projective coordinates. This represents a trifocal tensor (one of the three) associated with P 1 , P 2 , P 3 . ♦

We conclude this discussion by pointing out that although a trifocal tensor determines the joint image, the nine trilinear conditions that are traditionally encoded in the tensors only provide a weak characterization of the joint image. Given a trifocal tensor T which distinguishes the first view (β = (2, 1, 1) in the multi-graded Chow set-up), the so-called "point-point-point" (PPP) trilinear conditions are usually written in tensor (Einstein summation) notation as

u i 1 u j 2 u k 3 ( jpr kqs T pq i ) = 0 rs , (1.33)
where ijk the Levi-Civita permutation symbol. See [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]Chapter 15.2.1]. These trilinear constraints correspond to a subset of the maximal minors of the matrix U(u 1 , u 2 , u 3 ) defined in (1.18), more precisely to the nine minors obtained by considering all rows associated with the first camera, and two rows from each of the other two. The following result shows that the algebraic set defined by these nine conditions contains the joint image variety strictly. We believe that this fact has not been pointed out in previous literature (although it is closely related to the known degeneracies of transfer based on the trifocal tensor [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]Section 15.3.2]).

Proposition 1.5.7. For three cameras P 1 , P 2 , P 3 with non-collinear pinholes, the constraints (1.33) describe a set W which decomposes as W = M 3 ∪ S 12 ∪ S 13 , where

S 12 = {e 12 } × {e 21 } × P 2 ⊂ (P 2 ) 3 , S 13 = {e 13 } × P 2 × {e 31 } ⊂ (P 2 ) 3 , (1.34)
and e ij is the epipole in image i relative to the camera j.

The proof of this result is by direct computation, and a geometric justification is given in Figure 1.11. This fact is also consistent with the theory of multi-graded Chow forms. In fact, with the general notation introduced above, if we define S X,β as the set of points of points (u 1 , . . . , u k ) in

P n 1 × . . . × P n k such that every L 1 × . . . × L k containing (u 1 , . . . , u k ) meets X, then S X,

Experiments: trinocular geometry

As an application of some of theoretical results from this chapter, we consider the problem of the estimation of camera parameters (structure-from-motion) for three views.

In particular, we propose the minimization of a "trinocular-epipolar" error, given by the mean squared distance to the "epipolar" and "trinocular" lines, that we will define shortly. We present some experiments from [START_REF] Trager | Trinocular Geometry Revisited[END_REF] that show that epipolar constraints are sufficient for recovering camera parameters for non-degenerate configurations, while for collinear pinholes or for images of 3D points lying close to the trifocal plane, trilinear conditions are necessary.

In what follows, we assume that initial values for the projection matrices P 1 , P 2 , P 3 of three cameras have been estimated. This is possible using six triples of matching points, for a projective model [START_REF] Carlsson | Duality of reconstruction and positioning from projective views[END_REF][START_REF] Quan | Invariants of six points and projective reconstruction from three uncalibrated images[END_REF][START_REF] Ponce | Analytical methods for uncalibrated stereo and motion reconstruction[END_REF]. 6 The traditional approach for bundle-adjustment would proceed by minimizing a "geometric distance error" [START_REF] Hartley | Multiple view geometry in computer vision[END_REF], which measures the mean squared distance of each given triple to the nearest image points which actually correspond, according to the current estimate of the camera parameters. For example, the contribution of a triple of image points u 1 , u 2 , u 3 , assuming estimated parameters P 1 , P 2 , P 3 can be written as:

δ(u 1 , u 2 , u 3 | P 1 , P 2 , P 3 ) = min (û 1 ,û 2 ,û 3 ) {d(u 1 , û1 ) 2 + d(u 2 , û2 ) 2 + d(u 3 , û3 ) 2 }, (1.35)
where (û 1 , û2 , û3 ) are triples that form a correspondence for P 1 , P 2 , P 3 (i.e., belong to the joint image). This error function however has the disadvantage of not being directly computable from image data, since the nearest corresponding triple must be determined by solving a separate optimization problem. Indeed, bundle adjustment requires introducing a set of auxiliary variables that represent the coordinates of the original unknown 3D points. On the other hand, if we consider the "epipolar error" defined by

δ e (u 1 , u 2 , u 3 | P 1 , P 2 , P 3 ) = 3 i=1 d(u i , E i,i+1 (u i+1 )) 2 + d(u i , E i,i+2 (u i+2 )) 2 , ( 1.36)
where E ij (u j ) represents the epipolar line in the image i corresponding to the point u j in image j (defined by {u T j F ji u = 0}, where F ij is the fundamental matrix), and addition is done modulo 3, then δ e can easily be computed from image coordinates and camera parameters. In the next paragraph, we will define a "trinocular-epipolar" error that can also be computed very efficiently, and we will compare the effectiveness of these three error functions using both synthetic and real data. The geometric distance (1.35) will be used to evaluate the quality of the solutions recovered using the different approaches.

Non-collinear pinholes. In the case of three cameras with non-collinear pinholes, we define the trinocular-epipolar error as the average squared distance between each image point, two epipolar lines, and one "trinocular line" in each image. For a given triple (u 1 , u 2 , u 3 ), the trinocular line in image i is the projection of the unique transversal through the fundamental point e 4 = [0, 0, 0, 1] in P 3 and the viewing lines corresponding to u j , u k (with distinct i, j, k). Clearly, if the triple (u 1 , u 2 , u 3 ) were an exact correspondence, each image point would lie on the corresponding trinocular line. Moreover, the trinocular line in each image can be computed easily from the expression of trilinear constraint t 4 defined in Section 1.5.1. The contribution of a triple (u 1 , u 2 , u 3 ) to the trinocular-epipolar error can be written as

δ t (u 1 ,u 2 , u 3 | P 1 , P 2 , P 3 ) = 3 i=1 d(u i , E i,i+1 (u i+1 )) 2 + d(u i , E i,i+2 (u i+2 )) 2 + d(u i , T i+1,i+2 0 (u i+1 , u i+2 )) 2 , (1.37) (u i+1 , u i+2
) is the trinocular line in the i-th image. This error function can be easily computed using only image data. However, we should note that that the trinocular line is actually well-defined if and only if the point e 4 does not project onto (near) any of the image points. In practice, we can guarantee this by applying an appropriate homography of P 3 , imposing for example that e 4 it projects at infinity in the three images (which is possible in general). This corresponds to using a different trilinearity t p , as defined in Section 1.5.1.

We first evaluated this approach on synthetic data: we considered 50 random configurations of triples of camera matrices (simulating realistic extrinsic and intrinsic parameters) and clouds of 20 points (see Figure 1.12), and successively added various amounts of Gaussian noise to the projected points. The averaged results are shown in Figure 1. [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF]. We note that the trinocular-epipolar error gives slightly better results than the epipolar one, but bundle adjustment is the most accurate (as one would expect since it minimizes the geometric distance error (1.35) which is being reported). In this case, incorporating the trilinear constraint brings a slight improvement but does not seem to be essential: indeed, we know that for non-collinear pinholes, and three-dimensional points lying in general position, enforcing the epipolar constraints is sufficient for guaranteeing correspondence. However, if we position the 3D points gradually closer to the trifocal plane, we observe that the epipolar error gives increasingly worse results, ultimately failing in improving the initial reconstruction (Figure 1.14). In fact, when the epipolar lines are close to being coincident, measuring only epipolar distances can cause severe numerical instabilities. On the other hand, using the trinocular-epipolar error or bundle adjustment, the quality of the reconstruction seems to be independent of the distance to the trifocal plane. This experiment shows the same camera configuration and point sets increasingly close to the trifocal plane (constructed as points on the trifocal plane perturbed by gaussian noise).

We also evaluated the trinocular-epipolar approach on real data, using three images from the "house dataset" (courtesy of B. Boufama and R. Mohr). Figure 1.15 shows 38 correspondences between the images from this dataset, and the corresponding epipolar and trinocular lines, after camera parameters were recovered using the trinocular-epipolar approach. Table 1.4 shows the average distances between the data points and these lines. The mean distance to epipolar lines is on the order of 1pixel, and comparable to that obtained by classical techniques for estimating the fundamental matrix from pairs of images on the same data [START_REF] David | Computer Vision -A Modern Approach, Second Edition[END_REF]Ch. 8]. Finally, Table 1.5 compares the results of the Pinhole Cameras and their Multi-View Geometry different approaches, in terms of the geometric distance error. In this case, we see that the reconstructions obtained using the trinocular-epipolar and the epipolar approach are both very accurate. Table 1.4: Average distances (in pixels) to epipolar and trinocular lines for the house dataset (Figure 1.15). Here, "Epiij" refers to the distance between points in image i and the corresponding epipolar lines associated with image j, and "Trj" refers to the distance between points in image j and the corresponding trinocular line associated with the other two images. Tri-Epi Epi Bundle adj 0.73 0.73 0.72 Table 1.5: Quantitative comparison between different approaches for the house dataset (Figure 1.15), in terms of the geometric distance error (measured in pixels).

Collinear pinholes. For three cameras with collinear pinholes, the "trinocular-epipolar" error is defined as the average squared distance of each image point to the three epipolar lines and to two different "trinocular lines" (following Proposition 1.5.1).

δ t (u 1 , u 2 , u 3 | P 1 , P 2 , P 3 ) = 3 i=1 d(u i , E i,i+1 (u i+1 )) 2 + d(u i , E i,i+2 (u i+2 )) 2 + d(u i , T i+1,i+2 A (u i+1 , u i+2 )) 2 + d(u i , T i+1,i+2 B (u i+1 , u i+2 )) 2 ,
(1.38)

Three-View Geometry and Multifocal Tensors

where

T i+1,i+2 A (u i+1 , u i+2 ) and T i+1,i+2 B (u i+1 , u i+2
) are the trinocular lines in the i-th image, defined by the trilinear constraints t 3 , t 4 from Section 1.5.1. Once again, the trinocular lines are well defined if and only if e 3 and e 4 do not project onto (near) any of the given image points: in order to satisfy this condition, we select two points which project very far from the image data, and apply a homography of P 3 which maps e 3 and e 4 to these points. Remark 1.5.8. By restricting ourselves to configurations with collinear pinholes, the degrees of freedom of the camera geometry drop from 18 to 16. This suggests that it is possible to recover initial estimates of the matrices P i using fewer correspondences, and indeed we have implemented a method that recovers projection matrices for collinear pinholes given only five triples of correspondences and one correspondence between two images. This minimal method yields up to six real solutions. However, the simpler six-point algorithm from [START_REF] Carlsson | Duality of reconstruction and positioning from projective views[END_REF] still works in this case. Figure 1.16 shows the results of our experiments using synthetic data (20 points with various amounts of Gaussian noise added to the projections). We see that in this case the epipolar error fails in recovering the correct parameters, while the trinocular-epipolar error is able to achieve essentially optimal solutions. Indeed, we know that in the collinear case the epipolar constraints are never sufficient to guarantee correspondence. Finally, we evaluate our approach on real data, using three pictures taken from collinear viewpoints (allowing rotations about the camera's axis) and matching points using SIFT descriptors (Figure 1.17). Table 1.7 shows the average distances between the data points and the epipolar and trinocular lines for the dataset with collinear pinholes and Figure 1.18 displays the estimated epipolar and trinocular lines. Table 1.6 shows our quantitative results, which confirm that for collinear pinholes the trilinear components of the error are actually necessary.

Pinhole Cameras and their Multi-View Geometry

Tri-Epi Epi Bundle adj 0.68 5.33 0.67 Table 1.6: Quantitative comparison between different approaches for the office dataset (Figure 1.17), in terms of the geometric distance error (measured in pixels).

Figure 1.17: Three pictures taken from collinear viewpoints, and point correspondences obtained by matching SIFT descriptors. 1.7: Average distances (in pixels) to epipolar and trinocular lines for the collinear pinhole dataset (Figure 1.17). Here, "Epiij" refers to the distance between points in image i and the corresponding epipolar lines associated with image j, and "TrjA" and "TrjB" refer to the distances between points in image j and the corresponding two trinocular lines associated with the other two images.

Conclusions

Conclusions

We have presented some topics on the theory of pinhole cameras, focused mainly on the problem of explaining which sets of multi-view constraints can be used 1) to characterize point correspondences and 2) to determine a camera configuration. Indeed, one of the central themes of this chapter is the fact that these two properties are not exactly the same: this is why we defined "weak" and "strong" characterizations of the joint image in Section 1.3. For example, we have seen in Section 1.4 that fewer than 2n -3 epipolar constraints can be used to determine a camera configuration, even though these will characterize a set in (P 2 ) n that is much larger than the set of all point correspondences. This distinction is on the other hand relatively subtle, since a camera configuration clearly determines all point correspondences, and thus any weak characterization can be used to indirectly recover a strong one.

Several questions could deserve further work. In addition to the open problem mentioned at the end Section 1.4.6, we believe that it would be very useful to gain a complete understanding of the algebraic constraints that characterize the compatibility of fundamental matrices (i.e., study the algebraic set X ⊂ (P 8 ) N considered in Section 1.4.4). This is an issue that has not been considered much in classical multi-view geometry, but is clearly important in practice, since global compatibility constraints are not automatically satisfied using local measurements (e.g., pairwise matches) [START_REF] Sweeney | Optimizing the viewing graph for structure-from-motion[END_REF]. More generally, it would be helpful to develop more concrete models for representing the space of camera configurations (or, equivalently, for parameterizing joint-images), beyond traditional multifocal tensors.

Finally, the relationship between point correspondences and camera configurations resembles, in a vague sense, a sort of "duality". This idea can made more precise by appropriately restricting the choice of image coordinates, so that pinholes and scene points play (almost) symmetric roles. This is the basis of Carlsson-Weinshall duality [START_REF] Carlsson | Dual computation of projective shape and camera positions from mutliple images[END_REF]. We believe that this duality can be useful to gain further insight on many of the issues addressed in this chapter: we are currently revisiting it from the perspective of the joint images and point configurations [START_REF] Arthur B Coble | Point sets and allied Cremona groups[END_REF].

Proofs

This section collects some proofs. Propositions that appeared previously will be stated here with the same numbering, while auxiliary results will be numbered independently.

The joint image

We first prove Proposition 1.3.5 from Section 1.3, that describes the relationship between the open joint image M o n and its closure M n . Proposition 1.3.5. Given n ≥ 3 cameras with non-collinear distinct pinholes, one has

M o n = M n \ E n
, where

E n = n i=1 {e 1i } × . . . × P 2 (i) × . . . × {e ni }. (1.13)
Here P 2 (i) indicates P 2 at position i in the product, and e ij denotes the j-th epipole in the i-th image. However, if n = 2, or more generally if the cameras have collinear pinholes, then

E n = n i=1 {e 1 } × . . . × P 2 (i) × . . . × {e n } \ {(e 1 , . . . , e n )}, (1.14)
where e i now denotes the unique epipole in image i (so e i = P i (c j ) for all j = i).

Proof. In Section 1.3.3, we introduced the matrix

U(u 1 , . . . , u n ) =       P 1 u 1 0 . . . 0 P 2 0 u 2 . . . 0 . . . . . . . . . . . . . . . P n 0 0 . . . u n       , ( 1.39)
and pointed out that joint image variety can be characterized set-theoretically by

M n = {(u 1 , . . . , u n ) ∈ (P 2 ) n | U(u 1 , . . . , u n ) is not full rank}.
If the camera pinholes are not all collinear, the null space of U(u 1 , . . . , u n ) has dimension at most one, and (u 1 , . . . , u n ) belongs to E n = M n \M o n if and only if annihilating vectors [p, λ 1 , . . . , λ n ] ∈ R 4+n are such that λ i = 0 for some i: in this case p must be one of the camera centers, and we see that E n is given by (1.13). If the cameras are collinear, then the same reasoning applies except when (u 1 , . . . , u n ) = (e 1 , . . . , e n ) (because the null space of U(e 1 , . . . , e n ) has dimension two): in this case, it is easy to realize that (e 1 , . . . , e n ) ∈ M o n .

We next give a proof of Proposition 1.3.11. The result is implied by Theorem 3.6 in [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF], but the case of multilinear constraints (multidegree (1, . . . , 1)) is not pointed out. Our proof is also adapted from the one given in [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF], but we use more direct argument to show Proposition 1.7.3.

Proposition 1.3.11. The multilinear polynomials that vanish on M n form a vector space of dimension

d n = 3 n -n+3
Proofs Remark 1.7.1. In the rest of this section, we will assume that the camera matrices P 1 , . . . , P n satisfy a generality assumption, namely that all 4 × 4 matrices formed by any four distinct rows from P 1 , . . . , P n have full rank. If the camera pinholes are distinct, this condition can be guaranteed by changing bases appropriately in the different image planes. Since this kind of operation preserves the dimension of spaces generated by multilinearities (the vector spaces are mapped isomorphically onto themselves), this assumption does not affect the general validity of the result.

Proof. Let W n be the vector space generated by multlinear relations arising as maximal minors of U(u 1 , . . . , u n ) defined in (1.18); for simplicity we will refer to these constraints as the "fundamental" multilinearities. We will first compute the dimension of W n ; then we will show that W n coincides with the space of all multilinear constraints which vanish on M n . Our proof heavily relies on the fact that the fundamental multilinearities form a Gröbner basis for lexicographic ordering (and in fact, for all admissible term orderings): this will be shown in Proposition 1.7.3.

To compute the dimension of W n we will use the following well known property of Gröbner bases (see for example [START_REF] David A Cox | Ideals, Varieties, and Algorithms: An Introduction to Computational Algebraic Geometry and Commutative Algebra[END_REF]):

(P) If a set of homogeneous polynomials f 1 , . . . , f N forms a Gröbner basis (for a fixed monomial ordering), then dim

R (f 1 , . . . , f N ) = dim R (In(f 1 ), . . . , In(f N )),
where In(f i ) denotes the initial term of f i .

To use this result, we first introduce a lexicographic ordering for the monomials in the ring R[x 1 , y 1 , z 1 , . . . , x n , y n , z n ], so that

x 1 • • • x n y 1 • • • y n z 1 • • • z n .
In Lemma 1.7.2, we will show that for this ordering the set of leading terms of the fundamental mutlilinearities is given by B = C n \ A, where

C n = {x α y β z γ | α, β, γ ∈ N n , α + β + γ = (1, . . . , 1)}, ( 1.40)
is the set of all multilinear monomials, and

A ={x α y β z γ ∈ M n | α = 0, |β| ≤ 3} ∪ {x α y β z γ ∈ M n | |α| = 1, |β| ≤ 1}, (1.41)
where we used multi-index notation, so that for example

x α = k x α(k) k .
According to property (P), we know that dim(W n ) = dim(Span(B)). Since all monomials are linearly independent, we can now compute the dimension of W n using a simple

Pinhole Cameras and their Multi-View Geometry

counting argument: dim(W n ) = |T | = |M n | -|S| = = 3 n -1 + n + n 2 + n 3 -n 1 + (n -1) = 3 n -1 + n + n 2 + n 3 + n(n + 1) + n = 3 n - n + 3 3 + n.
(1.42)

Finally, to show that W n coincides with the space of all multilinear constraints, we recall that the fundamental multilinearities set-theoretically define the joint image variety M n (they give necessary and sufficient conditions for correspondence). Using the Nullstellensatz Theorem, it is enough to prove that they generate a radical ideal8 . This will also be shown in Proposition 1.7.3. Proof. From (1.40), we see that a monomial x α y β z γ in C n can be also identified by the vector ν = α + 2β + 3γ ∈ {1, 2, 3} n ; in this case we will write µ ν = x α y β z γ . One can verify that µ ν 1 can be a leading term of a fundamental multilinearity if and only if there exist ν 2 , ν 3 , ν 4 such that µ ν 1 µ ν i and |ν 1 -ν i | ≤ 4 for i = 2, 3, 4. Indeed, a fundamental mutilinearity P is defined by choosing n + 4 rows of U from (1.18), and the monomials µ ν appearing in P correspond to subsets of n rows (which are expressed by the vectors ν ∈ {1, 2, 3} n ); note also that the generality assumption guarantees that all possible coefficients are non-zero. It is straightforward to verify that µ ν 1 does not satisfy the previous property if and only if it belongs to the set A defined in (1.41).

Lemma 1.7.2. Using the ordering x

1 • • • x n y 1 • • • y n z 1 • • • z n ,

Proposition 1.7.3. The fundamental multilinearities form a universal Gröbner basis (i.e., a Gröbner basis relative to any monomial ordering). Moreover, they define a radical ideal.

Proof. Our proof is motivated by a classical result, which states that in the ring R = R[x ij ], where i = 1, . . . , m, j = 1, . . . , n, and m ≤ n, the maximal m × m minors of X = (x i,j ) form a universal Gröbner basis (generating the classical determinantal ideal) [START_REF] Miller | Combinatorial Commutative Algebra[END_REF].

Proofs

In order to obtain a similar statement for the maximal minors of the matrix U, we use the following two results:

1. If Z is a subset of variables in R[x ij ],
and X is the matrix obtained from X = (x i,j ) by setting the variables in Z to zero, then the nonzero maximal minors of X form a universal Gröbner basis [START_REF] Boocher | Free resolutions and sparse determinantal ideals[END_REF].

2. Let f 1 , . . . , f N be a set of polynomials forming a universal Gröbner basis in the ring R[s 1 , . . . , s k , t 1 , . . . , t j ], and consider the evaluation map

φ : R[s 1 , . . . , s k , t 1 , . . . , t j ] → R[s 1 , . . . , s k ] g(s 1 , . . . , s k , t 1 , . . . , t j ) → g(s 1 , . . . , s k , c 1 , . . . , c j ) (1.43)
for a fixed set c 1 , . . . , c j ∈ R. Thinking f 1 , . . . , f N as polynomials with ring coefficients, i.e., as elements of

R[s 1 , . . . , s k ] with R = R[t 1 , . . . , t j ], we denote with Lc(f i ) ∈ R the leading coefficient of f i . If φ(Lc(f 1 )), . . . , φ(Lc(f N )) are non-zero, then φ(f 1 ), . . . , φ(f N ) form a universal Gröbner basis in R[s 1 , . . . , s k ] [97].
We now consider the matrix U as in (1.18), but with indeterminate camera matrices: according to the first result given above, the maximal minors form a universal Gröbner basis. The fundamental multilinearities are obtained from these minors by substituting the actual entries of the camera matrices. If the leading coefficients of the minors are not mapped to zero under this specialization, then we can use the second result and obtain the claim. However, it is easy to realize that the leading coefficients are given by the determinants of the matrices obtained by selecting four rows from the camera matrices; assuming the generality condition from the Remark, these do not vanish for the actual camera entries. This implies that the fundamental multilinearities form a universal Gröbner basis.

Finally, we now see that the monomials in B = C n \ A (defined in (1.40) and (1.41)) generate the initial ideal for the ideal generated by fundamental multilinearities. The fact that this monomial ideal is squarefree, allows us to conclude that the fundamental multilinearities generate a radical ideal; see [START_REF] Herzog | Binomial edge ideals and conditional independence statements[END_REF]Corollary 2.2].

We next show that the joint image variety is closely related to the blow up, a fundamental construction in algebraic geometry [START_REF] Hartshorne | [END_REF]. For our purposes we can give the following definition for the blow up: Definition 1.7.4. Consider a smooth algebraic variety X of dimension n, and a subvariety Z ⊆ X of dimension d. Locally, we may write X as Z × W where W has dimension n -d and is transversal to Z. Let λ : X \ Z → P n-d-1 be the map associating with x = (z, w) ∈ Z × W the line x passing through w and z. The blow up of X in Z is given by closure of the graph of λ:

XZ = {(x, x ), x ∈ X \ Z} ⊆ X × P n-d-1 .
(1.44)

Pinhole Cameras and their Multi-View Geometry

This is a smooth variety, with a natural map π : XZ → X, known as the blow up map.

The exceptional locus is the inverse image Z = π -1 (Z) = Z × P n-d-1 of the center Z. It is is the set of points where π fails to be a local isomorphism: in fact, π always contracts the second factor P n-d-1 of Z to a point. See Figure 2 in the main part of paper.

Basically, the blow up replaces a subvariety Z ⊂ X with all the directions in X pointing out of Z. Blowups are an extremely important tool for the resolution of singularities, that is, the operation of constructing suitable smooth models for varieties with singularities. We refer to [START_REF] Hauser | The Hironaka theorem on resolution of singularities (or: A proof we always wanted to understand)[END_REF] for a nice introduction.

Proposition 1.7.5. If n ≥ 3 cameras do not have collinear pinholes, the joint image variety M n is isomorphic to the blow up of X = P 3 at Z = {c 1 , . . . , c n }, where c i , i = 1, . . . , n are the camera pinholes.

Proof. Iterating the construction given in Definition 1.7.4, we see that the blow up of

P 3 at Z = {c 1 , . . . , c n } is given by XZ = {(p, 1 p , . . . , n p ), p ∈ P 3 \ Z} ⊆ P 3 × (P 2 ) n , ( 1.45)
where i p denotes the line through p and c i . In particular, the projection of XZ onto (P 2 ) n is exactly the joint image variety M n , so we have a natural map

f : XZ → M n .
(

In order to prove that f is an isomorphism we have to show that: 1)

f is injective 2) the differential T f of f is injective.
The fact that f is injective for (p, 1 p , . . . , n p ) with p ∈ P 3 \ Z follows from the observation that the "joint projection map" P 3 P 2 × . . . × P 2 is injective (where it is defined) if one assumes non-collinear pinholes. Moreover, if p = c i for some i, then the exceptional set π -1 (c i ) is mapped isomorphically onto {(e 1i , . . . , , . . . e ni ), ∈ P 2 } ⊆ M n , where e ji is the epipole in the j-th image for the center c i (note that the image of the exceptional locus f (π -1 (Z)) is exactly the set E n given by Proposition 1.3.5).

To prove that the differential T f is injective at points (p, 1 p , . . . , n p ) with p ∈ P 3 \ Z, it is enough to observe that all lines in P 3 are mapped injectively by the joint projection map (since f is locally an affine map, if the differential were not injective, some line would have to be contracted). Similarly, if p = c i , then all lines contained in π -1 (c i ) are mapped isomorphically on some image P 2 , and the same holds for all lines passing through c i in P 3 . Proposition 1.3.7 (Singularities of the joint image variety). When the camera pinholes are not collinear, M n is smooth. When they are collinear (in particular for n = 2 views), then M n has a unique singular point given by the n-tuple of epipoles (e 1 , . . . , e n ).

Proofs

Proof. If the camera pinholes are not collinear, then Proposition 1.7.5 immediately implies that the joint image variety is smooth. If the pinholes are collinear, then the projection map f : XZ → M n considered in the previous proof is still well defined, and is a local isomorphism except at points that lie on the baseline ⊆ P 3 containing the centers (more precisely, on its strict transform in XZ , given by π -1 ( \ Z)). In fact, f contracts this set onto (e 1 , . . . , e n ). In general, however, birational morphisms between smooth varieties can only contract sets of codimension 1 [118, Theorem 2.2], while the baseline has codimension 2: we conclude that (e 1 , . . . , e n ) must be the only singular point of M n .

We now turn to algebraic characterizations of the joint image, and prove Theorem 1. Proof. Let us first assume that n = 4. If P 1 , . . . , P 4 are cameras with pinholes c 1 , . . . , c n in general position (i.e., non coplanar), then up to homographies of P 3 (that do not affect the joint image) and homographies of image planes (that simply result in linear changes of variables that map k-linearities isomorphically onto themselves) we may assume that:

P 1 =    1 0 0 1 0 1 0 0 0 0 1 0    , P 2 =    1 0 0 0 0 1 0 1 0 0 1 0    , P 3 =    1 0 0 0 0 1 0 0 0 0 1 1    , P 4 =    1 0 0 0 0 1 0 0 0 0 1 0    .
(1.47)

In this case, one can verify using Gröbner bases that the six epipolar relations are sufficient to generate, up to irrelevant components, the ideal associated to the joint image variety M 4 (see also [START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF]).

If P 1 , . . . , P 4 are cameras with pinholes c 1 , . . . , c 4 in general coplanar position (i.e., copla-

Pinhole Cameras and their Multi-View Geometry

nar with no subset of three that are collinear), then we may similarly assume that:

P 1 =    1 0 0 1 0 1 0 0 0 0 1 0    , P 2 =    1 0 0 0 0 1 0 1 0 0 1 0    , P 3 =    1 0 0 0 0 1 0 0 0 0 1 1    , P 4 =    1 0 0 1/3 0 1 0 1/3 0 0 1 1/3    .
(1.48)

This time, using Gröbner basis we see that the epipolar conditions describe an algebraic set W that is strictly larger than M 4 . In fact, W decomposes as

W = M 4 ∪ V t (1.49)
where

V t = {x i + y i + z i = 0 | i = 1, .
. . , 4} is the product of the "trifocal lines" (the projection of the plane containing the pinholes). We see that the epipolar constraints are only a weak characterization of the joint image: camera geometry can be recovered (for example by considering subsets of three cameras, see [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]) although correspondences are not directly characterized. On the other hand, one can verify that including trilinear relations is sufficient to exclude the spurious solutions and yields a strong characterization of M 4

If P 1 , . . . , P 4 are cameras with pinholes c 1 , . . . , c n that are coplanar, and with a subset of three that are collinear, then we can consider

P 1 =    1 0 0 1 0 1 0 0 0 0 1 0    , P 2 =    1 0 0 0 0 1 0 1 0 0 1 0    , P 3 =    1 0 0 0 0 1 0 0 0 0 1 1    , P 4 =    1 0 0 0 0 1 0 1/2 0 0 1 1/2    .
(1.50)

Once again, we see that that the bilinear and trilinear constraints completely characterize M 4 , while only bilinear conditions describe a set W that also decomposes as

W = M 4 ∪ V t (1.51) where V t = {x i + y i + z i = 0 | i = 1, .
. . , 4} is the product of the "trifocal lines".

Finally, if all P 1 , . . . , P 4 have collinear pinholes, we cannot simply change reference frame in P 3 and assume the cameras to be fixed, since four collinear points are not always projectively equivalent (the projective invariant is given by the cross-ratio). However,

Proofs

we can use a single parameter to describe all such camera configurations:

P 1 =    1 0 0 1 0 1 0 0 0 0 1 0    , P 2 =    1 0 0 0 0 1 0 1 0 0 1 0    , P 3 =    1 0 0 1/2 0 1 0 1/2 0 0 1 0    , P 4 =    1 0 0 t 0 1 0 1 -t 0 0 1 0    .
(1.52)

In this case, computing the primary decomposition we see that bilinear and trilinear constraints have spurious components only for t = 0, 1/2, 1 which correspond to nonvalid values for t (we assume the camera pinholes are distinct). Bilinear constraints, on the other hand, characterize (for feasible values of of t) a single irreducible component W, that strictly contains contains the joint image. With our choice of camera matrices, this component is described by

W = { -z 3 x 4 -z 3 y 4 + x 3 z 4 + y 3 z 4 = 0, -z 2 x 4 -z 2 y 4 + x 2 z 4 + y 2 z 4 = 0, -z 2 x 3 -z 2 y 3 + x 2 z 3 + y 2 z 3 = 0, -z 1 x 4 -z 1 y 4 + x 1 z 4 + y 1 z 4 = 0, -z 1 x 3 -z 1 y 3 + x 1 z 3 + y 1 z 3 = 0, -z 1 x 2 -z 1 y 2 + x 1 z 2 + y 1 z 2 = 0}. (1.53)
Note that these expressions do not depend on the parameter t, which shows that camera matrices cannot be determined by the epipolar conditions. We conclude that in this case the bilinearities do not give a weak characterization of the joint image.

This analysis proves Proposition 1.3.19 for n = 4, since we have exhausted all possible configurations of four pinholes in P 3 . The case n > 4 now follows easily. Indeed, bilinear and trilinear are always strongly sufficient because we have shown this to be true for all sets of four cameras (and this is enough thanks to Proposition 1.3.9). If the pinholes are non-coplanar, we may assume that cameras (1, 2, 3) have non-collinear pinholes and span a plane that does not contain any other pinhole, so applying the previous argument for all quadruplets of cameras (1, 2, 3, i) we can conclude that bilinearities give a strong characterization of the joint image. If the pinholes are only non-collinear, we can assume that pinholes (1, 2) span a line that doesn't contain other pinholes, and similarly use this to conclude that bilinearities give a weak characterization of the joint image. Finally, whenever the pinholes are all aligned, camera geometry cannot be uniquely determined from bilinearities: for example, the expressions in (1.53) show that, using appropriate image coordinates, all pairs of cameras with pinholes on a given line yield the same epipolar constraint. More generally, it is easy to realize that if P 1 , . . . , P n have collinear pinholes, then any set of camera matrices P 1 , . . . , P n , where P i is obtained from P i by adding to its columns multiples of the coordinate vector for the epipole, will yield the same set of epipolar constraints.

Viewing graphs

We next prove Proposition 1.4.16 from Section 1.4.6. We first need the following result, that shows that the conditions (1.27) are in fact linear constraints on h λ and h λ .

Proposition 1.7.6. If c = (c 0 , c 1 , c 2 , c 3 ) is a non-zero vector, then a matrix M = (m ij ) i,j=0,...,3 can be written in the form M = αI 4 + cv T for some arbitrary α and v if and only if the following linear expressions vanish:

m 31 c 2 -m 21 c 3 m 30 c 2 -m 20 c 3 m 32 c 1 -m 12 c 3 m 30 c 1 -m 10 c 3 m 23 c 1 -m 13 c 2 m 20 c 1 -m 10 c 2 m 32 c 0 -m 02 c 3 m 31 c 0 -m 01 c 3 m 23 c 0 -m 03 c 2 m 21 c 0 -m 01 c 2 m 13 c 0 -m 03 c 1 m 12 c 0 -m 02 c 1 m 22 c 1 -m 33 c 1 -m 12 c 2 + m 13 c 3 m 21 c 1 -m 11 c 2 + m 33 c 2 -m 23 c 3 m 30 c 0 -m 32 c 2 -m 00 c 3 + m 22 c 3 m 22 c 0 -m 33 c 0 -m 02 c 2 + m 03 c 3 m 20 c 0 -m 00 c 2 + m 33 c 2 -m 23 c 3 m 31 c 1 -m 32 c 2 -m 11 c 3 + m 22 c 3 m 11 c 0 -m 33 c 0 -m 01 c 1 + m 03 c 3 m 10 c 0 -m 00 c 1 + m 33 c 1 -m 13 c 3 . (1.54)
Proof. The result is easily shown using a computer algebra system. Inside the ring Q[m 00 , . . . , m 33 , c 0 , . . . , c 3 , v 0 , . . . , v 3 , α], we consider the ideal I obtained by eliminating the variables v 0 , v 1 , v 2 , v 3 and α from the coordinates of M -αI 4 + cv T . We can then verify that (1.54) generate an ideal that decomposes into two prime components: one of these is irrelevant for us (it describes the vanishing of c) and the other one is I. Proposition 1.4.16. The tangent space of T G (c 1 , . . . , c n ) at the identity can be represented as the space of e-tuple of matrices (h λ , λ ∈ E G ) where each h λ is in R 4×4 (not necessary invertible), and with compatibility conditions of the form

h λ -h λ = αI 4 + c i v T , (1.27)
where α ∈ R \ {0} and v ∈ R 4 are arbitrary, and λ and λ share the vertex i.

Proofs

Proof. According to Proposition 1.7.6, a matrix M can be written in the form M = αI 4 + cv T for some α ∈ R \ {0} and v ∈ R 4 if and only if it satisfies a set linear equations that depend on c. Let us write L c (M) = 0 for these linear conditions (so L c is a linear map). Note that necessarily L c (I 4 ) = 0.

A constraint of the form (1.26) can now be expressed as F (g λ , g λ ) = L c (g λ g -1 λ ) = 0. Writing the first order expansion or F (g λ , g λ ) = 0 at (I 4 , I 4 ) we obtain (≈ denotes equality up to higher order terms)

F (I 4 + h λ , I 4 + h λ ) ≈ L c ((I 4 + h λ )(I 4 -h λ )) ≈ L c (h λ -h λ ).
(1. [START_REF] Feldman | On the epipolar geometry of the crossed-slits projection[END_REF] This shows that L c (h λ -h λ ) = 0 is the tangent space at the identity of each constraint (1.26). By Proposition 1.7.6 we have that L c (h λ -h λ ) = 0 is equivalent to (1.27), and this concludes the proof.

By substituting M = h λ -h λ inside (1.54), we obtain explicit equations that can be used to determine finite solvability (as explained in Section 1.4.6).

We now discuss the computations required to analyze the viewing graph in Example 1.4.19 from Section 1.4.7. The following simpler example serves as a useful preparation.

Example 1.7.7. Let G be the four-cycle shown in Figure 1.19. This graph is clearly not solvable, because a solvable graph with four vertices must have at least e(4) = 5 edges. It is however useful to understand this example algebraically. Following the general approach described in Section 1.4.6, we assign the identity I 4 to the edge (1, 2), and unknown matrices g (2,3) , g [START_REF] Aholt | The ideal of the trifocal variety[END_REF][START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF] , g [START_REF] Abdeljaoued | Minors of Bezout matrices, subresultants and the parameterization of the degree of the polynomial greatest common divisor[END_REF][START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF] to the remaining edges. The compatibility equations yield

g (1,4) = α 1 I 4 + c 1 v T 1 g (2,3) = α 2 I 4 + c 2 v T 2 g (3,4) g -1 (2,3) = α 3 I 4 + c 3 v T 3 g (3,4) g -1 (1,4) = α 4 I 4 + c 4 v T 4 (1.56)
which imply that

g (3,4) = (α 3 I 4 + c 3 v T 3 )(α 2 I 4 + c 2 v T 2 ) = (α 4 I 4 + c 4 v T 4 )(α 1 I 4 + c 1 v T 1 ). (1.57)
Expanding (1.57) we obtain

(α 3 α 2 -α 4 α 1 )I 4 = c 1 w T 1 + c 2 w T 2 + c 3 w T 3 + c 4 w T 4 , (1.58)
where

w 1 = α 4 v 1 , w 2 = -α 3 v 2 , w 3 = -(α 2 I 4 + v 2 c T 2 )v 3 , w 4 = (α 1 I 4 + v 1 c T 1 )v 4 . (1.59)
From (1.58) we see that the vectors w i must be scalar multiples of the rows of the matrix C -1 where C has columns c 1 , c 2 , c 3 , c 4 . This easily implies that for fixed general coefficients α 1 , α 2 , α 3 , α 4 , there is a unique solution for v 1 , . . . , v 4 in (1.56). Moreover, since the matrices g (i,j) represent projective transformations, we can rescale these equations so that for example α 1 = α 2 = α 3 = 1. This shows that there is one degree of projective freedom corresponding to the choice of α 4 . This projective freedom can be explained in other ways: for example, our analysis implies that g [START_REF] Abdeljaoued | Minors of Bezout matrices, subresultants and the parameterization of the degree of the polynomial greatest common divisor[END_REF][START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF] can be any matrix of the form

g (1,4) = α 1 I 4 + c 1 v T 1 where v 1 • c 2 = v 1 • c 3 = v 1 • c 4 = 0.
Not counting the scale factor, this gives one degree of freedom, and fixing g [START_REF] Abdeljaoued | Minors of Bezout matrices, subresultants and the parameterization of the degree of the polynomial greatest common divisor[END_REF][START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF] of this type determines all other matrices up to scale. ♦ 9 Moreover, we can write

• v 4 = c 2 • v 4 = c 4 • v 4 = 0 while v 6 must satisfy c 2 • v 6 = c 3 • v 6 = c 6 • v 6 = 0).
g (4,5) g -1 (5,8) = I 4 + c 5 v T 8 g (5,8) g -1 (5,6) = αI 4 + c 5 v T 8 (1.60)
where

c 4 • v 8 = c 7 • v 8 = c 8 • v 8 = 0 and c 6 • v 8 = c 8 • v 8 = c 9 • v 8 = 0.
Multiplying these two expressions together we obtain

g (4,5) g -1 (5,6) = (I 4 + c 5 v T 8 )(αI 4 + c 5 v T 8 ) = (I 4 + c 5 v T 4 )(I 4 + c 5 v T 6 ), (1.61)
which yields

αI 4 + c 5 (v 8 + (α + c T 5 v 8 )v 8 ) T = I 4 + c 5 (v 6 + (1 + c T 5 v 6 )v 4 ). (1.62)
This relation can only be satisfied if α = 1 and

v 8 + (1 + c T 5 v 8 )v 8 = v 6 + (1 + c T 5 v 6 )v 4 .
(1.63)

Proofs

Because the orthogonality conditions that defined v 4 , v 6 , v 8 , v 8 are all independent (since the pinholes are generic), (1.63) can be satisfied only if these vectors are all zero. From this we deduce that all the matrices g (i,j) must be the identity, and the graph G is solvable. ♦

Three-view geometry

We focus on the case of n = 3 views. We first point out that any three cameras with non-collinear pinholes can be transformed by homographies of P 3 and of the image planes into

P 1 =    1 0 0 1 0 1 0 0 0 0 1 0    , P 2 =    1 0 0 0 0 1 0 1 0 0 1 0    , P 3 =    1 0 0 0 0 1 0 0 0 0 1 1    .
(1.64)

Similarly, three cameras with collinear pinholes can be transformed into

P 1 =    1 0 0 1 0 1 0 0 0 0 1 0    , P 2 =    1 0 0 0 0 1 0 1 0 0 1 0    , P 3 =    1 0 0 1/2 0 1 0 1/2 0 0 1 0    . (1.65)
These camera matrices will be used for our explicit computations.

We now consider the trilinear constraints encoded in the trifocal tensor. We recall that these conditions can be deduced from the maximal minors of the matrix

U(u 1 , u 2 , u 3 ) =    P 1 u 1 0 0 P 2 0 u 2 0 P 3 0 0 u 3    . (1.66)
More precisely, the nine trilinearities in the tensor that distinguishes the first view are given by

t ij , i ∈ {4, 5, 6}, j ∈ {7, 8, 9} (1.67)
where t ij is obtained by considering all rows of U excluding i and j. These are the same as (1.33).

Proposition 1.5.7. For three cameras P 1 , P 2 , P 3 with non-collinear pinholes, the constraints (1.33) describe a set W which decomposes as W = M 3 ∪ S 12 ∪ S 13 , where

S 12 = {e 12 } × {e 21 } × P 2 ⊂ (P 2 ) 3 , S 13 = {e 13 } × P 2 × {e 31 } ⊂ (P 2 ) 3 , (1.34)
and e ij is the epipole in image i relative to the camera j.

Proof. Considering the camera matrices (1.64), one can verify using primary decomposition that the nine relations describe a space W that decomposes as W = M 3 ∪ S 12 ∪ S 13 , where S 12 = {x 1 + y 1 = 0;

x 2 + y 2 = 0;

z 1 = 0; z 2 = 0} S 2 = {x 1 + y 1 = 0; x 3 + y 3 = 0; z 1 = 0; z 3 = 0}.
(1.68)

For our choice of cameras, these sets are exactly the ones described by (1.67), which in turn are a characterization that is independent of the choice of reference frames.

By using the collinear cameras of (1.65) one can also show the following Proposition 1.7.9. If P 1 , P 2 , P 3 have collinear pinholes, the constraints (1.67) strongly characterize the joint image M 3 .

Let us now clarify a subtle point relative to this last result: from Proposition 1.3.11, we know that 9 trilinearities can never generate all of the trilinear relations which vanish on M 3 (since this is always a vector space of dimension 10). However, this does not exclude the possibility for these to set-theoretically define M 3 : in other words, the zero-set of some relations can still describe M 3 , even though they do not algebraically generate the whole space of the trilinearities. This is the case for the nine trilinearities (1.67) for cameras with collinear pinholes. For example, one can verify that for the matrices (1.65) the trilinearity

t 26 = -z 1 x 2 x 3 -2z 1 y 2 x 3 + z 1 x 2 y 3 + x 1 x 2 z 3 + x 1 y 2 z 3 (1.69)
does not belong to the vector space generated by (1.67), although its zero-set clearly contains that of these nine trilinearities. ), that are obtained by multiplying bilinearities with a variable associated to the excluded image. In the case of non-collinear pinholes, one can verify that these constraints span a vector space of dimension 9. Any trilinearity that does not vanish on the product of the trifocal lines cannot belong to this space, so the three bilinearities together with such a trilinearity must describe the same zero-set as the whole space of trilinear constraints, i.e., exactly M 3 . See also [START_REF] Trager | Trinocular Geometry Revisited[END_REF].

2. This is well known: see [START_REF] Hartley | Multiple view geometry in computer vision[END_REF][START_REF] Heyden | Algebraic properties of multilinear constraints[END_REF], or the discussion in Example 1.3.18.

We now assume that P 1 , P 2 , P 3 have collinear pinholes.

1. Consider constraints {b 12 , b 13 , b 23 , t ij }, where t ij is chosen from (1.67). In generic conditions these describe a set W that decomposes as

W = M 3 ∪ V 1 ∪ V 2 ,
where V 1 and V 2 are each products of three corresponding (epipolar) lines in each image, respectively passing through the (i -2)-th coordinate points in the second image and through the (j -5)-th coordinate point in the third (e.g., if we consider T 48 then V 1 is the product of the three epipolar lines containing [1, 0, 0] ∈ P 2 in the second image and V 2 is the product of the three epipolar lines containing [0, 1, 0] ∈ P 2 in the third). Moreover, in general, if another trilinearity T kl among (1.67) is such that k = i and l = j, then {b 12 , b 13 , b 23 , t ij , t kl } are strongly sufficient, since the spurious sets associated with each trilinearity are excluded by the other constraint. The "generic" assumption that was used in this argument is that the triples of epipolar lines associated with the fundamental points are all distinct: in other words, we assumed that 1) no fundamental point in each image corresponds to any other fundamental point in another 2) no pair of fundamental points in a given image lies on the same epipolar line 10 . However, even if these conditions are not satisfied, there will always be at least two trilinearities among (1.67) that can be used together with the epipolar constraints to characterize M 3 : this follows from the fact that in each image one can always choose two fundamental points that do not lie on the same epipolar line.

2. Let t be any trilinearity whose zero-set does not contain that of {b 12 , b 13 }: then we may be sure that W = {b 12 = b 13 = t = 0} is a set of dimension 3, which must contain M 3 as a component of maximal dimension. By removing spurious components we can recover a strong characterization of the joint image, and thus camera geometry can be determined.

Chapter 2

Geometric Cameras

In the previous chapter, we modeled a pinhole camera as a linear map P 3 P 2 , described by a 3 × 4 matrix up to scale. We now adopt a more geometric viewpoint, and represent a camera as a mapping from P 3 to a two-dimensional family of lines, or a line congruence. This model ignores image planes and measurements, and focuses on the intrinsic geometry of the imaging process. This setting also allows us to study very general cameras in a unified manner: our model serves in fact as a natural abstraction of many existing devices, including pushbroom, panoramic and catadioptric cameras.

The material in this chapter is based on the following publication:

-Jean Ponce, Bernd Sturmfels, and Matthew Trager. "Congruences and concurrent lines in multi-view geometry". In: Advances in Applied Mathematics 88 (2017), pp. 62-91.

Introduction

Despite the practical usefulness of image coordinates for representing pixels in photographs, the geometry of vision is really a matter of lines in space. As pointed out for example by Grossberg and Nayar [START_REF] Michael | The raxel imaging model and raybased calibration[END_REF], the intensity value of a pixel in a photograph does not depend on the physical location of photosensitive elements on the retinal plane, but only on the "primary" 3D viewing ray associated with that pixel. In particular, any imaging system can be seen abstractly as a family of viewing rays, and this viewpoint is especially useful for studying non-central cameras [START_REF] Batog | Admissible linear map models of linear cameras[END_REF][START_REF] Pajdla | Geometry of two-slit camera[END_REF][START_REF] Pajdla | Stereo with oblique cameras[END_REF][START_REF] Ponce | What is a camera?[END_REF]. The set of all viewing rays associated with a camera forms in fact a line congruence, i.e., a surface inside Gr(1, P 3 ) (because every pixel in the two-dimensional image is associated with one viewing line). The representation of cameras as congruences has been used explicitly in [START_REF] Ponce | What is a camera?[END_REF][START_REF] Batog | Admissible linear map models of linear cameras[END_REF]. On the other hand, these works have focused mainly on "linear congruences", i.e.,

Geometric Cameras

congruences that are characterized by linear equations in Plücker coordinates. In this chapter, we study more general congruences that can be used to model cameras.

Our definition of a geometric camera is a map P 3 Gr(1, P 3 ) from 3-space into the Grassmannian of lines, with the property that the (closure of) the pre-image of a point (line) in the Grassmannian is that same line in P 3 . We will also always assume that the coordinates of the map from points to lines are algebraic functions. As mentioned above, a geometric camera is always associated with a congruence of lines [START_REF] Jessop | A Treatise on the Line Complex[END_REF], that is simply the image of the camera in Gr(1, P 3 ). An important subclass of these cameras are those associated with congruences of order one: this means that a generic point in P 3 is contained in exactly one line of the congruence. This type of congruence defines a rational geometric camera, where the map from points to image lines is given by rational functions. Congruences of order one have been classified in 1866 by Kummer [START_REF] Kummer | Über die algebraischen Strahlensysteme, insbesondere über die der ersten und zweiten Ordnung[END_REF]. Using this classification, we can give a concrete representation of all possible geometric cameras that are described by rational functions. Some of these models represent existing devices (e.g., 'two-slit" [START_REF] Feldman | On the epipolar geometry of the crossed-slits projection[END_REF][START_REF] Pajdla | Geometry of two-slit camera[END_REF], "pushbroom" [START_REF] Gupta | Linear pushbroom cameras[END_REF], "oblique" [START_REF] Pajdla | Stereo with oblique cameras[END_REF] cameras), but many others do not (yet) have physical realizations.

Our framework is also useful for studying the multi-view geometry of general camera models in a unified manner. Taking pictures with n rational cameras for congruences C 1 , . . . , C n defines a rational map

φ : P 3 C 1 × C 2 × • • • × C n ⊂ (Gr(1, P 3 )) n ⊂ (P 5 ) n . (2.1)
Here the congruence C i plays the role of the i-th image plane P 2 in classical multi-view geometry. The closure of the image of φ is a projective variety M (C 1 , C 2 , . . . , C n ) in (P 5 ) n , that we call the multi-image variety. This is analogous to the joint image variety in (P 2 ) n that was defined for pinhole cameras in Section 1.3. In order to characterize M (C 1 , C 2 , . . . , C n ) algebraically, we study the set V n in (Gr(1, P 3 )) n of all n-tuples of lines that are concurrent at a point. Under suitable genericity assumptions, the multiimage variety equals the intersection

(C 1 × C 2 × • • • × C n ) ∩ V n in Gr(1, P 3 ) n ⊂ (P 5 ) n . (2.2)
In particular, for any set of congruences C 1 , . . . , C n , we can recover "multi-view constraints" that describe concurrent visual rays by simply adding the equations defining the C 1 , . . . , C n to those defining V n .

Previous work

Congruences of lines were first studied by classical geometers of the nineteenth century [START_REF] Kummer | Über die algebraischen Strahlensysteme, insbesondere über die der ersten und zweiten Ordnung[END_REF]. They have however remained a popular topic in more modern mathematical works [START_REF] De | Congruences of lines with one-dimensional focal locus[END_REF][START_REF] De Poi | On a class of first order congruences of lines[END_REF][START_REF] Kohn | Secants, bitangents, and their congruences[END_REF]. Many of these papers are interested in differential properties of algebraic varieties (e.g., "focal loci" [START_REF] Arrondo | A focus on focal surfaces[END_REF]).

Introduction

The relationship between classical line congruences and non-central cameras was pointed out by Ponce [START_REF] Ponce | What is a camera?[END_REF], who noted that many existing devices, including two-slit [START_REF] Feldman | On the epipolar geometry of the crossed-slits projection[END_REF][START_REF] Pajdla | Geometry of two-slit camera[END_REF], pushbroom [START_REF] Gupta | Linear pushbroom cameras[END_REF] and oblique [START_REF] Pajdla | Stereo with oblique cameras[END_REF] cameras, could be modeled using a linear congruence. Batog, Goaoc, and Ponce took this further in [START_REF] Batog | Admissible linear map models of linear cameras[END_REF], where they classified all cameras associated with linear congruences, describing them as maps of the form x → x ∨ Ax where A is an appropriate linear transformation of P 3 ( they show that A must have minimal polynomial of degree 2). This representation was previously introduced by Pajdla for the case of oblique cameras [START_REF] Pajdla | Stereo with oblique cameras[END_REF].

The idea of using line geometry and Plücker coordinates for deriving multi-view constraints is not new and, indeed, it is quite common to recover the epipolar relation between cameras by imposing the concurrence of two visual rays (see, e.g., [START_REF] Pless | Using many cameras as one[END_REF]). In [START_REF] Trager | Trinocular Geometry Revisited[END_REF] we used a similar approach to study the case of n = 3 lines. On the other hand, to our knowledge a complete algebraic characterization of the concurrence conditions for general n-tuples of lines has not been proposed or used. In this sense, an interesting related work is by Sturm [START_REF] Sturm | Multi-view geometry for general camera models[END_REF], who deduced multi-view constraints for general camera systems and expressed them using Plücker coordinates of viewing lines. His derivation is very similar to the usual method for obtaining multi-view constraints for pinhole cameras (as presented in Section 1.3). However, Sturm's approach actually provides conditions on the (euclidean) camera motions, rather than on the cameras or congruences themselves. Thus, his equations are not well suited for describing the multi-view geometry of a system of cameras (indeed, the cameras are never actually modeled explicitly), although they can be used to recover camera poses from correspondences.

More works on non-central imaging models will be discussed in the next chapter.

Main contributions

Our main contributions can be summarized as follows.

• We study general "geometric cameras" that are associated with congruences of lines in Gr(1, P 3 ). We use Kummer's classification of congruences of order one [START_REF] Kummer | Über die algebraischen Strahlensysteme, insbesondere über die der ersten und zweiten Ordnung[END_REF] to characterize all rational geometric cameras (Section 2.3.2).

• We study the concurrent lines variety V n , that is the (2n + 3)-dimensional variety of n-tuples of concurrent lines in P 3 . We provide a complete description of the ideal generators associated with V n (Theorem 2.4.1).

• We introduce the multi-image variety M (C 1 , . . . , C n ) for n rational cameras (or congruences) C 1 , . . . , C n . This is analogous to the joint image variety (see Section 1.3), and indeed if each C i is a pinhole camera then the two varieties are isomorphic (Proposition 2.4.5). Assuming that cameras are in general position, an algebraic description of M (C 1 , . . . , C n ) is easily obtained from the generators of the ideal of V n (Theorem 2.4.3).

• We study some geometric cameras of order greater than one, for which point-toline map is algebraic but not rational. In particular, we describe the algebraic geometry of "panoramic" and "catadioptric" cameras (Section 2.5).

Before our main discussion, we briefly review in Section 2.2 some fundamental notions on the geometry of line congruences.

Conventions: Throughout the chapter, we write P n = P(C n+1 ) for the n-dimensional projective space over the complex numbers. This setting is most convenient for dealing with nonlinear algebraic varieties and the corresponding ideals (see Appendix B). On the other hand, our varieties will always be defined by polynomials that have coefficients in the field R of real numbers, and we will be mostly interested in the real locus of these varieties.

Compared to the previous chapter, the distinction between points in P n and their vector representatives will not be as important, so to avoid unnecessary clutter will use regular font to indicate both. For example, if x = [x] is a point in P 3 and A = [A] represents a projective transformation, then we may write Ax in place of Ax. This should not cause any confusion.

Preliminaries

This section reviews some basics on line geometry and congruences that will be central for our discussion. For more details on Plücker coordinates and the Grassmannian Gr(1, P 3 ) we refer the reader to [START_REF] Pottmann | Computational line geometry[END_REF] or to the Appendix A.

The Grassmannian of lines

The set of all lines of P 3 forms a four-dimensional variety, the Grassmannian of lines, denoted by Gr(1, P 3 ). The line passing through two distinct points x and y in P 3 can be represented using Plücker coordinates p = [p 12 , p 13 , p 14 , p 23 , p 24 , p 34 ] where p ij = x i y jx j y i . This defines a point in P 5 that is independent of the choice of x and y along the line. Plücker coordinates of lines satisfy the quadratic Plücker relation

p 12 p 34 -p 13 p 24 + p 14 p 23 = 0. (2.3)
Hence, Gr(1, P 3 ) can be identified with a quadric hypersurface in P 5 . We can also represent a line in P 3 by its dual Plücker coordinates [q 12 , q 13 , q 14 , q 23 , q 24 , q 34 ]. These are the 2 × 2-minors of a 2 × 4-matrix whose kernel is the line. Primal and dual Plücker coordinates are related by q ij = σ (ijkl) p kl , where i, j, k, l are distinct indices, and σ (ijkl) is the sign of the permutation (ijkl).

Preliminaries

To express incidences of lines with points and planes, it is convenient to write the Plücker coordinates of a line p = [p 12 , p 13 , p 14 , p 23 , p 24 , p 34 ] and its dual as the entries of two skewsymmetric 4 × 4-matrices: (2.4)

P =      0 p 34 -
With these definitions, the conditions rank(P ) = 2, rank(P * ) = 2, and trace(P P * ) = 0 are all equivalent to the Plücker relation (2.3). Moreover, the join of a line p and a point x in P 3 is the plane with dual coordinates P x. Similarly, the meet of p with a plane u in (P 3 ) * is the point with coordinates P u. Two lines p and q are concurrent if and only if trace(P Q * ) = trace(QP * ) = 0.

Congruences of lines

A two-dimensional family of lines in P 3 is a surface in Gr(1, P 3 ). This is classically known as a line congruence [START_REF] De | Congruences of lines with one-dimensional focal locus[END_REF][START_REF] De Poi | On a class of first order congruences of lines[END_REF]. Throughout this chapter (and this thesis) we will only consider algebraic congruences, that can be defined by polynomial equations in Plücker coordinates. Any algebraic congruence C is associated with a bidegree, that is a pair of nonnegative integers (α, β). This is analogous to the notion degree for a projective variety. The integer α is known as the order, and is the number of lines in C that pass through a general point of P 3 . The integer β is the class and represents the number of lines in C that lie in a general plane of P 3 . The study of congruences was an active area of research in the second half of the 19-th century. Many results from that period can be found in the book by Jessop [START_REF] Jessop | A Treatise on the Line Complex[END_REF] on line complexes, the classical term for threefolds in Gr(

F(C) = G(C).
Several natural congruences are derived from geometric objects in P 3 . Given a surface X in P 3 , the set of all lines that are tangent to X at two points forms the Geometric Cameras bitangent congruence Bit(X), while the set of lines that are tangent X with order of contact three form the inflectional congruence Infl(X). For a curve Y in P 3 , the set of lines that intersect Y in two points form the secant congruence Sec(Y ). We will return to these objects in Chapter 5 when we will study "aspects" of curves and surfaces in P

Rational Cameras

A geometric camera is a map that associates points in P 3 with "viewing lines" in Gr(1, P 3 ). A natural requirement for this map is that (the closure of) the pre-image of a generic element in Gr(1, P 3 ) is that same line in P 3 . We will also always assume that the coordinates of this mapping are algebraic functions, i.e., functions that can be defined as roots of polynomial equations. This guarantees that the Zariski closure of the image of a geometric camera inside Gr(1, P 3 ) is a line congruence defined by algebraic equations.

In this section, we focus on geometric cameras that are associated with line congruences of order one. We call these rational cameras, since the corresponding map P 3 Gr(1, P 3 ) is a rational map, i.e., its coordinates are homogeneous polynomial functions. Rational cameras are also completely determined by the corresponding congruence. Indeed, if C is a congruence of bidegree (1, β), we can define a map

P 3 C ⊂ Gr(1, 3) (2.6)
that associates with a generic point x in P 3 the unique line in C that passes through x. This map is defined everywhere except at the focal locus of C which, as was noted above, has dimension either one or zero, and coincides with the fundamental locus. We will often write C(x) for the image of x under the map (2.6).

We begin our discussion by analyzing in Section 2.3.1 the two simplest examples of rational cameras, namely pinhole cameras and two-slit cameras. These are "linear" congruences, obtained by intersecting the Plücker quadric with linear subspaces of P 5 . In Section 2.3.2, we provide a complete description of all possible rational cameras, based on a classical theorem of Kummer that classified all congruences of order one. In addition to general formulae, we present explicit examples that illustrate instances of these cameras.

Pinhole and two-slit cameras

We next describe geometric cameras associated with pinhole and two-slit cameras. These correspond to congruences of bidegree (1, 0) and (1, 1), respectively. We will consider rational cameras in full generality in the next subsection.

Pinhole cameras. If C is a (1, 0)-congruence, then C is an α-plane for some point c in P 3 , and (2.6) represents a traditional pinhole camera. The image of a point x is the line with Plücker coordinates

C(x) = x ∨ c =          c 1 x 2 -c 2 x 1 c 1 x 3 -c 3 x 1 c 1 x 4 -c 4 x 1 c 2 x 3 -c 3 x 2 c 2 x 4 -c 4 x 2 c 3 x 4 -c 4 x 3          ∈ Gr(1, P 3 ). (2.7)
Note that there is a complete symmetry between the center c and the projected point x, and if we write C c and C x for the α-planes of lines through c and x respectively, then

{x ∨ c} = C c ∩ C x .
Two slit cameras. A congruence C defined by two general linear forms in Plücker coordinates is a (1, 1)-congruence. To see this, we note that C is the intersection of Gr(1, P 3 ) with a linear space L in P 5 of dimension 3, and that the dual L ∨ of this space is a line in (P 5 ) * that intersects the (dual) Grassmannian in two points p * and q * (over the field C of complex numbers). These two points correspond to lines p and q in P 3 , and C is characterized geometrically as the family of common transversals to p and q. Each point of P 3 outside these two lines lies on a unique such transversal; similarly, a generic plane in P 3 contains exactly one transversal to p and q. The rational camera associated with C is a two-slit camera, and p and q are the two slits. These lines are also the focal locus for C.

As mentioned in Section 2.2, the plane containing a general point x in P 3 and the line p is the point in (P 3 ) * with coordinates x∨p = P x. Likewise, x∨q = Qx is the plane spanned by the point x and the line q. Here P and Q are the skew-symmetric 4 × 4-matrices that Geometric Cameras represent p and q. Intersecting these two planes gives the line in the congruence that contains x. In symbols,

C(x) = (x ∨ p) ∧ (x ∨ q) = P xx T Q -Qxx T P. (2.8)
The coordinates of the Plücker vector C(x) are quadratic in the coordinates of x, and they are bilinear in (p, q). For instance, the first coordinate of C(x), indexed by 12, is equal to (q 24 p 34 -q 34 p 24 )x 1 x 3 + (q 34 p 23 -q 23 p 34 )x 1 x 4 + (q 34 p 14 -q 14 p 34 )x 2 x 3 + (q 13 p 34 -q 34 p 13 )x 2 x 4 +(q 14 p 24 -q 24 p 14 )x 2 3 + (q 23 p 14 -q 14 p 23 -q 13 p 24 + q 24 p 13 )x 3 x 4 + (q 13 p 23 -q 23 p 13 )x 2 4 .

In summary, the picture of x taken with the two-slit camera C is the line given by (2.8).

Example 2.3.1 (Pushbroom cameras). A pushbroom camera [START_REF] Gupta | Linear pushbroom cameras[END_REF] is a device consisting of a linear array of sensors mounted on a platform that can move along a line perpendicular to the sensors (see Figure 2.1). As the platform moves, the camera scans a family of viewing planes. This type of optical system is commonly used in aerial and satellite cameras as well as CT systems. It was observed in [START_REF] Ponce | What is a camera?[END_REF] that pushbroom cameras are two-slit cameras where one of the two slits lies on the plane at infinity. If we identify euclidean 3-space with the affine chart U 1 = {x 1 = 0} then q can be any line of the form q = [0, 0, 0, q 23 , q 24 , q 34 ]. A standard choice is the line at infinity that is orthogonal to p, with respect to the usual scalar product on U 1 R 3 . That line has the Plücker coordinates q = [0, 0, 0, p 14 , -p 13 , p 12 ]. For this choice of q, the polynomial formula (2.8) for the image line C(x) specializes to This Plücker vector represents the picture of the point x taken by the pushbroom camera. ♦ Remark 2.3.2. If C is (1, 1)-congruence defined over the real numbers, then we can distinguish three possibilities for the focal locus: the two lines p and q in F(C) may be real and distinct, real and coincide (when the line L ∨ intersects Gr(1, P 3 ) in a double point), or they may form a complex conjugate pair of lines. In the first case, the (1, 1)congruence C is called hyperbolic. In the second case, C is said to be parabolic, and consists of a one-parameter family of flat pencils of lines centered on the line p = q. In the last case, the focal locus F C has no real points, and the (1, 1)-congruence C is said to be elliptic. The real geometry of linear cameras is discussed in more detail in [START_REF] Batog | Admissible linear map models of linear cameras[END_REF].

Classification of rational cameras

In this section we discuss the classification (1, β)-congruences C for any β, and we derive some explicit formulas for the rational maps x → C(x). Congruences of order one were first classified in 1866 by Kummer [START_REF] Kummer | Über die algebraischen Strahlensysteme, insbesondere über die der ersten und zweiten Ordnung[END_REF]. His result was then refined and extended by various authors in the 20th century. The following version was derived by De Poi in [START_REF] De | Congruences of lines with one-dimensional focal locus[END_REF]. We refer to his article for more information.

F(C

) is a twisted cubic in P 3 , and C consists of its secant lines. Here β = 3.

F(C)

is the union of a rational curve X of degree β and a line L that intersects X in β -1 points. The congruence C is the family of lines that intersects both L and X. Here we allow for degenerate cases: the points in X ∩ L are counted with multiplicity.

4. F(C) is (a non-reduced) line L. The congruence C is described by a morphism φ of degree β > 0 from L ∨ to L, where L ∨ denotes the planes containing L: a line is in C if it belongs to a pencil of lines lying in a plane Π in L ∨ and passing through φ(Π).

We next describe the rational cameras (2.6) for each of these families of congruences.

Type 1: F(C) is a point. This is the pinhole camera described in Section 2.3.1. (2.10) If we augment this ideal by the four entries of P x, where x = [x 1 , x 2 , x 3 , x 4 ] is an unknown world point in P 3 , then the radical of the resulting ideal is generated by the quadrics in (2.10) together with six bilinear equations that can be written in matrix-vector form as follows:

C(x) =         (x 1 x 3 -x 2 2 ) 2 (x 1 x 3 -x 2 2 )(x 1 x 4 -x 2 x 3 ) x 1 x 3 3 + x 3 2 x 4 -3x 1 x 3 x 2 x 4 + x 2 1 x 2 4 (x 2 x 4 -x 2 3 )(x 1 x 3 -x 2 2 ) (x 2 x 4 -x 2 3 )(x 1 x 4 -x 2 x 3 ) (x 2 x 4 -x 2 3 ) 2         . ( 2
         0 0 0 x 4 -x 3 x 2 0 0 0 x 3 -x 2 x 1 0 x 4 -x 3 0 0 x 1 0 x 3 -x 2 -x 2 x 1 0 x 4 0 -x 2 0 x 1 0 x 3 -x 2 0 x 1 0 0                   p 12 p 13 p 14 p 23 p 24 p 34          =          0 0 0 0 0 0          . (2.11)
This 6 × 6 matrix has rank 5. The solution space of (2.11) is spanned by the vector in (2.9).

The twisted cubic camera (2.9) has a nice interpretation in terms of tensor decompositions. For this, we identify P 3 with the space of symmetric 2 × 2 × 2-tensors. We seek to decompose an arbitrary tensor as the sum of two rank 1 tensors. Equivalently, we seek to write a binary cubic x 1 u 3 + 3x 2 u 2 v + 3x 3 uv 2 + x 4 v 3 as the sum of two cubes of linear forms in u and v. Rank 1 tensors correspond to points on the twisted cubic curve. The desired representation is unique, and is given by the intersection points of the twisted cubic with the secant line C(x).

Example 2.3.4. The binary cubic 3uv 2 + 3u 2 v corresponds to the point [0, 1, 1, 0] in P 3 . Its image under (2.9) is the line [1, 1, 0, 1, 1, 1]. This line intersects the twisted cubic at

[-2, -1+ √ -3, 1+ √ -3, 2] and [-2, -1- √ -3, 1- √ -3, 2] in P 3 . From this we deduce 3uv 2 + 3u 2 v = √ -3 3 u + 1 + √ -3 2 v 3 - √ -3 3 u + 1 - √ -3 2 v 3 .
(2.12)

♦ Type 3: F(C) is a rational curve X and a line L. After a change of coordinates we may assume that the line is

L = {[0, 0, x 3 , x 4 ] ∈ P 3 : [x 3 , x 4 ] ∈ P 1 }.
The dual line L ∨ parametrizes all planes in P 3 that contain L. A natural parametrization P 1 → L ∨ is given by identifying [x 1 , x 2 ] with the plane in P 3 with dual coordinates [x 2 , -x 1 , 0, 0].

To build our rational camera, we take an arbitrary rational curve X of degree β that intersects L in β -1 points. Each such curve X is given by a parametric representation

P 1 → X, [s, t] → sf (s, t), tf (s, t), g(s, t), h(s, t) , (2.13)
where f, g and h are arbitrary binary forms of degree β -1, β and β respectively.

Proposition 2.3.5. For the rational camera of Type 3, the map (2.6) is given by

C(x) =      x 1 x 2 x 3 x 4      ∨      x 1 f (x 1 , x 2 ) x 2 f (x 1 , x 2 ) g(x 1 , x 2 ) h(x 1 , x 2 )      . ( 2

.14)

Proof. The two column vectors in (2.14) represent two points in P 3 that lie on the plane in L ∨ with coordinates [x 1 , x 2 ], according to the parametrization above. We see from (2.13) that the second point lies on the curve X. Hence (2.14) is a line the intersects both L and X.

Remark 2.3.6. The ideal of the congruence C can be described in terms of Chow forms. We recall that Chow form Ch Z of an irreducible curve Z of degree γ in P 3 is a hypersurface of degree γ in the Grassmannian Gr(1, P 3 ), containing all lines in P 3 that intersect Z (see Section 1.5.2). The ideal of C is in fact the saturation of Ch L , Ch X with respect to ∩ β-1 i=1 P u i , where u i are the intersections between L and X, and Ch L and Ch X are the Chow forms of L and X respectively. Hence the ideal Ch L , Ch X represents all lines that intersect both L and X. The saturation removes β -1 extraneous components, namely the (1, 0)-congruences of lines passing through the points u i . We conjecture that the resulting ideal is generated by the Plücker quadric, the linear Chow form Ch L , and Geometric Cameras β linearly independent forms of degree β (including Ch X ). This description was observed experimentally.

Example 2.3.7. Fix β = 3 and let X be the twisted cubic curve given as in (2.13) with f = (s -t)(s + t), g = s 3 and h = t 3 . The ideal of X is generated by the 2 × 2-minors of

x 2 + x 4 x 3 -x 1 x 4 x 3 x 2 + x 4 x 3 -x 1 .
(2.15)

The line L = V (x 1 , x 2 ) meets the curve X in the two points [0, 0, 1, 1] and [0, 0,

Multi-view Geometry

In this section we study the multi-view geometry of systems of rational cameras. For this, we first investigate in Section 2.4.1 the conditions for multiple lines to be all concurrent at a single point, deriving a complete algebraic characterization of the concurrent lines variety. We apply this result in Section 2.4.2 to describe multi-view correspondences for arbitrary rational cameras.

The concurrent lines variety

The concurrent lines variety V n consists of ordered n-tuples of lines in P 3 that meet in a point x. The lines containing a fixed x form a linear space of constant dimension 2 (the α-plane for x). From this we infer that V n is irreducible of dimension 2n + 3, provided n ≥ 2. We regard V n as a subvariety in the product of projective spaces (P 5 ) n .

The following result fully characterizes the prime ideal Our proof rests on computations with the computer algebra system Macaulay2 [START_REF] Daniel | Macaulay2, a software system for research in algebraic geometry[END_REF].

I n = I(V n ) of
Proof. The case n = 2 is easy. We begin with n = 3. Let P, Q, R be skew-symmetric 4×4matrices representing three lines. These matrices have rank 2. The Plücker quadrics are trace(P P * ) = trace(QQ * ) = trace(RR * ) = 0.

(2.17)

Furthermore, the three lines are pairwise concurrent if and only if

trace(P Q * ) = trace(P R * ) = trace(QR * ) = 0. (2.18)
Using a computation with Macaulay2, we find that the ideal generated by the six quadrics in (2.17) and (2.18) is radical. It is the intersection of two prime ideals, each minimally generated by ten cubics in addition to (2.17) and (2.18). The first prime represents triples of lines that are coplanar, and is an extraneous component for us. The second prime is the concurrent lines variety. The cubic generators of that second prime ideal are the 3 × 3 minors of the 4 × 3 matrix (P u, Qu, Ru), where u is a column vector in R 4 . These span a ten-dimensional space of cubics. A basis for that space is obtained by selecting the vector u from the set

e 1 ,

Geometric Cameras

The initial monomial ideal is generated by the leading terms in the reduced Gröbner basis:

M 3 = p 14 p 23 , q 14 q 23 , r 14 r 23 , p 34 q 12 , p 34 r 12 , q 34 r 12 , p 23 q 13 r 12 , p 23 q 14 r 12 , p 23 q 14 r 23 , p 23 q 14 r 13 , p 24 q 13 r 12 , p 24 q 14 r 12 , p 24 q 14 r 23 , p 24 q 14 r 13 , p 34 q 14 r 13 , p 34 q 14 r 23 , p 34 q 24 r 13 , p 34 q 24 r 23 , p 23 q 13 q 24 r 13 , p 23 q 13 q 24 r 23 , p 24 q 13 q 24 r 13 , p 24 q 13 q 24 r 23 .

(2. [START_REF] Bottino | The visual hull of piecewise smooth objects[END_REF] This shows that the reduced Gröbner basis consists of 6 quadrics, 12 cubics and 4 quartics. All 22 leading terms are squarefree. This completes the proof of Theorem 2.4.1 for n = 3.

We next consider the case n = 4. A Macaulay2 computation verifies that Theorem 2.4.1 is true here. The ideal I 4 is minimally generated by the 10 quadrics trace(P i P * j ) together with 40 = 10 4 3 cubics, namely the 10 cubics from I 3 for any three of the four lines. The initial ideal M 4 = in(I 4 ) is minimally generated by 10 quadratic monomials, 48 = 12 4 3 cubic monomials, and 20 = 4 5 4 quartic monomials. The quadrics and cubics come from M 3 for any three of the four lines. Among the quartics are the 16 = 4 4 3 quartics from M 3 for any three of the four lines. However, the reduced Gröbner basis of I 4 now also contains four quadrilinear forms. These contribute four new generators of the monomial ideal M 4 : p 23 q 13 r 24 s 13 , p 23 q 13 r 24 s 23 , p 24 q 13 r 24 s 23 , p 24 q 13 r 24 s 13 .

(

We next assume n ≥ 5. We write G n for the union of the various reduced Gröbner bases, obtained from I 4 for any four of the n lines. The set G n has n+1 We claim that G n is the reduced Gröbner basis for the ideal G n it generates. This can be verified computationally with Macaulay2 for n ≤ 7. For n ≥ 8, we argue as follows. Consider any two polynomials in G n . We must show that their S-polynomial reduces to zero upon division with respect to G n . If their leading monomials are relatively prime then this is automatic, by Buchberger's First Criterion [START_REF] David A Cox | Ideals, Varieties, and Algorithms: An Introduction to Computational Algebraic Geometry and Commutative Algebra[END_REF]. Otherwise, the leading monomials have a Plücker variable in common. This means that at most seven of the n lines are involved in the two polynomials. But then their S-polynomial reduces to zero because the Gröbner basis property is already known for n ≤ 7. A similar argument shows that no trailing term in G n is a multiple of an leading term. Hence G n is the reduced Gröbner basis for its ideal.

The minimal generators of the ideal G n are obtained from the minimal generators of I 4 , for any four of the n lines. Hence G n is generated by the n+1 2 quadrics and the Finally, we must prove that the ideal G n equals the ideal I n we are interested in. By construction, all generators of G n vanish on the concurrent lines variety

V n = V (I n ). Therefore, G n ⊆ I n . (2.22)
Moreover, the initial ideal of G n is radical, and hence G n is a radical ideal. To complete the proof, all we now need is that the set G n cuts out the variety V n set-theoretically. This is equivalent to the statement that n ≥ 4 distinct lines in P3 are concurrent if and only if any three of the n lines are concurrent. This is indeed the case.

Remark 2.4.2. Suppose all P i satisfy the Plücker constraint trace(P i P * i ) = 0. The four 3×3-minors of P i u, P j u, P k u are scalar multiples of a single trilinear polynomial t u that expresses the condition for the planes u∨p i , u∨p j and u∨p k to be linearly dependent,i.e., for p i , p j , p k to admit a transversal line passing through u: this is exactly the trilinear relation t p considered in Section 1.5.1. In fact, as argued in Proposition 1.5.2, three lines are concurrent if and only if they are pairwise coplanar and they admit a transversal not contained in the planes defined by any two of them. From this we deduce that V n is cut out set-theoretically by n+1 2 bilinear quadrics trace(P i P * j ) and the 4 n 3 trilinear cubics T u where u runs over only {e 1 , e 2 , e 3 , e 4 }. This is confirmed by computation with Macaulay2.

The multidegree of V n in (P 5 ) n is a homogeneous polynomial of degree 3n -3 in n unknowns t 1 , t 2 , . . . , t n (see Section 1.5.2). Using the built-in command multidegree in Macaulay2, we found experimentally that the multidegree of the concurrent lines variety is the polynomial

[V n ] = (t 1 t 2 t 3 • • • t n ) 3 • 4 (i,j) t -2 i t -1 j + 8 {i,j,k} t -1 i t -1 j t -1 k . (2.23)
The first sum is over ordered pairs (i, j) with i = j; the second sum is over unordered triples {i, j, k}. After completion of our paper [START_REF] Ponce | Congruences and concurrent lines in multi-view geometry[END_REF], Laura Escobar and Allen Knutson [START_REF] Escobar | The multidegree of the multi-image variety[END_REF] found a proof for the formula (2.23). Their derivation rests on methods from representation theory.

Multi-image varieties

In this section, we use the concurrent lines variety V n to characterize multi-view correspondences for n rational cameras. We fix congruences C 1 , . . . , C n ⊂ Gr(1, P 3 ), where C i has bidegree (1, β i ) for some β i ∈ N. Combining their maps as in (2.6) gives

P 3 C 1 × • • • × C n , x → C 1 (x), . . . , C n (x) . (2.24)
The base locus of this rational map (i.e., its locus of indeterminacy) is the product of the focal loci,

F(C 1 ) × • • • × F(C n ).
We define the multi-image variety M (C 1 , . . . , C n ) to be the closure of the image of (2.24). This is an irreducible subvariety in the product of Grassmannians Gr(1, P 3 ) n ⊂ (P 5 ) n . We expect the map (2.24) to be birational in almost all cases, so M (C 1 , . . . , C n ) is a threefold.

The multi-image variety is clearly contained in the concurrent lines variety. In symbols,

M (C 1 , . . . , C n ) ⊆ V n ∩ (C 1 × • • • × C n ) ⊂ Gr(1, P 3 ) n . (2.25)
Our first result in this section shows that the left inclusion in (2.25) is usually an equality.

Theorem 2.4.3. Suppose that the n focal loci F(C 1 ), . . . , F(C n ) are pairwise disjoint. Then

M (C 1 , . . . , C n ) = V n ∩ (C 1 × • • • × C n ), (2.26)
i.e., the concurrent lines variety gives an implicit representation of the multi-image variety.

Proof. By (2.25), we only need to show one direction. For any n-tuple of lines (L Geometrically, we partition the set of six coordinate lines in P 3 into three pairs of disjoint lines (see Figure 2.3). Each pair defines a (1, 1)-congruence. Note that F(C 1 ), F(C 2 ) and F(C 3 ) are distinct, but they intersect in the four coordinate points. So, the hypothesis of Theorem 2.4.3 fails.

1 , . . . , L n ) in V n ∩ M (C 1 , . . . , C n ),

Multi-view Geometry

The ideal J + I 3 is radical but not prime. It is the intersection of five primes, each defining a threefold in P 5 ×P 5 ×P 5 . One of these is the variety M (C 1 , C 2 , C 3 ), with ideal p 14 p 23 -p 13 p 24 , q 14 q 23 + q 12 q 34 , r 13 r 24 -r 12 r 34 , p 14 q 23 + p 23 q 14 , p 24 r 13 + p 13 r 24 , q 34 r 12 + q 12 r 34 , p 23 q 34 r 24 + p 24 q 23 r 34 , p 13 q 34 r 24 + p 14 q 23 r 34 , p 14 q 23 r 24 + p 24 q 12 r 34 , p 13 q 23 r 24 + p 23 q 12 r 34 , p 13 q 14 r 24 -p 14 q 12 r 34 , p 14 q 34 r 13 + p 13 q 14 r 34 , p 14 q 23 r 13 -p 13 q 12 r 34 , p 14 q 23 r 12 -p 13 q 12 r 24 , p 13 q 23 r 12 + p 23 q 12 r 13 , p 24 q 14 r 12 + p 14 q 12 r 24 + J.

The other four associated primes define coordinate 3-planes in P 5 × P 5 × P 5 . They are p 23 , p 24 , q 23 , q 34 , r 24 , r 34 + J, p 13 , p 14 , q 14 , q 34 , r 13 , r 34 + J, p 14 , p 24 , q 12 , q 14 , r 12 , r 24 + J, p 13 , p 23 , q 12 , q 23 , r 12 , r 13 + J.

To understand the geometric meaning of these extraneous components, consider the last ideal. It represents all triples (L 1 , L 2 , L 3 ) where L 1 , L 2 , L 3 pass through [0, 0, 0, 1], and each line L i intersects one of the opposite coordinate lines, as is required for lines in C i . ♦ In the following we always consider congruences whose focal loci are pairwise disjoint, so the identity (2.26) holds. We present a detailed study of two special cases. First we allow arbitrary n, but assume β i ∈ {0, 1}. We then focus on n = 2, but with arbitrary β 1 and β 2 .

Multiple Views with Pinhole and Two-Slit Cameras. We begin with the most classical case, where C 1 , . . . , C n are pinhole cameras with distinct centers c 1 , . . . , c n . Each congruence C i is a plane in P 5 , and the map x → C i (x) = x ∨ c i is analogous to the linear projection P 3 P 2 with center c i . In the usual set-up with pinhole cameras, this map is represented by a 3 × 4 matrix A i whose null-space is given by c i , and P 2 is identified with the image of A i . Since A i and x → x ∨ c i have the same null-space, there exists a 6 × 3 matrix B i such that x ∨ c i = B i A i x; this is of course simply the "inverse projection matrix" from Chapter 1, Section 1.2.5.

Proposition 2.4.5. The joint image variety (see Definition 1.3.4) of the pinhole cameras

A 1 , . . . , A n is isomorphic to the multi-image variety M (C 1 , . . . , C n ) under the map (P 2 ) n → Gr(1, P 3 ) n , (u 1 , . . . , u n ) → (B 1 u 1 , . . . , B n u n ).
Here, the equation (2.26) holds ideal-theoretically, i.e., the prime ideal of the multi-view variety is the image of I n modulo the linear equations

P 1 c 2 = • • • = P n c n = 0 that define C 1 × • • • × C n .
Proof. The first statement is immediate from the discussion of the two realizations of P 2 , as the image of A i or as the plane C i in P 5 . The second statement about ideals is more subtle. It can be derived using the functorial set-up developed by Li [START_REF] Li | Images of rational maps of projective spaces[END_REF].

The ideal I n of the concurrent lines variety V n is minimally generated by n+1 2 quadrics and 10 n 3 cubics in the 6n Plücker coordinates. We add to this the 3n linear equations that define cubics in the 6n 1 + 6n 2 Plücker coordinates on the ambient space (P 5 ) n 1 +n 2 .

C 1 × • • • × C n .
Note that for n 2 = 0 we recover the known ideal generators of the joint image variety (Theorem 1.3.10).

Proof. The desired ideal is obtained from I n by adding 3 linear forms for every pinhole camera C i and 2 linear forms for every two-slit camera C i . We need to examine the extent to which the generators of I n become linearly dependent modulo these 3n 1 + 2n 2 linear forms. For n ≤ 3 cameras this examination amounts to computations with Macaulay2, one for each ordered partition (n 1 , n 2 ) of n. For n ≥ 4 cameras we group the minimal generators of I n according to their degree in the Z n -grading. Each graded component specifies a subset of cameras of size at most three. Hence all the linear relations arise from those for n = 3.

Epipolar Geometry for Rational Cameras. We now take a closer look at the case of two rational cameras C 1 and C 2 . We assume that C i is a congruence of bidegree (1, β i ) for i = 1, 2 and that F(C 1 ) ∩ F(C 2 ) = ∅. The associated multi-image variety M (C 1 , C 2 ) in P 5 × P 5 is defined by the ideal

I P (C 1 ) + I Q (C 2 ) + trace(P Q * )
⊂ C[p 12 , . . . , p 34 , q 12 , . . . , q 34 ],

(2.27)

where I P (C 1 ) and I Q (C 2 ) are respectively the ideals for C 1 and C 2 in the two sets of variables.

This set-up generalizes familiar objects from two-view geometry. For example, if p is a line in the congruence C 1 , then the epipolar curve Epi(p) in C 2 consists of all lines q such that (p, q) belongs to M (C 1 , C 2 ). The ideal of Epi(p) in the P 5 with coordinates q 12 , . . . , q 34 is given by I Q (C 2 ) + trace(P Q * ) . The curve Epi(p) has degree 1 + β 2 in Plücker coordinates (see Proposition 2.4.7 below). In particular, for pinhole cameras C 1 , C 2 , we recover the classical epipolar lines in two-view geometry [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. However there is an important (and well known [START_REF] Sturm | Camera models and fundamental concepts used in geometric computer vision[END_REF]) distinction compared to the classical case: if either C 1 or C 2 is not a pinhole cameras, then the families of curves Epi 12 = {Epi(p) : p ∈ C 1 } and Epi 21 = {Epi(q) : q ∈ C 2 } are not in general related by a one-to-one correspondence. More concretely, if q and q both belong to Epi(p), then we cannot conclude that Epi(q) = Epi(q ). This follows from the fact that the ideal from Theorem 2.4.6 is not multilinear, and contrasts with the case of pinholes, where there exists a homography relating the P 1 of epipolar lines in each image.

In traditional two-view geometry, the two camera centers in P 3 span the "baseline", which projects onto the two epipoles. This generalizes as follows to our setting. A line L in P 3 is a baseline for the two cameras C 1 and C 2 if it lies in the intersection C 1 ∩ C 2 in Gr(1, P 3 ). The baselines are precisely the loci that are contracted by the map (2.24), since for every point x in such a line L we have (C 1 (x), C 2 (x)) = (L, L). We expect C 1 ∩ C 2 to consist of finitely many points. Some of these points are defined over C, and are included in the following count. The epipolar curves Epi(p) and Epi(q) in Gr(1, P 3 ) have degrees 1 + β 2 and 1 + β 1 respectively. The number of baselines in P 3 for the camera pair

(C 1 , C 2 ) equals 1 + β 1 β 2 .
Proof. The intersection theory in the Grassmannian Gr(1, P 3 ) ⊂ P 5 works as follows. A hypersurface of degree d intersects an (α, β)-congruence in a curve of degree dα + dβ. Two congruences of bidegrees (α 2 , β 1 ) and (α 3 , β 2 ) intersect in α 2 α 3 + β 1 β 2 points. A classical reference is Jessop's book [START_REF] Jessop | A Treatise on the Line Complex[END_REF]. A modern one is any introduction to Schubert calculus.

We now illustrate these concepts with an example.

Example 2.4.8. Let β 1 = β 2 = 2 and consider the type 3 congruences C 1 and C 2 of common transversals to L 1 , X 1 and L 2 , X 2 , where

L 1 = V (x 2 , x 3 -x 4 ), X 1 = V (x 1 , x 2 2 + x 2 3 -x 2 4 ), L 2 = V (x 1 -x 2 , x 3 ), X 2 = V (x 2 1 -x 2 2 + x 2 3 , x 4 ). Note that that the intersection of F(C 1 ) = L 1 ∪ X 1 and F(C 2 ) = L 2 ∪ X 2 in P 3 is empty. The intersection points on the two focal loci are L 1 ∩ X 1 = {[0, 0, 1, 1]} and Geometric Cameras L 2 ∩ X 2 = {[1, 1, 0, 0]}.
The prime ideals of the two congruences in the coordinate ring of Gr(1, P 3 ) are given by If we fix a point p in C 1 then its corresponding cubic curve Epi(p) lives in C 2 , and vice versa. For example, the ideal I P (C 2 ) + 3p 12 + 4p 13 + 5p 14 -3p 23 -3p 24 + p 34 defines the epipolar curve in Gr(1, P 3 ) associated with p = [3, 4, 5, -3, -3, 1] in C 1 .

I P (C 1 ) = p 23 -
The ideal I P (C 1 ) + I P (C 2 ) defines five points in Gr(1, P 3 ), These represent the five baselines. One point is [0, 1, 1, 1, 1, 0]. It represents the line through [0, 0, 1, 1] and [1, 1, 0, 0]. The other four baselines have the Plücker vectors

5 2 a 2 - 1 2 , a, - 5 2 a 3 - 1 2
a, a, a, 1 where 5a 4 -2a 2 + 1 = 0.

We see that three of the five baselines are real, while the other two are defined over C. ♦

Higher-Order Cameras

In Sections 2.3.2 and 2.4.2 we considered congruences whose point-to-line maps x → C(x) are rational. However, there are important examples of real-world cameras that are associated with congruences of higher order (see e.g., [START_REF] Sturm | Camera models and fundamental concepts used in geometric computer vision[END_REF]). For example, a (2, β)-congruence associates a given point x with a pair of lines, but a physical camera might record only one line for x, because of orientation constraints. In this section, we study the geometry in Gr(1, P 3 ) for two types of devices that exist in practice, namely "non-central panoramic" and "catadioptric cameras". We consider here idealized and generalized versions of these models. For example, we describe catadioptric systems that use an arbitrary smooth algebraic surface as a mirror, and a general geometric camera to record lines.

Panoramic Cameras

A panoramic camera enables photographs with a 360 • field of view. One such panoramic device consists of a 1D-sensor measuring 2D-projections onto a fixed center, that is rotated about a vertical axis L not containing the center. The 1D-sensor travels on a circle X around the line L. The associated congruence C consists of all lines that intersect both L and X, and has bidegree (2, 2). Physical realizations come in two versions, depending on the orientation of the sensor (which causes precisely one of the Another system is the stereo panoramic camera, on the right in Figure 2.4. It is obtained by rotating a 1D-sensor about an axis parallel to the sensor. In each position the sensor records parallel lines tangent to the rotation. This is a variation of the "omnivergent" camera proposed in [START_REF] Shum | Omnivergent stereo[END_REF]. It produces stereo (binocular) panoramic images, since every 3D-point is observed from two sensor locations. Both of the cameras shown in Figure 2.4 are discussed in [START_REF] Sturm | Camera models and fundamental concepts used in geometric computer vision[END_REF].

We first discuss the non-central panoramic camera. The corresponding (2, 2)-congruence C is determined by a line L and a non-degenerate conic X, both in P 3 , such that L∩X = ∅. Then, as above, C consists of all lines in P 3 that intersect both L and X. Proof. Given any two pairs of disjoint conics and lines (X, L) and (X , L ) in P 3 , we may always apply a homography over C so that

X = X = V (x 1 , x 2 2 + x 2 3 + x 2 4
). Transformations that fix X are projectivizations of affine maps C 3 → C 3 , x → Ax + b such that AA T = λId, where x = (x 2 /x 1 , x 3 /x 1 , x 4 /x 1 ) are coordinates on the affine chart U 1 = {x 1 = 0}. These maps act transitively on points of U 1 and on points of V (x 1 )\X, so we conclude that L and L are equivalent. If we restrict to R, and both conics have real points, then we use

X = X = V (x 1 , x 2 2 + x 2 3 -x 2 4
), and a similar result holds.

Thanks to Lemma 2.5.1, we may choose

I L = x 2 , x 3 and I X = x 4 , x 2 2 + x 2 3 -x 2 1
as the ideals that represent the line L and the conic X. Proof. We compute the focal locus as described at the end of Section 2.2. For the congruence C given in (2.28), the quartic generator is found to be (x 2 2 + x 2 3 )x 2 4 . Remark 2.5.3. It is known that the focal locus of a general smooth (2, 2)-congruence (defined by a general linear form and a general quadratic form in Plücker coordinates) is a Kummer surface: this is a classical algebraic surface of degree four with 16 nodes. However, the congruence C in (2.28) is singular: its singular locus, V (p 14 , p 23 , p 24 , p 34 ), consists in of all lines in P 3 that meet L and lie in the plane spanned by X. The Kummer surface degenerates to the arrangement of four planes F(C) in Proposition 2.5.2.

We now discuss the stereo panoramic camera. Its congruence consists of the lines that are tangent to a singular quadratic surface Q and pass through a fixed line L. For the camera on the right in Figure 2.4, the quadric Q is a cylinder around the axis and L is a line at infinity. Interestingly, we note that the stereo panoramic camera is dual, in the sense of projective geometry, to the non-central panoramic camera. Specifically, its congruence C * is obtained by dualizing C in (2.28). The result is Here I L = x 1 , x 4 and

I C * = p 14 , p
I Q = x 2 1 -x 2 2 -x 2 3 .
Clearly, C * has bidegree (2, 2). As in Remark 2.5.3, C * is singular along a line. Singular points are lines that meet L and the cone point of Q. Proposition 2.5.4. The focal locus F(C * ) of the stereo panoramic camera consists of the singular quadric Q and the plane at infinity (spanned by L and the cone point of Q), which is taken with multiplicity 2. Algebraically, it is defined by the non-reduced quartic

x 2 1 (x 2 1 -x 2 2 -x 2 3 ).
Proof. This is verified by a computation, like Proposition 2.5.2.

Catadioptric Cameras

A catadioptric camera is an optical device that makes use of reflective surfaces (catoptrics) and lenses (dioptrics). For many applications it is desirable to have a single "effective viewpoint" [START_REF] Baker | A theory of single-viewpoint catadioptric image formation[END_REF]. This can be achieved by using a mirror that is a paraboloid or hyperboloid of revolution, placing a pinhole camera at one of the foci. In our setting, it is

Higher-Order Cameras natural to consider a more general catadioptric system that uses an arbitrary smooth algebraic surface S as a mirror, and a geometric camera C to record lines. We will describe the resulting line congruence in Gr(1, P 3 ).

We measure angles in P 3 according to the usual scalar product in

U 1 = {x 1 = 0}, so that cos ∠(x, y) = x 2 y 2 + x 3 y 3 + x 4 y 4 (x 2 2 + x 2 3 + x 2 4 )(y 2 2 + y 2 3 + y 2 4 )
.

(2.30)

The points x = [0, x 2 , x

(y) =          3 i=1 a 2 i y 1 3 i=1 a 2 i y 2 -2 3 i=0 a i y i a 2 3 i=1 a 2 i y 3 -2 3 i=0 a i y i a 3 3 i=1 a 2 i y 4 -2 3 i=0 a i y i a 4          . (2.31)
This map is a linear involution of P 3 that fixes H, and whose restriction to the real affine 3-space U 1 is the usual Euclidean reflection with respect to H ∩ U 1 . The reflection can also be applied to lines, by defining the image of p = x ∨ y as ρ H (x) ∨ ρ H (y). This is a linear involution of the Grassmannian Gr(1, P 3 ), defined by the 6 × 6 compound matrix

C 2 (ρ H ) of ρ H .
We now consider a smooth algebraic surface S in P 3 , defined by a polynomial f of degree d. We say that two lines L and L in P 3 are specular for S if there exists a point x ∈ S such that the tangent plane T x (S) is not isotropic, L and L meet in x, and they are reflections of each other respect to T x (S). We define the mirror variety M S to be the closure of the set of all pairs (L, L ) ∈ Gr(1, P 3 ) 2 that are specular for S. For a general line L there are d lines L such that (L, L ) ∈ M S , one for each point x in S ∩ L. Hence the mirror variety M S of a surface S is 4-dimensional.

To compute the defining equations of the mirror variety M S , we first construct the ideal J = f, P x, trace(P P * ), trace(QQ

* ) + ∧ 2 q | ρ TxS (p) .
This lives in R[x 1 , x 2 , x 3 , x 3 , p 12 , . . . , p 34 , q 12 , . . . , q 34 ]. The last summand is the ideal of 2 × 2-minors of a 6 × 2-matrix with columns q and ρ TxS (p), where ρ TxS (p) denotes the reflection of the line p with respect to the tangent plane T x (S). These constraints express the requirement that q is equal to that reflection. We then saturate J with respect to the isotropic ideal

I Iso = (∇ x f ) 2 1 + (∇ x f ) 2 2 + (∇ x f ) 2 3
and with respect to the irrelevant ideal x 1 , x 2 , x 3 , x 4 , before eliminating the variables x 1 , x 2 , x 3 , x 4 . Example 2.5.5. Let S be the ellipsoid given by f = 1 16 (x

2 2 + x 2 3 ) + 1 25 x 2 4 -x 2 1 .
The mirror variety M S has codimension 6 in P 5 × P 5 . Using the approach outlined above, we recover the ideal associated with M S . We observe that its bidegree (see Appendix B) is

4t 5 0 t 1 + 12t 4 0 t 2 1 + 18t 3 0 t 3 1 + 12t 2 0 t 4 1 + 4t 0 t 5 1 . ♦ Remark 2.5.6.
The intersection of the mirror variety M S with the diagonal ∆ in P 5 ×P 5 is the normal congruence. These are the lines that intersect S orthogonally (we assume that we have removed components associated with isotropic tangent planes). The focal locus of the normal congruence is the caustic surface [START_REF] Catanese | Focal loci of algebraic varieties I[END_REF]. In the language of differential geometry, this is the union of the centers of principal curvature for S. Moreover, the order α of the normal congruence coincides with the Euclidean distance degree (ED degree) of the surface S. That is the number of critical points on S of the squared distance function to a generic point [START_REF] Draisma | The Euclidean distance degree of an algebraic variety[END_REF].

Let C be any congruence, representing a geometric camera. The specular congruence of C with respect to S is another surface C S in the Grassmannian Gr(1, P 3 ). We define C S as the closure of the set of all lines L for which there exist L ∈ C and x ∈ L ∩ S such that T x S is not isotropic and L = ρ TxS L. Concretely, C S contains the lines of C after these are reflected by S (see Figure 2.5.) Thus, C S is the congruence associated with the catadioptric camera determined by S and C. Note that if L is in C then there exists L such that (L, L ) ∈ M S . This implies

C S ⊆ C S := π 2 M S ∩ (C × Gr(1, P 3 )) , (2.32)
where π 2 is the projection onto the second factor. For a general C, an appropriate application of Bertini's Theorem [START_REF] Harris | Algebraic geometry: a first course[END_REF] ensures that the right hand side C S is irreducible, and the containment (2.32) is an equality (set-theoretically). In this case, we can compute equations for C S by adding the equations defining the given congruence C (in the variables p 12 , . . . , p 34 ) to the ideal of the mirror variety M S , then saturating by the irrelevant Conclusions ideal p 12 , . . . , p 34 , and finally eliminating the variables p 12 , . . . , p 34 . We experimented with this in Macaulay2.

The next example shows that C S can have spurious components. These are removed by saturating the ideal of C S with respect to the Chow form Ch X Iso where X Iso = V (I Iso + f ). We note that the order and class of the specular congruence C S depend on the relative position of S and C (and the absolute quadric

V (x 1 , x 2 2 + x 2 3 + x 2 4
)). The focal locus of C S is a caustic by reflection in [START_REF] Josse | On caustics by reflection of algebraic surfaces[END_REF], but here we do not require for the light source to be a point.

Example 2.5.7. Let S be the ellipsoid from Example 2.5.5. We first consider a catadioptric camera with mirror S and a pinhole sensor at a point P . Let us start with P = [1, 0, 0, 3]. The radical ideal of C S (computed as described above) is the intersection of two prime ideals:

I 1 = q 23 ,
3q 13 -q 34 , 3q 12 -q 24 I 2 = q 14 q 23 -q 13 q 24 + q 12 q 34 , 625q 2 12 + 625q 2 13 + 256q 2 14 + 150q 12 q 24 + 9q 2 24 + 150q 13 q 34 + 9q 2 34 .

We observe that I 2 is a component of Ch X Iso . It is extraneous for us. More precisely, X Iso contains two quadric curves on S, and I 2 is generated by the Plücker quadric and the Chow form of one of these curves. On the other hand, I 1 is the ideal of C S . This (1, 0)-congruence is the α-plane for Q = [1, 0, 0, -3]. The points P and Q are the two foci of the ellipsoid S.

If we choose P randomly, then C S = C S . Using the computation explained above, we find that the bidegree of the specular congruence C S is [START_REF] Åström | Generalised epipolar constraints[END_REF][START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF]. According to Josse and Pène [START_REF] Josse | On caustics by reflection of algebraic surfaces[END_REF], the focal locus F(C S ), which is the caustic by reflection of S, is a surface of degree 18.

Finally, we consider the catadioptric camera given by S together with a general (1, 1)congruence (two-slit camera). The resulting specular congruence has bidegree [START_REF] Guenther | Geometric modeling for computer vision[END_REF][START_REF] Arrondo | Introduction to projective varieties[END_REF]. ♦

In closing, we wish to reiterate that the notion of order used in this chapter is always the algebraic one. The "physical" order of a catadioptric camera may be quite a bit lower, due to orientation constraints, with some of the rays reflected inside the body of the mirror.

Conclusions

We have studied in this chapter a model of geometric cameras that represents an imaging system as a line congruence inside Gr(1, P 3 ). Much of our exposition focused on cameras that are associated with congruences of order one, and that can be used to define rational projection mappings. In particular, we have classified all possible such cameras, and studied their multi-view geometry, by characterizing algebraically n-tuples of lines that are concurrent at a point. We have also described the geometry of some devices that are associated with congruences of higher order, namely panoramic and catadioptric cameras.

While our treatment may appear as very abstract, we believe that our approach is useful for several reasons. First, our framework can be used to compare and classify general imaging models, and to deduce geometric properties of camera systems that are independent of parameterizations and choices of image coordinates. For example, the degree of "epipolar curves", or the number of "baselines", are features that can be understood using the geometry of congruences, as explained in Section 2.4.2. More concretely, our results on multi-view correspondences can be carried over to image coordinates, and thus provide a methodology for characterizing multi-view correspondences for very general systems of cameras.

Further work could be devoted to non-rational cameras, which we did not study in full generality. For example, it could be useful to gain a better understanding of the action of reflections and refractions on congruences of lines. As discussed in Section 2.5.2, mirrors or lenses act on a congruence C by mapping it to another congruence C . Although we described a method for computing C given C, we were not able to establish a general relation between the bidegrees of these congruences.

Finally, we mention another direction that we think is worth pursuing, that is to relate our framework with (some version of) the plenoptic function [START_REF] Edward | The plenoptic function and the elements of early vision[END_REF]. This function describes the radiance along all rays in a region of three-dimensional space. Mathematically, we can model it as a mapping F : Gr(1, P 3 ) → R or F : Gr(1, P 3 ) → P1 . 1 In these terms, taking a picture with any type of camera means restricting F to the lines of the associated congruence: the output is a two-dimensional photograph of the scene represented by F. This viewpoint also fits nicely with the topics that will be studied in Chapters 4 and 5. In particular, F must satisfy certain properties in order to be a "valid" plenoptic function representing a scene. For example, if we assume that F is a discrete function F : Gr(1, P 3 ) → {0, 1} that produces binary images (silhouettes), then it must describe sets of lines which intersect an object in P 3 (e.g., a surface over the reals). This relates to our study of visual hulls in Chapter 4. Alternatively, we can assume that pictures of surfaces are "outlines", rather than silhouettes. In this case, F must describe sets of tangents to smooth surfaces. This idea will be important in Chapter 5.

Chapter 3

Photographic Cameras

The geometric cameras studied in Chapter 2 are maps from P 3 to Gr(1, P 3 ), and are independent of image measurements. Physical "photographic" cameras, on the other hand, will always represent image points using coordinates. In this chapter, we model this kind of camera as a rational map P 3 P 2 or P 3 P 1 ×P 1 . Since any photographic camera is obtained from a parameterization of a congruence of order one, we can build on the framework of rational geometric cameras to study arbitrary photographic imaging systems and their multi-view geometry.

The material in this chapter is based on the following publications:

-Jean Ponce, Bernd Sturmfels, and Matthew Trager. "Congruences and concurrent lines in multi-view geometry". In: Advances in Applied Mathematics 88 (2017), pp. 62-91.

-Matthew Trager, Bernd Sturmfels, John Canny, Martial Hebert, and Jean Ponce. "General models for rational cameras and the case of two-slit projections". In: IEEE Conference on Computer Vision and Pattern Recognition. 2017.

Introduction

Representing cameras as congruences of viewing rays is convenient for studying geometric properties of imaging processes, however, in practice, the use of image coordinates is very important, and is essential for the design of SfM algorithms. For this reason, we extend the geometric camera model considered in the previous chapter to include image measurements and coordinates. More precisely, we define a "photographic" camera to be a rational map P 3 P 2 or P 3 P 1 × P 1 with the property that the pre-image of a (generic) point is a line. Any such camera is obtained by composing a rational camera P 3 C, where C is an order-one congruence in Gr(1, P 3 ), with a (birational)

Photographic Cameras

"parameterization" C P 2 or C P 1 × P 1 . Thus, our photographic camera model is an analytical version of the geometric rational camera model from Chapter 2.

A natural strategy for obtaining a parameterization C P 2 of a congruence C is to use a retinal plane H, equipped with a system of homogeneous coordinates. This allows us to map a generic line in C to the coordinates of its point of intersection with H. This strategy models the geometry of a physical device (which could record an image using a plane of photosensitive sensors), and generalizes the coordinate mapping used in pinhole cameras. However, it is also convenient to consider more general parameterizations, and particularly mappings C P 1 × P 1 . Indeed, many (1, β)-congruences are naturally parameterized by P 1 × P 1 , since their focal locus consists of a line and a rational curve. For these congruences, a photographic camera P 3 P 1 × P 1 leads to simpler expressions. Moreover, many non-central cameras used in practice produce photographs using a 1D scanner moving along a curve, or by sampling columns or rows from sequences of 2D pictures [START_REF] Sturm | Camera models and fundamental concepts used in geometric computer vision[END_REF]. In these cases, the projectivized image is best identified with P 1 × P 1 (one component parameterizes "time", and the other component represents a 1D measurement).

The fact that we only consider rational (polynomial) projections allows us to use Theorem 2.3.3 to classify all photographic cameras based on the "type" of the associated congruence of order one. Moreover, we can apply the formulae for geometric rational cameras in Plücker coordinates from previous chapter to construct explicit examples of photographic cameras. The inverse of any parameterization C P 2 or C P 1 × P 1 also immediately provides the "inverse line projection", associating image points with viewing lines. This can be used together with our description of the concurrent lines variety to recover multi-view constraints for correspondences in image coordinates.

After some general considerations on photographic cameras, we present a detailed analysis of two-slit cameras. It has previously been noted that a two-slit projection onto a retinal plane leads to a map P 3 P 2 whose coordinates are quadratic expressions [START_REF] Feldman | On the epipolar geometry of the crossed-slits projection[END_REF][START_REF] Zomet | Mosaicing new views: The crossed-slits projection[END_REF]. On the other hand, the same mapping can be described using a pair of linear projections P 3 P 1 , that is, using a bilinear map P 3 P 1 × P 1 . This simple bilinear form is very convenient, and allows us for example to analyze orbits of two-slit cameras under affine, similarity, and euclidean transformations, and to introduce intrinsic parameters and calibration matrices that closely resemble the traditional ones defined for pinhole cameras. Moreover, point correspondences among two views are characterized by a 2 × 2 × 2 × 2 epipolar tensor, and we can completely describe the relationship between such tensors and camera configurations (each tensor encodes two configurations of two-slit cameras). We propose "linear" and "minimal" reconstruction algorithms, and discuss strategies for self-calibration for two-slit cameras.

The study of two-slit cameras is relatively simple because of the bilinear representation, however one could develop a more general theory of photographic cameras, that could be seen as an algebraic, non-linear version of classical multi-view geometry. We do not Introduction pursue this direction in detail here, but we describe some simple methods for manipulating photographic cameras, explaining for example how to compute forward and inverse projections of lines and points. Furthermore, we apply "multi-graded Chow forms" [START_REF] Osserman | Multigraded Cayley-Chow forms[END_REF] to show the existence of trifocal and quadrifocal forms, that characterize matching points and lines in different images, and directly generalize traditional multi-focal tensors for arbitrary photographic cameras.

Previous work

The popularity of the pinhole projection model is due to its simplicity, and to the fact that it naturally mimics the geometry of the (human) eye. However, in recent years there has been growing interest in alternative imaging systems, that often have practical advantages. For example, compared to a pinhole camera, a non-central imaging device can collect light in a more "distributed" way, so that a single photograph can provide more information about a scene [START_REF] Rademacher | Multiple-center-of-projection images[END_REF][START_REF] Shum | Omnivergent stereo[END_REF]. A broad overview of non-standard imaging technologies, with many pointers to the literature, can be found in the monograph by Sturm, Ramalingam, Tardif, Gasparini and Barreto [START_REF] Sturm | Camera models and fundamental concepts used in geometric computer vision[END_REF]. In that work, the authors make the distinction between "global", "local", and "discrete" models, in which each of the camera's parameters affects, respectively, the whole projection function, a local "patch" of the field of view, or a single ray recorded by the camera. From this viewpoint, we only consider global camera models in this chapter.

Among the most common techniques for generating non-perspective images is to use a moving "slit" that captures 1D images. This includes pushbroom cameras [START_REF] Gupta | Linear pushbroom cameras[END_REF], but also more general "non-central panoramic" systems, which sample columns of pixels from 2D photographs recorded with a moving (traditional) camera [START_REF] Yu | Panoramic representation of scenes for route understanding[END_REF][START_REF] Ishiguro | Omni-directional stereo[END_REF]. In this setting, we mention that by sampling two columns of pixels it is also possible to capture "stereo panoramas" from a single rotating camera [START_REF] Peleg | Stereo panorama with a single camera[END_REF][START_REF] Shum | Omnivergent stereo[END_REF]. In our terminology, the viewing rays of this type of device form a line congruence of order two (see Section 2.5.1). It has also been observed that sampling different columns of pixels varying together with the camera location is equivalent to a two-slit projection model [START_REF] Zomet | Mosaicing new views: The crossed-slits projection[END_REF]. As noted by Pajdla [START_REF] Pajdla | Geometry of two-slit camera[END_REF], two-slit projection includes the pushbroom camera model, which corresponds to one of the two slits lying at infinity. In general, the analytical form of a two-slit projection is a quadratic map P 3 P 2 [START_REF] Zomet | Mosaicing new views: The crossed-slits projection[END_REF]. Feldman, Pajdla and Weinshall used this representation to study epipolar geometry of two-slit cameras [START_REF] Feldman | On the epipolar geometry of the crossed-slits projection[END_REF]. Their description is based on a 6 × 6 "fundamental matrix", that corresponds to a bilinear constraint on the quadratic Veronese embedding P 2 → P 5 of image coordinates.

A unified representation for several types of non-central cameras is the "General Linear Camera" (GLC) model by Yu and McMillan [197]. This model is based on a two-plane parameterization of lines, and can be used to represent pinhole, two-slit, and linear oblique [START_REF] Pajdla | Stereo with oblique cameras[END_REF] cameras. As argued by Ponce [START_REF] Ponce | What is a camera?[END_REF], all these classes of cameras can also be understood using linear congruences in Gr(1, P 3 ). In subsequent work [START_REF] Batog | Admissible linear map models of linear cameras[END_REF], Batog, Goaoc, and Ponce investigated in more detail these linear imaging systems (which are actually represented by quadratic expressions in 3D coordinates: here "linear" refers to the geometry of the line congruence). They explained, for example, the role of the retinal plane in the imaging process, and argued that the choice of this plane is not completely irrelevant, as is the case for pinhole cameras. They also described epipolar geometry for these cameras using a 6 × 6 fundamental matrix, or using a 4 × 4 fundamental matrix assuming appropriate "normalized" image coordinates. These special image coordinates will also play a role in our study of two-slit cameras. We also mention that our 2×2×2×2 epipolar tensor can be seen as an instance of "Grassmann tensors", which were studied by Hartley and Schaffalitzky [START_REF] Richard | Reconstruction from projections using grassmann tensors[END_REF], and is also formally identical the radial quadrifocal tensor, proposed by Thirtala and Pollefeys for cameras with radial distortion [START_REF] Thirthala | Radial multi-focal tensors[END_REF]. Here we reintroduce this type of tensors in the context of two-slit cameras.

Finally, we mention that Hartley and Saxena introduced in [START_REF] Richard | The cubic rational polynomial camera model[END_REF] a "cubic rational polynomial" camera model that is similar to the rational photographic cameras that are studied in this chapter (their model is essentially a bicubic map P 3 P 1 × P 1 ). The motivation for this type of camera was to provide an accurate approximation for complex imaging systems, particularly those used in satellite imagery, whose real projection mappings are often non-rational. Hartley and Saxena presented an algorithm for estimating parameters of their model given world-to-image correspondences, and also discussed several strategies for dealing with noise and over-fitting issues. Interestingly, this model has since become a standard in photogrammetry, and is commonly used in commercial satellite sensors (see for example [START_REF] Hu | Understanding the rational function model: methods and applications[END_REF][START_REF] Vincent | A comprehensive study of the rational function model for photogrammetric processing[END_REF][START_REF] Zheng | Minimal solvers for 3d geometry from satellite imagery[END_REF]). A different application of a similar model was proposed by Claus and Fitzgibbon [START_REF] Claus | A rational function lens distortion model for general cameras[END_REF] to approximate distortion effects in wide-angle lenses. The main difference between these rational mappings and our photographic cameras is that we only consider projections that associate image points with lines. Rational maps with arbitrary coefficients, such as those used in [START_REF] Richard | The cubic rational polynomial camera model[END_REF] and [START_REF] Claus | A rational function lens distortion model for general cameras[END_REF], will not have this property, since the pre-image of a point will typically be an algebraic curve in space: from our strict perspective, these are not models of "cameras".

Main contributions

The main contributions of this chapter can be summarized as follows.

• We study photographic cameras, that are rational mappings P 3 P 2 or P 3 P 1 × P 1 with the property that the pre-image of generic points are lines in space (Section 3.2). By parameterizing line congruences of order one, we derive explicit representations for all such cameras.

• We present a study of two-slit cameras [START_REF] Batog | Admissible linear map models of linear cameras[END_REF][START_REF] Pajdla | Geometry of two-slit camera[END_REF][START_REF] Zomet | Mosaicing new views: The crossed-slits projection[END_REF], that we model as bilinear maps P 3 P 1 × P 1 . We explain the relationship between this model and the traditional description of these cameras as quadratic maps P 3 P 2 (Section 3.3.1). We then use the bilinear representation to introduce a pair of upper-triangular cal-ibration matrices that can be identified with the orbits of two-slit cameras under similarity transforms (Section 3.3.2). We also introduce a 2 × 2 × 2 × 2 epipolar tensor, that encodes the projective geometry of a pair of two-slit cameras, and generalizes the traditional fundamental matrix. We use these results to propose algorithms for structure from motion and self-calibration (Section 3.3.3).

• We illustrate some properties of general photographic cameras: for example, we characterize multi-view point correspondences using a generalized version of the joint image variety (Section 3.4.1), and explain how to compute forward and inverse projections of lines and points (Section 3.4.2). We also show the existence of multifocal forms, that characterize corresponding points and lines in three and four views, and directly generalize classical trifocal and quadrifocal tensors in the case of photographic cameras (Section 3.4.3).

Conventions.

We return to the notation used in Chapter 1: we write P n = P(R n+1 ) for the n-dimensional projective space over the real numbers, and use bold and normal font to distinguish between vectors and projective objects (e.g., we write x = [x] for a point in P n ). If p = (p 12 , p 13 , p 14 , p 23 , p 24 , p 34 ) is a vector representing Plücker coordinates of a line in P 3 , then we write p * = (p 34 , -p 24 , p 23 , p 14 , -p 13 , p 12 ) for the vector that represents the same line in dual coordinates (or, equivalently, that represents the dual of that line in primal coordinates).

Congruences and Coordinates

This section introduces our model for photographic cameras. After giving some preliminary definitions (Section 3.2.1), we apply our study of congruences of order one from the previous chapter to derive explicit algebraic representations for photographic cameras (Sections 3.2.2 and 3.2.3).

Basic definitions

We are interested in photographic devices that can be described using rational maps from P 3 into P 2 or P 1 × P 1 . The following is a precise definition for our model. We recall that a rational map between projective spaces is a map whose coordinates are homogeneous polynomial functions. Such a map is called "dominant" if the closure of its image is the whole codomain (i.e., the map is surjective up to closure).

Definition 3.2.1.

A photographic rational camera is a dominant rational map P 3 P 2 or P 3 P 1 × P 1 with the property that the fiber (pre-image) of a generic point is a line in P 3 .

Photographic Cameras

This clearly extends the traditional pinhole camera model studied in Chapter 1. As explained in the Introduction, we allow for the image to be the projective plane P 2 as well as a product of projective lines P 1 × P 1 , since the latter is more natural when the imaging system consists of a moving 1D sensor, or when images are formed from 1D samples in sequences of images, as in [START_REF] Rademacher | Multiple-center-of-projection images[END_REF][START_REF] Zomet | Mosaicing new views: The crossed-slits projection[END_REF], for example. Moreover, we will see shortly that rational cameras associated with congruences of higher order are more easily viewed as maps into P 1 × P 1 .

Concretely, we can describe a photographic camera using a triplet [f 1 , f 2 , f 3 ] of homogeneous polynomials in R[x 1 , x 2 , x 3 , x 4 ] of the same degree, or by two such pairs ([g 1 , g 2 ], [h 1 , h 2 ]). The projection maps are given by

Q 1 : P 3 P 2 , [x] → [f 1 (x), f 2 (x), f 3 (x)],
or

Q 2 : P 3 P 1 × P 1 , [x] → ([g 1 (x), g 2 (x)], [h 1 (x), h 2 (x)]) . (3.1)
We will indicate these mappings as

Q 1 = [f 1 , f 2 , f 3 ] and Q 2 = ([g 1 , g 2 ], [h 1 , h 2 ]
). The base locus of a rational camera is the set points in P 3 where the projection mapping is not defined. This means that the base locus of Q 1 and Q 2 as above is given by

V (f 1 , f 2 , f 3 ) and V (g 1 , g 2 ) ∪ V (h 1 , h 2 )
respectively. For a pinhole camera, the base locus is simply the pinhole. The degree of a rational camera is the degree of the f i , or the pair of degrees of the g i and h i (which in general are different).

It is worth noting that the polynomials in (3.1) cannot be arbitrary, because of the assumption that the fibers must lines. More concretely, to describe the pre-image a general image point u = [u 1 , u 2 , u 3 ] in P 2 for a camera

Q 1 = [f 1 , f 2 , f 3 ], we consider the locus of x = [x] in P 3 such that rank    f 1 (x) u 1 f 2 (x) u 2 f 3 (x) u 3    = 1. (3.2)
For generic polynomials f 1 , f 2 , f 3 , this describes a curve in P 3 that contains the base locus V (f 1 , f 2 , f 3 ). In order for Q 1 to be a camera, we require for this curve be the union of V (f 1 , f 2 , f 3 ) and a line. Analogous considerations hold for a camera

Q 2 = ([g 1 , g 2 ], [h 1 , h 2 ]
), for which we take the locus of

x = [x] in P 3 such that rank g 1 (x) u 1 g 2 (x) u 2 = 1 and rank h 1 (x) v 1 h 2 (x) v 2 = 1. Example 3.2.2. If three quadratic forms f 1 , f 2 , f 3 are such that V (f 1 , f 2 , f 3 ) is a twisted cubic, then Q = [f 1 , f 2 , f 3 ] is a camera. Indeed (3.
2) is a curve of degree four, so removing V (f 1 , f 2 , f 3 ) leaves a line. For three quadratic forms to have this property they must be expressible as the 2 × 2-minors of a 2 × 3-matrix of linear forms in x 1 , x 2 , x 3 , x 4 . For example, the map

x → [x 1 x 3 -x 2 2 , x 1 x 4 -x 2 x 3 , x 2 x 4 -x 2 3 ] (3.3)
is a photographic camera. ♦

In general, characterizing which triples f 1 , f 2 , f 3 or quadruples g 1 , g 2 , h 1 , h 2 of polynomials correspond rational photographic cameras is a challenging problem. We address this issue by using of our study of line congruences from Chapter 2.

From lines to measurements

To construct examples of rational cameras, we will use our classification of congruences from the previous chapter. A photographic rational camera Q 1 : P 3 P 2 or Q 2 : P 3 P 1 × P 1 is in fact uniquely associated with a congruence of order one, that is the closure of the image of the map

L Q 1 : P 2 Gr(1, P 3 ), or L Q 2 : P 1 × P 1 Gr(1, P 3 ),
associating image points with their fiber. The image of L Q 1 or L Q 2 is a congruence of order one since a general point x in P 3 will be contained in exactly one image line (that is the set of points in P 3 whose image under Q 1 or Q 2 is the same as for x). Conversely, given any (1, β)-congruence C, we may obtain a rational camera by composing the geometric camera C : P 3 Gr(1, P 3 ) with a birational "parameterization" C P 2 or C P 1 × P 1 . Here "birational" means that these maps are rational and (generically) invertible, and that their inverse is also rational.

In summary, any rational photographic camera Q : P 3 P 2 or Q : P 3 P 1 ×P 1 can be seen as an analytical representation of a rational geometric camera C : P 3

Gr(1, P 3 ), which is in turn defined by line congruence of order one. This is analogous to relationship between a pinhole projection P 3 P 2 and the corresponding bundle of lines. In particular, using the terminology from Chapter 2, the base locus of a photographic camera Q, where the projection mapping is not defined, will always contain the focal locus of the associated congruence C (possibly strictly, as we will see shortly). Both the base locus and the focal locus can be seen as generalizations of the pinhole in traditional cameras. We will also say that photographic camera has class β if the associated congruence has bidegree (1, β).

In order to construct parameterizations C P 2 or C P 1 × P 1 we will usually follow one of two strategies:

1. A natural approach for defining a map C P 2 is to fix a "retinal plane" H in P 3 , equipped with a system of homogeneous coordinates. This allows us to associate with a general line in C the coordinates of the point of intersection ∧ H on H Figure 3.1: A photographic camera (in the figure, a two-slit camera) can be obtained by associating a scene point x with a visual ray l, then mapping the ray l to its intersection y with some retinal plane H, and finally using a projective coordinate system on H to express y as a point u in P 2 .

(see Figure 3.1). This coincides of course with the usual coordinate mapping used for pinhole cameras.

2. The second strategy for constructing a birational map C P 1 × P 1 applies when the lines in C are naturally parametrized by pairs of points in P 1 . This happens for example when lines in C are characterized as the common transversals of two space curves (or lines).

We note that the first strategy can be made more explicit as follows. Let H is a plane in P 3 defined parametrically by

P 2 → H, [u 1 , u 2 , u 3 ] → [u 1 y 1 + u 2 y 2 + u 3 y 3 ] (3.4)
where y 1 = [y 1 ], y 2 = [y 2 ], y 3 = [y 3 ] are points on H. In other words, we view the columns of the 4 × 3 matrix Y = (y 1 , y 2 , y 3 ) as vectors forming a normalized basis inducing a system of homogeneous coordinates on H. We can then define a rational map Gr(1, P 3 ) P 2 associating a generic line with the coordinates of the point ∧ H in H. This map is linear in Plücker coordinates, and is described by the 3 × 6 matrix

N Y =    (y 2 ∨ y 3 ) * (y 3 ∨ y 1 ) * (y 1 ∨ y 2 ) *    . (3.5)
In other words, the matrix N Y induces a linear projection P 5 P 2 , with the property that a generic line in P 3 with Plücker coordinates [p] intersects the plane H at the point with coordinates u = [u] = [N Y p] in P 2 . As for pinhole cameras, the parameterized retinal plane H can be used to model a grid of photosensitive sensors that record photographs.

Congruences and Coordinates

Note that by composing a geometric camera C with a linear projection P 5 P 2 given by (3.5), we obtain a rational map Q that is not defined on the lines in C that lie on H. There are in general β such lines (the class of C), that will belong to the base locus of the photographic camera Q, but not to the focal locus of the congruence C. Example 3.2.3. Consider the plane H = {x 3 -x 4 = 0} in P 3 , equipped with the reference frame induced by the normalized basis y 1 = (1, 0, 0, 0) T , y 2 = (0, 1, 0, 0) T , y 3 = (0, 0, 1, 1) T . The matrix N Y takes in this case the form

N Y =    0 -1 1 0 0 0 0 0 0 -1 1 0 0 0 0 0 0 1    . (3.6)
The null-space of the corresponding linear projection is V (p 13 + p 14 , p 42 + p 23 , p 34 ), which characterizes lines contained in the plane H. ♦ Remark 3.2.4. The construction described above can in principle be generalized to curved "retinal surfaces". More precisely, given a rational surface S in P 3 together with an explicit parameterization ϕ : P 2 S, we can define, at least locally, an algebraic map from Gr(1, P 3 ) to P 2 that associates a line with the coordinates of one of its intersections with S. By composing this map with geometric cameras from Chapter 2, we can construct new analytical camera mappings. Note however that these are not photographic cameras in the sense of Definition 3.2.1, because the resulting expressions will in general be algebraic and not rational (i.e., they will involve square roots). Nevertheless, this could be a good mathematical model for many optical devices (e.g., compound eyes of insects).

Examples of photographic cameras

We next use our study of congruences from Chapter 2 together with the parameterizations discussed above to construct examples of photographic rational cameras. In particular, we consider the four types of congruences described by Theorem 2.3.3. Computations in this section were carried out in Sage [START_REF]SageMath, the Sage Mathematics Software System[END_REF].

Type 1: A congruence C of type 1 is an α-plane, i.e., a bundle of lines that meet at a point c. The classical analytical model for a pinhole camera as a projection matrix can be derived by composing a geometric camera of this type with the map Gr(1, P 3 ) P 2 defined by the 3 × 6 matrix N Y from (3.5), for an arbitrary plane H and normalized basis Y on H. Indeed, it is easy to see that

x → N Y (c ∨ x) =    (y 2 ∨ y 3 ∨ c) * (y 3 ∨ y 1 ∨ c) * (y 1 ∨ y 2 ∨ c) *    x,
which represents the usual projection matrix. On the other hand, by considering different parameterizations of C, it is also possible to construct "alternative" pinhole cameras, which differ only analytically from the traditional linear projection model. For example, a bilinear map [x] → (A 1 x, A 2 x) where A 1 , A 2 are 2 × 4-matrices whose null-spaces meet is, geometrically speaking, a pinhole camera. Any pinhole model can be obtained from a traditional one by applying a "non-linear" change of image coordinates P 2 P 2 or P 2 P 1 × P 1 .

Example 3.2.5.

If [x] = [x 1 , x 2 , x 3 , x 4
], the following three maps

P 3 P 2 , [x] → [x 1 , x 2 , x 3 ], P 3 P 2 , [x] → [x 2 x 3 , x 1 x 3 , x 1 x 2 ], P 3 P 1 × P 1 , [x] → [x 1 , x 2 ], [x 1 + x 2 , x 3 ]) , (3.7)
represent the same geometric pinhole camera with center c = [0, 0, 0, 1]. ♦ Type 2: A geometric camera of type 2 records lines which are secants of a twisted cubic in P 3 . An example of a photographic camera of this type can be obtained by composing the geometric camera from (2.9) with the map Gr(1, P 3 ) P 2 defined by the 3 × 6 matrix N Y from (3.5). Note that this results in a photographic rational map where the coordinate polynomials have degree 4.

A different approach for constructing this type photographic camera is illustrated in Example 3.2.2: if three quadratic forms f 1 , f 2 , f 3 are expressible as the 2 × 2-minors of a 2 × 3-matrix of linear forms in x 1 , x 2 , x 3 , x 4 , then V (f 1 , f 2 , f 3 ) is a twisted cubic, and

Q = [f 1 , f 2 , f 3 ]
is a photographic camera of type 2. Note that in this case the coordinate polynomials have degree 2.

Example 3.2.6. Applying the 3 × 6 projection matrix N Y from (3.6) to the geometric camera (2.9) we obtain the map

Q(x) =    -x 3 2 x 3 + x 1 x 2 x 2 3 + x 1 x 3 3 + x 1 x 2 2 x 4 + x 3 2 x 4 -x 2 1 x 3 x 4 -3x 1 x 2 x 3 x 4 + x 2 1 x 2 4 -x 2 2 x 2 3 + x 1 x 3 3 + x 2 x 3 3 + x 3 2 x 4 -x 1 x 2 x 3 x 4 -x 2 2 x 3 x 4 -x 1 x 2 3 x 4 + x 1 x 2 x 2 4 x 4 3 -2x 2 x 2 3 x 4 + x 2 2 x 2 4    .
This describes a photographic camera whose focal locus F is the standard twisted cubic (the image of [s, t] → [s 3 , s 2 t, st 2 , t 3 ]), and where the retinal plane H = {x 3 -x 4 = 0} has a coordinate frame defined by the basis (1, 0, 0, 0), (0, 1, 0, 0), (0, 0, 1, 1). The base locus of Q(x) (where the projection is not defined) consists of X together with the two lines L 1 = V (x 3 , x 2 ) and L 2 = V (x 2 -x 3 , x 1 -x 3 ): these are in fact the lines of the congruence that lie on H. We point out that are only two lines rather than the expected number β = 3 because H is tangent to F at the point [1, 0, 0, 0].

Congruences and Coordinates

Another photographic camera associated with the same congruence is given more simply by

Q (x) =    x 2 3 -x 2 x 4 x 2 x 3 -x 1 x 4 x 2 2 -x 1 x 3    . (3.8)
The base locus for Q(x) contains only the twisted cubic F: this coincides with the focal locus of the congruence. ♦ Type 3: A congruence C of type 3 contains a family of lines that intersect both a fixed line L and a rational curve X of degree β meets intersects L in β -1 points (counted with multiplicity). As argued in Section 2.3.2, if L is the line V (x 1 , x 2 ), the curve X can be represented parametrically as

P 1 → X, [s, t] → sf (s, t), tf (s, t), g(s, t), h(s, t) , ( 3.9)
where f, g and h are arbitrary binary forms of degree β -1, β and β respectively. The associated geometric camera P 3 → Gr(1, P 3 ) is described by (Proposition 2.3.5)

C(x) =      x 1 x 2 x 3 x 4      ∨      x 1 f (x 1 , x 2 ) x 2 f (x 1 , x 2 ) g(x 1 , x 2 ) h(x 1 , x 2 )      . ( 3.10)
Using this expression, we can construct an associated photographic camera by composing with the map Gr(1, P 3 ) P 2 defined by N Y as above. However, another (arguably simpler) strategy is instead to construct a map P 3 P 1 × P 1 that associates a point x with the coordinates of the points y 1 and y 2 where C(x) meets X and L respectively. Here we parameterize X as in (3.9) and L by [s, t] → [0, 0, s, t]. A simple calculation using (3.10) shows that

y 1 = [x 1 f (x 1 , x 2 ), x 1 f (x 1 , x 2 ), g(x 1 , x 2 ), h(x 1 , x 2 )], y 2 = [0, 0, g(x 1 , x 2 ) -f (x 1 , x 2 )x 3 , h(x 1 , x 2 ) -f (x 1 , x 2 )x 4 ],
so the resulting photographic camera Q : P 3 P 1 × P 1 is given by

Q(x) = x 1 x 2 , g(x 1 , x 2 ) -f (x 1 , x 2 )x 3 h(x 1 , x 2 ) -f (x 1 , x 2 )x 4 . (3.11)
General photographic cameras P 3 P 2 can be obtained by composing this expression with a birational map P 1 × P 1 P 2 .

Photographic Cameras

Example 3.2.7. The map

Q(x) = x 1 x 2 , x 2 1 + x 2 2 -x 1 x 3 x 1 x 2 -x 1 x 4
is a photographic camera P 3 P 1 × P 1 of type 3 with β = 2. It corresponds to the congruence of lines intersecting L = V (x 1 , x 2 ) and the conic X(s, t) = [s 2 , st, s 2 + t 2 , st]. These expressions can be obtained from (3.9) and (3.11) using f (s, t) = s, g(s, t) = s 2 +t 2 and h(s, t) = st.

A photographic camera P 3 P 2 associated with the same congruence is given by the composition of the geometric camera (3.10) and the map Gr(1, P 3 ) P 2 defined by the matrix N Y from (3.6). The resulting expression is

Q (x) =    -x 3 1 + x 2 1 x 2 -x 1 x 2 2 + x 1 x 2 x 3 -x 1 x 2 x 4 -x 2 1 x 2 + x 1 x 2 2 -x 3 2 + x 2 2 x 3 -x 2 2 x 4 x 1 x 2 x 3 -x 2 1 x 4 -x 2 2 x 4    .
♦ Type 4: A congruence C of type 4 can be viewed as a degeneration of a congruence of type 3, and the associated geometric map can be represented as (3.9) assuming the binary form f is identically zero. In this case, because the curve X and the line L coincide, we cannot map a point x in P 3 to the coordinates of the two intersection points y 1 , y 2 (in fact, the image (3.11) becomes one-dimensional in this case). However, we can construct photographic maps by composing (3.9) with the usual map Gr(1, P 3 ) P 2 defined by the matrix N Y . Example 3.2.8. An example of photographic camera P 3 P 2 of type 4 is

Q(x) =    -x 3 1 + x 2 1 x 2 -x 1 x 2 2 -x 2 1 x 2 + x 1 x 2 2 -x 3 2 x 1 x 2 x 3 -x 2 1 x 4 -x 2 2 x 4    .
This expression is obtained from the geometric camera (3.9) with g(s, t) = s 2 + t 2 and h(s, t) = st, composed with the map Gr(1, P 3 ) P 2 defined by N Y from (3.6). ♦

Two-Slit Cameras

In this section, we focus on a particular class of photographic rational cameras, namely two-slit cameras [START_REF] Feldman | On the epipolar geometry of the crossed-slits projection[END_REF][START_REF] Gupta | Linear pushbroom cameras[END_REF][START_REF] Pajdla | Geometry of two-slit camera[END_REF][START_REF] Zomet | Mosaicing new views: The crossed-slits projection[END_REF]. This type of camera is associated with congruences of class β = 1, and it can be viewed as the "simplest" rational camera model after pinhole Two-Slit Cameras projections. Photographic two-slit cameras can in fact be conveniently represented using a bilinear model

P 3 P 1 × P 1 , [x] → ([A 1 x], [A 2 x]),
where A 1 , A 2 are 2 × 4-matrices of full rank with disjoint null spaces. These null spaces identify the two slits of the cameras. We view each P 1 in the image as parameterizing the planes that contain one of two slits.

We begin our discussion by explaining in Section 3.3.1 the relationship between the bilinear representation and the quadratic mapping P 3 P 2 that has been previously used to model two-slit cameras [START_REF] Feldman | On the epipolar geometry of the crossed-slits projection[END_REF][START_REF] Zomet | Mosaicing new views: The crossed-slits projection[END_REF]. We then classify two-slit cameras based on their orbits under affine, similarity and euclidean transformations (Section 3.3.2). Our analysis leads to the definition of upper triangular calibration matrices, whose entries are invariant to similarity orbits. In Section 3.3.3, we describe two-view geometry for two-slit cameras, pointing out the existence of a 2 × 2 × 2 × 2 "epipolar tensor" that shares many similarities with the traditional fundamental matrix for pinhole cameras (although a tensor represents in general two configurations of cameras). We also present some algorithms for reconstruction and self-calibration based on these ideas.

Retinal planes and bilinear maps

Two-slit cameras record viewing rays that are the transversals to two fixed skew lines (the slits). The bidegree of the associated congruence is (1, 1), since a generic plane will contain exactly one transversal to the slits, namely the line joining the points where the slits intersect the plane. As explained in Chapter 2, this type of congruence is the intersection of Gr(1, P 3 ) with a 3-dimensional linear space in P 5 . The geometric camera C : P 3

Gr(1, P 3 ) associated with slits l 1 , l 2 was described in (2.8) as a Plücker matrix

C(x) = (x ∨ l 1 ) ∧ (x ∨ l 2 ) = [P 1 xx T P 2 -P 2 xx T P 1 ], (3.12)
where P 1 , P 2 are the Plücker matrices associated with the slits l 1 , l 2 .

We now assume that we have fix a retinal plane H with the coordinate system defined by Y = (y 1 , y 2 , y 3 ). A rational camera Q : P 3 P 2 is obtained by composing (3.12) with the 3 × 6 matrix N Y as in (3.5). A simple calculation shows that the resulting map can be written compactly as

Q(x) =    x T P 1 S * 1 P 2 x x T P 1 S * 2 P 2 x x T P 1 S * 3 P 2 x    , ( 3
= V (x 1 , x 3 ), l 2 = V (x 2 , x 3 + x 4 ).
The corresponding geometric map P 3

Gr(1, P 3 ) is given by

C(x) =          x 1 x 2 0 x 1 x 3 + x 1 x 4 -x 2 x 3 x 2 x 3 x 2 3 + x 3 x 4          . ( 3.14)
Composing with the projection N Y in Example 3.2.3, we obtain the formula for a rational camera with slits l 1 , l 2 :

Q(x) =    x 1 x 3 + x 1 x 4 2x 2 x 3 x 2 3 + x 3 x 4    . (3.15)
The same expression can be deduced using (3.13). ♦

It was noted in [START_REF] Feldman | On the epipolar geometry of the crossed-slits projection[END_REF][START_REF] Zomet | Mosaicing new views: The crossed-slits projection[END_REF] that using a different retinal plane H in (3.13) corresponds in general to composing Q : P 3 P 2 with a quadratic change of image coordinates P 2 P 2 . However, this transformation is in fact linear when H and H intersect along a transversal to the slits. This follows from the following property. Lemma 3.3.2. Let l 1 , l 2 be two skew lines in P 3 . For any point x not on the these lines, we indicate with C(x) the unique transversal to l 1 , l 2 passing through x. If H and H are two planes meeting at a line δ that intersects l 1 and l 2 , then the map f : H H defined, for points y not on δ, as

f (y) = C(y) ∧ H , (3.16)
can be extended to a homography between H and H .

Two-Slit Cameras

Proof. The result can be shown by fixing (convenient) coordinate frames on H and H , and using them to express f analytically: the result is a linear map.

We also sketch a more "geometric" argument. We need to show that a generic line m on H is mapped by (3.16) to a line on H . If m does not intersect l 1 or l 2 , the union of the common transversals to l 1 , l 2 , m is a quadric in P 3 . The intersection of this quadric with H will be a plane conic. However, this intersection contains the transversal line δ, and hence it is a degenerate conic, i.e., a union of two lines. Since δ does not belong to the image of (3.16), we deduce that (the closure of) image of m is a line in H .

This lemma also implies that two retinal planes that intersect at a transversal to the slits can define the same rational camera (using appropriate coordinate systems). Note the similarity with the traditional pinhole case, where the choice of the retinal plane is completely irrelevant since the pinhole c induces a homography between any planes H, H not containing c. See Figure 3.2.

Contrary to the case of pinhole cameras, two-slit cameras of the form (3.13) are not all projectively equivalent. This can be argued geometrically by noting that the coordinates u 1 , u 2 in P 2 of the points y 1 = l 1 ∧ H, y 2 = l 2 ∧ H (the intersections of the slits with the retinal plane) are always preserved by projective transformations of P 3 . For Batog et al. [START_REF] Batog | Classical problems in computer vision and computational geometry revisited with line geometry[END_REF][START_REF] Batog | Admissible linear map models of linear cameras[END_REF], the coordinates u 1 , u 2 are "intrinsic parameters" of the camera. More precisely, they are projective intrinsics, since they are invariant under projective transformations. Batog et al. also argue that choosing the points y 1 , y 2 as points in the projective basis on H leads to simplified analytic expressions for the projection. Here, we develop this idea further, and observe that a two-slit camera with this kind of coordinate system can be equivalently represented using the bilinear description P 3 P 1 × P 1 mentioned at the beginning of the section.

For any retinal plane H, we fix a coordinate system defined by a basis Y = (y 1 , y 2 , y 3 ) where [y 1 ] = l 2 ∧ H, [y 2 ] = l 1 ∧ H and y 3 is arbitrary: in this case, a straightforward computation shows that (3.13) can be written in the form

Q(x) =    (p T 1 x) (q T 2 x) (p T 2 x) (q T 1 x) (p T 2 x) (q T 2 x)    =    p T 1 x/p T 2 x q T 1 x/q T 2 x 1    , ( 3.17)
where p 1 = (l 1 ∨ y 3 ), p 2 = -(l 1 ∨ y 1 ), q 1 = (l 2 ∨ y 3 ), q 2 = -(l 2 ∨ y 2 ) are vectors representing planes in P 3 . This quadratic map can be described using two linear maps

P 3 P 1 , namely Q 1 (x) = p T 1 x p T 2 x = [A 1 x], Q 2 (x) = q T 1 x q T 2 x = [A 2 x]. (3.18)
In other words, (3.17) determines the 2 × 4 matrices A 1 and A 2 up to two scale factors. The two 2×4 matrices for a two-slit camera are analogues of the 3×4 matrix representing Photographic Cameras a pinhole camera: for example, the slits are associated with the null-spaces of these two matrices. The linear maps P 3 P 1 correspond in fact to the "line-centered" projections for the two slits.

Conversely, given a pair of 2 × 4 projection matrices A 1 , A 2 , we can use (3.17) to obtain a (quadratic) mapping P 3 P 2 . This represents a photographic two-slit camera with a retinal plane. The retinal plane is however not completely fixed: it may be any plane containing the line {p T 2 x = q T 2 x = 0}. This line passes is spanned by y 1 and y 2 , and is the locus of points where (3.17) is not defined.

In summary, representing a two-slit camera as a bilinear map P 3 P 1 ×P 1 is essentially equivalent to using a retinal plane with a reference frame in which the first two basis points are l 1 ∧ H, l 2 ∧ H. Clearly, we can assume this whenever the coordinates of l 1 ∧ H, l 2 ∧ H (the projective intrinsic parameters in [START_REF] Batog | Admissible linear map models of linear cameras[END_REF]) are known. In the remainder of the section, we will always represent a two-slit camera with a pair of 2 × 4 projection matrices, writing Q = ([A 1 ], [A 2 ]) for the camera represented by A 1 , A 2 . We will however sometimes also consider the associated "physical realization" with a retinal plane, using the projection P 3 P 2 given by (3.17).

Remark 3.3.3. It is clear from the representation in terms of projection matrices that our two-slit model has 7 + 7 = 14 degrees of freedom. In terms of retinal planes: there are 8 degrees of freedom corresponding to the choice of the slits, 2 for the intersection points of the retinal plane with the slits, and 4 for the choice of coordinates on the plane (since two basis points are constrained). The retinal plane belongs to the pencil of planes containing {x 3 = x 3 + x 4 = 0}, i.e., it is a plane of the form x 3 -dx 4 = 0. The choice d = 1 is natural since points of the form [x 1 , x 2 , 1, 1] are mapped to [x 1 , x 2 , 1]. ♦

Orbits and calibration matrices

The orbit of a photographic camera under the action of projective, affine, similarity, or euclidean transformations is a family of cameras that are geometrically and analytically equivalent for that "geometry". In the following, we will refer to an orbit as a primitive camera model, where the word "primitive" refers to the fact that the model represents a single device up to changes of world coordinates. We next describe some primitive models for two-slit cameras. This is analogous to the discussion on pinhole cameras from Section 1.2.4.

Two-Slit Cameras

Our analysis is based on the simple observation that applying a projective transformation 1 . This immediately implies that all photographic two-slit cameras represented as pairs 2×4-matrices are projectively equivalent (as noted above, this is in contrast with the quadratic model (3.13)). Indeed, given any pair of two-slit cameras

[T] to a two-slit camera Q = ([A 1 ], [A 2 ]) yields Q = ([A 1 T], [A 2 T]).
Q = ([A 1 ], [A 2 ]) and Q = ([B 1 ], [B 2 ]), a pro- jective transformation relating them is [A -1 B],
where A and B are obtained by stacking A 1 , A 2 and B 1 , B 2 respectively (note that A and B will always be invertible because slits are disjoint). In particular, we see that our bilinear model P 3 P 1 × P 1 is a projective primitive camera model, with 14 degrees of freedom.

We next consider affine transformations, and look at the orbit of the two-slit camera from Example 3.3.4. It is easy to see that all cameras Q = ([A 1 ], [A 2 ]) that are equivalent to (3.19) have the form

A 1 = m T 1 t 1 m T 3 t 3 , A 2 = m T 2 t 2 m T 3 t 4 , ( 3.20)
where m i are arbitrary 3-vectors. Viewed as maps P 3 P 2 , as in (3.17), these are two-slit cameras where the retinal plane is parallel to the slits: indeed, while this plane is not completely determined, it must contain the line

{[m T 3 , t 3 ]x = [m T 3 , t 4 ]x = 0}
, that is spanned by intersections of the slits with the plane at infinity. We will refer to (3.20) as a parallel two-slit camera. This describes a primitive affine model with 12 degrees of freedom.

We now consider the family of parallel cameras of the form .21) for d = 0 and 0 < θ < 2π (and θ = π). From the viewpoint of euclidean geometry, the slits for this camera are at an angle of θ and distance d. Note that (3.15) is of this form, with θ = π/2 and d = 1. Using (3.21) as a family of canonical euclidean devices, we can introduce expressions for the "intrinsic parameters" of two-slit cameras.

A 1 = 1 0 0 0 0 0 1 0 , A 2 = 2 cos θ 2 sin θ 0 0 0 0 1 d . ( 3
Proposition 3.3.5. If Q = ([A 1 ], [A 2 ]
) is a parallel two-slit camera (3.20), then we can uniquely write where K 1 and K 2 are upper-triangular 2 × 2 matrices defined up to scale with positive elements along the diagonal, and r 1 , r 2 , r 3 are unit vectors, with r 3 orthogonal to both r 1 , r 2 . Here, θ = arccos(r 1 • r 2 ) is the angle between the slits, and |t 4 -t 3 | is the distance between the slits.

A 1 = K 1 r T 1 t 1 r T 3 t 3 , A 2 = K 2 r T 2 t 2 r T 3 t 4 , ( 3
Moreover, if the matrices K 1 and K 2 are written as

K 1 = f u u 0 0 1 , K 2 = 2f v v 0 0 1 , ( 3.23)
then f u , f v can be interpreted as "magnifications" in the u and v directions, and (u 0 , v 0 ) as the position of the "principal point". See Figure 3.3.

Proof. The decomposition exists and is unique because of RQ-decomposition of matrices [START_REF] Gene | Matrix computations[END_REF]Theorem 5.2.3]. More precisely, if we write

A 1 = [M 1 | t 1 ], A 2 = [M 2 | t 2 ],
where M 1 , M 2 are 2 × 3, then K 1 , K 2 are the (normalized) upper triangular matrices in the RQ decomposition for M 1 , M 2 respectively.

We next observe that for a pair canonical matrices

A 1 = 1 0 0 0 0 0 1 0 , A 2 = 2 cos θ 2 sin θ 0 0 0 0 1 d , (3.24)
the corresponding euclidean orbit is of the form

r T 1 t 1 r T 3 t 3 , 2r T 2 2t 2 r T 3 t 3 + d , ( 3.25)
where θ = arccos(r 1 • r 2 ). This follows by applying a 4 × 4 euclidean transformation matrix to (3.24). These cameras decompose with K 1 being the identity and K 2 = Two-Slit Cameras diag(2, 1). Finally, if we indicate with p 1 , p 2 and 2q 1 , q 2 the rows of (3.25), so that the corresponding camera can be written as x → u = (p T 1 x/p T 2 x, 2q T 1 x/q T 2 x, 1), then the

composition of p T 1 p T 2 , q T 1 q T 2 with K 1 , K 2 as in (3.23) yields the camera [x] → f u p T 1 x p T 2 x + u 0 , f v 2q T 1 x q T 2 x + v 0 , 1 . (3.26)
From this we easily deduce the physical interpretations of the entries of K 1 and K 2 .

The parameters θ and d, and the matrices K 1 and K 2 are clearly invariant under euclidean transformations. Moreover, within the parallel model (3.20), two cameras belong to the same euclidean orbit if and only if all of their parameters are the same. In fact, the 12 degrees of freedom of a parallel camera are split into 6 corresponding to the "intrinsics" θ, d, K 1 and K 2 , and 6 for the "extrinsic" action of euclidean motion. 2 Compared to the traditional intrinsic parameters for pinhole cameras, we note the absence of a term corresponding to the "skewness" of the image reference frame. Indeed, the angle between the two axes must be the same as the angle between the slits, as a consequence of the "intrinsic coordinate system" (the principal directions correspond in fact to the fixed basis points y 1 , y 2 on the retinal plane). On the other hand, θ and d do not have an analogue for pinhole cameras. We can think of d and θ as the "3D" intrinsic parameters, since they are distinct from the "analytic" intrinsic parameters, that are entries of the calibration matrices K 1 , K 2 , and differentiate (euclidean orbits of) cameras only based on analytic part of their mapping. See also Figure 3.4, that shows images taken with virtual two-slit cameras with slits forming different angles. Finally, we point out that for two-slit cameras in (3.22), the euclidean orbit and the similarity orbit do not coincide: this implies that when the intrinsic parameters are known, some information on the scale of a scene can be inferred from a photograph [START_REF] Sturm | Camera models and fundamental concepts used in geometric computer vision[END_REF].

Pushbroom cameras. Pushbroom cameras are a degenerate class of projective twoslit cameras, in which one of the slits lies on the plane at infinity [START_REF] Feldman | On the epipolar geometry of the crossed-slits projection[END_REF]. This is quite similar to the class affine cameras for perspective projections. Pushbrooms are handled by our projective model (3.17), but not by our affine one (3.20), where both slits are necessarily finite. We thus introduce another affine model, namely where m 1 , m 2 , m 3 are arbitrary 3-vectors. All such cameras are equivalent up to affine transformations, so this describes an affine model with 11 degrees of freedom. The corresponding rational cameras P 3 P 2 can be written as

A 1 = m T 1 t 1 0 T 1 , A 2 = m T 2 t 2 m T 3 t 3 , ( 3
[x] → [m T 1 , t 1 ]x, [m T 2 , t 2 ]x [m T 3 , t 3 ]x , 1 . (3.28)
This coincides with the linear pushbroom model proposed by Hartley and Gupta [START_REF] Gupta | Linear pushbroom cameras[END_REF], who identify (3.28) with the 3

× 4 matrix with rows [m T 1 , t 1 ], [m T 2 , t 2 ], [m T 3 , t 3 ].
Let us now consider a family of affine pushbroom cameras of the form sin θ cos θ 0 0 0 0 0 1

0 1 0 0 0 0 1 0 , ( 3.29)
for 0 < θ < 2π (and θ = π). This represents a pushbroom camera where the direction of motion of the sensor is at an angle θ with respect to the parallel scanning planes. We use this family of canonical devices to define calibration matrices and intrinsic parameters for pushbroom cameras.

Proposition 3.3.6. Let Q = ([A 1 ], [A 2 ]
) be a pushbroom camera as in (3.27), and let us also assume that m 1 and m 3 are orthogonal. 3 We can uniquely write

A 1 = K 1 r T 1 t 1 0 T 1 , A 2 = K 2 r T 2 t 2 r T 3 t 3 . (3.30)
where

K 1 = diag(1/v, 1), K 2 = f u 0 1
(with positive v and f ) and r 1 , r 2 , r 3 are unit vectors, with r 3 orthogonal to both r 1 , r 2 . Here, θ = arccos(r 1 • r 2 ) is the angle between the two slits (or between the direction of motion of the sensor and the parallel scanning planes). Moreover, v can be interpreted as the speed of the sensor, and f and u as the magnification and the principal point of the 1D projection.

Proof. The proof is similar to that of Proposition 3.3.5. The decomposition is unique because of QR-factorization of matrices. The euclidean orbits of "canonical" pushbroom cameras have the form

r T 1 t 1 0 T 1 , r T 2 t 2 r T 3 t 3 .
All these cameras decompose with K 1 , K 2 being the identity. Finally, the physical interpretation of the parameters follows by noting that composing a pushbroom camera (with rows p 1 , (0, 0, 0, 1) T and q 1 , q 2 ) with calibration matrices

K 1 , K 2 yields [x] → 1 v p T 1 x, f q T 1 x q T 2 x + u, 1 .
From this we deduce the physical interpretations of the entries of K 1 and K 2 .

Similarly to parallel two-slit cameras, the entries of K 1 , K 2 can be seen as the "analytic" intrinsic parameters of the pushbroom camera, while θ is a "3D" intrinsic parameter.

Epipolar geometry and algorithms

We next describe the epipolar geometry for two-slit cameras using a 2 × 2 × 2 × 2 epipolar tensor. It is worth noting that image correspondences between two-slit cameras have previously been characterized using a 6 × 6 [START_REF] Feldman | On the epipolar geometry of the crossed-slits projection[END_REF], or a 4 × 4 fundamental matrix [START_REF] Batog | Admissible linear map models of linear cameras[END_REF]. The latter approach, due to Batog et al. [START_REF] Batog | Admissible linear map models of linear cameras[END_REF], is somewhat similar to ours, since it is based on the "intrinsic" image reference frame that is equivalent to our bilinear model.

Photographic Cameras

However, the tensor representation has the advantage of being easily described in terms of the elements of the four 2 × 4 projection matrices, in a form that closely resembles the corresponding expression for the traditional fundamental matrix.

We recall here that a pair of two-slit cameras

Q 1 = ([A 1 ], [A 2 ]), Q 2 = ([B 1 ], [B 2 ]) is projectively equivalent to another pair Q 1 = ([A 1 ], [A 2 ]), Q 2 = ([B 1 ], [B 2 ]
) if the two pairs of cameras are related by a projective transformation, i.e., if there exists a projective transformation T = [T] of P 3 such that

A i = A i T and B i = B i T for i = 1, 2.
An equivalence class of projectively equivalent pairs of cameras defines a "projective configuration".

Theorem 3.3.7. Let Q 1 = ([A 1 ], [A 2 ]), Q 2 = ([B 1 ], [B 2 ]
) be two general projective two-slit cameras. The set of corresponding image points (u, v), (u , v ) in P 1 × P 1 is characterized by the relation

2 i,j,k,l=1 f ijkl u i v j u k v l = 0, (3.31)
where F = (f ijkl ) is a 2 × 2 × 2 × 2 "epipolar tensor". Its entries are

f ijkl = (-1) i+j+k+l • det (A 1 ) T 3-i (A 2 ) T 3-j (B 1 ) T 3-k (B 2 ) T 3-l , (3.32)
where (A i ) j denotes the j-th row of A i , and similarly for B i .

All pairs of two-slit cameras in the same projective configuration determine the same 2 × 2 × 2 × 2 epipolar tensor F. Conversely, in general, a valid epipolar tensor F determines two distinct projective configurations of pairs of two-slit cameras.

We note that this result is consistent with the theory presented by Hartley and Schaffalitzky in [START_REF] Richard | Reconstruction from projections using grassmann tensors[END_REF], where it is shown that "Grassmann tensors" (generalizations of multi-view tensors for projections in arbitrary dimension), lead to a two-fold ambiguity in the reconstruction when the image is a P 1 . A more mathematical proof of the same result is given in [START_REF] Ito | Projective reconstruction in algebraic vision[END_REF]. Our epipolar tensor is also formally identical to the "radial quadrifocal tensor", defined by Thirthala and Pollefeys in [START_REF] Thirthala | Radial multi-focal tensors[END_REF] for quadruples of cameras with axial symmetry (indeed, each such camera defines a linear map P 3 P 1 ).

Proof. The inverse line projection P 1 × P 1 → Gr(1, P 3 ) of a two-slit camera is a bilinear map given by

ξ(u, v) =   ij (-1) i+j u i v j (A 1 ) 3-i ∧ (A 2 ) 3-j   ∈ Gr(1, P 3 ). (3.33)
The definition of F now follows immediately by imposing the condition that ξ(u, v) and ξ(u , v ) are concurrent.

Two-Slit Cameras

The definition of F is clearly invariant under projective transformations of P 3 . Hence, assuming that the vectors (A 1 ) 1 , (A 2 ) 1 , (B 1 ) 1 , (B 2 ) 1 are independent (which is true generically) we can apply a change of reference frame in P 3 so that the projection matrices have the form

A 1 = 1 0 0 0 c 11 c 12 c 13 c 14 , A 2 = 0 1 0 0 c 21 c 22 c 23 c 24 , B 1 = 0 0 1 0 c 31 c 32 c 33 c 34 . B 2 = 0 0 0 1 c 41 c 42 c 43 c 44 .
(3.34)

The 16 entries of F are now, up to sign, the principal minors of the 4×4-matrix C = (c ij ). More precisely,

f ijkl = (-1) i+j+k+l det C [i-1,j-1,k-1,l-1]
where

C [i-1,j-1,k-1,l-1]
is the submatrix of C where the selected rows and columns correspond to the binary vector [i-1, j -1, k -1, l-1] (e.g., C [1,0,0,0] = (c 11 )). Thus, determining the projection matrices (A 1 , A 2 ), (B 1 , B 2 ) corresponding to the tensor F, is equivalent to finding the entries of a 4 × 4-matrix given its principal minors. This problem is studied in [START_REF] Lin | Polynomial relations among principal minors of a 4× 4-matrix[END_REF]. The set of all matrices with the same principal minors as C have the form

D -1 CD or D -1 C T D,
where D is a diagonal matrix. These two families of matrices, viewed as elements of (3.34), correspond to two distinct projective configurations of cameras.

The set of all epipolar tensors forms a 13-dimensional variety in P 15 : this agrees with 14 + 14 -15 = 13, where 14 represents the degrees of freedom of two-slit cameras, and 15 is to account for projective ambiguity. Two equations are sufficient to characterize an epipolar tensor locally, however a result in [START_REF] Lin | Polynomial relations among principal minors of a 4× 4-matrix[END_REF] implies that a complete algebraic characterization actually requires 718 polynomials of degree 12.

Remark 3.3.8. Our study of canonical forms and calibration matrices in Section 3.3.2 leads to a natural definition of essential tensors: for example, an essential tensor could be defined by (3.32) where

Q 1 = ([A 1 ], [A 2 ]), Q 2 = ([B 1 ], [B 2 ]
) are all of the form (3.22) with K 1 , K 2 being the identity. Proposition 3.3.5 then guarantees that for any pair of "parallel" two-slit cameras as in (3.20), we can uniquely write the epipolar tensor as

F ijkl = E ijkl (K 1A ) i (K 2A ) j (K 1B ) k (K 2B ) l , (3.35)
where E ijkl is an essential tensor. This closely resembles the analogous decomposition of fundamental matrices. Recovering an algebraic characterization of essential tensors, similar to the classical Demazure equations [START_REF] Demazure | Sur deux problemes de reconstruction[END_REF], could be an interesting topic for future work (the problem can also be generalized to linear projections in arbitrary dimensions).

123

Photographic Cameras

Linear SFM. Using Theorem 3.3.7, we can design a linear algorithm for SfM, that proceeds as follows:

1. Using at least 15 image point correspondences, estimate F linearly using (3.31).

2. Recover two projective camera configurations that are compatible with F.

Clearly, for noisy image correspondences, the linear estimate from step 1 will not be a valid epipolar tensor, since tensors of the form (3.32) are not generic (as noted above, they form a 13-dimensional variety in P 15 ). On the other hand, it is possible to recover projection matrices from only 13 of the entries of F, and this avoids the problem of using a valid tensor. A simple scheme for this is as follows:

1. We set out to recover the entries of a 4 × 4-matrix C given its principal minors.

Since we can always replace C with D -1 CD, where D is a diagonal matrix, we can assume that c 12 = c 13 = c 13 = c 14 = 1 (at least generically). Other similar assignments are possible.

2. Elements on the diagonal and on the first column of C are easily computed using seven of the entries of F: Here the elements to the right of the equal signs have already been assigned. Hence, we recover 10 entries of C from linear equalities. This leads to 8 possible matrices C, i.e., a finite number of camera configurations. Note however that the entries f 1111 and f 2111 of F were never used (which is why we can assume the tensor to be generic): in an ideal setting with no noise, exactly two of the 8 solutions will be consistent with the remaining constraints.

c
This approach for recovering two-slit projections from the corresponding epipolar tensor relies on some genericity assumptions (e.g., we have often divided by element without verifying that it is not zero), and developing an optimal strategy for this task is outside the scope of our work. Nevertheless, we include as a proof of concept an example that applies this method on synthetic data.

124

Two-Slit Cameras

Example 3.3.9. We first illustrate concretely some basic properties of the fundamental tensor. We consider the following pairs of projection matrices: Computing the epipolar tensor (3.31) for both of these pairs yields F as in (3.37). On the other hand, the two camera configurations are not projectively equivalent: indeed, if a projective transformation between the two existed, it would need to be the identity, because five of the eight rows coincide. It is straightforward to verify that it is in fact the second pair that corresponds to the configuration of the original cameras (3.36).

A 1 = -1 7 4 0 8 -1
We next try to estimate the cameras (3.36) from image correspondences. We consider 70 random points in space, project them onto an image plane using (3.36) together with (3.17). We then add noise to the images of the points. Although none of the eight solutions in the method outlined above will be exactly consistent with the last two entries of F, we can consider the two solutions that minimize an "algebraic residual" for these constraints. For image sizes of about 100 × 100, and noise with a standard deviation of 10 -5 , we recover the following pairs of cameras (that should be compared with (3.38)):

A 1 1 = 1. 0.

♦

Minimal SFM. It is also possible to design a 13-point algorithm that recovers projection matrices (3.34) and the corresponding tensor F from a minimal amount of data, namely 13 point correspondences between images. Substituting these correspondences in (3.31), we obtain an under-determined linear system, which implies that the epipolar tensor is a linear combination αT 1 + βT 2 + γT 3 for some T 1 , T 2 , T 3 that generate the corresponding null-space. Since the variety of epipolar tensors has codimension 2 in P 15 , we expect to find a finite number of feasible tensors in this linear space (up to scale factors). According to [START_REF] Lin | Polynomial relations among principal minors of a 4× 4-matrix[END_REF]Remark 14], the variety of epipolar tensors (that is viewed there as the projective variety for the principal minors of 4 × 4 matrices) has degree 28. Hence, this minimal approach should lead to 28 complex solutions for F, and 56 projective configurations of cameras. Using the computer algebra system Macaulay2 [START_REF] Daniel | Macaulay2, a software system for research in algebraic geometry[END_REF] we have verified (over finite fields) that imposing 13 general linear conditions on the 16 principal minors of the matrix C, and fixing c 12 = c 13 = c 14 = 1, we obtain 56 solutions C in the algebraic closure of the field.

Self-calibration.

Any reconstruction based on the epipolar tensor will be subject to projective ambiguity. On the other hand, using results from Section 3.3.2, it is possible to develop strategies for self-calibration. Let us assume that we have recovered a projective reconstruction A i 1 , A i 2 for i = 1, . . . , n for finite two-slit cameras (that we assume were originally "parallel"). We indicate with Q a "euclidean upgrade", that is, a 4 × 4-matrix that describes the transition from a euclidean reference frame to the frame corresponding to our projective reconstruction. According to Proposition 3.3.5, we may write

A i 1 Q = K i [R i 1 | t i 1 ], A i 2 Q = K i 2 [R i 2 | t i 2 ]
, where R i 1 , R i 2 are 2 × 3 matrices with orthonormal rows (for simplicity, we remove the factor 2 from the canonical euclidean form). In particular, for all i = 1, . . . , k, we have

A i 1 QΩ * Q T A i 1 T = K i 1 K i 1 T A i 2 QΩ * Q T A i 2 T = K i 2 K i 2 T , (3.43)
where equality is up to scale and Ω * = diag(1, 1, 1, 0). Geometrically, the matrix Ω * Q = QΩ * Q T represents the dual of the absolute conic, in the projective coordinates used in the reconstruction. The equations (3.43) identify in fact the dual of the image of the absolute conic in the two copies of P 1 . These are the set of planes containing each slit that are tangent to the absolute conic in P 3 .

Photographic Cameras

We now assume that the principal points cameras are at the "origin", so that K i 1 , K i 2 (and hence K i 1 K iT 1 and K i 2 K iT 2 ) are diagonal. Each row in (3.43) gives two linear equations in the elements of Ω * Q , corresponding to the zeros in the matrices on the right hand side. For example, imposing that the (1, 2)-entry of

K i 1 K iT 1 is
Q such that Q Ω * Q T = Ω * Q .
The matrix Q is however not uniquely determined, and indeed we can actually only recover a similarity upgrade, since any similarity transformation will fix the absolute conic in P 3 . Example 3.3.10. To apply our self-calibration scheme, we consider 10 cameras

A i 1 , A i 2 , i = 1, . . . , 10, of the form A i 1 = K i 1 [R i 1 | t i 1 ]Q -1 , A i 2 = K i 2 [R i 2 | t i 2 ]Q -1 , where R i 1 , t i 1 , R i
2 , t i 2 are random parameters for euclidean primitive parallel cameras, K i 1 , K i 2 are random diagonal calibration matrices, and Q is a random 4×4 matrix describing a projective change of coordinates. We also add small amounts of noise to the entries of

A i 1 , A i 2 . The matrices A i 1 , A i 2 represent
     , QΩ * Q T =      1.
-0.58 -0.34 0.28 -0.58

     , Q Ω * Q T =      1.
-0.59 -0.34 0.29 -0.59 1.44 -0.51 -0.56 -0.34 -0.51 1.35 -0.48 0.29 -0.56 -0.48 0.75

     .
The matrices Q, Q are not close, however one easily verifies that Q -1 Q is (almost) a similarity transformation. In particular, the cameras

A i 1 Q , A i 2 Q , i = 1, .
. . , 10 are a "similarity upgrade" of the projective solution. For example, for the first of our 10 original cameras we had K 1 1 = diag(4.04, 1), K 2 = diag(1.37, 1), and indeed

A 1 1 Q = -2.

Properties of Photographic Cameras

After having studied two-slit cameras, we broaden our perspective and return to general rational photographic cameras. Much of the analysis carried out for pinhole cameras in Chapter 1, including the classification of orbits, the description multi-view constraints, and the study of camera configurations, can in principle be replicated for any camera model. We present in this section some first results and definitions that could be a point of departure for an in-depth treatment of all rational imaging models in the future.

In particular, we introduce in Section 3.4.1 a generalized version of the joint image, that characterizes point correspondences for systems of general photographic cameras.

We also derive as a simple example the epipolar geometry between a pinhole and twoslit camera. In Section 3.4.2 we discuss some basic properties of rational photographic cameras, explaining in particular how to compute forward and inverse projections of lines and points. Finally, in Section 3.4.3 we introduce trifocal and quadrifocal forms, that characterize matching points and lines in different images, and generalize traditional multi-view tensors.

Multi-view geometry

In order to describe the multi-view geometry for arbitrary photographic cameras, we can use a generalized version of the joint image variety from Chapter 1. For any collection of photographic cameras

Q 1 , . . . , Q n 1 , Q 1 , . . . , Q n 2
where Q i : P 3 P 2 and Q j : P 3 P 1 × P 1 , we consider the closure of the image of

P 3 (P 2 ) n 1 × (P 1 × P 1 ) n 2 , x → Q 1 (x), . . . , Q n 1 (x), Q 1 (x), . . . , Q n 2 (x) . (3.45)
This clearly extends the usual notion of joint image from Section 1.3 and [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF][START_REF] Trager | The joint image handbook[END_REF][START_REF] Triggs | Matching constraints and the joint image[END_REF].

Photographic Cameras

The following immediate result relates the joint image and the "multi-image variety" of lines, introduced in Section 2.4.2. This is a generalization of Proposition 2.4.5 from pinhole cameras to arbitrary photographic cameras. We write C Q i and C Q j for the congruences associated with Q i and Q j .

Proposition 3.4.1. The joint image variety for

Q 1 , . . . , Q n 1 , Q 1 , . . . , Q n 2 is birationally equivalent to the multi-image variety M (C Q 1 , . . . , C Qn 1 , C Q 1 , . . . , C Q n 2
), defined in Section 2.4.2, under the map

L Q 1 × • • • × L Qn 1 × L Q 1 × • • • × L Q n 2 : (P 2 ) n 1 × (P 1 × P 1 ) n 2 Gr(1, P 3 ) n , (3.46)
where

L Q i , L Q j are "inverse projections" of L Q i : P 2 Gr(1, P 3 ), L Q j : P 1 × P 1
Gr(1, P 3 ) for the congruences

C Q i , C Q j .
The practical importance of this observation is that we can use it together with the theory from Chapter 2 to recover multi-view constraints characterizing the joint image for any set of photographic cameras. More precisely, from Theorem 2.4.3 we deduce that, if the base loci of Q 1 , . . . , Q n 1 , Q 1 , . . . , Q n 2 are pairwise disjoint, then the joint image variety is birationally equivalent to a slice of the concurrent lines variety V n 1 +n 2 . Since the closure of the image of (3.46

) is C M 1 × • • • × C Qn 1 × C Q 1 × • • • × C Q n 2
, we can obtain multi-view constraints in image coordinates by replacing the Plücker variables with the coordinates of L Q i (w) and L Q j ((u, v)) in the multilinear polynomials defining V n 1 +n 2 .

Epipolar geometry: pinhole + two-slit. As a simple example (which doesn't require the concurrent lines variety but only the incidence condition for two lines) we characterize the joint-image for a pinhole camera Q 1 : P 3 P 2 with Q 1 = [A] and a two-slit camera Q 2 :

P 3 P 1 × P 1 with Q 2 = ([B], [C]
). The inverse projection L Q 1 : P 2 → Gr(1, P 3 ) is given by (cf. Section 1.2.5)

[u] → [u 1 (A 2 ∧ A 3 ) -u 2 (A 1 ∧ A 3 ) + u 3 (A 1 ∧ A 2 )], (3.47)
where the A i are row vectors of A. The analogous map

L Q 2 : P 1 ×P 1 → Gr(1, P 3 ) is ([v], [w]) → [v 1 w 1 (B 2 ∧ C 2 ) -v 1 w 2 (B 2 ∧ C 1 ) -v 2 w 1 (B 1 ∧ C 2 ) + v 2 w 2 (B 1 ∧ C 1 )]. (3.48)
Now if (u, (v , w )) is an element in P 2 × (P 1 × P 1 ), we replace Plücker coordinates in the incidence constraint trace(PQ * ) = 0 with the images of u and (v , w ) under (3.47) and (3.48). From this we deduce that the joint image for Q 1 and Q 2 is characterized by the multi-linear constraint i,j,k f ijk u i v k w l = 0,

f ijk = (-1) i+j+k • det A T l A T m B T ĵ C T k , (3.49)
where (i, l, m), (j, ĵ), (k, k) are distinct sets of indices. This condition can be identified with a 3 × 2 × 2 epipolar tensor F = (f ijk ), which is of course very similar to the epipolar tensor for pairs of two-slit cameras studied in Section 3.3.1, and to the traditional fundamental matrix for pinhole cameras. A direct computation with Macaulay2 also allows us to characterize these tensors algebraically.

Proposition 3.4.2. The variety of 3×2×2 epipolar tensors as in (3.49) has codimension one in P 11 . Its defining polynomial is

f 2 111 f212f221f 2 322 -f 2 111 f212f222f321f322 -f 2 111 f221f222f312f322 + f 2 111 f 2 222 f312f321 -f111f112f211f221f 2 322 + f111f112f211f222f321f322 -f111f112f212f221f321f322 + f111f112f212f222f 2 321 +f111f112f 2 221 f312f322 + f111f112f221f222f311f322 -f111f112f221f222f312f321 -f111f112f 2 222 f311f321 -f111f121f211f212f 2 322 + f111f121f211f222f312f322 + f111f121f 2 212 f321f322 -f111f121f212f221f312f322 +f111f121f212f222f311f322 -f111f121f212f222f312f321 + f111f121f221f222f 2 312 -f111f121f 2 222 f311f312 +f111f122f211f212f321f322 + f111f122f211f221f312f322 -2f111f122f211f222f312f321 -f111f122f 2 212 f 2 321 -2f111f122f212f221f311f322 + 2f111f122f212f221f312f321 + f111f122f212f222f311f321-f111f122f 2 221 f 2 312 +f111f122f221f222f311f312 + f 2 112 f211f221f321f322 -f 2 112 f211f222f 2 321 -f 2 112 f 2 221 f311f322 +f 2 112 f221f222f311f321 + f112f121f 2 211 f 2 322 -f112f121f211f212f321f322 -f112f121f211f221f312f322 +f112f121f 2 222 f 2 311 + 2f112f121f211f222f312f321 + 2f112f121f212f221f311f322 -f112f121f212f222f311f321 -2f112f121f211f222f311f322 -f112f121f221f222f311f312-f112f122f 2 211 f321f322+f112f122f211f221f311f322 +f112f122f211f212f 2 321 -f112f122f211f221f312f321 + f112f122f211f222f311f321-f112f122f212f221f311f321 +f112f122f 2 221 f311f312-f112f122f221f222f 2 311 +f 2 121 f211f212f312f322-f 2 121 f211f222f 2 312 -f 2 121 f 2 212 f311f322 +f 2 121 f212f222f311f312 -f121f122f 2 211 f312f322 + f121f122f211f212f311f322 -f121f122f211f212f312f321 +f121f122f211f221f 2 312 + f121f122f211f222f311f312 + f121f122f 2 212 f311f321 -f121f122f212f221f311f312 -f121f122f212f222f 2 311 +f 2 122 f 2 211 f312f321-f 2 122 f211f212f311f321-f 2 122 f211f221f311f312+f 2 122 f212f221f 2 311 .
Remark 3.4.3. In [START_REF] Thirthala | Radial multi-focal tensors[END_REF], Thirtala and Pollefeys also point that a 3 × 2 × 2 tensor as in (3.49) (which, from their perspective, represents a pinhole camera and two radial cameras) must satisfy a constraint. They derive this constraint geometrically, showing in fact that any such tensor must satisfy det(W) = 0 where W = (w ij ) is a 3 × 3 matrix with entries

w ij = t i22 t j11 + t j22 t i11 -t i12 t j21 -t j12 t i21 .
One can verify that this is the same constraint as the one given in Proposition 3.4.2. We were not aware of this result at the time of our publication [START_REF] Ponce | Congruences and concurrent lines in multi-view geometry[END_REF].

Orbits and projections of subspaces

Given any photographic camera Q : P 3 P 2 or Q :

P 3 P 1 ×P 1 , changing coordinates in P 3 amounts to replacing Q([x]) with Q([Tx]
) for some T in GL(4, R). As noted for pinhole cameras in Chapter 1, understanding the action of projective transformations on cameras is important, since a coordinate frame in 3-space is not physically defined, so a projective camera model is best viewed as a GL(4, R)-orbit (or a union of orbits). 5

Photographic Cameras

For example, given a photographic camera Q, it is natural to ask what is the associated stabilizer group Stab(Q) = Stab GL(4,R) (Q), that is, the set of projective transformations that leave Q fixed. It is easy to realize that Stab(Q) only depends on the line congruence C Q associated with Q. Indeed, a transformation fixes Q if and only if it preserves all lines in the congruence (not necessarily point-wise).

Computationally, we observe the following fact. Assuming that Q is a photographic camera associated with a (1, β)-congruence C Q , then:

1. If β = 0, Stab G (Q) is five-dimensional in G (including the scale factor). 2. If β = 1, Stab G (Q) is two-dimensional in G (including the scale factor). 3. If β > 1, Stab G (Q) is one-dimensional in G (
including the scale factor). In other words, the identity is the only projective transformation of P 3 that fixes Q.

Perhaps slightly unintuitively, this implies that the parameter space of photographic cameras of higher class is actually simpler than that of pinhole and two-slit cameras, since when β > 1 we can identify every camera in the orbit of Q with a projective transformation of P 3 . On the other hand, the action of projective transformations will in general not be transitive on the set of photographic cameras of a given class (that is, two general photographic cameras will not be projectively equivalent), even among those of the same degree. This was already noted for the quadratic model of two slit cameras in Section 3.3.

Another difference between pinhole cameras and general photographic cameras is related to changes of image coordinates. Let us assume that the image of a photographic camera Q is P 2 . Then, for general projective transformation S of P 2 , the camera x → S(Q(x))

will not be equivalent to Q under a projective transformation of P 3 . One way to argue this is that, contrary to pinhole cameras, general photographic cameras will present "focal points" in the image, where the inverse projection P 2 C Q is not defined. It is easy to see that the coordinates of these points must be invariant under projective transformations of P 3 ; these points are in fact the "projective intrinsics" for two-slit cameras from Section 3.3.1. Similarly, if the camera Q was defined using a retinal plane H equipped a coordinate frame, then changing this plane will generally not correspond to the action of a projective transformation on Q (neither in P 3 nor in P 2 ).

Projections and Inverse Projections of Subspaces. General photographic cameras are non-linear maps, so projections and inverse projections of linear spaces will not be linear spaces in general. This is of course not true for the inverse projections of generic image points which are, by construction, lines in P 3 . For an arbitrary camera

Q : P 3 P 2 , with Q = [f 1 , f 2 , f 3 ]
, a formula for the inverse projection P 2 Gr(1, P 3 ) can be obtained computationally by considering the ideal

I in Q[x 1 , x 2 , x 3 , x 4 , u 1 , u 2 , u 3 ] Properties of Photographic Cameras of 2 × 2 minors of    f 1 (x) u 1 f 2 (x) u 2 f 3 (x) u 3    .
(3.50)

Saturating I with respect to f 1 , f 2 , f 3 yields an ideal J that contains two linear forms in x 1 , x 2 , x 3 , x 4 with coefficients in u 1 , u 2 , u 3 : these two linear forms describe the preimage of a generic point u = [u 1 , u 2 , u 3 ]. A similar general procedure gives the inverse projection of an image point if

Q : P 3 P 1 × P 1 . If L = {au 1 + bu 2 + cu 3 = 0} is a line in P 2 , and Q = [f 1 , f 2 , f 3
] is any rational map, then the pre-image of L in P 3 is a surface described simply S L = {af 1 + bf 2 + cf 3 = 0}. In the case of a camera, this surface will always be ruled, and will contain the focal locus of the associated congruence C Q . As the L varies in (P 2 ) * , S L will vary a two dimensional family of surfaces, or a linear system [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF]. This is basically a "dual" viewpoint of cameras.

Finally, for a camera

Q = [f 1 , f 2 , f 3 ]
, the projection of a general line M in P 3 will be an algebraic curve in P 2 . The equation of this image of curve can described parametrically in terms of the Plücker coordinates of M . An explicit relation can be obtained computationally as follows. We assume that we know (or have previously computed) the geometric camera associated with Q, in the form of a six vector q x of polynomial expressions in x 1 , . . . , x 4 , representing the Plücker coordinates of the image line of Note that this system of conics is linearly parameterized by the Plücker coordinates of the line that is being projected. ♦

x = [x 1 , x 2 , x 3 , x 4 ]. We now consider the ideal I in Q[u 1 , u 2 , u 3 , x 1 , . . . ,

Photographic cameras and the Veronese embedding.

A useful way to view photographic cameras using the Veronese embedding. We recall that the Veronese embedding V n,d : P n → P ( n+d d )-1 maps a point x in P n to the point in P ( n+d d )-1 whose coordinates are all possible monomials of total degree d evaluated at the coordinates of x. The image of V n,d is the Veronese variety. For example, for n = 1, d = 3, we have V 3,1 : [s, t] → [s 3 , s 2 t, st 2 , t 3 ], and the Veronese variety is the twisted cubic in P 3 .

If Q : P 3 P 2 is a photographic camera of degree d, we consider the Veronese embedding V 3,d : P 3 → P N with N = 3+d 3 -1. The action of Q can always be seen as the composition of V 3,d with a linear projection Q V : P N P 2 and, in particular, the projection Q V uniquely determines Q. In other words, we can represent the non-linear map Q on P 3 as a linear map Q V on P N . The action of Q corresponds to the action of Q V restricted the Veronese variety in P N .

Representing a non-linear map as a linear map in a higher dimensional space can be useful in practice, and indeed the Veronese embedding has been used for calibration of rational cameras [START_REF] Claus | A rational function lens distortion model for general cameras[END_REF][START_REF] Richard | The cubic rational polynomial camera model[END_REF]. However, it is important to note that while all linear projections Q V : P N P 2 describe rational maps P 3 P 2 of degree d, from our perspective these maps will not be cameras in general. In fact, the condition for a rational map to be a photographic camera can be stated in terms of how the inverse projections of Q V intersect the Veronese variety in P N . It would be interesting to see whether this idea can be used to directly characterize rational cameras.

Multifocal forms

We now present a generalization of multifocal tensors for nonlinear photographic cameras. We will make use of the general theory of multi-graded Chow forms that we introduced in [START_REF] Osserman | Multigraded Cayley-Chow forms[END_REF]. This theory was already applied in Section 1.5.2 to describe traditional multifocal tensors for pinhole cameras. We refer to that section for a very brief summary on Chow and multi-graded Chow forms, and to our paper [START_REF] Osserman | Multigraded Cayley-Chow forms[END_REF] for more details.

We assume that Q 1 , . . . , Q n are photographic cameras P 3 P 2 (in fact, most of what follows only requires that Q i are rational maps). We recall that the joint image variety

M n = M (Q 1 , . . . , Q n ) in (P 2
) n is defined as the closure of the image of

P 3 (P 2 ) n , x → (Q 1 (x), . . . , Q n (x)).
Just as for pinhole cameras, the joint image is birationally equivalent to P 3 , and thus has dimension 3, whenever at least two of the cameras are geometrically distinct. As argued in Section 3.4.1, an algebraic characterization of M n can always be derived from the concurrent lines variety defined in Chapter 2.

On the other hand, we can also use multi-graded Chow forms to describe the joint image.

As explained in Section 1.5.2, this requires choosing β = (β 1 , . . . , β n ) so that 0 ≤ β i ≤ 2 and n i=1 β i = 4. For any such β, we consider

Z Mn,β = {(L 1 , . . . , L k ) | M n ∩ (L 1 ∩ . . . ∩ L k ) = ∅} ⊂ Gr(n -β 1 , P 2 ) × . . . × Gr(n -β k , P 2 ).
This set contains n-tuples (L 1 , . . . , L n ), where each L i is a linear space of codimension β i in P 2 , and for which there exists a correspondence (u 1 , . . . , u n ) in M n with u i in L i . According to Theorem 1.5.5, the set Z Mn,β has "nice" properties when i∈I β i ≤ 3

Properties of Photographic Cameras

for all I {1, . . . , k}, which means β i > 0 for all i and n ≤ 4. In this case, the set Z Mn,β will be described by a single equation F Mn,β in primal and dual coordinates image coordinates.

Just as for pinhole cameras, there are only three possibilities:

• n = 2, β = (2, 2): in this case F Mn,β is a polynomial describing corresponding points (PP) in two images.

• n = 3, β = (2, 1, 1) (and permutations): in this case F Mn,β is a polynomial describing "point-line-line" (PLL) correspondences across three images.

• n = 4, β = (1, 1, 1, 1): in this case F Mn,β is a polynomial describing "line-line-lineline" (LLLL) correspondences for four images.

From the general theory of multi-graded Chow forms, we also know that F Mn,β characterizes the joint image M n (i.e., even for n = 3 and n = 4 the form encodes point correspondences). In summary, we have the following result.

Theorem 3.4.5. For any type of photographic cameras (rational maps) P 3 P 2 , there exist bifocal, trifocal and quadrifocal forms, which describe corresponding linear spaces of type (P P ), (P LL), (LLLL) across 2, 3, 4 images respectively. The knowledge of these forms is equivalent to the knowledge of the joint image M n (n = 2, 3, 4) of the cameras.

The forms in Theorem 3.4.5 generalize traditional multifocal tenors in multiview geometry. An obvious difference here is the fact that multi-view forms are in general not multilinear, but only multihomogeneous in each set of image variables. On the other hand, Theorem 1.5.5 describes how the multidegree of a Chow form F Mn,β (as a multihomogeneous polynomial) can be computed from the multidegree of the M n (as a variety in (P 2 ) n ). 6 Recent results by Escobar and Knutson on the multidegree of multi-image varieties [START_REF] Escobar | The multidegree of the multi-image variety[END_REF] might lead to a general formula for the multidegree of generalized joint images, and for the degrees of the associated multifocal forms. For the moment, the multidegree of M n can be computed directly in simple cases. Example 3.4.6. Consider the following three maps Q i : P 3 P 2 :

Q 1 (x) =    x 1 + x 4 x 2 + x 4 x 3 + x 4    , Q 2 (x) =    x 2 1 + 2x 1 x 3 + x 2 3 x 1 x 3 + x 1 x 4 + x 3 x 4 + x 2 4 x 2 1 + x 1 x 3 + x 1 x 4 + x 3 x 4    , Q 3 (x) =    x 3 1 -x 1 x 2 x 4 x 2 1 x 2 + x 3 2 -x 2 2 x 3 x 2 1 x 2 -x 2 2 x 4    .
These are three photographic cameras:

Q 1 is a pinhole (1, 0)-camera, Q 2 is a two-slit (1, 1)-camera, and Q 3 is a (1, 2)-camera.
These examples were chosen to have disjoint focal loci. Using the command multidegree in Macaulay2, we can verify that the multidegree of the joint image is

2T 2 1 T 2 + 3T 2 1 T 3 + T 1 T 2 2 + 6T 1 T 2 T 3 + T 1 T 2 3 + 3T 2 2 T 3 + 2T 2 T 2 3 . (3.52) Conclusions 2. If the map (3.53) is injective for the camera models Q 1 , . . . , Q k , then the multifocal form uniquely represents [Q 1 , . . . , Q k ] inside the configuration space C(Q 1 , . . . , Q k ).
Proof. Both statements follow immediately from the fact that the knowledge of a multiview form is equivalent to the knowledge of the the joint image.

We conjecture that apart from the case where n = 2 and Q 1 , Q 2 are two-slit cameras, the map (3.53) is usually injective.

Conclusions

We have introduced a general model of photographic cameras, that are described by rational functions, and that are associated with congruences of order one. Among the simplest examples of this model are two-slit cameras viewed as bilinear projections P 3 P 1 × P 1 . We have studied such cameras, analyzing the corresponding projective orbits, and describing the associated epipolar geometry. A similar methodology can in principle be applied to any photographic camera. While cameras described by rational mappings have previously been considered [START_REF] Claus | A rational function lens distortion model for general cameras[END_REF][START_REF] Richard | The cubic rational polynomial camera model[END_REF], these works did not explicitly require image points to be projections of 3D lines.

Overall, the goal of this chapter was to provide a framework for studying properties of general photographic devices in a unified way. In particular, we believe that it is important to distinguish between two main parts of the imaging process: the "geometric" part, which characterizes the set of light rays that are recorded by the camera, and the "analytic" part, that describes how these rays are mapped to image coordinates. Moreover, it is useful to emphasize and study the action of (subgroups of) projective transformations on classes of cameras, particularly for the purpose of parameterizing camera configurations in multi-view reconstruction. For pinhole cameras, for example, affine changes of image coordinates have the same effect as a projective transformation of space. As argued in Section 3.4.2, this property is not true for arbitrary photographic cameras, and this makes the projective framework arguably less useful in practice: in this sense, for calibrating systems of general cameras, solving for euclidean motions using the method described by Sturm [START_REF] Sturm | Multi-view geometry for general camera models[END_REF] is probably simpler than applying projective methods. On the other hand, using a projective framework is much more convenient for understanding the geometry of rational mappings and of the corresponding cameras.

In the future, we hope to understand the relation between configurations of photographic cameras and the corresponding joint image varieties, as discussed at the end Section 3.4.3. Although the use of trifocal and quadrifocal forms for camera calibration would probably be very difficult (given the large number of coefficients that would need to be estimated), whether or not a joint image determines a unique configuration of cameras is a central theoretical question for multi-view reconstruction. On the more tometric information, and treating objects and their images as simple "sets" in projective spaces.

We begin our discussion by introducing a simple notion of "geometric consistency" for image sets. More precisely, for fixed cameras P 1 , . . . , P n , we say that sets T 1 , . . . , T n in P 2 are consistent if there exists a set S in P 3 such that P i (S) = T i (Figure 4.1). This setup can be seen as an extension of classical multi-view geometry, which provides conditions for points (and sometimes lines) to be consistent (i.e., to correspond) in terms of given camera projections [START_REF] Faugeras | The geometry of multiple images: the laws that govern the formation of multiple images of a scene and some of their applications[END_REF][START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. This also expands an analogy initiated by the "generalized epipolar constraint" introduced in [START_REF] Åström | Generalised epipolar constraints[END_REF], that essentially characterizes consistency for two image sets. In our study, we point out several results that hold for arbitrary image sets (or sometimes convex image sets), that have identical counterparts in the theory of point correspondences.

Given a family of consistent image sets, the largest set in space that projects onto them is the associated visual hull [START_REF] Guenther | Geometric modeling for computer vision[END_REF]. In the literature, the term "visual hull" has also been used for another related but different concept: for Bottino and Laurentini [START_REF] Laurentini | The Visual Hull Concept for Silhouette-Based Image Understanding[END_REF], the visual hull of a set S in space is the largest set whose images from all viewpoints are the same as for S. In our presentation, we differentiate between these two notions, calling them the "finite" and the "complete" visual hull, respectively. After introducing both types of visual hulls in a very general setting, we point out a close analogy with (projective) convexity. Loosely speaking, the the visual and convex hull of a set in space can both be obtained from exactly the same construction, with the latter using "line-centered" projections P 3 P1 instead of pinhole projections.

Regarding the relationship between consistency and visual hulls, we note that a family of consistent image sets always provides a complete, albeit implicit, representation of a finite visual hull. This is similar to the idea of representing 3D points by their projections, or, equivalently, identifying 3D space with the joint image (see Section 1.3). Moreover, this viewpoint suggests a general strategy for investigating three-dimensional geometric features directly from image data. For example, the combinatorial components that have previously been defined for finite visual hulls ("cone strips", "intersection curves" and "triple points" [START_REF] David | Computer Vision -A Modern Approach, Second Edition[END_REF]) admit simple formal descriptions in image space. This leads to a simple but effective strategy for computing an explicit representation of the boundary of the finite visual hull (an arbitrarily dense point cloud, that can be turned into a mesh using standard geometry processing algorithms [START_REF] Bernardini | The ball-pivoting algorithm for surface reconstruction[END_REF]).

The definitions of visual hulls and image consistency do not require metric information, which is why we adopt a projective setting. This eliminates various degenerate situations and allows, for example, to unify the cases of orthographic and perspective projections (as in traditional multi-view geometry). 1 Another advantage of the projective language

Introduction

is that it provides homogeneous formulations for duality and line geometry. In particular, we can exploit the fact that perspective projections are related to planar sections in the dual space, to define a very natural "dual" notion of consistency, expressing conditions for a family of planar sets to be sections of the same object. Finally, we point out that for most of our presentation we do not use any regularity assumption on the observed sets (silhouettes) and 3D shapes: similar to "volumetric approaches" in shape-fromsilhouettes [START_REF] Worthy | Volumetric Descriptions of Objects from Multiple Views[END_REF][START_REF] Chien | Volume/surface octrees for the representation of three-dimensional objects[END_REF][START_REF] Slabaugh | A Survey of Methods for Volumetric Scene Reconstruction from Photographs[END_REF], this bypasses the difficulties that arise when considering an idealized setting for smooth surfaces.

Previous work

Consistency and duality. The most widespread application of consistency (for silhouettes) has been for designing alternatives to point-based methods for camera calibration, required for dealing with smooth and textureless surfaces. Indeed, ever since the seminal work of Rieger [START_REF] Jh Rieger | Three-dimensional motion from fixed points of a deforming profile curve[END_REF], the problem of estimating camera motion or calibration parameters using only silhouettes has received considerable attention: see, e.g., [START_REF] Boyer | On using silhouettes for camera calibration[END_REF][START_REF] Furukawa | Structure and motion from images of smooth textureless objects[END_REF][START_REF] Hernández | Silhouette coherence for camera calibration under circular motion[END_REF][START_REF] Joshi | Structure and motion estimation from dynamic silhouettes under perspective projection[END_REF][START_REF] Paulo Rs Mendonça | Epipolar geometry from profiles under circular motion[END_REF][START_REF] Sudipta | Camera network calibration from dynamic silhouettes[END_REF][START_REF] Kwan-Yee | Reconstruction of sculpture from its profiles with unknown camera positions[END_REF]. Albeit with some variations, all these methods exploit (more or less directly) the geometric constraints provided by the epipolar tangencies [START_REF] Åström | Generalised epipolar constraints[END_REF]. In addition to camera calibration, silhouette consistency has been enforced explicitly for other tasks; for example multi-view segmentation [START_REF] Neill Df Campbell | Automatic 3d object segmentation in multiple views using volumetric graphcuts[END_REF], or 3D-reconstruction [START_REF] Cremers | Multiview stereo and silhouette consistency via convex functionals over convex domains[END_REF] and recognition [START_REF] Lazebnik | On pencils of tangent planes and the recognition of smooth 3D shapes from silhouettes[END_REF]. Another interesting "artistic" application is discussed in [START_REF] Niloy | Shadow art[END_REF]. There exists limited theoretical work on silhouette consistency, and it is always restricted to special situations. In particular, the problem of determining whether a family of silhouettes can correspond to a real object is considered in [START_REF] Bottino | Introducing a new problem: Shape-fromsilhouette when the relative positions of the viewpoints is unknown[END_REF][START_REF] Jacobs | Judging whether multiple silhouettes can come from the same object[END_REF][START_REF] Yi | Efficiently Determining Silhouette Consistency[END_REF], but results are only given for the case of orthographic projections and somewhat restricted camera motion. Some theoretical properties, such as the fact that epipolar tangency conditions do not imply global consistency, can be found in [START_REF] Boyer | On using silhouettes for camera calibration[END_REF][START_REF] Man | Visual hull construction, alignment and refinement for human kinematic modeling, motion tracking and rendering[END_REF].

Consistency of Image Sets and Visual Hulls

The duality between projections and planar sections (in 3-space) is well known in a variety of different settings. For orthographic projections, this principle is most naturally described for smooth surfaces (taking sections of the Gauss sphere [START_REF] Patrick | Silhouette-Slice Theorems[END_REF]), or for convex bodies (using convex polarity [START_REF] Schneider | Convex bodies: the Brunn-Minkowski theory[END_REF]). For perspective projections, it is typical to consider smooth or algebraic surfaces, and define the dual surface as the union of tangent planes at smooth points inside the dual projective space (P 3 ) * . The convex projective setting discussed here is probably less common, although it is very similar to the polarity defined in terms of oriented projective geometry in [START_REF] Stolfi | Oriented projective geometry: A framework for geometric computations[END_REF]. Despite the use of different mathematical languages (and different hypotheses, particularly on the smoothness of the projected object), these approaches are all very closely related.

Visual hulls. The idea of approximating a shape by intersecting the visual cones associated with many silhouettes was, remarkably, already popular in the 1800s [START_REF] Sturm | A historical survey of geometric computer vision[END_REF].

In the modern literature, the problem was first addressed in 1974 in Baumgaurt's PhD thesis [START_REF] Guenther | Geometric modeling for computer vision[END_REF]. Bottino and Laurentini studied the "complete" visual hull for polyhedral [START_REF] Laurentini | The Visual Hull Concept for Silhouette-Based Image Understanding[END_REF], smooth [START_REF] Bottino | The Visual Hull of Smooth Curved Objects[END_REF], and piecewise smooth [START_REF] Bottino | The visual hull of piecewise smooth objects[END_REF] surfaces in R 3 . They showed for example that the boundary of the complete visual hull of a smooth object contains ruled surface patches, corresponding to two types of "event surfaces" ("tritangent" and "edge") [START_REF] Bottino | The Visual Hull of Smooth Curved Objects[END_REF]. This viewpoint is actually closer to the topic of the next chapter, where we will study projections of smooth (algebraic) surfaces as "contours"; here we focus on more primitive set-theoretic relations between sets of points, lines, and planes.

The concept of "image-based" visual hulls for reconstruction was introduced by Matusik et al. [START_REF] Matusik | Image-based visual hulls[END_REF], who proposed an efficient method for rendering visual hulls without explicitly constructing a 3D model, exploiting instead epipolar geometry to reproject points across different images. A limitation of this method is that it is view-dependent, i.e., rendering a novel view requires running the entire algorithm again. However, the basic ray casting procedure used to compute "intersection intervals" along visual rays (see Section 4.6.2), has inspired subsequent work for computing explicit models of the visual hull by similarly reducing 3D computations to 2D [START_REF] Franco | Exact polyhedral visual hulls[END_REF][START_REF] Franco | Efficient polyhedral modeling from silhouettes[END_REF][START_REF] Matusik | Polyhedral Visual Hulls for Real-Time Rendering[END_REF]. Our strategy for recovering a point cloud representation of the visual hull also exploits the image-based procedure from [START_REF] Matusik | Image-based visual hulls[END_REF]; however, compared to [START_REF] Franco | Exact polyhedral visual hulls[END_REF][START_REF] Franco | Efficient polyhedral modeling from silhouettes[END_REF][START_REF] Matusik | Polyhedral Visual Hulls for Real-Time Rendering[END_REF], it is simpler, since it does not deal with the problem of recovering the polyhedral structure, and because it exploits image contours as well as silhouettes viewed as regions.

The combinatorial structure of the finite visual hull was noted by Lazebnik, Boyer, and Ponce in [START_REF] Lazebnik | On computing exact visual hulls of solids bounded by smooth surfaces[END_REF]. Moreover, similar to our projective framework, Lazebnik, Furukawa, and Ponce [START_REF] Lazebnik | Projective visual hulls[END_REF] use oriented projective geometry for dealing with visual hulls. Contrary to [START_REF] Lazebnik | Projective visual hulls[END_REF], however, our approach does not require a global notion of orientation, since "region correspondences" across multiple images are well defined in purely projective spaces.

Main contributions

The following is a brief summary of the main contributions in this chapter.

• We study a notion of geometric consistency for arbitrary image sets and projective cameras. We also define a more general form of k-consistency, and investigate the relationship between pairwise (k = 2) consistency, epipolar tangencies, and global consistency (Sections 4.2.2 and 4.2.3).

• We restate the notion of consistency in terms of duality, expressing the condition for planar sets to be sections of the same object. For convex silhouettes, we show that the dual of the visual hull coincides with the convex hull of the dual image of the silhouettes (Proposition 4.3.6).

• We describe the "complete" and the "finite" visual hulls for arbitrary sets in projective space. We point out connections with notions of convexity and duality in P n and Gr(1, P 3 ).

• We present an image-based description of the combinatorial features of the finite visual hull, and propose a simple but efficient method for recovering an arbitrarily dense point cloud on the boundary of the visual hull. We also introduce a "hardness" function that can be computed directly from image data and provides an estimate of the quality of the visual hull reconstruction.

Conventions. Throughout the chapter, we let P n = P(R n+1 ). For any pinhole camera P : P 3 P 2 and set of points T in P 2 , the associated visual cone is P -1 (T ), where P -1 denotes the pre-image set (this definition excludes the pinhole from the visual cone, however this will be irrelevant for our discussion).

Consistency of Image Sets

In this section, we introduce our notion of geometric consistency for arbitrary sets in different images. The definition we give is very natural, and similar concepts have previously been used to introduce "incoherence" measures for silhouettes [START_REF] Boyer | On using silhouettes for camera calibration[END_REF][START_REF] Hernández | Silhouette coherence for camera calibration under circular motion[END_REF]. After presenting some basic definitions in Section 4.2.1, we consider in Section 4.2.2 the important case of two silhouettes, pointing out how the consistency condition is basically equivalent to the popular "epipolar tangency" constraint, but only applied to extremal tangents. We then investigate in more detail the difference between pairwise consistency and general consistency in Section 4.2.3. We conclude in Section 4.2.4 with a discussion on the topology of the visual hull for two views, which further clarifies the relationship between consistency and epipolar tangents.

Basic definitions

Let P 1 , . . . , P n be n projective cameras P 3 P 2 with distinct pinholes c 1 , . . . , c n , and let T 1 , . . . , T n be a family of sets in P 2 , which we view as belonging to different images. We allow these sets to be completely arbitrary: for example, each set T i could be a finite collection of points, curves or regions in each image. For each i = 1, . . . , n, we write

C i = P -1 i (T i )
R = M (P 1 , . . . , P n ) ∩ (T 1 × . . . × T n ) ⊂ (P 2 ) n
projects surjectively onto each T i . Note that, contrary to the case of point correspondences, the set R in Definition 4 is in general not unique. However, we can always associate T 1 , . . . , T n with the maximal set that projects onto them. This set is given by It is actually customary to define the visual hull simply as the intersection of the cones i C i for arbitrary, not necessarily consistent sets (or "silhouettes"). We will discuss this ambiguous terminology (and explain the meaning of the word "finite") later in the chapter. Here we only note that if i C i is a non-empty intersection of arbitrary visual cones, then it can be viewed as the visual hull associated with the subsets Ti = P i ( j C j ) ⊆ T i , which will always be consistent. In fact, consistency is clearly equivalent to the fact that Ti = P i ( j C j ) = T i for all i = 1, . . . , n, or to T i ⊆ P i ( j C j ), since the opposite inclusion is always true. We collect a few other simple but useful properties. Proposition 4.2.3. Let T 1 , . . . , T n be arbitrary sets in P 2 . 1. T 1 , . . . , T n are consistent if and only if for each i = 1, . . . , n, and for all u i ∈ T i , the visual ray P -1 (u i ) intersects j =i C j .

H = i C i where each C i = P -1 i (T i ) is a visual cone.

Consistency of Image Sets

2. T 1 , . . . , T n are consistent if and only if

T i ⊆ P i   j =i C j   , ∀i ∈ {1, . . . , n}. (4.1)
See Figure 4.2.

3. If T 1 , . . . , T n are consistent, then any subfamily T i 1 , . . . , T is is consistent (for the associated cameras P i 1 , . . . , P is ).

Proof. The first property follows from the fact that T i ⊆ P i ( j C j ) can be expressed as

P -1 i (u i ) ∩ j C j = ∅ for all u i ∈ T i , which in turn is equivalent to P -1 i (u i ) ∩ j =i C j = ∅, since P -1 i (u i ) ⊆ C i .
The second and third properties are consequences of the first one.

The last point in this result suggests the following natural definition. Definition 4.2.4. The sets T 1 , . . . , T n are k-consistent (for some k ≤ n) if every subfamily of sets T i 1 , . . . , T i k is consistent. Proposition 4.2.3 implies that consistency (or n-consistency) implies k-consistency for every k. Much of our discussion in the following sections will be aimed at understanding whether some version of the converse holds, i.e., if k-consistency can be used to deduce consistency.

This might be a good moment to point out that the notion of geometric consistency discussed in this chapter is somewhat independent from a more intuitive (but less formal) concept of "similarity" of appearance. Indeed, consistent silhouettes may actually look completely different (as in the example in Figure 4.1), while, on the other hand, almost identical silhouettes may be geometrically inconsistent. Thus, the concept might be well suited for being used alongside more traditional feature-based methods for recognition.

Finally, for the rest of this section we will make the following assumption for all n-tuples of closed image sets T 1 , . . . , T n and cameras P 1 , . . . , P n : (A) For each camera center c i and visual cone C j , with i = j, c i does not belong to C j . This condition is useful for excluding various degenerate situations.

Pairwise consistency

Let us assume that we are given only two image sets T 1 , T 2 and two cameras P 1 , P 2 . According to Proposition 4.2.3, the sets T 1 , T 2 are consistent if and only if

T 1 ⊆ P 1 (C 2 ) , and T 2 ⊆ P 2 (C 1 ) . ( 4.2)
In other words, we require for each set to be contained in the projection of the visual cone associated with the other one. Pairwise consistency is closely related to the popular epipolar tangency constraint [START_REF] Åström | Generalised epipolar constraints[END_REF][START_REF] Kwan-Yee | Reconstruction of sculpture from its profiles with unknown camera positions[END_REF]. Indeed, we can restate the condition (4.2) in terms of epipolar geometry as follows. We recall here that an "epipolar line" in an image is any line that contains an epipole (the the projection of the other pinhole); the set of epipolar lines in the two images are in "epipolar correspondence", i.e., they are related by a fixed homography P 1 → P 1 .

Proposition 4.2.5. Two image sets T 1 , T 2 are consistent if and only if the set of epipolar lines in the first image which intersect T 1 is in epipolar correspondence with the set of epipolar lines in the second image which intersect T 2 .

Proof. The statement can be seen as a consequence of the first property in Proposition 4.2.3. In fact, the epipolar correspondence condition guarantees that for all i = 1, 2, and for every point u i ∈ T i there exists at least one corresponding point u j ∈ T j (j = i), so that triangulating all pairs of associated points (i.e., intersecting the cones C 1 , C 2 ) we obtain a set R ⊆ P 3 \ {c 1 , c 2 } that projects exactly onto T 1 and T 2 . Note that assumption A is equivalent to the fact that, in each image, the epipole lies outside of the given set.

If we assume that T 1 , T 2 are connected regions bounded by smooth curves, then pairwise consistency essentially reduces to the fact that extremal epipolar tangents (i.e., "outermost" epipolar lines that are tangent to the contours) are in epipolar correspondence [START_REF] Boyer | On using silhouettes for camera calibration[END_REF]. See Figure 4.3. On the other hand, pairwise consistency does not require non-extremal epipolar tangents to be matched. In fact, perhaps somewhat unintuitively, the visual hull generated by two silhouettes with non-extremal epipolar tangencies (matched or unmatched) will always present a complicated topology (see Figure 4.4). The difference between the two examples in the figure is that in the case of unmatched internal epipolar tangencies (top), any object which is consistent with the silhouettes must project with occlusions (because the inconsistent "branches" are necessarily projections of different parts of the observed object); on the other hand, when internal epipolar tangencies are matched (bottom), there exists an object (different from the visual hull) that projects onto the silhouettes without occlusions. We refer to Section 4.2.4 for a more detailed discussion.

Returning to the case of an arbitrary number of sets, we have previously observed that if T 1 , . . . , T n are consistent, then they are 2-consistent, or "pairwise consistent" (see Definition 4.2.4). The converse however is not true: this was pointed out in [START_REF] Boyer | On using silhouettes for camera calibration[END_REF][START_REF] Man | Visual hull construction, alignment and refinement for human kinematic modeling, motion tracking and rendering[END_REF], and is illustrated by our example in Figure 4.6. On the other hand, the following result shows that pairwise consistency implies an "approximate" version consistency in the case of three sets. This can be seen as a generalization of the fact that three non-coplanar visual rays that intersect pairwise are all concurrent.

Proposition 4.2.6. Let T 1 , T 2 , T 3 be connected closed sets that are pairwise consistent.

If the pinholes of the cameras c 1 , c 2 , c 3 are not collinear, and if each visual cone C i does not intersect the plane spanned by c 1 , c 2 , c 3 , then i C i is not empty.

Proof. It is sufficient to prove that, say,

T 1 ∩ P 1 (C 2 ∩ C 3 ) is not empty. Let R = P 1 (C 2 ) ∩ P 1 (C 3
). The assumptions on the pinholes guarantee that R is a quadrilateral (it is the intersection of two connected projected cones; see Figure 4.5). From pairwise consistency (4.2), we know that T 1 is tightly "inscribed" in R, meaning that T 1 ⊆ R and T 1 intersects of the four edges of R. The same holds for This statement closely resembles "Helly-type" theorems in computational geometry [START_REF] Csaba | Handbook of discrete and computational geometry[END_REF], and is a generalization of the fact that point correspondence is always implied by tripletwise point correspondence. An inspection of the proof also shows that it is not actually necessary for the sets T i to be convex, but only for their intersections with epipolar lines (rather than all lines) to be intervals. Returning to the case of arbitrary image sets, we see from (4.2) that pairwise consistency is equivalent to

P 1 (C 2 ∩ C 3 ): indeed P 1 (C 2 ∩ C 3 ) ⊆ P 1 (C 2 ) ∩ P 1 (C 3 )
T i ⊆ j =i P i (C j ), i = 1, . . . , n. (4.3)
Moreover, if the sets T i are consistent there is a chain of inclusions T i ⊆ P i j =i C j ⊆ j =i P i (C j ). In fact, as pointed out in [START_REF] Boyer | On using silhouettes for camera calibration[END_REF], pairwise and general consistency can be seen as the same formal condition except for the inverted order of the projection and intersection of the visual cones.

It is also natural to ask how consistency constrains the shape of the image sets. For this, we observe that if T 1 , . . . , T n are consistent for P 1 , . . . , P n , then a set, say T 1 , may be replaced by any strictly larger one T1 without affecting consistency, provided that T1 ⊆ P 1 j =1 C j : indeed, this guarantees that condition (4.1) remains satisfied in each image. For example, as shown in Figure 4.8, we are allowed to modify the shape of a set T i with a small protrusion, in the neighborhood of any point that does not belong to the boundary of P i j =i C j . On the other hand, points belonging to the intersection of T i with the boundary of P i j =i C j are more constrained: we will refer to these points as "tangential points", since the associated visual rays are "tangent" to the visual hull associated with the remaining silhouettes j =i C j . A full understanding of tangential points and their relationship with consistency requires further study. For the moment, we point out that the visual ray corresponding to a tangential point will typically graze the surface of one of the remaining visual cones, so that the ray will correspond to an epipolar tangent (not necessarily extremal) for the associated image (Figure 4.9, left). In general, it is also possible for the ray to intersect the set j =i C j only at an "intersection curve" [START_REF] Lazebnik | Projective visual hulls[END_REF] (so that it is not tangent to any cone) giving rise to a "tangential triple point" on the visual hull surface (Figure 4.9, right). However, it is not hard to realize that this second case will not occur for generic projections of smooth solids, because points on a smooth surface are never visible from three generic viewing directions.

Topology of the visual hull from two views

In this section, we study the topology of the visual hull arising from two consistent closed sets. This clarifies our discussion in Section 4.2.2, particularly in reference to Figure 4.4. The main tool for our analysis will be the (circular) Reeb graph [START_REF] Biasotti | Reeb graphs for shape analysis and applications[END_REF]. Definition 4.2.8. Let X be a topological space, and let f : X → S 1 be a continuous function, where S 1 denotes unit circle. Define an equivalence relation ∼ on X where p ∼ q whenever p and q belong to the same connected component of a level set f -1 (c) for some c ∈ S 1 . The circular Reeb graph is defined as the quotient space X / ∼ endowed with the quotient topology. 4In practice, the circular Reeb graph is obtained simply by contracting every connected component of a level set to a point: see Figure 4.10. For general functions, the (circular) Right: a tangential point may also be the projection of a "tangential triple point": in this case, the associated viewing ray is not tangent to any visual cone. Reeb graph will be an actual graph (1-manifold with singularities). Let us now consider two cameras P 1 , P 2 with distinct centers c 1 , c 2 , two consistent closed sets T 1 , T 2 , and their associated visual hull H. We also indicate with = c 1 ∨ c 2 the line passing through the centers: we assume that H does not intersect (this is guaranteed by our assumption (A) from Section 4.2.1). Let us choose an arbitrary parameterization of the pencil of planes through , so that the pencil may be seen as a topological circle P 1 ∼ = S 1 . We thus can define a projection map Q :

P 3 \ → P 1 ∼ = S 1
, by associating a point p with the plane passing through p in the pencil of planes through . The parameterization of the planes through also induces parameterizations of the epipolar lines in each image: we thus can define two projection maps Q ij : P 2 \ {e ij } → P 1 ∼ = S 1 that associate a point in image i with the corresponding epipolar line relative to the j-th epipole e ij . We have the following commutative diagram:

Consistency of Image Sets H T 1 P 1 < T 2 P 2 > S 1 Q ∨ Q 21 < Q 12 >
The visual hull H is actually the fiber product of the maps Q ij : T i → S 1 [START_REF] James R Munkres | Topology: a first course[END_REF]: in other words, H is in bijection with

T 1 × S 1 T 2 = {(u, v) ∈ T 1 × T 2 | Q 12 (u) = Q 21 (v)} (4.4)
(and, in fact, one can also show that H is topologically equivalent to T 1 × S 1 T 2 with the topology induced by the product topology). As a consequence, we obtain following simple but useful result:

Proposition 4.2.9. If G T 1 , G T 2 are the Reeb graphs of T 1 , T 2 for Q 12 , Q 21 respectively,
then the Reeb graph of H for P is given by the (topological) fiber product of

G T 1 × S 1 G T 2 , that is: G H = {(u, v) ∈ G T 1 × G T 2 | Q 12 (u) = Q 21 (v)}, (4.5)
where u, v denote the level sets associated with u and v respectively.

Proof. From (4.4) we see that

Q -1 (c) ∼ = Q -1
12 (c)×Q -1 21 (c), which implies that the claim is true set-theoretically. To show that the result also holds topologically, we first note that the diagram illustrated above passes to the quotient on the associated Reeb graphs, so by universality of the fiber product we know that there exists a continuous map α :

G H → G T 1 × S 1 G T 2 ,
given by α(p) = (P 1 (p), P 2 (p)). To prove that α is an homeomorphism we only need to show that it is an open map. This is a simple verification: if V ⊆ G H is an open set, it corresponds to an open set V ⊆ H, which is mapped via P 1 , P 2 to two open sets V1 , V2 in T 1 , T 2 respectively (since projections are open maps), which pass to the quotient as open sets

V 1 , V 2 in H T 1 , H T 2 .
We also point out that the projections induce maps on the Reeb graphs

P 1 : G H → G T 1 , P 2 : G H → G T 2 that
are surjective (because of consistency). In fact, it is easy to realize that the same property must hold more generally for any set S ⊆ P 3 \ {c 1 , c 2 } such that P 1 (S) = T 1 and P 2 (S) = T 2 : Proposition 4.2.10. Let S ⊆ P 3 \ {c 1 , c 2 } such that P 1 (S) = T 1 and P 2 (S) = T 2 , and let G S be the associated Reeb graph for the map P : S → S 1 (as previously defined). Then the projections P 1 , P 2 induce surjective continuous maps

P 1 : G S → G T 1 , P 2 : G S → G T 2 .

Consistency of Image Sets and Visual Hulls

Proof. The maps P i : G S → G T i , i = 1, 2 defined by P i (p) = P i (p) are surjective because of the assumption that P i (S) = T i .

We can now use the Reeb graph to clarify Figure 4.4 in Section 4.2.2. For example, Figure 4.11 shows two silhouettes T 1 , T 2 and their Reeb graphs G T 1 , G T 2 (we assume rectified epipolar geometry), together with the Reeb graph of the associated visual hull R H , which is easily computed using (4.5): points in G H represent possible pairs in G T 1 , G T 2 that are at the same "height". 4.12 shows that when the Reeb graphs G T 1 , G T 2 have non-extremal maxima of different heights (which represent unmatched internal epipolar tangencies), then H must have some level sets with at least three connected components (i.e., the visual hull must have least three "local maxima") in order for surjective continuous projections to G T 1 , G T 2 to exist (Proposition 4.2.10). On the other hand, if the silhouettes had matching internal epipolar tangencies, two connected components would have been sufficient. Convex sets in projective space. Let us recall here that an affine chart in projective space P n is simply the complement of any hyperplane A = P n \ H: in practice, A may be seen as a copy of affine space in P n . In fact, the complement of the hyperplane

H = {[x] | ϕ(x) = 0} in P n can be written as A = {[x] ∈ P n | ϕ(x) = 1}, which is in bijective correspondence with {x ∈ R n+1 | ϕ(x) = 1} in R n+1 .
In the following, it will be convenient to say that a set T ⊆ P n is finite if its closure is contained in some affine chart (i.e., if the closure of T does not intersect all hyperplanes in P n ).

For any set S ⊆ R N +1 , we can consider the projectivization PS = {[x] ∈ P n | x ∈ S} ⊆ P n , consisting of all classes of vectors up to scale that contain elements of S. Conversely, if T ⊆ P n is connected and finite, there are exactly two pointed cones C 1 , C 2 such that PC i = T , and these are such that C 1 = -C 2 . We will say that C i is a cone over T . We now give the following definition: As mentioned in Section 4.3.1, the following fact holds. Proposition 4.3.2. A finite set K ⊆ P n is convex if and only if there exists a affine chart where K is compact and convex in the affine sense.

Proof. If C is a pointed convex cone in R n+1 , there exists ϕ ∈ (R n+1 ) * such that ϕ(x) < 0 for all x ∈ C \{0}. Thus, if K = PC, K is contained in the affine chart {[x] ∈ P n | ϕ(x) = 1}
, and in this chart it is compact and convex as an affine set. Conversely, if K is compact and convex in {

[x] ∈ P n | ϕ(x) = 1}, then C = {x ∈ R n+1 | [x] ∈ K, ϕ(x) ≥ 0} is a closed pointed convex cone in R n+1 such that PC = K.
For any finite and connected set T ⊆ P n , we define the convex hull in P n of T as PC where C ⊆ R n+1 is the convex hull of one of the two pointed cones over T (the projectivization makes the choice of C irrelevant). Finally, we can define the dual T • ⊆ (P n ) * of a finite set T ⊆ P n as PC • for a pointed cone over C over S (again, the choice is irrelevant). Note that since Definition 4.3.1 requires a convex set to be the projectivization of pointed convex cones, it is possible that the dual of a convex set K ⊆ P n may not be convex: indeed, if C ⊆ R n+1 is a convex cone with empty interior, then C • is not pointed. The next result shows that the convex dual of a set T ⊆ P n coincides with the closure of the hyperplanes that do not meet T in P n :

Proof. By definition

T • = PC • and C • = {ϕ ∈ (R n ) * | ϕ(x) ≤ 0, ∀x ∈ C}: it is clear now that if [ϕ] ∈ PC • then necessarily ϕ ∈ C • or -ϕ ∈ C • .

Duality and visual hulls

Given a camera P : P 3 \ {c} → P 2 with pinhole c we write P ∨ : (P 2 ) * → (P 3 ) * for the dual embedding that associates lines in P 2 with planes in P 3 through c (see Section 1.2.5). The image of P ∨ is a plane in the dual space (P 3 ) * , namely the dual c ∨ of the pinhole c (the set of planes containing c). The following result describes perspective projections of convex sets from the dual point of view. Proposition 4.3.4. Let P be a perspective projection with center c, and let K ⊂ P 3 \{c} and L ⊂ P 2 be a convex sets. Then

P (K) = L if and only if P ∨ (L • ) = K • ∩ c ∨ . (4.7)
Indeed, the geometric intuition for (4.7) is that a line in P 2 does not meet L if and only if its preimage plane in P 3 does not meet K. An analytical proof is as follows.

Proof. It is enough to show the analogous property for cones in R 4 . For any finite connected set S ⊆ P n , we denote by S an arbitrary pointed cone in R n+1 over S. We also indicate with P a linear projection map P : R 4 → R 3 associated with a perspective projection P on P 3 . We need to show that P ( K) = L is equivalent to P ∨ ( L• ) = K• ∩c ⊥ , where P ∨ : (R 3 ) * → (R 4 ) * is the dual map associated with P (i.e., the map ϕ → ϕ • P for ϕ ∈ (R 3 ) * ), and c ⊥ denotes the hyperplane in (R 4 ) * of all linear functionals that vanish on the null-space of P . Indeed, we have that:

L = { P (x) | x ∈ K} ⇔ L• = {ϕ ∈ (R 3 ) * | ϕ( P (x)) ≤ 0, ∀x ∈ K} ⇔ L• = {ψ ∈ (R 4 ) * | ψ(x) ≤ 0, ∀x ∈ K} ∩ Im( P ∨ ) ⇔ L• = K• ∩ c ⊥ ,
where we used the fact that Im( P ∨ ) = c ⊥ . This result shows that projecting a convex object in P 3 is equivalent to taking a planar section of its dual. We also refer to [START_REF] Ponce | On Image Contours of Projective Shapes[END_REF], where Ponce and Hebert used a similar idea in a smooth setting to investigate the qualitative relationship between image contours and projective shapes. Motivated by Proposition 4.3.4, we introduce a notion of "sectional consistency". We assume that we are given projective maps J 1 , . . . , J n , where each J i : P 2 → P 3 identifies P 2 with a plane π i in P 3 . The planes π 1 , . . . , π n are distinct. Definition 4.3.5. A family L 1 , . . . , L n of sets in P 2 is said to be sectionally consistent (relative to the embeddings J 1 , . . . , J n ) if there exists

K ⊆ P 3 such that K ∩ π i = J i (L i ) for all i = 1, . . . ,
H = K • , where K is the convex hull of P ∨ 1 (L • 1 ), . . . , P ∨ n (L • n ). 5
Proof. The first claim is a direct consequence of (4.7). For the second part, we can use the fact that for arbitrary closed convex cones

K 1 , . . . , K n in R N [Conv(K 1 ∪ . . . ∪ K n )] • = K • 1 ∩ . . . ∩ K • n . (4.8)
See for example [START_REF] Schneider | Convex bodies: the Brunn-Minkowski theory[END_REF]. In fact, it is sufficient to take K i = P ∨ i (L • i ), and observe that K • i is the visual cone C i . This follows from the following relations for cones in R 4 (we use the same notation as in the proof of Proposition 4.3.4):

( P ∨ i ( L• i )) • = {x ∈ R 4 | ψ(x) ≤ 0 ∀ψ ∈ P ∨ i ( L• i )} = {x ∈ R 4 | ϕ( P (x)) ≤ 0 ∀ϕ ∈ L• i } = {x ∈ R 4 | P (x) ∈ Li }.
Note that while {x ∈ R 4 | P (x) ∈ Li } is a convex cone in R 4 , its projectivization coincides with the two sided projective cone C i , which is not convex for Definition 4.3.1 (in fact, the set P ∨ i (L • i ) has empty interior). Consistency is arguably more intuitive in its dual formulation, since planar sets in space are conceptually easier to grasp than families of cones. For example, the pairwise consistency constraint for two convex silhouettes T 1 , T 2 dualizes to the fact that P ∨ 1 (T • 1 ) and P ∨ 2 (T • 2 ) have the same intersection with the "dual baseline" (c 1 ∨ c 2 ) ∨ : see Figure 4.15.

5 The convex hull of a closed set S contained in an affine chart in P n is well-defined only when the set S is connected: this is true for

P ∨ 1 (L • 1 ) ∪ . . . ∪ P ∨ n (L • n ) since assumption (A) guarantees that P ∨ i (L • i ) ∪ P ∨ j (L • j )
is connected for all i, j. We also point out a duality between the "combinatorial" structure of the visual hull H and of its dual K = H • : by interpreting the supporting planes of H as points on the boundary of K, we see that a plane π supports a cone (patch) C i if and and only if, in the dual space, it represents a point belonging to the planar curve that is the boundary of P ∨ i (L • i ); on the other hand, if a supporting plane π meets the visual hull at the intersection of two cones C i , C j (i.e., at an intersection curve, see Section 4.6), then it is dual to a point on a ruled patch, joining the two boundaries of P ∨ i (L • i ) and P ∨ j (L • j ); finally, if π supports H at the intersection of three cones C i , C j , C k (i.e., a triple point), then it represents a point of the dual hull K that belongs to a planar patch, spanning across the three boundaries of P ∨ i (L • i ), P ∨ j (L • j ), P ∨ k (L • k ) (note that ruled and planar patches are typical for convex hulls of curves in space [START_REF] Koenderink | Solid shape[END_REF]). See Figure 4.16.

The notion of sectional consistency given in Definition 4.3.5 is reminiscent of questions in geometric tomography [START_REF] Richard | Geometric tomography[END_REF], or stereology [START_REF] Howard | Unbiased stereology: three-dimensional measurement in microscopy[END_REF]. In tomography, for example, the duality between projections and sections is well studied, but typically in an affine setting that considers only orthographic projections. Nevertheless, it is quite possible that tools from these neighboring fields could provide interesting new insight for problems in computer vision.

Consistency and Line Incidences

We now re-introduce consistency as a property of families of lines in 3D. Our previous definition of consistency did not in fact depend on the representation of the image sets in terms of coordinates P 2 , but only on the incidence properties of the associated visual rays. After giving some basic definitions in Section 4.4.1, we summarize in Section 4.4.2 some results from our paper [START_REF] Bukh | Consistent sets of lines with no colorful incidence[END_REF], where we investigated the relationship between consistency and k-consistency for finite families of lines.

Consistency for lines

If L 1 , . . . , L n are n sets of lines P 3 , we define an n-incidence as an n-tuple of lines 1 , . . . , n that are all concurrent and where each i belongs to L i . More generally, for any subset of indices I = {i 1 , . . . , i k } ⊂ {1, 2 . . . n} we say that a k-incidence for I is a k-tuple of lines i 1 , . . . , i k that are all concurrent and where i j belongs to L i j . It is easy to see that the image sets T 1 , . . . , T n in P 2 are consistent (resp. k-consistent) for some cameras P 1 , . . . , P n in the sense of Definition 4.2.1 (resp. Definition 4.2.4) if and only if the families of lines L i associated with the visual cones C i = P -1 1 (T i ) are consistent in the sense of Definition 4.4.1 (resp. Definition 4.4.2). This is completely analogous to the fact that the correspondence of points across different images can be viewed as a geometric condition on the concurrence of visual rays. Note that we did not require for each family of lines L i to be concurrent at a point, even though this will always be the case for families of visual rays coming from pinhole cameras: the more Consistency of Image Sets and Visual Hulls general setting can be useful for studying consistency for non-central imaging systems like the ones described in Chapters 2 and 3.

In [START_REF] Bukh | Consistent sets of lines with no colorful incidence[END_REF] we studied the consistency for families of lines from the viewpoint of discrete geometry. In particular, our main result in that paper shows that it is possible to construct finite but arbitrarily large families of lines L 1 , . . . , L n that are k-consistent but have not even one (k + 1)-incidence. Our discussion in [START_REF] Bukh | Consistent sets of lines with no colorful incidence[END_REF] mainly focuses on mathematical questions, that are not always directed connected to vision. On the other hand, consistency in this finite setting is related to multi-view point correspondences, and might be useful for resolving ambiguities in SIFT matching. For this reason, we summarize without proofs some of the main results of [START_REF] Bukh | Consistent sets of lines with no colorful incidence[END_REF] in the following subsection. The interested reader is referred to the paper for more details.

k-consistency with no colorful incidence

We have seen in Figure 4.6 that there exist image sets (and thus families of lines) that are pairwise consistent but not globally consistent. The following example illustrates that is also possible to construct arbitrarily large finite families of concurrent lines L 1 , L 2 , L 3 that are 2-consistent but have no 3-incidences.

Example 4.4.3. Let c 0 , c 1 , c 2 be three non-collinear points in P 3 . For any n ≥ 1, we construct a sequence of points x 0 , x 1 . . . , x 3n so that x 0 is arbitrary, and x i+1 is a generic point on the line (i,i+1) = c (i mod 3) ∨ x i . For each j ∈ {0, 1, 2} define L j to be the set of lines (i,i+1) with i ≡ j (mod 3). These families are 2-consistent but have no 3-incidences. ♦

For k ≥ 3, showing the existence of families of lines that are k-consistent but have no (k + 1)-incidence requires much more work. In the rest of this section, it will convenient to refer to each family L a color. An n-incidence will sometimes be referred to as a colorful incidence. Two proofs of the following result are given in [START_REF] Bukh | Consistent sets of lines with no colorful incidence[END_REF].

Theorem 4.4.4. For any k ≥ 3, there exist arbitrarily large families L 1 , . . . , L k+1 of concurrent lines in P 3 that are k-consistent but have no colorful incidence.

Both proofs of this result in [START_REF] Bukh | Consistent sets of lines with no colorful incidence[END_REF] construct families of lines L 1 , . . . , L k+1 in R k+1 ⊂ P k+1 with desired properties (k-consistent with no colorful incidence) and with color classes consisting of parallel lines selected from a regular grid. A generic projection to P 3 , preserves incidences and therefore the properties of the construction. To obtain the families L 1 , . . . , L k+1 in R k+1 , we use two different strategies: the first approach is probabilistic, while the second approach uses linear algebra over finite vector spaces.

In both constructions, every color class is concurrent. The probabilistic argument is asymptotic and proves the existence of configurations where every line is involved in many k-incidences for every choice of k -1 other colors. The algebraic construction is explicit and is minimal in the sense that removing any line breaks the k-consistency. In algorithmically, by enforcing appropriate algebraic constraints on coordinates using a computer algebra system (see, e.g., [START_REF] Sturmfels | Computational algebraic geometry of projective configurations[END_REF]).

Visual Hulls: Theory

In this section, we introduce a basic theory of visual hulls in projective space. We begin with an informal discussion, explaining the distinction between "finite" visual hulls, defined from a finite set of images, and previously considered in Sections 4.2 and 4.3 (see Definition 4.2.2), and "complete" visual hulls, associated with objects in space. Precise definitions are given in Section 4.5.2, where we also introduce a duality between sets in P 3 and in Gr(1, P 3 ). This duality is very similar to the duality between convex projective sets used in Section 4.3, and indeed visual hulls and convex hulls can be defined using almost the same construction.

What is a visual hull?

As mentioned in the introduction to the chapter, the term "visual hull" is often used for two different but related concepts. According to Laurentini [START_REF] Laurentini | The Visual Hull Concept for Silhouette-Based Image Understanding[END_REF], the visual hull of an object R is the largest set containing R whose opaque perspective projections are the same as those of R for all viewpoints. 6 More commonly, however, the term "visual hull" refers to the intersection of the visual cones obtained by observing R from a finite number of viewpoints. This object is the largest set whose perspective projections are the same as those of R only from the fixed viewpoints. To emphasize the distinction between the two notions, we use complete visual hull for the former, and finite visual hull for the latter. The complete visual hull of an object is, in general, quite similar to the original shape, the main difference being "filled in" concavities. The finite visual hull, on the other hand, will gradually "converge" to the complete visual hull as the number of views increases, but a particular instance may be significantly different from the original object being observed (see Figure 4.19). As remarked in Section 4.2.2, the finite visual hull is also not necessarily the most "intuitive" shape associated with the given projections, especially for a small number of views.

The theory of visual hulls has been studied for polyhedral [START_REF] Laurentini | The Visual Hull Concept for Silhouette-Based Image Understanding[END_REF], smooth [START_REF] Bottino | The Visual Hull of Smooth Curved Objects[END_REF], or piecewise smooth [START_REF] Bottino | The visual hull of piecewise smooth objects[END_REF] surfaces in R 3 . Here, we consider instead arbitrary sets in P 3 . Extending definitions from surfaces to sets does not require additional effort and, as we will argue shortly, the more general viewpoint can be useful for dealing with noisy (non-consistent) silhouettes. Regarding the projective setting, it is noted in [START_REF] Lazebnik | Projective visual hulls[END_REF] that the construction of the visual hull only relies on contacts between viewing rays and the observed object, and such properties can be considered independently of the euclidean or affine structure on 3D space. In [START_REF] Lazebnik | Projective visual hulls[END_REF], Lazebnik et al. actually make use of oriented projective geometry [START_REF] Stolfi | Oriented projective geometry: A framework for geometric computations[END_REF] in order to triangulate cone strips, since they exploit the notion of "near" and "far" intersection edges relative to the viewing direction. Oriented projective geometry is indeed necessary (and useful) to model the fact that that visible points lie "in front" of a camera. On the other hand, this setting requires some awkward constructions, since every geometric entity is doubled to take into account the possible orientations. Our approach is arguably simpler, and does not require orientation.

Visual and convex hulls

Let S be a set in P 3 , which for simplicity we assume to be closed, connected and with non-empty interior. For any point c in P 3 not in S, we write (S | c) for the set of lines through c that meet S:

(S | c) = { ∈ Gr(1, P 3 ) | c ∈ , ∩ S = ∅} ⊂ Gr(1, P 3 ). (4.9)
The set (S | c) can be viewed as the image of S for the geometric camera associated with the pinhole c (see Chapter 2). With this notation, we are interested in recovering an approximation of S from (S | c 1 ), . . . , (S | c k ). This is equivalent to recovering S from some perspective images in P 2 , assuming that all the camera parameters are known. From this definition we see that if S = VH (S) is the complete visual hull of S, then (S | c) = (S | c) for all points c in P 3 (because the set of lines meeting S is the same as the set of lines meeting S ), and that S is the largest set with this property. Note also that taking the visual hull can be viewed a "closure operator", i.e., VH (VH (S)) = VH (S).

Intuitively, the visual hull resembles the more conventional notion of the convex hull. We can make this idea precise by noting that the convex hull of a connected closed set S with non-empty interior in P 3 can be defined geometrically as the set CH (S) of points x in P 3 such that all planes through x meet S:

CH(S) = {x ∈ P 3 | x ∈ H ∈ (P 3 ) * ⇒ H ∩ S = ∅} ⊂ P 3 . (4.11)
When S is contained in an affine chart P 3 \ H of P 3 , this is equivalent to the convex hull in the traditional sense of convex geometry. Moreover, we can extend the notation from (4.9) so that if is a line and S is a closed connected set in P 3 , then (S | ) is the set of planes through that meet S. The convex hull S = CH (S) of a set S now has the property that (S | ) = (S | ) for all lines in P 3 , and it is the largest set with this property.

Our definitions of visual and convex hulls suggest a more general notion of "k-convex hull" of a set S in P n (with k ≤ n), which consists of points x in P n such that every (n -k)-dimensional linear space through x meets S. By construction, this is the largest set whose image under any linear projection P n P k is the same as for S. Similar notions have indeed been introduced, although to our knowledge always in a Euclidean setting (see for example [START_REF] Petrovich | Some extensions of the class of k-convex bodies[END_REF][START_REF] Dg Larman | On visual hulls[END_REF]). In our opinion, this is less convenient for dealing geometrically with linear projections.

Returning to case of visual hulls, the problem of recovering a set S in P 3 from a finite number of projections (S | c 1 ), . . . , (S | c n ) leads to the following natural definition. As before, S is a closed and connected set, and c 1 , . . . , c n are points in P 3 not beloning to S. Definition 4.5.2. The finite visual hull associated with S and c 1 , . . . , c n is the set of points x in P 3 such that the lines i = x ∨ c i , for i = 1, . . . , n, all meet S:

FVH (S | c 1 , . . . , c n ) = {x ∈ P 3 | (x ∨ c i ) ∩ S = ∅, i = 1, . . . ,
F V H(S | c 1 , . . . , c n ).
It is also interesting to note that the largest set in P 3 such that (S | i ) = (S | i ) for a finite number of lines i = 1, . . . , n is a polyhedron. Hence, in a rather precise sense, a finite visual hull can be seen as a "generalized polyhedron". See also [START_REF] Lazebnik | Projective visual hulls[END_REF] and Section 4.3, where the combinatorial polyhedral structure of the visual hull is discussed. Finally, visual hulls can also be understood using a form of "generalized duality". As explained in Section 4.3, we can define the convex dual of a finite connected set S in P 3 as the closure of the set of all hyperplanes H such that H ∩ K is empty:

S • = cl{H ∈ (P 3 ) * | ∀x ∈ S, x / ∈ H} ⊂ (P 3 ) * , ( 4.13)
where we write cl for the topological closure operator. Identifying P 3 and (P 3 ) * , we can symmetrically define the dual for sets in (P 3 ) * . Now if S is closed with a non-empty interior, then CH (S) = S •• holds. This leads to a similar definition for the visual hull: we associate with a set S in P 3 the (closure of the) set of lines that do not intersect S:

S λ = cl{ ∈ Gr(1, P 3 ) | ∀x ∈ S x / ∈ } ⊂ Gr(1, P 3 ). (4.14)
Conversely, we associate with a set of lines L in Gr(1, P 3 ) the set of points that do not belong to any of the lines on L:

L µ = cl{x ∈ P 3 | ∀ ∈ L x / ∈ } ⊂ P 3 . (4.15)
If S is closed with non-empty interior, then with these definitions VH (S) = S λµ . This duality also suggests a notion of convexity in the Grassmannian Gr(1, P 3 ), where convex sets in Gr(1, P 3 ) are "double duals" L µλ of sets of lines L. Thus, convex sets of lines can Figure 4.20: Assuming that a set of cameras has been fixed, the association between consistent silhouettes and finite visual hulls is bijective.

Silhouette consistency is however unstable for arbitrarily small measurement and calibration errors, which means that in practice silhouettes recovered separately in each image will never be exactly consistent, even if they are generated by the same object. In fact, as argued in [START_REF] Franco | Efficient polyhedral modeling from silhouettes[END_REF][START_REF] Franco | Exact polyhedral visual hulls[END_REF][START_REF] Lazebnik | Projective visual hulls[END_REF], real contours associated with measurement errors lack the true epipolar tangencies predicted in an idealized setting. Accordingly, "epipolar tangents" to silhouettes either miss the contour or cut it transversally (Figure 4.21, left) and, likewise, the associated cone strips do not intersect at frontier points, but appear to be interrupted. On the other hand, as previously noted (and illustrated in Figure 4.20), naively intersecting the visual cones associated with a set of inconsistent silhouettes T 1 , . . . , T n , yields in fact the finite visual hull associated with the reprojected silhouettes Ti = P i (H) ⊆ T i , which are, by construction, exactly consistent. These reprojected silhouettes Ti will present sharp cuts, occurring where an epipolar tangent intersected the original silhouette (see Figure 4.21, right). Importantly, these cuts will not occur in idealized projections of smooth surfaces, but they are admissible in images of polyhedra (as in the "exact polyhedral visual hulls" discussed in [START_REF] Franco | Exact polyhedral visual hulls[END_REF]) or for more general sets considered here. In this regard, the boundaries of the reprojected consistent silhouettes Ti = P i (H) ⊆ T i will present "degenerate" but exact epipolar tangencies, in the form of a point in one contour and a segment in the other. Such features correspond to special "frontier segments" on the visual hull surface, which are the intersections between the cone strips associated to the new silhouettes (in particular, the new cone strips are not disconnected). See Figure 4.21 (right). In conclusion, by not restricting ourselves to objects bounded by smooth surfaces, we obtain a more robust approach for dealing with finite visual hulls.

From silhouettes to finite visual hulls

We now let T 1 , . . . , T n be arbitrary silhouettes in n images, and assume that n projection matrices P 1 , . . . , P n have been fixed. It will be convenient to write H = (T 1 , . . . , T n ) for the finite visual hull associated with T 1 , . . . , T n :

H = (T 1 , . . . , T n ) = {x | P i (x) ∈ T i , i = 1, . . . , n}. (4.16)
As noted above, this representation is also equivalent to a unique one H = ( T1 , . . . , Tn ), where the sets Ti are exactly consistent (taking Ti = P i (H)).

Equation (4.16) effectively provides a complete, albeit implicit, characterization of a finite visual hull and, indeed, this description is actually used by volumetric approaches [START_REF] Worthy | Volumetric Descriptions of Objects from Multiple Views[END_REF][START_REF] Chien | Volume/surface octrees for the representation of three-dimensional objects[END_REF][START_REF] Slabaugh | A Survey of Methods for Volumetric Scene Reconstruction from Photographs[END_REF] to verify whether a voxel belongs to the visual hull or not. "Surface based" methods [START_REF] Lazebnik | Projective visual hulls[END_REF][START_REF] Matusik | Polyhedral Visual Hulls for Real-Time Rendering[END_REF][START_REF] Franco | Exact polyhedral visual hulls[END_REF], on the other hand, typically do not use (4.16) directly, and recover an explicit representation (usually a mesh) of the visual hull boundary by combining the geometric information provided by the contours in each image. However, we argue in this section that an implicit description (i.e., a family of silhouettes and camera matrices) can be used to characterize directly the boundary of the visual hull, and, in fact, all of its combinatorial features. For this, it will be convenient to think of visual hulls in terms of the joint image, which we studied in Section 1.3. We recall here that the joint image M n is an algebraic variety in (P 2 ) n (or (R 2 ) n , if we consider only finite points) that contains all point correspondences (u 1 , . . . , u n ) for the cameras P 1 , . . . , P n . Here we will use the joint image mainly as a conceptual tool, treating it as an "image-based replica" of 3D space. Indeed, a point x is uniquely associated with the n-tuple of image points (P 1 (x), . . . , P n (x)), and vice-versa: by definition, a correspondence (u 1 , . . . , u n ) may always be triangulated to a 3D point (any two of the image points are actually sufficient for this). If the camera centers are not all collinear, this association is completely bijective. 9 In this section, we assume that the cameras P 1 , . . . , P n are fixed, and always identify a 3D point with the n-tuple of its projections, We also point out that if we identify 3D space with the joint image, a visual hull H = (T 1 , . . . , T n ) for a family of arbitrary (not necessarily consistent) image sets T 1 , . . . , T n , can be characterized as

H = (T 1 , . . . , T n ) = M n ∩ (T 1 × . . . × T n ). (4.17)
Indeed, this relation expresses the fact that that x = (u 1 , . . . , u n ) belongs to H if and only if u i belongs to T i for all i = 1, . . . , n.

Intersection intervals using epipolar geometry. In this section, we will make use of a fundamental procedure from [START_REF] Matusik | Image-based visual hulls[END_REF] that computes, for a given point (pixel) in a reference image, the intersections of the associated ray with the visual hull (for convenience we will refer to these sets as "intersection intervals"). The basic idea is to compute the intersection of the viewing ray and each visual cone by intersecting the epipolar line associated with the original point and the silhouette that generated the cone (thus reducing the computation from 3D to 2D). After this operation is carried out for each visual cone, the resulting intervals are lifted back to the viewing ray (or, equivalently, to an epipolar line in a reference image) and intersected. The whole procedure does not require an explicit representation of the visual hull. See Figure 4.22.

Identifying the combinatorial structure of the visual hull. The implicit description of equation (4.16) characterizes a finite visual hull as the set of points which project inside the given silhouettes. Identifying 3D space with the joint image, this is equivalent to considering all image correspondences in which each image component belongs to the silhouette, as in equation (4.17). In particular, it is possible to use multi-view geometry to investigate the topological and combinatorial structure of the visual hull.

In the following, we write ∂S i to indicate the boundary of S i .

• The cone strips of H = (T 1 , . . . , T n ) can be characterized as:

∂ i H ={ x | P i (x) ∈ ∂T i , P j (x) ∈ T j ∀j} = { (u 1 , . . . , u n ) | u i ∈ ∂T i , u j ∈ T j ∀j}, (4.18)
for i = 1, . . . , n, since these are the points belonging to H that project onto (at least) one of the silhouette contours.

• The intersection curves are the intersections between two cone strips:

∂ ij H ={ x | P i (x) ∈ ∂T i , P j (x) ∈ ∂T j , P k (x) ∈ T k ∀k} = { (u 1 , . . . , u n ) | u i ∈ ∂T i , u j ∈ ∂T j , u k ∈ T k ∀k} (4.19)
for all i, j = 1, . . . , n, and i = j (note that such sets might be empty).

• The triple points are the intersections of three (or more) cone strips:

∂ ijk H ={ x | P i (x) ∈ ∂T i , P j (x) ∈ ∂T j , P k (x) ∈ ∂T k , P l (x) ∈ T l ∀l} = { (u 1 , . . . , u n ) | u i ∈ ∂T i , u j ∈ ∂T j , u k ∈ ∂T k , u l ∈ T l ∀l} (4.20)
for all distinct i, j, k in {1, . . . , n}.

Frontier points can also be characterized in image space, since these are points that project onto pairs of "epipolar tangencies". However, as mentioned in Section 3.2, epipolar tangencies are unstable and will typically be degenerate in the case of real noisy data (Figure 4.21).

The image-based characterizations given above imply that the combinatorial components of the visual hull can be computed by walking along each silhouette contour and inspecting the intersection intervals (Figure 4. 23, top). For example, points on the intersection curves are the endpoints of all the intersection intervals for each point on an image contour, since the extremes belong to the visual hull and project onto (at least) two of the contours. Although this strategy does not immediately provide the connectivity of the points, it is simpler (and more efficient) than the tracing procedure described in [START_REF] Lazebnik | Projective visual hulls[END_REF], where the curves generated by all pairs of cameras are first computed and then "clipped" based on whether they actually belong to the visual hull or not. 10 From silhouettes to point clouds. We have observed that the combinatorial structure of the visual hull can be understood from intersection intervals at all points on the contours. In particular, the cone strips, which compose the visual hull's surface, are the union of all points on these intervals (see Figure 4.23). An explicit characterization of the surface of the visual hull can thus be obtained by sampling the intersection intervals for points on contours. This procedure, which requires the choice of two sampling parameters n and d, can be summarized as follows:

for all silhouettes S do Sample one out of every n ≥ 1 points on the contour of S. for p in the sampled points in the contour of S do Compute the intersection intervals I(p) on the ray from p (cf. Section 4.1).

for Each interval J in I(p) do Sample points at regular intervals of width d in J.

Store the sampled points together with the extremes of J. end for end for end for Boyer and Franco [START_REF] Boyer | A hybrid approach for computing visual hulls of complex objects[END_REF] actually used a similar strategy to recover points on the surface of the visual hull, however they only considered endpoints of the intersection intervals, so they did not obtain an arbitrarily dense cloud (in fact, such points always project onto two contours, and thus lie on intersection curves). 11 The sampling is simple but robust, since the contours are not assumed to be smooth or polygonal, and the silhouettes used as input need not be exactly consistent. Figures 4.24-4.28, show some results. Finally, the point cloud can be sampled with any desired density, so it may be converted to a surface mesh using standard geometry processing algorithms (see Figure 4.24, bottom where we used a "ball-pivot" algorithm [START_REF] Bernardini | The ball-pivoting algorithm for surface reconstruction[END_REF]). Figure 4.24: Top: Four views of a reconstructed point cloud (with approximately 75000 points), computed with 24 silhouettes using the procedure described in the text. Bottom: Four views of a mesh obtained using "ball-pivot" algorithm [START_REF] Bernardini | The ball-pivoting algorithm for surface reconstruction[END_REF] from a dense point cloud.

A hardness function for points on a cone strip

We pointed out in Section 4.5 that the reconstruction of a visual hull is essentially always different from the actual object being observed (Figure 4.19). In particular, even in an ideal noiseless setting, when a point x belongs to the reconstructed visual hull H, it may not belong to the actual object R that is being observed. However, there exist special points in H, sometimes known as hard points [START_REF] Laurentini | How far 3D shapes can be understood from 2D silhouettes[END_REF], which are guaranteed to belong to R: these are points for which (at least) one of the visual rays emanating from the camera centers intersects H only at x. This characterization follows from the fact that every Visual Hulls: Practice visual ray intersecting H must also intersect R (and that R is necessarily contained in H). In principle, we can be certain that the surface ∂H of a reconstructed visual hull coincides with the surface ∂R of the object being observed only if all points on ∂H are hard. On the other hand, it is easy to see that for general surfaces observed from a finite number of viewpoints this is never the case. For this reason, we introduce a simple generalization of the concept of hardness in the form of a function: Definition 4.6.1. Let x = (u 1 , . . . , u n ) be a point belonging to a cone strip ∂H i of H = (T 1 , . . . , T n ) (so that u i ∈ ∂S i ). The hardness of the point x (relative to ∂ i H), or of the image point u i , is defined as:

h(x) = h(u i ) = len(CH ( i ∩ H)) (4.21)
where CH denotes the convex hull, i is the visual ray associated to u i and len is the length measured using the Euclidean distance in R 3 .

In other words, the hardness is simply the total length of the intersection intervals along the viewing ray associated to u i . It is easy to realize that a point x has hardness zero (relative to some strip) if and only if it is a hard point, and that more generally the hardness h(x) provides an upper bound on the distance between x and a point belonging to observed object R: indeed, as mentioned above, at least one point on the intersection intervals in a viewing ray must belong to R. 12 See Figure 4.29. In practice, the hardness of a point u i ∈ ∂S i can be computed directly from image data, by recovering the intersection intervals as described in Section 4.1. Moreover, the "average hardness", for a cloud of points on the surface of the visual hull may provide a meaningful measure for the quality of the visual hull approximation. In particular, it can be used to estimate whether the current images provide a sufficiently accurate reconstruction. Figure 4.30 shows some examples.

Conclusions

In this chapter, we have studied geometric properties of image consistency and of visual hulls. These notions are in some sense complementary, since consistent image sets are projections in images of the same object in space, while visual hulls are 3D shapes corresponding to families of images. We have pointed out that image consistency can be seen as an extension of traditional multi-view geometry, which is devoted to consistency of points and lines. We also presented a "dual" interpretation of consistency, expressing conditions for planar sets to be sections, rather than projections, of a single object.

Similarly to convex hulls, we argued that visual hulls are arguably easier to understand for arbitrary (projective) sets, rather than for smooth or polyhedral surfaces. By spelling out the topological relationship between image sets and 3D shapes, we presented a simple image-based method for obtaining a representation of the boundary of the visual hull.

Our discussion was focused almost entirely on pinhole projections, however our definitions can be easily adapted to more general imaging models. For example, if C 1 , . . . , C n are geometric cameras (see Chapter 2), so that C i : P 3 Gr(1, P 3 ), then the finite visual hull of a set S for these cameras can be defined as

FVH (S | C 1 , . . . , C n ) = {x ∈ P 3 | C i (x) ∩ S = ∅, i = 1, . . . , n} ⊂ P 3 . (4.22)
This is the largest set whose whose images under C 1 , . . . , C n are the same as for S. On the other hand, our discussion on discrete sets of lines in Section 4.4.1 points to some differences between consistency for line bundles more and general sets of lines (compare

Conclusions

Theorems 4.4.4 and 4.4.6). It would be interesting to see whether these differences have any impact on reconstruction methods.

We conclude with a brief discussion on two related problems which we believe could be topics for future work. The question of whether silhouettes can be projections of the same object for unknown cameras was discussed by Bottino and Laurentini in [START_REF] Bottino | Introducing a new problem: Shape-fromsilhouette when the relative positions of the viewpoints is unknown[END_REF]. In that paper, the authors assume the extrinsic parameters of the cameras to be unknown, and analyze in detail the geometric constraints for compatibility in a particular case (orthographic viewing directions parallel to the same plane). Bottino and Laurentini also make an interesting observation, namely that if one assumes projective cameras (unknown intrinsic and extrinsic parameters), then any family of silhouettes can be projections of a single object: they observe in fact that considering a convex object, and applying local protrusions with appropriate shapes, one is able to produce arbitrary silhouettes by placing cameras near the surface. On the other hand, this property would seem to imply that SfM methods can never exploit the geometry of the silhouettes in order to recover projective camera parameters (contrary for example to the results in [START_REF] Hernández | Silhouette coherence for camera calibration under circular motion[END_REF]): indeed, if T 1 , . . . , T n are image sets that uniquely determine a configuration of cameras, then it is clear that another set T n+1 in a new image must be constrained in order for it to be consistent with T 1 , . . . , T n . We note however that the construction proposed in [START_REF] Bottino | Introducing a new problem: Shape-fromsilhouette when the relative positions of the viewpoints is unknown[END_REF] and sketched above violates our assumption (A) for all pairs of cameras and silhouettes (viewing cones must be extremely "wide", and thus contain all other centers which lie near the surface of the convex object). We believe instead that if one excludes certain regions in the space of parameters, then silhouettes will not always be compatible.

Silhouette

As an example, consider two pictures with two arbitrary silhouettes T 1 , T 2 , one in each image. In this case, the camera configurations such that T 1 , T 2 are consistent and such that each epipole lies outside the corresponding silhouette (as in condition (A)), have

Introduction

In Chapter 4 we studied projections of arbitrary sets in projective space. We now take a different look at projections of smooth algebraic curves and surfaces. In this setting, we always represent the image of an object using a plane curve. If the observed object is a curve, then the image is simply its point-wise projection. For a surface, the image is described by the outline curve, or image contour, generated by the projection. The outline is the natural sketch one might use to depict the surface (with transparencies), and it is the projection of the critical points where viewing lines are tangent to the surface. In both cases, the image curve will present singularities that arise from the projection, even if the original curve or surface was smooth. If we now let the camera travel along a path in 3-space, this path naturally breaks up into segments according to how the image looks like: within each segment, the topology and singularities of the image curve do not change. The sharp transitions in the appearance of the image curve are known as visual events, and play an important role in the study of visual perception. For example, any object has only a finite number of "stable appearances", or aspects, which correspond to regions of viewpoints in the complement of the events. The overall structure of aspects and events can also be encoded in an aspect graph (or view graph) [START_REF] Bowyer | Aspect graphs: An introduction and survey of recent results[END_REF], in which vertices correspond to aspects, and edges correspond to visual events between stable views. Hence, the aspect graph describes all possible evolutions of appearances of a given surface.

Traditionally, visual events have been defined mathematically using catastrophe theory [START_REF] Thom | Structural stability and morphogenesis[END_REF][START_REF] Bröcker | Differentiable germs and catastrophes[END_REF], a branch of singularity theory [START_REF] Whitney | On singularities of mappings of Euclidean spaces. I. Mappings of the plane into the plane[END_REF][START_REF] William | Curves and Singularities: a geometrical introduction to singularity theory[END_REF]. Roughly speaking, singularity theory tries to classify stable germs (i.e., local equivalence classes) of differential maps, exhibiting whenever possible "normal forms" that can be used to represent them. Catastrophe theory, in particular, studies stable and non-stable critical points of "potential functions". This theory can be used to describe sudden qualitative changes of behaviors of very general dynamical systems, with applications ranging from biology to behavioral sciences [START_REF] Christopher | Catastrophe theory[END_REF]. On the other hand, the classification of germs is based on analytical tools, which do not seem to capture directly the special geometry involved in visual problems. Moreover, the theory is most well-suited for dealing with "local" instabilities, while "multi-local" events are also important in vision.

In this chapter, we revisit visual events from a more geometric (and projective) viewpoint.

In the language of algebraic geometry, any curve or surface in projective space has various "associated hypersurfaces". For example, if X is a smooth surface, its dual surface X ∨ in (P 3 ) * contains all tangent planes to X; its associated Hurwitz hypersurface in Gr(1, P 3 ) contains all tangent lines to X. It is intuitively clear that these objects determine the evolution of the contours generated by X. We make this connection precise, explaining how all visual events can be obtained from the iterated singular loci of associated hypersurfaces. We also discuss effective ways of computing event surfaces using algebraic methods.

Previous work

The appearance of a solid object under a continuously varying viewpoint was studied in the 1970's by Koenderink and van Doorn [START_REF] Koenderink | The singularities of the visual mapping[END_REF][START_REF] Koenderink | The internal representation of solid shape with respect to vision[END_REF], in the context of visual perception in psychology and artificial intelligence. A detailed discussion can be found in

Introduction

Koenderink's book Solid Shape [START_REF] Koenderink | Solid shape[END_REF]. On the mathematical side, the topic was studied in singularity theory by Arnol'd and others [START_REF] Vladimir | Singularities of smooth mappings[END_REF][START_REF] Louis | La famille des projections orthogonales d'une surface et ses singularités[END_REF][START_REF] Oa Platonova | Projections of smooth surfaces[END_REF]. In that setting, the transitions between locally stable views are the non-generic singularities from catastrophe theory. These catastrophes have been classified for "projection-generic" smooth surfaces, and the catalogue consists of the following six visual events. The first three names are due to René Thom [START_REF] Thom | Structural stability and morphogenesis[END_REF]:

(L) Local events: lip, beak-to-beak, and swallowtail.

(M) Multi-local events: tangent crossing, cusp crossing, and triple point.

In the 80's and 90's, visual events became a popular research topic in computer vision [START_REF] Bowyer | Aspect graphs: An introduction and survey of recent results[END_REF][START_REF] Sylvain Petitjean | Computing exact aspect graphs of curved objects: Algebraic surfaces[END_REF][START_REF] Ponce | Computing exact aspect graphs of curved objects: Parametric surfaces[END_REF]. We refer to Chapter 13 in the textbook by Forsyth and Ponce [START_REF] David | Computer Vision -A Modern Approach, Second Edition[END_REF] for a general introduction. The interest in these topics was due to the fact that aspect graphs and visual events can in principle be used as compact representations for all qualitative appearances of objects. These ideas actually never found much practical use, mainly because of the computational complexity of aspect graphs, and the difficulties of dealing with real-life objects. Nevertheless, several algorithms for computing aspect graphs of algebraic surfaces were proposed, with test implementations involving both numerical and symbolic methods. For example, Ponce and Kriegman [START_REF] Ponce | Computing exact aspect graphs of curved objects: Parametric surfaces[END_REF] and Rieger [START_REF] Jh Rieger | Global bifurcation sets and stable projections of nonsingular algebraic surfaces[END_REF] studied the case of orthographic projections of parametric algebraic surfaces; methods for implicit algebraic surfaces were introduced by Petitjean et al. [START_REF] Sylvain Petitjean | Computing exact aspect graphs of curved objects: Algebraic surfaces[END_REF] for orthographic projections, and by Rieger [START_REF] Jh Rieger | Computing view graphs of algebraic surfaces[END_REF] for perspective projections. All the examples shown in these articles are very special low-degree surfaces: we revisit this literature, now 25 years old, and develop it further from the viewpoint of today's applied algebraic geometry.

Main contributions

Our main contributions can be summarized as follows.

• We revisit visual events for curves and surfaces in a projective and algebraic setting. 2 The different visual events are ruled surfaces associated with sets of lines and planes that meet the original curve or surface in exceptional ways. In particular, we can characterize the events in terms of "associated hypersurfaces" for the observed object (the dual surface in (P 3 ) * , or the Chow or Hurwitz three-folds in Gr(1, P 3 )). This viewpoint is more geometric, and arguably simpler, compared to the traditional approach for defining visual events based on singularity theory.

• We explain how to manipulate algebraically ruled and developable surfaces, that can be effectively represented as curves in Gr(1, P 3 ) or (P 3 ) * . We then discuss practical methods for computing the visual events associated with a curve or surface X in P 3 . Particularly when X is a surface, this is a non-trivial matter because the degrees of the ruled surfaces in the output are very high, as seen in Table 5.2. This challenge can be addressed by pre-computing equations for multiple-root loci. For example, when X is a quintic surface, this allows us to compute a representation for its flecnondal surface that has degree 260. We present case studies and examples showing the computation of different visual events.

Conventions.

In this chapter, we write P n = P(C n+1 ), and our model for the object to be viewed is a variety X in P 3 = P(C 4 ). We will however usually assume that our varieties are defined by equations with real coefficients, and we will often be interested in the associated locus X R of real points of X. We will not distinguish between projective points and their vector representatives, and both will be written in regular font.

Preliminaries

We present some background and tools that will be used in our discussion on visual events. In particular, Section 5.2.1 is devoted to ruled surfaces in P 3 and its subclass of developable surfaces. We introduce effective representations of ruled surfaces, and show how to compute with these. As we will see, this is relevant because all visual event surfaces are ruled. Section 5.2.2 then focuses on ruled surfaces arising from "associated hypersurfaces", i.e., as iterated singular loci of Chow and Hurwitz threefolds in Gr(1, P 3 ), and of dual varieties in (P 3 ) * .

Ruled surfaces and developable surfaces

An irreducible surface in P 3 is ruled if it is a union of straight lines (see Figure 5.1, left). These lines are parameterized by some curve C, and they are known as the generators of the surface. A first example are smooth quadratic surfaces in P 3 : over C, these always possess two rulings of lines. Many classical results on ruled surfaces can be found in the book by Edge [START_REF] Leonard | The Theory of Ruled Surfaces[END_REF]. In this section we develop tools for computing and representing ruled surfaces algebraically. This will be useful because all of the visual event surfaces associated with an object are ruled.

A key player in our study is the Grassmannian of lines in space Gr(1, P 3 ). We will write p 12 , p 13 , p 14 , p 23 , p 24 , p 34 for the Plücker coordinates representing lines in P 3 . We refer to Section 2.2 or to Appendix A for an introduction to the Grassmannian Gr(1, P 3 ) and to Plücker coordinates. Throughout the chapter we will also use the fact that any line

          x 1 x 2 x 3 x 4      =      0 0 0 0      . ( 5.2)
We add these four bilinear forms to the ideal I C , and we then saturate with respect to the irrelevant ideal p 12 , p 13 , p 14 , p 23 , p 24 , p 34 of P 5 . 3 The resulting ideal describes the incidence correspondence of points on lines that are in the curve C. Now, by eliminating the variables p ij , we obtain an ideal in R[x 1 , x 2 , x 3 , x 4 ] defined by a single polynomial of degree d, which defines the desired surface. This computation can also be reversed: given a surface S in P 3 , we can compute the Fano scheme of all lines on S, that lives in Gr(1, P 3 ). Its ideal in R[p 12 , p 13 , p 14 , p 23 , p 24 , p 34 ] is obtained by substituting (5.1) into the equation of S, extracting the coefficients of the resulting polynomial in t, and saturating their ideal by p 34 . The Fano scheme is usually empty or consists of points. However, if it is a curve C, then the surface is ruled and S = S C . In summary, we can describe developable surfaces in P 3 using curves in Gr(1, P 3 ) (or P 5 ), and it is possible to convert each representation into the other algebraically.

We now turn to developable surfaces (see Figure 5.1, right). Given an arbitrary subvariety X in P 3 , its dual X ∨ ⊂ (P 3 ) * is the Zariski closure of the set of all planes that are tangent at a smooth point of X (see Appendix B). Typically, the dual S ∨ of a surface of degree at least two will be a surface in (P 3 ) * . However, it can happen that S ∨ is a curve, in which case we say that S is a developable surface. In other words, S is developable if it has a one-dimensional family of tangent planes: this coincides with the informal definition that it can be "rolled" on a plane. It is clear that each developable surface S in P 3 is encoded by its dual curve S ∨ in (P 3 ) * since we can recover the surface by the biduality relation S = (S ∨ ) ∨ . The following result shows that developable surfaces in P 3 are a special subfamily of ruled surfaces. Figure 5.1: A ruled surface (left) and a developable surface (right). Note that the developable surface is the union of lines tangent to a space curve. This curve is the edge of regression.

A developable surface S = S C that is not a cone has three distinct encodings as a curve. First, there is the curve C in the Grassmannian Gr(1, P 3 ). Second, there is the dual curve S ∨ in (P 3 ) * . We saw how to recover S from these encodings. Finally, there is the edge of regression E(S) which lies on the surface S in P 3 . Points in E(S) correspond to planes in (P 3 ) * that intersect the curve S ∨ with multiplicity three ("osculating planes"). The surface S is the tangential surface of E(S), i.e., it is the union of lines that are tangent to E(S) (see [140, page 111]). This also verifies that S is indeed a ruled surface.

The curves E(S) and S ∨ are also related by a biduality relation; namely, S ∨ is the set of planes in P 3 that intersect E(S) with multiplicity three [141, Thm. 5.1]. Moreover, the tangent lines of E(S) and S ∨ are dual to each other. This situation degenerates when the surface S is a cone, which means that its dual S ∨ is a plane curve: in that special When S is developable, the specialization (5.6) can also be obtained from the ideal I(S ∨ ).

Let J be a Jacobian matrix for the ideal I(S ∨ ) in R[y 1 , y 2 , y 3 , y 4 ]. This matrix has four columns. Let J x be the matrix obtained from J by adding one more row, namely the vector (x 1 , x 2 , x 3 , x 4 ) in (5.5). We now add the 3 × 3-minors of J x to the ideal I(S ∨ ), we saturate with respect to the ideal of 2 × 2-minors of J, and then we eliminate the unknowns y 1 , y 2 , y 3 , y 4 . The result is the desired principal ideal generated by (5.6) in R[s, t]. See Example 5.4.4 for an application of this method.

These strategies can be adapted to compute the curve that is obtained as the intersection of a ruled or developable surface S with fixed plane H in P 3 . For event surfaces, this corresponds to restricting the camera movement to a plane, or to assuming that all projections are orthographic (which implies that the viewpoint lies on the plane at infinity). It is sufficient to parameterize the points on H by substituting x i = sa i + tb i + uc i in (5.6).

Associated ruled surfaces

The ruled surfaces of interest to us arise from an arbitrary curve or surface X in P 3 , and represent families of planes and lines that intersect X with prescribed multiplicities. These correspond to the bottom rows of the diagrams in Figures 5.2 and 5.3, that can be interpreted as follows. For a general curve or surface X in P 3 , the rows of these diagrams correspond to codimension in (P [START_REF] Kohn | Coisotropic hypersurfaces in the Grassmannian[END_REF][START_REF] Sturmfels | The Hurwitz form of a projective variety[END_REF]). These codimension one loci are:

• the dual surface X ∨ in (P 3 ) * ,

• the Chow threefold Ch(X) in Gr(1, P 3 ), consisting of lines that meet the curve X,

• the Hurwitz threefold Hur(X) in Gr(1, P 3 ), of lines that are tangent to the surface X.

Every irreducible hypersurface in Gr(1, P 3 ) is defined by one equation in Plücker coordinates, which is unique up to scaling and modulo the Plücker quadric. In the two cases above, this equation is called Chow or Hurwitz form of X. As we will see in Section 5.3, the irreducible components of the visual event surface of X are (iterated) singular loci of the coisotropic hypersurfaces associated to X. The developable components are dual to the singular curves in the dual surface X ∨ ; the non-developable components are parameterized by the singular curves in the singular locus of the Chow or Hurwitz threefold of X.

Let us first consider a general smooth curve X in P 3 . The left diagram in Figure 5.2 depicts the landscape in (P 3 ) * . The dual surface X ∨ consists of planes that meet X with multiplicity 2. The singular locus of X ∨ is the union of two irreducible curves, whose points are osculating planes (m = 3) and bitangent planes (m = 2 + 2). The symbols that denote our loci, like T p (X) and E p (X), will be explained in Section 5.3. The right diagram in Figure 5.2 shows the landscape in the Grassmannian Gr(1, P 3 ). We refer to [103, Theorem 1.1] for precise statements and their proofs, also for the right diagram in Figure 5.3. The singular locus of the Chow threefold Ch(X) is the surface (or congruence, cf. Chapter 2) Sec(X) in Gr(1, P 3 ) of secant lines, i.e., lines that meet X twice. The singular locus of Sec(X) is the curve D (X) of all trisecant lines. The curve T (X) of tangent lines is contained in Sec(X) but it does not belong to the singular locus. In Figure 5.3, we consider various loci associated with a general smooth surface X in P 3 . The dual surface X ∨ is singular along two irreducible curves. The nodal component E p (X) of its singular locus is the set of all bitangent planes, and the cuspidal component P p (X) is the set of all planes that intersect X with multiplicity 3 at a point (see also Figure 5.4). The Hurwitz threefold Hur(X) is singular along two irreducible surfaces. Its nodal component Bit(X) is the congruence of all bitangent lines, and its cuspidal component PT(X) is the congruence of principal tangents, i.e., lines that meet X with multiplicity 3. The latter was denoted Infl(X) in [START_REF] Kohn | Secants, bitangents, and their congruences[END_REF]. These surfaces contain three special curves F (X), C (X) and T (X), indicating lines that meet X with multiplicity 4, or 3+2, or 2+2+2. For instance, 2+2+2 refers to tritangent lines. Sections 5.3.2 and 5.4 are devoted to the ruled surfaces in P 3 that are represented by these curves.

(P 3 ) * m = 1 X ∨ m = 2 T p (X) m = 3 E p (X) m = 2 + 2 Gr(1, P 3 ) m = 0 Ch(X) m = 1 Sec(X) m = 1 + 1 D (X) m = 1 + 1 + 1 T (X) m = 2
For each of the five curves at the bottom of Figure 5.3, there is also an associated curve on X, which consists of the points on X where the special intersection occurs. For example, the curve associated with E p (X) is the locus of points on X that lie on bitangent planes. These are the contact points on a curved object when it rolls on a table. Our favorite terminology for this curve is due to Cayley: he calls it the node-couple curve. For P p (X) and F (X), the special contact occurs at a single point, and we can give a more detailed description. At a general point x, the surface has two principal tangents, that are the tangent lines of the nodal curve obtained by intersecting X with its tangent plane at x. The same lines are the intersection of the tangent plane with the Hessian quadric at x defined by (here we assume that X is defined by a polynomial f in R[x 1 , x 2 , x 3 , x 4 ]) y H f (x) y T = 0, where y = [y 1 , y 2 , y 3 , y 4 ] and

(P 3 ) * m = 1 X ∨ m = 2 P p (X) m = 3 E p (X) m = 2 + 2 Gr(1, P 3 ) m = 1 Hur(X) m = 2 PT(X) m = 3 F (X) m = 4 Bit(X) m = 2 + 2 C (X) m = 3 + 2 T (X) m = 2 + 2 + 2
H f = ∂ 2 f ∂x i ∂x j .
Exceptional situations occur at flecnodal and parabolic points x. At a flecnodal point, one of the two principal tangents has intersection multiplicity 4. At a parabolic point, the Hessian matrix H f (x) drops rank, and the two principal tangents degenerate to a double line. At these points, the intersection of X with its tangent plane has a cusp at x. The locus of all parabolic points is the curve given by the intersection of X with the Hessian surface {det(H f ) = 0}. Over the real numbers, the parabolic curve is the boundary between the elliptic and hyperbolic regions on X, where the two principal tangents are respctively both complex or both real. The curve P p (X) is the set of tangent planes at parabolic points, and the curve F (X) is the set of higher order principal tangents at flecnodal points. The parabolic and flecnodal curves always intersect tangentially, at special points known as godrons (or cusps of the Gauss map). Interestingly, the nodecouple curve also passes through the godrons, and has the same tangent as the parabolic and flecnodal curves [START_REF] Salmon | A treatise on the analytic geometry of three dimensions[END_REF].

Remark 5.2.4. Questions regarding the singular loci of the families of lines and planes described in this section are often delicate to answer formally. This is the topic of [START_REF] Kohn | Secants, bitangents, and their congruences[END_REF]. For example, in the course of examining parabolic surfaces, we discovered a small error Lemma 4.1 (b) in [START_REF] Arrondo | A focus on focal surfaces[END_REF] states that the surface Y 2 is singular at points (x, L) for which x is parabolic. This is incorrect. A general cubic surface X has a degree 12 curve of parabolic points. However, the incidence variety Y 2 is smooth. This is shown by direct computation.

Event Surfaces

In this section we describe the geometry of visual events for curves (Section 5.3.1) and surfaces (Section 5.3.2). In the case of curves we also present strategies for recovering the events computationally. For surfaces this task is more challenging, and will be addressed in Section 5.4.

Definition of events.

It is convenient introduce some precise terminology and notation for the rest of the chapter. As explained in the introduction, our model for the object to be viewed is a smooth variety X of in (complex) projective space P 3 . Taking a picture of X is modeled by a linear projection π : P 3 P 2 with pinhole z (for zentrum). This defines a curve C z (X) in the image plane P 2 . If X is a surface, then C z (X) is the branch locus of π restricted to X: this is the closure of the set of points in the image plane whose corresponding viewing lines are tangent to X at some smooth point. If X is a curve, then C z (X) is simply the closure of the image of X under π. Even if X is smooth in P 3 , the image curve C z (X) has in general many singular points in P 2 . For a surface viewed from a general viewpoint z, the only singularities in the contour are nodes and cusps; for a space curve, the image curve has only nodes (see Appendix B for a discussion on nodes and cusps of plane curves). As the center z changes, the structure of its singularities is locally constant, however at some points a transition occurs, and the singularity structure changes: the visual event surface V(X) is the Zariski closure in P 3 of the set of these transition points. The visual event surface V(X) usually has multiple components, corresponding to the different types of visual events. We will explain the geometry of these irreducible components in the rest of this section.

Views of curves

We begin by studying the visual events for a general curve X in P 3 of degree d and genus g. 4 This automatically implies that X is smooth and irreducible. There are three possible visual events, corresponding to the three Reidemeister moves that are well-known in classical knot theory [START_REF] Richard | Introduction to knot theory[END_REF]. These are shown in Figure 5.5. The three events correspond to three components of the visual event surface V(X) associated with the curve X. These components can be characterized as follows:

1. The tangential surface T (X), also known as the tangent developable, is the union of all tangent lines to X. It represents viewpoints z such that the plane curve C z (X) has a cusp. When z crosses T (X), a node on C z (X) transitions from being real to complex.

2. The edge surface E(X) is the union of all secant lines that are edges. An edge is the line spanned by two points on X whose tangent lines lie in a common plane. This surface represents viewpoints z such that the plane curve C z (X) has a tacnode. When z crosses E(X), a pair of nodes transitions between being real and being complex.

3. The trisecant surface D(X) is the union of all lines that are spanned by triples of collinear points on X (the symbol D stands for drei). This represents viewpoints z such that C z (X) has a triple point. When z crosses D(X), the real curve C z (X) experiences a triangle crossing, but the number of real singularities is unchanged.

This classification helps explain the notation used in Section 5.2. The symbols T (X), E (X) and D (X) denote the curves in the Grassmannian Gr(1, P 3 ) that represent the surfaces T (X), E(X) and D(X). The surfaces T (X) and E(X) are developable, so they can also be represented by their dual curves in (P 3 ) * :

T p (X) := T (X) ∨ and E p (X) := E(X) ∨ .
Here the index "p" stands for planes. The earlier used upper index " " stands for lines. The trisecant surface D(X) is ruled but not developable, so it has no associated curve in (P 3 ) * .

The following classical theorem characterizes the expected degrees of the ruled surfaces T (X), E(X) and D(X). These coincide with the degrees of the curves T (X), E (X) and D (X) inside P 5 .

Theorem 5.3.1. For a general space curve X of degree d and genus g, the degrees of the tangential surface T (X), the edge surface E(X) and the trisecant surface D(X) are as follows:

deg T (X) = 2(d + g -1) deg E(X) = 2(d -3)(d + g -1) deg D(X) = (d-1)(d-2)(d-3) 3 -(d -2)g
Proof. The degree of the tangential surface T (X) is the Riemann-Hurwitz number 2d + 2g -2. This coincides with the degree of the dual surface X ∨ . See [140, page 111] for a geometric derivation and [START_REF] Pal | The geometry of the tangent developable[END_REF] for computational examples. The formula for the degree of the edge surface E(X) appears in [START_REF] Ranestad | On the convex hull of a space curve[END_REF]Theorem 2.1]. The proof given there is based on De Jonquiéres' Formula. The degree of the trisecant surface D(X) is due to Berzolari who first found it in 1895. One can find Berzolari's formula in Bertin's article [START_REF] Bertin | On the singularities of the trisecant surface to a space curve[END_REF] on the geometry of D(X).

Table 5.1 summarizes the conclusion of Theorem 5.3.1 for space curves of degree d at most 6. The genus g ranges from 0 to Castelnuovo's bound (which describes the maximal possible genus among irreducible non-degenerate curves of given degree). Note that, for fixed d and increasing g, the degree of D(X) decreases while that of the others increases. In particular, there is no trisecant surface for twisted cubic curves (d = 3, g = 0) and elliptic quartic curves (d = 4, g = 1) (cf. [START_REF] Bertin | On the singularities of the trisecant surface to a space curve[END_REF]Proposition 1]). We also point out that the edge surface E(X) is of importance in convex geometry because the algebraic boundary of the convex hull of a real affine curve X consists of E(X) and the tritangent planes of X [START_REF] Ranestad | On the convex hull of a space curve[END_REF].

Consider now the projection π z : X ⊂ P 3 P 2 from a center z ∈ P 3 \ X. The image C z (X) of X is equivalent to appropriate planar sections of the associated Chow hypersurface Ch(X) in Gr(1, P 3 ) and of the dual hypersurface X ∨ in (P 3 ) * . More precisely, writing α(z) the set of all lines in P 3 that contain z and z ∨ for the set of all planes in P 3 that contain z, we have the following result. Proposition 5.3.2. The image C z (X) of the curve X is projectively equivalent to the curve α(z) ∩ Ch(X) in the Grassmannian Gr(1, P 3 ). The curve (C z (X)) ∨ ⊂ (P 2 ) * that is dual to the contour is projectively equivalent to the curve z ∨ ∩ X ∨ in the dual projective space (P 3 ) * . These facts follow from basic projective geometry (compare this with Proposition 4.3.4). The two planar curves α(z)∩Ch(X) and z ∨ ∩X ∨ can be viewed as "intrinsic realizations" of the image curve C z (X) (i.e., they are geometric objects that do not depend on choices of coordinates). Moreover, the visual cone associated with X (the union of all lines in the pre-image of C z (X)) is a particular developable surface: the two curves from Proposition 5.3.2 can be seen as representatives for this surface in Gr(1, P 3 ) and (P 3 ) * , as discussed in Section 5.2.1.

Remark 5.3.3. Inside Gr(1, P 3 ) and (P 3 ) * , visual events correspond to planes that intersect Ch(X) and X ∨ in non-generic ways. In particular, let us consider Ch(X). As illustrated in Figure 5.2 (right), the singular locus of Ch(X) is the secant congruence Sec(X) of lines meeting X twice. It is shown in [START_REF] Kohn | Secants, bitangents, and their congruences[END_REF] that this congruence has order ν = 1 2 (d -1)(d -2) -g , which means that a general α-plane in Gr(1, P 3 ) will always meet Sec(X) in exactly ν distinct points (see Chapter 2). These points correspond to the nodes that are generically visible in the image curve C z (X). On the other hand, the number of distinct nodes will be different from ν if only if α(z) intersects Sec(X) at a singular point, or at a smooth point non-transversely. According to Figure 5.2, singular points of Sec(X) are trisecant lines to X. Moreover, one can show that non-transversal of a plane α(x) at smooth points of Sec(X) can occur at two types of points in Gr(1, P 3 ) (or lines in P 3 ): lines that are tangent to X, and lines that are "edges" for X. This gives a geometric interpretation of the visual events for curves.

Computing events for curves. We now address the task of computing three event surfaces T (X), E(X) and D(X) for a given space curve X. In our paper [START_REF] Kohn | Changing Views on Curves and Surfaces[END_REF], we first focused on a few distinguished classes of curves, such as curves that lie on a quadric surface, or rational curves (g = 0) of arbitrary degree, and described some specialized computational techniques that are effective for these particular cases. Here we only present "general purpose" methods. The interested reader is referred to our paper for [START_REF] Kohn | Changing Views on Curves and Surfaces[END_REF] for a more detailed discussion on special curves.

We assume that the curve X is given by its ideal

I = f 1 , f 2 , . . . , f k in R[x 1 , x 2 , x 3 , x 4 ].
Our goal is to compute the defining polynomial F of each ruled event surface S C in P 3 . If the polynomial F is too large, then we compute the ideal of the curve C in Gr(1, P 3 ) or when S C is developable, the ideal of the dual S ∨ C in (P 3 ) * . The edge surface E(X) was already discussed in [START_REF] Ranestad | On the convex hull of a space curve[END_REF][START_REF] Seigal | Real rank two geometry[END_REF]. We therefore focus on the other two ruled surfaces in Theorem 5.3.1. The easier among them is the tangential surface T (X). At any given point p on the curve X, the tangent line is defined by the linear equations

    ∂f 1 ∂x 1 (p) ∂f 1 ∂x 2 (p) ∂f 1 ∂x 3 (p) ∂f 1 ∂x 4 (p) . . . . . . . . . . . . ∂f k ∂x 1 (p) ∂f k ∂x 2 (p) ∂f k ∂x 3 (p) ∂f k ∂x 4 (p)          x 1 x 2 x 3 x 4      =      0 0 0 0      . ( 5.7)
To find the polynomial F defining T (X), we take a variable point p = [y 1 , y 2 , y 3 , y 4 ], and augment the ideal I with the constraints from (5.7). This gives an ideal inside R[x 1 , x 2 , x 3 , x 4 , y 1 , y 2 , y 3 , y 4 ]. From that ideal, we saturate and eliminate the variables y 1 , y 2 , y 3 , y 4 . The resulting ideal is F .

The trisecant surface D(X) will be represented by its curve D (X) in Gr(1, P 3 ). To compute this curve, we parameterize lines in P 3 using (5.1). Suppose for now that our curve is defined by the vanishing of two forms (i.e., it is a complete intersection) X = V (f 1 , f 2 ). We want the univariate polynomials f 1 (z(t)) and f 2 (z(t)) to have three common roots, i.e., their greatest common divisor (GCD) must have degree ≥ 3. This can be expressed using subresultants [START_REF] Abdeljaoued | Minors of Bezout matrices, subresultants and the parameterization of the degree of the polynomial greatest common divisor[END_REF]. The vanishing of all subresultants of order i = 0, . . . , r -1 for two polynomials in t means that their GCD has degree at least r. In our case, we form the ideal given by the subresultant coefficients of f 1 (z(t)) and f 2 (z(t))

the singular loci shown on the left in Figure 5.3. The remaining three surfaces T (X), C(X) and F(X) arise from the curves in the Grassmannian Gr(1, P 3 ) seen on the right of that diagram.

Remark 5.3.7. The curves of lines and planes from Figure 5.3 capture both the local and multi-local features of the surface X. This is an advantage compared to the traditional approach for studying the appearance of surfaces based on differential geometry and singularity theory. In the computer vision literature [START_REF] Bowyer | Aspect graphs: An introduction and survey of recent results[END_REF][START_REF] Sylvain Petitjean | Computing exact aspect graphs of curved objects: Algebraic surfaces[END_REF][START_REF] Ponce | On Image Contours of Projective Shapes[END_REF][START_REF] Ponce | Computing exact aspect graphs of curved objects: Parametric surfaces[END_REF][START_REF] Jh Rieger | Global bifurcation sets and stable projections of nonsingular algebraic surfaces[END_REF], prominent local features of a surface were defined in terms of the euclidean Gauss map and the asymptotic spherical image. These are maps from the surface to the unit sphere S 2 , taking a point on X R to its normal direction, or to the direction of one of its principal tangents. In our algebro-geometric setting, the role of S 2 is played by the dual surface X ∨ ⊂ (P 3 ) * and the principal tangent surface P T (X) ⊂ Gr(1, P 3 ). These surfaces carry much more information than the unit sphere S 2 .

Consider now the projection π z : X ⊂ P 3 P 2 from a center z ∈ P 3 \X. The follwing result, analogous to Proposition 5.3.2, describes "intrinsic realizations" of the contour C z (X). Proposition 5.3.8. The contour C z (X) of our surface X is projectively equivalent to the curve α(z) ∩ Hur(X) in the Grassmannian Gr(1, P 3 ). The curve (C z (X)) ∨ ⊂ (P 2 ) * that is dual to the contour is projectively equivalent to the curve z ∨ ∩ X ∨ in the dual projective space (P 3 ) * .

As noted for curves in Remark 5.3.3, visual events are associated with planes α(z) that intersect Hur(X) and X ∨ non-generic ways. In particular, let u be a point in the image curve C z (X), and let L u = π -1 z (u) be the associated pre-image line in Gr(1, P 3 ). This line is a point of contact between α(z) and Hur(X) inside Gr(1, P 3 ). Assuming that the intersection of α(z) and Hur(X) is transversal at L u , then we have that:

• If L u is a general point of Hur(X), then u is a smooth point of the curve C z (X).

• If L u is a general point of Bit(X) ⊂ Hur(X), then u is a simple node of C z (X).

• If L u is a general point of PT(X) ⊂ Hur(X), then u is a simple cusp of C z (X).

Another interesting fact (shown e.g. in [START_REF] Ponce | On Image Contours of Projective Shapes[END_REF]) is that if L u is a (non-principal) tangent line at a parabolic point, then u is a flex point of C z (X).

Cuspidal and nodal singularities exist for any viewpoint z, since α(z) always intersects the congruences Bit(X) and PT(X) in Gr(1, P 3 ). In fact, it is shown in [START_REF] Kohn | Secants, bitangents, and their congruences[END_REF] that the orders of these congruences are respectively ν

= 1 2 d(d -1)(d -2)(d -3) and κ = d(d -1)(d -2)
, which means that a general image curve will have exactly ν distinct nodes and κ distinct cusps (over C). On the other hand, for non generic planes α(z) this will not be the case. First, α(z) can intersect the singular loci of Bit(X) and PT(X). According to Figure 5.3, these singular loci consist of T , C and F . These sets of lines correspond to special singularities in the image curve C z (X).

Event Surfaces

(T ) If L u is a tritangent line, then u is a triple point. This is a triple point event.

(C) If L u is a principal bitangent, then C z (X) has a smooth branch and a cuspidal branch that meet at u. This is a cusp crossing event.

(F) If L u is a flecnodal line, then u is the limit of two cusps and a node, i.e. an infinitesimal change of the viewpoint produces two cusps and a node. This is a swallowtail event.

Alternatively, α(z) can meet Bit(X) or PT(X) at a smooth point but not transversely.

One can show that this occurs when α(z) contains a line L u that is either a bitangent line on a bitangent plane, or a principal tangent at a parabolic point. This corresponds to the singular loci E p (X) and P p (X) of the dual surface X ∨ . Once again this gives rise to special singularities in C z (X).

(E) If L u is a bitangent line on a bitangent plane, then the associated image point u is a tacnode. It is obtained as the limit of two smooth branches coming together at u. This is a tangent crossing event.

(P) If L u is the principal tangent at a parabolic point p, then, over the real numbers, two behaviors are possible: either u is an isolated node, which corresponds a lip event, or u is a tacnode, obtained as the limit of two cusps, which is a beak-to-beak event. See below for a discussion.

The six visual events are shown in Parabolic events. We now briefly explain how to distinguish two possible local behaviors ("lip" vs. "beak-to-beak") of the contour when the viewpoint z belongs to the parabolic surface P(X). As argued in Section 5.2.1, the parabolic surface P(X) is a developable surface, since it is dual to the curve P p (X) in (P 3 ) * . In particular, all principal tangents at parabolic points are the tangents of the edge of regression curve, denoted by E(P(X)). This curve is different from the parabolic curve X, and in fact we can associate each parabolic point x with another point e x , where the principal tangent from x meets (and in fact is tangent to) E(P(X)). In real projective space, the complement of {x, e x } in that line has two connected components. The distinction between lip and beak-to-beak is made by which of these two components the viewpoint z belongs to. This fact is shown in [START_REF] Oa Platonova | Projections of smooth surfaces[END_REF]Theorem 4.10]. Real-life visual events. We conclude this section with an informal discussion that should provide an intuitive understanding of our five event surfaces. The following are some real life situations where these events can actually be observed.

We first note that cuspidal and nodal singularities of image contours are stable features, which are visible in most surfaces. Nodes occur whenever occlusions create discontinuities in the contour. Cusps only appear for non-convex objects. For instance, they can be observed on the folds of a piece of cloth. From an exceptional viewpoint, it is possible that several of these singularities occur along the same visual ray. This gives rise to a multi-local visual event (cusp crossing, tangent crossing, or triple points). One can spot these events with a napkin or a towel.

The three local events on the left in Figure 5.6 are more complicated. It takes some practice to discover them in the real world. Here are some concrete examples:

• Lip event: If we observe a small hill from a high aereal viewpoint (say, from a hot air balloon), then all points on the ground are visible. The hill does not generate an image contour. However, as we descend closer to the ground, the profile of the hill suddenly becomes visible in the contour. This qualitative change of appearance is a lip event.

• Beak-to-beak. Observe a glass bottle from the bottom, with your eye close to the base. You see a part of the contour generated by the convex region where the sectional diameter of the bottle decreases. Now, tilt the bottle slowly towards its upright position. At some point, you see a complete path from the base to the top of the bottle. Previously your view had been blocked. This is a beak-to-beak event. Contrary to the lip event, the contour does not disappear at the transition point, but it breaks into two pieces.

• Swallowtail. The traditional drawing of a (transparent) torus presents two swallowtails. We see both cuspidal and nodal singularities in the contour [155, Fig. 2].

As we rotate the torus, a visual event occurs, and these singularities disappear.

Computing Visual Events

We now return to the computation of visual event surfaces. Theorem 5.3.5 gives the degrees of the irreducible components of the visual event surface when X is a general surface of degree d in P 3 . Table 5.2 summarizes these degrees for d ≤ 7. One notices that the degrees are now much larger than those for curves in The degrees in Table 5.2 pose a challenge because a homogeneous polynomial in four unknowns of degree δ can have as many as δ+3 3 terms. For instance, if X is a quintic surface then its flecnodal surface F(X) has degree δ = 260, so the expected number of terms is δ+3 3 = 2997411. In this section we address this challenge.

Multiple-root loci

Throughout this section, we make use of multiple root loci for binary forms. These are varieties that identify the coefficients of polynomials with multiple roots. More precisely, we consider the coefficients c 0 , c 1 , . . . c d of

c 0 t d + c 1 t d-1 + c 2 t d-2 + • • • + c d-1 t + c d .
(

For a partition λ = (λ 1 , . . . , λ k ) ∈ N k with k i=1 λ i ≤ d, we write ∆ λ (d) for the ideal in R[c 0 , . . . , c d ] representing polynomials (5.8) that have k complex roots with multiplicities λ 1 , . . . , λ k . The varieties are called multiple root loci in [START_REF] Lee | Duality of multiple root loci[END_REF]. For example, if d ≥ 4, then ∆ (4) (d) is the ideal of coefficients of polynomials of degree d with one quadruple root.

We refer to [START_REF] Lee | Duality of multiple root loci[END_REF] for details on the ideals ∆ λ (d). Some relevant instances are listed in Table 5.3. Its entries are copied from [START_REF] Lee | Duality of multiple root loci[END_REF]Table 1]. For instance, the entry "6 10 , 8 38 " in the last column means that ∆ (3,2) (7) is minimally generated by 10 polynomials of degree six and 38 polynomials of degree eight.

The ideals ∆ λ (d) can be computed explicitly using subresultants [START_REF] Abdeljaoued | Minors of Bezout matrices, subresultants and the parameterization of the degree of the polynomial greatest common divisor[END_REF]. The i-th subresultant S i (h 1 , h 2 ) of two polynomials h 1 (t), h 2 (t) is a polynomial of degree at most i whose coefficients are the determinants of particular minors of the Sylvester matrix of h 1 and h 2 . The vanishing of S i (h 1 , h 2 ) for 0 ≤ i ≤ d -1 means that the greatest common divisor (GCD) of h 1 and h 2 has degree at least d. Moreover, if S d (h 1 , h 2 ) not zero, it is exactly this GCD. If we let h d be the polynomial (5.8) and h d be its derivative with respect to t, then the condition that h d has roots with multiplicity λ = (λ 1 , . . . , λ k ) is equivalent to the fact that the GCD of h d and h d has degree k i=1 (λ i -1) and has roots with multiplicities λ = (λ 1 -1, . . . , λ k -1).

aspect graph [START_REF] Bowyer | Aspect graphs: An introduction and survey of recent results[END_REF]. This graph describes the topology of the complement of the event surface inside real projective space, so its computation requires tools from real algebraic geometry (such as the tracing procedure described in [START_REF] Sylvain Petitjean | Computing exact aspect graphs of curved objects: Algebraic surfaces[END_REF]).

In this chapter we have considered only pinhole projections, however it is possible to define visual events for more general imaging models as well. More precisely, if C is a line congruence in Gr(1, P 3 ) representing a geometric camera (cf. Chapter 2), and if X is the curve or surface that is being observed, then a visual event will occur when C contains a line that is a generator for one of the ruled surfaces described in Section 5.3, that are associated with X. Equivalently, C must intersect certain special curves inside Gr(1, P 3 ). A difficulty here is that we cannot treat the event surfaces as actual "boundaries", since a camera is not identified with a single viewpoint. On the other hand, we can apply different motions to a camera, and ask when the resulting congruence intersects the aforementioned curves in Gr(1, P 3 ). For example, if we apply a translational (1D) motion to the congruence, then by dimensionality arguments there will be a finite number of points where these visual events take place. We expect the nature (and appearance) of the "generic" visual events to be the same as in the traditional pinhole case. However, allowing for a higher dimensional motion space, there will be more complex behaviors for exceptional (higher codimension) camera locations.

Although algebraic curves and surfaces are usually not well suited for modeling real-life objects, we believe that our idealized approach provides useful tools for understanding visual events. In particular, restricting ourselves to an algebraic framework does not affect the overall geometric picture, but avoids some technical difficulties that arise when using the traditional differential geometric viewpoint. More precisely, we can use the fact that in an algebraic setting, and particularly in projective space and over the complex numbers, parameterized geometric systems typically have a unique "general" behavior. This allows us to easily characterize exceptional projections, without having to use notions of stability of germs for differential functions. On a more practical side, describing visual events in terms of the incidences of a variety with lines and planes is useful for capturing simultaneously local and multilocal instabilities. As noted in Remark 5.3.7, previous geometric descriptions of visual events made use of the Euclidean Gauss map and the asymptotic spherical image [START_REF] Forsyth | Computer Vision: A Modern Approach[END_REF]. These maps can be used to understand the three local visual events, i.e., those corresponding to the parabolic surface (lip and beak-to-beak events) and to the flecnodal surface (swallowtail events). On the other hand, in our study we considered analogous objects in (P 3 ) * and Gr(1, P 3 ): the dual surface X ∨ and the congruence of principal tangents PT(X) (which respectively correspond to the two maps mentioned above), as well as the congruence of bitangents Bit(X) (which was not previously represented as a map into S 2 ). We have seen that these sets of planes and lines can be used to characterize all visual events.

Conclusions

This thesis was devoted to mathematical models for describing the geometry of imaging processes. The first three chapters focused on models for cameras: Chapter 1 investigated topics on the multi-view geometry of pinhole cameras; Chapter 2 studied general imaging systems as mappings from points to lines; Chapter 3 extended this last geometric model with an analytical map associating viewing lines to coordinates. The final two chapters discussed, from different perspectives, the relationship between 3D objects and their images. Chapter 4 considered arbitrary sets in P 3 , and the consistency conditions that are naturally satisfied by their projections in a finite set of images. Chapter 5 focused on smooth curves and surfaces, and described the corresponding visual events by studying sets of lines in Gr(1, P 3 ) with special incidence properties with the observed curve or surface.

Possible extensions of our work were discussed in detail in the conclusions of the chapters, but we recall here some general directions that we believe are worth pursuing. For example, it could be useful to develop more concrete models for representing the space of configurations of pinhole cameras (or, equivalently, for parameterizing joint images). In some mathematical papers, the space of camera configurations has been identified with a component the of a "Hilbert scheme" parameterizing varieties with an assigned Hilbert function [START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF][START_REF] Lieblich | Two Hilbert schemes in computer vision[END_REF]. On the other hand, by using so-called "reduced" cameras, one immediately sees that the configuration space of n cameras is intimately related with the space of configurations of n + 4 points in P 3 . We are currently working on spelling out this relationship, using Carlsson-Weinshall duality [START_REF] Carlsson | Duality of reconstruction and positioning from projective views[END_REF] and some classical works on point configurations [START_REF] Arthur B Coble | Point sets and allied Cremona groups[END_REF].

Turning to the non central imaging systems from Chapters 2 and 3, we hope to gain a better understanding of the action of refractions and reflections on line congruences, and use this to develop better models for representing catadioptric cameras. We also find it amusing to build physical cameras associated with general congruences. For example, we have assembled a prototype of a (1,2)-camera: this device consists of a box with two slits, one straight and one circular, that generates an image corresponding to the set of light rays passing through both slits. In the future, we would like to construct more general systems as well, possibly using programmable directional sensors similar to the Conclusions flexible cameras described in [START_REF] Shree K Nayar | Programmable imaging: Towards a flexible camera[END_REF].

Finally, we believe that the study of contact properties between viewing rays and objects, along the lines of our discussion in Chapters 4 and 5, is a topic of great importance for both computer vision and computer graphics. The plenoptic function [START_REF] Edward | The plenoptic function and the elements of early vision[END_REF] and the light field [START_REF] Levoy | Light field rendering[END_REF], for example, are fundamental representations of a 3D scene, that encode photometric information for producing images from all viewpoints. These functions are complex (they have 5D and 4D parameter space), but the representations also have some redundancies, since the maps must satisfy certain conditions in order to describe an actual scene. This is in some sense a "continuous" version of our notion of multiview consistency from Chapter 4. Mathematically, this also seems related to the study of varieties in Gr(1, P 3 ) that are defined by objects in P 3 . These are essentially the "coisotropic" varieties described in [START_REF] Israel M Gelfand | Discriminants, resultants, and multidimensional determinants[END_REF][START_REF] Kohn | Coisotropic hypersurfaces in the Grassmannian[END_REF] and considered in Chapter 5. Developing more concrete models for these varieties, perhaps in terms of discrete differential geometry [START_REF] Bobenko | Discrete differential geometry[END_REF] relative to 3D meshes, could be useful to apply many theoretical ideas from this thesis in practical situations.

Projective Geometry

Note that fixing a projective reference frame on P(V ) is the same as fixing a projective isomorphism between P(V ) and P n . More generally: Theorem A.1.2. If P(V ) and P(W ) are two projective spaces of dimension n, then given two projective reference frames P 1 , . . . , P n+2 and Q 1 , . . . , Q n+2 on P(V ) and P(W ) respectively, there exists a unique projective isomorphism ϕ : P(V ) → P(W ) such that ϕ(P i ) = Q i for i = 1, . . . , n + 2.

Analytic representation of projective maps.

Let us consider two projective spaces P(V 1 ) and P(V 2 ) of dimension n and m, equipped with fixed projective reference frames R 1 , R 2 respectively. We also let B 1 , B 2 be two normalized vector bases on V 1 , V 2 associated with the reference frames R 1 , R 2 .

If f : P(V 1 ) \ K → P(V 2 ) is a projective map induced by a linear map ϕ : V 1 → V 2 , with K = P(Ker ϕ), then we associate with f an (m + 1) × (n + 1) matrix A describing ϕ with respect to the bases B 1 , B 2 . The matrix A is determined by f up to a non-zero multiplicative coefficient. If a point P in P(V 1 )\K has homogeneous coordinates P = [v] with v in K n+1 , then f (P ) will have homogeneous coordinates [Av]. In the case V 1 = V 2 , invertible square matrices represent changes of projective coordinates.

Affine Geometry. For many concrete applications of projective geometry, including computer vision, it is often convenient to view projective geometry as an extension of affine geometry. Inside P n , we fix H ∞ = {x n+1 = 0} as the "hyperplane at infinity". The complement U = P n \ H ∞ is the set of "finite points". There is a natural bijection j : K n → U given by j : (u 1 , . . . , u n ) → [u 1 , . . . , u n , 1] j -1 : [x 1 , . . . , x n+1 ] → x 1 x n+1 , . . . , x n x n+1 .

The map j induces an immersion of K n in P n . We can thus interpret P n as an extension of K n with the hyperplane at infinity H ∞ .

The projective transformations f of P n such that f (H ∞ ) = H ∞ correspond to matrices of the form.

A = B t 0 1 .
With the identification j introduced above, these describe classical affine affine maps.

More generally, an affine structure can be defined for any set U H = P n \ H where H is an arbitrary hyperplane of P n . The set U H is called an affine chart.

A.2.1 Grassmannians and Plücker coordinates

Let P n = P(K n+1 ) denote the n dimensional projective space over a field K inside C. For any 0 ≤ k ≤ n, the Grassmann variety (or Grassmannian) Gr(k, P n ) is the set of all projective subspaces of P n that have dimension k. It is possible to equip any Grassmannian with a set of homogeneous coordinates as follows.

Let H = P(W ) be a projective subspace of P n of dimension k, spanned by the independent points P 1 = [w 1 ], . . . , P k+1 = [w k+1 ]. If W is the (n + 1) × (k + 1) matrix W whose columns are the vectors w 1 , . . . , w k+1 , we associate with the subspace H the point [p H ] in P ( n+1 k+1 )-1 whose homogeneous coordinates are the maximal minors of M, ordered lexicographically. It is customary to write p i 1 ...,i k+1 (with 0 ≤ i 1 < . . . < i k+1 ≤ n) for the coordinate corresponding to the determinant of the (k +1)×(k +1) matrix defined by the columns i 1 , . . . , i k+1 of W. It easy to see that the point [p H ] in P ( n+1 k+1 )-1 is independent of the choice of P 1 , . . . , P k+1 in H. The association Gr(k, P n ) → P ( n+1 k+1 )-1 is known as the Plücker embedding, and the homogeneous coordinates in P ( n+1 k+1 )-1 of a subspace H are known as its Plücker coordinates.

It is also possible to represent H using dual Plücker coordinates. If H is the intersection of n -k hyperplanes H 1 , . . . , H n-k represented by points [u 1 ], . . . , [u n-k ] in (P n ) * , we consider the (n -k) × (n + 1) matrix U whose rows are u 1 , . . . , u k+1 . We then associate with H the point [q H ] in P ( n+1 n-k )-1 = P ( n+1 k+1 )-1 whose coordinates are the maximal minors of U, ordered lexicographically. Note that this point also represents the primal Plücker coordinates of the dual space H ∨ of H inside Gr(n -k -1, P n ). For dual Plücker coordinates it is customary to write q j 1 ...,j n-k (with 0 ≤ j 1 < . . . < j n-k ≤ n) for the coordinate corresponding to the determinant of the matrix defined by the columns j 1 , . . . , j n-k of U. Primal and dual plucker coordinates are related by q j 1 ...,j n-k = (-1) σ(i 1 ,...,i k+1 ) p i 1 ...,i k+1 , (A.1)

where j 1 , . . . , j n-k are complementary indices of i 1 , . . . , i k+1 (ordered appropriately), and σ(i 1 , . . . , i k+1 ) is the sign of the permutation (i 1 , . . . , i k+1 , j 1 , . . . , j n-k ).

Finally, we mention that it is sometimes convenient to allow for indices of (primal or dual) Plücker to be any sequence of distinct numbers i 1 , . . . , i k+1 or j 1 , . . . , j n-k between 1 and n. In this case, the transposition of two indices is simply a change of sign of p i 1 ,...,i k+1 or q j 1 ,...,j n-k .

Plücker relations. The image of Plücker embedding of Gr(k, P n ) is a subvariety of P ( n+1 k+1 )-1 . Indeed, Plücker coordinates always satisfy certain quadratic relations, known as Plücker relations. These are completely characterized by the following result (see for example [START_REF] Sturmfels | Algorithms in invariant theory[END_REF] for a proof).

[p 12 , p 13 , p 14 , p 23 , p 24 , p 34 ] and its dual as the entries of two skew-symmetric 4×4-matrices: Two lines [p] = [p 12 , p 13 , p 14 , p 23 , p 24 , p 34 ] and [q] = [q 12 , q 13 , q 14 , q 23 , q 24 , q 34 ] are concurrent at a point in P 3 if and only if p * • q = q * • p = p 12 q 34 -p 13 q 24 + p 14 q 23 + p 23 q 14 -p 24 q 13 + p 34 q 12 = 0.

(A.5)

If P, P * , Q, Q * are the matrices associated with [p] and [q] as above, then (A.5) is equivalent to trace(PQ * ) = trace(P * Q) = 0 (A.6)

Note that the concurrence condition is bilinear in Plücker coordinates. In particular, all lines that intersect a fixed line [p] form a threefold in P 5 , obtained by intersecting Gr(1, P 3 ) with a hyperplane.

If x = [x] is a point and = [p] is a line in P 3 , then the join x ∨ is the plane with dual homogeneous coordinates [u] given by u = Px. This plane is not defined if and only if x belongs to , in which case Px = 0. (A.7)

For a fixed point x = [x], this yields three independent linear equations in the entries of P. These equations define a plane in P 5 that is actually completely contained in the Grassmannian Gr(1, P 3 ). This is known as the α-plane of x.

Similarly, if u = [u] is a dual point representing a plane in P 3 and = [p] is a line, then the meet u ∧ is the point [x] with x = P * u. This point is not defined if and only if u contains , that is if

P * u = 0. (A.8)
For a fixed u = [u], this defines a plane in P 5 contained in Gr(1, P 3 ). This is known as the β-plane of u.

Remark A.2.3. The families of α and β-planes form two disjoint rulings on the Plücker quadric. Two different planes in the same family (α or β) always intersect in exactly one point in Gr(1, P 3 ): this reflects the fact that in P 3 there is exactly one line through two generic points, and exactly one line contained in two generic planes. On the other hand, the α-plane of a point x and the β-plane of a plane u do not meet unless x lies on u. In other words, there is no line contained in a given plane and passing through a given point, unless the point lies on the plane. When X = V (S), we say that X is the variety defined by S. ii) The operators V and I reverse containments: if S ⊂ S in C[x 1 , . . . , x n+1 ] then V (S) ⊃ V (S ) in P n , and if X ⊂ X in P n then I(X) ⊃ I(X ) in C[x 1 , . . . , x n+1 ].

iii) If X is a variety, then V (I(X)) = X.

iv) If I is a homogeneous ideal, then I(V (I)) ⊃ I.

The last point can be mode precise. We define the radical √ I of an ideal I as

√ I = {F | ∃ m ≥ 1 s.t. F m ∈ I}.
The radical √ I is an ideal that contains I. We say that I is a radical ideal if I = √ I. The following important result, known as the (projective) Nullstellensatz, fully explains Irreducible Components the correspondence of ideals and projective sets, showing that varieties are uniquely associated with radical ideals in C[x 1 , . . . , x n+1 ]. The proof can be found for example in [START_REF] Arrondo | Introduction to projective varieties[END_REF]. The ideal M = x 1 , . . . , x n+1 is called the "irrelevant ideal", and it plays a special role because it is the largest polynomial set such that V (M) = ∅ (any projective point has at least one non-zero coordinate). For this ideal we have I(V (M)) = C[x 1 , . . . , x n+1 ], even though M is radical. Excluding the irrelevant ideal, there is a one-to-one correspondence between projective varieties in P n and homogeneous radical ideals in C[x 1 , . . . , x n+1 ]:

projective varieties homogeneous radical ideals

X I -→ I(X) V (I) V ←- I
Finally, we note that varieties can be viewed as closed sets of a a topology on P n . This is known as the Zariski topology. In the context of varieties, we always consider this topology. For example, the closure of any set Y ⊂ P n is the smallest variety containing Y .

B.2 Irreducible Components

A projective variety X in P n is said to be irreducible if it is not the union X = Z 1 ∪ Z 2 of two proper algebraic subsets Z 1 X and Z 2 X. It is clear that inside an irreducible variety X, (Zariski) open sets U ⊂ X are always dense (i.e., their closure is the whole set U = X).

If X is an irreducible projective variety, the corresponding ideal I(X) is a prime ideal. Algebraically, an ideal I is prime if and only if it satisfies

F G ∈ I ⇒ F ∈ I or G ∈ I.
If an ideal I is prime it is always radical (I = √ I).

A projective variety X in P n can always be decomposed as the union of a finite number of irreducible components

X = Z 1 ∪ . . . ∪ Z k , (B.2)
and moreover this decomposition is unique up to order. The decomposition of X has an associated decomposition of ideals (notice the union now becomes an intersection):

I(X) = I(Z 1 ) ∩ . . . ∩ I(Z k ). (B.3)

Projective Varieties

In this decomposition I(X) is radical, and I(Z 1 ), . . . , I(Z k ) are prime. We note that a non-radical ideal I cannot be expressed as an intersection of primes, but it admits a more general decomposition based on primary ideals, which is not completely unique. In this thesis we mostly deal with radical ideals associated with varieties, and that can be decomposed into primes. However, we briefly recall for completeness the more general notion of primary decomposition.

An ideal I is said to primary if F G ∈ I and G / ∈ I implies that there exists m ≥ 1 such that F m ∈ I. It is easy to see that if I is primary then √ I is prime. The following result now holds.

Theorem B.2.1. Any ideal I can be decomposed as

I = I 1 ∩ . . . ∩ I r
where each I j is primary, ∩ j =k I j ⊂ I k holds, and the radicals √ I 1 , . . . , √ I r are distinct prime ideals.

Moreover, the prime ideals √ I 1 , . . . , √ I r are uniquely determined by I, and any primary ideal I j contained a minimal prime (with respect to the inclusion) is also uniquely determined. However primary ideals associated with non-minimal prime ideals are in general not unique.

Finally, we define here the operation of ideal saturation, which is used many times in the computations described in this thesis. The saturation of an ideal I with respect to another ideal J is the ideal I : J ∞ defined as

I : J ∞ = {F | ∀G ∈ J ∃m ≥ 0 s.t. F G m ∈ I}.
This ideal has the property that

V (I : J ∞ ) = V (I) \ V (J),
where the bar denotes the (Zariski) closure (see for example [START_REF] David A Cox | Ideals, Varieties, and Algorithms: An Introduction to Computational Algebraic Geometry and Commutative Algebra[END_REF]). In other words, saturating I with respect to J excludes from V (I) all the irreducible components of V (I) that are contained in V (J). A particularly important case is when J = M = x 1 , . . . , x n+1 is the irrelevant ideal.

B.3 Dimension and Degree

Let X be a variety in P n . The dimension of X can be defined as the maximum integer r = dim X such that any linear subspace of P n of codimension at most r meets X. Another equivalent definition is the maximum length r of a strictly increasing chain Z 0 Z 1 . . . Z r of irreducible varieties contained in X. Many other characterizations of the dimension exist. When the dimension of X is n -1, X is called a hypersurface.

Tangent Spaces

Two projective varieties X in P n and Y in P m are said to be isomorphic when there exist morphisms F : X → Y and G : Y → X that are inverses of each others. If X and Y both live in P n , then X and Y are isomorphic if they are in particular projectively equivalent, i.e., if there exists a projective transformation T : P n → P n such that T (X) = Y . The converse however is not true, and two isomorphic varieties need not be projectively equivalent: for example, a line and a conic in P 2 are isomorphic (see Example B.5.1).

Two projective varieties X in P n and Y in P m are said to be birational or birationally equivalent when there exist dominant rational maps F : X Y and G : Y X that are inverses of each other. This is clearly a weaker condition than being isomorphic (or projectively equivalent). A variety X that is birationally equivalent to P n for some n is said to be rational.

B.6 Tangent Spaces

Let X be a variety in P n with I(X) = F 1 , . . . , F r . If p is a point on X, the embedded tangent space of X at p is the linear space T p (X) ⊂ P n defined by the equations ∂F i ∂x 1

(p)x 1 + . . . ∂F i ∂x n (p)x n = 0, i = 1, . . . , r.

Geometrically, T p (X) ⊂ P n coincides with the union of all lines meeting X at p with multiplicity at least two.

If we define local dimension of X at p as the maximum dimension of a component of X through p, then one can show that T p (X) always has dimension at least equal to this local dimension. When the dimension T p (X) coincides with the local dimension, we say that p is smooth, otherwise it is singular. More concretely, if the local dimension of X at p is r (or in particular if X is irreducible of dimension r), then p is singular if and only if the Jacobian matrix has rank strictly smaller than n -r. We say that X is smooth if all points p on X are smooth. Two varieties X and Y in P n intersect transversely at a point p if T p (X) and T p (Y ) span P n . We mention in this context the following important result.

J(p) =   
Theorem B.6.1 (Bertini's Theorem). If X is a variety in P n , then there exists an open subset U in (P n ) * such that for any hyperplane H in U the set X ∩ H is smooth at points where X is smooth. In particular, if X is smooth, then its intersection with a general hyperplane is smooth.

B.7 Multiple Points on Hypersurfaces

We now take a closer look at singular points of hypersurfaces in P n . This is the simplest case to describe, since hypersurfaces in projective space are always defined by a single polynomial.

We let X = V (F ) be a hypersurface in P n , where F is homogeneous polynomial in C[x 1 , . . . , x n+1 ] of degree d. We also fix a point p on X, and another point q not necessarily on X (to simplify notation, we use p, q for the corresponding coordinates vectors in C n+1 ). We now write the Taylor expansion of F at p: F (p + tq) = t∆ q F (p) + (t 2 /2)∆ 2 q F (p) + . . .

+ (t d /d!)∆ d q F (p),
where

∆ k q F (p) = i 1 ,...,i k q i 1 . . . q i k ∂ k F ∂x i 1 . . . ∂x i k (p).
The intersection multiplicity of the line L = p ∨ q with X at p is m if ∆ q F (p) = . . . = ∆ m-1 q F (p) = 0 and ∆ m q F (p) = 0.

We say that p is an m-fold point for X (or has multiplicity m for X) if the intersection multiplicity with X is at least m for all lines through p. This is equivalent to the fact that all (m -1)-derivatives of F vanish at P (this automatically implies that all lower order derivatives vanish as well). Thus, 1-fold points are smooth points, and m-fold points for m ≥ 2 are singular.

If p is a point of multiplicity m on X, we define the tangent cone C p (X) ⊂ P n at p as the hypersurface of degree m defined by ∆ m x F (p) = i 1 ,...,im

x i 1 . . . x im ∂ m F ∂x i 1 . . . ∂x im (p) = 0.
This represents the union of all lines that intersect X at p with multiplicity higher than m. Thus, the tangent cone generalizes the tangent space when p is a singular point.

Within the tangent cone C p (X), we can consider special lines that intersect X with ever higher multiplicity. For this, we write C h p (X) for the union of all lines that intersect X at p with multiplicity at least m + h. This set is defined by In general, C h p (X) contains a line only for h ≤ n -1. For h = n -1, we have that C n-1 p (X) contains in general a finite number of lines that have intersection multiplicity exactly m + n -1 at p: these lines are known as the principal tangents of X at p. In Dual Varieties particular, if p is a smooth point X, we expect to find (n -1)! principal tangents that have intersection multiplicity n with X.

For example, let us consider a smooth surface X in P 3 . For any point p on X, we have that C p (X) is simply the embedded tangent plane T p (X) at p, while the set C 2 p (X) generally contains two distinct principal tangents that have intersection multiplicity 3 with X. From the perspective of real geometry, one distinguishes points p where these two tangent are both real (hyperbolic points), or complex (elliptic points). We also note that exceptional situations occur at flecnodal and parabolic points: at a flecnodal point, one of the two principal tangents has intersection multiplicity at least 4; at a parabolic point, ∆ 2

x (p) is a degenerate quadric, and the two principal tangents coincide. See Our analysis is also useful for describing singularities of plane curves X in P 2 . We first point out that if p is a smooth point on X, then there is only one principal tangent, which coincides with the tangent line T p (X), and usually has intersection multiplicity 2 with X. There are also special flex points that are smooth points p where the unique tangent line has intersection multiplicity 3 with X. On the other hand, if p is a 2-fold singular point on X (so that only the first order derivatives of F vanish at p) then the tangent cone C p (X) contains a pair of principal tangents through p that have multiplicity 3 with X. When these two lines are distinct, we say that p is a nodal singularity or simply a node; when the two lines coincide (which means that ∆ 2

x (p) is the square of a linear form), we say that p is a cuspidal singularity or a cusp. See

B.8 Dual Varieties

Let X be an irreducible projective variety in P n . We say that a hyperplane H in P n is tangent to X if there exists a smooth point p on X such that H contains the tangent space T p (X). The closure X ∨ in (P n ) * of the set of all tangent hyperplanes to X is an Computations with Sage culus, number theory, cryptography, group theory, combinatorics, and many more. Both softwares are free and open-source. Sage is actually designed as a front-end interface to other existing open-source packages, and for algebraic computations it is mostly based on the system Singular [START_REF] Decker | Singular 4-1-1 -A computer algebra system for polynomial computations[END_REF]. In terms of functionality, Macaulay2 has more advanced tools that are intended for research in algebraic geometry. For the purposes of this thesis, however, the functions available in Sage are almost always sufficient (one exception being the Macaulay2 command multidegree, which is not available in Sage, and is used several times in the thesis). Sage is based on the Python programming language, and can be used in a Jupyter interactive notebook. It is also possible to call functions in Macaulay2 from within Sage. For these reasons, we describe here only some basic examples of commands that can be used for computations in Sage.

To define a polynomial ring over Q (which is normal in symbolic computation) one can use the command This is a conic in P 2 . Note that the pinhole of the projection [0, 0, 0, 1] belonged to the original curve C, which explains is why the degree dropped.

In general, we can always compute projections for the camera [x 1 , x 2 , x 3 , x 4 ] → [x 1 , x 2 , x 3 ] by simply eliminating the variable x4. However, projecting with arbitrary cameras requires a different approach. A possible strategy is illustrated by following code (as in normal Python syntax, hash characters indicate comments in the code).

Résumé

Cette thèse étudie les modèles mathématiques destinés à décrire la géométrie des processus d'imagerie en vision par ordinateur. Notre approche est enracinée dans le langage de la géométrie projective, qui fournit le cadre le plus général pour l'étude des propriétés des lignes et des incidences qui sont au coeur de la vision géométrique. Nous appliquons également des outils de la géométrie algébrique, car la plupart des objets que nous rencontrons sont décrits par des équations polynomiales. Par exemple, la géométrie de n caméras peut être encodée dans une variété algébrique en (P 2 ) n formée par les correspondances de points. La Grassmannienne des lignes Gr(1, P 3 ) joue également un rôle central dans notre étude. Les surfaces en Gr(1, P 3 ) (ou "congruences") peuvent par example être utilisées pour représenter des caméras abstraites, qui associent des points à des lignes. Nous étudions aussi la relation entre les formes 3D et leurs images.

En particulier, pour les ensembles arbitraires se projetant sur des silhouettes, l'image est déterminée par l'ensemble des lignes qui rencontrent l'objet observé ; pour les surfaces lisses, le "contour visuel" est déterminé par les lignes qui sont tangentes à la surface. Cette perspective est appliquée dans l'étude des "coques visuelles" et des "événements visuels".

Mots Clés

Modèles géométriques, géométrie projective, caméras, silhouettes, contours visuels.

Abstract

This thesis studies mathematical models for describing the geometry of imaging processes in computer vision. Our approach is rooted in the language of projective geometry, which provides the most general setting for studying properties of lines and incidences that are at the heart of geometric vision. We also apply some tools from algebraic geometry, since many of the objects that we encounter are described by polynomial equations. For example, the multiview geometry of n pinhole cameras (or in fact any type of cameras) can be encoded in the "joint image", that is an algebraic variety in (P 2 ) n formed by all point correspondences. The Grassmannian of lines Gr(1, P 3 ) also plays a central role in our study. In particular, surfaces in the Grassmannian (or "line congruences") can be used to represent abstract cameras, that are mappings from points to viewing lines. In addition to modeling cameras, we investigate the relationship between 3D shapes and their images. For arbitrary sets projecting onto opaque silhouettes, the image is determined by the set of viewing lines that meet the observed object; for smooth surfaces, the "image contour" is determined by the set viewing lines that are tangent to the surface. This perspective is applied to the study of "visual hulls" and "visual events".

Figure 1 . 1 :

 11 Figure 1.1: The geometry of the camera obscura.

  yields a camera defined by [kMR | Mt + b]. This projection matrix can be rescaled as [MR | Mt + b], where t is such that Mt + b = 1/k(Mt + b) (recall that M has full rank). In particular, [M|b]T is equivalent to [M|b] T, where T is a euclidean transformation:

Proposition 1 . 2 . 3 .

 123 If the r-dimensional linear space M in P m has Plücker coordinates [w M ] in P ( m+1 r+1 )-1 , then its image U = J(M ) in P n for the linear embedding J = [B] has Plücker coordinates [v U ] in P ( n+1 r+1 )-1 , where

Theorem 1 . 3 . 16 .

 1316 The map J from Proposition 1.3.15 is injective. In particular, two families of pinhole cameras have the same joint image if and only if they are in the same configuration.

Example 1 . 3 . 18 .

 1318 Consider three cameras P 1 , P 2 , P 3 with non-collinear pinholes. It is easy to verify that the three epipolar constraints b 12 , b 13 , b 23 do not cut out the joint image M 3 . However, they define a set W whose irreducible components are given by

Figure 1 . 3 :

 13 Figure 1.3: Graphs representing epipolar conditions between n = 8 cameras: a "strong" characterization based on 3n -6 = 18 constraints (left), and a "weak" one based on 2n -3 = 13 constraints (right).

9 Figure 1 . 4 :

 914 Figure 1.4: Examples of minimal solvable viewing graphs for n ≤ 9 views (see Section 1.4.5).

Figure 1 . 5 :

 15 Figure 1.5: A viewing graph of this form (where G 1 , G 2 , G 3 , G 4 represent arbitrary subgraphs) is not solvable

Figure 1 . 8 :

 18 Figure 1.8: The viewing graph from Example 1.4.17.

Figure 1 . 9 :

 19 Figure 1.9: The viewing graph from Example 1.4.18.

Figure 1 . 10 :

 110 Figure 1.10: The viewing graph from Example 1.4.19.

Figure 1 . 11 :

 111 Figure 1.11: Geometric explanation for Proposition 1.5.7: the points e and e are epipoles; for any choice of u ∈ P 2 in the second image, all lines through e and e will give rise to a point-line-line correspondence with u.

  β determines X, but will in general contain it strictly a distinguished irreducible component. See[START_REF] Osserman | Multigraded Cayley-Chow forms[END_REF] Proposition 3.6].

7

 7

Figure 1 . 12 :Figure 1 . 13 :Figure 1 . 14 :

 112113114 Figure 1.12: Examples of synthetic data: non-collinear pinholes (left) and collinear pinholes (right). The camera parameters defined random rotations and translations for camera motions, and produced feasible image sizes (around 500 pixels per dimension).

Figure 1 . 15 :

 115 Figure 1.15: Estimated epipolar and trinocular lines. Note that the two families of epipolar lines associated with an image typically contain (near) degenerate pairs (such as the pair shown in the first image): these can be disambiguated using trilinearities.

Figure 1 . 16 :

 116 Figure 1.16: Collinear pinholes: Quantitative results using synthetic datasets (20 points) with different amounts of Gaussian noise, in terms of the geometric distance error. The results are averages of 50 configurations of points and cameras.

Figure 1 . 18 :

 118 Figure 1.18: Estimated epipolar and trinocular lines. Note that the pairs of epipolar lines essentially coincide, while trinocular lines are always transversal.

  and assuming the condition in Remark 1.7.1, the leading terms of the fundamental multilinearities are given by B = C n \ A, where C n and A are defined in (1.40) and (1.41).

Figure 1 . 19 :Example 1 . 7 . 8 .

 119178 Figure 1.19: The viewing graph from Example 1.7.7.

Figure 1 . 20 :

 120 Figure 1.20: The viewing graph from Example 1.7.8.

Figure 2 . 1 :

 21 Figure 2.1: The geometry of a pushbroom camera.

Theorem 2 . 3 . 3 .

 233 Let C be a (1, β)-congruence with focal locus F(C). Then one of the following four situations is the case (see Figure 2.2): 1. F(C) is a point c, and C is the α-plane of lines through c. Here β = 0.

Figure 2 . 2 :

 22 Figure 2.2: The four types of rational cameras associated with congruences of order one.

Type 2 :

 2 F(C) is a twisted cubic. After a change of coordinates, the twisted cubic in P 3 is the image of the map [s, t] → [s 3 , s 2 t, st 2 , t 3 ]. The corresponding rational camera is

2 4 = 4 n 3 + 4 n 4

 44 quadrics trace(P i P * j ), and it has 12 n 3 cubics, namely those having the 12 leading terms in (2.20), for any three lines. Finally, there are 4 n+1 quartics in G n . Their leading monomials are the quartics in (2.20), for any three lines, and the quartics in (2.21), for any four of the n lines.

Figure 2 . 3 :

 23 Figure 2.3: Three two-slit cameras C 1 , C 2 , C 3 with intersecting focal loci. The triple of concurrent lines belongs to C 1 ×C 2 ×C 3 but not to the multi-image variety M (C 1 , C 2 , C 3 ). See Example 2.4.4 for details.

Proposition 2 . 4 . 7 .

 247 Let C 1 and C 2 be general congruences of bidegree (1, β 1 ) and (1, β 2 ).

Figure 2 . 4 :

 24 Figure 2.4: Panoramic cameras: non-central (left) and stereo (right)

Lemma 2 . 5 . 1 .

 251 Any two such congruences are equivalent up to projective transformations of P 3 .

Figure 2 . 5 :

 25 Figure 2.5: A general algebraic surface S maps a congruence C (red lines) to a "specular congruence" C S (blue lines).

3

 3 are dual Plücker matrices for [y 2 ∨y 3 ], [y 3 ∨y 1 ], [y 1 ∨y 2 ] respectively[START_REF] Feldman | On the epipolar geometry of the crossed-slits projection[END_REF][START_REF] Zomet | Mosaicing new views: The crossed-slits projection[END_REF].

Figure 3 . 2 :

 32 Figure 3.2: Left: A pinhole c induces a homography between any two retinal planes not containing c. Right: Two skew lines l 1 , l 2 induce a homography between planes intersecting at a transversal δ to l 1 , l 2 .

Example 3 . 3 . 4 .

 334 The two-slit projection from Example 3.3.1 is of the form (3.17) with

. 22 )Figure 3 . 3 :

 2233 Figure 3.3: Physical interpretation of the entries the calibration matrices for parallel two-slit cameras: the parameters f u , f v , u 0 , v 0 describe the change of retinal plane coordinates, with respect to some camera in the euclidean orbit of (3.21).

. 27 )Figure 3 . 4 :

 2734 Figure 3.4: Simulated two-slit images of an artificial scene. The figures on the left show the imaging device, consisting of a retinal plane and two slits forming different angles (top: θ = π/2; bottom: θ = 5/8π). The figures on the right show the resulting images.

11 =

 11 -f 1222 ; c 22 = -f 2122 ; c 33 = -f 2212 ; c 44 = -f 2221 ; c 21 = (c 11 c 22 -f 1122 )/c 12 ; c 31 = (c 11 c 33 -f 1212 )/c 13 ; c 41 = (c 11 c 44 -f 1221 )/c 14 .

3 .

 3 The remaining six entries of C are pairwise constrained by six elements of F. For example, using the minors f 2112 , f 1112 (corresponding to rows/columns 2, 3 and 1, 2, 3 of C) we deduce that c 32 must satisfy ac2 32+bc 32 + c = 0 where a = c 13 c 21 , b = f 1112 + c 11 f 2112 -c 13 c 31 c 22 -c 12 c 21 c 33 , c = c 12 c 31 c 22 c 33 -c 12 c 31 f 2112 , and that c 23 = (c 22 c 33 -f 2112 )/c 32 . Similar relations hold for the pairs c 24 , c 42 and c 34 , c 43 .

  a projective configuration of two-slit cameras. Using (3.43), we can recover an estimate for Ω * Q = QΩ * Q T by solving an over-constrained linear system (with 40 equations). From this, we compute a matrix Q such that Q Ω * Q T Ω * Q . For our example, the original data was 43 0.88 -0.93 1.52 -0.38 -0.21 1.83 -0.55 0.83 -0.95 -0.63 0.93

Figure 4 . 1 :

 41 Figure 4.1: Geometrically consistent silhouettes are feasible projections of a 3D object.

Definition 4 . 2 . 2 .

 422 If T 1 , . . . , T n are consistent for the cameras P 1 , . . . , P n , then H = i C i is the finite visual hull associated with T 1 , . . . , T n .

2

 2

Figure 4 . 2 :

 42 Figure 4.2: The silhouettes from Figure 4.1 are consistent (relative to three orthogonal orthographic projections), since each silhouette is contained in the reprojection (in gray) of the intersected visual cones associated with the other views. See Proposition 4.2.3.

Figure 4 . 3 :

 43 Figure 4.3: Parwise consistency: the sets of epipolar lines intersecting each silhouette must be in epipolar correspondence. In particular, the extremal lines lines A 1 , A 2 and B 1 , B 2 must correspond.

  holds, and P 1 (C 2 ∩ C 3 ) intersects all edges of R since C 2 , C 3 are themselves pairwise consistent (an extremal epipolar line in P 1 (C 2 ) is the projection of a line in C 2 which must intersect C 2 ∩C 3 ). The claim follows from continuity arguments: for example, by considering paths in T 1 and P 1 (C 2 ∩ C 3 ) connecting different pairs of opposite edges.

Figure 4 . 4 :

 44 Figure 4.4: The visual hull generated by two "hook-shaped" silhouettes always presents self-occlusions, whether the internal epipolar tangents are unmatched (top) or matched (bottom). In these examples we considered orthogonal orthographic projections, but the behavior is completely general.

Figure 4 . 5 :

 45 Figure 4.5: Proof of Proposition 4.2.6. The set T 1 (dark gray) and P 1 (C 2 ∩ C 3 ) are both "inscribed" in the quadrilateral P 1 (C 2 ) ∩ P 1 (C 3 ) (light gray), and must thus intersect.

Figure 4 . 7 :

 47 Figure 4.7: Proof of Proposition 4.2.7: the visual ray corresponding to p meets the other visual cones in intervals that intersect pairwise, and thus must all intersect.

Figure 4 . 8 :

 48 Figure 4.8: If we modify the second silhouette from Example 4.1 so that the new silhouette is still contained in the reprojection of the visual hull associated to the other views, consistency is preserved. Note that the reprojection onto the third view is not the same as in the previous case, however condition (4.1) remains satisfied since the new reprojected set is larger.

Figure 4 . 9 :

 49 Figure 4.9: Tangential points are the intersections of a silhouette T i with the boundary of P i j =i C j . Left: the silhouette of a torus and the reprojection of the visual hull generated by three different views. The six tangential points are epipolar tangencies (colors indicate projections of cone surfaces and the associated epipolar tangency points).Right: a tangential point may also be the projection of a "tangential triple point": in this case, the associated viewing ray is not tangent to any visual cone.

Figure 4 .

 4 Figure 4.10: A plane region (left), and the associated Reeb graph (right), with respect to a projection (that may be viewed as a map to S 1 ).

Figure 4 . 11 :

 411 Figure 4.11: Two silhouettes, with inscribed Reeb graphs associated to the epipolar projections (left and center) and the Reeb graph of their visual hull (right).

Figure

  Figure 4.12 shows that when the Reeb graphs G T 1 , G T 2 have non-extremal maxima of different heights (which represent unmatched internal epipolar tangencies), then H must have some level sets with at least three connected components (i.e., the visual hull must have least three "local maxima") in order for surjective continuous projections to G T 1 , G T 2 to exist (Proposition 4.2.10). On the other hand, if the silhouettes had matching internal epipolar tangencies, two connected components would have been sufficient.

Figure 4 .

 4 Figure 4.12: A three dimensional set S projecting onto T 1 , T 2 (that have unmatched internal epipolar tangencies) must present level sets with three connected components (left and center). However, the visual hull of T 1 , T 2 actually has level sets with four connected components (right).

Definition 4 . 3 . 1 .

 431 A closed finite set K ⊆ P n is convex if there exists a closed pointed convex cone C ⊆ R n+1 such that K = PC. See Figure 4.14.

Figure 4 .

 4 Figure 4.14: A convex set in P n is the projectivization of a closed pointed convex cone in R n+1

Proposition 4 . 3 . 3 .

 433 Let T ⊆ P n be a finite connected set and let C be a pointed cone over T . A hyperplane [ϕ] ∈ (P n ) * belongs T • if and only if ϕ(x) ≥ 0 for all x ∈ C or ϕ(x) ≤ 0 for all x ∈ C.

Figure 4 . 15 :

 415 Figure 4.15: Dual pairwise-consistency. The dual images of the silhouettes P ∨ 1 (T • 1 ) and P ∨ 2 (T • 2 ) are sectionally consistent if and only if they have the same intersection with the "dual baseline" (c 1 ∨ c 2 ) ∨ .

Figure 4 . 16 :

 416 Figure 4.16: Structure of the "dual visual hull". The boundary of the dual visual hull is composed of planar curves, ruled surface patches and planar triangular patches: each of these are associated with specific components of the visual hull. See text for details.

Definition 4 . 4 . 1 .Definition 4 . 4 . 2 .

 441442 The sets of lines L 1 , . . . , L n are consistent if every line in L = n i=1 L i belongs to a n-incidence. The sets L 1 , . . . , L n are k-consistent if any subfamily L i 1 , . . . , L i k is consistent. In other words, L 1 , . . . , L n are k-consistent if for every subset I = {i 1 , . . . , i k } ⊂ {1, 2 . . . n}, every line in L i 1 ∪ . . . ∪ L i k belongs to a k-incidence for I.

Figure 4 . 18 :

 418 Figure 4.18: Two non-planar examples of 12 lines in 4 colors that are 3-consistent and have no 4-incidence. Left: A variation of Desargues' configuration. Right: A subset of the configuration of Reye; note that triples of parallel lines intersect at infinity.

Figure 4 . 19 :

 419 Figure 4.19: An object (left), its complete visual hull (center), and its finite visual hull associated with four viewpoints (right). Note that the finite visual hull is not topologically equivalent to the original object.

Definition 4 . 5 . 1 .

 451 The complete visual hull associated with S is the set VH (S) of points x in P 3 such that all lines through x meet S:VH (S) = {x ∈ P 3 | x ∈ ∈ Gr(1, P 3 ) ⇒ ∩ S = ∅} ⊂ P 3 .(4.10)

Remark 4 . 5 . 3 .

 453 If c is a point in P 3 and is a line containing c, then (S | c) determines (S | ): indeed, a plane H belongs to (S | ) if and only if there exists a line in (S | c) contained in H. In particular, given two points c 1 , c 2 the sets of lines L 1 = (S | c 1 ) and L 2 = (S | c 2 ) must induce the same set of planes (S | (c 1 ∨ c 2 )). This is essentially a geometric way of expressing the consistency of L 1 , L 2 (see Definition 4.4.1).

8

 8

Figure 4 . 21 :

 421 Figure 4.21: Inconsistencies in real data. Left: epipolar tangencies in real images do not correspond exactly. Right: In the presence of noise, frontier points degenerate to "frontier segments" on the visual hull.

Figure 4 .

 4 Figure 4.22: A viewing ray intersects the visual hull at "intersection intervals", which may be computed using epipolar geometry without an explicit model of the visual hull.

Figure 4 .

 4 23 (bottom) shows the set of all intersection curves[START_REF] Lazebnik | Projective visual hulls[END_REF] computed using 24 silhouettes.

Figure 4 . 23 :

 423 Figure 4.23: Top: Recovering the combinatorial components from intersection intervalsfor points on a contour. Using the characterizations given in the text, we see that the green segment contains general points on a cone strip (they project onto one contour), the two red points belong to an intersection curve (they project onto two contours), and the blue point is a triple point (it projects onto three contours). Bottom: Four views of the intersection curves, computed using 24 silhouettes.

Figure 4 .

 4 Figure 4.25: "Alien" dataset: 24 silhouettes and projection matrices. Top: Four views of the intersection curves. Bottom: Four views of a point cloud on the visual hull surface (approximately 70000 points).

Figure 4 .

 4 Figure 4.26: "Predator" dataset: 24 silhouettes and projection matrices. Top: Four views of the intersection curves. Bottom: Four views of a point cloud on the visual hull surface (approximately 130000 points).

Figure 4 .

 4 Figure 4.27: "Roman" dataset: 48 silhouettes and projection matrices. Top: Four views of the intersection curves. Bottom: Four views of a point cloud on the visual hull surface (approximately 240000 points).

Figure 4 .

 4 Figure 4.28: "Skull" dataset: 24 silhouettes and projection matrices. Top: Four views of the intersection curves. Bottom: Four views of a point cloud on the visual hull surface (approximately 140000 points).

Figure 4 . 29 :

 429 Figure 4.29: The hardness of a point on the surface of the visual hull is essentially the width of the cone strip at the point, i.e., the length of the intersection interval.

Figure 4 . 30 :

 430 Figure 4.30: Colored point clouds, using 4,8, 12, and 24 silhouettes respectively, where the color of each point represents the hardness. In these four cases, the average hardness is 149.82, 60.26, 46.88, 31.06 respectively, compared with the dimensions of object that are approximately 250 × 450 × 550. These quantitative measurements confirm the visible fact that there is a significant improvement in the quality of the visual hull reconstruction when passing from 4 to 8 silhouettes. Note also how, even in the case of 4 views, points with low hardness correctly belong to the actual object.

Theorem 5 . 2 . 2 .

 522 Every developable surface S in P 3 is a ruled surface, i.e., it satisfies S = S C for some curve C in Gr(1, P 3 ). For a curve C in Gr(1, P 3 ), the corresponding ruled surface S C is developable if and only if all tangent lines of C viewed as a curve in P 5 are contained in the Plücker quadric. Proof. For the first statement see [50, Chapter V, §344]. The second is [13, Prop. 12.4.1].

Figure 5 . 2 :

 52 Figure 5.2: Loci of planes and lines that meet a curve X with assigned multiplicities.

Figure 5 . 3 :

 53 Figure 5.3: Loci of planes and lines that meet a surface X with assigned multiplicities.

Figure 5 . 4 :

 54 Figure 5.4: A simple quartic surface X and its dual X ∨ . Note that the dual presents singular curves: the nodal curve E p (X) corresponds to bitangent planes for the original surface, while the cuspidal curve P p (X) represents osculating planes.

Figure 5 . 5 :

 55 Figure 5.5: Changing views of a curve correspond to Reidemeister moves. The viewpoint z crosses the tangential surface (left), edge surface (middle), or trisecant surface (right).

Figure 5 . 6 .

 56 Similar pictures are quite common in the computer vision literature: for instance, see [139, Figures5 and 6], and Figures 13.20 through 13.25 in the textbook[START_REF] David | Computer Vision -A Modern Approach, Second Edition[END_REF]. In Figure5.7, we show some actual examples of parabolic and flecnodal surfaces.

Figure 5 . 6 :

 56 Figure 5.6:The catalogue of visual events for the projections of a smooth surface from a viewpoint that moves. The local events (left, from top to bottom) are lip, beak-to-beak, swallowtail. The multi-local events (right, from top to bottom) are tangent crossing, cusp crossing, triple point. Reprinted from[START_REF] Pae | On computing structural changes in evolving surfaces and their appearance[END_REF] with permission of Springer.

Figure 5 . 7 :

 57 Figure 5.7: Visual event surfaces. The first two figures show two parabolic surfaces, corresponding to a lip event (left) and a beak-to-beak event (center). The parabolic curves on the surface are drawn in red. The last figure shows a flecnodal curve and a flecnodal surface.

[ 4 )

 4 If x and y are column vectors representing points on the line, then our definition for the associated matrix P * is xy T -yx T . Similarly, if u and v are column vectors representing planes containing the line, then the matrix P is uv T -vu T . The conditions rank(P) = 2, rank(P * ) = 2, and trace(PP * ) = 0 are all equivalent to the Plücker relation (A.3).

Definition B. 1 . 1 .

 11 A projective variety (or simply variety) X ⊂ P n is the zero-set of a family of homogeneous polynomials. More precisely, X is a projective variety if there exists a set S ⊂ C[x 1 , . . . , x n+1 ] of homogeneous polynomials such that X = V (S) where V (S) = {a ∈ P n | F (a) = 0, ∀F ∈ S}.(B.1)

I

  (X) = {F ∈ C[x 1 , . . . , x n+1 ] | F (a) = 0, ∀A ∈ X}. This set is always a homogeneous ideal in C[x 1 , . . . , x n+1 ].According to Definitions B.1.1 and B.1.2, a variety X has an associated ideal I(X), and an ideal I has an associated variety V (I). This association between varieties and polynomial ideals is the first brick in the foundation of algebraic geometry. The following are some basic properties of this correspondence. i) If S is a general set of polynomials in C[x 1 , . . . , x n+1 ], then V (S) = V (I) where I is the smallest ideal containing S.

Theorem B. 1 . 3 (

 13 Projective Nullstellensatz). If I is contained in M = x 1 , . . . , x n+1 , then I(V (I)) = 1 . Otherwise I(V (I)) = √ I.

F 1 xF

 1 (p) = . . . = ∆ m+h-(p) = 0.

Figure B. 1 Figure B. 1 :

 11 Figure B.1: Local shape of a surface: an elliptic point (left), a hyperbolic point (center), and a parabolic point (right).

  Figure B.2.

  R.<x1,x2,x3,x4> = PolynomialRing(QQ)Here the ring R has variables x1,x2,x3,x4. We define an ideal in R as follows.I_C = ideal(x1*x3 -x2^2, x1*x4 -x2*x3, x2*x4 -x3^2)The ideal I_C represents a twisted cubic curve C in P 3 . The projection of this curve under the pinhole camera [x 1 , x 2 , x 3 , x 4 ] → [x 1 , x 2 , x 3 ] can be obtained simply be eliminating the variable x4: J_C = I_X.elimination_ideal(x4); print(J)The output is:Ideal (x2^2 -x1*x3) ofMultivariate Polynomial Ring in x1, x2, x3, x4 over Rational Field

#

  xi are space variables, uj are image variables R.<x1,x2,x3,x4,u1,u2,u3> = PolynomialRing(QQ) # ideal of curve (can be arbitrary) I_C = ideal(x1*x3 -x2^2, x1*x4 -x2*x3, x2*x4 -x3^2) # vectors of variables: X = matrix([[x1],[x2],[x3],[x4]])

Proposition 1.3.15.

  If P 1 , . . . , P n and Q 1 , . . . , Q n are projectively equivalent, then they define the same joint image M (P 1 , . . . , P n

Table 1 .

 1 Bilinear constraints alone weakly characterize M n if and only if the pinholes are not all collinear.

		Non coplanar Coplanar	Collinear
	Bil.	Strong	Weak	Not sufficient
	Bil.+Tril.	Strong	Strong	Strong

Theorem 1.3.19. Assume n cameras are given. 1. Bilinearities and trilinearities (the sets S 2 , S 3 from Definition 1.3.8) always strongly characterize M n , independently of the pinhole configuration. 2. Bilinear constraints alone strongly characterize M n , if and only if the pinholes are not all coplanar 3. 1: Summary of the results of Theorem 1.3.19.

  Pinhole Cameras and their Multi-View GeometryProof. If (u 1 , . . . , u n ) is an n-tuple of image points that satisfy all of the constraints given in (A), then Proposition 1.3.19 guarantees that the visual rays associated to (u 1 , u 2 , u 3 , u i ) converge for all i = 4, . . . , n. Since c 1 , c 2 , c 3 are necessarily not collinear, this implies that all the visual rays intersect, so that (u 1 , . . . , u n ) is in fact a correspondence. Similarly, the set of constraints given in (B) allow one to determine a consistent set of camera parameters: it is enough to note that F 12 , F 1i , F 2i can be used to recover the camera configurations for all triples 1, 2, i, so by fixing cameras P 1 , P 2 compatible with F 12 , we can uniquely recover all of the remaining cameras.

Proposition 1.4.1. Consider n ≥ 4 cameras with pinholes c 1 , . . . , c n . We write F ij for the fundamental matrix associated with cameras i and j. (A) If three pinholes (say) c 1 , c 2 , c 3 are not coplanar with any other c i for i ≥ 4, then the fundamental matrices {F 12 , F 13 , F 23 } ∪ {F 1i , F 2i , F 3i } i=4,...,n identify 3n -6 bilinear constraints that strongly characterizes the joint image. (B) If two pinholes (say) c 1 , c 2 are not collinear with any other c i for i ≥ 3, then the fundamental matrices {F 12 } ∪ {F 1i , F 2i } i=3,...,n identify 2n -3 bilinear constraints that weakly characterize the joint image.

Lemma 1.4.5. A

  P 1 , . . . , S n P n ) and (S 1 P 1 , . . . , S n P n ). This implies that F G (P 1 , . . . , P n ) is solvable if and only F G (S 1 P 1 , . . . , S n P n ) is. viewing graph G is solvable if and only if, for generic cameras P 1 , . . . , P n , the fundamental matrices F G (P 1 , . . . , P n ) = {F (P i , P j ) | (i, j) ∈ E G } uniquely determine the remaining fundamental matrices {F (P i , P j

	Pinhole Cameras and their Multi-View Geometry
	In the rest of the section we will only consider generic configurations of cameras/pinholes
	(so a complete graph will always be solvable). This covers most cases of practical interest,
	although in the future degenerate configurations (including some collinear or coplanar
	pinholes) could be studied as well.
	Definition 1.4.4. A viewing graph G is said to be solvable if F G (P 1 , . . . , P n ) is solvable
	for generic cameras P 1 , . . . , P n .
	In other words, solvable viewing graphs describe sets of fundamental matrices that are
	generically sufficient to recover a camera configuration. Despite its clear significance, the
	problem of characterizing which viewing graphs are solvable has not been studied much,
	and only partial answers are available in the literature (mainly in [113, 157]). It is quite
	easy to produce examples of graphs that are solvable, but it is much more challenging,
	given a graph, to determine whether it is solvable or not. The following observation
	provides another useful formulation of solvability (note that the "if" part requires the
	genericity assumption, as shown in Example 1.4.3).
		on the graph G and
	on the pinholes c 1 , . . . , c n of P 1 , . . . , P n .
	Proof. The statement expresses the fact that changes of image coordinates are only a
	"relabeling" of a camera configuration and the associated fundamental matrices. More
	precisely, if S 1 , . . . , S n are arbitrary projective transformations of P 2 , then (P 1 , . . . , P n )
	and (P 1 , . . . , P n ) are in the same configuration if and only if the same is true for
	(S 1 Example 1.4.3. If G is a complete graph with n ≥ 3 vertices, then F G (P 1 , . . . , P n ) is
	solvable if and only if the pinholes of the cameras P 1 , . . . , P n are not all aligned. Indeed,
	if the pinholes are aligned, then the fundamental matrices between all pairs of cameras
	are not sufficient to completely determine the configuration: replacing any P i = [P i ]
	with P i = [P i (I 4 + c j v T )], where c j = [c j ] is the pinhole of another camera and v T
	is arbitrary, yields a new set of cameras which belongs to a different configuration but
	has the same set of fundamental matrices. Conversely, the complete set of fundamental
	matrices determines a unique camera configuration whenever there are at least three
	non-aligned pinholes (cf. Theorem 1.3.19).	♦

1 , . . . , P n ) implies that (P 1 , . . . , P n ) and (P 1 , . . . , P n ) are in the same projective configuration. In other words, a set of fundamental matrices is solvable if and only if it uniquely determines a projective configuration of cameras. Proposition 1.4.2. The solvability of F G (P 1 , . . . , P n ) only depends

  If c i = [c i ] is the pinhole of the camera P i associated with i, then we consider two distinct projective transformations of the form T 1 = [I 4 +α 1

	represented by the edges of G.
	Proposition 1.4.7. Any solvable graph is 2-connected, i.e., it has the property that after
	removing any vertex the graph remains connected.
	Proof. Assume that a vertex i disconnects the graph G into two components G 1 , G 2 , and
	let P 1 , . . . , P n be a set of n generic cameras, whose pairwise fundamental matrices are

  2 ) n whose dimensions is larger than 3 (and thus does not contain the joint image M n as a maximal dimensional component) but still completely determines M n . Mathematically, this is an interesting phenomenon that we hope to investigate in the future.

	Some values of e(n) are listed in Table 1.2 (here d(n) represents the minimal number
	of constraints on the fundamental matrices, and will be discussed shortly). Note that
	e(n) < 2n -3 for all n ≥ 5.
	n	2 3 4 5 6 7	8	9	10 11 12 13 14 15 16
	e(n) 1 3 5 6 8 9 11 12 14 16 17 19 20 22 23
	d(n) 0 3 6 2 5 1	4	0	3	6	2	5	1	4	0
		Table 1.2: The relation between n, e(n), and d(n).

  12 , F 23 , F 31 relating three pairs of cameras with non-aligned pinholes are compatible if and only if ij = [e ij ] is the epipole in image i relative to the camera j [78, Theorem 15.6].

	e T 13 F 12 e 23 = e T 21 F 23 e 31 = e T 32 F 31 e 12 = 0,	(1.23)
	where e

Table 1

 1

	.2).
	We now use Proposition 1.4.11 to deduce a new necessary condition for solvability.

Theorem 1.4.12. Let

  G be a solvable graph with n vertices and e edges. Then for any subgraph G of G with n vertices and e edges we must have

	d(n , e ) ≤ d(n, e),	(1.24)
	where d(n, e) = 7e -11n + 15. More generally, if G 1 , . . . , G k are subgraphs of G, each
	with n i vertices and e i edges, with the property that the edge sets E G i ⊂ E G are pairwise
	disjoint, then we must have
	k
	d(n i , e i ) ≤ d(n, e).	(1.25)
	i=1

  Assume that e 21 = P 2 c 1 and e 32 = P 3 c 2 are fixed epipoles in images 2 and 3, and that the fundamental matrices F 24 = F (P 2 , P 4 ), F 34 = F (P 3 , P 4 ) are also fixed. If c 1 , c 2 , c 4 are not aligned, we can use F 24 to "transfer" the point e 21 , and obtain a line l 41 in image 4 that contains epipole e 41 . Similarly, if c 1 , c 3 , c 4 are not aligned, we obtain another line m 41 using the same procedure with F 34 and e 31 . If the pinholes c 1 , c 2 , c 3 , c 4 are not all coplanar, the lines l 41 and m 41 will be distinct, and their point of intersection will be e 41 . This implies that we can draw a dashed arrow from 1 to 4.Move III: Assume that the epipoles e 21 and e 12 are fixed, and that the images of three other pinholes c 3 , c 4 , c 5 are fixed in both images 1 and 2. If the planes c 1 , c 2 , c i for i = 3, 4, 5 are distinct, then the images of c 3 , c 4 , c 5 give three correspondences that fix the "epipolar line homography"[START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. This completely determines F 12 , and we can draw a solid edge between 1 and 2.

	Pinhole Cameras and their Multi-View Geometry
	Move I: The second diagonal of the square is deducible from the other edges because
	the square with one diagonal is a solvable graph (this is a simple consequence of
	Proposition 1.4.8).
	Move II:
	.6).
	I) If there are solid edges defining a four-cycle with one diagonal, draw the other
	diagonal.
	II) If there are dashed arrows 1 → 2, 1 → 3, and solid edges 2 -4 and 3 -4, draw a
	dashed arrow 1 → 4.
	III) If there are double dashed arrows 1 ↔ 2, together with three pairs of dashed arrows
	i → 1, i → 2 for i = 3, 4, 5, make the arrow between 1 and 2 a solid (undirected)
	edge.
	Figure 1.6: Three moves (left: I, center: II, right: III) that can be used to prove
	solvability.
	Theorem 1.4.14. Let G be a viewing graph. If applying the three moves described above
	iteratively to G we obtain a complete graph, then G is solvable.
	Proof. For each of the three moves we need to show that the new edges contain infor-
	mation about the unknown fundamental matrices that can actually be deduced from
	F G (P 1 , . . . , P n ).

Example 1.4.15. Using

	Theorem 1.4.14, we can show that all graphs from Figure 1.4
	are solvable. Figure 1.7 illustrates this explicitly for two cases (n = 6 and n = 8).	♦
	II (2x)	III (2x)	I (6x)
	II (6x)	III (3x)	I (9x)
	Figure 1.7: Two applications of Theorem 1.4.14 to prove that viewing graphs are solvable.

Example 1.4.17. The

  graph shown in Figure1.8 is one of the five cases with n = 8, e = 11 that are "candidates" but are not "solvable with moves". However, we can show that this graph is actually solvable by arguing that the image of the pinhole 1 in the view 7 is fixed, even if this is not a consequence of the moves of Theorem 1.4.14 (this is represented by the gray dashed arrow in the figure).

								6, or using other
	arguments. The following examples present a few interesting cases.
	(n, e(n))	(3,3) (4,5) (5,6) (6,8) (7,9) (8,11) (9,12)
	connected	1	1	5	22	107	814	4495
	candidates	1	1	1	4	3	36	28
	solvable with moves	1	1	1	4	3	31	5
	finite solvable	1	1	1	4	3	36	27
	Table 1.3: Solvability of minimal viewing graphs using our methods.

  The epipolar constraints {b 12 , b 13 , b 23 } uniquely determine a camera configuration, i.e., they provide a weak characterization of the joint image.

	Pinhole Cameras and their Multi-View Geometry
	2. • If the cameras have collinear pinholes:
	1. A strong characterization of the joint image is given by {b 12 , b 13 , b 23 , t 1 , t 2 }
	where t 1 and t 2 are (sufficiently general) trilinear constraints.
	2. Two epipolar constraints together with one (sufficiently general) trilinearity
	{b 12 , b 13 , t} uniquely determine camera geometry, i.e., they give a weak char-
	acterization of the joint image.
	According to Theorem 1.3.19, for n = 3 views epipolar constraints cannot be used
	to provide a strong characterization of correspondences. This is also illustrated by
	Example 1.3.18. The following proposition explains how many constraints are actually
	necessary. The proof is computational; details are given at the end of the chapter.
	Proposition 1.5.1. Consider n = 3 cameras. We write b ij for the bilinear (epipolar)
	constraint among cameras i and j.
	• If the pinholes are non-collinear:
	1. For any trilinearity t that does not vanish on the product of the trifocal lines,
	the constraints {b 12 , b 13 , b 23 , t} provide a strong characterization of the joint
	image.

  1, P 3 ) ⊂ P 5 . Its ideal in Q[p 12 , p 13 , p 14 , p 23 , p 24 , p 34 ] is generated by the Plücker quadric and by p 3 23 -p 13 p 23 p 24 + p 12 p 2 24 + p 2 13 p 34 -p 12 p 14 p 34 -2p 12 p 23 p 34 .

Table

	Epi12	Epi13	Tr1A	Tr1B	Epi23	Epi21	Tr2A	Tr2B	Epi31	Epi32	Tr3A	Tr3B
	0.60	0.19	0.49	2.95	0.74	0.76	0.77	0.74	0.21	0.66	2.80	0.84

  3.19.

Theorem 1.3.19. Assume n cameras are given. 1. Bilinearities and trilinearities (the sets S 2 , S 3 from Definition 1.3.8) always strongly characterize M n , independently of the pinhole configuration. 2. Bilinear constraints alone strongly characterize M n , if and only if the pinholes are not all coplanar 3. Bilinear constraints alone weakly characterize M n if and only if the pinholes are not all collinear.

  To motivate this fact, one can show that (t 26 ) 2 Proofs does belong to the ideal generated by (1.67) (i.e., (t 26 ) 2 can be written as an algebraic combination of the nine trilinearities) even if t 26 doesn't: this clearly implies that adding the constraint t 26 to (1.67) does not impose any additional conditions on the zero-set (since t 26 = 0 if and only if t 2 26 = 0). Consider n = 3 cameras. We write b ij for the bilinear (epipolar) constraint among cameras i and j. • If the pinholes are non-collinear: 1. For any trilinearity t that does not vanish on the product of the trifocal lines, the constraints {b 12 , b 13 , b 23 , t} provide a strong characterization of the joint image.

	Proposition 1.5.1.

2. The epipolar constraints {b 12 , b 13 , b 23 } uniquely determine a camera configuration, i.e., they provide a weak characterization of the joint image. • If the cameras have collinear pinholes: 1. A strong characterization of the joint image is given by {b 12 , b 13 , b 23 , t 1 , t 2 } where t 1 and t 2 are (sufficiently general) trilinear constraints. 2. Two epipolar constraints together with one (sufficiently general) trilinearity {b 12 , b 13 , t} uniquely determine camera geometry, i.e., they give a weak characterization of the joint image. Proof. Let us first assume that P 1 , P 2 , P 3 have non-collinear pinholes. 1. The epipolar constraints describe the same projective locus as nine trilinearities (namely, t 12 , t 13 , t 23 , t 45 , t 46 etc.

  3 . The focal locus F(C) of a congruence C can be recovered computationally as follows. Let I be the ideal in C[p 12 , p 13 , p 14 , p 23 , p 24 , p 34 ] that defines C. Of course, p 14 p 23 -p 13 p 24 + p 12 p 34 ∈ I. The set of lines in C that pass through a point x in P 3 is given by the ideal

	Remark 2.2.2. I + P x ,	(2.5)
	where P is the 4 × 4 Plücker matrix with indeterminate coefficients from (2.4). For a
	generic x in P

3

, the ideal (2.5) has exactly α complex zeros in P 5 . To compute the focal locus, we treat the coordinates of x as parameters, and we add to (2.5) the 5 × 5 minors of the Jacobian matrix of (2.5) with respect to the Plücker coordinates. This gives an ideal in C[p 12 , . . . , p 34 , x 1 , . . . , x 4 ]. By saturating and eliminating p 12 , . . . , p 34 , we obtain the ideal in C[x 1 , x 2 , x 3 , x 4 ] that defines the focal locus F(C) in P 3 .

  -p 13 p 23 x 2 1 -p 14 p 24 x 2 1 + p 2 13 x 1 x 2 + p 2 14 x 1 x 2 -p 12 p 13 x 1 x 3 -p 12 p 14 x 1 x 4 p 12 p 23 x 2 1 -p 14 p 34 x 2 1 -p 12 p 13 x 1 x 2 + p 2 12 x 1 x 3 + p 2 14 x 1 x 3 -p 13 p 14 x 1 x 4 p 12 p 24 x 2 1 + p 13 p 34 x 2 1 -p 12 p 14 x 1 x 2 -p 13 p 14 x 1 x 3 + p 2 12 x 1 x 4 + p 2 13 x 1 x 4 p 12 p 23 x 1 x 2 -p 14 p 34 x 1 x 2 -p 12 p 13 x 2 2 + p 13 p 23 x 1 x 3 + p 14 p 24 x 1 x 3 +p 2 12 x 2 x 3 -p 2 13 x 2 x 3 +p 12 p 13 x 2 3 -p 13 p 14 x 2 x 4 +p 12 p 14 x 3 x 4 p 12 p 24 x 1 x 2 + p 13 p 34 x 1 x 2 -p 12 p 14 x 2 2 -p 13 p 14 x 2 x 3 + p 13 p 23 x 1 x 4 +p 14 p 24 x 1 x 4 +p 2 12 x 2 x 4 -p 2 14 x 2 x 4 +p 12 p 13 x 3 x 4 +p 12 p 14 x 2 4 p 12 p 14 x 2 x 3 -p 12 p 24 x 1 x 3 -p 13 p 34 x 1 x 3 + p 13 p 14 x 2 3 + p 12 p 23 x 1 x 4 -p 14 p 34 x 1 x 4 -p 12 p 13 x 2 x 4 -p 2 13 x 3 x 4 +p 2 14 x 3 x 4 -p 13 p 14 x 2

		
	  	   .
	4

  1, -1]. The corresponding (1, 3)-congruence C is parametrized by (2.14). The ideal of C equals p 12 , p 14 p 23 -p 13 p 24 , p 13 p 2 14 -p 2 23 p 24 -p 13 p 14 p 34 + p 23 p 24 p 34 , p 3 14 -p 23 p 2 24 -p 2 14 p 34 + p 2 24 p 34 , p 2 13 p 14 -p 3 23 -p 2 13 p 34 + p 2 23 p 34 . = 0. The ideal of the resulting (1, 3)-congruence is p 12 , p 14 p 23 -p 13 p 24 , p 13 p 2 14 -p 2 23 p 24 , p 3 14 -p 23 p 2 24 , p 2 13 p 14 -p 3 23 .

	(2.16)
	♦

♦

Type 4: F(C) is a non-reduced line L. This is the degenerate case of Type 3 congruences when the binary form f is identically zero. The degree β morphism φ :

L ∨ → L promised in Theorem 2.3.3 sends [x 1 , x 2 ] to the point [0, 0, x 3 , x 4 ] = 0, 0, g(x 1 , x 2 ), h(x 1 , x 2 )

on the line L ⊂ P 3 . The corresponding rational camera is given by the formula (2.14) with f = 0.

Example 2.3.8. Let β = 3 as in Example 2.3.7 but now with f = 0, g = s 3 and h = t 3 . The non-reduced structure of L is the ideal x

2

1 , x 1 x 2 , x 2 2 , obtained from (2.15) by setting x 4

  the concurrent lines variety in the polynomial ring of 6n Plücker coordinates. Let P 1 , P 2 , . . . , P n be skew-symmetric 4×4-matrices of unknowns that represent lines in P 3 , and let P * 1 , P * 2 , . . . , P * n be the dual matrices. The ideal I n is minimally generated by the n+1 2 quadrics trace(P i P * j ) and the 10 n 3 cubics obtained as 3×3minors of P 1 u, P 2 u, . . . , P n u where u runs over {e 1 , e 2 , e 3 , e 4 , e 1 +e 2 , e 1 +e 3 , . . . , e 3 +e 4 }, and e 1 , e 2 , e 3 , e 4 represent the fundamental points in P 3 . For the reverse lexicographic order, the reduced Gröbner basis of I n consists of n+1

	Theorem 2.4.1. 2	quadrics, 12 n 3 cubics and 4 n+1 4
	quartics.

  e 2 , e 3 , e 4 , e 1 + e 2 , e 1 + e 3 , e 1 + e 4 , e 2 + e 3 , e 2 + e 4 , e 3 + e 4 . (2.19)We note that the cubics for coplanar triples of lines are the 3 × 3-minors of the 4 × 3matrix (P * u, Q * u, R * u), where u ∈ R 4 . A basis of 10 cubics is obtained from the same set(2.19).Using Macaulay2, we now compute the reduced Gröbner basis of our prime ideal from the 6 + 10 = 16 generators with respect to the reverse lexicographic order determined by p 12 >p 13 >p 14 >p 23 >p 24 >p 34 > q 12 >q 13 >q 14 >q 23 >q 24 >q 34 > r 12 >r 13 >r 14 >r 23 >r 24 >r 34 .

  Let β 1 = β 2 = β 3 = 1 and fix three two-slit cameras C 1 , C 2 , C 3 defined by J = p 12 , p 34 , q 13 , q 24 , r 14 , r[START_REF] Boyd | Convex optimization[END_REF] ⊂ R[p 12 , . . . , p 34 , q 12 , . . . , q 34 , r 12 , . . . , r 34 ].

there exists x ∈ P 3 such that x ∈ L i for all i. If x does not lie in any of the n focal loci then L i = C i (x) and we are done. Otherwise, x lies in exactly one of the focal loci, say, x ∈ F(C i ). We then consider a sequence of nearby pairs (x , L i, ) that converges to (x, L i ) and satisfies x ∈ L i, \F(C i ) and C i (x ) = L i, . For each j ∈ {1, 2, . . . , n}\{i} the locus F(C j ) is closed. Since it does not contain x, we can assume that it also does not contain x . Hence (C 1 (x ), . . . , C n (x )) is a well-defined sequence of points in the variety M (C 1 , . . . , C n ). It converges to (L 1 , . . . , L n ), which therefore also lies in M (C 1 , . . . , C n ). We now give an example for n = 3 that shows the necessity of the hypothesis on the focal loci in Theorem 2.4.3. Example 2.4.4.

  This reduces the minimal generators to n 2 quadrics and n

	3 n 1 2 n 2 + 6n 1	n 2 2 + 10 n 2 3	2	+ n 2 quadrics, and n 1 3 +

3

cubics. These are the bilinearities and trilinearities from Theorem 1.3.10.

We next generalize Proposition 2.4.5 to arrangements of n 1 pinhole cameras C 1 , . . . , C n 1 and n 2 two-slit cameras C 1 , . . . , C n 2 . These n = n 1 + n 2 cameras are assumed to satisfy the hypothesis of Theorem 2.4.3. Thus, the pinholes are distinct, the slits are pairwise disjoint, and no pinhole is allowed to lie on a slit. Theorem 2.4.6. The ideal of the multi-image variety M (C 1 , . . . , C n 1 , C 1 , . . . , C n 2 ) is minimally generated by 3n 1 + 2n 2 linear forms, n 1 +n 2

  p 24 , p 2 12 + p 2 13 -p 2 14 , p 12 p 24 + p 13 p 34 + p 14 p 34 , p 12 p 34 -p 13 p 24 + p 14 p 23 , I P (C 2 ) = p 13 -p 23 , p 2 14 -p 2 24 + p 2 34 , p 12 p 14 + p 12 p 24 -p 23 p 34 , p 12 p 34 -p 13 p 24 + p 14 p 23 . In both expressions, the first two polynomials are the Chow forms of L i and X i respectively. The ideal of the two-image variety M (C 1 , C 2 ) is given by (2.27).

  The ideal of the congruence C in R[p 12 , . . . , p 34 ] is generated by the Chow forms of L and X together with the Plücker quadric: I C = p 23 , p 2 14 -p 2 24 -p 2 34 , p 12 p 34 -p 13 p 24 + p 14 p 23 . (2.28)We can see that C has bidegree (2, 2) by counting the intersections with generic α-planes and β-planes. Since C has order higher than one, its focal locus is two-dimensional.

	Proposition 2.5.2. The focal locus F(C) of the non-central panoramic camera consists
	of the plane spanned by the conic X, taken with multiplicity 2, and a conjugate pair of
	complex planes that intersect in the line L. Algebraically, it is defined by the non-reduced
	quartic (x 2 2 + x 2 3 )x 2 4 .

  3 , x 4 ] and y = [0, y 2 , y 3 , y 4 ] lie on the plane at infinity, P 3 \U 1 , and they represent directions in U 1 . Let H be the plane in P 3 defined by a 1 x 1 +a 2 x 2 +a 3 x 3 + a 4 x 4 = 0. Here a i ∈ C is allowed, but we assume that H is non-isotropic, meaning that a 2 2 + a 2 3 + a 2 4 = 0. With the convention above, the reflection of a point y = [y 1 , y 2 , y 3 , y 4 ] with respect to H is ρ H

  The pair A 1 , A 2 represents a parallel finite two-slit camera, while B 1 , B 2 is a pushbroom camera. The associated epipolar tensor (3.32) is

	Photographic Cameras
	and
			A 2 1 =	1. -3.87 1. 1. 1. 0 0 0
	B 1 =	13 4 A 2 2 = 14 9 -3 8 0 0 0 1 , B 2 1 =	, A 2 = 0. -14.22 8.33 9.25 4.24 11 6 -2 4 1. 0. 0. , 8 -1 13 -5 B 2 = -3 8 10 3 6 13 5 13 0.44 0.20 0.27 -0.07 0. 0. 1. 0.	.	,	(3.36)	(3.39)
			B 2 2 =	0. -0.86 -1.34 -2.26 0.88 0. 0. 1.	.
		F =	0 21816 -25650 0	1906 -2090 -3642 5510
								(3.37)
			880 18600 -11875 475	97 -1259 1425 -380	,
		A 1 1 =	1. -3.87 1. 1. 1. 0. 0. 0.
		A 1 2 =	0. -14.22 8.33 -6.67 -22.17 1. 0. 0.	,
								(3.38)
		B 1 1 =	0. 0.44 -0.28 0.27 1.14 0. 1. 0.
		B 1 2 =	0. -0.86 0.26 0.15 0.88 0. 0. 1.	,

where each 4 × 4 matrix represents a block (f ijkl ) kl for fixed i, j. Note that f 1111 and f 1112 are zero, since the second rows of A 1 , A 2 , B 1 are linearly dependent. Using the approach outlined above, we can use this tensor to recover two matrices C 1 , C 2 whose principal minors are the entries of F (we must normalize F so that f 2222 = 1). We use these matrices to construct two pairs of two-silt cameras, namely

4

  zero yields a 11 a 21 m 11 + a 11 a 22 m 12 + a 11 a 23 m 13 + a 11 a 24 m 14 + a 12 a 21 m 21 + a 12 a 22 m 22 + a 12 a 23 m 23 + a 12 a 24 m 24 + a 13 a 21 m 31 + a 13 a 22 m 32 + a 13 a 23 m 33 + a 13 a 24 m 34 + a 14 a 21 m 41 + a 14 a 22 m 42 + a 14 a 23 m 43 + a 14 a 24 m 44 = 0,

	(3.44)
	where Ω * Q = (m ij ), and the elements of A i 1 = (a ij ) are known. A sufficient number of views allows us to estimate Ω * Q linearly. Finally, from the singular value decomposition
	of Ω *

Q , we can compute a matrix

  for the visual cone associated with T i . The sets T 1 , . . . , T n are consistent (relative to the cameras P 1 , . . . , P n ) if there exists a non-empty setR ⊆ P 3 \ {c 1 , . . . , c n } such that P i (R) = T i for all i = 1, . . . , n.If each of T 1 , . . . , T n is a single point, then consistency reduces to the usual notion of point correspondence (see Definition 1.3.1). Extending this analogy, consistent image sets can be seen as n-tuples of projections of all possible objects in space. Consistency can be equivalently defined using the joint image variety M (P 1 , . . . , P n ) in (P 2 )

	Definition 4.2.1.

n for the cameras P 1 , . . . , P n (see Definition 1.3.4). More precisely, T 1 , . . . , T n are consistent if and only if the set

  n. A family L 1 , . . . , L n of convex sets in P 2 is consistent for a set of projections P 1 , . . . , P n if and only if L • 1 , . . . , L • n are sectionally consistent for the embeddings P ∨ 1 , . . . , P ∨ n . Moreover, if consistency holds, and H is the finite visual hull associated with L 1 , . . . , L n , then

	This notion of sectional consistency is "dual" to geometric consistency from Defini-
	tion 4.2.1.
	Proposition 4.3.6.

  n} ⊂ P 3 . (4.12) By construction, the finite visual hull FVH (S | c 1 , . . . , c n ) is the largest set in P 3 such that (S | c i ) = (S | c i ) for all i = 1, . . . , n (rather than for all viewpoints c, as was case for the complete visual hull). The relationship with the notion of finite visual hull from Definition 4.2.2 is clear: if S is a set in P 3 that projects onto image sets T 1 , . . . , T n in P 2 for some cameras P 1 , . . . , P n with pinholes c 1 , . . . , c n , then T 1 , . . . , T n are consistent and their finite visual hull for P 1 , . . . , P n in the sense of Definition 4.2.2 is

  compatibility. In our study of consistency, we considered families of image sets T 1 , . . . , T n together with known camera projections P 1 , . . . , P n . On the other hand, an arguably more interesting question is to understand, given a family of sets T 1 , . . . , T n in P 2 , what are the possible cameras P 1 , . . . , P n that make these sets consistent. For this, we can revisit the notion of consistency from Definition 4.2.1 as a condition on both image sets as well as cameras, characterizing when T 1 , . . . , T n and P 1 , . . . , P n are "compatible". This idea is also important in multi-view geometry, where correspondence constraints can be viewed as conditions on image coordinates as well as on camera parameters. More generally, we can say that image sets T 1 , . . . , T n are compatible if there exist cameras P 1 , . . . , P n for which they are consistent according to our usual notion from Section 4.2.

  Consider now an irreducible curve C in Gr(1, P 3 ) that has degree d in P 5 . We write I C for its prime ideal in the R[p 12 , p 13 , p 14 , p 23 , p 24 , p 34 ]. The union of all lines on C forms a ruled surface S C in P 3 . The ruled surface S C is irreducible and it has degree d in P 3 . Conversely, every irreducible ruled surface in P 3 arises in this way from some irreducible curve C in Gr(1, P 3 ).Proof. This is one of the basic facts derived in Edge's book[START_REF] Leonard | The Theory of Ruled Surfaces[END_REF] Chapter I,[START_REF] Bröcker | Differentiable germs and catastrophes[END_REF].In practice, the defining polynomial of the surface S C ⊂ P 3 can be computed from C using the incidence conditions

						Preliminaries
	satisfying p 34 = 0 has the parametric representation
					p 12	
			z(t) =	    tp 13 -p 23 tp 12	   	∈ P 3 .	(5.1)
				tp 14 -p 24
	Lemma 5.2.1. 	0	p 34 -p 24 p 23
	 -p 34    p 24 -p 14 0	p 14 -p 13 0 p 12
	-p 23 p 13 -p 12	0

Table 5 .

 5 1: Degrees of the components of the visual event surface of a space curve.

	d g	deg(T (X)) deg(E(X)) deg(D(X))
	3 0	4	0	0
	4 0	6	6	2
	4 1	8	8	0
	5 0	8	16	8
	5 1	10	20	5
	5 2	12	24	2
	6 0	10	30	20
	6 1	12	36	16
	6 2	14	42	12
	6 3	16	48	8
	6 4	18	54	4

Table 5

 5 .1.

	Changing Views on Curves and Surfaces
	d	deg(F(X)) deg(C(X)) deg(T (X)) deg(E(X)) deg(P(X))
	3	0	0	0	0	30
	4	80	0	0	160	128
	5	260	510	0	930	330
	6	576	2448	624	3168	672
	7	1064	7308	3808	8260	1190

Table 5 .

 5 2: Degrees of the components of the visual event surface of a general surface.

1

  A (polynomial) ideal is a subset I of C[x 1 , . . . , x n+1 ] that forms an additive group (F 1 , F 2 ∈ I implies F 1 + F 2 ∈ I) and that is closed under multiplication with any other polynomial (F ∈ I then GF ∈ I for all G ∈ C[x 1 , . . . , x n+1 ]). One can show that all polynomials ideals are finitely generated, i.e., there always exist polynomials F 1 , . . . , F r such thatI = {A 1 T 1 + . . . + A r T r | A i ∈ C[x 1 , . . . , x n+1 ]}.In this case, we say that the polynomials F 1 , . . . , F k are generators of I, and we write I = F 1 , . . . , F r . The ideal I is a homogeneous ideal if I = F 1 , . . . , F r where each F i is a homogeneous polynomial.

Definition B.1.2. If X ⊂ P n is a variety, we define

  

	∂F 1 ∂x 1 (p) . . . ∂F 1 ∂x 1 (p) . . . . . . ∂Fr ∂x 1 (p) . . . ∂Fr ∂x 1 (p)	   

For concreteness, we consider the actions of GL(4, R) and GL(3, R) rather than of PGL(4, R) and PGL(3, R). Clearly, matrices that are scalar factors of each other will act the same way.

Here by "feasible joint image" we mean a variety in (P 2 ) n that arises as the joint image of a configuration of pinhole cameras.

This is the codimension of XG in H e where H ⊂ P 8 is the determinant hypersurface.

Here we actually need that TG(c1, . . . , cn) is smooth: this follows from a technical result, which states that an algebraic group (more properly a "group scheme") over a field of characteristic zero is always smooth[START_REF] Mumford | Abelian Varieties[END_REF] Sec.11].

One way to take this information into account is to define a new type of edge together with additional moves. We did not do this in Theorem 1.4.14, because it would have made the statement more complicated, and because this type of edge is never necessary for smaller graphs.

Three projection matrices provide an over-parameterization of the configuration space of three cameras. In[START_REF] Trager | Trinocular Geometry Revisited[END_REF], we also presented a minimal parameterization of three view geometry (defined on an open set of all configurations).

We should point out that we could not place all the points exactly on the trifocal plane, since the minimal reconstruction method we used for initialization requires points to be in general position[START_REF] Carlsson | Duality of reconstruction and positioning from projective views[END_REF]. Moreover, coplanar points give rise to a critical configuration for three views[START_REF] Hartley | Multiple view geometry in computer vision[END_REF], meaning that projectively non-equivalent triples of cameras can produce the same images.

+ n.

Assuming the ideal J generated by the fundamental multilinearities is radical, we can write J = I ∩ i (xi, yi, zi) where I = I(Mn) is the prime ideal describing Mn (defined by bilinear, trilinear, and quadrilinear relations[START_REF] Aholt | A Hilbert Scheme in Computer Vision[END_REF]). Any multilinearity that vanishes on Mn lies in I and in (xi, yi, zi) for all i, thus it must belong to J.

Note that the inverse of a matrix of the form I + cv T 1 is given by I + cvT 2 where v2 = -1 1+c T v 1 v1. In particular v2 is a scalar multiple of v1.

We point out that the matrices (1.65) do not satisfy these assumptions.

cubics that are listed in the statement of Theorem 2.4.1. Its leading terms are square-free.

This is actually closer to the "4D light field"[START_REF] Levoy | Light field rendering[END_REF], since the plenoptic function has a five-dimensional parameter space, that includes the point where a ray is being sampled.

Here by "applying" we actually mean that Q(x) = Q(T (x)), or Q (T -1 (x)) = Q(x).Exchanging the role of T and T -1 has no effect on our discussion.

Intrinsic parameters describing euclidean orbits among more general (non-parallel) two-slit cameras can also be defined, but two more parameters are required. We chose to consider only two-slits with retinal plane parallel to the slits, since this is a natural assumption, and because the parameters have a simpler interpretation in this case.

The result can be extended to the case of arbitrary pushbroom cameras, by allowing for the calibration matrix K2 to be a general 2 × 2 matrix with positive entries along the diagonal. In that case, however, the geometric interpretation for the parameters is less clear.

This is a floating point approximation of the matrices, because recovering C1, C2 requires solving quadratic equations. Since for the moment we do not assume any noise, it would also possible to use exact symbolic representations.

As elsewhere in our presentation, we consider for concreteness the action of GL rather than PGL. Clearly, matrices that are scalar factors of each other will act the same way.

Unfortunately the term "multidegree" has different meanings which are both necessary in this setting.

The most significant difference between the euclidean and projective frameworks in our setting is that in the latter case visual cones are two-sided. An alternative (but perhaps less natural) approach would have been the use of oriented projective geometry[START_REF] Stolfi | Oriented projective geometry: A framework for geometric computations[END_REF].

This condition is not actually necessary for all of our results, and weaker assumptions may often be considered. However, for the sake of simplicity, we give a single condition that is valid throughout the section.

To be precise, it should be noted that, although in projective space, none of the cones Cj for j = i contain ci (because of our assumption (A)). This means that we can treat the sets P -1 i (ui) ∩ Cj as intervals on an affine line.

Traditionally, the Reeb graph is defined for real-valued functions f : X → R, however for our purposes the "circular" definition is more natural. Note however any function f : X → S 1 that is not surjective may actually be viewed as real-valued.

Because he represents viewing rays as half-lines rather than lines, Laurentini actually defines an "internal" and an "external" (complete) visual hull, for distinguishing whether or not the viewpoint is allowed to lie inside the convex hull of the observed object. This distinction will not be relevant for us.

Available at http://www.cse.wustl.edu/~furukawa/research/visual_hull/index.html.

One could argue that the reprojection is not necessarily the "best" way to enforce consistency for noisy silhouettes. Ideally, one might try to compute a finite visual hull associated with inconsistent silhouettes T1, . . . , Tn that is "optimal" for an appropriate measure (such as Hausorff distance between Ti and Ti). We will not deal with this issue here.

In the case of collinear centers one needs to exclude the 3D points belonging to the baseline, which are all mapped to the product of the epipoles. We will disregard this very special degenerate case.

Moreover, a tracing procedure of the vertices of the intersection intervals on the viewing rays can be designed, along lines similar to[START_REF] Lazebnik | Projective visual hulls[END_REF], but without the additional cost of computing the part of intersection curves lying outside the final visual hull, or a special treatment for frontier points.

Presumably, Boyer and Franco considered only endpoints of the intersection intervals because they exploited only image contours, while, for robustly computing the intersection intervals, it is convenient to have access to the silhouette as a region (i.e., as a binary mask). Our approach, on the other hand, uses contours and masks. This means that a basic pre-processing step might be required to obtain both representations (e.g., if a continuous contour is given, a mask can be obtained using a "flood-fill" algorithm), however this is not an issue in practice. This aspect also points out how our viewpoint is somewhat in between "volumetric" and "surface-based" methods.

This upper bound is often not very tight, and, in fact, one could introduce variations of our definition which take into account the position of x relative to the intersection interval, or the intersection intervals associated with all of the views (rather than just the one associated to the cone strip). We chose this definition because it is much simpler and easily interpretable, and also because it provides a notion that is defined for image contour points as well.

degrees of freedom, compared to 7 in the unconstrained case. Indeed, if we fix the epipoles e 1 , e 2 outside of T 1 , T 2 , then the image sets impose two constraints on camera

Another PhD student, Kathlén Kohn, worked on this paper as well. This chapter does not include a technical section on intersection theory which was mainly her contribution.

In the computer vision literature, visual events are typically studied only for surfaces (mainly because surfaces are more suited for representing real-life objects). From a theoretical standpoint, however, it is quite natural to consider first the simpler case of projections of curves.

Saturation is an algebraic operation that removes components from an algebraic set. In our presentation, it is often used to remove components associated with the "irrelevant" ideal, that is generated by the coordinate variables. See Appendix B

The genus is an important birational invariant of algebraic curves (and more general algebraic varieties). For smooth complex curves, it coincides with the topological notion of genus for the associated Riemann surface. Here we will not need the actual definition of the genus, but we will use the fact that space curves of fixed degree and genus admit a uniform algebraic parameterization, known as a "Hilbert scheme"[START_REF] Arrondo | Introduction to projective varieties[END_REF]. In particular, this allows us to speak about the "general behavior" of space curves with a given degree and genus.

Note that if S = P(W ), then S ∨ is not the same as P(W * ). For example, (P n ) ∨ = ∅ = (P n ) * .

Projective Geometry and Plücker Coordinates

For some authors this is actually definition of an affine projective set, since they require projective varieties to be also irreducible (see Section B.2).

We mention that it is also possible to perform the computation above without saturating, by defining the incidence ideal more simply as I_incidence = ideal((M*X-U).list()). However, this is a "trick", since the correct incidence condition is given by the minors of [M*X|U].

Acknowledgments

am grateful to Jean for caring so much about my work, and for countless interesting discussions that always made me feel excited about doing research. I thank Martial for his support and wise advice, and for the months I spent at CMU, which I greatly enjoyed. I acknowledge the ERC advanced grant VideoWorld, the Insitut Universitaire de France, the GAYA Inria/CMU associated team for financial support. Many thanks go to my jury members -Tomas Pajdla,

Thesis summary

The following is a brief description of the main contributions of the thesis. The relevant previous literature for each topic is discussed at the beginning of the corresponding chapter.

Photographic Cameras

According to Theorem 1.5.5, the trifocal form for these cameras for β = (2, 1, 1) has degrees (d 1 , d 2 , d 3 ) given by the coefficients for degrees (1, 1, 1), (0, 2, 1), (0, 1, 2) in (3.52), so (d 1 , d 2 , d 3 ) = [START_REF] Arrondo | Introduction to projective varieties[END_REF][START_REF] Aholt | The ideal of the trifocal variety[END_REF][START_REF] Edward | The plenoptic function and the elements of early vision[END_REF]. Indeed, a computation shows that the trifocal form for these cameras is a polynomial with 579 terms that starts with T = u 6 1 r 3 1 s 2 1 -2u 5 1 u 3 r 3 1 s 2 1 -8u 4 1 u 2 u 3 r 3 1 s 2 1 + ... + 9u 3 1 u 3 3 r 3 3 s 2 3 .

Here the variables u 1 , u 2 , u 3 are projective coordinates representing points in the first image, while r 1 , r 2 , r 3 and s 1 , s 2 , s 3 are dual projective coordinates representing lines in the second and third image. ♦

Finally, classical multifocal tensors uniquely encode a configuration of pinhole cameras, and it is natural to ask whether this important property holds for non-linear multifocal forms as well. To address this question we need to introduce an appropriate "configuration space" for general rational cameras. For this we consider a set Q 1 , . . . , Q n of projective orbits of photographic cameras (i.e., Q i = {Q i T | T ∈ PGL(4, R)} for some camera Q i ), which we view as n fixed "camera models". We then define the set of configurations for these models as

This generalizes the space of configurations of pinhole cameras. Indeed, a configuration is always defined as the product of independent projective orbits, up to global projective equivalence.

It is easy to see that if (Q 1 , . . . , Q n ) and (Q 1 , . . . , Q n ) are projectively equivalent n-tuples, the joint images M (Q 1 , . . . , Q n ) and M (Q 1 , . . . , Q n ) are the same. This is because a joint image is independent of the choice of coordinates in P 3 (see also Proposition 1.3.15 for the case of pinhole cameras). This implies that we can associate a configuration in C(Q 1 , . . . , Q n ) with a joint image, i.e., there is a map

where [Q 1 , . . . , Q n ] denotes the configuration (equivalence class) of (Q 1 , . . . , Q n ). If all Q i are pinhole camera models, this association is injective (Theorem 1. 3.16). In general, this may not be the case: indeed, if Q 1 , Q 2 are a pair of two-slit cameras, then we have already noted that the joint image M (Q 1 , Q 2 ) corresponds in general to two projective configurations (Theorem 3.3.7). The following, however, is true.

Proposition 3.4.7. Let Q 1 , . . . , Q k be photographic cameras where k is either 2, 3 or 4.

We write of Q 1 , . . . , Q k for the corresponding "projective camera models".

1. Any multifocal form (as defined in Theorem 3.4.5) is invariant to changes of projective coordinates in P 3 .

Photographic Cameras

"practical" side, we would like to extend our photographic model to include non-rational mappings, particularly to be able to study catadioptric imaging systems. As explained in the previous chapter, this might require deriving an effective description for the action of refractions and reflections on line congruences.

Chapter 4

Consistency of Image Sets and Visual Hulls

This chapter studies the geometric relationship between 3D objects and their projections in multiple images. For most our discussion, we let an object be an arbitrary subset S of P 3 , so that its images for n different cameras are subsets T 1 , . . . , T n of P 2 . Being projections of the same object in space imposes constraints on image sets, leading to a notion of geometric consistency that resembles (and generalizes) the traditional "correspondence" for image points. Conversely, image sets can be "triangulated" to a 3D shape, that is known as the (finite) visual hull. We study the theory of consistency and projective visual hulls, pointing out connections with line geometry and duality. Some of these ideas are put into practice in a simple algorithm for computing the visual hull.

The material in this chapter is based on the following publications:

-Matthew Trager, Martial Hebert, and Jean Ponce. "Consistency of silhouettes and their duals". In: IEEE Conference on Computer Vision and Pattern Recognition. 2016.

-Boris Bukh, Xavier Goaoc, Alfredo Hubard, and Matthew Trager. "Consistent sets of lines with no colorful incidence". In: International Symposium on Computational Geometry. 2018.

Introduction

What images can be obtained by taking pictures of the same 3D object? Conversely, what are the 3D objects that can produce a given set of images? In this chapter, we investigate this type of questions from a purely geometric standpoint, disregarding pho-

Consistency of Image Sets

From pairwise to general consistency

We have seen that pairwise consistency does not imply general consistency [START_REF] Boyer | On using silhouettes for camera calibration[END_REF][START_REF] Man | Visual hull construction, alignment and refinement for human kinematic modeling, motion tracking and rendering[END_REF]: in fact, as shown by the example in Figure 4.6, the two notions are not equivalent even in the restricted case of convex image sets (contrary to a claim in [START_REF] Boyer | On using silhouettes for camera calibration[END_REF]). Interestingly, however, we can show that for convex sets (i.e., sets that are convex in an affine chart: see Section 4.3 for a discussion on projective convexity) general consistency is actually implied by 3-consistency. A similar fact was pointed out in [START_REF] Yi | Efficiently Determining Silhouette Consistency[END_REF].

Because of convexity, the sets P -1 i (u i ) ∩ C j are intervals; moreover, they intersect pairwise because of the assumption of 3-consistency: this implies that they all intersect. 3 See Figure 4.7.

A Dual View of Consistency

Finally, Figure 4.13 shows that Reeb graph for the visual hull of two connected silhouettes may be disconnected; this implies that the visual hull itself is also disconnected. This observation is actually closely related to the so-called "mountain-climbing" problem: this problem asks for conditions on functions describing two-dimensional "mountains", so that two "climbers" in different locations can coordinate their movements to meet while always staying at the same height [START_REF] James | A mountain-climbing problem[END_REF].

A Dual View of Consistency

In this section we revisit the notion of consistency from the viewpoint of duality. In particular, we discuss the relationship between consistency for projections, as presented in the previous section, and a different notion of consistency for planar sections of solids. We focus in our presentation on the case of convex sets, since the duality is much simpler in this setting. Section 4.3.1 is an informal introduction to convex duality in the projective setting; the interested reader can find more precise definitions in Section 4.3.3; our main results are given in Section 4.3.3.

Duality

The basis of many similar notions of duality is the fact that points and hyperplanes in some n-dimensional space can play symmetric roles. For example, in R n any hyperplane through the origin can be represented with its orthogonal vector. In projective space P n , all hyperplanes correspond to points of the dual projective space (P n ) * ; in fact, any k-dimensional linear subspace H in P n , can be associated with an (n -k)-dimensional linear subspace H * in (P n ) * (see Appendix A). The identification of hyperplanes in P n with points in the dual space (P n ) * can be used to introduce notions of duality for more general objects in P n . If S is a smooth or algebraic hypersurface in P n , (the closure of) the set of tangent hyperplanes at points of S forms a dual hypersurface S ∨ in (P n ) * [START_REF] Evgueni | Projective duality and homogeneous spaces[END_REF]. In this setting, however, the dual hypersurface will typically have self-intersections: for example, in the case of plane curves, crossings in the dual correspond to bitangents of Consistency of Image Sets and Visual Hulls the original curve. For this reason, we will consider here duality for (properly) convex sets in projective space [START_REF] Goldman | Projective geometry on manifolds[END_REF]. In the following, we will say that a set K in P n is convex if there exists an affine chart so that K is compact and convex in the usual affine sense. This is equivalent to a more analytical definition, for which K is convex if it is the projectivization of a cone K in R n+1 that is convex, closed, and pointed (i.e., it does not contain any line) [START_REF] Boyd | Convex optimization[END_REF]. To any convex set K in P n , we can now associate a dual set K • in (P n ) * , that can be characterized geometrically as the closure of the set of all hyperplanes H such that H ∩ K is empty. Alternatively, the same set K • is the projectivization of the polar cone ( K) • of a convex cone K in R n+1 that corresponds to K.

For an appropriate choice of coordinates, convex duality in projective space coincides with the usual concept of polarity for affine sets [START_REF] Boyd | Convex optimization[END_REF]. However, while many properties of affine polarity only apply for convex sets containing the origin (in particular, biduality: K •• = K), the projective framework is completely "homogeneous", and does not require similar conditions. Moreover, this setting is useful for dealing with general perspective projections. All the results in Section 4.3.3 can be understood geometrically and proven using purely synthetic arguments, by exploiting the characterization of duality as the set of "complementary" hyperplanes. More formal analytical proofs can be given using cones in R n+1 .

Convex cones and polarity

We now clarify some technical aspects on convex duality in the projective setting. We first recall some notions on convex cones. More details can be found in [START_REF] Boyd | Convex optimization[END_REF][START_REF] Schneider | Convex bodies: the Brunn-Minkowski theory[END_REF].

A set C in a real vector space V is a cone if x ∈ C implies λx ∈ C for all λ ≥ 0. A cone C is pointed if it contains no line, i.e., if x ∈ C and -x ∈ C then x = 0. Finally, a cone C is solid if it has a non-empty interior.

For any cone C ⊆ V , we can define the polar cone C • ⊆ V * (we denote by V * the space of linear functionals on V ):

We should note that many authors only consider V = R N and identify R N and (R N ) * by using the standard scalar product; for our purposes it is useful to keep the two spaces distinct. For any cone C, the polar cone C • is closed and convex. Moreover, the following properties hold (see [START_REF] Boyd | Convex optimization[END_REF]):

Consistency and Line Incidences

the second explicit approach, each family of lines L i has 2 k 2 -k-1 lines per set, which means 32 lines of each color class for k = 3 (see Figure 4.17).

Figure 4.17: Four families of lines L 1 , L 2 , L 3 , L 4 obtained using the explicit algebraic construction from [START_REF] Bukh | Consistent sets of lines with no colorful incidence[END_REF] for k = 3 (projecting a regular 4-grid to P 3 ). The four color classes are 3-consistent but have no colorful incidence. Each family L i has 32 lines.

Constructions with few lines. For k = 3, the smallest configuration in our constructive proof of Theorem 4.4.4 has 32 lines per color. In fact, if we restrict ourselves to small configurations, k-consistency does imply the existence of some colorful incidences.

Theorem 4.4.5. Let L = L 1 ∪ L 2 ∪ L 3 ∪ L 4 be a 3-consistent colored set of lines in P 3 with no colorful incidence and concurrent colors. If |L| < 24, then L is contained in a 2-plane.

We do not know whether the number 24 in this result is optimal. Interestingly, if we drop the assumption that all color classes are collinear then smaller examples exist: for example, we can show that there are exactly two configurations of 12 lines with four colors each of size 3 that are 3-consistent but have no colorful incidence. The two configurations in Figure 4.18 are very closely related to the classical projective configurations Desargues and Reye, respectively (see [START_REF] Hilbert | Geometry and the Imagination[END_REF]Chapter III]). The proof of Theorem 4.4.6 proceeds in two steps: first we show that there are only two possible combinatorial "incidence structures" so that each line meets exactly one line of the other three colors. Then we show that both these incidence structures can be realized geometrically in P 3 (as illustrated in Figure 4.18). The second step can be addressed

Consistency of Image Sets and Visual Hulls

be viewed as counterparts inside the Grassmannian of visual hulls in space. A different notion of convexity in Grassmannian was described by Goodman and Pollack in [START_REF] Jacob E Goodman | When is a set of lines in space convex[END_REF][START_REF] Jacob | Convexity on affine Grassmann manifolds[END_REF]: for them, however, convex sets of lines are defined as "duals" of families of convex sets in affine space. Although we will not investigate the geometric properties of these notions generalized duality and convexity, we point out that if S is a region bounded by a smooth surface Z, then the boundary of S λ in Gr(1, P 3 ) will be contained in the 3-fold of lines that are tangent to Z (the "Hurwitz 3-fold", see Chapter 5).

Visual Hulls: Practice

In this section we discuss some "practical" aspects related to consistency and visual hulls. In particular, we will show how (almost) consistent silhouettes can be used as implicit representations of a finite visual hull.

We begin by discussing an important issue that arises in practice, namely that consistency is an unstable feature and will not be exactly satisfied in the presence of noise (Section 4.6.1). We note that for dealing with this problem it is actually quite useful that we do not restrict ourselves to smooth surfaces. In Section 4.6.2, we explain how implicit image-based representations can be used to recover explicit 3D features, including a dense point cloud on the surface of the visual hull. Finally, we introduce in Section 4.6.3 a "hardness" function for points on the visual hull surface, that provides an upper bound the distance between a point and the actual (unknown) surface that is being observed. This function can be used to estimate how well the finite visual hull approximates the complete visual hull of the observed object.

Examples in this section are obtained using real data from Y.Furukawa. 7

How consistent are projections of the same object in practice?

In the previous sections, we introduced finite visual hulls from two slightly different perspectives: starting from a set S in space and a finite number of viewpoints (Definition 4.5.2), and also without specifying S, simply as an intersection of visual cones (Definition 4.2.2). These definitions are essentially equivalent, since for any set of cameras P 1 , . . . , P n with centers c 1 , . . . , c n , the finite visual hull FVH (S | c 1 , . . . , c n ) of an "object" S coincides with the intersection of the visual cones associated with the "silhouttes" T i = P i (S). The exact relationship between, objects, finite visual hulls, and silhouettes is illustrated in Figure 4.20. The important fact to note here is that there is a one-to-one correspondence between all possible finite visual hulls and all families of consistent silhouettes. This observation allows us to treat consistent silhouettes as representations of finite visual hulls.

Consistency of Image Sets and Visual Hulls

geometry, namely correspondence of two the epipolar tangents. Fixing instead e 1 , e 2 in the interiors of T 1 and T 2 , the homography of epipolar lines is unconstrained. Compatibility constraints for silhouettes in two views have actually been used recover camera parameters (directly) when the ambiguity of camera motion has only two degrees of freedom, namely for pure translation motion [START_REF] Sato | Affine reconstruction of curved surfaces from uncalibrated views of apparent contours[END_REF], or when a visible plane provides a homography between the views [START_REF] Cross | Parallax geometry of smooth surfaces in multiple views[END_REF]. For general camera motions, however, these conditions are difficult to use because they cannot be written in closed form for arbitrary image sets T 1 , T 2 . Explicit constraints can in principle be formulated when the image sets admit an algebraic description: see for example [START_REF] Kahl | Using conic correspondences in two images to estimate the epipolar geometry[END_REF], which essentially addresses compatibility for regions bounded by conics in P 2 .

In general, a better understanding of silhouette compatibility can be important for developing some theoretical foundations for silhouette-based SfM. Moreover, it would be interesting to understand whether a family of silhouettes alone (without camera parameters) can be used provide a unique representation of an object.

Visual hull approximations.

As pointed out several times throughout the chapter, the finite visual hull is the largest set that projects onto a given family of silhouettes, but is usually different from the actual object that generated the silhouettes. In particular, especially for a small number of views, the difference between the (unknown) shape that is being observed and the visual hull can be significant (see Figure 4.4). Understanding the relationship between an object and its reconstructed visual hull, as a function of the location of the viewpoints, is an interesting and, to the best of our knowledge, largely unexplored problem. This kind of study could be important for applications, for example in viewpoint planning strategies [START_REF] Konstantinos | A survey of sensor planning in computer vision[END_REF].

The simple hardness function we proposed in Section 4.6.3 can be seen as a step in this direction, but it does not provide any kind of guarantee regarding the topological correctness of the reconstruction. In this sense, the kind of analysis described for two views in Section 4.2.4 might be useful, since it relates the topological features of the silhouettes with those of the visual hull, and also of all 3D shapes with the correct projections. An important tool in the study of these issues could be the "line-dual" representation S λ in Gr(1, P 3 ) of an object S (see Section 4.5). This set represents the family of all lines that meet S, and captures visual features of S in a very direct way: for example, images of S are (appropriate) planar sections of S λ and, as we will see in Chapter 5, special singularities on the boundary S λ correspond to "visual events". On the other hand, to our knowledge there is not much literature on data structures for representing line-geometric data (see [START_REF] Pottmann | Computational line geometry[END_REF]). Developing effective representations S λ for real objects or 3D models will first require work on these topics.

Chapter 5

Changing Views on Curves and Surfaces

In this chapter we study visual events from an algebraic viewpoint. Given a sufficiently general curve or surface in 3-space, we consider the contour curve associated with a moving viewpoint. Qualitative changes in the contour occur when the viewpoint crosses the visual event surface. Our algebraic framework allows us to give a new characterization of the components of this ruled surface as the iterated singular loci of the higher associated hypersurfaces of the original curve or surface: these associated hypersurfaces include the dual surface in (P 3 ) * , and the Chow and Hurwitz hypersurfaces in Gr(1, P 3 ). We also present formulas, first derived by Salmon and Petitjean, for the degrees of these surfaces, and show how to compute exact representations for all visual events using algebraic methods.

The material of this chapter is based on the following publication 1 :

-Kathlén Kohn, Bernd Sturmfels, and Matthew Trager. "Changing Views on Curves and Surfaces". In: Acta Mathematica Vietnamica 43.1 (2018), pp. 1-29.

Preliminaries

case, the edge of regression E(S) is the vertex of the cone S. We illustrate the three curve encodings of a developable surface with a simple example.

Example 5.2.3. Let S be the surface of degree six in P 3 that is defined by the polynomial

This is the surface in [162, §3, eq. ( 9)]. We verify that S is developable by computing the ideal of its dual variety S ∨ ⊂ (P 3 ) * . This shows that S ∨ is a smooth rational quartic curve:

3)

The curve C in the Grassmannian Gr(1, P 3 ) that encodes the ruling of S = S C has the ideal This ideal defines the Fano scheme of S in P 5 . Finally, the edge of regression E(S) is the rational quartic curve {(s 4 : s 3 t : st 3 : t 4 )} in P 3 . The ideal of this curve equals

This curve has S as its tangential surface. All of these computations can also be reversed: this shows how various objects can serve as a representation of the surface S. ♦ Many of the ruled surfaces S C we shall encounter in later sections have the property that their defining polynomial f is extremely large and impossible to compute symbolically. In such cases, the curve C in Gr(1, P 3 ) ⊂ P 5 is more manageable, and we can often compute generators for its ideal I C . This encoding of the curve also enables us to carry out directly computations that involve the surface S C . For example, we may wish to compute the points of intersection of the ruled surface S C with a general line L: indeed, if a camera travels along L, the real intersection points with the visual event surfaces are precisely the visual events that can be observed from that camera motion.

To perform this computation, we fix two points [a 1 , a (5.6) Thus, even when f is unknown, we can compute its specialization (5.6) directly from I C .

Changing Views on Curves and Surfaces of order 0, 1 and 2 (together with the Plücker relation). The ideal of the trisecant curve D (X) is obtained by saturating by the ideal of the leading coeffcients of f 1 (z(t)) and f 2 (z(t)). This approach also generalizes to the case when X is not a complete intersection. Indeed, if X is defined by f 1 , . . . , f k , then we can use the same strategy to impose that

) and f k (z(t)) have three roots in common for any choice of s 1 , . . . , s k-1 .

We conclude this section with an example that illustrates the last row of Table 5.1.

Example 5.3.4. Let X be the smooth curve of degree 6 and genus 4 in P 3 defined by

The above method easily yields the equation of degree 18 for the tangential surface T (X):

This polynomial has 1094 terms. Its largest coefficient is underlined. A compact encoding is given by the 11 quadratic generators of the ideal of T (X). It is also easy to compute the quartic surface D(X), and it takes a little longer to compute the degree 54 curve E (X). ♦

Views of surfaces

We now turn to the visual events for a general surface X in projective 3-space P 3 . The six visual events associated with X were mentioned in Section 5.1.1 in items (L) and (M). These correspond to the following five irreducible components of V(X):

1. The flecnodal surface F(X) is the union of all lines L with contact of order 4 at a point of X. In other words, the equation of X restricted to L has a root of multiplicity 4.

2. The cusp crossing surface C(X) is the union of all lines L with contact of order 3 + 2 at two points of X, i.e., the equation for X ∩ L on L has a triple root and a double root.

3. The tritangent surface T (X) is the union of all lines L with contact of order 2+2+2 at three points of X, i.e., the equation for X ∩ L on L has three double roots.

4. The edge surface E(X) is the envelope of the bitangent planes of X. It is the union of all bitangent lines arising from these planes. This surface was denoted (X [2] ) ∨ in [START_REF] Ranestad | On the convex hull of a space curve[END_REF].

5. The parabolic surface P(X) is the envelope of all tangent planes that have contact of order 3 with X. It is the union of all principal tangents at parabolic points [139, §A.1.2].

Event Surfaces

The following theorem characterizes the expected degrees of these five surfaces.

Theorem 5.3.5. For a general surface X of degree d in P 3 , the visual event surface V(X) decomposes into the five components listed above. The degrees of these surfaces are: We first learned these degree formulas from Petitjean's article [START_REF] Sylvain Petitjean | The complexity and enumerative geometry of aspect graphs of smooth surfaces[END_REF]. Only much later did we notice that all five formulas already appeared in Salmon's 1882 book [START_REF] Salmon | A treatise on the analytic geometry of three dimensions[END_REF]. The precise pointers to both sources are given above.

Remark 5.3.6. In Section 5 of our paper [START_REF] Kohn | Changing Views on Curves and Surfaces[END_REF] we present new proofs for (some of) the degree formulas shown in Theorem 5.3.5. Our exposition there makes use of modern intersection theory [START_REF] Eisenbud | 3264 and all that: A second course in algebraic geometry[END_REF][START_REF] Fulton | Intersection theory[END_REF]. Our arguments are different from those of Petitjean [START_REF] Sylvain Petitjean | The complexity and enumerative geometry of aspect graphs of smooth surfaces[END_REF], that require instead a full understanding of Colley's multiple point theory [START_REF] Susan | Enumerating stationary multiple-points[END_REF][START_REF] Susan | Lines having specified contact with projective varieties[END_REF], and also leave out several steps. On the other hand, the derivations in Salmon's book [START_REF] Salmon | A treatise on the analytic geometry of three dimensions[END_REF] are inspiring, but lack the rigor of 20th century intersection theory. Unfortunately, although techniques we used are rather well-known in algebraic geometry (they can be learned from the textbook by Eisenbud and Harris [START_REF] Eisenbud | 3264 and all that: A second course in algebraic geometry[END_REF]), they also involve a lot of technical machinery, and are beyond the scope of this thesis. The interested reader is referred to [START_REF] Kohn | Changing Views on Curves and Surfaces[END_REF] for these proofs.

In this section we explain the geometry of the five irreducible event surfaces. We begin with possible formal argument which shows that the list of visual events in Theorem 5.3.5 is indeed exhaustive.

Proof sketch of the first part of Theorem 5.3.5. Let u be an image point in C z (X) and let L u := π -1 z (u) be the the pre-image line associated with u. Platonova [142, Main Theorem] characterizes all possible local singularities in the contour curve. In particular, she shows in the case L u ∈ PT(X) that u is not a simple cusp on its branch of the contour curve if and only if L u is either a flecnodal line or the unique principal tangent at a parabolic point. Hence, we only have to characterize the possible multi-local events. We noted above that tritangent lines and principal bitangent lines do not yield simple nodes in the contour. The final observation is that a line L u which has contact order of exactly 2 at exactly two distinct points of the surface X projects to a simple node u if and only if L u is not contained in a bitangent plane to X which is tangent at the same two points as L u .

The five ruled surfaces in Theorem 5.3.5 are encoded by the curves shown in the last row in Figure 5.3. The surfaces E(X) and P(X) are developable, and are the duals of

Computing Visual Events

This allows us to compute ideal ∆ λ (d) recursively. For example, let us assume that we have recovered the conditions for a polynomial to have a double root (λ = (2)) and a triple root (λ = (3)) for all degrees up to d -1. We consider the ideal I defined by S 0 (h d , h d ), S 1 (h d , h d ) and S 2 (h d , h d ) = 0, saturated by the leading coefficient c 0 . This ideal decomposes into three components corresponding to ∆ (4) (d), ∆ ( Table 5.3: The ideals ∆ (λ) (d) of multiple root loci relevant for visual events of surfaces.

Events for surfaces

In what follows we assume that the ideals ∆ λ (d) have been pre-computed for d ≤ 7. We use these ideals to compute the curves F (X), C (X) and T (X) in the Grassmannian Gr(1, P 3 ). The correspondence between the three multi-local events F, C, T and the three special partitions λ was seen on the right side in Figure 5.3, where λ was denoted by m.

Let f = f (x 1 , x 2 , x 3 , x 4 ) be the polynomial of degree d that defines the surface X. We parameterize points z(t) on the line in P 3 with Plücker coordinates p using a parameter t as in (5.1). We substitute z(t) into the polynomial f , and we regard f (z(t)) as a univariate polynomial in t, written as in (5.8). The coefficients c i are now homogeneous expressions of degree d in the Plücker coordinates q. At this point, we substitute these expressions c i (q) into the generators of ∆ λ (d). The result is an ideal in the Plücker coordinates q that defines the desired curve set-theoretically.

Example 5.4.1. Let d = 5 and consider the smooth quintic surface X defined by

We compute the curve F (X) in Gr(1, P 3 ) that represents the flecnodal surface. Its prime ideal has degree 260 and is generated by 10 sextics plus the Plücker quadric. This computation was done with the method above, starting from the ideal ∆ (4) [START_REF] Vladimir | Singularities of smooth mappings[END_REF]. ♦

Changing Views on Curves and Surfaces

Let us shift gears and focus on the local events P and E, seen on the left in Figure 5.3. We start with the parabolic surface P(X). Let X be defined by a polynomial f ∈ R[x 1 , x 2 , x 3 , x 4 ]. The ideal I(P ) of the parabolic curve P is defined by f and the determinant of the Hessian matrix H f . Consider the incidence variety of the parabolic curve and its tangent planes, that is {(x, T x (X)) | x ∈ P } ⊂ P 3 × (P 3 ) * . We compute the ideal of the incidence variety by adding the 2 × 2-minors of the matrix ∂f /∂x 1 ∂f /∂x 2 ∂f /∂x 3 ∂f /∂x 4 y 1 y 2 y 3 y 4

to I(P ). We then saturate the resulting ideal by x 1 , x 2 , x 3 , x 4 and afterwards eliminate x 1 , x 2 , x 3 , x 4 . This furnishes the ideal of the dual curve P p (X) in (P 3 ) * , which encodes the developable surface P(X).

For a general parabolic point x of X, the Hessian matrix H f (x) has rank three, so its nullspace represents a unique point p x in P 3 . We use the following simple fact to compute P (X). Lemma 5.4.2. For x ∈ P , the point p x lies on the unique principal tangent of X at x.

Proof. The relation x H f (x) p T

x = 0 holds. Euler's relation shows that x H f (x) is the gradient vector of f at x. Hence p x lies on the tangent plane to X at x. Furthermore, p x belongs to the principal tangent since p x H f (x) p T x is zero. Hence x and p x span the principal tangent.

The curve P (X) in Gr(1, P 3 ) is precisely the collection of the lines spanned by a general parabolic point x and the corresponding point p x from Lemma 5.4.2. This allows us to compute the ideal of P (X) in Plücker coordinates p 12 , p 13 , . . . , p 34 . First, we recover the ideal I of the incidence variety {(x, y) | x ∈ P, y ∈ ker H f (x)} by adding the four entries of the column vector H f (x) • y to the ideal I(P ) = f, det H f (x) . Secondly, we consider the map from the coordinate ring of the Grassmannian to the quotient ring of I that maps Plücker coordinates q ij to the 2 × 2-minors of ( x 1 x 2 x 3 x 4 y 1 y 2 y 3 y 4 ). The kernel of this ring map is the ideal of the curve P (X) in Gr(1, P 3 ).

Example 5.4.3. Let d = 3 and consider the Fermat cubic X defined by f = x 3 1 + x 3 2 + x 3 3 + x 3 4 . We can easily compute the ideal of the curve P (X) as described above, and from this we find the parabolic surface P(X). It decomposes into irreducible components of low degree:

This is one of the few cases where symbolic computation of the equation of P(X) is easy. ♦

Conclusions

Example 5.4.4. Let d = 3 and let X be the cubic surface defined by f = x 3 1 + x 3 2 + x 3 3 + x 3 4 + (x 1 + 2x 2 + 3x 3 + 4x 4 ) 3 . Using the method above, we easily compute the ideal of P (X). We next demonstrate how to find the visual events of type P as the camera moves along a line.

Consider the line with parametric representation z(t) = [t, 1, t -1, t + 1] in P 3 . Let Q be the skew-symmetric 4 × 4 matrix obtained from (5.2) by substituting to Plücker coordinates. We add the four coordinates of z(t) • Q to the ideal of P , we then saturate with respect to p 12 , . . . , p 34 , and thereafter we eliminate the unknowns p ij . The result is

This polynomial has 30 distinct complex roots. Precisely 8 of them are real, namely -1.0135860298526, -1.01135228952, -0.60097492358065, -0.3501467610081, -0.266855069244, -0.19167605663, -0.081116156693251, 0.37894374773077 . These 8 roots mark the visual events of type P as the viewpoint travels along the line z(t). ♦

We found that the computation the edge surface E(X) is more challenging than that of the parabolic surface P(X). Consider the case when X is a general quartic: the surface E(X) has in this case degree 160, and hence so does the curve E (X) in Gr(1, P 3 ). We succeeded in computing the ideal of this curve only for quartics X that are singular or very special. For instance, if X is the Fermat quartic then E(X) a surface of degree 80, with multiplicity 2. Since E(X) is developable, we could also try to use E p (X) as an encoding; unfortunately, the degree is then even higher. By Proposition 5.1 in [START_REF] Kohn | Changing Views on Curves and Surfaces[END_REF], in fact, the dual curve E p (X) has degree 480 in (P 3 ) * . The computation of edge surfaces E(X) definitely requires further research.

Conclusions

We have studied visual events of general algebraic curves and surfaces in projective space.

All the events can be characterized geometrically in terms of "associated hypersurfaces" in (P 3 ) * and Gr(1, P 3 ), which contain planes and lines that intersect the observed object in exceptional ways. Computing the visual events for general algebraic surfaces is challenging, because the degrees of the event surfaces increase very rapidly (see Table 5.

2).

We have discussed some strategies for addressing this issue: in particular, it is convenient to represent ruled and developable surfaces as curves in Gr(1, P 3 ) of (P 3 ) * , and to pre-compute ideals of multiple-point loci. Further work will be necessary for efficiently computing the edge surface. Moreover, we did not discuss here the computation of the Appendix A

Projective Geometry and Plücker Coordinates

This Appendix contains a brief introduction to projective geometry (Section A.1), and then collects some basics on Grassmannians and line geometry (Section A.2).

A.1 Projective Geometry

The following is a very brief introduction to projective geometry. This material (including proofs) can be found in any standard book on the subject, for example [START_REF] Scott | Projective geometry[END_REF][START_REF] Greenlees | Algebraic projective geometry[END_REF].

On the other hand, many classical references such as these use a different style and older language. Our organization follows the first part of the exercise book [START_REF] Fortuna | [END_REF], which is modern and concise.

Basic definitions.

Let K be a field, and let V be a finite-dimensional vector space over K. The projective space associated with V is defined as

where two vectors v and w satisfy the relation v ∼ w if and only if v = λw for some λ in

) is the standard projective space of dimension n over K. We will write [x] = [x 1 , . . . , x n+1 ] for the class of the vector x = (x 1 , . . . , x n+1 ) in K n+1 .

Projective Geometry and Plücker Coordinates

If W is a linear subspace of V , then P(W ) is a projective subspace (or linear subspace) of P(V ). Concretely, a projective subspace of dimension d corresponds to the set of vector lines within a linear vector space of dimension d + 1.

Incidence between subspaces. The intersection of two (or more) projective subspaces is a always a projective subspace. Indeed, if S 1 = P(W 1 ), S 2 = P(W 2 ) are projective subspaces of P(V ), then P(W 1 )∩P(W 2 ) = P(W 1 ∩W 2 ). We will often write the intersection of two projective subspaces using the meet (∧) operator, so that

We say that S 1 , S 2 are incident if S 1 ∧ S 2 = ∅, otherwise they are skew.

The union of two projective subspaces is not a projective subspace. However, we use the join (∨) operator, and write S 1 ∨ S 2 for the smallest projective subspace containing S 1 ∪ S 2 . For example, given two points P, Q in P 3 , we write P ∨ Q for the line they span.

Proposition A.1.1. Let S 1 = P(W 1 ), S 2 = P(W 2 ) be projective subspaces of P(V ).

In particular, if dim S 1 + dim S 2 ≥ dim P(V ), then S 1 and S 2 cannot be skew.

Projective Transformations. Linear maps between vector spaces induce transformations between the corresponding projective spaces. More precisely, if V and W are vector spaces, and ϕ : V → W is a linear map with nullspace K, then the associated projective map φ : P(V ) \ P(K) → P(W ) is defined by φ([v]) = [ϕ(v)] for all v not in K. In particular, if ϕ is a linear isomorphism (so K = {0}), then φ : P(V ) → P(W ) is called a projective isomorphism, or homography.

Homogeneous coordinates. Points

inside a projective space P(V ) of dimension n are said to be linearly independent if v 1 , . . . , v k are linearly independent vectors in V . Clearly, if P(V ) has dimension n, at most n + 1 points can be linearly independent. More generally, P 1 = [v 1 ], . . . , P k = [v k ] with k ≥ n + 1 are in general position if any subset of at most n + 1 points among them is linearly independent.

An ordered set R = {P 1 , . . . , P n+2 } of n + 2 points in general position in P(V ) is called a projective reference frame. The points P 1 , . . . , P n+1 are called fundamental points, while P n+2 is the unit point. A projective reference frame induces a system of homogeneous coordinates on P(V ) as follows. For any u ∈ V such that [u] = P n+2 , there exists a unique vector basis

) are the coordinates of v with respect to the basis B u , then we take [x 1 , . . . , x n+1 ] to be the homogeneous coordinates of P with respect to R.

It is easy to verify that these coordinates are determined by P and R up to a non-zero multiplicative coefficient.

Projective Geometry and Plücker Coordinates

Duality. Let V be a vector space and let V * be its dual, i.e., the vector space of all linear functionals ψ : V → K. The projective space P(V * ) is known as the dual projective space of P(V ), and is denoted by P(V ) * . The spaces P(V ) and P(V ) * are projectively isomorphic. If ψ is an element of V * , the corresponding projective point [ψ] in P(V ) * uniquely represents the hyperplane {[v] | ψ(v) = 0} in P(V ). Hence, points in P(V ) * can be naturally identified with hyperplanes in P(V ).

More concretely, let us consider the standard projective space P n , so V = K n+1 . In this case, it customary to use the usual scalar product to identify K and K * . With this identification, a hyperplane {a 1 x 1 + . . . + a n+1 x n+1 = 0} in P n is the "dual point" [a 1 , . . . , a n+1 ] in (P n ) * . Equivalently, the hyperplane {a 1 x 1 + . . . + a n+1 x n+1 = 0} is represented by [a 1 , . . . , a n+1 ] using "dual homogeneous coordinates" on (P n ) * : these are induced by the normalized basis on (K n+1 ) * that is dual to the standard vector basis on

If W is a vector subspace of K n , we write W ⊥ for the orthogonal complement of W in (K n+1 ) * :

If S = P(W ) is a projective subspace of P n , then we define the associated dual space S ∨ = P(W ⊥ ) in (P n ) * as the projectivization of the orthogonal complement W ⊥ . This represents the set of hyperplanes which contain all points in S. If S has dimension k in P n , then S ∨ has dimension n -k -1 in (P n ) * . The association S → S ∨ is known as the duality correspondence, and it satisfies S ∨∨ = S. 1 Moreover, duality reverses containments and the operations of join and meet:

A.2 Grassmannians and Line Geometry

In this section we introduce Grassmannians and Plücker coordinates, with particular focus on the Grassmannian of lines Gr(1, P 3 ). We should mention that a popular formalism for expressing geometric properties in projecive spaces is the Grassmann-Cayley algebra (see for example [START_REF] Sturmfels | Algorithms in invariant theory[END_REF]). We will not introduce this theory here, since it is not directly used in the thesis. We define Plücker coordinates for arbitrary Grassmannians, but describe the operations of "join" and "meet" only for points, lines, and planes in P 3 .

Projective Geometry and Plücker Coordinates

Theorem A.

A.2.2 Lines in projective space

In this thesis we are mostly interested in the Grassmannian Gr(1, P 3 ) of lines in P 3 . According to the general discussion above, this set can be embedded in P 5 . In particular, the line through the points x = [x 1 , x ), then the dual Plücker coordinates of are [q 12 , q 13 , q 14 , q 23 , q 24 , q 34 ] where q ij = a i b j -a j b i . As explained above, primal and dual coordinates satisfy p ij = σ(ijkl)q kl , where i, j, k, l are distinct indices and σ(ijkl) is the sign of the permutation (ijkl).

Note that if we identify P 3 and (P

Projective Varieties

This appendix is a very short (and incomplete) introduction to the algebraic geometry of projective varieties. Introductory courses on algebraic geometry usually first focus on affine varieties, however in this thesis we are only interested in varieties in projective space. We found that a useful reference for our purposes are the lecture notes by Arrondo [START_REF] Arrondo | Introduction to projective varieties[END_REF], which present many results on projective varieties in a very concrete fashion. Another excellent and quite accessible reference is the book by Beltrametti, Carletti, Gallarati, and Monti Brigadin [START_REF] Beltrametti | Lectures on curves, surfaces and projective varieties: a classical view of algebraic geometry[END_REF], that introduces algebraic geometry from a rigorous but very classical perspective ("classical" here is opposed to the abstract approach of modern algebraic geometry, usually based on the language of schemes). Another wellknown and slightly more advanced introductory textbook is by Harris [START_REF] Harris | Algebraic geometry: a first course[END_REF].

B.1 Ideals and Varieties

Classical algebraic geometry works over the field C of complex numbers. Throughout this chapter we let P n = P(C n+1 ). We also write C[x 1 , . . . , x n+1 ] for the ring of polynomials in x 1 , . . . , x n+1 with complex coefficients.

We recall that a homogeneous polynomial F in C[x 1 , . . . , x n+1 ] is a polynomial that has all terms of the same degree d. In that case, we have F (λa 1 , . . . , λa n+1 ) = λ d F (a 0 , . . . , a n+1 ) for any a 1 , . . . , a n+1 , λ in C. Any polynomial F of degree d can be decomposed as F = F 0 + . . . + F d where F k is homogeneous of degree k. In the following, if a = [a] is a projective point in P n , and F is a homogeneous polynomial C[x 1 , . . . , x n+1 ], we write F (a) = 0 if F evaluated at a = (a 1 , . . . , a n+1 ) is zero. This is well-defined because F is homogeneous. More generally, even if F is not homogeneous, we write F (a) = 0 to say that F k (a) = 0 for all F k in the homogeneous decomposition F = F 0 + . . . + F d .

Products of Projective Spaces

The degree of a variety X of dimension r is the maximum number d = deg X of points in the intersection of X with a linear space of codimension r (when this intersection is finite).

In general we expect the intersection of two varieties

Here the intersection between X 1 and X 2 is said to be generically transverse if each of the components of X 1 ∩ X 2 contains a point where the intersection is transverse (we refer to Section B.6 for a discussion of transversality).

B.4 Products of Projective Spaces

Many important algebraic varieties studied in this thesis are subsets of products of projective spaces. For dealing with these sets, it is convenient to introduce the Segre map.

For a product P n 1 × P n 2 , the Segre map σ is defined as In other words, the Segre map associates a pair ([x], [y]) to a point in P (n+1)(m+1)-1 whose coordinates are the pairwise products of the coordinates of [x] and [y]. This definition can be iterated for more general products P n 1 × . . . × P n k (and everything that follows can be extended to this case: here we restrict ourselves to the product of two projective spaces only for notational simplicity). The image of the Segre map σ(P n ×P m ) inside P (n+1)(m+1)-1 is known as the Segre variety.

A variety X in P n ×P m is by definition a set such that σ(X) is a variety in P (n+1)(m+1)-1 .

The following result characterizes varieties in P n × P m more directly. We recall that a polynomial F (x 1 , . . . , x n+1 , y 1 , . . . , y m+1 ) is bihomogeneous if it is independently homogeneous in the variables x i and in the variables y i (in general, with different degrees).

Proposition B.4.1.

A set X in P n ×P m is a variety if and only if it can be characterized as the zero-set of a family bihomogeneous polynomials.

The dimension of a variety X in P n × P m is the dimension σ(X) in P (n+1)(m+1)-1 . The notion of "degree" is however more subtle, and is best expressed using a multidegree polynomial. If X has dimension r and codimension p = n + m -r, then the multidegree is a homogeneous polynomial in

is the number of intersections of X with a product L 1 × L 2 of general linear spaces L 1 in P n and L 2 in P m of dimensions d and p -d respectively.

B.5 Morphisms and Rational Maps

We now introduce two types of maps between algebraic varieties. Roughly speaking, morphisms (or regular maps) are polynomial maps that can be evaluated at every point of the domain, while rational maps are polynomial maps that can only be evaluated on an open dense set of the domain. The exact definition requires some extra work because these maps may not have a single polynomial representation, but may need to be defined by "stitching" together different local representations.

More precisely, let X ⊂ P n be a variety. A morphism F : X → P m is a map such that for every x in X, there exists an open neighborhood x ∈ U ⊂ X and homogeneous polynomials f 1 , . . . , f m+1 of the same degree so that F (p) = [f 1 (p), . . . , f m+1 (p)] for all p in U . Note that the polynomials f 1 , . . . , f m+1 cannot simultaneously vanish on U . The image under a morphism of a projective (irreducible) set is a projective (irreducible) irreducible set.

is not defined at the point p = [0, 1, 1] on X. However, ϕ 1 can be extended to a morphism F : X → P 1 by setting F (x) = ϕ 1 (x) for x = p and F (p) = [START_REF] Abdeljaoued | Minors of Bezout matrices, subresultants and the parameterization of the degree of the polynomial greatest common divisor[END_REF]0]. Indeed, if we consider

then ϕ 1 (x) = ϕ 2 (x) for all x ∈ X excluding the points p and q = [0, -1, 1], and ϕ 2 (p) = [START_REF] Abdeljaoued | Minors of Bezout matrices, subresultants and the parameterization of the degree of the polynomial greatest common divisor[END_REF]0]. Hence, the morphism F is defined by stitching together the polynomial representations ϕ 1 and ϕ 2 . ♦

A rational map F from an algebraic variety X to P m is only defined on a open dense set W of X. It is usually written as F : X P m . The definition is analogous to that of a morphism but we restricted to points in W : for every x in W , there must exist an open neighborhood x ∈ U ⊂ W and homogeneous polynomials f 1 , . . . , f m+1 of the same degree so that F (p) = [f 1 (p), . . . , f m+1 (p)] for all p in W . When X = P n , one can show that every rational map F : P n P m admits a global representation, i.e., we can write F (p) = [f 1 (p), . . . , f m+1 (p)] whenever F is defined. An important example for this thesis is when F : P n P m is a linear projection (see Chapter 1). A rational map F : X Y is dominant if the closure of the image of F is Y . Two rational maps F : X Y and G : Y Z can be composed only if F is dominant.

Projective Varieties irreducible projective variety, known as the dual variety of X. This name comes from the following important result (see [START_REF] Israel M Gelfand | Discriminants, resultants, and multidimensional determinants[END_REF] for a proof).

Theorem B.8.1. For any projective variety X in P n , we have X ∨∨ = X. Moreover, if p is a smooth point of X, then H is tangent to X at p if and only if p ∨ , that is a hyperplane in (P n ) * , is tangent to X ∨ at H ∨ .

Independently of the dimension of X in P n , the dual variety is typically a hypersurface in (P n ) * . Indeed, one can show that if the dual variety X ∨ has codimension r + 1 then X is the union of r-dimensional linear projective spaces lying on X (i.e., it is ruled).

Even if X is smooth, the dual variety X ∨ usually has singularities. For example, if X is a smooth curve in P 2 , the dual curve X ∨ in (P 2 ) * will have cusp points that correspond to tangent lines at flex points of X, and it will have nodes that correspond to bitangent lines for X (i.e., lines that are tangent at two distinct points on X). The expected number of these dual singularities is given by the classical Plücker formulae, which we state here in the case of smooth curves.

Theorem B.8.2 (Plücker formulae). If X is a smooth curve in P 2 of degree d, the dual X ∨ is a curve in (P 2 ) * of degree d(d -1). If X is generic, then X ∨ contains κ ∨ = 3(d -2) cusps and ν ∨ = 1/2d(d -1)(d 2 -9) nodes. These correspond to κ ∨ flex points ν ∨ bitangents for X.

B.9 Computations with Sage

This thesis contains many algebraic results and examples that were derived using a computer algebra system. In particular, we used mainly Sage [START_REF]SageMath, the Sage Mathematics Software System[END_REF] or Macaulay2 [START_REF] Daniel | Macaulay2, a software system for research in algebraic geometry[END_REF]. These two softwares are different, in that Macaulay2 is intended exclusively for computations in algebraic geometry and commutative algebra, while Sage [START_REF]SageMath, the Sage Mathematics Software System[END_REF] is a "general purpose system", that can be used to study a wide variety of topics in mathematics, such as cal- After defining the vectors X and U representing variable points in P 3 and P 2 respectively, as well as the projection matrix M, we impose that U is the image of X by requiring that the matrix [M*X|U] has rank one (by defining the ideal I_incidence). We then define the ideal I_full which collects the constraints that X belongs to the original curve, and that U is the image of X. Having written all the necessary conditions, we need to eliminate the variables x1,x2,x3,x4. However, we must first remove from I_full a spurious component, corresponding to the values of X for which M*X is zero. For this reason, we saturate the ideal I_full with respect to the ideal defined by the coordinates of M*X, using command I_full.saturation(ideal((M*X).list()))[0] (the last argument 0 is required since the Sage command also returns a "saturation exponent"). After this step we can finally eliminate the variables x1,x2,x3,x4, and obtain the ideal J C that represents the projection of the given curve in P 2 . 2 For the curve and projection matrix used in the example above, the output is: