David F Fouhey

Vincent Delaitre

Abhinav Gupta

Alexei A Efros

Ivan Laptev

Josef Sivic

People Watching: Human Actions as a Cue for Single View Geometry

Fig. 1. What can human actions tell us about the 3D structure of a scene? Quite a lot, actually. Consider the people depicted on the left. They were detected in a time-lapse sequence in one of rooms A, B, or C.

Introduction

The human body is a powerful and versatile visual communication device. For example, pantomime artists can convey elaborate storylines completely nonverbally and without props, simply with body language. Indeed, body pose, gestures, facial expressions, and eye movements are all known to communicate a wealth of information about a person, including physical and mental state, intentions, reactions, etc. But more than that, observing a person can inform us about the surrounding environment with which the person interacts.

Consider the two people detections depicted in Figure 1. Can you tell which one of the three scenes these detections came from? Most people can easily see Fig. 2. Overview of the proposed approach. We propose the use of both appearance and human action cues for estimating single-view geometry. Given an input image or set of input images taken by a fixed camera (e.g., a time-lapse), we estimate human poses in each image (a), yielding a set of human-scene interactions (b), which we aggregate over time (for time-lapses). We use these to to infer functional surfaces (c) in the scene: sittable (red), walkable (blue). We simultaneously generate multiple room hypotheses (d) from appearance cues alone. We then select a final room hypothesis and infer the occupied space in the 3D scene using both appearance and human action cues. See all results on the project website.

that it is room A. Even though this is only a static image, the actions and poses of the disembodied figures reveal a lot about the geometric structure of the scene. The pose of the left figure reveals a horizontal surface right under its pelvis ending abruptly at its knees. The right figure's pose reveals a ground plane under its feet as well as a likely horizontal surface near the hand location. In both cases we observe a strong physical and functional coupling between people and the 3D geometry of the scene. In this work, we aim to exploit this coupling. This paper proposes to use human pose as a cue for 3D scene understanding. Given a set of one or more images from a static camera, the idea is to treat each person as an "active sensor," or probe that interacts with the environment and in so doing carves out the 3D free-space in the scene. We represent human poses following J.J. Gibson's notion of affordances [START_REF] Gibson | The ecological approach to visual perception[END_REF] -each pose is associated with the local geometry that permits or affords it. This way, multiple poses in space and time can jointly discover the underlying 3D structure of the scene.

In practice, of course, implementing this simple and elegant scenario would be problematic. First of all, the underlying assumption that the humans densely explore the entire observed 3D scene is not realistic. But more problematic is the need to recover high-quality 3D pose information for all people in an image. While several very promising 2D pose estimation approaches exist [START_REF] Andriluka | Pictorial structures revisited: People detection and articulated pose estimation[END_REF][START_REF] Yang | Articulated pose estimation using flexible mixtures of parts[END_REF][START_REF] Johnson | Learning effective human pose estimation from inaccurate annotation[END_REF], and it is possible to use anthropometric constraints to lift the poses into 3D [START_REF] Taylor | Reconstruction of articulated objects from point correspondences in a single image[END_REF], the accuracy of these methods is still too low to be used reliably.

As a result, in this paper we take a soft, hybrid approach. We first employ the single-view indoor reconstruction method of Hedau et al. [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF] which produces a number of possible 3D scene hypotheses. We then use existing human detection machinery to generate pose candidates. The crux of our algorithm is in simultaneously considering the appearance of the scene and perceived human actions in a robust way to produce the best 3D scene interpretation given all the available evidence. We evaluate our approach on both time-lapses and still images taken from the Internet, and demonstrate significant performance gains over state-of-the-art appearance-only methods.

Background

Our goal is to understand images in terms of 3D geometry and space. Traditional approaches in computer vision have focused on using correspondences and multiple view geometry for 3D reconstruction. While these methods have been successful, they are not applicable when only a single view of the scene is available. Since humans can infer scene structure from a single image, single-view reconstruction is a critical step towards vision systems with more human-like capabilities. Furthermore, 3D scene estimates from a single image not only provide a richer interpretation of the image but also improve performance of traditional tasks such as object detection [START_REF] Hoiem | Putting objects in perspective[END_REF][START_REF] Hedau | Thinking inside the box: Using appearance models and context based on room geometry[END_REF].

In recent years, there have been significant advances in using statistical approaches for single-view 3D image understanding [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF][START_REF] Yu | Inferring spatial layout from a single image via depth-ordered grouping[END_REF][START_REF] Lee | Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces[END_REF][START_REF] Hoiem | Geometric context from a single image[END_REF][START_REF] Wang | Discriminative learning with latent variables for cluttered indoor scene understanding[END_REF][START_REF] Gupta | Blocks world revisited: Image understanding using qualitative geometry and mechanics[END_REF][START_REF] Barinova | Geometric image parsing in man-made environments[END_REF][START_REF] Del Pero | Sampling bedrooms[END_REF][START_REF] Payet | Scene shape from texture of objects[END_REF][START_REF] Schwing | Efficient structured prediction for 3D indoor scene understanding[END_REF][START_REF] Del Pero | Bayesian geometric modeling of indoor scenes[END_REF]. To make progress on the extremely difficult and severely underconstrained problem of estimating scene geometry from a single image, these approaches impose domain specific constraints, mainly regarding the human-made nature of the scenes. However, although they assume a human-centric scene structure, each of these approaches treats humans as clutter rather than as a cue. This work aims to demonstrate that humans are not a nuisance, but rather another valuable source of constraints.

Other work on the interface between humans and image understanding has mostly focused on modeling these constraints at a semantic level [START_REF] Gupta | Objects in action: An approach for combining action understanding and object perception[END_REF][START_REF] Turek | Unsupervised learning of functional categories in video scenes[END_REF][START_REF] Delaitre | Learning person-object interactions for action recognition in still images[END_REF]. For example, drinking and cups are functionally related and therefore joint recognition of the two should improve performance. Semantic-level constraints have been also shown to improve object discovery and recognition [START_REF] Turek | Unsupervised learning of functional categories in video scenes[END_REF][START_REF] Prest | Weakly supervised learning of interactions between humans and objects[END_REF][START_REF] Gall | Functional categorization of objects using realtime markerless motion capture[END_REF], action recognition [START_REF] Gupta | Objects in action: An approach for combining action understanding and object perception[END_REF][START_REF] Delaitre | Learning person-object interactions for action recognition in still images[END_REF][START_REF] Kjellstrom | Simultaneous visual recognition of manipulation actions and manipulated objects[END_REF][START_REF] Desai | Discriminative models for static humanobject interactions[END_REF], and pose estimation [START_REF] Yao | Classifying actions and measuring action similarity by modeling the mutual context of objects and human poses[END_REF][START_REF] Gupta | Context and observation driven latent variable model for human pose estimation[END_REF].

In this paper we specifically focus on modeling relationships at a physical level between humans and 3D scene geometry. In this domain, most earlier work has focused on using geometry to infer human-centric information [START_REF] Grabner | What makes a chair a chair?[END_REF][START_REF] Gupta | From 3d scene geometry to human workspace[END_REF]. For instance, Gupta et al. [START_REF] Gupta | From 3d scene geometry to human workspace[END_REF] argued that functional questions such as "Where can I sit?" are more important than categorizing objects based on name, and used estimated 3D geometry in images to infer Gibsonian affordances [START_REF] Gibson | The ecological approach to visual perception[END_REF], or "opportunities for interaction" with the environment. Our work focuses on the inverse of the problem addressed in [START_REF] Grabner | What makes a chair a chair?[END_REF][START_REF] Gupta | From 3d scene geometry to human workspace[END_REF]: we want to observe human actors, infer their poses and then use the functional constraints from these poses to improve 3D scene understanding. Our goal is to harness the recent advances in person detection and pose estimation [2-4, 30, 31], and design a method to improve single-view indoor geometry estimation. Even though the building blocks of this work, human pose estimation [START_REF] Yang | Articulated pose estimation using flexible mixtures of parts[END_REF] and 3D image understanding [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF][START_REF] Lee | Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces[END_REF], are by no means perfect, we show that they can be robustly combined. We also emphasize our choice of the monocular case, which sets our work apart from earlier work on geometric reasoning using human silhouttes [START_REF] Guan | 3d occlusion inference from silhouette cues[END_REF] in multi-view setups. In single-view scenarios, the focus has been on coarse constraints from person tracks [START_REF] Krahnstoever | Bayesian autocalibration for surveillance[END_REF][START_REF] Rother | What can casual walkers tell us about the 3D scene[END_REF][START_REF] Schodl | Depth layers from occlusions[END_REF], whereas we focus on fine-grained physical and functional constraints using human actions and poses.

Overview

Our work is an attempt to marry human action recognition with 3D scene understanding. We have made a number of simplifying assumptions. We limit our focus to indoor scenes: they allow for interesting human-scene interactions and several approaches exist specifically for estimating indoor scene geometry [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF][START_REF] Lee | Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces[END_REF][START_REF] Hedau | Thinking inside the box: Using appearance models and context based on room geometry[END_REF]. We use a set of commonly observed physical actions: reaching, sitting, and walking to provide constraints on the free and occupied 3D space in the scene. To achieve this, we manually define surface constraints provided by each action, e.g., there should be a sittable horizontal surface at the knee height for the sitting action. We adopt a geometric representation that is consistent with recent methods for scene layout estimation [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF][START_REF] Lee | Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces[END_REF]. Specifically, we build upon the work of Hedau et al. [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF]: each scene is modeled in terms of the layout of the room (walls, floor, and ceiling) and the 3D layout of the objects. It is assumed that there are three principal directions in the 3D scene (Manhattan world [START_REF] Coughlan | The Manhattan world assumption: Regularities in scene statistics which enable bayesian inference[END_REF]) and therefore estimating a room involves fitting a parametric 3D box.

While temporal information can be useful for detecting human actions and imposing functional and geometrical constraints, in this work, we only deal with still images and time-lapse videos with no temporal continuity. Time-lapses are image sequences recorded at a low framerate, e.g., one frame a second. Such sequences are often shot with a static camera and show a variety of interactions with the scene while keeping the static scene elements fixed. People use time lapses to record and share summaries of events such as home parties or family gatherings. This type of data is ideal for our experiments since it has a high diversity of person-scene interactions. It also enables us to test our method on realistic data with non-staged activities in a variety of natural environments.

The overview of our approach is shown in Figure 2. First, we detect humans performing different actions in the image and use the inferred body poses to extract functional regions in the image such as sittable and reachable surfaces (Section 3). For time-lapses, we accumulate these detections over time for increased robustness. We then use these functional surface estimates to derive geometrical constraints on the scene. These constraints are combined with an existing indoor scene understanding method [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF] to predict the global 3D geometry of the room by selecting the best hypothesis from a set of possible hypotheses (Section 4.1). Once we have the global 3D geometry, we can use these human poses to reason about the free-space of the scene (Section 4.2).

Local Scene Constraints from People's Actions

Our goal is to predict functional image regions corresponding to walkable, sittable and reachable surfaces by analyzing human actions in the scene. We achieve this by detecting and localizing people performing three different actions (standing, sitting, reaching) and then using their pose to predict contact points with the surfaces in the scene. For time-lapses, contact points are aggregated over multiple frames to provide improved evidence for the functional image regions.

Given a person detected performing an action, we predict contacts with surfaces as follows: (i) for walkable surfaces we define a contact point as the mean location of the feet position, and use all three types of actions; (ii) for sittable surfaces, we define a contact point at the mean location of the hip joints, and consider only sitting actions; and (iii) for reachable surfaces, we define a contact point as the location of the hand further from the torso, and use only reaching actions. These surfaces are not mutually exclusive (e.g., beds are sittable and reachable). To increase robustness, we place a Gaussian at the contact points of each detection and weight the contribution of the pose by the classifier confidence. The standard deviation of each Gaussian is set to a fraction of the detection bounding box, 1/4 in X-and 1/40 in Y-direction, respectively. This yields probability maps h for the different types of functional image regions, as illustrated in Figures 2c and4c,d.

Our approach is agnostic to the particular method of pose detection; in this work, we use two complementary approaches. We build primarily on the articulated pose model of Yang and Ramanan [START_REF] Yang | Articulated pose estimation using flexible mixtures of parts[END_REF]. Here, we employ the model for detecting human action by training a separate model for each of the three actions. Additionally, we use the model of Felzenszwalb et al. [START_REF] Felzenszwalb | A discriminatively trained, multiscale, deformable part model[END_REF] for sitting and standing: the low variance of the relevant joints of these actions (e.g., feet for standing) enable us to accurately approximate poses by simply transferring a fixed pose. Since the sitting detector may also respond to standing people, we discriminate between different actions by jointly calibrating the detectors of each model with respect to each other by fitting a multinomial logistic regression

Space Carving Via Humans

In the previous section we discussed how we estimate human poses and functional regions such as sittable and walkable surfaces. Using the inferred human poses, we now ask: "What 3D scene geometry is consistent with these human poses and functional regions?" We build upon [START_REF] Gupta | From 3d scene geometry to human workspace[END_REF], and propose three constraints that human poses impose on 3D scene geometry: Containment: The volume occupied by a human should be inside the room.

Free space: The volume occupied by a human cannot intersect any objects in the room. For example, for a "standing pose," this constraint would mean that no voxels below 5ft can be occupied at standing locations. Support: There must be object surfaces in the scene which provide sufficient support so that the pose is physically stable. For example, for a "sitting" pose, there must exist a horizontal surface beneath the pelvis (such as a chair). This constraint can also be written in terms of the functional regions; for example, sittable regions must be supported by occupied voxels in the scene.

Our goal is to use these constraints from observed human poses to estimate room geometry and the occupied voxels in the scene. Estimating voxels occupied by the objects in the scene depends on the global 3D room layout as well as the free-space and support constraints. On the other hand, estimating 3D room layout is only dependent on the containment constraint and is independent of the free-space and support constraints. Therefore, we use a two-step process: in the first step, we estimate the global 3D room layout, represented by a 3D "box," using appearance cues and the containment constraints from human actors. In the second step, we use the estimated box-layout to estimate the occupied voxels in the scene. Here, we combine cues from scene appearance and human actors to carve out the 3D space of the scene.

Estimating Room Layout

Given an image and the set of observed human poses, we want to infer the global 3D geometry of the room. We build on the approach of Hedau et al. [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF] which first estimates three orthogonal vanishing points and then samples multiple room layout hypotheses that are aligned with the estimated three scene directions. The best hypothesis is selected using a learned scoring function based on global appearance cues, such as detected straight lines and classifier outputs for different surfaces (walls, floor, ceiling). However, estimating the room layout from a single view is a difficult problem and it is often almost impossible to select the right layout using appearance cues alone. We propose to further constrain the inference problem by using the containment constraint from human poses. This is achieved with a scoring function that uses appearance terms, as in [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF], and terms to evaluate to what degree the hypothesized room layout is coherent with observed human actors.

Given input image features x and the observed human actors H (represented by functional surface probability maps h), our goal is to find the best room layout hypothesis y * . We use the following scoring function to evaluate the coherence of image features and human poses with the hypothesized room layout y: f (x, H, y) = ψ(x, y) + φ(H, y) + ρ(y),

where ψ(x, y) measures the compatibility of the room layout configuration y with the estimated surface geometry computed using image appearance, φ(H, y) measures compatibility of human poses and room layout, and ρ(y) is a regularizing penalty term on the relative floor area that encourages smaller rooms.

As we build upon the code of Hedau et al., the first term, ψ(x, y) is the scoring function learned via Eqns. 3-4 of [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF]. The second term enforces the containment constraints and expands as

φ(H, y) = h∈H ϕ(ζ(h), y), (2)
where ζ(h) is the mapping of support surfaces onto the ground plane and ϕ measures the normalized overlap between the projection and floor in the hypothesized room layout. Intuitively, φ(H, y) enforces that both the human body and the objects it is interacting with should lie inside the room. We approximate ζ(h) by using the feet locations of detected actors, which produces accurate results for our action vocabulary. Finally, the term ρ(y) = -c•max(0, (A-M )/M ) imposes a penalty for excessive floor area A, measured with respect to the minimum floor area M out of the top three hypotheses; in our experiments, c = 1/8. We need this regularization term since φ(H, y) can only expand the room to satisfy the containment constraint.

Estimating Free Space in the Scene

Once we have estimated room layouts we now estimate the voxels occupied by objects. However, this is a difficult and ill-posed problem. Hedau et al. [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF] use an appearance based classifier to estimate pixels corresponding to objects in the scene. These pixels are then back-projected under the constraint that every occupied voxel must be supported. Lee et al. [START_REF] Lee | Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces[END_REF] and Gupta et al. [START_REF] Gupta | From 3d scene geometry to human workspace[END_REF] further constrain the problem with domain-specific cuboid object models and constraints such as "attachment to walls". We impose functional constraints: a human actor carves out the free space and support surfaces by interacting with the scene. The room layout and camera calibration gives a cuboidal 3D voxel map in which we estimate the free space. We first back project the clutter mask of Hedau et al. [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF], and then incorporate constraints from different human poses to further refine this occupied voxel map. Specifically, we backproject each functional image region h at its 3D height1 , yielding a horizontal slice inside the voxel map. This slice is then used to cast votes above and below in voxel-space: votes in favor of occupancy are cast in the voxels below; votes against occupancy are cast in the voxels above. The final score for occupancy of a particular voxel is a linear sum of these votes, weighed by the confidence of human pose detections; as the result is probabilistic, to produce a binary interpretation, we must threshold the results.

Experiments

We validate the proposed approach on a collection of consumer time-lapse videos of indoor scenes and a collection of indoor still images. Both datasets were collected from the Internet and depict challenging scenes capturing one or more people engaged in everyday activities interacting with the scene. The code and data for the experiments is available on the project webpage.

Baselines. For both time-lapses and single images, we compare our estimates of room geometry to a number of approaches. Our primary baseline is the appearance-only system of Hedau et al. [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF]. To provide context, we also include another baseline, in which we impose the box model on the approach of Lee et al. [START_REF] Lee | Geometric reasoning for single image structure recovery[END_REF]. Finally, to show that all methods are operating better than chance, we use location only to predict the pixel labels: after resizing all scenes to common dimensions, we use the majority label in the training images for each pixel.

We use the standard metric of per-pixel accuracy. We compare the estimated layout with a manual labeling of room geometry; note that since the camera is fixed in time-lapses, only a single annotation is needed.

Implementation details. We train detectors on example images using the Yang and Ramanan model for all three actions [START_REF] Yang | Articulated pose estimation using flexible mixtures of parts[END_REF] and the Felzenszwalb et al.

Input Image

Functional Regions Scene Geometry Fig. 5. Example time-lapse sequence results: given an input image, we use functional regions (walkable: blue; sittable: red; reachable: green) to constrain the room layout; having selected a layout, we can also infer a more fine-grained geometry of the room via functional reasoning. See all results on the project website.

(a) Appearances Only (Hedau et al ).

(b) Appearances + People (Our approach).

Fig. 6. Timelapse experiment: A comparison of (a) appearance only baseline [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF] with (b) our improved room layout estimates. In many cases, the baseline system selects small rooms due to high clutter. On the right, even though the room is not precisely a cuboid, our approach is able to produce a significantly better interpretation of the scene. [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF], indoor scenes from [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF], and a subset of Pascal 2008 classification training data. None of the three negative image sets contains people. On testing sequences, adaptive background subtraction is used to find foreground regions in each frame and remove false-positive detections on the background. We also use geometric filtering similar to [START_REF] Hoiem | Putting objects in perspective[END_REF] to remove detections that significantly violate the assumption of single ground plane.

Experiment 1: Understanding time-lapse sequences

This dataset contains 40 videos of indoor time-lapse sequences totaling more than 140,000 frames. The videos were collected from YouTube by using keywords such as "time-lapse," "living room," "dinner," "party," or "cleaning." We treat each sequence as a collection of still images of a particular scene. Most of the frames contain one or more people interacting with the scene or each other. Examples include: people sitting on beds or sofas; people walking and people reaching into drawers and on tables. On average each video has around 3500 frames and 1200, 1300 and 400 detections of standing, sitting and reaching action, respectively. Figure 5 shows the performance of our approach on a set of time-lapses. The second column shows the probabilistic estimates of "walkable", "sittable" and "reachable" surfaces in blue, red and green respectively. We use these functional region estimates to select the best room hypothesis and estimate the free space of the scene, which is shown in the third column. These results show that human actors provide lot of information about the scene as they interact with it. For example, in the first case, the far away couches and their corresponding sittable surfaces are hard to recognize, even for human observers. Because our approach observes human actors walking and sitting in those areas, it can easily infer the sittable surface. Similarly, in the second row, even though the scene is cluttered, human reaching actions help us to infer a horizontal surface on the left.

We also qualitatively compare the 3D room layout estimated by our approach to that of Hedau et al. [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF]. Figure 6 shows some examples of the relative performance; comparisons for all time-lapse sequences may be found on the project page. Quantitatively, as shown in Table 1, our method is able to consistently improve on the baseline, averaging a gain of 7.6% (bootstrapped 95% confidence Following [START_REF] Hedau | Recovering free space of indoor scenes from a single image[END_REF], we also quantitatively evaluate estimated free space. We obtain ground-truth by manually labeling the floor occupancy map in the estimated room. Compared to the appearance-only backprojected clutter labeling, our approach achieves a 15.1% average precision gain in estimating floor free space.

Experiment 2: Understanding single images

In the second experiment, we consider a dataset of 100 still images of indoor scenes. As existing work treats humans as clutter, previous data sets have deliberately excluded humans from their scenes; we therefore must gather a new dataset. The images were collected with Internet image search engines using keywords such as "living room," "eating," or "sitting," and in collections of pictures of celebrities and political figures. We emphasize that since our approach is general, it can be applied to the wealth of still images available on the Internet. Our results show that functional constraints from human actions provide strong evidence of 3D geometry even in a single image. Figure 7 shows few examples of our estimated room geometry as compared to Hedau et al. [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF]. Comparisons for all images may be found on the project website. Figure 8 shows examples of estimated 3D room geometry and the 3D occupied voxels. Quantitatively, we demonstrate 2.6% improvement (bootstrapped 95% confidence interval: 0.9%-4.8%) over all still images, as seen in Table 2, and our performance is as good or better than [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF] in 88% of cases. The gain over the baseline is lower in the still image case than in the time-lapse case; this is largely due to rooms in which functional reasoning does not significantly adjust the interpretation, leading to equivalent accuracy with and without people: in many cases, human actions cannot be exploited, even if the person detections are perfect (Table 2), e.g., if the room selected with appearance alone is correct or if all actors are contained within an inaccurate room.

Discussion

While recognizing actions and estimating poses for a given person is still a very challenging problem, we have shown that noisy pose detections can significantly improve estimates of scene geometry and 3D layout even in a single image. This suggest other ways of using statistically aggregated noisy pose estimates, for example, learning relations between human actions and semantic objects (beds, chairs, tables) in the scene [START_REF] Delaitre | Scene semantics from long-term observation of people[END_REF]. We expect further gains in accuracy of the proposed method when better pose estimators become available in future.

Input: Image

Output: Room

Fig. 3 .

 3 Fig. 3. Example action detection and pose estimation results. The predicted surface contact points are shown by ellipses: blue (walkable), red (sittable), green (reachable). Shown actions are: standing (1-2), sitting (3-5), and reaching (6-8).

Fig. 4 .

 4 Fig. 4. Predicting functional image regions. (a) An image from a time-lapse sequence. (b) Overlaid example person detections from different frames: standing (blue), reaching (green). (c,d) Probability maps of predicted locations for (c) walkable and (d) reachable surfaces. Note that the two functional surfaces overlap on the floor.

  Fig. 7. Still Image Experiments:The correct person in the correct place can very easily disambiguate complex scene interpretation problems. In the last example, although the vanishing points are inaccurate, we produce a more accurate intepretation.

Fig. 8 .Fig. 9 .

 89 Fig. 8. Still Image Experiments: Functional reasoning to detect sittable surfaces in still images.

Table 1 .

 1 Time-lapse experiment: Average pixel accuracy for geometry estimation on time-lapse sequences. Our method achieves significant gains; further, using humans alone produces competitive performance.LocationAppearance Only People Only Appearance + People Lee et al.Hedau et al.

	Overall	64.1%	70.4 %	74.9%	70.8%	82.5%
	model for sitting and standing. For the standing action, we use a subset of 196
	images from [3] containing standing people. For sitting and reaching, we collect
	and annotate 113 and 77 new images, respectively. All images are also flipped,
	doubling the training data size. As negative training data we use the INRIA out-
	door scenes

Table 2 .

 2 Single Image Experiment: Average pixel accuracy for geometry estimation on single images. With even a single pose, our method achieves significant gains. To demonstrate the value of cues from people, we show results only using human action cues to select room hypotheses; specifically, we use only our human action compatibility and room size terms, φ and ρ, to rank the hypotheses. Even with only people as cues, our system performs only 4.1% worse on average than Hedau et al. and equivalently to Lee et al.

		Location	Appearance Only	Appearance + People
		Lee et al. Hedau et al. Ours with Ground Truth Poses
	Overall	66.4%	71.3%	77.0%	79.6%	80.8%
	interval: 4.5% to 11.3%). Further, our performance is worse than the baseline
	in only 7.5% of cases.

Because our classes are fine-grained, we can use human dimensions for the heights of the surfaces: for reaching, it is waist height (3ft), and sitting, knee-height (1ft).

Acknowledgments: This work was supported by a NSF Graduate Research Fellowship to DF, and by ONR-MURI N000141010934, Quaero, OSEO, MSR-INRIA, EIT-ICT, and ERC grant Videoworld.


Vincent Delaitre

David F Fouhey

Ivan Laptev

Josef Sivic

Abhinav Gupta

Alexei A Efros

Scene semantics from long-term observation of people

Our everyday objects support various tasks and can be used by people for different purposes. While object classification is a widely studied topic in computer vision, recognition of object function, i.e., what people can do with an object and how they do it, is rarely addressed. In this paper we construct a functional object description with the aim to recognize objects by the way people interact with them. We describe scene objects (sofas, tables, chairs) by associated human poses and object appearance. Our model is learned discriminatively from automatically estimated body poses in many realistic scenes. In particular, we make use of time-lapse videos from YouTube providing a rich source of common human-object interactions and minimizing the effort of manual object annotation. We show how the models learned from human observations significantly improve object recognition and enable prediction of characteristic human poses in new scenes. Results are shown on a dataset of more than 400,000 frames obtained from 146 time-lapse videos of challenging and realistic indoor scenes.

Introduction

What are people expected to do with a Christmas tree just set up in a living room? Is it common to see a person sitting on a stove? Current computer vision methods provide no answers to such questions. Meanwhile, resolving these and many other questions by recognizing functional properties of objects and scenes would be highly relevant for addressing the tasks of abnormal event detection and predicting future events in image and video data.

Object functions can be derived from the known associations between object categories and human actions (the mediated perception of function approach [START_REF] Palmer | Vision science: photons to phenomenology[END_REF]), for example chair→sittable, window→openable. Actions such as sitting, however, can be realized in many different forms which can be characteristic for some objects but not for others, as illustrated in Figure 1. Moreover, some objects may not support the common function associated with their category: for example, windows in airplanes are usually not openable. These and numerous other examples suggest that the category-level association between objects and their functions is not likely to scale well to the very rich variety of the types and forms of person-object interactions. Instead, we argue that the functional descriptions of objects should be learned directly from observations of visual data. ? Fig. 1. Different ways of using objects. While all people depicted on the left are sitting, their sitting poses can be rather unambiguously associated with the objects on the right. In this paper we build on this observation and learn object descriptions in terms of characteristic body poses.

In this work we design object descriptions by learning associations between objects and spatially co-occurring human poses. To capture the rich variety of person-object interactions, we automatically detect people and estimate body poses in long-term observations of realistic indoor scenes using the state-of-theart method of [START_REF] Yang | Articulated pose estimation using flexible mixtures of parts[END_REF]. While reliable pose estimation is still a challenging problem, we circumvent the noise in pose estimation by observing many person interactions with the same instances of objects. For this purpose we use videos from hours-lasting events (parties, house cleaning) recorded with a static camera and summarized into time-lapses 1 . Static objects in time-lapses (e.g., sofas) can be readily associated with hundreds of co-occurring human poses spanning the typical interactions of people with these objects (see Figures 234). Equipped with this data, we construct statistical object descriptors which combine the signatures of object-specific body poses as well as the object's appearance. The model is learned discriminatively from many time-lapse videos of variety of scenes.

To summarize our contributions, we propose a new statistical model describing objects in terms of distributions of associated human poses. Notably, we do not require human poses to be annotated during training and learn the rich variety of person-object interactions automatically from long-term observations of people. Our functional object description generalizes across realistic and challenging scenes, provides significant improvements in object recognition and supports prediction of human poses in new scenes.

Background. Semantic object labeling and segmentation has been mainly considered for outdoor scenes, e.g. [START_REF] Kohli | Robust higher order potentials for enforcing label consistency[END_REF][START_REF] Shotton | Textonboost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation[END_REF]. For indoor scenes the focus has been on recovering spatial layout [START_REF] Hedau | Thinking inside the box: Using appearance models and context based on room geometry[END_REF][START_REF] Lee | Geometric reasoning for single image structure recovery[END_REF][START_REF] Fouhey | People watching: Human actions as a cue for single view geometry[END_REF], possibly since many indoor objects are often better defined by their function rather than appearance.

The interplay between people and objects has recently attracted significant attention. Interactions between people and semantic objects has been studied in still images with the focus on improving action recognition [START_REF] Gupta | Observing human-object interactions: Using spatial and functional compatibility for recognition[END_REF][START_REF] Delaitre | Learning person-object interactions for action recognition in still images[END_REF], object localization [START_REF] Gupta | Observing human-object interactions: Using spatial and functional compatibility for recognition[END_REF][START_REF] Desai | Discriminative models for static humanobject interactions[END_REF][START_REF] Stark | Functional object class detection based on learned affordance cues[END_REF] and discovery [START_REF] Prest | Weakly supervised learning of interactions between humans and objects[END_REF] as well as pose estimation [START_REF] Yao | Modeling mutual context of object and human pose in humanobject interaction activities[END_REF][START_REF] Yao | Classifying actions and measuring action similarity by modeling the mutual context of objects and human poses[END_REF]. In video, constraints between human actions and objects (e.g., drinking from a coffee cup) have been investigated in restricted laboratory setups [START_REF] Gupta | Observing human-object interactions: Using spatial and functional compatibility for recognition[END_REF][START_REF] Gall | Functional categorization of objects using realtime markerless motion capture[END_REF][START_REF] Kjellstrom | Simultaneous visual recognition of manipulation actions and manipulated objects[END_REF] or ego-centric scenarios [START_REF] Fathi | Learning to recognize objects in egocentric activities[END_REF]. In both still images and video the focus has been typically on small objects manipulated by hands (e.g., coffee cups, footballs, tennis rackets) rather than scene objects such as chairs, sofas or tables, which exhibit large intra-class variability. In addition, manual annotation of action categories [START_REF] Gupta | Observing human-object interactions: Using spatial and functional compatibility for recognition[END_REF][START_REF] Kjellstrom | Simultaneous visual recognition of manipulation actions and manipulated objects[END_REF] or human poses [START_REF] Yao | Modeling mutual context of object and human pose in humanobject interaction activities[END_REF] in the training data is often required and the models typically do not allow predicting poses in new scenes without people.

Functional scene descriptions have been developed for surveillance setups, e.g., [START_REF] Peursum | Combining image regions and human activity for indirect object recognition in indoor wide-angle views[END_REF][START_REF] Turek | Unsupervised learning of functional categories in video scenes[END_REF][START_REF] Wang | Learning semantic scene models by trajectory analysis[END_REF], but the models are usually designed for specific scene instances and use only coarse-level observations of object/person tracks [START_REF] Turek | Unsupervised learning of functional categories in video scenes[END_REF][START_REF] Wang | Learning semantic scene models by trajectory analysis[END_REF], or approximate person segments obtained from background subtraction [START_REF] Peursum | Combining image regions and human activity for indirect object recognition in indoor wide-angle views[END_REF]. In contrast, our method generalizes to new challenging scenes, and uses finer grain descriptors of estimated body configuration enabling discrimination between object classes such as sofas and chairs.

Recent attempts [START_REF] Grabner | What makes a chair a chair?[END_REF][START_REF] Gupta | From 3d scene geometry to human workspace[END_REF] have inferred functions or affordances [START_REF] Gibson | The ecological approach to visual perception[END_REF] from automatically obtained noisy 3D reconstructions of indoor scenes. These methods infer affordance based on the geometry and physical properties of the space. For example, they find places where a person can sit by fitting a 3D human skeleton in a particular pose at a particular location in the scene. While people can sit at many places, they tend to sit in sofas more often than on tables. Moreover, they may sit on sofas in a different way than on a floor or on a chair. In this work we aim to leverage these observations and focus on statistical affordances by learning typical human poses associated with each object.

In a similar setup to ours, Fouhey et al. [START_REF] Fouhey | People watching: Human actions as a cue for single view geometry[END_REF] have looked at people's actions as a cue for a coarse 3D box-like geometry of indoor scenes. Here we investigate the interplay between object function and object semantics, rather than scene geometry. In addition, in [START_REF] Fouhey | People watching: Human actions as a cue for single view geometry[END_REF] the geometric person-scene relations are designed manually. In this work, we learn semantic person-object interactions from data.

Method overview

In this section we give a brief overview of the proposed approach. Our main goal is to learn functional object descriptions from realistic observations of personobject interactions. To simplify the learning task, we assume input videos to contain static objects with fixed locations in each frame of the video. Annotation of such objects in the whole video can be simply done by outlining object boundary in one video frame as illustrated in Figure 2. Moreover, person interactions with static objects can be automatically recorded by detecting people in the spatial proximity of annotated objects.

We start by over-segmenting input scenes into super-pixels, which will form the candidate object regions (details given in Section 5). For each object region R we construct a descriptor vector h(R) to be used for subsequent learning and recognition. The particular novelty of our method is a new descriptor representing an object region by the temporal statistics h P (R) of co-occurring people (Section 3). This descriptor contains a distribution of human body poses and their relative location with respect to the object region. We also represent each object region by appearance features, denoted h A (R), and the absolute location in the frame, denoted h L (R), as described in Section 4.

R R X 1 X 2 X 3 X 4 SIFT 1 SIFT 2 SIFT 3 SIFT 4 Appearance distribution (BOF) Location distribution h P (R) h A (R) h L (R)           h(R)
Given descriptor vectors, one for each object region, containing statistics of characteristic poses, appearance and image locations, a linear support vector machine (SVM) classifier is learnt for each object class from the labelled training data in a discriminative manner. At test time, the same functional and appearance representation is extracted from candidate object regions of the testing video. Individual candidate object regions are then classified as belonging to one of the semantic object classes.

Modeling long-term person-object interactions

This section presents our model of the relationship between objects and surrounding people. We start by introducing a new representation describing an object by the statistics of co-occurring human poses. We then explain the details of the extraction and quantization of human poses in time-lapses.

Describing an object by a distribution of poses

We wish to characterize objects by the typical locations and poses of surrounding people. While 3D reasoning about people and scenes [START_REF] Gupta | From 3d scene geometry to human workspace[END_REF] has some advantages, reliable estimation of scene geometry and human poses in 3D is still an open problem. Moreover, deriving rich person-object co-occurrences from a single image is difficult due to the typically limited number of people in the scene and the noise of automatic human pose estimation. To circumvent these problems, we take advantage of the spatial co-occurrence of objects and people in the image plane. Moreover, we accumulate many human poses by observing scenes over an extended period of time.

In our setup we assume a static camera and consider larger objects such as sofas and tables which are less likely to change locations over time. We describe object region R in the image by the temporal statistics h P of co-occurring human poses. Each person detection d is represented by the locations of J(= 14) body joints, indexed by j, and the assignment q d k of d's pose to a vocabulary of K P discrete pose clusters; see Figure 3 and Sections 3.2-3.3 for details. To measure the co-occurrence of people and objects, we define a spatial grid of 9 cells c around each body joint j. We measure the overlap between the object region R and the grid cell B d j,c by the normalized area of their intersection I(B j,c , R) = |Bj,c∩R| |Bj,c| . We then accumulate overlaps from all person detections D in a given video and compute one entry h P k,j,c (R) of the histogram descriptor h P (R) for region R as

h P k,j,c (R) = d∈D I(B d j,c , R) 1 + exp(-3s d ) q d k , (1)
where k, j, and c index pose clusters, body joints and grid cells, respectively. The contribution of each person detection in ( 1) is weighted by the detection score s d . The values of q d k indicate the similarity of the person detection d with a pose cluster k. In the case of the hard assignment of d to the pose cluster k, q d k = 1 for k = k and q d k = 0 otherwise. In our experiments we found that better results can be obtained using soft pose assignment as described in the next section.

Building a vocabulary of poses

We represent object-specific human actions by a distribution of quantized human poses. To compute pose quantization, we build a vocabulary of poses from person detections in the training set by unsupervised clustering.

In order to build the pose vocabulary, we first convert each detection d in the training video into a 2J-dimensional pose vector x d by concatenating midpoint coordinates of all detected body joints. We center and normalize all pose vectors in the training videos and cluster them by fitting a Gaussian Mixture Model (GMM) with K P components via expectation maximization (EM). The components are initialized by the result of a K-means clustering and during fitting we constrain the covariances to be diagonal. The resulting mean vectors µ k , diagonal covariance matrices Σ k and weights π k for each pose cluster k = 1, • • • , K P form our vocabulary of poses (see Figure 4). A pose vector x d for a detection d can be described by a soft assignment to each of the µ k by computing the posterior probability vector q d , where

q d k = p(x d |µ k , Σ k )π k K P j=1 p(x d |µ j , Σ j )π j . ( 2
)

Person detection and pose estimation

We focus on detecting people in three body configurations common in indoor scenes: standing, sitting and reaching. We use the person detector from Yang and Ramanan [START_REF] Yang | Articulated pose estimation using flexible mixtures of parts[END_REF], which was shown to perform very well at both people detection and pose estimation and train three separate models, one for each body configuration. We found that training 3 separate models improved pose estimation performance over using a single generic pose estimator (Section 7).

The three detectors are run separately on all frames of each time-lapse video in a sliding window manner at multiple scales. As all our videos have fixed viewpoint, we use background subtraction (Section 7) to remove some false positive detections. Additional false positives can be removed via geometric filtering: we use the vanishing point estimation method proposed in [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF] to compute the horizon height y h . We then assume a linear relationship h p (y p ) = α(y p -y h ) between a person's height h p and the feet y-coordinate y p in the image [START_REF] Rodriguez | Density-aware person detection and tracking in crowds[END_REF], and learn the scaling coefficient α via RANSAC and robust least square fitting. We discard detections for which the difference between the detected person height and the expected person height is greater than a given threshold ǫ. Finally we normalize the output of the detectors by making the mean and standard deviation of the detection scores equal to 0 and 1 on training videos, respectively. The filtering and normalization is performed separately for each detector.

To obtain the final set of detections, we perform standard non-maxima suppression on the combined outputs of the three detectors in each frame: if bounding boxes of several person detections overlap (i.e., have intersection over union bigger than 0.3), the detection with the highest normalized response is kept. This leads to a set D i of confident person detections for the i th video. Each detection d ∈ D i is represented by an associated normalized score s d and an estimated limb-configuration consisting of J bounding boxes B d j , j = 1, • • • , J corresponding to J = 14 locations of body joints.

As our time-lapse videos are sparsely sampled in time, the reasoning about temporal evolution of human poses is not straightforward. We therefore currently discard any temporal information about detected people. Nevertheless, the temporal re-occurrence of characteristic body poses for particular objects is a very powerful cue which we exploit to (i) reduce the noise in pose estimation and (ii) to span the rich variety of person-object interactions.

Modeling appearance and location

In addition to the distribution of poses we also model the appearance and absolute position of image regions. We build on the orderless bag-of-features representation [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF] and describe the appearance of image regions by a distribution of visual words. We first densely extract SIFT descriptors [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] f ∈ F k from image patches B f of multiple sizes s k for k = 1, • • • , S for all training videos and quantize them into visual words by fitting a GMM with K A components. Each feature f is then soft-assigned to this vocabulary in the same manner as described in Eq. ( 2). This results in an assignment vector q f for each feature.

The K A -dimensional appearance histogram h A (R) for region R is computed as a weighted sum of assignment vectors q

f h A (R) = S k=1 f ∈F k s 2 k I(B f , R) q f , (3)
where s 2 k I(B f , R) is the number of pixels belonging to both object region R and feature patch B f .

Similar to [START_REF] Hoiem | Geometric context from a single image[END_REF], we also represent the absolute position of regions R within the video frame. This is achieved by spatially discretizing the video into a grid of m × n cells, resulting in a (m × n)-dimensional histogram h L (R) for each region R. Here the i th bin of h L (R) is simply the proportion of pixels of the i th cell of the grid falling into R.

Learning from long-term observations

We now detail how we obtain candidate object regions from multiple super-pixel segmentations and learn the model of person-object interactions. We then show how to recognize objects in testing videos and predict likely poses in new scenes.

Obtaining candidate object regions. As described in previous sections, we represent objects by accumulating statistics of human poses, image appearance and location at object regions R. Candidate object regions are obtained by oversegmenting video frames into super-pixels using the method and on-line implementation of [START_REF] Felzenszwalb | Efficient graph-based image segmentation[END_REF]. As individual video frames may contain many people occluding the objects in the scene, we represent each video using a single "background frame" containing (almost) no people (Section 7). Rather than relying on a single segmentation, we follow [START_REF] Hoiem | Geometric context from a single image[END_REF] and compute multiple overlapping segmentations by varying the parameters of the segmentation algorithm.

Learning object model. We train a classifier for each object class in a oneversus-all manner. The training data for each classifier is obtained by collecting all (potentially overlapping) super-pixels, R i for i = 1, • • • , N , from all training videos. For each region, we extract their corresponding pose, appearance and location histograms as described in Sections 3 and 4. The histograms are separately L 1 -normalized and concatenated into a single K-dimensional feature vector

x i = [ hP (R i ), hA (R i ), hL (R i )]
, where h denotes L 1 -normalized histogram

h. An object label y i is then assigned to each super-pixel based on the surface overlap with the provided ground truth object segmentation in the training videos. Using the surface overlap threshold of 34%, each super-pixel can be assigned up to two ground truth object labels. Finally we train a binary support vector machine (SVM) classifier with the Hellinger kernel for each object class using the labelled super-pixels as training data. The Hellinger kernel is efficiently implemented using the explicit feature map Φ(x i ) = x i /L 1 (x i ) and a linear classifier. Finally, the outputs of individual SVM classifiers are calibrated with respect to each other by fitting a multinomial regression model from the classifiers output to the super-pixel labels [START_REF] Hastie | The Elements of Statistical Learning[END_REF]. The output of the learning stage is a K-dimensional weight vector w y of the (calibrated) linear classifier for each object class y. At test time, multiple super-pixel segmentations are extracted from the background frame of the test video and the individual classifiers are applied to each super-pixel. This leads to a confidence measure for each label and super-pixel. The confidence of a single image pixel is then the mean of the confidences of all the super-pixels it belongs to.

Inferring probable pose. Here we wish to predict the most likely pose within a manually provided bounding box in an image, given an object layout (segmentation) of the scene. This is achieved by choosing the pose cluster, for which the sum of learnt object weights for all joints most agree with the given per-pixel object labels in the image. More formally, denoting w y (k, j, c) the weight learnt for label y, pose cluster k, joint j and grid cell c, we select the pose cluster k that maximizes the sum of per-pixel weights under each joint grid cell

B k j,c k = arg max k J j=1 9 c=1 pixels i∈B k j,c w yi (k, j, c), (4)
where y i is the label for pixel i.

Time-lapse dataset

We extend the dataset of [START_REF] Fouhey | People watching: Human actions as a cue for single view geometry[END_REF] to 146 time-lapse videos containing a total of around 400,000 frames. Each video sequence shows human actors interacting with an indoor scene over a period of time ranging from a few minutes to several hours. The captured events include parties, working in an office, cooking or roomcleaning. The videos were downloaded from YouTube by placing queries such as "time-lapse party". Search results were manually verified to contain only videos captured with a stationary camera and showing an indoor scene. All videos are sparsely sampled in time with limited temporal continuity between consecutive frames. The dataset represents a challenging uncontrolled setup, where people perform natural non-staged interactions with objects in a variety of real indoor scenes.

We manually annotated each video with ground truth segmentation masks of eight frequently occurring semantic object classes: 'Bed', 'Sofa/Armchair', 'Coffee Table ', 'Chair', 'Table ', 'Wardrobe/Cupboard', 'Christmas tree' and 'Other'. Similar to [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF], the 'Other' class contains various foreground room clutter such as clothes on the floor, or objects (e.g., lamps, bottles, or dishes) on tables. In addition to objects we also annotated three room background classes: 'Wall', 'Ceiling' and 'Floor'. As the camera and majority of the objects are static, we can collect hundreds or even thousands of realistic person-object interactions throughout the whole time-lapse sequence by providing a single object annotation per video. The dataset is divided into 5 splits of around 30 videos with approximately the same proportion of labels for different objects. The dataset including the annotations is available at http://www.di.ens.fr/willow/research/scenesemantics/.

Experiments

In this section we give the implementation details and then show results for (i) pose estimation (ii) semantic labeling of objects in time-lapse videos and (iii) predicting likely poses for new scenes.

Implementation details. The foreground/background segmentation in each video frame is estimated using a pixel-wise adaptive mixture of Gaussian with 5 components [START_REF] Staufer | Adaptive background mixture models for real-time tracking[END_REF] (with α = 0.01 and T = 0.2). We also compute a single "background image" for each video that contains no people by taking the median of background segments across all video frames. Person detections and human pose estimates in each frame are obtained using the method and code of [START_REF] Yang | Articulated pose estimation using flexible mixtures of parts[END_REF]. Detections in the background segments and with confidence smaller than -1.1 are removed. The threshold ǫ for the ground-plane based geometric filter [START_REF] Rodriguez | Density-aware person detection and tracking in crowds[END_REF] is set to 30%. Super-pixels for each video are generated using the code of [START_REF] Felzenszwalb | Efficient graph-based image segmentation[END_REF] with parameters σ ∈ {0.2, 0.3}, k = 80 and min = 600. SIFT features are extracted from patches of size s ∈ {8, 16, 32, 64} pixels, with 50% spatial overlap. To train the proposed model, we use 3 splits of the dataset (see section 6) to cross-validate the C parameter of the SVM and use the 4 th split to calibrate the outputs of the individual classifiers. The resulting model is tested on the 5 th split. This is repeated five times for the different test splits to obtain the mean and standard deviation of the classification performance.

Pose estimation. To evaluate person detection and pose estimation performance we have annotated poses of at least ten (randomly chosen) person occurrences in each video, resulting in 1606 pose annotations. Person (bounding box) detection performance is measured using the standard average precision (AP) and pose estimation performance is measured by the Percentage of Correct Parts (PCP) score among the detected people as proposed in [START_REF] Ferrari | Progressive search space reduction for human pose estimation[END_REF]. We first compare our individually trained pose estimators for each action (see section 3.3) with a single model trained on images from all 3 action classes. Both have a similar recall of around 52% but the individually trained models achieve an average PCP of 50% compared to 47% for the single model. We then evaluate the effect of the background subtraction and geometric filtering for person detection. The individually trained models achieve an AP of 33%, which is significantly improved by background subtraction (51%) and geometric filtering (56%).

Semantic labeling of objects. Semantic labeling performance is measured by pixel-wise precision-recall curve and average precision (AP) for each object. Table [START_REF] Palmer | Vision science: photons to phenomenology[END_REF] shows the average precision for different object and room background classes for different feature combinations of our method. Performance is compared to two baselines: the method of [START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF], trained on our data with semantic object annotations, and the deformable part model (DPM) of [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] trained over manually defined bounding boxes for each class. At test time, the DPM bounding boxes are converted to segmentation masks by assigning to each testing pixel the maximum score of any overlapping detection. Note that combining the proposed pose features with appearance (A+P) results in a significant improvement in overall performance, but further adding location features (A+L+P) brings little additional benefit, which suggests that spatial information in the scene is largely captured by the spatial relation to the human pose. The proposed method (A+L+P) also significantly outperforms both baselines. Example classification results for the proposed method are shown in Figure 5. Finally, learnt weights for different objects are visualized in Figure 6.

We have also evaluated our model on functional surface estimation. For training and testing, we have provided ground truth functional surface masks for the dataset of [START_REF] Fouhey | People watching: Human actions as a cue for single view geometry[END_REF]. Our model achieves AP of 76%, 25% and 44% for 'Walkable', 'Sittable' and 'Reachable' surfaces, respectively, averaging a gain of 13% compared to [START_REF] Fouhey | People watching: Human actions as a cue for single view geometry[END_REF], which could be attributed to the discriminative nature of our model.

Predicting poses in new scenes. Figure 7 shows qualitative results of predicting likely human poses in new scenes. Given a person bounding box and the manually labelled object regions, the most likely pose is predicted using Eq. ( 4). As can be seen, the automatically generated poses are consistent with object classes as well as with the scene geometry despite no explicit 3D reasoning is included in our model.

Discussion

We have proposed a statistical descriptor of person-object interactions and have demonstrated its benefits for recognizing objects and predicting human body poses in new scenes. Notably, our method requires very little annotation and relies on long-term observations of people in time-lapse videos. Given the mutual dependence of objects and human poses, the current method can be further extended to perform joint pose estimation and object recognition.

Fig. 2 .

 2 Fig. 2. Overview of the proposed person-based object description. Input scenes are oversegmented into super-pixels; each super-pixel (denoted R here) is described by the distribution of co-occurring human poses over time as well as by the appearance and location of the super-pixel in the image.

Fig. 3 .

 3 Fig. 3. Capturing person-object interactions. An object region R is described by a distribution (histogram) over poses k (left), joints j (middle) and cells c (right). The 3×3 grid of cells c is placed around each joint to capture the relative position of an object region R with respect to joint j. The pixel overlap between the grid cell c and the object region R weights the contribution of the j th joint and the k th pose cluster.

Fig. 4 .

 4 Fig. 4. Pose cluster and detection examples. Left: example cluster means from our pose vocabulary. Right: person detections in multiple frames of time-lapse videos assigned to the pose clusters on the left.

Fig. 5 .Fig. 6 .

 56 Fig. 5. Object soft segmentation. Scene background with no people (left). Object ground truth (middle). Mean probability map for inferred objects (right).

Fig. 7 .

 7 Fig. 7. Plausible poses prediction. The proposed model supports automatic prediction of plausible human poses in new scenes. This is achieved by selecting a pose cluster leading to the best agreement between the (manually provided) scene object layout and the object weights learned for each joint.

Table 1 .

 1 Average precision (AP) for baselines of Felzenszwalbet al.[START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF] and Hedauet al.[START_REF] Hedau | Recovering the spatial layout of cluttered rooms[END_REF] compared to four different settings of our method: appearance and location features only (A+L), person features only (P), appearance and person features (A+P), appearance, location and person features combined (A+L+P).

		DPM [33] [24] (A+L)	(P)	(A+P) (A+L+P)
	Wall	-	75±3.9 76±1.6 76±1.7 82±1.2 81±1.3
	Ceiling	-	47±20 53±8.0 52±7.4 69±6.7 69±6.6
	Floor	-	59±3.1 64±5.5 65±3.6 76±3.2 76±2.9
	Bed	31±20 12±7.2 14±5.0 21±5.8 27±13 26±13
	Sofa/Armchair	26±9.4 26±10 34±3.3 32±6.5 44±5.4 43±5.8
	Coffee Table	11±5.4 11±5.2 11±4.4 12±4.3 17±10 17±9.6
	Chair	9.5±3.9 6.3±2.8 8.3±2.7 5.8±1.4 11±5.4 12±5.9
	Table	15±6.4 18±3.8 17±3.9 16±7.1 22±6.2 22±6.4
	Wardrobe/Cupboard 27±10 27±8.2 28±6.4 22±1.1 36±7.4 36±7.2
	Christmas tree	50±3.3 55±12 72±1.8 20±6.0 76±6.2 77±5.5
	Other Object	12±6.4 11±1.2 7.9±1.9 13±4.2 16±8.3 16±8.2
	Average	23±1.8 31±2.0 35±2.4 30±1.7 43±4.4 43±4.3

Time-lapse http://en.wikipedia.org/wiki/Time-lapse_photography is a common media type used to summarize recordings of long events into short video clips by temporal subsampling. We use time-lapses widely available on public video sharing web-sites such as YouTube, which are typically sampled at one frame per 1-60 seconds.

Acknowledgments: This work was supported by a NSF Graduate Research Fellowship to DF, by ONR-MURI N000141010934, ONR Grant N000141010766, Quaero program funded by OSEO, MSR-INRIA, EIT-ICT and ERC grant Videoworld.