Keywords: Scene understanding, action prediction, interaction analysis, spatiotemporal visual representation, object detection in videos, deep convolutional neural networks, representation learning

If I have seen further it is by standing on the shoulders of giants.

-Isaac Newton

Experience life in all possible waysgood-bad, bitter-sweet, dark-light, summer-winter. Experience all the dualities. Don't be afraid of experience, because the more experience you have, the more mature you become.

Chapter 1 Introduction

Computer vision aims to automatically understand and interpret visual scenes around us. The field has come a long way since early attempts to address the problem [START_REF] Seymour | The summer vision project[END_REF]. Nowadays, many applications of computer vision have found their way out of research labs to numerous on-line services and consumer devices.

Vision technologies are gradually shaping a new future where people are relieved from tedious, repetitive tasks. Moreover, intelligent devices are now able to provide human-like assistances in our daily activities. For example, face recognition has matured into a reliable security technology on smart phones and computers, replacing the passcode. Driver assistance systems have evolved from simple warnings and speed control, to the so called "Level 3" where human intervention is only required in rare difficult situations. Intelligent systems, like Amazon Echo Look 1 , can now be placed in the living room not only to answer questions but also to understand emotions.

People are arguably most important objects in images and video. Indeed, about 35% of pixels in movies and YouTube videos as well as about 25% of pixels in photographs belong to people [START_REF] Laptev | Modeling and visual recognition of human actions and interactions[END_REF]. Not surprisingly, human-centric tasks usually serve as core components in many computer vision pipelines. Analyzing people in images and videos has many practical applications in security, entertainment, education and other domains. Airport security can benefit from face recognition to Person Detection is another human-centric computer vision task aiming to localize people in images and video. It often serves as a backbone for many other human analysis tasks, e.g. human verification, action recognition, behavior understanding, crowd counting and others. Given the needs of time-critical applications, the performance of person detectors is important both in terms of speed and accuracy [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF][START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF][START_REF] Girshick | Fast r-cnn[END_REF]. Similar to action prediction, the task can be addressed in the context of still images [START_REF] Girshick | Fast r-cnn[END_REF][START_REF] Vu | Context-aware CNNs for person head detection[END_REF][START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] and videos [Zhu et al., 2017b].

2. http://www.chinadaily.com.cn/a/201804/08/WS5ac9bfe9a3105cdcf6516b7b.html 3. https://www.theverge.com/2018/6/26/17479068/ai-guardman-security-camera-shoplifterjapan-automated-surveillance This thesis addresses action prediction and person detection in visual data. In the scope of these two problems, our study spans a range of topics from human intentions, interactions and appearance to visual recognition in the spatio-temporal domain. Below we identify the goals of the thesis in Section 1.1 and outline main contributions of the thesis in Section 1.2.

Goals

Our first goal is to study and leverage correlation between human intentions and static scenes. We formulate a prediction task and introduce a predictive model to anticipate human actions given natural scene images. Our second goal is to improve person detection by exploring spatial interactions of people in images and the spatiotemporal evolution of appearance in video. We also aim to improve the speed and accuracy of general object detection.

Action Prediction in Static Scenes. As illustrated in Figure 1-1, scenes are frequently correlated with typical actions. Action prediction in static scenes could therefore provide scene-specific priors when recognizing human actions. In this work, we want to explore and make use of the strong action-scene associations using statistical methods. Based on such associations, we address a new task of predicting human actions for images of static scenes. The success of this task is expected to improve general scene understanding, reasoning about affordance and action recognition. This work also enables potential applications such as affordance geo-localization.

Person Detection in Videos. Understanding complex visual scenarios like movies is a challenging task where person detection plays an important role. Standard detection approaches often perform local search on the object hypothesis space, generated by techniques like exhaustive sliding windows [Fleuret and [START_REF] Vu | together with probabilities for the five most likely actions, predicted manually by people (red) and by an automatic method (green). 1-2 Illustration of contextual models[END_REF] together with probabilities for the five most likely actions, predicted manually by people (red) and by an automatic method (green). 2014]. Recent advances in object detection are strongly related to the progress in learning deep visual representations, i.e. features, extracted from deep convolutional neural networks. Even equipped with powerful features, state-of-the-art detection methods are still behind human-level performance. We argue that a visual scene is not a random composition of objects, therefore independently recognizing objects while ignoring the whole context is sub-optimal. Indeed, there exists an underlying latent structure that bind things together to make the scene "look-real" under human perception. Such an argument becomes even stronger in the case of videos where the temporal continuity is also a critical recognition cue [START_REF] Ellis | A Source Book of Gestalt Psychology[END_REF].

We aim at modeling human dynamics in movies in both spatial and spatio-temporal domains. Detection in videos is the testbed for our models where we demonstrate performance improvement over state-of-the-art baselines. In the spatial domains, i.e. individual frames, we try to exploit human-specific cues, e.g. human interactions, as additional evidence for detecting people. In the spatio-temporal domain, i.e. videos, we design a framework that can efficiently make use of temporal consistency to improve detection performance.

In the first line of work, we exploit structural contexts to enhance person head detection on single frames. Two types of structural context are considered: holistic information and interactions between people. First, we observe that head locations can be predicted from other image parts. In particular, other parts of the human body and the rest of the scene constrain locations of heads in the video frame. In addition, interactions between people put constraints on the relative locations and appearance of heads. For example, heads appear differently while people are talking and kissing. We exploit such constraints by explicitly modeling relations between pairs of people. Figure 1-2 illustrates examples of the two contextual cues.

In the second line of work, we build a temporal model for changes of head appearance in consecutive video frames. Our model can be applied to a more generic case of detecting multiple object classes. Unlike images, video provides additional cues which can help to disambiguate the detection problem. For example, sequences with objects undergoing dynamic occlusions and rotations are structured and contain space-time patterns with object-specific properties, see Figure 123. In this work, we argue that dynamic changes of appearance provide discriminative information that can improve the detection performance. Our goal is to learn discriminative models for the temporal evolution of object appearance and to use such models for detection.

The learned spatio-temporal features are robust in difficult cases, for example, limited lighting, fast motions and occlusions.

In the third line of work, we learn video representations that incorporate multiscale information on longer time horizons and design practical frameworks that achieve accuracy, efficiency and predictive power. Analogous to the Gestalt principle of common fate [START_REF] Wertheimer | Laws of organization in perceptual forms[END_REF][START_REF] Ellis | A Source Book of Gestalt Psychology[END_REF], we hypothesize that temporal coherence by accounting for motion across frames allows learning powerful representations while achieving greater invariance to blur, lighting, pose and occlusions. Our goal is to design a memory-based causal framework that can encapsulate video representations over time. At each time step, memories in the past can be propagated and aggre-gated to enhance the current video feature, improving object detection in videos, see

Contributions and outline

The rest of the thesis is organized in seven chapters. In Chapter 2, we review related work. We then present prior methods and models that are used in this thesis in Chapter 3. The remaining chapters are split into two parts corresponding to our two main goals as follows:

Action Prediction in Static Scenes. In Chapter 4, we present the following three contributions. First, we introduce a new dataset with manual annotations of typical actions for 397 scene classes and use it to analyze action-scene correlations. Second, based on the discovered correlations, we demonstrate successful action prediction for images of static scenes. Finally, we propose a new task of geo-localized action prediction. We apply our method to geo-tagged images on the web and show encouraging results of searching maps for locations suitable for particular activities. This work corresponds to the publication presented in ECCV 2014 [START_REF] Vu | Predicting actions from static scenes[END_REF].

Person Detection in Videos.

-Chapter 5 presents our use of two contextual cues to improve person detection in videos. We build on a state-of-the-art CNN model for object detection and extend it to contextual reasoning. First, we leverage person-scene relations and propose a Global CNN model trained to predict positions and scales of heads directly from a full image. Second, we explicitly model pairwise relations among objects and train a Pairwise CNN model using a structured-output surrogate loss. To train and test our model, we introduce a new large dataset with 369, 846 human heads annotated in 224, 740 video frames from 21 movies. We show the importance of our large dataset for training and evaluate our method on the new and two existing datasets. The results demonstrate improvements of the proposed contextual CNN model compared to other recent baselines including R-CNN [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF]] on all three datasets. We also demonstrate a speed-up of object detection provided by our Global model. Our new dataset and the code are publicly available from the project web-page 4 . This work was presented at ICCV 2015 [START_REF] Vu | Context-aware CNNs for person head detection[END_REF]].

-To model the temporal evolution of appearance, we introduce space-time tubes corresponding to sequences of object bounding boxes in consecutive frames. In Chapter 6, we propose two CNN architectures for generating and classifying tubes, respectively. Our tube proposal network (TPN) first generates a large number of spatio-temporal tube proposals maximizing the objectness score and guaranteeing high recall for the ground truth object tubes. The Tube-CNN implements a tube-level object detector in the video using tube proposals as input.

Our tube models show notable improvement in comparison with frame-level detector baselines on HollywoodHeads, a large-scale person detection dataset.

We achieve comparable results with the winner of ImageNet VID challenge of object detection in videos [Kang et al., 2016a]. On the Youtube-Objects (YTO) dataset [START_REF] Prest | Learning object class detectors from weakly annotated video[END_REF], a weakly annotated dataset closely related to ImageNet VID dataset, our framework surpasses state-of-the-art localization performance by a large margin.

-In Chapter 7, we propose a novel network structure, termed MemNet, that encodes a memory of the feature representation, which is updated at every frame based on image observations and warped from one frame to the next to account for observer and scene motions. In order to encapsulate a long-term video representation, we introduce a hierarchical architecture, named ClockNet, operating on multiple temporal scales with minimal memory consumption. Compared to baseline methods on object detection in videos, our proposed methods achieve state-of-the-art results on ImageNet VID dataset, while running at faster speed.

In addition, our networks can propagate memories over time, which leads to the 4. http://www.di.ens.fr/willow/research/headdetection two potential applications: (i) fast object detection with two-stream architecture on multiple GPUs and (ii) feature anticipation.

Chapter 8 concludes the thesis and discusses some possible future works.

11

Chapter 2

Related Work

In this chapter we review previous works that is closely related to this thesis.

The chapter is comprised of two sections: Section 2.1 discusses some works on action recognition and action prediction; Section 2.2 gives a brief overview of person and object detection methods.

Action Recognition and Prediction

We give a brief overview of the literature related to our problem of action prediction in static scenes. The section is organized in two parts: action recognition and action prediction.

Action Recognition

There is a large body of research addressing human action recognition in videos.

Early methods mainly involve human body parts tracking [START_REF] Rohr | Towards model-based recognition of human movements in image sequences[END_REF] and human

motion analysis [START_REF] Bobick | The recognition of human movement using temporal templates[END_REF]Davis, 2001, Efros et al., 2003]. Follow up methods focused on statistical representations for action recognition. [START_REF] Laptev | On space-time interest points[END_REF] represented motion patterns with space-time local features. The idea is to localize spatio-temporal interest points corresponding to characteristic events. Using such interest points, Bagof-Words approach has been used to represent actions in the video. Schuldt et al.

Action Prediction.

Recognition of functional properties of objects and scenes is an interesting but less explored area of computer vision. Action prediction has been addressed in [START_REF] Kitani | Activity forecasting[END_REF] and Walker [START_REF] Walker | Patch to the future: Unsupervised visual prediction[END_REF], where the authors want to model future motion of people and cars using priors derived from the scene.

Person and Object Detection

Person is an important object class for many computer vision tasks. Indeed, early detection frameworks mostly focused on finding human faces or bodies [START_REF] Martin | The representation and matching of pictorial structures[END_REF][START_REF] Papageorgiou | A trainable system for object detection[END_REF][START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF][START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF]. Localizing people plays an important role in higher-level tasks such as behavior and interaction understanding. In this section, we review previous work on person and object detection in videos. We start by giving an overview of some early face detection methods. We continue by reviewing more general approaches that could detect human bodies as well as other object classes. Most of this section is dedicated to the related literature of object detection built upon deep convolutional neural networks, more details of which will be given in Chapter 3.

Face Detection

During the last decades, many methods have been focused on detecting human faces in images and videos. We review three categories of face detection methods: low-level features analysis, template matching and appearance feature learning.

Low-level feature analysis. Edge and color are two common low-level features used in face detection. Edge-detection-based approaches [START_REF] Govindaraju | Locating human faces in photographs[END_REF][START_REF] Saad | Human face segmentation and identification[END_REF]] try to assign labels, e.g. left/right sides, to edges then fit them to a face model using the golden ratio or elliptic hough transform. Edges were extracted by classical techniques such as Canny filters [START_REF] Canny | A computational approach to edge detection[END_REF]. When it comes to color, most approaches focus on localizing skin areas, which most-likely belongs to human parts like faces. On color spaces like HSV, YIQ and YCrCb, skin-color pixels are tightly clustered. Some methods simply set a threshold on color histogram to determine skincolor [START_REF] Sobottka | Extraction of facial regions and features using color and shape information[END_REF][START_REF] Cai | Detecting human faces in color images[END_REF][START_REF] Yoo | A fast algorithm for tracking human faces based on chromatic histograms[END_REF]. More complex methods use statistical models, e.g. mixture of Gaussians, to parameterize skin-color clusters [START_REF] Yang | A real-time face tracker[END_REF][START_REF] Stephen | Modelling facial colour and identity with gaussian mixtures[END_REF][START_REF] Oliver | Lafter: a real-time face and lips tracker with facial expression recognition[END_REF].

In general, low-level-feature-based detection methods are susceptible to illumination change and noise.

Template matching. In template-matching-based methods, a template, i.e. face pattern, is manually predefined [START_REF] Sakai | Line extraction and pattern detection in a photograph[END_REF][START_REF] Craw | Automatic extraction of face-features[END_REF], 1992]

Support Vector Machines for Image Classification

Image classification is a task of assigning input images with one or more labels from a given set of categories. A traditional approach to this problem includes a construction of image representation in terms of image features followed by a classifier.

In this section, we first outline two common pipelines for image representation: Bag of visual Words (BoW) and Fisher Vector (FV) in Section 3.1.1. We then revisit Support Vector Machines (SVM), one of the most common classification technique in Section 3.1.2.

Image Representation

A common image representation pipeline is composed of three steps: extracting local image descriptors, learning a visual dictionary and encoding descriptors. In the first step, local descriptors like SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF], CSIFT [Abdel-Hakim and Farag, 2006] and HOG [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] are extracted from the input image.

In practice, we often use a dimensionality reduction technique like PCA to reduce the size of those descriptive vectors. This step helps decreasing the computational complexity as well as removing noises. Bag of visual words (BoW) [Fei-Fei and AlexNet.

The network consists of 8 layers: 5 convolutional layers and 3 fully-connected layers. Figure 3-1 depicts the network architecture. AlexNet takes fixed size 224×224 images as inputs. Kernel sizes of the first and the second convolutional layers are 11x11 and 5x5 respectively. The rest 3 convolutional layers have kernels of size 3 × 3.

In total, the network has about 60 million parameters and 650, 000 neurons. We modified the original R-CNN by replacing the SVM classifier with a sub-network for classification, which is similarly done in Fast-RCNN and Faster-RCNN later.

Fast R-CNN

Fast R-CNN [START_REF] Girshick | Fast r-cnn[END_REF] is an upgrade of R-CNN. The two frameworks therefore share similar detection pipeline. Both are based on pre-generated region proposals. Also top convolutional layers of base networks are used for feature extraction.

However, instead of forwarding cropped image regions, in Fast R-CNN we extract appearance feature of the whole image by passing it through the stacked convolutional layers. Output features are termed as feature maps. Fast R-CNN speeds up R-CNN by a large magnitude with the idea of directly performing regional-pooling on feature maps. Regional-pooling is a spatial-aware pooling operation which take box proposals and feature maps as inputs. Given an input image coupled with a feature map, for each box proposal, we proportionally localized the corresponding feature region on the feature map then scale it to a pre-defined size, e.g. 5 × 5. The scaling step is done by dividing the feature region into equal-sized cells then performing max-pooling on each one. Effectively, outputs of the regional-pooling operation are fixed-size vectors.

In this work, the authors introduced the Region-Of-Interest (ROI) pooling layer to this purpose. The ROI-pooling is speed efficient as, for a given input image, it only requires one forward pass to extract the shared feature map for all 𝑁 region proposals, while in R-CNN 𝑁 forward passes are done. The same argument is also applied for the back-propagation phase on the feature extraction component.

The Fast-RCNN replaces the SVM classifier with a neural network, which not only takes away the high storage-demand but also unifies the whole detection pipeline.

Effectively, the model can be trained end-to-end given the precomputed Selective Search proposals. While R-CNN is trained with only one classification loss, Fast-RCNN has two learning branches: region classification and bounding box regression.

Moreover, the hard-negative mining step is replaced by a simple heuristic based on overlap ratios of region proposals and object ground-truths. Such modifications have been proven effective as Fast-RCNN performs better than R-CNN on the detection task. pothesis at every pixel location. 𝐾 is the number of pre-defined aspect ratios, e.g. Our environments, such as living rooms, cafés and offices, vary in terms of objects and geometry, but also in actions typically performed at these places (e.g., people typically work in offices and cook or eat in kitchens). In this chapter, our goal is to explore such correlation and to develop algorithms able to answer questions such as "What are typical actions for a given scene?", "Where is a good place to have a picnic?" or "Can I cycle along this path?". Automatic answers to such questions could be useful for several purposes. First, action prediction could provide scenespecific priors when recognizing human actions. For example, relaxing is common on beaches but not on motorways; cooking is common in kitchens but not in meeting rooms. Second, deviations from an expected set of actions could be used to identify abnormal activities. Moreover, automatic action prediction for geo-localized images could support the search of places suited for particular purposes.

SUN Action dataset

Dataset annotation. To analyze correlations over a wide range of scene categories and a rich set of actions, we gather the novel SUN Action dataset (short for "Scene UNderstanding -Action") with manual annotations of typical human actions for images of static scenes. We use scene images from the SUN dataset [START_REF] Vu | together with probabilities for the five most likely actions, predicted manually by people (red) and by an automatic method (green). 1-2 Illustration of contextual models[END_REF]. For each of the 397 well-sampled scene categories we collect free-form annotations of typical actions for the twenty "most typical" images in that category [START_REF] Ehinger | Estimating scene typicality from human ratings and image features[END_REF], for a total of 7940 images. Annotations were crowdsourced using Amazon Mechanical Turk (AMT) 1 . AMT workers were shown images of scenes and were asked to list between one and three words or short phrases for each scene describing an action that one would typically perform in a given scene. Scene category labels were not provided. All together we collected 137,558 responses: each image received The majority of images in the SUN Action dataset contain no people. We found this property to be important for collecting unbiased annotations of typical actions.

For a few images containing people we have observed a bias in action annotations towards actions depicted in the image. An example of such a bias is shown in Figure 4-2 illustrating two crosswalk scenes, one without people and one with a cycling person.

1. AMT workers gave consent (set by the MIT IRB) for each HIT they chose to perform. In the scene containing the cyclist, the predominant response was "bike", unlike other images in the crosswalk category.

Processing of action responses. Action responses were gathered in free-form natural language and require preprocessing for our further analysis. Many of responses contain nearly identical information but differ in grammatical structure, such as "read the book while on the flight" and "read a book". Our first pass of preprocessing converts responses into simplified action annotations by extracting verbs or verbnoun patterns from each response. This strategy reduces the response space while preserving the meaning. For example, responses like "read the book while on the flight" or "avoid eye contact with neighbours" are trimmed to "read book" and "avoid eye contact" respectively. We use the Stanford NLP toolbox [START_REF] Toutanova | Feature-rich part-of-speech tagging with a cyclic dependency network[END_REF] for part-of-speech tagging, stemming, and removal of stop words, and extract either verbs or verb-noun patterns from each response. Responses containing no verbs are removed. The words extracted in this stage of preprocessing are used as input to predict scene categories in Section 4.3.

For the action prediction task in Section 4.4 we manually group semantically similar action responses into action classes. To define action classes, we automatically extract 100 most frequent verb patterns, i.e. single verbs, verb+noun, etc., from action responses. Patterns with similar meaning are then manually merged yielding action labels, for example, "walk on grass" and "walk on sand" are merged into "walk".

We note that the automatic parsing of natural language into action categories is an open problem beyond our work. In particular, we separate scenes into 197 outdoor and 203 indoor categories and define corresponding 38 outdoor and 23 indoor action classes as listed in Figure 456.

Given the average of 17.3 action responses per image in our database and a potentially large number of typical actions for a scene, our per-image annotation is not exhaustive. To address this problem, we assume that instances of the same scene category share the same functional properties. We found this assumption to be valid in most cases in our database. We therefore assign the same action labels to all instances of a given scene category using the following label propagation

Analysis of scene-action correlation

Are different scene categories correlated with distinctive sets of actions? Scene categories are often defined by what a person would typically do there: for example, in an office one would typically work, whereas in a kitchen one would typically cook.

Indeed, most man-made scenes around us have been created to facilitate certain actions.

This section verifies and quantifies relations between actions and scenes. We demonstrate successful recognition of a large number of scene categories from associated actions descriptions. We further investigate the structure of action-scene correlations with a hierarchical clustering analysis.

Predicting scenes from actions

To verify the hypothesized correlation between scene categories and actions, we conduct two classification experiments using action annotations in the SUN Action dataset. We take inspiration from the field of text classification. In the SUN Action dataset, each image is associated with a collection of natural-language action descriptions. Classifying images based on a collection of associated responses is reminiscent of classifying documents based on their contents. However, there are two notable differences in our approach. First, the number of responses available per image (17.3 responses on average) is significantly lower compared to the number of words in a typical text document. Secondly, we wish to probe category membership using only a small collection of responses per image, to simulate asking a handful of people to provide a most typical action for the image and then performing classification based on the consensus of that set of responses.

Classification methods.

We classify images using two simple bag-of-words techniques -Nearest Centroid and Naive Bayes. We divide the images in each class into 10 folds for cross-validation. For example, "mountain, snowy" is placed in a category of its own according to the visual similarity, whereas it is grouped together with other outdoor places on the basis of action similarity.

and classes in the dataset, min 𝑤,𝑐 (Pr(𝑤|𝑐)), and use this probability to stand in as the class-conditional likelihood for unobserved words. We assume a uniform prior over scene categories, enabling maximum likelihood estimation: that is to say, bag-of-words queries are classified according to which class provides the largest class-conditional likelihood.

Results. due to the sharing of common actions. For example, scene categories "basement" and "attic" are both often annotated by actions "store" and "clean", while scene categories "river" and "lake" are frequently labeled with "swim" and "fishing".

Quantitative classification results of the two methods for the 33-category subset and all 397 scene classes are shown in

Visual Action Prediction

People can easily determine appropriate actions to perform in a given place. Are machines able to do the same thing? In Section 4.3 we have addressed the related problem of predicting scene categories from a set of associated actions. Here we turn to the problem of predicting typical actions for an image of a scene. We approach the problem of visual action prediction using standard image classification techniques in terms of local features and binary classifiers. To train image classifiers we use action labels derived from action annotations as described in Section 4.2. We predict actions separately for indoor and outdoor scenes.

We test two different schemes for action prediction. Under the first scheme (S1),

we train action classifiers directly from images using action labels only. Under the second scheme (S2), we first classify images into scene categories as an intermediate step and then assign action scores based on the obtained scores of scene classifiers. We assume that any particular test image belongs to one scene category only, therefore a score for an action 𝑎 in a given image is defined as a max scene score over scene categories 𝑆 associated with 𝑎.

Implementation Details

Image representation. We use mean Average Precision (mAP) as the performance measure. To get mAP, we first compute the area under precision-recall curve, or Average Precision (AP), for each class. Then mAP is determined as the mean of average precisions across all classes. We obtain best prediction mAP of 60.99% for outdoor actions and 52.09%

for indoor actions using combination of HOG_BOW, HOG_FV and CSIFT_FV kernels. Our result is significantly higher than the mAP at chance level, i.e. 6.32%

for outdoor action classes and 4.24% for indoor action classes. We also aggregate predicted action scores for test images and try to estimate the score contribution. Figure 4-5 shows some test images along with manual action annotations and automatic action predictions. For this visualization, we map SVM scores of test images to probabilities using Platt's sigmoid [START_REF] Platt | Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods[END_REF], with parameters estimated during the training phase. Even though the results are not perfect, we still observe a good match between distributions of annotated and predicted actions. Our predictors successfully give reasonable responses like "swim", "take a walk" and "relax" to a beach image, or "take picture", "pray" and "sightsee" to a temple 48 For more quantitative analysis, we now consider for geo-localizing actions on a map and answering map queries of the type "Where can I do X?". Results are derived from geo-localized scene images publicly available on the Internet. IGMA is the first attempt to automatically answer geo-localized action queries. Our strategy of predicting actions at a broad spatial scale using geo-localized images enables us to go beyond manual location-action labels: for example, one can "have a picnic" not only at a designated picnic area, but also in a grassy countryside field.

Collecting Qualitatively, our predictions have a high degree of correspondence the locations of skiing areas. Similarly, in Figure 4-9(left) we illustrate predictions for "hike" and "swim" in France. These results visually correspond to the sea-coast and mountain areas of France, confirming good geo-localization of actions. with associated actions, not all scenes within a scene category share the same action affordances in practice. One possible approach to this issue might be to subdivide scene categories according to more fine-grained functional affordances, e.g. separately identifying rivers where people can and cannot swim.

Conclusion

In We assume that the ground-truth labels ŷ𝑖 are available for all candidates in training images.

For each pair of nodes we choose an order based on the coordinates of corresponding bounding boxes: the left box is defined to be the first, the right one -the second.

Let ℰ denote the set of oriented pairs of candidates (set of edges). We cluster all edges based on relative locations and scales of bounding boxes 1 and denote the cluster index of edge (𝑖, 𝑗) ∈ ℰ by 𝑘 𝑖𝑗 ∈ {1, . . . , 𝐾}. The clustering step helps to group 1. To cluster edges we apply k-means algorithm with 𝐾 = 20 to a subset of oriented edges in training images. Edges in this subset connect object candidates with positive labels as well as any other candidates with high scores of the pre-trained Local model. For the clustering we use relative location features (horizontal and vertical displacements, ratio of sizes) converted to the log scale and normalized to have zero mean and unit standard deviation. pairs of heads with similar spatial arrangement and relative scale.

Inspired by [Desai et When the number of candidates is small, i.e. |𝒱| ≤ 20, both maximization problems of (5.4) can be solved exactly using exhaustive search. When the number of candidates becomes larger, the exhaustive search becomes too slow. In this case one can use the cascade of QPBO [START_REF] Kolmogorov | Minimizing non-submodular functions with graph cuts -a review[END_REF] and TRW-S [START_REF] Kolmogorov | Convergent tree-reweighted message passing for energy minimization[END_REF] methods to approximate 𝑠 𝑖 . Specifically, QPBO allows to quickly determine the optimal label for some candidates. On our dataset QPBO works surprisingly well, i.e., in many cases it is able to label all nodes. If some nodes are unlabeled by QPBO, one can apply the exhaustive search when the number of unlabelled nodes is at most 20 and TRW-S otherwise.

We have tried using 16 and 32 candidates per image. The exact inference is tractable only in the first case. In this work, we use 16 candidates per image as the large number of candidates did not improve performance on our validation set.

Training the model

We train parameters of our model by minimizing a structured surrogate loss using the stochastic gradient descent algorithm 2 . The algorithm for parameter update 2. As common in the deep learning literature we ignore the non-differentiability issues and assume that in practice we can always compute the gradient. Gradient of the structured loss. To optimize the structured loss w.r.t. the model parameters w, we need to compute the gradient of the objective w.r.t. model parameters. We can always achieve this goal using the back-propagation method under two assumptions: 1) the gradient can be back-propagated through the modules of the model, i.e. all the partial derivatives of 𝜙 𝐸 , 𝜙 𝑈 , 𝜙 𝑃 w.r.t. the input and the parameters can be computed; 2) the scores of the candidates (5.4) can be computed exactly.

To start the back-propagation procedure, we compute the gradient of structured loss w.r.t. potentials 𝜃 𝑈 𝑖 , 𝜃 𝑃 𝑖𝑗,𝑘 of the joint score function 𝑆. [Jaderberg et al., 2015a] have in details explained how to do this for the SSVM loss (5.5). Here we explain how to differentiate the loss (5.6). First, the gradient of the loss (5.6) w.r.t. the scores can be expressed as 𝑑ℓ 𝑑𝑠 𝑖 = (-1) ŷ𝑖 +1 𝑣 ′ (𝑠 𝑖 (-1) ŷ𝑖 +1 ), 𝑣 ′ (𝑡) = exp(-𝑡) 1 + exp(-𝑡) .

The gradient of the score (when existent) w.r.t. potentials can be computed exactly if we can compute all max-marginals exactly:

𝑑𝑠 𝑖 𝑑𝜃 𝑈 𝑝 = 𝑦 𝑖,1 𝑝 -𝑦 𝑖,0 𝑝 , 𝑑𝑠 𝑖 𝑑𝜃 𝑃 𝑝𝑞,𝑘 = (𝑦 𝑖,1 𝑝 𝑦 𝑖,1 𝑞 -𝑦 𝑖,0 𝑝 𝑦 𝑖,0 𝑞 )[𝑘 𝑖𝑗 = 𝑘]
where 𝑦 𝑖,𝑡 𝑞 is the 𝑞-th component of y (5.8)

Notice that all the derivatives of potentials w.r.t. parameters and features can be computed by propagating the gradient through networks 𝜙 𝑈 and 𝜙 𝑃 . Finally, propagation of the gradient (5.8) through 𝜙 𝐸 gives us the direction of the update for parameters w 𝐸 of the FE.

Implementation details

In this section we provide implementation details for our models, our approach to merging the models, and some technical information.

Local model

To train the Local model, we assign each candidate region to the positive (head)

or negative (background) class. For a given image, we make this assignment based on the intersection-over-union (IoU) overlap ratio 𝑜 of the candidate bounding box with the best matching ground-truth bounding box. Specifically, candidates with 𝑜 > 0.6 are labeled as positives and candidates with 𝑜 < 0.5 are labeled as negatives.

The remaining candidates are considered ambiguous and are not used at the training.

Following [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] we exploit the context padding. Each candidate is resized to 188 × 188 square patch which is extended with 18 pixels on each side filled from the original image. The input images of our CNN are of size 224 × 224. For each image, we form a training batch by sampling 64 proposals such that the balance between classes is roughly maintained.

We initialize parameters of the network using the ImageNet pre-trained network of [START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF]. We optimize the parameters of the network by minimizing the sum of independent log-losses with a stochastic gradient descent (SGD) algorithm with momentum 0.9 and weight decay 0.0005. We initialize the learning rate at 0.01, and decrease it several times by a factor of 10 after the validation error reaches saturation.

Global model

The Global model takes the whole image (isotropically rescaled and zero-padded to size 224 × 224) as the input and provides a vector of 284 numbers as the output.

Each element of the output vector is associated with a cell of our multi-scale grid.

For each cell we construct a target objective: 1 is the corresponding image patch has at least 0.3 IoU ratio with at least one ground-truth object bounding box. To train the Global model we optimize the sum of independent log-losses with an SGD algorithm. We initialize the model with the ImageNet pre-trained network [START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF]. The learning rate of SGD is set to 0.0001, momentum -to 0.9, weight decay -to 0.0005.

Pairwise model

The number of candidates from one image that our Pairwise model can process is quite limited due to the complexity of the inference procedure. To train the Pairwise model we assign each selected candidate a target binary label based on the maximum IoU ratio with the ground-truth bounding boxes (threshold 0.5). We form a training batch from 64 candidates coming from 4 different images.

The FE part of the model is initialized from the Local model. The weights of the UN and PN were initialized randomly using zero-mean Gaussians with standard deviation 0.01. The structured surrogate objective is optimized with and SGD with momentum 0.9, weight decay 0.000005, and learning rate 0.00001. We decreased the learning rate by a factor of 10 after 4 passes over the training data.

Combining models

Datasets

In this section we present our new head detection dataset, HollywoodHeads (HH), In summary, the training set of HollywoodHeads contains 216, 719 frames from 15 movies, the validation set contains 6, 719 frames from 3 movies and the test set contains 1, 302 frames from another set of 3 movies. Human heads with poor visibility (e.g., strong occlusions, low lighting conditions) were marked by the "difficult" flag and were excluded from the evaluation. The HollywoodHeads dataset is available from http://www.di.ens.fr/willow/research/headdetection. for training and 714 for testing. To evaluate our method using upper-body annotation, we have regressed head bounding boxes to upper-body bounding boxes [START_REF] Mathias | Face detection without bells and whistles[END_REF]. The parameters of regression were tuned on the TVHI training subset for each tested method.

TVHI dataset

Casablanca dataset

The Casablanca dataset [START_REF] Ren | Finding people in archive films through tracking[END_REF] contains 1, 466 frames from the movie "Casablanca".

The frames are annotated with head bounding boxes, however, the annotation of frontal heads is typically reduced to face bounding boxes and, therefore differs in the scale and aspect ratio from the HollywoodHeads annotation. Given some mistakes in the original annotation in [START_REF] Ren | Finding people in archive films through tracking[END_REF], we have added missing bounding boxes for the heads of all people in the foreground. Similar to the TVHI dataset, we have applied bounding-box regression to compensate dataset-specific annotation biases.

Experiments

This section presents our experimental results. First, we demonstrate the effect of different combinations of proposed models (Section 5. Note that our evaluation on the Casablanca dataset differs from [START_REF] Ren | Finding people in archive films through tracking[END_REF] due to the improved annotation and the use of VOC evaluation procedure. Animated results of our method are available from the project web-page http://www.di.ens. fr/willow/research/headdetection. In Table 5.2 we report values of AP produced by different models together with the train/test speed. We measure the speed as the number of image patches processed per second. For each model we choose the size of the training batch such that the training speed is maximal. In all cases it happens to be the maximum batch size that fits into the GPU memory. Experiments of this section were run on NVIDIA TITAN X with 12G RAM.

5. http://www.vlfeat.org/matconvnet/pretrained/

Size of the training set

In 5.3. We observe that the amount of the training data and, maybe more importantly, its diversity significantly helps the performance.

Complexity reduction with the Global model

Here we show that the Global model can suppress false candidates and reduce the computational complexity of R-CNN and our Local model at test time. We achieve this by transferring scores of the Global model detection proposals. We then filter out low-score candidates and thus reduce the number of candidates that have to pass through Local CNN. We evaluate the performance of detectors with different percentage of candidates remaining after the filtering.

Conclusion

In this chapter we have addressed the task of detecting people in still images. We proposed two context-aware CNN-based models. To train and evaluate our method, we have collected the new large-scale HollywoodHeads dataset consisting of movie frames and human head annotations. The combination of our context-aware models and the CNN-based local detector achieves state-of-the-art results on our dataset and the two existing human detection datasets, TVHI and Casablanca.

We believe that our context-aware models can be extended to tackle general object classes. In particular, the Microsoft COCO dataset [START_REF] Lin | Microsoft COCO: Common objects in context[END_REF]] contains many small object classes with implied spatial constraints. Another possible direction for future work is to take into account motion information to extend our methods to perform long-term tracking.

Chapter 6 Tube-CNN: Modeling temporal evolution of appearance for object detection in video

In the previous chapter, we have shown how the detection performance can benefit from contextual cues in static images, such as the global image context and the pairwise context capturing relations between objects. In this chapter we explore temporal cues and construct spatio-temporal models for object detection. Indeed,static images may sometimes lack information to identify the presence of an object. Video, on the other hand, provides additional cues which can help to disambiguate the detection problem. For example, sequences with objects undergoing dynamic occlusions and rotations are structured and contain space-time patterns with object-specific properties, see Figure 6-1. We argue that dynamic changes of appearance provide discriminative information that can boost object detection in difficult situations.

Object tracking in video can compensate some failures of object detectors by imposing temporal smoothness. Tracking, however, often assumes the temporal continuity of appearance and can be misleading when such assumptions break due to strong occlusions or fast motions. In contrast to tracking, our goal in this paper is to learn discriminative models for the temporal evolution of object appearance and to use such models for object detection. People are among the most frequent and difficult objects in images and video.

Changes in clothing, poses and hair-style as well as frequent occlusions and dynamic variations of appearance make person detection particularly challenging. In this work, we pay close attention to this object class. To span the variety of dynamic scenes and activities, we consider people in Hollywood movies and address detection of person heads as in Chapter 5. To train our models we use the recent HollywoodHeads dataset presented in Section 5.3 and extend it with the annotation of head tracks in 3, 8 hours of video from 21 different movies. We test our method on two datasets and report significant improvements compared to the state of the art. We also evaluate and demonstrate advantages of our method in difficult scenes with strong dynamic occlusions.

The following presentation in this chapter is organized as follows. Section 6.1 de- scribes in detail our Tube-CNN model. In Section 6.2 we present our two approaches to generate tube proposals and describe our Tube Proposal Network (TPN). Section 6.3 explains our experimental setups and demonstrates our results. Section 6.4 concludes the chapter.

Tube-CNN for object detection

We now describe the Tube-CNN model for detecting objects in video. Our model operates on short chunks of consecutive video frames. Instead of rectangular image regions, our elementary units are spatio-temporal tubes spanning several frames. In this section, we assume that a set of candidate tubes (tube proposals ) is provided and describe our Tube-CNN that classifies and refines them.

The number of all possible tubes is huge and does not allow to consider them exhaustively. In this work, we consider only linear tubes, i.e., tubes corresponding to uniform linear motion in the video. Such an approximation is reasonable only locally for small chunks of consecutive frames. In what follows, we always represent a linear tube by the two rectangles on its first and last frames. We study the effect of different tube lengths in Section 6.3.5. Tube classification. The classification starts with Tube-of-Interest (TOI) pooling on the feature volume. Within the TOI-pooling layer, we first map the coordinates of tube proposals to subvolumes of features. Next, we max-pool within each frame to obtain a fixed-size feature map. The last stage consists in the temporal aggregation of the maps coming from all the frames of a chunk. We have tried max-pooling, average pooling, and 1-dimensional convolution. Our conclusion is that temporal max-pooling works best (without significant difference although) and we use it in what follows.

Note, that if temporal aggregation is done by max or average pooling, the number of parameters in Tube-CNN does not depend on the chunk length 𝑇 , and the learned model can be readily applied to chunks of arbitrary length.

After the TOI-pooling, we obtain a feature map of a fixed size for each tube proposal. This map is fed into the final part of the classification network. For example, in the case of ResNet-101 pipeline [START_REF] He | Deep residual learning for image recognition[END_REF], it is the 4-th block. The tube classification branch ends with the cross-entropy loss.

1. https://github.com/BVLC/caffe/tree/master/models/bvlc_reference_caffenet Tube regression. Tube regression adjusts spatial positions of tube proposals to better localize the object. Tube regression consists of two networks for bounding box regression [START_REF] Girshick | Fast r-cnn[END_REF] at the two ends of the tube. Both branches of the tube regression start with the ROI pooling [START_REF] Girshick | Fast r-cnn[END_REF] on the corresponding frame (the start and end frames of the chunk). Tube regression branches of the network end with the smooth L1 loss [START_REF] Girshick | Fast r-cnn[END_REF].

Supervision

To train the Tube-CNN model, one needs a dataset with annotated object tracks.

Details of the datasets used for our training are given in Section 6.3.1.

At the training time, Tube-CNN model takes a chunk of 𝑇 consecutive frames and a set of tube proposals as inputs. Given the chunk, we find a set of ground-truth tracks passing through it. Each ground-truth track is approximated within a chunk by a linear ground-truth tube. We use these tubes to assign labels to tube proposals in this chunk.

Label assignment for a tube proposal is based on the tube overlap between the tube proposal and the ground-truth tubes. We define the tube overlap between two tubes as minimum of the spatial Intersection-over-Union (IoU) overlaps at their ends.

Each sampled tube proposal of a training chunk is assigned with a label: positive (class label of an object) and negative (background). Tube proposals having tube overlap 𝜃 𝐺𝑇 ≥ 0.5 with the best matched ground-truth are marked as positives. Tube proposals with 0.1 ≤ 𝜃 𝐺𝑇 < 0.5 are marked as negatives. Additional details on the training setup are given in Section 6.3.2.

Generating tube proposals

The Tube-CNN model uses tube proposals as input. Tube proposals are defined as the sequences of region proposals on consecutive video frames that hypothesize spatio-temporal positions of objects. To reduce the number of potential proposals, we consider only the ones corresponding to uniform linear motion. In this section, we introduce a CNN model that generates tube proposals given an input chunk of frames: Tube Proposal Network. We also propose a tracking-based baseline to build tube proposals on top of single-frame region proposals. Evaluation of different tube proposals is reported in Section 6.3.3.

Tube proposal network

We now present our Tube Proposal Network (TPN). This is a CNN model that can generate a small set of high quality class-agnostic tube proposals used for object detection.

Architecture. Tube Proposal Network is a fully convolutional network [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF], taking a chunk of consecutive frames as input and producing a set of tube proposals. Each tube proposal has an objectness score and a refined region location. We refer to all spatial positions of the feature volume as seed locations and the corresponding features as seed features. Each seed feature vector is associated with a set of 𝐾 reference tubes, which we call tube anchors. Tube anchors sharing the same center axis in the feature volume correspond to the same seed location as well as the same seed feature vector.

Each seed feature vector is then passed through an anchor network to produce two outputs per each of the 𝐾 tube anchors: the objectness score and parameters for the tube regression. In our setting, the anchor network consists of two fully-connected layers. Finally, we obtain the tube proposals by collecting the regressed locations of all the tube anchors. Scores of the tube proposals directly come from the objectness scores of the anchors.

To prune tube proposals, we apply Non Maximum Suppression (NMS) based on the objectness score. We modify the standard NMS to the case of tubes by replacing the spatial IoU overlap ratio with the tube overlap ratio.

Training. TPN and Tube-CNN have many common properties, thus, their training procedures are similar. Compared to Tube-CNN operating with tube proposals, TPN is trained on mini-batches of tube anchors.

Each seed feature vector is connected to 𝐾 tube anchors. Tube anchors have the same center axis with the receptive field of the seed features. In the related work [START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], anchors for region proposals correspond to several scales and aspect ratios. While adopting this setup to tubes, we have tried to add anchors with motion between the start and end regions, but it has not improved the recall while significantly increasing the number of anchors, thus complexity at the testing stage. We therefore use tube anchors, with spatial location fixed, but with varying scales and aspect ratios.

Tube proposals by tracking box proposals

We introduce a tracking-based method to produce tube proposals from region proposals (which we refer to as box proposals ) of single frames. Given a chunk of 𝑇 consecutive frames, we generate box proposals of the start frame by the Selective

Search method [START_REF] Jasper Rr Uijlings | Selective search for object recognition[END_REF]. For each of those box proposals, we hypothesize its corresponding positions on the end frame by using a tracker. Specifically, we use the Kanade-Lucas-Tomasi (KLT) tracker [START_REF] Shi | Good features to track[END_REF] to obtain point tracks between the start and the end frames of the chunk. Each box proposal is associated with a set of interior point tracks, corresponding to a set of movement directions.

Those directions are then clustered into 𝑁 𝑏 directional bins. We apply RANSAC on 𝑁 ℎ most populated bins to construct 𝑁 ℎ candidate temporal paths in the interval of 𝑇 frames 2 . Tube proposals are then constructed by linearly moving the box along all hypothesized temporal paths.

The above process could be built upon the output of any box proposal method.

To have a rough comparison with [Kang et ImageNet VID dataset. We also evaluate our method on the multi-class object detection benchmark. ImageNet VID [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF]] is a large-scale dataset for object detection providing complete annotation of 30 object classes in 1. Youtube-Objects dataset. Youtube-Objects (YTO) is a video dataset for object localization providing annotation for 10 object classes on a sparse set of key frames [START_REF] Prest | Learning object class detectors from weakly annotated video[END_REF]. The 10 object classes in YTO form a subset of 30 object classes in ImageNet VID. We use YTO for additional evaluation of the models pretrained on ImageNet VID.

Training details

Base networks. In our experiments, Tube-CNN is initialized either from Caf-feNet [START_REF] Girshick | Fast r-cnn[END_REF] or from Resnet-101 [START_REF] He | Deep residual learning for image recognition[END_REF]] models pre-trained on ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. The TPN model is based on the CaffeNet architecture.

CaffeNet. Tube-CNN initialized from CaffeNet has a CNN feature extractor block composed of the first 5 convolutional layers. The last fully-connected (FC) layers are used for tube classification and tube regression. The TOI pooling used for tube classification, and the ROI pooling used for tube regression are applied to the feature volume created by stacking 𝑐𝑜𝑛𝑣5 feature maps. As described in Section 6.1.1, our TOI pooling consists of: spatial max-pooling and temporal max-pooling. Similarly to [START_REF] Girshick | Fast r-cnn[END_REF], the spatial max-pooling produces fixed-size 6 × 6 output features.

On the other hand, temporal max-pooling collapses all values along temporal dimension into one max value. Therefore, for every input tube proposal, TOI pooling layer outputs a 256-channels feature map with a fixed spatio-temporal extent of 6 × 6 × 1. object aspect ratio in ImageNet VID. Figure 6-5 illustrates recall depending on the number of tube proposals. In all cases TPN achieves a significant reduction in the number of proposals by the cost of a minor drop in recall. This property can be used to significantly reduce computational complexity of detection at test time.

HollywoodHeads

Detection results

In this section we report results for object detection. Following the PASCAL VOC detection challenge [Everingham et with the analysis of recall in Section 6.3.3 confirms the advantage the proposed TPN scheme.

Tube-CNN performs slightly better on SS-Track proposals compared to TPN. We believe that Tube-CNN + TPN detection results could be further improved if using deeper networks for the TPN model.

Effect of the tube length. We restrict our method to tubes with linear motion to enable a tractable approach for generating tube proposals. We believe linear tubes might be sufficient to represent short time intervals, which is the main target in this work. In Table 6.6 we report results comparing performance of Tube-CNN on the HollywoodHeads dataset for different values of tube length 𝑇 (all models are based on CaffeNet without hard negative mining). We observe the best performance obtained for 𝑇 = 10.

Conclusion

We have addressed the task of object detection in video. To this end, we proposed two CNN models for generating and classifying object tube proposals respectively. Our tube classification model achieves state-of-the-art results on all four tested datasets for object detection in video. The unified framework of the two models is both accurate and computationally efficient at the running time. Our method particularly improves object detection in difficult situations with dynamic occlusions. We therefore believe it could be highly valuable when used as input for object tracking and other challenging scenes. Finally, we note that our contributions are architecture-independent. The speed, accuracy and predictive benefits of our online representation are available for any detection method on video inputs.

1. To put this accuracy gain in context for object detection, the gains from hard example mining [START_REF] Wang | A-Fast-RCNN: Hard positive generation via adversary for object detection[END_REF] or hard positive generation [START_REF] Shrivastava | Training region-based object detectors with online hard example mining[END_REF] are around 2% mAP. Then, the memory feature map is warped via bilinear sampling based on a learned displacement field. The detection output in every frame is computed from the current memory.

Feature propagation via memory networks

The goal of this work is to improve the feature representation for objects in videos, by leveraging temporal information and motion. Exploiting past frames can also help predictions in the current frame when occlusions or motion blur distorts image evidence, c.f. [Zhu et al., 2017b]. We propose to continuously aggregate and update features over time to provide a stable and powerful representation of the scene captured by the video, which is illustrated in Figure 7-2. In Sections 7.1.1 and 7.1.2, we describe our approach to learn long-term video representations where feature memory is accumulated and transformed over time to account for observer and scene motions.

Aggregating features over time

Given a single image ℐ ∈ R ℎ ℐ ×𝑤 ℐ ×3 , a convolutional neural network (CNN) with parameters 𝛩 ℱ first extracts a feature map ℱ ∈ R ℎ ℱ ×𝑤 ℱ ×𝑑 ℱ , where 𝑑 ℱ is the number of feature maps and we typically have ℎ ℱ = 1 16 ℎ ℐ and 𝑤 ℱ = 1 16 𝑤 ℐ . In the following, we show how these single image feature representations are effectively aggregated over time. While we use a single feature map per image for ease of presentation, note that we can make use of multiple feature maps at different resolutions to handle scale variations, which was shown to be useful in [Lin et al., 2017a, Yang et al., 2016].

Tracking features over time: In every frame 𝑡, we hold a feature map ℳ 𝑡 ∈ R ℎ ℱ ×𝑤 ℱ ×𝑑 ℱ that acts as a memory on the feature representation of the video. Since the scene is dynamic and the camera is moving, the same objects will appear at different locations of the image plane in frames 𝑡 -1 and 𝑡. In order for the memory of the past frame ℳ 𝑡-1 to benefit detection in the current frame 𝑡, ℳ 𝑡-1 needs to be transformed according to the scene dynamics. Similar to [Zhu et Updating with image evidence: After having transformed the memory to the current frame 𝑡, i.e., M𝑡 , we need to aggregate the newly available image evidence ℱ 𝑡 extracted by the feature CNN into the memory,

ℳ 𝑡 = 𝜓( M𝑡 , ℱ 𝑡 ) , (7.2)
which defines one step of the proposed MemNet. We implement (and experimentally evaluate) two variants of the aggregation function 𝜓(•). The first is a parameter-free combination that leads to exponential decay of memory over time,

𝜓( M, ℱ) := 1 2 (ℳ + ℱ) , (7.3)
and the second is a weighted combination of memory and image features, While the network architecture in theory allows gradients to flow over the memory warping module to learn a good feature propagation, it also opens a shortcut for minimizing the loss because image evidence is available at every frame. While for some tasks past information is truly essential for prediction in the present, for several tasks the image of the current frame already provides most evidence for a good prediction (or at least a signal to minimize the loss). To encourage the network to learn a good feature propagation module, we randomly drop image evidence with probability 0.8 states is simple, we did not observe any improvements for our particular applications with more complex architectures like LSTM [START_REF] Hochreiter | Long Short-Term Memory[END_REF] or GRU [START_REF] Cho | Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation[END_REF].

𝜓( M, ℱ) := 𝛼 ℳ • ℳ + 𝛼 ℱ • ℱ , (7.4) with 𝛼 ℳ , 𝛼 ℱ ∈ R ℎ ℱ ×𝑤 ℱ ×1 and 𝛼 ℳ + 𝛼 ℱ = 1.

Detection, Propagation and Anticipation in Videos

To demonstrate the benefits of propagating features over time with the proposed MemNet and ClockNet, we show its impact for three practical applications. cable to all of these detectors. In this paper, we choose R-FCN [START_REF] Dai | R-FCN: Object detection via regionbased fully convolutional networks[END_REF] because it has shown a good trade-off between the accuracy and speed and given its publicly available implementation. Here, we explore another option to speed-up a modern object detector. Instead of using a single model, we demonstrate how to exploit two models with complementary properties running simultaneously (but asynchronously) on two threads (two GPUs)

Object detection in videos

to achieve both speed and accuracy, using our feature propagation.

We run a fast detector, R-FCN-18 (i.e., R-FCN with ResNet-18 [START_REF] He | Deep residual learning for image recognition[END_REF]) in one thread and a slower but also stronger detector, ClockNet-101, in the other thread. R-FCN-18 runs at the required frame rate and can provide output for every frame, however at a lower quality than ClockNet-101 could do if no time requirements existed. The main problem with the strong object detector is that it will always have some delay (or latency) 𝛥 to produce an output. If 𝛥 is too large for a practical system, the strong detector is not usable. It is important to note that achieving a speed-up with two GPUs is not trivial in a real-time setting. For the offline case it is easy to distribute computation of different images on multiple GPUs. However, this is not an option for streaming data. In Section 7.3.3, we still empirically compare with two alternative baselines that also leverage two GPUs.

With our design, on the other hand, we can still leverage the strong features by making up for the delay via feature propagation. We compute the displacement field between frames at times 𝑡 + 𝛥 and 𝑡 and warp the strong features ℳ 101 𝑡 into the current frame 𝑡 + 𝛥, where the fast object detector has already computed features

ℱ 18
𝑡+𝛥 , see Figure 7-5. We boost the representational power of R-FCN-18 by combining the feature maps. Again, we take the average of both features (the dimensionality is the same), but more advanced aggregation schemes are possible. We experimentally evaluate this application in Section 7.3.3.

Anticipating features

In the previous application of Section 7.2.2, we deploy feature propagation over several frames, but the displacement fields are still computed from image evidence, similar to [Zhu et al., 2017a]. For a true visual anticipation, however, future images are not available.

We propose to extrapolate the displacement fields into future frames and use them to propagate the feature (or memory) maps. For demonstration, we use a simple extrapolation technique. Given the two previous displacement fields 𝒟 𝑡-1,𝑡-2 and 𝒟 𝑡,𝑡-1 , we compute the difference of aligned displacement vectors (with bilinear sampling), which gives us the acceleration of pixels. We then apply a constant acceleration motion model to each displacement vector and extrapolate for one or multiple frames.

Obviously, this extrapolation technique has limitations but it is sufficient for our demonstrations of feature anticipation. We analyze the quality of the anticipated features in Section 7.3.2 by measuring the object detection quality in future frames.

Experiments

Our experimental evaluation focuses on the performance of our feature propagation and aggregation methods for object detection in videos. In Sections 7. However, this variant also leads to a large performance drop.

In summary, Table 7. Effect of different sequence lenghts and temporal scales: We investigate the importance of the length of the sequences used for training MemNet. In the testing phase there is no limitation on the sequence length, but GPU memory is a hard constraint during training as gradients need to flow back through the whole sequence.

We can observe in Table 7. Table 7.2 demonstrates results of this ablation study. We first compare our weighting approach with the adaptive weight [Zhu et al., 2017b] and the recently proposed AdaScan [START_REF] Kar | Adascan: Adaptive scan pooling in deep convolutional neural networks for human action recognition in videos[END_REF]. Interestingly, we found that simply averaging features from the current frame and the warped memory gives on par results with more complex weighting schemes, which is also consistent with the results reported in Table 7.1.

Next, we investigate the impact of employing the memory module at different representation levels of the underlying feature extraction network, i.e., ResNet-50. We Propagation: In this experiment, we provide image information up to frame 𝑡 -𝛿 and then propagate features up to frame 𝑡. Note that displacement fields are still available but only for warping the memory; no image evidence in form of appearance features is available to the detector after frame 𝑡-𝛿. The propagated features are then used to compute the detections. We compare our model with a baseline that takes the baseline is more evolved and similar to our proposed design. However, instead of propagating features, (delayed) detections from the strong model are propagated to align with the faster detector ("BoxProp"), as in Section 7.3.2. As in the previous experiment, we observe that feature-level propagation is superior to propagating bounding boxes.

Conclusions

Our work demonstrates how to learn a long-term video representation that can effectively leverage past information. We rely on the concept of a memory that is being updated regularly with image evidence and is being warped over time for a proper spatio-temporal alignment of features. The proposed ClockNet is a temporal multi-scale architecture that can store information at multiple temporal scales. Our experimental evaluation illustrates benefits of introduced components and demonstrates applications in terms of: feature anticipation and a fast multi-threaded object detector.

In future work it will be interesting to investigate the benefits of our learned video representations to other tasks such as action recognition or semantic segmentation in videos. We also plan to further explore the utility of feature anticipation and postulate that learning representations in causal settings is beneficial for visual anticipation. Images are not random collections of pixels. We strongly believe that the latent structure of images and video should be more closely explored and used as a regularizer for feature learning. Other strong cues for self-supervised feature learning could include motion and sound, as these modalities are widely used by people to understand the world [START_REF] Ellis | A Source Book of Gestalt Psychology[END_REF]. This motivates self-supervised multi-modal learning algorithms that can exploit the underlying structure not only of images alone but in combination with other modalities. Domain Adaptation. There is a growing interest in using synthetic data to overcome the difficulty of collecting annotations. Althought the rendering engines are producing more and more realistic images, the underlying distributions of synthetic images and the real ones still differ. That raises a problem of adapting or transferring features and knowledges learned from synthetic data to real data. A direct use of such features often leads to poor performance on real datasets. An ambitious solution for domain adaptation is to learn a semantic embedding playing as a common protocol that can translate the semantic world, either real or synthetic. Such a system could then be deployed to solve practical problems such as reliable recognition of traffic scenes in various weather conditions.

Reliable Predictive Model. From a very young age, people possesses a natural gift of inferring others' intentions. Such an ability is important for social adaptation in an interconnected society. We believe that intelligent systems should be able to not only recognize but also anticipate human intentions and act accordingly to. This is an ambitious goal which needs advancement in multiple areas including recognition, end-to-end deep reinforcement learning and uncertainty deep modelling.

  Geman, Amit et al., 2004, Dalal and Triggs, 2005, Lampert et al., 2008, Dollár et al., 2010, Kokkinos, 2011, Felzenszwalb et al., 2010a] or heuristic object proposals [Uijlings et al., 2013, Krähenbühl and Koltun, 2014, Zitnick and Dollár, 2014, Cheng et al., 2014, Girshick et al.,

Figure 1 - 1 -

 11 Figure 1-1 -Illustration of scene-action correlation [Vu et al., 2014]: Images of scene classes sandbar and temple_east_asia from the SUN dataset [Xiao et al., 2010]

  (a) Predicting person head position and scale from holistic information. Columns from the left to the right correspond to the original image and the output of the context model at different resolutions. Red color corresponds to cells with high scores for the "head" class, blue color indicates cells with low scores for the "head" class. Local Pairwise Local Pairwise (b) Filtering head hypotheses with the pairwise interaction cues. Yellow boxes correspond to correct detections, red -to false positives. Pairwise links are plotted with either yellow (attractive) or red (repulsive).

Figure 1 -

 1 Figure 1-2 -Illustration of contextual models [Vu et al., 2015].

Figure 1 - 3 -

 13 Figure 1-3 -Space-time object tubes used in our work to capture evolution of object appearance over short sequences of video frames.

Figure 1 -

 1 Figure 1-4 -A schematic comparison between a typical per-frame method (left) and the memory-based video representation learning approach (right) for object detection in videos.

- 4 .

 4 Figure 1-4. Compared to per-frame features, our video representations show best detection improvement on frames degraded by fast motions.

[

  2004],[START_REF] Laptev | Learning realistic human actions from movies[END_REF] classified actions by applying Support Vector Machines (SVMs) on the occurrence histograms.[START_REF] Wang | Action Recognition by Dense Trajectories[END_REF] proposed an action recognition framework with dense trajectory descriptors. Feature points are first localized and then tracked with optical flow to densely produce point trajectories. Each trajectory is represented by descriptors, e.g. HOG, HOF and MOH, within its neighborhood space-time volume. Action recognition is performed with the standard bag-of-features approach.Deep convolutional neural networks have been applied for action recognition. Simonyan and Zisserman [2014] designed a two-stream architecture separately processing RGB images and optical flows. Late fusion is applied on the 𝐿 2 -normalized softmax outputs of the two streams. The network achieved comparable performance with state-of-the-art methods using "hand-crafted" features. Despite relatively small improvements, this work showed promising potential of CNNs for action recognition. More recently, Tran et al. [2015] introduces C3D, a 3D convolutional neural networks for action recognition. C3D architecture extends 2D CNNs to videos. The learned C3D features computed from RGB input have been used for video representation, followed by SVM for action classification. Varol et al. [2016] extended C3D to learn long-term video representation and confirmed the advantage of using optical flows for human action recognition. Like how CNN models for recognition tasks on images benefit from the pretraining phase on the ImageNet dataset, CNN models for videos considerably benefit in pretraining on big datasets such as Sport-1M [Karpathy et al., 2014] and Kinetics [Kay et al., 2017]. The "Two-Stream Inflated 3D ConvNets" (I3D) extends state-of-the-art architectures on image classification to handle spatiotemporal 3D information in videos. I3D models pretrained on the Kinetics dataset and finetuned on HMDB-51 [Kuehne et al., 2011] and UCF-101 [Soomro et al., 2012] datasets achieve state-of-the-art performance on the both action recognition benchmarks. In general, training CNNs for videos is a challenging task due to the difficulty to collect data annotation and high memory consumption of the deep networks. Action recognition in stills images received less attention compared to videos. The work of Ikizler et al. [2008] was one of the first attempts to recognize actions in static images using human poses. The authors argued that poses often characterize actions; so one can extract and classify poses to derive action labels of images. Yao and Fei-Fei [2010a,b] have shown how an explicit model of human-object interaction can also benefit action recognition in images. Human-object interactions are represented by stochastic AND/OR structures. Training is done with a modified Apriori mining algorithm. Delaitre et al. [2010] classified Bag-of-Features representations to action categories. In that work, the authors also explore the use of latent SVM (LSVM) to learn deformable part-based representations of actions. The combination of the two types of action representation showed significant improvement in action recognition performance in still images.

Figure 3 - 1 -Figure 3

 313 Figure 3-1 -AlexNet architecture for image classification task: 5 successive convolutional layers followed by 3 fully connected layers [Krizhevsky et al., 2012].

Figure 3 - 3 -

 33 Figure 3-3 -A comparison between VGG-19 and ResNet-34 with respect to the depth [He et al., 2016].

3. 2

 2 Figure 3-4 -R-CNN object detection pipeline [Girshick et al., 2014].

Figure 3 -

 3 5 depicts the Fast-RCNN object detection pipeline. In this thesis, Fast-RCNN serves as the frame-base baseline in our work of learning video representations for object detection. Also, inspired by the ROI-pooling, we design a nouvel pooling layer operating on spatio-temporal domain. Details are presented in Chapter 6.

Figure 3 - 5 -

 35 Figure 3-5 -Fast-RCNN object detection pipeline [Girshick, 2015].

Figure 3 -

 3 Figure 3-6 shows the Faster-RCNN object detection pipeline. The RPN branch uses the same feature map used for classification in the Fast-RCNN branch. RPN slides a small convolutional network on the feature map and generate 𝐾 object hy-

1 : 1 , 1 : 2 and 2 : 1 .Figure 3 - 7 -

 11137 Figure 3-7 -Position-sensitive score maps and Position-sensitive RoI Pooling [Dai et al., 2016].

Figure 3 -

 3 Figure 3-7 demonstrates the pooling process of R-FCN. Such a modification allows a fully-convolutional design for the object classification part of the R-FCN framework, which in turn significantly improves the running speed. In Chapter 7, R-FCN is the frame-base baseline, to which we compare our long-term representation learning framework for object detection in videos. We also use R-FCN as the backbone detection architecture, in this work.

  Computer vision has a rich body of work on recognizing human actions[START_REF] Laptev | Learning realistic human actions from movies[END_REF][START_REF] Niebles | Modeling temporal structure of decomposable motion segments for activity classification[END_REF][START_REF] Sadanand | Action bank: A high-level representation of activity in video[END_REF][START_REF] Wang | Action Recognition by Dense Trajectories[END_REF][START_REF] Marszalek | Actions in context[END_REF] and scenes[START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF][START_REF] Oliva | Modeling the shape of the scene: A holistic representation of the spatial envelope[END_REF][START_REF] Quattoni | Recognizing indoor scenes[END_REF][START_REF] Vogel | Natural scene retrieval based on a semantic modeling step[END_REF]. Most of this work addresses the problems of action and scene recognition separately. Recently, several methods have shown advantages of recognizing actions or tracking people in the context of their environments[START_REF] Marszalek | Actions in context[END_REF][START_REF] Kitani | Activity forecasting[END_REF]. Similarly, the interplay between human poses and objects has been studied in[START_REF] Grabner | What makes a chair a chair?[END_REF][START_REF] Gupta | From 3d scene geometry to human workspace[END_REF][START_REF] Delaitre | Scene semantics from long-term observation of people[END_REF]. While previous work has looked at functional properties for a few selected classes of scenes and objects, here we aim to exploit correlation between scenes and actions at a large scale of hundreds of scene categories.Using the discovered correlations, we demonstrate prediction of human actions for test images of outdoor scenes such as, for example, found on Google maps.To reach our goal, we construct a new SUN Action dataset and collect manual annotations of human actions for 7940 images of 397 scene categories from the SUN dataset[START_REF] Xiao | Sun database: Large-scale scene recognition from abbey to zoo[END_REF]. Analysis of this data reveals strong action-scene correlation for the majority of scene categories. Notably, we show that an image's scene category can be determined from corresponding textual descriptions of characteristic actions for that image. We present the SUN Action dataset in Section 4.2 and then study the action-scene correlation in Section 4.3.Using the discovered relations between scenes and actions, we next address the task of automatic action prediction for images of static scenes. We consider 38 action classes and assign action labels to 397 scene categories. Using such scene-based action annotation we learn visual classifiers for each action category and predict actions for images of static scenes. In Section 4.4, we demonstrate successful action prediction for images of static scenes. In Section 4.5 we propose and investigate the new task of geo-localized action prediction. Our motivation comes from the large amount of publically-available geotagged images (e.g., on Flickr, Instagram and Google Maps) which number constantly increases with the popularity of social networks and the availability of connected camera devices. Application of automatic action prediction on such images will enable the search for places based on their function, including specific actions such as swimming, having picnic, hiking and many others. In our experiments reported in Section 4.5, we use geo-localized images collected from panoramio.com and demonstrate examples of successful map-based action prediction for the region of France. Section 4.6 concludes the chapter.

17. 3

 3 responses on average, and each category received an average of 346.4 responses. Example images and corresponding responses from the SUN Action dataset are shown in Figure 4-1. We have observed a varying diversity of responses for different scene categories. The top row of Figure 4-1 shows a few examples of scene classes with low entropy of response histograms (high annotator agreement, low response diversity). Such scenes often correspond to places that have been designed for specific purposes (tennis court) or where the natural environment limits the set of possible actions (wave). In contrast, scene classes with high entropy of responses (Figure 4-1, bottom) are places that afford many actions (e.g., a television studio, where many actions need to take place over the course of filming) or unfamiliar places (an anechoic chamber).

Figure 4 - 1 -Figure 4

 414 Figure 4-1 -SUN Action scene categories with corresponding histograms of action responses. Top row: Scene categories with low entropy of response histograms. Bottom row: scene categories with higher entropy of response histograms. Low-entropy categories are often places designed for specific purposes (tennis court) or where the environment limits possible actions (wave). By comparison, high-entropy categories are places that afford many actions (television studio) or are unfamiliar (anechoic chamber)

Figure 4 - 3 -Figure 4 - 4 -

 4344 Figure 4-3 -Results of action-based scene classification. (a): Confusion matrix for the 33-category subset using Maximum Likelihood Naive Bayes estimation. The high values along the diagonal indicate excellent classification performance. A few pairs of categories e.g., (basement,attic) and (river,lake) are confused due to similarity in their characteristic actions. (b): Average accuracy (%) of scene classification for the 33-category subset and for all 397 scene categories.

Figure 4 -

 4 Figure 4-3 illustrates results of scene classification. As visualizing results across 397 individual classes is difficult, we select a 33-category subset of well-recognized and semantically important scene categories. To select the 33-category subset, we have asked four of our collaborators to nominate 20-40 most important scene types. Out of the 80 scene types with most annotated images, 35 received at least two nominationsand were slated for inclusion. "Cathedral" was removed for not being different enough from "church", and "abbey" and "coast" was removed for containing only aerial shots, leaving a final slate of 33 scene categories.

Figure 4 -

 4 3(b). Notably, both methods perform considerably better than chance while Naive Bayes provides better performance than Nearest Centroid. The fact that such simple classification methods yield very good performance indicates a strong correlation between scene categories and human actions: different scene categories have distinct patterns of associated actions. This confirms our initial hypothesis of a very strong relation between scene categories and their functional properties.4.3.2 Action-based scene clusteringWe seek to further investigate the structure of correlations between scene categories and actions: Which scene categories are more similar in terms of their function?We use hierarchical clustering and group scene descriptors at multiple scales. At the finest scale, only the most similar scene types cluster together, whereas at coarse scales, clusters are larger and encompass more dissimilar scene types. Dendrogram visualizations in Figure4-4show the progression of clustering patterns from fine to coarse: categories are represented as "leaves"; branchings closer to the leaves of the tree connect classes that cluster together under fine-grained clusterings; and branchings closer to the trunk of the tree encompass broader clusterings. The height of each linkage in the dendrogram indicates the distance between the subclusters it connects.The two dendrograms in Figure4-4 illustrate image-based and action-based scene clustering. In the first case, distances between scenes were obtained as Euclidean distances of corresponding image descriptors (seeSection 4.4). Clustering based on human action annotations was obtained using 𝜒 2 -distances between scene representations in terms of bag-of-words histograms (see Section 4.3.1). We observe that image-based similarity in Figure4-4(left) captures substantially different information about scene classes as compared to action similarity Figure4-4(right). For example, "alley", "bathroom" and "subway interior" are grouped together according to visual similarity due to their similar geometry and texture, but are separated according to action similarity since alleys, bathrooms and subways have different function. Another example is that visual similarity places "mountain, snowy" is a category of its own because no other class commonly depicts open white peaks, whereas action similarity places mountains together with other outdoor places that are associated with hiking, taking photos, and related actions.

  Figures 4-5-4-7 are produced with our best kernel combination.In Figure4-6, we show classification results with 38 outdoor and 23 indoor action classes sorted by AP. For better visualization of prediction results for some action classes, we show example images in Figure4-7. The last two columns depict some hard positive and hard negative samples for each class. For outdoor scenes, action classes such as "hike", "pray", having rather typical color/structure, are easier to classify than other "can-do-almost-every-where" action classes like "learn". While people can often differentiate universities from other buildings based e.g., on the text and other cues, the task is still difficult for current vision systems, especially, for those exploiting global image representations. We notice that indoor actions are more structuredependent than outdoor actions. In our experiment APs of indoor action classes are generally lower than APs of outdoor action classes. We also observe different levels of difficulty among indoor actions, e.g., detecting bowling lanes is easier than detecting sink-like structures. We found building action classifiers challenging, because positive samples are possibly images from very different scene categories, thus covering much larger range of visual texture and structure.

Figure 4 - 5 -Figure 4 - 6 -

 4546 Figure 4-5 -Automatic visual action prediction for test images in SUN Action dataset.

Figure 4 - 7 -

 47 Figure 4-7 -Selected SUN Action classification results -both outdoor (cyan) and indoor (orange) -with our best kernel combination.

4. 5

 5 Image-based Geo-Mapping of ActionsOne possible application of scene-based action prediction is to search for places affording specific action. For example, a user may ask "Where can I camp in the Mont Blanc valley?" or "Where can I sunbathe in Tuscany?". Such queries are currently not supported by map services such as Google Maps or Bing Maps. To address this problem, we introduce Image-based Geo-Mapping of Action (IGMA), an application

  the Panoramio dataset. We use Panoramio image sharing service [Google, 2007] to collect a dataset with geo-localized images. Like the SUN Action dataset, the images in Panoramio contain few to no people. The Panoramio service provides a REST API for selecting images: given a range of longitude and latitude values, Panoramio returns a JSON file of image properties including image URL and geographical position. For our experiment, we collected Panoramio images of France, with longitude from -5 ∘ to 8 ∘ and latitude from 41 ∘ to 51 ∘ . In total, our dataset contains over 38, 000 distinct geo-tagged images.

Figure 4 -

 4 Figure 4-8 -"Where can I ski in France?" -(Top left) Official skiing stations in France [ski, 2013]. (Top middle) Suggested places for skiing by IGMA. (Top right) Dense map of action "ski" generated by IGMA. (Bottom) Panoramio images of suggested places for skiing.

Figure 4 -

 4 Figure 4-8 illustrates IGMA's suggestions for the question "Where can I ski in France?". We compare the estimated dense map produced by IGMA for the action "ski" with the the map of official skiing stations in France, acquired from [ski, 2013].

Figure 4 -

 4 Figure 4-9(right) illustrates an interesting result of predicting the "swim" action in Paris. This result suggests an area for further investigation: the recommended locations for swimming in Paris fall mainly along the river Seine, where swimming is very uncommon. While it is true that scene categories often have strong correlation

5 . 1 . 1

 511 Figure 5-2). Object hypotheses form a grid of c = 284 square cells of four sizes (28x28, 56x56, 112x112 and 224x224 pixels) and the stride corresponding to the 50% of cell size. Except the output layer, the architecture of the Global CNN is identical to our Local model described in Section 5.1.1. The Global CNN is trained with SGD, minimizing the sum of c log-loss functions, one per each grid cell 𝑐 ∈ {1 • • • c},

5. 1 . 3

 13 Pairwise modelIn this section we describe our Pairwise model that aims to jointly reason about multiple object candidates. Following[START_REF] Desai | Discriminative models for multi-class object layout[END_REF] we formulate the model as a joint score function where variables correspond to object candidates. In the prior work[Desai et al., 2011, Felzenszwalb et al., 2010a, Hoai and Zisserman, 2014] unary potentials of the score function are defined by the response of the local object detector at corresponding locations, whereas higher-order potentials model spatial relations between candidates. Our Pairwise model enriches the model of [Desai et al., 2011] by making all potentials of the score function (5.2) dependent on the image data and, in contrast to [Chen and Yuille, 2014], allows to perform the joint training of all parameters. We describe details of our model in Section 5.1.3. We train parameters of our model by minimizing the structured surrogate loss using stochastic gradient descent algorithm. The details of our training procedure are presented in Section 5.1.3. Model formulation Score function. Consider a set of 𝒱 candidate bounding boxes (nodes) extracted from an image. Let each bounding box have a binary variable 𝑦 𝑖 , 𝑖 ∈ 𝒱 assigned to it. We associate label 1 with the object class and label 0 with the background class.

(5. 4 )

 4 The individual scores are used in the standard precision-recall evaluation pipeline[START_REF] Everingham | The PASCAL visual object classes (VOC) challenge[END_REF].

consists of the following four steps: 1 .

 1 Select the set of candidates by applying the non-maximum suppression [Felzenszwalb et al., 2010a] on top of the scores produced by the Local model. 2. Perform the forward pass through the model to compute potentials of the joint score function.

3 .

 3 Perform the inference to compute the structured loss and its gradient (see below).4. Back-propagate the gradient through the model.We explain details of the algorithm below.Structured surrogate loss. A structured loss is a function that maps the current values of parameters, image data x = (x 𝑖 ) 𝑖∈𝒱 and the ground-truth labeling ŷ = (ŷ 𝑖 ) 𝑖∈𝒱 to a real number. A popular choice for the surrogate loss for structured-prediction tasks is the structured SVM (SSVM) objective[START_REF] Taskar | Max-margin markov networks[END_REF][START_REF] Tsochantaridis | Large margin methods for structured and interdependent output variables[END_REF]:ℓ SVM (w, ŷ, x) = maxy ( 𝑆(y; w, x) + ℎ(y, ŷ) ) -𝑆(ŷ; w, x) (5.5) where ℎ(y, ŷ) ≥ 0 measures the agreement between the two labelings. Possible choices for ℎ include the Hamming loss, the Hamming loss with penalties normalized by the frequency of classes, or higher-order losses making use of assumption that each ground-truth object is assigned to exactly one object candidate [Osokin and Kohli, 2014]. Notice, that in (5.5) the joint score 𝑆 depends on parameters w and image data x implicitly through potentials 𝜃 𝑈 and 𝜃 𝑃 . However, in our experiments we have observed that the SSVM loss is less suited for the detection task, i.e. optimizing the objective (5.5) does not lead to good results in terms of precision-recall measure. To tackle this problem, we propose a new surrogate loss which directly imposes penalties on the wrong values of individual scores (5.4) extracted from the joint score 𝑆. Specifically, this loss can be written as ℓ(w, ŷ, x) = ∑ 𝑖:ŷ 𝑖 =1 𝑣(𝑠 𝑖 (w, x)) + ∑ 𝑖:ŷ 𝑖 =0 𝑣(-𝑠 𝑖 (w, x)) (5.6) where 𝑣 can be any non-increasing function bounded from below. We use 𝑣(𝑡) = log(1 + exp(-𝑡)) which brings us closer to the training of conventional detector with a soft-max loss.

  Local and Pairwise models. We now describe the process of computing the scores of the joint model. First, we compute the scores of the Local model for all candidates and perform the non-maximum suppression[Felzenszwalb et al., 2010a] using NMS threshold 0.3. The 16 top-scoring detections produced by NMS are then used as input for the Pairwise model. This number of candidates is sufficient on scenes with a few people, but can cause the drop of recall for crowded scenes (especially for some scenes of Casablanca dataset). To compensate for this drop, we combine scores produced by the Local and Pairwise models 𝑠 𝑙 ,𝑠 𝑝 respectively. For candidates with both scores existing, we use the affine combination 𝑠 𝑙𝑝 = 𝛼𝑠 𝑙 + (1 -𝛼)𝑠 𝑝 + 𝛽. For candidates with the score of the Pairwise model non-existent, we use the score of the Local model 𝑠 𝑙𝑝 = 𝑠 𝑙 . Parameters 𝛼 ∈ [0, 1] and 𝛽 ∈ [-10, 10] are selected by maximizing AP on the validation set using grid search. Local, Pairwise and Global models. To combine scores 𝑠 𝑙𝑝 with the Global model, we associate each candidate with the output cell of the Global model having maximum IoU overlap-ratio. The score of the joint model 𝑠 * is computed as an affine combination of the detection score 𝑠 𝑙𝑝 and the grid cell score 𝑠 𝑔 , i.e. 𝑠 * = 𝛾𝑠 𝑙𝑝 +(1-𝛾)𝑠 𝑔 where 𝛾 ∈ [0, 1] is obtained by maximizing AP on the validation set.

3 . 3 .

 33 and discuss two other datasets we use for evaluation: TVHI [Patron-Perez et al., 2012,[START_REF] Hoai | Talking heads: Detecting humans and recognizing their interactions[END_REF] and Casablanca[START_REF] Ren | Finding people in archive films through tracking[END_REF].5.3.1 HollywoodHeads datasetHollywoodHeads dataset contains 369, 846 human heads annotated in 224, 740 video frames from 21 Hollywood movies The movies vary in genres and represent different time epochs. To create annotation, we have manually annotated tracks of human heads in action-rich movie clips. For each head track, head bounding boxes, i.e., the smallest axis-parallel rectangles including all visible pixels of the head, were manually annotated on several key frames. The bounding boxes on remaining frames were linearly interpolated and manually verified to be correct. In total, we have collected 2, 380 clips with 3, 872 human tracks, spanning over 3.5 hours of video. The dataset is divided into the training, validation and test subsets which 3. List of movies used in HollywoodHeads dataset. Training set: American beauty, As Good As It Gets, Big Fish, Big Lebowski, Bringing out the dead, Capote, Clerks, Crash, Dead Poets Society, Double Indemnity, Erin Brockovich, Fantastic 4, Fargo, Fear And Loathing In Las Vegas, Fight Club.Validation set: Five Easy Pieces, Forrest Gump, Gang Related. Test set: Gandhi, Charade, I Am Sam. 69 have no overlap in terms of movies Given the redundancy of consequent video frames, we have temporally subsampled videos in the validation and test subsets.

  The extended TV Human Interaction (TVHI) dataset [Patron-Perez et al., 2012, Hoai and Zisserman, 2014] consists of 1, 313 frames of TV show episodes annotated with bounding boxes of human upper bodies. Frames are split into the two sets: 599

  4.1) and provide the comparison with the state-of-the-art (Section 5.4.2). Section 5.4.3 compares different architectures of the Local model. We then justify the need of our new large dataset for training (Section 5.4.4) and show improvements in computational complexity that can be achieved with the Global model (Section 5.4.5). To evaluate the detection performance, we use the standard Average Precision (AP) measure based on the Precision-Recall (PR) curve [Everingham et al., 2010]. Detections having high overlap ratio with the ground truth (IoU > 0.5) are considered as true positives. Multiple detections assigned to the same ground truth are penalized and declared as false positives. Matches to "difficult" head annotations are ignored in the evaluation, i.e. such detections are considered neither as true positives nor as false positives. 5.4.1 Results of context-aware models We compare performance of the following four models: the Local model (Sec .5.1.1), the combination of the Local and Global models (Section 5.1.2), the combination of the Local and Pairwise models (Section 5.1.3) and the combination of all the three proposed models. The performance of head detection is evaluated on HollywoodHeads, Casablanca and TVHI datasets. Qualitative results of the Global and Pairwise models are illustrated in Figures 5-2 and 5-4 respectively. Table5.1 presents quantitative results for all models. We observe that the Global and Pairwise models consistently improve the performance of the baseline Local model. The combination of all three models demonstrates the best performance on all three datasets.5.4.2 Comparison with the state-of-the-art methodsWe compare our approach against several baselines: the CNN-based object detector[START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF]] (R-CNN), DPM-based face detector[START_REF] Mathias | Face detection without bells and whistles[END_REF]

Figure 5 -

 5 Figure 5-2 -Qualitative results for the Global model. The top row shows detections produced by the Local model. The middle row illustrates the multi-scale score map produced by our Global model. Red color correspond to high score values for the "head" class, blue color -to low score values. The bottom row demonstrates detections by the combination of the Local and Global models.

Figure 5 - 3 -

 53 Figure 5-3 -Results of our method compared to the state-of-the-art on the three datasets.

5. 4 . 3

 43 Architectures of the Local model. In this section we compare the performance and speed of different architectures of the Local model. We consider AlexNet architecture [Krizhevsky et al., 2012], VGG-S [Chatfield et al., 2014], VGG-verydeep-16 [Simonyan and Zisserman, 2015] provided with the MatConvNet framework [Vedaldi and Lenc, 2015] 5 and the model of [Oquab et al., 2014]. All models were pre-trained on the ImageNet dataset [Deng et al., 2009] and fine-tuned on the training set of the HollywoodHeads dataset as the Local model.

Figure 5 - 4 -

 54 Figure 5-4 -Qualitative results for the Pairwise model. For each video frame we show results of the Local model (left) and the Pairwise model (right). For both methods we choose the score thresholds such that the precision equals the recall on the validation set. The plotted bounding boxes show the detections with the scores above the selected thresholds. Yellow boxes correspond to correct detections, redto false positives. For the Pairwise model we show the strength of links between the candidates detected by at least one method. Links above a strength threshold (attractive) are plotted yellow and others -red (repulsive).

Figure 6 - 1 -

 61 Figure 6-1 -Space-time object tubes used in our work to capture evolution of object appearance over short sequences of video frames.

Figure 6 - 2 -

 62 Figure 6-2 -Architecture of Tube-CNN for object detection. The input is a chunk of images and a set of tube proposals. The model starts with the CNN feature extractor to get a spatio-temporal feature volume. The following network splits into the three branches: tube classification (starts with Tube-of-Interest, TOI, pooling), regression on the start and end frames, which begins with the frame-level Region-of-Interest pooling, ROI.

6. 1 . 1

 11 ArchitectureTube-CNN is a convolutional neural network operating on chunks of consecutive frames and the corresponding sets of tube proposals. For each proposal, the model outputs the class scores and the refined proposal position. Tube-CNN is an end-toend model consisting of three main blocks: CNN feature extraction, tube classification and tube regression. The overall network architecture is shown in Figure6-2.CNN feature extractor. The first block of the network extracts a feature map independently for every frame in the input chunk of frames. Extracted features are stacked along the temporal dimension to form a spatio-temporal feature volume. To achieve good performance, we reuse the idea of sharing computations from first CNN layers among all proposals[START_REF] He | Spatial pyramid pooling in deep convolutional networks for visual recognition[END_REF][START_REF] Girshick | Fast r-cnn[END_REF]. For the architecture on this block, we can reuse most of single frame CNNs (we try CaffeNet1 and ResNet[START_REF] He | Deep residual learning for image recognition[END_REF]).

Figure 6 - 3 -

 63 Figure 6-3 -Architecture of Tube Proposal Network (TPN). Tube anchors are plotted in red, yellow and green.

Figure 6 -

 6 Figure 6-3 illustrates the TPN model. Similar to the Tube-CNN model, TPN passes input chunks of consecutive frames through several CNN layers. The output feature maps are stacked along the temporal dimension and passed through a 3 × 3 × 3 volumetric convolutional layer (conv3D) [Tran et al., 2015] to form a feature volume.

Figure 6 -Figure 6 - 4 -

 664 Figure 6-4 -Examples of TPN proposal. Examples of TPN proposals. Tube anchors and final proposal are plotted by green and magenta respectively.

3

 3 million video frames. The dataset contains 5 354 short videos split into 3 862 videos for training, 555 videos for validation and 937 videos for testing. The 30 object classes of the ImageNet VID dataset are a subset of 200 classes used in ImageNet DET challenge. As the ground-truth for the test set is not published, we evaluate and compare our results on the validation set with 176 126 frames.

Figure 7 - 1 -

 71 Figure 7-1 -A schematic comparison between a typical per-frame method (left) and the proposed video representation learning approach (right) for object detection in videos. Wepropose a multi-scale memory that efficiently aggregates image evidence over longer time horizons and also accounts for camera and object motion by feature warping, which enables learning better representations that lead to higher accuracy.

Figure 7 -

 7 Figure 7-2 -A detailed illustration of the proposed MemNet running for three frames. At every time step, features from the current frame (blue) are aggregated with the memory of the previous frame (purple), either by simple averaging or with a learned adaptive weighting.

Section 7 .

 7 1.3 discusses the differences between our models and other memory-base architectures such as FGFA [Zhu et al., 2017b] and convolutional RNNs [Pinheiro and Collobert, 2014, Liang and Hu, 2015].

  The weights are computed by a separate CNN, named weight-CNN, with parameters 𝜉 ℳ and 𝜉 ℱ operating on ℳ and ℱ, respectively, and the constraint 𝛼 ℳ + 𝛼 ℱ = 1 is always satisfied by passing the concatenated output of the CNNs through a per-pixel softmax function. The parameters of weight-CNN are automatically learned together with the rest of the network without any additional supervision. In the first frame 𝑡 = 1, we simply assign the memory ℳ 1 to be the feature representation of the image ℱ 1 . Training MemNet: Training the proposed video representation requires a supervisory signal from a task module acting on top of the memory features ℳ. In general, ℳ can be used for different tasks, for example to predict future frames [Mathieu et al., 2016]. In this work we explore object detection in videos, where the supervisory signal comes from a combination of object localization and classification loss functions, see Section 7.2.1. All parts of the video representation can be trained end-to-end. Since the bilinear sampling and the grid generation of the warping module are both differentiable [Jaderberg et al., 2015b], we can back-propagate the training signal over time to previous frames, to the image feature extractor, as well as to the FlowNet generating the displacement fields.

Figure 7 - 3 -

 73 Figure 7-3 -Our ClockNet extends the MemNet by adding multiple time axes with increasing time scales to aggregate more information from further back in time. Each additional time axis 𝑘 > 1 skips 2 𝑘-1 -1 frames. We only illustrate two time scales to avoid clutter.

Figure 7 - 4 -Figure 7 - 4 .

 7474 Figure 7-4 -Relation between standard convolutional recurrent neural networks (left) and the proposed video representation (right).

  Given a representation ℳ 𝑡 of a video sequence at frame 𝑡, the object detector first computes object proposals with a region proposal network (RPN) as proposed in[START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF]]. Object proposals define potential locations of objects of interest (independent of the actual category) and reduce the search space for the final classification stage. Each proposal is then classified into one of the 𝐶 categories and the corresponding proposal location is further refined. In contrast to Faster-RCNN[START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], the per-proposal computation costs in R-FCN are minimal by using position-sensitive ROI pooling. This special type of ROI pooling is applied on the

Figure 7 - 5 -

 75 Figure 7-5 -In a multi-GPU setup, a fast but weak object detector (R-FCN-18, green blocks) leverages the features of a strong but slow object detector (ClockNet-101, red blocks), see Section 7.2.2. The width of blocks represent computation time. At a frame 𝑡, strong features from ClockNet-101 are only available from 𝑡 -𝛥, but efficiently warped into frame 𝑡 with our propagation module. The warped features boost the representational power of R-FCN-18 significantly, without increasing latency of the real-time system.

3 . 1 , 7 . 3 . 2 and 7 . 3 . 3 ,

 31732733 we evaluate the performance of the proposed MemNet and ClockNet on the three applications introduced in Section 7.2, respectively. Dataset: All our experiments are conducted on the ImageNet VID data set [Russakovsky et al., 2015], which is most suitable for object detection in videos and has been used to evaluate recent approaches for this task [Feichtenhofer et al., 2017b, Kang et al., 2016b, Zhu et al., 2017a,b]. ImageNet VID is a large scale data set consisting of 5344 video clips, captured at frames rates between 25 and 30 FPS and divided into training, validation and testing sets with 3862, 555 and 937 clips respectively. Each clip is fully annotated with bounding box tracks of 30 different object classes. Implementation details: We use the ResNet-101 architecture [He et al., 2016] as the basic feature extractor in all experiments with the exception of some ablation studies. In particular, we use à trous convolutions as in [Chen et al., 2016, Zhu et al., 2017b] to increase the feature resolution. Similar to [Zhu et al., 2017b] the extracted features are passed through a 3 × 3 convolutional layer and a non-linear activation (ReLU [Nair and Hinton, 2010]) before we provide them as inputs to MemNet and ClockNet. To estimate displacement fields we use FlowNet [Dosovitskiy et al., 2015]. All the parameters are jointly fine-tuned end-to-end. In general, we closely follow the experimental setup of [Zhu et al., 2017b] using their publicly available MXNet [Chen et al., 2015] implementation. All models are trained for 2 epochs with an initial learning rate of 0.001, which is decreased by a factor of 10 after 4 3 epochs. We train our models on the same mix of ImageNet DET and ImageNet VID training sets as in [Zhu et al., 2017b]. All experiments, including runtime measurement, are done with NVIDIA TITAN Xp GPUs. For training, we use a setup with 4 parallel GPUs.

Figure 7 - 6 -

 76 Figure 7-6 -Accuracy and runtime trade-off of various methods.

Figure 7 - 7 -

 77 Figure 7-7 -Qualitative examples of our memory models and the RFCN baseline.

1 and Figure 7 - 6

 76 demonstrates the advantages of our memory propagation mechanism and our temporal multi-scale architecture ClockNet in terms of accuracy and runtime. Moreover, our memory-based architectures have the additional benefit (over FGFA) of being able to propagate features into future frames, which we analyze in Section 7.3.2 and demonstrate in a practical application in Section 7.3.3.

  1 that a longer temporal window for training MemNet (MemNet-6 vs. MemNet-3) is indeed beneficial. In addition, we analyze the importance of multiple temporal scales in ClockNet, which operates on three time axes 𝑘 =[START_REF]-4 A schematic comparison between a typical per-frame method (left) and the memory-based[END_REF] 3, 4], corresponding to temporal strides of 1, 4 and 8, respectively. To emphasize the benefit of feature aggregation across multiple temporal scales, we compare ClockNet to MemNet trained with longer strides (MemNet-6-strd-4 and MemNet-6strd-8), given it access to larger temporal horizon during training. We can see from Table7.1 that both baselines perform worse than MemNet-6 (with temporal stride of 1), which illustrates the importance of the information provided by actual neighboring frames. Therefore, the success of ClockNet provides a signal for the benefits of a temporal multi-scale architecture, where different temporal scales are complementary to each other. More experiments on various details of our methods can be found in the supplemental material. Different aggregation schemes and aggregation levels: We further want to investigate the impact of several aspects of the proposed model and conduct an ablation study on (i) different feature aggregation schemes and (ii) memories at different feature representation levels. For all these experiments, we use a ResNet-50 model and train only on half resolution images.

Figure 7 -

 7 Figure 7-8 -We illustrate the impact of jointly fine-tuning the FlowNet with the object detector. For each example consists of 4 images: the 2 input images ("Image0, Image1") and the displacement field before and after fine-tuning. The figure shows 2 columns and 4 rows of examples.

  This work has addressed problems of action prediction in still images as well as object detection in videos. Towards action prediction, we have collected a new dataset with manual annotations of typical actions for 397 scene categories. The strong correlations between actions and scenes are justified by statistical methods. Based on such correlations, we have proposed a new task of predicting actions in still images and introduced a potential application of affordance geo-localization. In the second part of the thesis we have addressed object detection in videos from different angles.First, on the frame-level, we have leveraged contextual cues to improve detection performance with Global and Pairwise models. Second, on the clip-level, we have designed CNN models that can learn spatio-temporal representations to capture the temporal evolution of object appearance. Last, we have designed an efficient and effective memory-base framework that can store and transfer long-term appearance information. Our approaches show significant improvements over the baselines. Despite recent progress computer vision is facing important challenges. Some of these challenges are described below.Self-supervised Feature Learning. The power of learning visual features currently comes with the high cost of annotating very large datasets. In recent years much work has been focused on creating large and high quality fully-annotated datasets with the goal of learning more robust visual representations[START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF], Neuhold 123 et al., 2017]. Although such a brute-force approach will advance computer vision in the short-term, its sustainability in the long run is questionable. Motivated by this issue, another direction of recent work addresses self-supervised feature learning from a massive amount of unannotated data using adhoc tasks[START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF][START_REF] Pathak | Learning features by watching objects move[END_REF][START_REF] Larsson | Colorization as a proxy task for visual understanding[END_REF][START_REF] Doersch | Multi-task self-supervised visual learning[END_REF]. Despite some progress, the performance of such methods is still not on a par with supervised results by fully-supervised approaches.

125 6 - 7 7 - 4

 6774 Qualitative results for HollywoodHeads, Casablanca and ImageNet VID datasets. On each frame we illustrate results of the Box-CNN model and the Tube-CNN model. For both methods we choose thresholds such that precision equals recall on the corresponding dataset. All detections with scores above threshold are plotted. Yellow boxes: correct detections, red boxes: false detections. . . . . . . . . . . . . . 7-1 A schematic comparison between a typical per-frame method (left) and the proposed video representation learning approach (right) for object detection in videos. We propose a multi-scale memory that efficiently aggregates image evidence over longer time horizons and also accounts for camera and object motion by feature warping, which enables learning better representations that lead to higher accuracy. . . . . . . . . . . . . . . . . . . . . . . . 7-2 A detailed illustration of the proposed MemNet running for three frames. At every time step, features from the current frame (blue) are aggregated with the memory of the previous frame (purple), either by simple averaging or with a learned adaptive weighting. Then, the memory feature map is warped via bilinear sampling based on a learned displacement field. The detection output in every frame is computed from the current memory. . . 7-3 Our ClockNet extends the MemNet by adding multiple time axes with increasing time scales to aggregate more information from further back in time. Each additional time axis 𝑘 > 1 skips 2 𝑘-1 -1 frames. We only illustrate two time scales to avoid clutter. . . . . . . . . . . . . . . . . . Relation between standard convolutional recurrent neural networks (left) and the proposed video representation (right). . . . . . . . . . . . . . . .

  attacks and violent behavior 2 . Video surveillance systems require reliable detection of malicious human activities like robbery, burglary or violence 3 . Aside from security purposes, person detection/tracking and action analysis can be used to assist sport coaches in planning strategies. Human gaze detectors in smart TVs can capture human attention and help generating targeted advertisements. Gaze detection and emotion recognition is also relevant for designing more efficient education schemes [Asteriadis et al., 2009, Rodrigo and Baker, 2009, D'Mello et al., 2012, Sümer et al., 2018]. Analyzing student behaviors and interactions gives more insight on how students teach each other which helps to optimize peer instruction [Jermann et al., 2011, Chong et al., 2017, Hayashi, 2018]. At a larger scale, crowd counting and crowd motion analysis are useful in scenarios such as demonstrations or mass evacuations.

1. http://money.cnn.com/2017/04/26/technology/amazon-echo-look/index.html prevent Among a large range of human-related vision tasks, Action Prediction attempts to anticipate human behaviors based on available visual cues. Action prediction can be addressed in images [Delaitre et al., 2010, Yao and Fei-Fei, 2010a, Delaitre et al., 2011, Vu et al., 2014] or videos [Kitani et al., 2012, Walker et al., 2014], in settings that contain people [Ikizler et al., 2008, Ikizler-Cinbis et al., 2009, Ferrari et al., 2009, Yang et al., 2010] or without people [Vu et al., 2015]. Importantly, action prediction can provide reliable priors for other action analysis tasks such as action recognition or action localization. Besides helping to improve recognition performance, action priors are useful to detect abnormal behavior.

  Viola-Jones, is the use of CNN features instead of hand-crafted ones. In the original work, the authors used CNN features produced by AlexNet [Krizhevsky et al., 2012] pretrained on image classification task. To reduce the search space, R-CNN uses object proposals generated by Selective Search [Uijlings et al., 2013]. Each regional proposal is cropped, resized and forwarded through a CNN to produce an appearance feature. The pipeline ends with SVM classifiers, taking the CNN feature as input and producing object scores. With a relatively simple design, R-CNN outperformed other contemporary methods by a large margin.The limitation of R-CNN is its heavy computations and high storage-demand.Inspired by the idea of the Spatial Pyramid Pooling network (SPPnet)[He et al.,

	2.2.2 Early Object Detection Frameworks video signal as an entire combination of the two factors. however, this information is not used to transform features over time to compensate
	In [Kang et al., 2017], the authors propose a Tubelet Proposal Networks for gen-for observer and scene motions.
	Here we extend our review to more general approaches that can detect arbitrary
	modification allows a fully-convolutional design for the object classification part of erating tubelet proposals, which are then scored by an encoder-decoder CNN-LSTM. The recently proposed flow-guided feature aggregation framework (FGFA) [Zhu
	objects. Viola-Jones [Viola and Jones, 2001] is an efficient detection framework ex-
	ploiting a cascading architecture with multiple stages, built upon fast Haar features and Adaboost. The framework works well for rigid objects which appearance variants the R-FCN framework. R-FCN is later improved with deformable convolution and deformable pooling layers in [Dai et al., 2017]. Deformable R-FCN learns where to aggregate and pool information in the image and demonstrates better robustness to Regression is done on every frame to adjust box location along the tubelet. The authors show that their proposal network performs best with temporal window size of 5 frames. et al., 2017a,b], on the other hand, explicitly warps convolutional feature maps be-tween frames for better alignment when aggregating them. The warping function is triggered by a learned displacement field initialized with FlowNet [Dosovitskiy et al., Chapter 3
	are low, e.g. frontal faces.
	In cases of non-rigid deformations and large variations in appearance due to chang-ing viewpoints or intra-class variability, more complicated frameworks using pictorial variations in scale and rotation. Deformable R-FCN currently shows best performance for object detection in still images on standard benchmarks. While performance of recent object detection methods is getting close to human-Object detection in video and tracking-by-detection have a tight relation. Graph-based methods [Pirsiavash et al., 2011, Tang et al., 2015] formulate object tracking as a global optimization problem on association graph. The association graph is con-2015]. FGFA [Zhu et al., 2017b] uses a fixed temporal window of nearby frames, requires as many warp computations as the length of the window, and compromises causality, i.e., integrates features from future frames. Background
	structures [Fischler and Elschlager, 1973, Felzenszwalb and Huttenlocher, 2005] like
	level, such methods often require expensive computational power. For real-time appli-structed by measuring appearance similarity between objects. In [Leal-Taixé et al., Object detection in videos has recently attracted much interest, partly due to the
	cations, along side with performance, speed is an important factor. Works like Single-2016], the authors propose a Siamese CNN to learn such appearance similarity effi-introduction of the ImageNet VID challenge [Russakovsky et al., 2015]. Besides [Fe-This chapter introduces terminology and methods that will be used in the rest of
	Shot-Detector (SSD) [Liu et al., 2016] and You-Only-Look-Once (YOLO) [Redmon ciently. ichtenhofer et al., 2017b, Zhu et al., 2017b] with a focus on the feature level, most the thesis. In Section 3.1 we review SVM-based image classification used for action
	et al., 2016, Redmon and Farhadi, 2017, 2018] endeavour to reduce detection time recent approaches for detection in video leverage the temporal data on a higher level, prediction in Chapter 4. Section 3.2 gives an overview of CNN architectures used in
	without loosing much performance. Video representation learning for object detection in videos. Learning rep-e.g., by tracking objects or specifically designed post-processing mechanisms [Han our work for object detection in Chapters 5-7.
	resentations for videos has been a long-standing goal in computer vision and many et al., 2016, Kang et al., 2016b], which are orthogonal to contributions in this thesis.
	Object Detection in Videos Object detection in video is closely related to ob-directions have been explored. Donahue et al. [Donahue et al., 2015] or Srivas-Besides object detection, the proposed feature representations can also be used for
	ject detection in images, as one can apply image object detectors to each frame of tava et al. [Srivastava et al., 2015] rely on recurrent neural networks (RNNs) like other tasks that operate on videos like semantic segmentation [Kundu et al., 2016,
	a video. Video, however, provides additional cues, such as temporal continuity. A LSTMs [Hochreiter and Schmidhuber, 1997] to propagate feature representation from Shelhamer et al., 2016] or action recognition [Feichtenhofer et al., 2016, 2017a, Si-
	common practice for video object detection is to post-process detections by means of still frames over time. However, unlike our approach, features are propagated with-monyan and Zisserman, 2014]. Visual anticipation has also been shown useful for the
	a tracker [Bojanowski et al., 2013, Ordonez et al., 2011, Everingham et al., 2006, Pir-out explicit knowledge of motion in the scene. 3D CNNs provide more freedom to tasks of segmentation [Luc et al., 2017] and action recognition [Vondrick et al., 2016].
	siavash et al., 2011]. The tracker first links high-confident detections on consecutive learn motion-specific kernels (although motion is also not explicitly used) and were Our work targets object detection and explicitly accounts for motion in the learned
	frames into tracks. As object detector often produces unstable scores along an object successfully used in [Tran et al., 2015, Yao et al., 2015] for tasks like video caption-representation.
	track, the linking step helps to find missing detections. An additional smoothing step ing or action recognition, but come with considerably more parameters to learn and
	could be done to stabilize object position on the track. Recently, Kang et al. [Kang typically more computational costs. Recent works have also considered unsupervised
	et al., 2016b] use this practice to generate object tracks. In their work, the tracker learning with pretext tasks [Wang and Gupta, 2015], however, we focus on efficient
	output is named tube proposal. It then becomes input for the next post-processing learning of supervised representations.
	ture learning with Neural Networks [Feraund et al., 2001], or Deep Convolutional steps including box-perturbation, max pooling and rescoring. To avoid possible con-Two-stream architectures like [Simonyan and Zisserman, 2014, Feichtenhofer et al.,
	Neural Network [Girshick et al., 2014, Girshick, 2015, Ren et al., 2015], also falls into fusion, we stress that the term tube proposal is later used in Chapter 6 with a different 2016, 2017a] combine features extracted from both images and motion (optical flow)
	this category. Such approaches often require a large amount of annotated training meaning. Unlike other video detection methods [Pirsiavash et al., 2011, Tang et al., to boost the representational power. While 3D CNNs learn motion-specific features
	data to achieve good performance. 2015], which separate appearance and motion processing phases, our models treat implicitly, two-stream architectures explicitly take optical flow as input. In both cases,

or parameterized [Yuille et al., 1992, Lanitis et al., 1995, da Vitoria Lobo and Kwon, 1998, Cootes et al., 2001]. Object of interest, i.e. human faces, is determined based on the correlation scores between the template and local regions in the image. Correlation score represents the likelihood of having face appearance on the local region.

Learning methods. Such methods use statistical analysis and machine learning to learn face appearance feature

[START_REF] Kirby | Application of the karhunen-loeve procedure for the characterization of human faces[END_REF] Sirovich, 1990, Turk and[START_REF] Turk | Eigenfaces for recognition[END_REF]

, instead of manually designing face templates. Most popular techniques are Eigenfaces

[START_REF] Turk | Eigenfaces for recognition[END_REF]

, Support Vector Machine

[START_REF] Osuna | Training support vector machines: an application to face detection[END_REF][START_REF] Constantine P Papageorgiou | A general framework for object detection[END_REF]

], Naive Bayes Classifier

[Schneiderman and Kanade, 1998, 2000]

and Hidden Markov Model

[START_REF] Samaria | Hmm-based architecture for face identification[END_REF] Young, 1994, Nefian and[START_REF] Ara | Face detection and recognition using hidden markov models[END_REF]

]. Fea-Deformable Part Models (DPM)

[Felzenszwalb et al., 2010a

] are more effective. DPM considers object as a combination of parts in a pre-defined star-like structure. Detection and parts discovery are done simultaneously by solving a joint optimization problem. DPM's main bottleneck is the speed. Follow-up works accelerate DPM using different techniques such as cascade

[Felzenszwalb et al., 2010b]

, coarse-tofine

[START_REF] Pedersoli | A coarse-to-fine approach for fast deformable object detection[END_REF]

, branch-and-bound

[START_REF] Kokkinos | Rapid deformable object detection using dual-tree branch-andbound[END_REF]

, Fast Fourier Transform (FFT)

[START_REF] Dubout | Exact acceleration of linear object detectors[END_REF]

2.2.3 Object Detection using Convolutional Neural Networks

Prior to 2011, most computer vision models relied on hand-crafted features, e.g. Haar, HOG, SIFT. Since 2011, the success of deep convolutional neural networks (CNNs) [Krizhevsky et al., 2012] has opened a new epoch for computer vision. Combining the massive amounts of data and the powerful computing hardwares, i.e. GPUs, CNNs are able to learn strong appearance features that can be well adapted to multiple vision tasks, including object detection. In the following, we give an overview of the most recent object detection frameworks that are built on Convolutional Neural Networks. Object Detection in Images. R-CNN [Girshick et al., 2014] presents a successful object detector based on recent CNN features. The main difference between R-CNN and traditional frameworks, e.g. 2014], Fast-RCNN [Girshick, 2015] improves R-CNN speed and performance with the efficient Region-Of-Interest (ROI) pooling layer. Features of region proposals are extracted from a shared feature map. Moreover, the SVM classifier is replaced by a neural network, which not only removes the high storage-demand but also unifies the detection pipeline. In the following work, Faster-RCNN [Ren et al., 2015] improves Fast-RCNN speed and performance with the Region Proposal Networks (RPN). The RPN replaces the only non-CNN component in the fast-RCNN pipeline, i.e. region proposal generator, and results in the end-to-end CNN-based detection pipeline. Compared to Fast-RCNN, Faster-RCNN is both faster and more accurate. All aforementioned methods use image features produced by pre-trained CNNs on image classification task. Detection performance thus heavily depends on the quality of features generated by the base-networks. In practice, the stronger the base-network is, the better performance we get, i.e. Faster-RCNN based on ResNet-101 [He et al., 2016] outperforms Faster-RCNN based on AlexNet [Krizhevsky et al., 2012] by a large margin. R-CNN, Fast-RCNN and Faster-RCNN are designed in such a way that we can easily 'upgrade' the feature extraction part. However, when it comes to Residual Networks (ResNet) [He et al., 2016], such a simple modification is suboptimal. In [Dai et al., 2016], the authors argue that detection frameworks based on ResNet should be fully-convolutional in order to achieve the optimal performance. R-FCN is designed in such a spirit by using Position-Sensitive ROI pooling layer. ROI pooling is no more a class-agnostic operation but a class-and position-dependent operation. That

  𝐾 visual centers 𝜇 1 , 𝜇 2 , ..., 𝜇 𝐾 . On a given image, 𝑛 local descriptors {x 1 , x 2 , ..., x 𝑛 } are densely extracted and are assigned to their closest centers. Considering 𝑠 th descriptor, we have the 𝑠 th assignment determined as: 𝑎 𝑠 = argmin 𝑖∈T 𝑑(x 𝑠 , 𝜇 𝑖 ) with 𝑑(, ) is the distance function. Finally, the set of descriptors is encoded into a histogram vector ℎ ∈ R 𝐾 , with:ℎ 𝑖 = |{𝑠 | 𝑎 𝑠 = 𝑖}| X|𝜃) w.r.t the GMM parameters 𝜃 = (𝜋 1 , 𝜇 1 , 𝛴 1 , ..., 𝜋 𝐾 , 𝜇 𝐾 , 𝛴 𝐾 ).One way is using the Fisher information matrix 𝐹 𝜃 = 𝐸 X [log 𝑃 (X|𝜃) log 𝑃 (X|𝜃) ′ ]. {𝑥 𝑖 , 𝑦 𝑖 |𝑖 = 1..𝑁 } with data points 𝑥 𝑖 ∈ R 𝑀 and labels 𝑦 𝑖 ∈ {1, -1}.𝑦 𝑖 𝑦 𝑗 𝛼 𝑖 𝛼 𝑗 ⟨𝑥 𝑖 , 𝑥 𝑗 ⟩The important point of using the dual form is that the optimization objective can be written entirely in terms of the inner products ⟨𝑥 𝑖 , 𝑥 𝑗 ⟩. Let us assume that there exist a feature mapping function 𝜙 which maps the data points in the original vector space to another vector space on which data can be linearly separated. The trick here is: instead of explicitly formulating 𝜙, which is a complicated and expensive task, we only have to define the inner product of the mapped features on the target vector space.Effectively, we define kernel 𝐾 : R 𝑀 × R 𝑀 → R such that 𝐾(𝑥 𝑖 , 𝑥 𝑗 ) = ⟨𝜙(𝑥 𝑖 ), 𝜙(𝑥 𝑗 )⟩. kernel trick to SVM is simply replacing ⟨𝑥 𝑖 , 𝑥 𝑗 ⟩ by 𝐾(𝑥 𝑖 , 𝑥 𝑗 ) everywhere.Intuitively, 𝐾(𝑥 𝑖 , 𝑥 𝑗 ) can be viewed as dissimilarity, or distance, between the mapped features 𝜙(𝑥 𝑖 ) and 𝜙(𝑥 𝑗 ) on the target feature space. In such a way, we

	The normalized gradient is then 𝐹 𝜃 -1/2 Kernel SVM. The dual form of SVM optimization problem is given as follows: ∇ 𝜃 log 𝑃 (X|𝜃). In practice, to achieve better layer is mainly characterized by its number of filters, kernel size, and spatial strides.
	performance, there are other good practices such as 𝑙 2 and power normalizations [Per-ronnin et al., 2010]. max 𝛼 𝑁 ∑ 𝑖=1 Since the success of AlexNet in image classification [Krizhevsky et al., 2012], CNNs 𝑁 1 ∑ 𝛼 𝑖 -𝑖,𝑗=1 2 have been widely used for many computer vision tasks, including object detection [Gir-
	Perona, shick et al., 2014, Girshick, 2015, Ren et al., 2015, Vu et al., 2015, Mordan et al., 2017] s.t. 𝛼 𝑖 ≥ 0, 𝑖 = 1, . . . , 𝑁 𝑁 and segmentation [Long et al., 2015, Chen et al., 2016], human pose and shape re-
	2005, Avila et al., 2013] and Fisher Vector (FV) [Perronnin and Dance, 2007, Sanchez ∑ 𝛼 𝑖 𝑦 𝑖 = 0. covery [Güler et al., 2018, Varol et al., 2018], image captioning [You et al., 2016],
	𝑖=1
	et al., 2013] pipelines are different at dictionary learning and feature encoding steps. 3.1.2 SVM Classification visual question answering [Antol et al., 2015, Ben-Younes et al., 2017], motion esti-
	While BoW uses K-means clustering and Histogram Encoding, FV uses Gaussian mation [Dosovitskiy et al., 2015] and others [Jain et al., 2017, Li et al., 2017, Lee
	Mixture Models and Fisher Encoding. Support Vector Machines (SVM) is a supervised machine learning technique which et al., 2017, Kim et al., 2018, Engilberge et al., 2018]. In the first part of this sec-
	is used in classification and regression tasks. In this part, we revisit the use of SVM tion, we revisit the base CNN architectures originally designed for image classification
	Quantization. The main goal of quantization is to construct a base for descrip-in binary classification. In a nutshell, SVM is a discriminative classifier trying to tasks. In the second part, we discuss CNN frameworks designed for object detection
	tors, usually named as visual dictionary. A visual dictionary is built by grouping separate data points lying on a p-dimensional feature space with an optimal (p-1)-in images and videos.
	descriptors into clusters, called visual words. Two most used clustering techniques dimensional hyperplane. The optimal hyperplane maximizes the distances, called
	are K-mean and Gaussian Mixture Models (GMM). Encoding. Extracted descriptors are encoded into a single feature vector. The two common encoding techniques are Histogram Encoding and Fisher Encoding. margins, between nearest data points of either class and the hyperplane. Originated from the work of Cortes and Vapnik [1995], SVM is formulated as a linear classifier. To solve nonlinear tasks, we can use the kernel trick [Boser et al., 1992]. 3.2.1 Architectures CNN is a feed-forward network consisting of successive layers stacking on top of each other. The first CNN architecture (AlexNet) that has shown success for Applying can choose a reasonable kernel given a particular classification or regression prob-classification of natural images [Krizhevsky et al., 2012] had five convolutional layers
	Histogram Encoding. Assumed that we already constructed a visual dictionary lem. Some most common kernel functions used in Computer Vision are Gaussian and relatively large filter sizes. Further work has shown advantages of CNNs with
	Linear SVM. We formulate the binary classification problem as follows: Given a dataset 𝒟 = The kernel: 𝐾(𝑥 𝑖 , 𝑥 𝑗 ) = exp(-||𝑥 𝑖 -𝑥 𝑗 || 2 2𝜎 2 ), 𝜒 2 -kernel: 𝐾(𝑥 𝑖 , 𝑥 𝑗 ) = exp(-𝛼 ∑ 𝑀 𝑚=1 (𝑥 𝑚 𝑖 -𝑥 𝑚 𝑗 ) 2 𝑥 𝑚 𝑖 +𝑥 𝑚 𝑗 ) and Histogram Intersection kernel: 𝐾(𝑥 𝑖 , 𝑥 𝑗 ) = ∑ 𝑀 𝑚=1 𝑚𝑖𝑛(𝑥 𝑚 𝑖 , 𝑦 𝑚 𝑖 ) more convolutional layers and smaller filters such as VGG architecture [Chatfield et al., 2014]. While the increased network depth has led to the vanishing gradient with K-means, consisting of Fisher Encoding. Fisher Vector (FV) is a popular encoding technique for scene SVM classifier is written as: ℎ(𝑥) = sign(𝑤 𝑇 𝑥 + 𝑏). In this way, the hyperplane is characterized by 𝜃 = ⎡ ⎣ problem, [He et al., 2016] enabled training of very deep networks by introducing ⎤ 𝑤 𝑏 ⎦ with 𝑤 ∈ R 𝑀 and 𝑏 ∈ R. Effectively, the line 𝜃 𝑇 [ 𝑥 1 ] = 0 is the decision boundary. Our goal is to find 𝜃 * that maximizes the margin. Slack ResNet architectures with skip connections. We overview these three types of CNN 3.2 Convolutional Neural Networks architectures below.
	classification [Perronnin and Dance, 2007]. FV is the gradients of the log-likelihood variable 𝜉 is often added to relax the linear separability constraint and make the A deep neural network (DNN) is a feed-forward artificial neural networks com-
	function log 𝑃 (In algorithm less susceptible to outliers. The optimization problem is formulated as: posed of input and output layers, hidden layers and nonlinear activation functions.
	other words, FV stands for the direction to which the parameters should change to move the input descriptor toward the generative model's modes. Because FV is the min 𝑤,𝑏 𝑁 There are many types of nonlinear activation functions, e.g. Sigmoid, Tanh, Softmax, 1 2 ||𝑤|| 2 + 𝐶 ∑ 𝑖=1 𝜉 𝑖 RELU [Nair and Hinton, 2010], etc.. A convolutional neural networks (CNN or Con-
	concatenation of different types of gradients, a normalization step is often needed. s.t. 𝑦 𝑖 (𝑤 𝑇 𝑥 𝑖 + 𝑏) ≥ 1 -𝜉 𝑖 , 𝑖 = 1, . . . , 𝑁 vNet) is a DNN equipped with additional layers dedicated for processing numerical
	𝜉 𝑖 ≥ 0, 𝑖 = 1, . . . , 𝑁. images, i.e. convolutional and pooling layers [LeCun et al., 1998]. A convolutional

  Within the training set, the responses for each image are split into individual words.

	Method	33-cat	397-cat
	Chance level	3.00	0.25
	Nearest Centroid 85.80	40.31
	ML Naive Bayes 91.97	55.86
	These word counts are combined and normalized across all images within a given
	class, to generate a word distribution histogram for each scene category. Within the
	test set, responses for each image are randomly grouped into chunks of 7 responses for
	that image, to simulate asking a handful of people at a time to provide a most typical
	action for each image. Responses within each chunk are then split into individual

  Our image classification pipeline follows standard approaches and consists of densely extracted local image features, a learned visual vocabulary and a feature encoding step. For local image features we use HOG2x2[START_REF] Vu | together with probabilities for the five most likely actions, predicted manually by people (red) and by an automatic method (green). 1-2 Illustration of contextual models[END_REF], SIFT[START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] and CSIFT [Abdel-Hakim and Farag, 2006, van de Sande et al., 2011a] descriptors. Descriptor dimension is reduced by PCA. For the encoding phase we consider two popular encoding techniques: histogram encoding (BoW) and Fisher Vector encoding (FV). To exploit spatial information, we apply Spatial Pyramid framework [Lazebnik et al., 2006], using grids of size 1x1, 2x2, 3x1. Each grid cell is represented either by BoW or FV vectors. The resulting vectors are normalized and concatenated to create the final representation. In the rest of this section we use the format <descriptor >_<encoding technique > to denote image representation techniques, e.g. CSIFT_FV as Fisher Vector encoding for CSIFT descriptors.

	Classification.
	4.4.2 Experimental results
	For training and testing the classifiers, we randomly divide the dataset into two
	equal parts. Our training and testing splits are balanced in the number of images per
	scene category. Our results for action prediction are summarized in Figures 4-5-4-7
	and Table 4.1.

For the classification, we train SVM classifiers using LIBSVM toolbox

[START_REF] Chang | Libsvm: A library for support vector machines[END_REF]

. Linear kernels are used for image representation by Fisher Vector. For the histogram representation (SIFT_BoW /CSIFT_BoW), we use 𝜒 2 kernel

[START_REF] Zhang | Local features and kernels for classification of texture and object categories: A comprehensive study[END_REF]

. With HOG_BoW, we exploit Histogram Intersection kernel. Training by SVM, we can boost up the performance by simply using a linear combination of kernels. In our experiments, we aggregate kernels with equal weights.

Table 4 .

 4 1. The table shows action prediction mAPs of two proposed training schemes combined with different image representation techniques. By comparing results of the two schemes, as shown in two columns S1 and S2, we can conclude that learning action classifiers directly achieve better prediction performance than aggregating multiple scene classifiers. This improvement can be attributed to sharing similar functional properties across different scene classes. In terms of image representation techniques, Fisher Vector encoding

	Action	Precision Recall	Correct Predictions	Most Confident False Positives	Least Confident False Negatives
	take
	pic-
	ture
	hike
	pray
	learn
	bowl
	watch
	sleep
	wash
	dish

Table 4 .

 4 1 -Scene-based outdoor (cyan) and indor (orange) action prediction results with different approaches. Note that mAP at chance level is 6.32% (outdoor) an d 4.24% (indoor). S1 and S2 columns respectively show classification mAP (%) of the two aforementioned training schemes. Column (S1-S2) shows the different mAP between two schemes. We observe consistently better performance of scheme

	Method	S1	S2	S1-S2	S1	S2	S1-S2
	SIFT_BoW	40.92	40.68	0.24	31.75	28.71	3.04
	SIFT_FV	41.15	34.51	6.64	31.04	27.13	3.91
	CSIFT_BoW	47.78	44.43	3.35	32.53	27.90	4.63
	CSIFT_FV	49.52	41.65	7.87	36.29	29.70	6.59
	HOG_BoW	47.03	45.93	1.10	37.91	35.50	2.41
	HOG_FV	52.66	47.75	4.91	42.78	43.89	-1.11
	HOG_BoW+ SIFT_FV+ CSIFT_FV	56.60	50.06	6.54	46.11	40.04	6.07
	HOG_BoW+ HOG_FV+ CSIFT_FV	60.99 54.25	6.74	52.09 45.98	6.11
	SIFT_FV+ HOG_FV+ CSIFT_FV	56.48	49.61	6.87	45.76	41.41	4.35

S1: directly training binary action classifiers over scheme S2: aggregating scene classifiers.

  al., 2011], we construct a joint score function 𝑆(y; w) that ties together the labels of candidates in the same image: 𝑈 maps the feature vector f 𝑖 of a candidate 𝑖 to the value of the corresponding unary potential, i.e. 𝜃 𝑈 𝑖 = 𝜙 𝑈 (f 𝑖 , w 𝑈 ). The pairwise network 𝜙 𝑃 maps the concatenated feature vectors of its two candidates to a vector 𝜃 𝑃 𝑖𝑗 where the 𝑘-th component 𝜃 𝑃 𝑖𝑗,𝑘 corresponds to the one of 𝐾 cluster indices, i.e. 𝜃 𝑃 𝑖𝑗,𝑘 𝑖𝑗 = 𝜙 𝑃 𝑘 𝑖𝑗 (f 𝑖 , f 𝑗 , w 𝑃 ). 𝑈 and w 𝑃 are the trainable parameters of the UN and PN, correspondingly.In our experiments we found the following architectures to work best. The FE was of the same structure as our Local model (based on the network of[START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF]) leading to 2048 features. In both UN and PN we use just one fully-connected layer. Addition of more hidden layers did not improve results.

	Precision-recall evaluation. Object detection methods are typically evaluated
	𝑆(y; w) = in terms of precision-recall (PR) and average precision (AP) values. To construct ∑ 𝜃 𝑈 𝑖 (𝑦 𝑖 ; w) + ∑ 𝜃 𝑃 𝑖𝑗 (𝑦 𝑖 , 𝑦 𝑗 , 𝑘 𝑖𝑗 ; w), (5.2)
	𝑖∈𝒱 the precision-recall curve given the joint score (5.3), we follow the approach of [Desai (𝑖,𝑗)∈ℰ
	where w denotes trainable parameters, 𝜃 𝑈 𝑖 and 𝜃 𝑃 𝑖𝑗 are unary and pairwise potentials et al., 2011]. For each candidate bounding box 𝑖, we compute an individual score 𝑠 𝑖 (w)
	depending on w, and y = (𝑦 𝑖 ) 𝑖∈𝒱 is a vector of all binary variables. defined as the difference of the max-marginals of the joint score
	Note, that different values of potentials in (5.2) can lead to exactly the same score
	function 𝑆. We rewrite Eq. (5.2) in the more compact form (the set of all representable 𝑠 𝑖 (w) = max y:𝑦 𝑖 =1 𝑆(y; w) -max y:𝑦 𝑖 =0 𝑆(y; w).
	functions of binary variables stays the same):
	𝑆(y; w) =	∑	𝑦 𝑖 𝜃 𝑈 𝑖 (w) +	∑	𝑦 𝑖 𝑦 𝑗 𝜃 𝑃 𝑖𝑗,𝑘 𝑖𝑗 (w)	(5.3)
		𝑖∈𝒱		(𝑖,𝑗)∈ℰ
	where unary potentials 𝜃 𝑈 𝑖 and pairwise potentials 𝜃 𝑃 𝑖𝑗,𝑘 𝑖𝑗	are represented by real values.

Connecting the score function and the image. Now we connect the image with potentials of the score function (5.3) using several feed-forward neural networks.

First, from the Local model described in Section 5.1.1 we create a feature extractor (FE), i.e. a function 𝜙 𝐸 that constructs feature vector f 𝑖 for the image data x 𝑖 of candidate 𝑖: f 𝑖 = 𝜙 𝐸 (x 𝑖 , w 𝐸 ). Here w 𝐸 is a vector of trainable parameters of FE.

To connect features f 𝑖 with potentials in (5.3) we construct two additional feedforward networks: the unary network (UN) and the pairwise network (PN). The unary network 𝜙 Vectors w

  Back-propagation of the gradient. The next step of the back-propagation procedure is to compute the derivatives of the loss w.r.t. parameters of the UN and PN

	get
	𝑑ℓ 𝑑𝜃 𝑈 𝑝	=	∑ 𝑖∈𝒱	𝑑ℓ 𝑑𝑠 𝑖	𝑑𝑠 𝑖 𝑑𝜃 𝑈 𝑝	,	𝑑ℓ 𝑑𝜃 𝑃 𝑝𝑞,𝑘	=	∑ 𝑖∈𝒱	𝑑ℓ 𝑑𝑠 𝑖	𝑑𝑠 𝑖 𝑑𝜃 𝑃 𝑝𝑞,𝑘
	𝑑ℓ 𝑑w 𝑈 = ∑ 𝑖∈𝒱	𝑑ℓ 𝑑𝜃 𝑈 𝑖	𝑑𝜃 𝑈 𝑖 𝑑w 𝑈 ,	𝑑ℓ 𝑑w 𝑃 = ∑ (𝑖,𝑗)∈ℰ	𝐾 ∑ 𝑘=1	𝑑ℓ 𝑑𝜃 𝑃 𝑖𝑗,𝑘	𝑑𝜃 𝑃 𝑖𝑗,𝑘 𝑑w 𝑃	(5.7)
	and w.r.t. the output of the feature extractor
	𝑑ℓ 𝑑f 𝑖	=	𝑑ℓ 𝑑𝜃 𝑈 𝑖	𝑑𝜃 𝑈 𝑖 𝑑f 𝑖	+	∑ 𝑗:(𝑖,𝑗)∈ℰ	𝑑ℓ 𝑑𝜃 𝑃 𝑖𝑗,𝑘 𝑖𝑗	𝑑𝜃 𝑃 𝑖𝑗,𝑘 𝑖𝑗 𝑑f 𝑖
									+	∑ 𝑗:(𝑗,𝑖)∈ℰ	𝑑ℓ 𝑑𝜃 𝑃 𝑗𝑖,𝑘 𝑗𝑖	𝑑𝜃 𝑃 𝑗𝑖,𝑘 𝑗𝑖 𝑑f 𝑖	.

𝑖,𝑡 = arg max y:𝑦 𝑖 =𝑡 𝑆(y; w) for 𝑡 ∈ {0, 1}. Here, [•] is the Iverson bracket notation. Combining the two derivatives via the chain rule we

.

  To select the "good" candidates out of the thousands produced by the selective search[START_REF] Jasper Rr Uijlings | Selective search for object recognition[END_REF] we use the non-maximum suppression (with threshold 0.3) on top of the scores provided by the Local model. We find that 16 candidates per image produced this way provide good balance between quality and speed.To construct clusters of candidate pair (edges) incorporating the layout information we use the three features representing the vertical and horizontal displacements and the ratio of the candidate sizes. To be precise, if the position of each candidate is defined by a tuple (𝑥 𝑖 , 𝑦 𝑖 , 𝑤 𝑖 , ℎ 𝑖 ) we define the size of the candidate as𝑠 𝑖 = (𝑤 𝑖 + ℎ 𝑖 )/2,its horizontal position as 𝑥 𝑐 𝑖 = 𝑥 𝑖 + 𝑤 𝑖 /2 and its vertical position as 𝑦 𝑐 𝑖 = 𝑦 𝑖 + ℎ 𝑖 /2. For the two candidates sorted such that 𝑥 𝑖 ≤ 𝑥 𝑗 we compute the features as follows: 𝑓 1 𝑖𝑗 = log(𝑠 𝑖 /𝑠2), 𝑓 2 𝑖𝑗 = 𝜙((𝑥 𝑐 𝑗 -𝑥 𝑐 𝑖 )/𝑠 𝑖 ), 𝑓 3 𝑖𝑗 = 𝜙((𝑦 𝑐 𝑗 -𝑦 𝑐 𝑖 )/𝑠 𝑖 ), where 𝜙(𝑥) = sign(𝑥) log(|𝑥| + 1). All the three features are normalized to have zero mean and unit standard deviation on the training set. We find that increasing number of clusters beyond 20 does not improve the performance.

Table 5 .

 5 this experiment we analyze the amount of training data required to train our models. Our full training set is constructed from 15 movies. We also examine the use of smaller training sets corresponding to the first 8 movies and the first 4 movies 3 -Performance of models trained on the training sets of different sizes. We report % AP for each test set.

		AlexNet	Oquab	VGG-S	verydeep-16
	AP Train speed Test speed	76.3 445 1490	76.7 284 980	77.2 147 510	78.5 30 74
	Table 5.2 -Performance (% AP) of Local models of different architectures on the
	HollywoodHeads validation set. Bottom lines report training and testing speed, mea-
	sured by the number of image patches processed per second.
	Test set	4 movies	8 movies	15 movies
	Casablanca HollywoodHeads TVHI	51.2 63.3 88.6	62.5 67.7 88.8	72.7 72.7 89.8

of the full training set respectively. We use each training set to train parameters of our full model and evaluate it on three datasets. Corresponding results are reported in Table

Table 5

 5

	.4 presents results of

Table 5 .

 5 4 -Performance of the R-CNN method and of our Local model (% AP) on the test set of HollywoodHeads with different percentage of candidates remaining after filtering using the Global model.

  al., 2016a], we generate tube proposals 2. We use 𝑁 𝑏 = 16 and 𝑁 ℎ = 4 in our experiments HollywoodHeads 4 is a recent large-scale dataset for person head detection with video clips from 21 movies. The original dataset provides head annotation for all the frames. To train tube models in this work, we extend the annotation by combining detections to form tracks. Our extended HollywoodHeads dataset contains 331 746 video frames with complete annotation of head tracks, split into 216 719, 67 181 and 47 846 frames for training, validation and test respectively. We have removed 10 test frames from the original test set, because they belonged to short video clips of length less than 10 frames. in Section 5.3.3 with the annotation of missing ground-truth heads.

	from high-scoring detections produced by our Fast R-CNN baseline	3 .
	6.3 Experiments
	In this section, we present and analyze experimental results of our method. We
	first introduce video datasets for object detection in Section 6.3.1. We then clarify
	our experimental setup in Section 6.3.2. Section 6.3.3 evaluates the quality of tube
	proposals. Section 6.3.4 presents our main results for object detection and compares
	them to the state of the art. In Section 6.3.5 we present an ablation study analyzing
	parameters and design choices of our method.
	6.3.1 Datasets
	HollywoodHeads dataset. For evaluation and comparison to the previous work, we keep the test set of [Vu et al.,
	2015] with 1 302 annotated frames 5 .
	Occlusions are among the most challenging factors for object detection. To study
	the detection performance under occlusions, we select a subset of video clips, named
	HollywoodHeads-Hard, with partially visible heads. HollywoodHeads-Hard is auto-
	matically composed from clips with multiple annotated heads, where ground-truth
	bounding boxes have significant overlap on the horizontal axis. In total, we obtain
	266 difficult clips and select one frame for testing from each of them.
	Casablanca dataset. In addition to HollywoodHeads, we evaluate head detection
	on the Casablanca dataset. This dataset was introduced in [Ren, 2008] and extended
	3. This baseline is described in details in Section 6.3.4
	4. http://www.di.ens.fr/willow/research/headdetection/ 5.
	86

Table 6 .

 6 1 -Evaluation of tube proposals in terms of tube-recall and box-recall onHollywoodHeads and VID datasets. Avg.#TP stands for the average number of tube proposals per chunk of consecutive frames.

	ImageNet VID

Table 6 .

 6 5 -Detection performance mAP (%) on HollywoodHeads and ImageNet VID dataset. CN and RN and HN stands for CaffeNet, Resnet and hard negative mining respectively.

	al., 2010], we use Average Precision (AP)

Table 6 .

 6 6 -Tube-CNN performance for different values of T.

	Tube length	T=1	T=5	T=10	T=15	T=20
	AP (%)	71.2	76.6		76.9 73.4 70.1
	Box-CNN	fc7 max	fc7 avg	Tube-CNN
	76.4	77.0		77.3	79.1

Table 6 .

 6 7 -Aggregation of Box-CNN and Tube-CNN.Advantage of feature-level aggregation. In this experiment, we show the advantages of early feature aggregation with TOI pooling and learning features for tube classification. We compare our Tube-CNN framework with baselines of temporal aggregation of single-frame image features. In particular, we use our best CaffeNetbased single frame Box-CNN detector to extract FC7 features on all the frames of training tubes and train a linear SVM classifier based on the temporally aggregated features. SVM parameters for each setup are chosen with 5-fold cross-validation. In Table6.7 we report results for max and average aggregation baselines. The improve-

	ment of Tube-CNN over baselines confirms the advantage of jointly learning features
	and their temporal aggregation as proposed in this work.
	Running time. Table 6.8 compares running times of object detectors (in frames
	per second) for Tube-CNN on SS-Track and TPN proposals. To obtain results for SS-
	Track, we accumulate the time required for per-frame Selective Search proposals, KLT
	point tracking and Tube-CNN evaluation. Results for TPN are obtained by summing
	times for the evaluation of the TPN network, applying tube-NMS and evaluating
	Tube-CNN. When using ResNet-based Tube-CNN object detectors, our method with
	100 TPN is about 28 times faster compared to Tube-CNN + SS-Track. The improve-
	ments of running time come from the efficient generation of TPN proposals and from
	the reduced number of proposals required for the Tube-CNN evaluation.

  , but use a fixed set of nearby frames and do not maintain causality or efficiency. In contrast, we propose a video representation that composes information across time in an online fashion (see Figure7-1), which is not only faster, but also enables predictive applications.

	et al. [2017b]We apply our video representation to object detection in Section 7.2.1 and evalu-
	ate it on the ImageNet VID Russakovsky et al. [2015] data set in Section 7.3.1. Our
	proposed architectures improve over per-frame baselines by up to 2.2% in mean aver-
	age precision (mAP).	1 We achieve state-of-the-art results close to flow-guided feature
	aggregation (FGFA) [Zhu et al., 2017b] without having access to future frames and
	with considerably lower runtime, while outperforming a causal variant of FGFA.
	In recent years, object detection in videos has attracted significant interest with
	benchmarks such as ImageNet VID [Russakovsky et al., 2015] or Youtube-8M [Abu-Sections 7.2.3 and 7.3.2 demonstrate this for the novel application of anticipating
	El-Haija et al., 2016]. A popular approach has been to detect bounding boxes inde-objects in future frames.
	pendently for each frame using CNN-based methods [Lin et al., 2017a,b, Liu et al.,
	2016, Ren et al., 2015], followed by temporal reasoning in terms of tracking [Kang
	et al., 2016b], re-scoring detections [Feichtenhofer et al., 2017b] and performing non-
	maximum suppression [Han et al., 2016]. While such methods improve over per-frame
	baselines, we here wish to investigate alternative and more powerful strategies to ag-
	gregate temporal information for object detection. A few recent works temporally
	aggregate features to improve representation power Feichtenhofer et al. [2017b], Zhu

A key benefit of the online nature of our video representation is that it imparts predictive abilities, which enables novel applications. First, in Section 7.2.2, we enhance the accuracy of an online real-time detector, by leveraging a stronger but less efficient detector in another thread. While the strong detector lags due to higher latency, our memory warping enables propagating and aligning its representation with the real-time detector, boosting the accuracy of the latter by more than 10% mAP, with no impact on speed or online operation (see

Section 7.3.3)

. This is non-trivial, since parallelizing standard detectors in an online setup is not straightforward. Next, our predictive warping of video representations enables anticipating features in future frames, which allows solving visual tasks without actually observing future images.

  is the bilinear sampling function and 𝒟 (𝑡,𝑡-1) ∈ R ℎ ℱ ×𝑤 ℱ ×2 is a displacement (or flow) field between frames 𝑡 and 𝑡-1, which is estimated by a CNN with parameters 𝛩 𝒟 . This CNN is a pre-trained FlowNet[START_REF] Dosovitskiy | FlowNet: Learning Optical Flow With Convolutional Networks[END_REF], which takes images

		al., 2017b], we use
	bilinear sampling to implement this transformation,
	M𝑡 = 𝜙(ℳ 𝑡-1 , 𝒟 (𝑡,𝑡-1) ) ,	(7.1)

where 𝜙(•) ℐ 𝑡 and ℐ 𝑡-1 as input and predicts the displacement, but we fine-tune the parameters 𝛩 𝒟 for the task at hand. Note that for fast computation of the displacement field, we feed FlowNet with half-resolution images and up-scale the displacement field. Also note that in the absence of ground truth data for the displacement field, this CNN predicts displacements suitable for the task at hand, which is demonstrated in Section 7.3.1.

  While our representation ℳ is generic and can be used for various tasks, we here focus on object detection in video. Modern object detectors such as Faster-RCNN[START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF], R-FCN[START_REF] Dai | R-FCN: Object detection via regionbased fully convolutional networks[END_REF], SSD[START_REF] Liu | SSD: Single Shot MultiBox Detector[END_REF] or Reti-naNet[Lin et al., 2017b] share a similar high-level structure and rely on a convolutional neural network to extract features ℱ from a single image. The detectionspecific modules applied on top of ℱ define the differences between the detectors,

e.g., proposal-based

[START_REF] Shaoqing Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF][START_REF] Dai | R-FCN: Object detection via regionbased fully convolutional networks[END_REF]

or proposal-free

[START_REF] Liu | SSD: Single Shot MultiBox Detector[END_REF], Lin et al., 2017b]

, making ℱ an interface between one generic module and detection-specific modules. Our proposed MemNet and ClockNet operate on ℱ and compute a novel feature representation ℳ, making our video representation appli-

Acknowledgements

Parents, sister, wife

The rest of this chapter is organized as follows. In Section 5.1, we describes our contextual CNN models. Implementation details are presented in Section 5.2. Section 5.3 introduces benchmark datasets followed by the presentation of experimental results in Section 5.4. Section 5.5 concludes the chapter.

Each feature map is flattened into a 9126-dimensional feature vector, which is then passed through FC layers to compute the classification score. We use the same spatial hyper-parameters of 6 × 6 for ROI pooling. Given an input tube proposal, the ROI pooling layer outputs two 9126-dimensional feature vectors corresponding to the two ends of the tube proposal. We pass the two feature vectors through FC layers to produce regression parameters for the beginning and the end of the tube.

Resnet-101. The TOI pooling and the ROI pooling layers are inserted between the third and the fourth blocks of the network, i.e. conv_4x and conv_5x of Table 1 in [START_REF] He | Deep residual learning for image recognition[END_REF]. In detail, 91 convolutional layers before and including conv_4x belong to CNN feature extractor block of Tube-CNN. All layers after and including conv_5x construct tube classification and tube regression blocks. The spatial extent of both TOI and ROI pooling layers has a fixed-size of 7 × 7. The temporal maxpooling is done similarly as for CaffeNet. Without being flattened, pooled feature maps are passed into 𝑐𝑜𝑛𝑣_5𝑥 followed by an average pooling layer. In our setting, output size of 𝑐𝑜𝑛𝑣_5𝑥 is different from the one of the original network. We adapt to this change by adjusting kernel size of the average pooling layer to 4 × 4. The last FC layers of tube classification and tube regression blocks take the average pooling output to produce final object score and position.

Training parameters. We optimize Tube-CNN and TPN with the stochastic gradient descent (SGD) algorithm with momentum 0.9 and weight decay 0.0005 on minibatches [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. The classification and regression are trained with the cross-entropy loss and smooth L1 loss [START_REF] Girshick | Fast r-cnn[END_REF], respectively. We fix tube length 𝑇 = 10 for all tube models. This design choice is analyzed in Section 6.3.5.

Hard negative mining. Hard negative mining enables training on difficult samples and has been proven effective for object detection with HOG and DPM models [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF]Triggs, 2005, Felzenszwalb et al., 2010a] and more recently with CNN methods [START_REF] Arandjelović | Netvlad: Cnn architecture for weakly supervised place recognition[END_REF][START_REF] Shrivastava | Training region-based object detectors with online hard example mining[END_REF]. In our experiments, we find that a few iterations of hard negative mining improves detection performance for all of our models. We here describe the procedure in an abstract manner. The term proposal stands for box proposal in case of Box-CNN, and tube proposal in case of Tube-CNN. The overlap ratio between proposal and ground-truth is defined separately for each model. An iteration of hard negative mining consists of three steps: (1) a forward pass on a subset of input proposals to find hard negatives; (2) composition of several training batches with hard negatives and random positives;

(3) training the network on the constructed batches. We define hard negatives as high-scoring false positives with no overlap to any ground-truth. The hard negative training batch contains 25% positives and 50% hard negatives at most. The rest of the batch is composed of negatives having overlap ratio to ground-truth in the range [0.1, 0.5). Section 6.3.5 analyzes the effect of hard negative mining on the detection performance.

Evaluation of tube proposals

In this section, we evaluate the quality of tube proposals produced by different methods on HollywoodHeads and ImageNet VID datasets.

Tube proposal baselines. Tube proposals have been previously proposed for the task of action detection in video. We here evaluate and compare two state-of-the-art methods for action proposals [START_REF] Jc Gemert | APT: Action localization Proposals from dense Trajectories[END_REF][START_REF] Oneata | Spatio-temporal object detection proposals[END_REF]. For the 3D-Proposal method [START_REF] Oneata | Spatio-temporal object detection proposals[END_REF], we extract 10, 000 tube proposals for every shots of 10 frames. For APT [START_REF] Jc Gemert | APT: Action localization Proposals from dense Trajectories[END_REF], we use longer shots due to the length constraint imposed by the method.

Evaluation of recall. Object proposals should maximize the coverage of groundtruth object bounding boxes. We evaluate the quality of tube proposals by examining their recall on both frame level and video level. We refer to the two measures as boxrecall and tube-recall, respectively. Tube-recall is the percentage of ground-truth tubes having tube overlap-ratio more than 0.5 with at least one tube proposal (tube overlapratio is defined in Section 6.1.2). Baselines. We compare our method to the single-frame Fast R-CNN object detectors [START_REF] Girshick | Fast r-cnn[END_REF] which we re-train on our data. We refer to such detectors as Box-CNN to contrast with the tube-level models. Box-CNNs are trained on single frames of training videos and frame-level annotations.

Single-frame object detectors often produce noisy detections along object tracks [Kang et al., 2016b]. A common practice to improve object detection in video is to link detections on consecutive frames with tracking-by-detection methods [START_REF] Everingham | Hello! my name is... buffy"automatic naming of characters in tv video[END_REF][START_REF] Pirsiavash | Globally-optimal greedy algorithms for tracking a variable number of objects[END_REF]. We here follow the procedure in [START_REF] Everingham | Hello! my name is... buffy"automatic naming of characters in tv video[END_REF]] and link object detections across frames based on the consensus of KLT point tracks. This procedure enables to overcome some failures of object detectors by bridging the gaps in object tracks along time and by discarding short tracks. We apply tracking-by-detection to the Box-CNN method and denote this strong baseline as "Box-CNN + track".

Head detection result. In the first and second columns of Table 6

Ablation study

In this section, we analyze design choices and parameters of our method. We also investigate and demonstrate advantages of TPN in terms of computational complexity. For the ImageNet VID dataset we only report results of models based on Resnet-101.

We train and test Tube-CNN models using tube proposals generated by our TPN model and by tracking box proposals (i.e. SS-Track). In Table 6.5, we denote these two Tube-CNN models as "Tube-CNN + SS-Track" and "Tube-CNN + TPN" respectively. In both cases, Tube-CNN models outperform Box-CNN and other baselines.

The main advantage of TPN is in test time, which grows linearly with the number of proposals. Using a smaller set of proposals, Tube-CNN + TPN is able to achieve similar (or even better) results compared to Tube-CNN + SS-Track. This together Chapter 7

Long-Term Representation Learning for Efficient Object Detection in Videos

In this chapter, we continue addressing the problem of object detection in videos.

Different to the approaches presented in Chapter 5 and Chapter 6, this work explores the feature learning aspect in videos. We aim at learning good video representation which can both improve detection performance and increase the running speed.

Motion is an important cue enabling the human visual system to perceive its environments [START_REF] Cutting | Perception with an Eye for Motion[END_REF][START_REF] Gibson | The Ecological Approach to Visual Perception[END_REF]. By encapsulating motion information, video provides a rich medium for computer vision to understand and analyze the visual world. While the advent of convolutional neural networks (CNNs) has led to rapid improvements in learning spatial features, a persistent challenge remains to learn efficient representations that derive significant benefits from long-term temporal information in videos.

In this work we aim to learn video representations that incorporate multi-scale information on longer time horizons and design practical frameworks that achieve accuracy, efficiency and predictive power. While our frameworks are applicable to diverse problems, we demonstrate its application to the problem of object detection in videos. Motivated by the Gestalt principle of common fate [Ellis, 1938, Wertheimer, at frame 𝑡, which we found to improve results by a few percentage points.

Extending the temporal scale

The basic MemNet operates on just a single temporal scale, which has limited capability to leverage information at a larger temporal horizon. While, in theory, information from the whole video sequence is contained in the feature representation of the current frame 𝑡, this portion can be vanishingly small, particularly for the aggregation function relying on the exponential decay.

We Note that when 𝐾 = 1, the basic MemNet is obtained.

In order to exchange information across the different time scales 𝑘, we aggregate all memory maps at a single frame 𝑡 by simply averaging them, i.e., ℳ 𝑡 = 1

As with the feature map aggregation in the basic MemNet, different strategies for combining feature maps are possible. We chose the simpler parameter-free averaging, as a more complex learning-based weighting scheme did not show any performance gains. The aggregated memory ℳ 𝑡 can then be used as input to any task-specific modules.

Discussion

Our proposed video representations have a simple and intuitive structure, can be trained end-to-end and fulfill the basic requirements for a fast and causal system that can be applied to videos in any real-world application. In contrast to FGFA Zhu et al.

[2017b], the proposed model does not look at future frames and is also not limited to a specific temporal horizon in the past, rather can carry information from the whole

Object detection in videos

Object detection in videos aims at localizing objects in every video frame, i.e., estimating bounding boxes around objects associated with a confidence score.

Evaluation metrics: We measure detection performance as mean average precision (mAP) over all object classes, where we additionally differentiate between fast, medium and slowly moving objects using the subsets of videos introduced in [Zhu et al., 2017b]. We also measure the average runtime per frame in milliseconds (ms)

for each model (using the same framework and GPU setup).

Baselines: Main results: to process an incoming frame.

We also note that the computation time of Causal FGFA can be reduced by aggregating displacement fields in an online manner, see Cau. FGFA* in compared the memory module at "conv3", "conv4", "conv5", and a combination of the latter two. Combining the combination of feature maps outperforms the lowerlevel feature maps but using the highest level representation, i.e., "conv5", still gives the overall best results.

Fine-tuning FlowNet: Finally, we want to understand the effect of fine-tuning FlowNet during training the video representation for the detection task. Similar to [Zhu et al., 2017b], we observe a performance drop in mAP if FlowNet is not fine-tuned. Given that displacement fields after fine-tuning are apparently better for the detection task, we visualize the difference for some examples in Figure 78. One difference that we can observe is that the object tends to move as a whole and ignores the motion of individual parts.

Propagating and anticipating features

In the previous experiment, the appearance features of the current frame are always available and the main purpose was to analyze the influence of additional information from the past. In this experiment, we want to give more insights into the quality of the feature propagation and feature anticipation, which has several applications as discussed in Section 7.2. We also compare our results to the box-propagating baselines. detected bounding boxes at frame 𝑡 -𝛿 and propagates them with the computed displacement fields to frame 𝑡. The mean displacement inside a bounding box serves as the translational vector. Comparing the anticipation results with the propagation model in Table 7. 3, we can see that our flow extrapolation strategy works well for MemNet which has short temporal stride, although the runtime speed drops due to our current non-optimized implementation of flow extrapolation. Reversely, on ClockNet memories, feature anticipation performs worse than feature propagation. One explanation is that quality of the extrapolated flows is heavily degraded with the long temporal strides of ClockNet.

Methods

Fast detection by propagating strong features

In this section we analyze the application described in Section 7.2.2 and Figure 7-5. A fast but weak object detector is running in the main thread providing detection output at every frame, while leveraging features from a strong but slow feature extractor. In order to use the strong features, they have to be propagated to the current frame, i.e., aligned over time, to compensate for the delay 𝛥. Quantitative results: Comparison to two-thread baselines: Recall that achieving speed-up with two GPUs in a real-time (data-streaming) setting is non-trivial as parallelizing frames over multiple GPUs is not possible. We still evaluate alternative baselines that leverage two GPUs. First, a trivial two times speed-up can be achieved by splitting the image into two halves, running the detector individually and merging the detections before NMS, denoted "(split)" in Table 7.4. While this gives the expected speed-up, the performance drop is significant, which can be explained by the fact that objects in Imagenet-VID are mostly centered, thus effectively truncating them. The second .

Abstract

In this thesis, we address person detection and action prediction in visual data. We develop models that learn representations for visual data and the structure in the output space while making use of contextual cues and temporal consistency. We also propose a predictive model to anticipate person's attention in given static scenes.

In the first part of the thesis, we explores the strong association between scene categories and actions. Based on that understanding, we formulate a new task of predicting human actions in static scenes. To train and evaluate the proposed model, we collect a new dataset of scene-action associations, named SUN Action dataset. The success of this task enables potential applications such as affordance geo-localization.

The second part of the thesis is focused on person and generic object detection in videos. First, we construct contextual models to enhance person detection in individual frames. We train and evaluate our method on our new HollywoodHeads dataset with annotated human heads in movies. Our models consistently improve detection performance over baseline detectors. Second, we introduce a novel convolutional neural network architecture operating on short clips of frames to leverage temporal consistency and to learn spatio-temporal representations. By empirical experiments, we demonstrate the benefit of our spatio-temporal representations for object detection in videos. Last, we learn video representations that incorporate multiscale information on coarse time scales and design practical frameworks that achieve accuracy, efficiency and predictive power. Compared to per-frame features, our video representations show best detection improvement on frames degraded by fast motions.

Keywords

Scene understanding, action prediction, interaction analysis, spatio-temporal visual representation, object detection in videos, deep convolutional neural networks, representation learning.