Vicky Kalogeiton

Philippe Weinzaepfel

Vittorio Ferrari

Cordelia Schmid

Action Tubelet Detector for Spatio-Temporal Action Localization

 datasets, in particular at high overlap thresholds.

Introduction

Action localization is one of the key elements to video understanding. It has been an active research topic for the past years due to various applications, e.g. video surveillance [START_REF] Hu | A survey on visual surveillance of object motion and behaviors[END_REF][START_REF] Oh | A large-scale benchmark dataset for event recognition in surveillance video[END_REF] or video captioning [START_REF] Venugopalan | Sequence to sequence-video to text[END_REF][START_REF] Yao | Describing videos by exploiting temporal structure[END_REF]. Action localization focuses both on classifying the actions present in a video and on localizing them in space and time. Action localization task faces significant challenges, e.g. intra-class variability, cluttered background, low quality video data, occlusion, changes in viewpoint. Recently, Convolutional Neural Networks (CNNs) have proven well adapted for action localization, as they provide robust representations of video frames. Indeed, most state-of-the-art action localization approaches [START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF]29,[START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF] are based on CNN object detectors [START_REF] Liu | SSD: Single shot multibox detector[END_REF][START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] that detect human actions at the 1 Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France. 2 University of Edinburgh 3 Naver Labs Europe standing up?

si,ng down? frame level. Then, they either link frame-level detections or track them over time to create spatio-temporal tubes. Although these action localization methods have achieved remarkable results [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF], they do not exploit the temporal continuity of videos as they treat the video frames as a set of independent images on which a detector is applied independently. Processing frames individually is not optimal, as distinguishing actions from a single frame can be ambiguous, e.g. person sitting down or standing up (Figure 1). In this paper, we propose to surpass this limitation and treat a video as a sequence of frames. Modern object detectors for images, such as Faster R-CNN [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] and Single Shot MultiBox Detector (SSD) [START_REF] Liu | SSD: Single shot multibox detector[END_REF], proceed by classifying and regressing a set of anchor boxes to the ground-truth bounding box of the object. In this paper, we introduce a spatio-temporal tubelet extension of this design. Our Action Tubelet detector (ACT-detector) takes as input a short sequence of a fixed number of frames and outputs tubelets, i.e., sequences of bounding boxes over time (Figure 2). Our method considers densely sampled anchors of cuboid shape with various sizes and aspect ratios. At test time, we generate for each anchor cuboid a score for a given action and regressed coordinates transforming it into a tubelet. Importantly, the score and regression are based on convolutional feature maps from all frames in the sequence. While the anchor cuboids have fixed spatial extent across time, the tubelets change size, location and aspect ratio over time, following the actors. Here we build upon the SSD framework, but the proposed tubelet extension is applicable to other detectors based on anchor boxes, such as Faster R-CNN. Our experiments show that taking as input a sequence of frames improves: (a) action scoring, because the ambiguity between different actions reduces and (b) localization accuracy, because frames in a cuboid are regressed jointly and hence, they share information about the location of the actor in neighboring frames, see Figure 1. Our ACT-detector obtains state-of-the-art frame-mAP and video-mAP performance on the J-HMDB [START_REF] Jhuang | Towards understanding action recognition[END_REF] and UCF-101 [START_REF] Soomro | UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild[END_REF] datasets, in particular at high overlap thresholds. In summary, we make the following contributions:

• We introduce the ACT-detector, an action tubelet detector that proceeds by scoring and regressing anchor cuboids.

• We demonstrate that anchor cuboids can handle moving actors for sequences up to around 10 frames.

• We provide an extensive analysis demonstrating the clear benefit of leveraging sequences of frames instead of operating at the frame level.

The code of our ACT-detector is available at http://thoth.inrialpes.fr/src/ACTdetector.

Related work

Almost all recent works [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF]29,[START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF] for action localization build on CNN object detectors [START_REF] Liu | SSD: Single shot multibox detector[END_REF][START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF]. In the following, we review recent CNN object detectors and then state-of-the-art action localization approaches. Object detection with CNNs. Recent state-of-the-art object detectors [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF][START_REF] Liu | SSD: Single shot multibox detector[END_REF][START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF][START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] are based on CNNs. R-CNN [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] casts the object detection task as a region-proposal classification problem. Faster R-CNN [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] extends this approach by generating bounding box proposals with a fullyconvolutional Region Proposal Network (RPN). RPN considers a set of densely sampled anchor boxes, that are scored and regressed. Moreover, it shares convolutional features with proposal classification and regression branches. These branches operate on fixed-size features obtained using a Region-of-Interest (RoI) pooling layer. In a similar spirit, YOLO [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF] and SSD [START_REF] Liu | SSD: Single shot multibox detector[END_REF] also use a set of anchor boxes, which are directly classified and regressed without a RoI pooling layer. In YOLO, all scores and regressions are computed from the last convolutional feature maps, whereas SSD adapts the features to the size of the boxes. Features for predicting small-sized boxes come from early layers, and features for big boxes come from the latter layers, with larger receptive fields. All these object detectors rely on anchor boxes. In our work, we extend them to anchor cuboids leading to significant improvement for action localization. Action localization. Initial approaches for spatio-temporal action localization are extensions of the sliding window scheme [START_REF] Cao | Cross-dataset action detection[END_REF][START_REF] Laptev | Retrieving actions in movies[END_REF], requiring strong assumptions such as a cuboid shape, i.e., a fixed spatial extent of the actor across frames. Other methods extend object proposals to videos. Hundreds of action proposals are extracted per video given low-level cues, such as super-voxels [START_REF] Jain | Action localization with tubelets from motion[END_REF][START_REF] Oneata | Spatiotemporal object detection proposals[END_REF] or dense trajectories [START_REF] Chen | Action detection by implicit intentional motion clustering[END_REF][START_REF] Gemert | APT: Action localization proposals from dense trajectories[END_REF][START_REF] Puscas | Unsupervised tube extraction using transductive learning and dense trajectories[END_REF]. They then cast action localization as a proposal classification problem. More recently, some approaches [START_REF] Li | VideoLSTM convolves, attends and flows for action recognition[END_REF][START_REF] Wang | Actionness estimation using hybrid fully convolutional networks[END_REF][START_REF] Yu | Fast action proposals for human action detection and search[END_REF] rely on an actionness measure [START_REF] Chen | Actionness ranking with lattice conditional ordinal random fields[END_REF], i.e., a pixel-wise probability of containing any action. To estimate actionness, they use low-level cues such as optical flow [START_REF] Yu | Fast action proposals for human action detection and search[END_REF], CNNs with a two-stream fully-convolutional architecture [START_REF] Wang | Actionness estimation using hybrid fully convolutional networks[END_REF] or recurrent neural networks [START_REF] Wang | Actionness estimation using hybrid fully convolutional networks[END_REF]. They extract action tubes either by thresholding [START_REF] Li | VideoLSTM convolves, attends and flows for action recognition[END_REF] the actionness score or by using a maximum set coverage formulation [START_REF] Yu | Fast action proposals for human action detection and search[END_REF]. This, however, outputs only a rough localization of the action as it is based on noisy pixel-level maps.

Most recent approaches rely on object detectors trained to discriminate human action classes at the frame level. Gkioxari and Malik [START_REF] Gkioxari | Finding action tubes[END_REF] extend the R-CNN framework to a two-stream variant [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF], processing RGB and flow data separately. The resulting per-frame detections are then linked using dynamic programming with a cost function based on detection scores of the boxes and overlap between detections of consecutive frames. Weinzaepfel et al. [START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF] replace the linking algorithm by a tracking-by-detection method. More recently, two-stream Faster R-CNN was introduced by [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF]. Saha et al. [START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF] fuse the scores of both streams based on overlap between the appearance and the motion RPNs. Peng and Schmid [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF] combine proposals extracted from the two streams and then classify and regress them with fused RGB and multi-frame optical flow features. They also use multiple regions inside each action proposal and then link the detections across a video based on spatial overlap and classification score. Singh et al. [29] perform action localization in real-time using (a) the efficient SSD detector, (b) a fast method [START_REF] Kroeger | Fast optical flow using dense inverse search[END_REF] to estimate the optical flow for the motion stream, and (c) an online linking algorithm. All these approaches rely on detections at the frame level. In contrast, we build our ACT-detector by taking as input sequences of frames and demonstrate improved action scores and location accuracy over frame-level detections.

ACtion Tubelet (ACT) detector

We introduce the ACtion Tubelet detector (ACTdetector), an action tubelet approach for action localization in videos. The ACT-detector takes as input a sequence of K frames f 1 , ..., f K and outputs a list of spatio-temporal detections, each one being a tubelet, i.e., a sequence of bounding boxes, with one confidence score per action class. The idea of such an extension to videos could be applied on top of various state-of-the-art object detectors. Here, we apply our method on top of SSD, as it has lower runtime than other detectors, which makes it suitable for large video datasets. In this section, we first describe our fdafd4dfposed ACT-detector (Section 3.1), and then our full framework for video detection (Section 3.2). Finally, Section 3.3 describes our method for constructing action tubes.

ACT-detector

In this paper, we claim that action localization benefits from predicting tubelets taking as input a sequence of frames instead of operating at the frame level. Indeed, the appearance and even the motion may be ambiguous for a single frame. Considering more frames for predicting the scores reduces this ambiguity (Figure 1). Moreover, this allows to perform regression jointly over consecutive frames. Our ACT-detector builds upon SSD, see Figure 2 for an overview of the approach. In the following we review the SSD detector in details and then present our ACT-detector. SSD detector. The SSD detector (Single Shot MultiBox Detector) [START_REF] Liu | SSD: Single shot multibox detector[END_REF] performs object detection by considering a set of anchor boxes at different positions, scales and aspect ratios. Each of them is (a) scored for each object class and for a background class, and (b) regressed to better fit the object extent. SSD uses a fully convolutional architecture, without any object proposal step, enabling fast computation. Classification and regression are performed using different convolutional layers depending on the scale of the anchor box. Note that the receptive field of a neuron used to predict the classification scores and the regression of a given anchor box remains significantly larger than the box.

ACT-detector. Given a sequence of K frames, the ACTdetector computes convolutional features for each one. The weights of these convolutional features are shared among all input frames. We extend the anchor boxes of SSD to anchor cuboids by assuming that the spatial extent is fixed over time along the K frames. We then stack the corresponding convolutional features from each of K frames (Figure 2). The stacked features are the input of two convolutional layers, one for scoring action classes and one for regressing the anchor cuboids. For instance, when considering an anchor cuboid for which the prediction is based on the 'red' feature maps of Figure 2, the classification and regression are performed with convolutional layers that take as input the 'red' stacked feature maps from the K frames. The classification layer outputs for each anchor cuboid C + 1 scores: one per action class plus one for the background. This means that the tubelet classification is done based on the sequence of frames. The regression outputs 4 × K coordinates (4 for each of the K frames) for each anchor cuboid. Note that although all boxes in a tubelet are regressed jointly, they result in a different regression for each frame.

The initial anchor cuboids have a fixed spatial extent over time. In Section 4.2 we show experimentally that such anchor cuboids can handle moving actors for short sequences of frames. Note that the receptive field of the neurons used to score and regress an anchor cuboid is larger than its spatial extent. This allows us to base the prediction also on the context around the cuboid, i.e., with knowledge for actors that may move outside the cuboid. Moreover, the regression significantly deforms the cuboid shape. Even though anchor cuboids have fixed spatial extent, the tubelets obtained after regressing the 4×K coordinates do not. We display two examples in Figure 3 with the anchor cuboid (cyan boxes) and the resulting regressed tubelet (yellow boxes). Note how the regression outputs an accurate localization despite the change in aspect ratio of the action boxes across time. Training loss. For training, we consider only sequences of frames in which all frames contain the ground-truth action. As we want to learn action tubes, all positive and negative training data come from sequences in which actions occur. We exclude sequences in which the action starts or ends. Let A be the set of anchor cuboids. We denote by P the set of anchor cuboids for which at least one groundtruth tubelet has an overlap over 0.5, and by N the complementary set. Overlap between tubelets is measured by averaging the Intersection over Union (IoU) between boxes over K frames. Each anchor cuboid from P is assigned to ground-truth boxes with IoU over 0.5. More precisely, let x y ij ∈ {0, 1} be the binary variable whose value is 1 if and only if the anchor cuboid a i is assigned to the ground-truth tubelet g j of label y. The training loss L is defined as:

L = 1 N Lconf + Lreg , (1)
with N = i,j,y x y ij the number of positive assignments and Lconf (resp. Lreg) the confidence (resp. regression) loss as defined below.

The confidence loss is defined using a softmax loss. Let ĉy i be the predicted confidence score (after softmax) of an anchor a i for class y. The confidence loss is:

Lconf = - i∈P x y ij log (ĉ y i ) - i∈N log ĉ0 i . (2)
The regression loss is defined using a Smooth-L1 loss between the predicted regression and the ground-truth target. We regress an offset for the center (x, y) of each box in the tubelet, as well as for the width w and the height h. The regression loss is averaged over K frames. More precisely, let rx k i be the predicted regression for the x coordinate of anchor a i at frame f k and let g j be the ground-truth target. The regression loss is defined as:

Lreg = 1 K i∈P c∈{x,y,w,h} x y ij K k=1 SmoothL1 rc k i -g c k ij , with g x k ij = g x k j -a x k i a w k i g y k ij = g y k j -a y k i a h k i , g w k ij = log g w k j a w k i g h k ij = log   g h k j a h k i   .
(3)

Two-stream ACT-detector

Following standard practice for action localization [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF][START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF], we use a two-stream detector. We train an appearance detector, for which the input is a sequence of K consecutive RGB frames, and a motion detector, which takes as input the flow images [START_REF] Brox | High accuracy optical flow estimation based on a theory for warping[END_REF] obtained following [START_REF] Gkioxari | Finding action tubes[END_REF].

Each stream outputs a set of regressed tubelets with confidence scores, originating from the same set of anchor cuboids. For combining the two streams at test time we compare two approaches: union fusion and late fusion. For the union fusion [29], we consider the set union of the outputs from both streams: the tubelets from the RGB stream with their associated scores and the tubelets from the flow stream with their scores. For the late fusion [6], we average the scores from both streams for each anchor cuboid, as the set of anchors is the same for both streams. We keep the regressed tubelet from the RGB stream, as appearance is more relevant for regressing boxes, in particular for actions with limited motion. Our experiments show that late fusion outperforms the union fusion (Section 4.3).

From action tubelets to spatio-temporal tubes

For constructing action tubes, we build upon the frame linking algorithm of [29], as it is robust to missed detections and can generate tubes spanning different temporal extents of the video. We extend their algorithm from frame linking to tubelet linking and propose a temporal smoothing to build action tubes from the linked tubelets. The method is online and proceeds by iteratively adding tubelets to a set of links while processing the frames. In the following, t is a tubelet and L a link, i.e., a sequence of tubelets. Input tubelets. Given a video, we extract tubelets for each sequence of K frames. This means that consecutive tubelets overlap by K -1 frames. The computation of overlapping tubelets can be performed at an extremely low cost as the weights of the convolutional features are shared. We compute the convolutional features for each frame only once. For each sequence of frames, only the last layers that predict scores and regressions, given the stacked convolutional features (Figure 2), remain to be computed. For linking, we keep only the N = 10 highest scored tubelets for each class after non-maximum suppression (NMS) at a threshold 0.3 in each sequence of frames. Overlap between a link and a tubelet. Our linking algorithm relies on an overlap measure ov(L, t) between a link L and a tubelet t that temporally overlaps with the end of the link. We define the overlap between L and t as the overlap between the last tubelet of the link L and t. The overlap between two tubelets is defined as the average IoU between their boxes over overlapping frames. Initialization. In the first frame, a new link is started for each of the N tubelets. At a given frame, new links start from tubelets that are not associated to any existing link. Linking tubelets. Given a new frame f , we extend one by one in descending order of scores each of the existing links with one of the N tubelet candidates starting at this frame. The score of a link is defined as the average score of its tubelets. that meets the following criteria: (i) is not already selected by another link, (ii) has the highest score, and (iii) verifies ov(L, t) τ , with τ a given threshold. In our experiments we use τ = 0.2. Termination. Links stop when these criteria are not met for more than K -1 consecutive frames.

Temporal smoothing: from tubelet links to action tubes.

For each link L, we build an action tube, i.e., a sequence of bounding boxes. The score of a tube is set to the score of the link, i.e., the average score over the tubelets in the link.

To set the bounding boxes, note that we have multiple box candidates per frame as the tubelets are overlapping. One can simply use the box of the highest scored tubelet. Instead, we propose a temporal smoothing strategy. For each frame, we average the box coordinates of tubelets that pass through that frame. This allows us to build smooth tubes.

Temporal detection. The initialization and termination steps result in tubes spanning different temporal extents of the video. Each tube determines, thus, the start and end in time of the action it covers. No further processing is required for temporal localization.

Experimental results

In this section we study the effectiveness of our ACTdetector. After presenting the datasets used in our experiments (Section 4.1), we provide an analysis of our ACTdetector: we validate anchor cuboids (Section 4.2), evaluate input modalities (RGB and flow) and their fusion (Section 4.3), examine the impact of the length K of the se-quence of frames (Section 4.4), and present an error analysis (Section 4.5). We finally compare our method to the state of the art (Section 4.6).

Datasets, metrics and implementation details

Datasets. The UCF-Sports dataset [START_REF] Rodriguez | Action MACH: A spatio-temporal maximum average correlation height filter for action recognition[END_REF] contains 150 videos from 10 sports classes such as diving or running. The videos are trimmed to the action. We use the train/test split of [START_REF] Lan | Discriminative figure-centric models for joint action localization and recognition[END_REF].

The J-HMDB dataset [START_REF] Jhuang | Towards understanding action recognition[END_REF] contains 928 videos with 21 actions, including brush hair and climb stairs. The videos are trimmed to the action. We report results averaged on the three splits defined in [START_REF] Jhuang | Towards understanding action recognition[END_REF], unless stated otherwise.

The UCF-101 dataset [START_REF] Soomro | UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild[END_REF] contains spatio-temporal annotations for 24 sports classes in 3207 videos. The videos are not trimmed. Following [START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF][START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF], we report results for the first split only. Metrics. We use metrics at both frame and video level. Frame-level metrics allow us to compare the quality of the detections independently of the linking strategy. Metrics at the video level are the same as the ones at the frame level, replacing the Intersection-over-Union (IoU) between boxes by a spatio-temporal overlap between tubes, i.e., an average across time of the per-frame IoU [START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF][START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF]. To measure our performance at the frame level, we take into account the boxes originating from all tubelets that pass through the frame with their individual scores and perform NMS. In all cases, we only keep the detections with a score above 0.01.

We report frame and video mean Average Precision (mAP). A detection is correct if its IoU with a ground-truth box or tube is greater than 0.5 and its action label is correctly predicted [START_REF] Everingham | The PASCAL Visual Object Classes Challenge 2007[END_REF]. For each class, we compute the average precision (AP) and report the average over all classes.

To evaluate the localization accuracy of the detections, we report MABO (Mean Average Best Overlap) [START_REF] Uijlings | Selective search for object recognition[END_REF]. We compute the IoU between each ground-truth box (or tube) and our detections. For each ground truth box (or tube), we keep the overlap of the best overlapping detection (BO) and, for each class, we average over all boxes (or tubes) (ABO). The mean is computed over all classes (MABO).

To evaluate the quality of the detections in terms of scoring, we also measure classification accuracy. In each frame, assuming that the ground-truth localization is known, we compute class scores for each ground-truth box by averaging the scores from the detected boxes or tubelets (after regression) whose overlap with the ground-truth box of this frame is greater than 0.7. We then assign the class having the highest score to each of these boxes and measure the ratio of boxes that are correctly classified. Implementation details. We use VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] with ImageNet pre-training for both appearance and motion streams [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF]. Our frames are resized to 300 × 300. We use the same hard negative mining strategy as SSD [START_REF] Liu | SSD: Single shot multibox detector[END_REF], i.e., to avoid an unbalanced factor between positive and negative samples, only the hardest negatives up to a ratio of 3 negatives for 1 positive are kept in the loss. We perform data augmentation to the whole sequence of frames: photometric transformation, rescaling and cropping. Given the K parallel streams, the gradient of the shared convolutional layers is the sum over the K streams. We find that dividing the learning rate of the shared convolutional layers by K helps convergence, as it prevents large gradients.

Validation of anchor cuboids

This section demonstrates that an anchor cuboid can handle moving actions. We first measure how much the actors move in the training sets of the three action localization datasets by computing the mean motion overlap. For each box in a ground-truth tube, we measure its motion overlap: the overlap between this box and the ground-truth box n frames later for varying n. For each class, we compute the average motion overlap over all frames and we report the mean over all classes in Figure 5 (a). We observe that the motion overlap reduces as n increases, especially for UCF-Sports and UCF-101 for which the motion overlap for a gap of n = 10 frames is around 60%. This implies that there is still overlap between the ground-truth boxes that are separated by n = 10 frames. It also means that in many cases, this overlap is below 50% due to the motion of the actor.

In practice, we want to know if we have positive training anchor cuboids. Positive cuboids are the ones that have an overlap of at least 50% with a ground-truth tubelet; the overlap being the average IoU between boxes over the K frames in the sequence. Such cuboids are required for training the classifier and the regressor. Thus, we consider all possible training sequences and compute for each class the recall of the anchor cuboids with respect to the ground-truth tubelets, i.e., the ratio of ground-truth tubelets for which at least one anchor cuboid has an overlap over 0.5. We report the mean recall over the classes for varying IoU thresholds for the three datasets in Figure 5 (b-d). For all datasets, the recall at IoU= 0.5 remains 98% up to K = 6 and over 95% for K = 10. This confirms that cuboid-shaped anchors can be used in case of moving actors. When increasing K, for instance to 32, the recall starts dropping significantly.

Given that sequences of up to K = 10 frames result in high recall of the anchor cuboids, we examine in Sections 4.3 and 4.4 the performance of our tubelet detector for sequences of length ranging between K = 1 and K = 10.

Tubelet modality

In this section, we examine the impact of the RGB and flow modalities and their fusion on the performance of our ACT-detector. For all datasets, we examine the frame-mAP when using (i) only RGB data, (ii) only flow data, (iii) union of RGB + flow data [START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF], and (iv) late fusion of RGB + flow data for varying sequence length from 1 to 10 frames.

For all datasets and for all K, the RGB stream (blue line) outperforms the flow stream (red line), showing that appearance information is on average a more distinctive cue than motion (Figure 6). In all cases, using both modalities (green and black lines) improves the detection performance com- pared to using only one. We observe that late fusion of the scores (green line) performs consistently better than union fusion (black line), with a gain between 1% and 4% in terms of frame-mAP. This can be explained by the fact that union fusion considers a bigger set of detections without taking into account the similarity between appearance and motion detections. Instead, the late fusion re-scores every detection by taking into account both RGB and flow scores. Given that late fusion delivers the best performance, we use it in the remainder of this paper.

Tubelet length

In this section, we examine the impact of K. We consider K = 1 as the baseline, and we report results for our method with K = 2, 4, 6, 8, 10. We quantify the impact of K by measuring (i) the localization accuracy (MABO), (ii) the classification accuracy, (iii) the detection performance (frame-mAP), and (iv) the motion speed of actors. MABO. MABO allows us to examine the localization accuracy of the per-frame detections when varying K. Results are reported in Figure 7 (top). For all three datasets we observe that using sequences of frames (K > 1) leads to a significant improvement. In particular, MABO increases up to K = 4, and then remains almost constant up to K = 8 frames. For instance, MABO increases by 5% on UCF-Sports, 2% on J-HMDB and 5% on UCF-101 when using K = 6 instead of K = 1. This clearly demonstrates that performing detection at the sequence level results in more accurate localization, see Figure 3 for examples. Overall, we observe that K = 6 is one of the values for which MABO obtains excellent results for all datasets. Classification accuracy. We report classification accuracy on the three action localization datasets in Figure 7 (bottom). Using sequences of frames (K > 1) improves the classification accuracy of the detections for all datasets. For UCF-Sports, the accuracy keeps increasing with K, while for J-HMDB it remains almost constant after K = 6. For UCF-101, the accuracy increases when moving from K = 1 to K = 4 and after K = 8 it starts decreasing. Overall, using up to K = 10 frames improves performance over K = 1. This shows that the tubelet scoring improves the classification accuracy of the detections. Again, K = 6 is one of the values which results in excellent results for all datasets. Frame-mAP. Figure 6 shows the frame-mAP when training the ACT-detector with varying K. On all three datasets, we observe a gain up to 10% when increasing the tubelet length up to K = 6 or 8 frames depending on the dataset, compared to the standard baseline of per-frame detection. This result highlights the benefit of performing detection at the sequence level. For J-HMDB and UCF-101, we also observe a performance drop for K > 8, because regressing from anchor cuboids is harder as (a) the required transformation is larger when the actor moves, and (b) there are more training parameters for less positive samples, given that the recall of anchor cuboids decreases (Section 4.2).

The above results show that K = 6 gives overall good results. We use this value in the following sections. Figure 4 shows some qualitative examples comparing the performance for K = 1 and K = 6. We observe that our tubelets lead to less missed detections and to more accurate localization compared to per-frame detection (first and second rows). Moreover, our ACT-detector reduces labeling mistakes when one frame is not enough to disambiguate between classes. For instance, in the last row we predict the correct label catch, whereas in the third row there is a big variance in the labels (swing basketball, kick ball, catch).

Handling moving actors. To validate that our ACTdetector can handle moving actors, we measure frame-mAP with respect to the speed of the actor. We group actors into three categories (slow, medium, fast) with 1/3 of the data in each category. Speed is computed using the IoU of an actor with its instances in ±10 neighboring frames. Table 1 reports the frame-mAP at IoU = 0.5 for the three categories. For all datasets there is a clear gain between K = 1 and K = 6 for all speeds. In particular, for actors with fast motion the gain is +8% for UCF-Sports, +9% for J-HMDB, and +3% for UCF-101. This confirms that our tubelets can successfully handle large displacements. A potential explanation is the fact that the receptive fields are significantly larger than the the spatial extent of the anchor cuboid.

Error breakdown analysis

In this section, we examine the cause of errors in frame-mAP to better understand the reasons why our tubelets improve detection performance. More precisely, we consider five mutually exclusive factors and analyze which percentage of the mAP is lost due to each of them:

1. localization error E L : the detection is in a frame containing the correct class, but the localization is wrong, i.e., IoU < 0.5 with the ground-truth box. 2. classification error E C : the detection has IoU 0.5 with the ground-truth box of another action class. 3. time error E T : the detection is in an untrimmed video for the correct class, but the temporal extent of the action does not cover this frame. 4. other errors E O : the detection appears in a frame without the class, and has IoU < 0.5 with ground-truth boxes of any other class. 5. missed detections E M : we do not have a detection for a ground-truth box. The first four factors are categories of false positive detections, while E M refers to the ones we did not detect at all. For the first four factors, we follow the frame-mAP computation and measure the area under the curve when plotting the percentage of each category at all recall values. The missed detections (E M ) factor is computed by measuring the percentage of missed detections, i.e., ratio of groundtruth boxes for which there are no correct detections.

Figure 8 shows the percentage that each of these factors contributes to errors in the mAP for K = 1 and K = 6 with late fusion of RGB and flow as input modalities. For all datasets, we observe that when going from K = 1 to K = 6 there is almost no change in E L or in E O . In particular, for UCF-Sports and J-HMDB their values are extremely small even for K = 1. We also observe a significant decrease of E C between K = 1 and K = 6, in particular on the UCF-Sports and J-HMDB datasets. This highlights that including more frames facilitates the action classification task (Figure 1). This drop is lower on the UCF-101 dataset. This can be explained by the fact that most errors in this dataset come from false detections outside the tem-detector method UCF-Sports J-HMDB UCF-101 actionness [START_REF] Wang | Actionness estimation using hybrid fully convolutional networks[END_REF] -39. we report the results with and without their multi-region (+MR).

poral extent of the actions (E T ). Note that E T = 0 on UCF-Sports and J-HMDB, as these datasets are trimmed. For all datasets, a big gain comes from the missed detections E M : for K = 6 the percentage of missed detections drops significantly compared to K = 1. For instance, on J-HMDB the percentage of missed detections is reduced by a factor of 2. This clearly shows the ability of our proposed ACT-detector not only to better classify and localize (E C and MABO) actions but also to detect actions missed by the single-frame detector (see Figure 4).

Figure 1 .

 1 Figure 1. Understanding an action from a single frame can be ambiguous, e.g. sitting down or standing up; the action becomes clear when looking at a sequence of frames.

Figure 2 .

 2 Figure 2. Overview of our ACT-detector. Given a sequence of frames, we extract convolutional features with weights shared between frames. We stack the features from subsequent frames to predict scores and regress coordinates for the anchor cuboids (middle figure, blue color). Depending on the size of the anchors, the features come from different convolutional layers (left figure, color coded: yellow, red, purple, green). As output, we obtain tubelets (right figure, yellow color).

Figure 3 .

 3 Figure 3. Examples of regressed tubelets (yellow) from cuboids (cyan) in our ACT-detector. Note the accurate localization of the tubelet, despite the fact that the aspect ratio of the cuboid is changing over time.

Figure 4 .

 4 Figure 4. Examples when comparing per-frame (K = 1) and tubelet detections (K = 6). The yellow color represents the detections and their scores for the classes shown, the red color highlights errors due to missed detections (first row) or wrong labeling (third row) and the green color corresponds to correct labels. Our ACT-detector outputs one class label with one score per tubelet, we thus display it once.

Figure 5 .

 5 Figure 5. (a) Motion overlap: Mean motion overlap between a box in a ground-truth tube and its box n frames later for varying n. (b-d) Recall of the anchor cuboids for various IoU thresholds on the training set of three action localization datasets. The numbers in parenthesis indicate the recall at IoU = 0.5.

Figure 6 .

 6 Figure6. Frame-mAP of our ACT-detector on the three datasets when varying K for RGB data (blue line), flow (red line), union and late fusion of RGB + flow data (black and green lines, resp.).

Figure 7 .

 7 Figure 7. MABO (top) and classification accuracy (bottom) of our ACT-detector on the three datasets when varying K.

Figure 8 .

 8 Figure 8. Error analysis of our ACT-detector for K = 1 and K = 6 on three action localization datasets. We show frame-mAP and different sources of error, see Section 4.5 for details.

Table 1 .

 1 Frame-mAP for slow, medium and fast moving actors.

	K	UCF-Sports slow medium fast	J-HMDB (split 1) slow medium fast	UCF-101 (split 1) slow medium fast
	K = 1 79.5	84.0	68.1 61.2	55.5	49.0 69.6	73.5	67.3
	K = 6 85.5	89.7	76.8 69.8	66.9	58.0 75.4	78.5	70.7

Table 2 .

 2 Comparison of frame-mAP to the state of the art. For[START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF],

	9	-

Acknowledgments. This work was supported in part by the ERC grants ALLEGRO and VisCul, the MSR-Inria joint project, a Google research award, a Facebook gift, an Intel gift and an Amazon research award. We gratefully acknowledge the support of NVIDIA with the donation of GPUs.

Comparison to the state of the art

We compare our ACT-detector to the state of the art. Note that our results reported in this section are obtained by stacking 5 consecutive flow images [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF] as input to the motion stream, instead of just 1 for each of the K = 6 input frames. This variant brings about +1% frame-mAP. Frame-mAP. We report frame-mAP on the three datasets in Table 2. We compare our performance with late fusion of RGB+5flows when K = 6 to [START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF], that use a twostream R-CNN, and to [START_REF] Wang | Actionness estimation using hybrid fully convolutional networks[END_REF], which is based on actionness. We also compare to Peng and Schmid [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF] that build upon a two-stream Faster R-CNN with multiscale training and testing. We report results of [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF] with and without their multi-region approach. The latter case can be seen as the baseline Faster R-CNN with multiscale training and testing for K = 1. Our ACT-detector (i.e., with K = 6) brings a clear gain in frame-mAP, outperforming the state of the art on UCF-Sports, J-HMDB and UCF-101. We also observe that overall the performance of the baseline SSD (K = 1) is somewhat lower (by around 3 to 5%) than Faster R-CNN used by the state of the art [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF], see Figure 6. SSD, however, is much faster than Faster R-CNN, and therefore more suitable for large video datasets. Video-mAP. Table 3 reports the video-mAP results for our method and the state of the art at various IoU thresholds (0.2, 0.5, and 0.75). We also report results with the protocol 0.5:0.95 [START_REF] Lin | Microsoft COCO: Common objects in context[END_REF], which averages over multiple IoU thresholds, i.e., over 10 IoU thresholds between 0.5 and 0.95 with a step of 0.05. At rather low IoU thresholds (0.2, 0.5) on UCF-Sports and J-HMDB the performance of our ACTdetector is comparable to the state-of-the-art methods that rely on Faster R-CNN [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF] 3. Comparison of video-mAP to the state of the art at various detection thresholds. The columns 0.5:0.95 correspond to the average video-mAP for thresholds with step 0.05 in this range. For [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF], we report the results with and without their multi-region (+MR) approach.

outperform [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF] by 31% and 4%. In particular, our performance drops slower than the state of the art as the IoU threshold increases. This highlights the high localization accuracy of our tubelets and, therefore of our tubes. On UCF-101, we significantly outperform the state of the art at all overlap thresholds, with a larger gap at high thresholds. For instance, we outperform [29] by 5% at IoU = 0.5, and by 7.5% at IoU = 0.75. To validate our tubelet linking strategy (Section 3.3), we experiment with an approach that transforms tubelets into individual boxes and links them with [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF]. We observe a consistent gain of 1% on all datasets. As a summary, our ACT-detector improves over the state of the art, especially at high thresholds.

Runtime. We compare our runtime using two streams (appearance and flow) to the frame-based SSD approach of Singh et al.

[29] and to frame-based Faster R-CNN approaches [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF]. We report runtime on a single GPU without flow computation. Faster R-CNN based approaches [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF] run at 4fps and the SSD-based method [29] at 25-30fps.

Our ACT-detector also runs at 25-30fps (K = 6). Computing tubelets has a low overhead, since the convolutional features are computed once per frame due to the parallel architecture with shared weights. The post-processing is extremely fast (∼300fps) for all methods.

Conclusions

We introduced the ACT-detector, a tubelet detector that leverages the temporal continuity of video frames. It takes as input a sequence of frames and outputs tubelets instead of operating on single frames, as is the case with previous state-of-the-art methods [START_REF] Peng | Multi-region two-stream R-CNN for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF]29]. Our method builds upon SSD and introduces anchor cuboids that are scored and regressed over sequences of frames. An extensive experimental analysis shows the benefits of our ACT-detector for both classification and localization. It achieves state-ofthe-art results, in particular for high overlap thresholds.


Vicky Kalogeiton

Philippe Weinzaepfel

Vittorio Ferrari

Cordelia Schmid

Joint learning of object and action detectors

.

Introduction

Video understanding has received increased attention over the past decade leading to significant advances [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF][START_REF] Venugopalan | Sequence to sequence-video to text[END_REF]. However, most existing approaches focus either on object recognition [START_REF] Kang | Object detection from video tubelets with convolutional neural networks[END_REF][START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] or on human action recognition [START_REF] Peng | Multi-region two-stream r-cnn for action detection[END_REF][START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF] separately. For both tasks, the community has moved from small datasets [START_REF] Rodriguez | Action mach: a spatio-temporal maximum average correlation height filter for action recognition[END_REF] to large ones with thousands of videos and hundreds of classes [START_REF]Imagenet large scale visual recognition challenge (ilsvrc)[END_REF][START_REF] Heilbron | Activitynet: A large-scale video benchmark for human activity understanding[END_REF], from controlled environments [START_REF] Schüldt | Recognizing human actions: a local svm approach[END_REF] to videos in-the-wild [START_REF] Karpathy | Large-scale video classification with convolutional neural networks[END_REF]. Given the impressive success of Convolutional Neural Networks (CNNs) for object detection [23,[START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF], action localization has benefited as well from this improvement. In particular, Faster R-CNN [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] has been enhanced for videos by using a two-stream variant [START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Peng | Multi-region two-stream r-cnn for action detection[END_REF][START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF], in which both appearance and motion are used as inputs. Modern approaches first use such a detector to localize human actions in individual frames, and then either link them or track them over time to create spatio-temporal detections [START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Peng | Multi-region two-stream r-cnn for action detection[END_REF][START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF]. These methods focus exclusively on human action recognition.

While humans or actions alone are building blocks of video understanding, the relationship between objects and actions can yield a more complete interpretation. For instance, an autonomous car should not only be able to detect another car (object) or a human walking (action), but also a dog running or a ball flying (object-action). Other applications include content-based retrieval, video captioning [START_REF] Venugopalan | Sequence to sequence-video to text[END_REF][START_REF] Yao | Describing videos by exploiting temporal structure[END_REF] and health-care robots, for instance helping blind people crossing streets. Therefore, to better understand videos, we need to go beyond these two independent tasks of object and human action recognition and understand the relationship between objects and actions.

In this paper, we propose to jointly detect object-action instances in uncontrolled videos, e.g. cat eating, dog running or car rolling, see Figure 1. We build an end-to-end two stream network architecture for joint learning of objects and actions. We cast this joint learning problem by leveraging a multitask objective. We compare our proposed end-toend multitask architecture with alternative ones (Figure 3): (i) treating every possible combination of actions and objects as a separate class (Cartesian) and (ii) considering a hierarchy of objects-actions: the first level corresponds to objects and the second one to the valid actions for each object (hierarchical). We show that our method performs as well as these two alternatives while (a) requiring fewer parameters and (b) enabling zero-shot learning of the actions performed by a specific object, i.e., when training for an ob- ject class alone without its actions, our multitask network is able to predict actions for that object class by leveraging actions performed by other objects.

Interestingly, our multitask objective not only allows to effectively detect object-action pairs but also leads to performance improvements on each individual task (i.e., detection of either objects or actions). This is because the features learned for one task help learning the other one. We compare to the state of the art for object-action detection on the Actor-Action (A2D) dataset [START_REF] Xu | Can humans fly? Action understanding with multiple classes of actors[END_REF] that contains segmentation annotation for object-action pairs. For a direct comparison we transform our detections into pixelwise segmentation maps by using segmentation proposals [START_REF] Grundmann | Efficient hierarchical graph-based video segmentation[END_REF][START_REF] Pinheiro | Learning to refine object segments[END_REF]. Our approach significantly outperforms the state of the art [START_REF] Xu | Can humans fly? Action understanding with multiple classes of actors[END_REF][START_REF] Xu | Actor-action semantic segmentation with grouping-process models[END_REF] on this dataset. We finally apply our multitask objective to detect object-action relationships in images on the Visual Relationship Detection (VRD) dataset [START_REF] Lu | Visual relationship detection with language priors[END_REF]. In summary, we make the following contributions:

• We propose an end-to-end multitask architecture for joint object-action detection.

• We show that this multitask objective can be leveraged for zero-shot learning of actions.

• We demonstrate the generalization of our multitask architecture by applying it to (a) object-action semantic segmentation and (b) object-action relationships in images.

Related Work

Most existing approaches for detection in videos focus either on object or on action localization. Over the past few years, the methods range from low-level features [START_REF] Klaser | A Spatio-Temporal Descriptor Based on 3D-Gradients[END_REF][START_REF] Laptev | On space-time interest points[END_REF][START_REF] Malisiewicz | Ensemble of exemplar-svms for object detection and beyond[END_REF][START_REF] Prest | Learning object class detectors from weakly annotated video[END_REF][START_REF] Uijlings | Selective search for object recognition[END_REF][START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF][START_REF] Wang | A robust and efficient video representation for action recognition[END_REF], structured models that mine mid-level elements [START_REF] Lan | Action recognition by hierarchical mid-level action elements[END_REF][START_REF] Ma | Action recognition and localization by hierarchical space-time segments[END_REF] to parts [START_REF] Felzenszwalb | Object detection with discriminatively trained part based models[END_REF][START_REF] Pandey | Scene recognition and weakly supervised object localization with deformable part-based models[END_REF][START_REF] Raptis | Discovering discriminative action parts from mid-level video representations[END_REF] and attributes [START_REF] Liu | Recognizing human actions by attributes[END_REF]. However, CNNs currently constitute the dominant approach for large-scale and high-quality video detection. Object or action detection. Recent work on object detection [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF][START_REF] Kang | Object detection from video tubelets with convolutional neural networks[END_REF][START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] has shown remarkable progress, mainly due to the use of CNNs [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]23]. R-CNN [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] tackles object detection with CNNs by casting the task as a regionproposals classification problem. Faster R-CNN [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] goes a step further and generates proposals using a Region Proposal Network (RPN), which shares convolutional features with the proposal classification branch.

These per-frame detectors are also used by state-of-theart human action localization methods [START_REF] Peng | Multi-region two-stream r-cnn for action detection[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF] to obtain spatial information; then the detections are linked across time resulting in video-level localizations [START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF]. To leverage video data, the detector operates on two streams [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF]: RGB and optical flow. The two streams are trained separately and the scores are averaged at test time [START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Peng | Multi-region two-stream r-cnn for action detection[END_REF][START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF], i.e., late fusion of scores. In contrast, our architecture is a two-stream Faster R-CNN trained end-to-end based on a fusion by a fully-connected layer that operates on concatenated features from both streams. Moreover, it is trained with a multitask objective that allows us to detect objects and actions jointly. Joint modeling of objects and actions. Joint modeling of objects and actions in videos has received little attention so far. For the action localization task, some works [START_REF] Gupta | Observing humanobject interactions: Using spatial and functional compatibility for recognition[END_REF][START_REF] Prest | Explicit modeling of human-object interactions in realistic videos[END_REF] propose to model the interactions of humans and objects. However, the task we tackle in this paper is significantly different as objects are not used for the actions, but they are the actors. Bojanowski et al. [START_REF] Bojanowski | Finding actors and actions in movies[END_REF] have considered the case in which different entities can perform a set of actions, but these entities correspond to names of different actors, i.e., to person identification. Closely related to object-action detection in videos is the work [START_REF] Xu | Actor-action semantic segmentation with grouping-process models[END_REF][START_REF] Xu | Can humans fly? Action understanding with multiple classes of actors[END_REF] on segmenting objectaction pairs. They use Conditional Random Fields at the supervoxel level to output a semantic segmentation at the pixel level. We show that our detections based on a multitask objective also improve the semantic segmentation performance by leveraging segmentation proposals [START_REF] Grundmann | Efficient hierarchical graph-based video segmentation[END_REF][START_REF] Pinheiro | Learning to refine object segments[END_REF].

In images, however, object-action pairs have been modeled implicitly in the context of predicting sentences for images [START_REF] Mao | Deep captioning with multimodal recurrent neural networks (m-rnn)[END_REF][START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF] and more recently by visual phrases [START_REF] Sadeghi | Recognition using visual phrases[END_REF] relationships between objects [START_REF] Lu | Visual relationship detection with language priors[END_REF]. The task consists in detecting triplets of two objects and their relationship [START_REF] Lu | Visual relationship detection with language priors[END_REF][START_REF] Sadeghi | Recognition using visual phrases[END_REF].

Most approaches rely on object detectors. The relationship label is predicted from the bounding box around the two objects, and sometimes from additional modalities such as languages or frequency priors in the training set. We show that our multitask objective allows to predict the relationships between objects without (a) the need to see the whole bounding box and (b) the need to include any priors. In particular, we transform each triplet into two pairs, each consisting of one of the two objects and the interaction. Then, we train our network to detect bounding boxes around objects and also predict an interaction label. Zero-shot learning. Most existing approaches for zeroshot learning of categories rely on attributes [START_REF] Bucher | Improving semantic embedding consistency by metric learning for zero-shot classiffication[END_REF][START_REF] Escorcia | On the relationship between visual attributes and convolutional networks[END_REF][START_REF] Lampert | Attributebased classification for zero-shot visual object categorization[END_REF]. Attributes have also been used for human actions [START_REF] Liu | Recognizing human actions by attributes[END_REF][START_REF] Yao | Human action recognition by learning bases of action attributes and parts[END_REF]. Liu et al. [START_REF] Liu | Recognizing human actions by attributes[END_REF] were the first to represent actions by sets of attributes. They consider that each action class has an intra-class variability, which they try to model by searching which attributes are relevant for each class. They apply zero-shot learning by manually labeling attributes for all classes, including new ones without visual examples. In contrast, our approach does not require any attribute labels.

End-to-end multitask network architecture for joint learning of objects and actions

Given a video, we aim to detect the objects as well as the actions they are performing. Let O (resp. A) be the set of objects (resp. actions) labels. Some combinations of actions and objects may not be valid, e.g. car eating. We denote by V ⊂ O × A the set of valid object-action combinations.

End-to-end network architecture

We build an end-to-end two-stream multitask network that proceeds at the frame level (Figure 2). As most stateof-the-art methods for object and action detection in videos, we rely on Faster R-CNN [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] and its two-stream variant [START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF][START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF]. However, instead of training each stream separately, we propose to fuse both streams, thus enabling effective end-to-end learning. Our end-to-end network has two streams: (a) appearance, which takes as input the RGB data and (b) motion, which operates on the optical flow [3]. Following [START_REF] Gkioxari | Finding action tubes[END_REF], the input of the motion stream is a tensor of three channels with the x and y coordinates of the flow and its magnitude, represented as a 3-channel image. A Region Proposal Network (RPN) extracts candidate bounding boxes independently for each stream. We use the set union of the two RPNs and we aggregate features for each candidate box with a Region-of-Interest (RoI) pooling layer in each stream. After one fully-connected layer, the two streams are concatenated and fed to another fully-connected layer. The remaining network layers operate on the fused stream, enabling end-to-end training. This allows us to learn the most relevant features among all possible combinations of appearance and motion. In contrast, late fusion of the softmax probabilities of the two streams [START_REF] Peng | Multi-region two-stream r-cnn for action detection[END_REF] assumes that both appearance and motion are equally relevant for every class. As we show in Section 4.1.1, our proposed fusion significantly outperforms the late fusion.

Finally, we use a multitask loss for detecting objects, actions, and regressing the bounding box coordinates according to the object classes. The total loss L of the network is:

L = L RPN R + L RPN F + L cls + L reg , (1)
with L RPN R and L RPN F the losses of the RPN operating on the RGB and flow stream, respectively, L cls the classification loss, i.e., for recognizing objects and actions, and L reg the bounding box regression loss.

Joint learning of objects and actions

Given the candidate boxes, the network aims at jointly predicting whether a box contains a particular object and which action this object is performing. Let o (resp. a) be the ground-truth object (resp. action) label of a region proposal in the training set. To classify the boxes, we use a multitask architecture: one component predicts the object class, and a second one predicts the action class, independently of which object is performing it. Besides our proposed multitask architecture, we consider two alternatives to jointly predict object-action pairs: Cartesian product and hierarchy of classes. We now present details for these three objectives. We illustrate them in Figure 3 and summarize their main differences in Table 1. Multitask. Our multitask architecture relies on a multitask loss, for classifying candidate boxes with both object and action labels. The first branch predicts the object la-loss # outputs probability # params Multitask

-log pO(o) -log pA(a) |O| + |A| + 2 pO(o) • pA(a) 0.9M Cartesian -log pV (o, a) |V| + 1 pV (o, a) 54.6M Hierarchical -log pO(o) -log pA o (a) |O| + |V| + 1 pO(o) • pA o (a) 55.4M
Table 1: Comparison of different losses for object-action learning. We give the number of parameters in the classification layers from the VRD dataset [START_REF] Lu | Visual relationship detection with language priors[END_REF] where |O| = 100, |A| = 140, |V| = 13344 (Section 4.4).

bel. It is composed of a fully-connected layer that outputs |O| + 1 scores (one per object class and another one for background) followed by softmax. Let p O be the output of this branch. In the same way, p A denotes the output of the second branch that predicts the action label, i.e., of dimension |A| + 1. We use a log loss on both object and action classification:

L Multitask cls = -log p O (o) -log p A (a) . (2)
This version uses |O| + |A| + 2 outputs (Figure 3 ). In this case, there is only one branch for classification with |V| + 1 outputs. We denote as p V the output of this branch. The classification loss is:

L Cartesian cls = -log pV (o, a) . (3)
This version uses |V| + 1 outputs, which is in the order of |A| × |O|. For instance, for |V| = 13344 (VRD dataset [START_REF] Lu | Visual relationship detection with language priors[END_REF]) the number of parameters in the classification layer is 54.6M, i.e., 50× more than in the multitask (Table 1). This makes it less scalable than our multitask objective and does not allow sharing of action labels across object classes, which is required for zero-short learning. In the multitask case, samples of an object-action pair help training the detector of this object, which in turn helps detecting it doing other actions; e.g. adult-running and adultwalking samples help improving the adult detector. In contrast, by using the Cartesian product, each training sample helps training only one particular object-action detector. At test time, the probability of being an object-action instance (o, a) is given by p V (o, a). Hierarchy of classes. We also consider the set of valid object-action classes as a hierarchy (Figure 3 (c)). The first branch p O predicts the object. For each object o, any branch p Ao predicts the actions among the valid ones A o for o. In this case, the classification loss is:

L Hierarchy cls = -log pO(o) -log pA o (a) . (4)
This version uses a total of |O| + 1 outputs for the first level and |V| for the second level, see Figure 3 Per-object regression. In all cases, we refine the proposal output by the RPN using a per-object regression of the bounding box coordinates. The RPN minimizes the geometric difference between the proposals and the groundtruth boxes. We follow [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] and make the regression target scale-invariant by normalizing it by the size of the proposal. We denote by t o,a the regression target for a proposal that covers an object. By using a per-object regression, we obtain the following regression loss:

Lreg = Smooth-L1(uo -to,a) , (5)
with u o the output of the regression branch u corresponding to object o, and:

Smooth-L1(x) = 0.5x 2 if |x| < 1, |x| -0.5 otherwise. ( 6
)

Experimental Results

In this section, we study the impact of each of our contributions separately. We first examine joint detection of objects and actions (Section 4.1) and zero-shot learning (Section 4.2). Next, we compare our proposed multitask architecture to the state of the art on semantic segmentation of object-action pairs (Section 4.3) and relationship detection in images (Section 4.4). Implementation details. Our framework is based on Faster R-CNN [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] using the VGG-16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] as the underlying CNN architecture. We initialize both streams using the standard pre-training on ILSVRC 2012 [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. This is in line with [START_REF] Wang | Temporal segment networks: towards good practices for deep action recognition[END_REF], which shows that pre-training on ILSVRC 2012 instead of UCF-101 [START_REF] Soomro | UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild[END_REF] improves video classification accuracy.

Joint detection of objects and actions in videos

In this section, we evaluate our proposed end-to-end architecture for joint detection of object-action pairs. We start by validating the effectiveness of our end-to-end network (Section 4.1.1) and then, we examine the joint learning with the multitask objective (Section 4.1.2). Video datasets. Actor-Action (A2D) dataset [START_REF] Xu | Can humans fly? Action understanding with multiple classes of actors[END_REF], which has sparse framelevel annotations for both objects and actions in videos.

To the best of our knowledge, it is the only video dataset with bounding box and semantic segmentation annotations for object-action pairs. It contains 7 objects (adult, baby, ball, bird, car, cat, and dog) performing 8 different actions (climb, crawl, eat, fly, jump, roll, run, walk) or no action. We also use two video datasets for object detection: the YouTube-Objects (YTO) dataset [START_REF] Kalogeiton | Analysing domain shift factors between videos and images for object detection[END_REF][START_REF] Prest | Learning object class detectors from weakly annotated video[END_REF] and the 'object detection in video' (VID) track of the ILSVRC [START_REF]Imagenet large scale visual recognition challenge (ilsvrc)[END_REF]. YTO consists of videos collected from YouTube with 10 classes of moving objects, e.g. aeroplane, car. VID contains bounding boxes for 30 object classes including rigid objects, e.g. motorcycle, watercraft, and animals, e.g. fox, monkey. Protocol. We measure the detection performance using the PASCAL VOC protocol [START_REF] Everingham | The PASCAL Visual Object Classes Challenge 2007 Results[END_REF]: a detection is correct if its intersection-over-union overlap (IoU) with a ground-truth box is greater than 0.5 and its labels (object and action) are correctly predicted. The performance for a class is the average precision (AP), and the overall performance is captured by the mean over all classes (mAP).

End-to-end architecture

We want to quantify the effectiveness of our proposed endto-end architecture that consists of two streams fused (a) at the proposal (RoI) level and (b) at the feature level (Figure 2). We evaluate the impact of fusion for object detection alone. We perform experiments on the three video datasets (A2D, YTO and VID). Table 3 shows all the mAP results for the different cases we consider. Impact of RGB and Flow cues. To examine the impact of the RGB and flow cues, we train each stream separately. The first two rows of Table 3 show that the RGB stream significantly outperforms the flow one. This is due to the fact that the RGB stream is able to learn information about how the objects look, which is a distinctive cue across different object classes. The flow stream performs worse than the RGB one in general, and is particularly poor on the VID dataset. This is because most objects in VID move only slightly, or their motion is not discriminative for the class. Impact of end-to-end training. Our proposed fusion of the two streams enables end-to-end training. We examine the impact by comparing our proposed fusion of streams with late fusion of scores [START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Peng | Multi-region two-stream r-cnn for action detection[END_REF]. In the latter, i.e., late fusion of scores, we train the two-stream network fusing only the region-proposal layers and then average the classification scores of each stream as [START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Saha | Deep learning for detecting multiple space-time action tubes in videos[END_REF]. Results in Table 3 show that, for all video datasets, using late score fusion reduces the detection performance compared to using the RGB stream alone. Interestingly, this is opposite of the findings in human action localization [START_REF] Gkioxari | Finding action tubes[END_REF][START_REF] Peng | Multi-region two-stream r-cnn for action detection[END_REF], where performance increases due to the the significance of motion cues for actions. This shows that the two-stream architecture cannot be used as it is for object detection in videos and highlights a clear difference between object and human action detection. In contrast, on all object detection datasets, our proposed fusion outperforms the other cases: it leads to an increment over the late score fusion of approximately 2-3%. This shows that the network successfully learns when to leverage motion information and more importantly, how to jointly learn features coming from the two stream.

Multitask learning

In this section, we evaluate our proposed multitask learning of objects and actions. We start by evaluating the performance only on object or on action detection. Therefore, we train and test our network with only object or only action labels (first two rows of Table 4). We also compute a baseline (third row of Table 4) for object-action detection in which we combine the object and the action detector trained separately. More precisely, for each object detection, we obtain object-action scores by multiplying the object scores with the action scores from the most overlapping action box. Table 4 also reports the results of our proposed multitask architecture trained with objects and actions from the A2D dataset. The most interesting finding is that our multitask training improves the performance on each task separately (Table 4 We also consider two alternative ways to jointly detect objects and actions (Section 3.2 and Figure 3): (a) Cartesian product of object-action labels and (b) hierarchy of objectaction classes. Table 4 (Cartesian and hierarchical) reports the results when we train these two networks on the A2D dataset. We observe that they both perform similarly to our multitask network. The Cartesian and hierarchical networks have the advantage of being able to distinguish different ways objects perform each action (Table 1). Discussion. In practice there are similarities in the way different objects perform the same action (e.g. dog and cat eating) and in the way the same object performs different actions (e.g. dog walking and running). Thus, our multitask objective allows the network to exploit the commonality among the two tasks, and hence, what is learned for each task facilitates the learning of the other. In a nutshell, our multitask architecture is a simpler model, able to reach the same performance as the alternative architectures while requiring much fewer parameters (Table 1 # params) and enabling zero-shot learning (Section 4.2). For instance, in Section 4.4 we clearly show the benefit of our multitask architecture compared to the Cartesian and hierarchical architectures for a large number of objects and actions due to its lower number of parameters.

Note that both losses (object and action) contribute equally to the overall loss (Equation 2), as they are of the same type (softmax), and the tasks they address are of the same difficulty. To validate this, we vary the weight of the action loss over 0.5, 1, 2 and observe insignificant variations (< 0.5%) in the object-action mAP on A2D.

Zero-shot learning of actions

An important advantage of our end-to-end multitask architecture is its capability of predicting actions for an object without having trained for these particular object-actions combinations. To validate this intuition, we experiment on the A2D dataset (Table 2), which contains annotations for 7 objects performing 8 different actions in videos. We train the network seven times, where each time we remove for one object o all its action labels. For instance, we remove all action labels for the object cat, but keep the cat examples for training the object detector. Equation 2 is replaced by:

L Multitask cls zero-shot = -log p O (o) -[o = o] log p A (a) . (7)
Note that the object classifier is not changed, while the action classifier is learned only on the actions performed by the objects different from o . This approach to zero-shot learning does not assume any prior knowledge such as attributes of the unseen classes [START_REF] Liu | Recognizing human actions by attributes[END_REF].

We report the results of zero-shot learning in Table 5. We also report the AP when training with all object-action pairs. The results show that our network is able to infer information about actions not seen at training time for a given object. We observe that there are some object-action pair for which the AP is only slightly decreased, e.g. cat rolling or dog eating. This is because these objects share commonalities with others, e.g. cat and dog eating. In contrast, we observe poor performance for objects like ball which do not share similarities with other objects of the dataset. For object classes that share similarities in actions, such as cat and dog, our multitask architecture outperforms chance level classification of unknown actions by a large margin (+15%), while for classes that do not share commonalities with other classes, like adult the gain is smaller (+5%).

Object-action segmentation

A2D comes with annotations for semantic segmentation of object-action pairs. In this section, we extend our bounding box detections to pixelwise segmentation and we compare our results to the state of the art. Metrics. Following [START_REF] Xu | Can humans fly? Action understanding with multiple classes of actors[END_REF], we measure class-average pixel accuracy and global pixel accuracy. Accuracy is the percentage of pixels for which the label is correctly predicted, either over all pixels (global) or first computed for each class separately and then averaged over classes (classaverage). We also evaluate our segmentations using mIoU, i.e., the IoU between the ground-truth segmentation and output segmentation averaged over all classes. mIoU is better suited as it is not biased towards background which is the most present class and it penalizes errors when too many pixels are set to a particular label instead of background. Setup. Our multitask model predicts bounding boxes for each object-action pair. We extend our detections to pixelwise segmentations of object-action pairs by using segmentation proposals from either (a) the recently proposed SharpMask [START_REF] Pinheiro | Learning to refine object segments[END_REF] or (b) the hierarchical video segmentation method GBH [START_REF] Grundmann | Efficient hierarchical graph-based video segmentation[END_REF], which is the one used by the state-of-the- Table 6: Comparison to the state of the art for object, action and object-action segmentation on A2D using class-average pixel accuracy (ave), global pixel accuracy (glo) and mean Intersection over Union (mIoU) metrics.

art GPM method [START_REF] Xu | Actor-action semantic segmentation with grouping-process models[END_REF]. For each frame, we first apply nonmaximum suppression on the detections that have a score greater than 0.5. Then, for each detection, we select the segmentation proposal that overlaps the most with it (according to IoU). If there is no such proposal, we directly use the rectangular detection itself as a segmentation mask. While our setup is simple, it serves as a baseline to evaluate our detections for semantic segmentation.

Results. The first three rows of Figure 4 show correctly labeled and segmented object-action pairs. We observe that our segmentation results are accurate, even in difficult cases, such as small objects (e.g. birds) or cluttered scenes (e.g. adults running). The two last rows show typical failure cases. In the fourth row, the action label of one adult is incorrect and there are some detections considered as wrong due to missing annotations. In the last row we miss the adult for which only one arm is visible.

Table 6 provides a quantitative comparison between our results and the state of the art [START_REF] Xu | Actor-action semantic segmentation with grouping-process models[END_REF][START_REF] Xu | Can humans fly? Action understanding with multiple classes of actors[END_REF] on A2D. When using SharpMask, we outperform the previous state of the art for all metrics and all tasks, except for average accuracy on action segmentation, where we match [START_REF] Xu | Actor-action semantic segmentation with grouping-process models[END_REF]. Our improvements are particularly significant for object segmentation (+14% class-average accuracy, +16% mIoU) and joint object and action segmentation (more than +5% on all metrics). Note that we do not use any training segmentation from the A2D dataset (SharpMask is pre-trained on MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF]). Furthermore, we observe that even when using the same underlying method (GBH [START_REF] Grundmann | Efficient hierarchical graph-based video segmentation[END_REF]), we perform on par or better than [START_REF] Xu | Actor-action semantic segmentation with grouping-process models[END_REF][START_REF] Xu | Can humans fly? Action understanding with multiple classes of actors[END_REF] in all metric-task combinations.

Relationship detection of objects and actions

In this section we use only images, and therefore we use only the RGB stream as there is no flow for images. We apply our model to visual relationship detection, where we detect relationships between objects, defined as triples: ob-ject1 -interaction -object2. To do so, we transform each triplet into two pairs, each consisting of an object and an interaction and use them to train our multitask architecture.

Dataset and protocol. We employ the Visual Relationship Detection (VRD) dataset [START_REF] Lu | Visual relationship detection with language priors[END_REF] that examines object relationships. It contains 4k training and 1k test images with 38k relationships between objects, such as person kick ball, per- son wear shirt, motorcycle has wheel. There are 100 different objects and 70 interaction types. We consider here visual phrase detection [START_REF] Sadeghi | Recognition using visual phrases[END_REF], where the goal is to output a triplet object1 -interaction -object2 and localize it with one box having an IoU over 0.5 with the ground-truth box. We also evaluate relationship detection: the task consists in detecting a triplet object1 -interaction -object2 with two bounding boxes on object1 and object2, both having an IoU over 0.5 with their ground-truth boxes.

For evaluation, the metric used is recall @100 and recall @50 (denoted as R@N ) and not mAP, as not all possible interactions are annotated in the test images. In each image, the top N detections are kept and recall is measured. Model. To detect relationships using our multitask architecture, we transform each object1-interaction-object2 triplet into two pairs, each consisting of an object and an interaction label. More precisely, we double the set of all possible interactions, by including their passive forms. For example, the triplet human kicks ball becomes two pairs: (i) one with object human and action kick, and (ii) another pair with ob- Table 7: Comparison to different architectures and to the state-of-the-art visual relationships on the VRD dataset for phrases and relationship detection. We report R@100 and R@50 for methods using only visual cue (V) or also language and frequency priors (V+L+F).

ject ball and action kick = being kicked. In that way, our training set consists of 100 object classes performing 140 different actions. Note here that the possible number of outputs is 100 + 140 + 2 for our multitask objective.

At test time, we keep all detection with score over 0.5 and apply non-maximum suppression. For each pair of object detections, we score each possible interaction using the multiplication of the object scores and the interaction score. The interaction score is defined as the combination of the score of an interaction from the first object and its passive form from the second object, i.e., the interaction score of kick in human kicks ball includes both scores of kick for the human and being kicked for the ball.

Results. Table 7 reports the R@100 and R@50 for the two tasks we examine, i.e., phrase and relationship detection. We outperform all previous state-of-the-art results on both tasks and at both operating points, when comparing to methods based purely on the images ( [START_REF] Lu | Visual relationship detection with language priors[END_REF][START_REF] Sadeghi | Recognition using visual phrases[END_REF][START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF]). Moreover, our results are only a little worse than those of [START_REF] Lu | Visual relationship detection with language priors[END_REF], where they enhance their visual model with some frequency prior as well as language priors by leveraging the semantic similarities of relationships in term of words. In particular, we perform on par on phrase detection (+1% at R@100 and -2% at R@50). Note how our method features a clear increment from R@50 to R@100, which shows its potential to correctly detect interactions that may be lower in the recall list. Hence, including some language or spatial priors could significantly increase our performance. Figure 5 shows some qualitative results.

Benefits of the multitask training. We compare our multitask architecture with a baseline approach where we multiply the scores of two separate networks, one trained on objects and another one trained on interactions. Table 7 shows that our multitask architecture outperforms this alternative ('Baseline' row). This comparison highlights the benefit of joint training compared to training for each task separately. We have also evaluated the Cartesian and hierarchical combination of objects and actions (Section 3.2) and found that they perform poorly (for both R@100 is around 0%). This can be explained by lack of training data necessary to determine the large number of parameters (55M in Table 1). Table 8: Comparison to the state-of-the-art zero-shot detection of visual relationships on the VRD dataset. We report R@100 and R@50 for methods using only visual cue (V) or also language and frequency priors (V+L+F).

Zero-shot learning. The test set of the VRD dataset contains 1.9k triplets that never occur in the training set. Our architecture allows zero-shot learning and we report the results on these triplets in Table 8. Our method outperforms the state-of-the-art method [START_REF] Lu | Visual relationship detection with language priors[END_REF] when using only the visual modality (no language or frequency prior). Additionally, for phrase detection we detect unseen-at-training interactions better than [START_REF] Lu | Visual relationship detection with language priors[END_REF], even when they also use language and frequency priors. Finally, our multitask architecture outperforms the baseline by a significant margin, highlighting the benefit of joint training compared to separate one.

Conclusions

Most state-of-the-art works for video detection aim at localizing either objects or actions. Instead, we jointly detect objects and actions in uncontrolled video scenes. To this end, we propose an end-to-end network built upon Faster R-CNN [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF]. The key point is that our network operates with a multitask objective. We show that this joint training: (a) outperforms training alone with objects or with actions, as the network can generalize better, is less prone to overfit and benefits from sharing statistical strength between classes, (b) performs as well as other variants while requiring fewer parameters and (c) allows zero-shot learning of actions performed by an object, for which no action labels are present at training time. Our network can also be applied to different tasks including semantic segmentation and visual relationship detection.

1

  Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France 2 University of Edinburgh 3 Naver Labs Europe

Figure 1 :

 1 Figure 1: Detection examples of different object-action pairs for the videos of the A2D dataset [50].

Figure 2 :

 2 Figure 2: Overview of our end-to-end multitask network architecture for joint object-action detection in videos. Blue color represents convolutional layers while green represents fully connected layers. The end-to-end training is done by concatenating the fully connected layers from both streams. Here, p O and p A are the outputs of the two branches that predict the object and action labels, resulting in the loss described in Equation 2.

  and

Figure 3 :

 3 Figure 3: Illustration of the three different ways we consider for jointly learning objects and actions. The blue nodes represent objects and the red ones action classes, while the yellow ones represent the background class.

  (a)). For |O| = 100 and |A| = 140 the number of parameters in the classification layers is 0.9M (VRD dataset [24] used in Section 4.4). At test time, the probability of a box to be the object-action instance (o, a) is given by p O (o) • p A (a). Cartesian product. Another solution is to consider each object-action pair as a separate class, e.g. bird flying (Figure 3 (b)

Figure 4 :

 4 Figure4: Examples of semantic segmentation with (from left to right): the frame, the ground-truth and the segmentation output obtained when combining our approach with proposals from SharpMask[START_REF] Pinheiro | Learning to refine object segments[END_REF]. The colors of the segmentations represent an object-action pair. Note that we do not use any object-action segmentation at training time.

Figure 5 :

 5 Figure 5: Qualitative object-action relationships on the VRD dataset: (yellow): our correct boxes with their green label, (red): missed interactions for 100 retrieved boxes.ModalityMethod Phrase detection Relationship detection R@100 R@50 R@100 R@50 V VRD [24] 1.1 0.8 0.8 0.7 Baseline 4.3 2.3 2.4 1.3 Ours Multitask 5.5 3.4 2.9 1.9 V+L+F VRD [24] 3.8 3.4 3.5 3.1

Table 2 :

 2 Overview of the video datasets we use.for |O| = 100 and |V| = 13344 the number of parameters in the classification layers is 55.4M, i.e., 50× more than in the multitask (Table1). At test time, the probability of being an object-action instance (o, a) is given by p O (o) • p Ao (a).

Table 3 :

 3 Table2shows some statistics of the datasets we use. For object-action detection we use the Impact of end-to-end training: mAP for object detection of different training scenarios on the A2D, YTO and VID datasets.

	input RGB Flow RGB Flow Fusion RoI Stream	A2D YTO VID
	-	-	-	63.1	58.9	45.2
	-	-	-	32.0	32.3	5.0
			late	61.6	57.3	33.9
			ours	65.3	62.2	48.1

Table 5 :

 5 Evaluation of zero-shot learning for object-action pairs on A2D. For each object, we report the AP when excluding all actions of this object at training. The numbers in parenthesis indicate the AP when training with all object-action pairs.

Acknowledgments. This work was supported in part by the ERC grants ALLEGRO and VisCul, the MSR-Inria joint project, a Google research award, a Facebook gift, an Intel gift and an Amazon research award. We gratefully acknowledge the support of NVIDIA with the donation of GPUs used for this research.