Dave Achal

Pavel Tokmakov Cmu

Cordelia Cmu

Schmid

Deva Ramanan Cmu

Learning to Track Any Object

Object tracking can be formulated as "finding the right object in a video". We observe that recent approaches for class-agnostic tracking tend to focus on the "finding" part, but largely overlook the "object" part of the task, essentially doing a template matching over a frame in a slidingwindow. In contrast, class-specific trackers heavily rely on object priors in the form of category-specific object detectors. In this work, we repurpose category-specific appearance models into a generic objectness prior. Our approach converts a category-specific object detector into a categoryagnostic, object-specific detector (i.e. a tracker) efficiently, on the fly. Moreover, at test time the same network can be applied to detection and tracking, resulting in a unified approach for the two tasks. We achieve state-of-the-art results on two recent large-scale tracking benchmarks (OxUvA and GOT, using external data). By simply adding a mask prediction branch, our approach is able to produce instance segmentation masks for the tracked object. Despite only using box-level information on the first frame, our method outputs high-quality masks, as evaluated on the DAVIS '17 video object segmentation benchmark.

Introduction

Tracking is an essential element of video analysis. Extracting spatio-temporal regions corresponding to objects from a video is not only the end goal for surveillance and video labeling [START_REF] Jain | Click carving: Segmenting objects in video with point clicks[END_REF], but also an important intermediate representation for tasks such as action recognition [START_REF] Weinzaepfel | Learning to track for spatio-temporal action localization[END_REF][START_REF] Zhang | A structured model for action detection[END_REF].

Unfortunately, tracking in general is notoriously challenging and potentially ambiguous. Consider the example in Figure 1 of tracking a bus with only one side visible initially. Without prior knowledge, it is unclear whether the back side of the bus (visible in future frames) is a different viewpoint of the same bus, or a new object itself. In practice, many tracking approaches struggle to resolve such ambiguities, tending to diverge to an object part which is most similar to the initial template (e.g., the back window of the bus). Successful tracking in these scenarios necessitates object priors. Indeed, approaches for category-specific tracking, where the tracked object categories are known before hand, heavily rely on priors in the form of categoryspecific object detectors [2,48,[START_REF] Wojke | Simple online and realtime tracking with a deep association metric[END_REF]5]. By contrast, approaches for user-initialized tracking have largely eschewed such priors [START_REF] Bolme | Visual object tracking using adaptive correlation filters[END_REF]6,[START_REF] Zhu | Distractor-aware siamese networks for visual object tracking[END_REF] in the pursuit of tracking generic objects, sometimes known as model-free tracking [START_REF] Wu | Object tracking benchmark[END_REF][START_REF] Kristan | A novel performance evaluation methodology for single-target trackers[END_REF]. However, generic objects still share a common set of objectness traits [START_REF] Alexe | Measuring the objectness of image windows[END_REF]. How can we operationalize this implicit constraint into a useful prior?

In this work, we repurpose category-specific appearance models into a generic objectness prior that can be used for category-agnostic tracking. In essence, we show that model-free tracking is far easier with better models! Doing so requires tackling two key challenges, shown in Figure 1: (1) How do we best adapt a category specific prior into a 1 arXiv:1910.11844v1 [cs.CV] 25 Oct 2019 generic objectness prior? (2) How do we further adapt this generic prior to the particular instance of interest?

To address [START_REF] Alexe | Measuring the objectness of image windows[END_REF], we build a joint model for categoryspecific object detection and category-agnostic tracking (Figure 2). It is based on the Mask R-CNN [START_REF] He | Mask R-CNN[END_REF] object detection architecture. For tracking, it takes as an additional input an object template in the first frame and computes its feature embedding. This template is then used to compute the similarity between the object of interest and a new frame. The similarity map is in turn applied to reweight spatial features from the new frame to detect only the object of interest. Importantly, training the network jointly on image and video datasets, allows us to both capture a generic object appearance model from the diverse image data and learn to use it in a category-agnostic way for tracking.

To address (2) -e.g., better separating the bus in Figure 1 from other vehicles, such as the van on the right -we propose a lightweight on-the-fly adaptation strategy. We compute a linear separator (T d in Figure 1) between the object of interest and other objects in the first frame. This separator is computed in closed form in a fully differentiable manner, and applied in future frames to compute similarities.

An intriguing property of our proposed architecture is that it can be used both as a single-object tracker and an object detector. Moreover, by capitalizing on the mask prediction branch of [START_REF] He | Mask R-CNN[END_REF], we are able to train and test the same network for instance and video object segmentation. To sum up, we present a single unified approach for object detection, tracking, instance and video object segmentation.

We evaluate our model on two recent, large scale datasets for object tracking: OxUvA [START_REF] Valmadre | Long-term tracking in the wild: a benchmark[END_REF] and GOT [START_REF] Huang | GOT-10k: A large highdiversity benchmark for generic object tracking in the wild[END_REF]. The former is focused on long-term object tracking, with objects undergoing a lot of appearance variation and occlusion. In contrast, the videos in GOT are shorter, but contain diverse object categories, covering more than 560 object classes. Our method outperforms prior work on OxUvA, and outperforms state-of-the-art approaches that use external data on GOT by a large margin. Next, we show results competitive with the state-of-the-art on the LTB-35 dataset from the VOT 2018 Long Term challenge [START_REF] Kristan | The sixth visual object tracking vot2018 challenge results[END_REF]. Finally, we validate the quality of our masks on DAVIS'17 dataset for video segmentation [START_REF] Pont-Tuset | The 2017 DAVIS challenge on video object segmentation[END_REF], demonstrating that our unified approach performs on par with specialized video segmentation methods that don't finetune on the test videos.

Our contributions are three-fold: (1) we incorporate an objectness prior in a generic tracker with a joint model for object detection, tracking, instance and video object segmentation;

(2) we propose a lightweight strategy for computing discriminative object templates in an end-to-end fashion for efficiently handling distractors; (3) our method demonstrates state-of-the-art results on three benchmark datasets for object tracking and video object segmentation.

Related work

Single object tracking. Classical approaches for single object tracking, which requires tracking an object given a bounding box annotation in the first frame, were based on the tracking-by-detection paradigm: in many cases the detector is used to first to localize all the objects in a frame. The box corresponding to the object of interest was then selected by a discriminative classifier trained on the first frame annotation [3,[START_REF] Hua | Online object tracking with proposal selection[END_REF][START_REF] Kalal | Tracking-learningdetection[END_REF]. Correlation filters were commonly used for classification due to their efficiency [START_REF] Bolme | Visual object tracking using adaptive correlation filters[END_REF]. To address appearance variation, some models updated the object template over time [START_REF] Hua | Online object tracking with proposal selection[END_REF]36]. Recent approaches learn correlation filters on top of deep features [START_REF] Danelljan | Eco: Efficient convolution operators for tracking[END_REF][START_REF] Valmadre | End-to-end representation learning for correlation filter based tracking[END_REF].

Current methods for tracking largely ignore the objectness prior provided by detectors. Instead, they rely on a Siamese network architecture (initially introduced for signature verification [START_REF] Bromley | Signature verification using a" siamese" time delay neural network[END_REF]) adapted for tracking [6,16,[START_REF] Tao | Siamese instance search for tracking[END_REF].

Recently, there have been several attempts to introduce ideas from CNN-based detection architectures into Siamese trackers. In particular, Li et al. [START_REF] Li | High performance visual tracking with Siamese region proposal network[END_REF] use the similarity map obtained by matching the object template to the test frame as input to an RPN-like module adapted from Faster R-CNN [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF]. Later this architecture was extended by introducing hard negative mining and template updating [START_REF] Zhu | Distractor-aware siamese networks for visual object tracking[END_REF], adding a mask prediction branch [START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF], and using deeper models [START_REF] Li | Siamrpn++: Evolution of siamese visual tracking with very deep networks[END_REF]. Our approach differs in that instead of integrating components of object detectors into a tracking pipeline in a heuristic way, we turn a state-of-the-art object detection framework into a tracker. This allows our model to fully utilize the objectness prior learned on COCO, outperforming the heuristic-based approaches significantly.

Video object segmentation. Methods for video object segmentation take a precise object mask as input in the first frame and output pixel-level segmentations for the object in each frame. Early methods for this task were based on mask propagation through a graph connecting superpixels in the neighboring frames [START_REF] Tsai | Video segmentation via object flow[END_REF][START_REF] Wen | Jots: Joint online tracking and segmentation[END_REF]. More recently, these methods have been outperformed by deep-learning based approaches, which capitalize on the success of image segmentation architectures [9,[START_REF] Perazzi | Learning video object segmentation from static images[END_REF]. In particular, they fine-tune a model trained for foreground-background segmentation using the annotation in the first frame and evaluate it on the remaining frames of the video. Some approaches also update the model using its own predictions to handle appearance variation [START_REF] Voigtlaender | Online adaptation of convolutional neural networks for video object segmentation[END_REF]. While these methods demonstrate impressive accuracy, they remain slow due to the need to update the model during evaluation. Alternative approaches, that do not require network fine-tuning have been proposed recently [START_REF] Cheng | Fast and accurate online video object segmentation via tracking parts[END_REF][START_REF] Yang | Efficient video object segmentation via network modulation[END_REF][START_REF] Voigtlaender | Feelvos: Fast end-to-end embedding learning for video object segmentation[END_REF], but remain inferior in performance.

These approaches treat video object segmentation as a problem independent from object tracking, with the re- First, we use a state-of-the-art object detector [START_REF] He | Mask R-CNN[END_REF] to extract features for the template image containing the object to be tracked (top left). Next, we compute a discriminative template that separates the features corresponding to the tracked object from the distractors in the first frame using linear regression (top right). Finally, attention masks computed with this template are used to reweight the feature maps of the detector to focus on the object of interest (bottom). Note that unlike standard, category-specific detectors, our box-head and mask-head output a single, categoryagnostic prediction for the tracked object.

cent exception of [START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF]. In contrast, we adapt the intuition from [START_REF] He | Mask R-CNN[END_REF] that instance masks can be computed as a by-product of object detection. Our tracker with a mask prediction branch achieves competitive performance on DAVIS'17 video object segmentation benchmark without requiring mask-level supervision on the first frame.

Object detection. CNN architectures for detection have brought significant progress, replacing classical methods for object detection that relied on hand-crafted features and part-based models [12]. Early approaches [START_REF] Babenko | Visual tracking with online multiple instance learning[END_REF][START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] trained CNNs to classify pre-computed object proposals. More recent approaches solve the detection problem in an end-toend way [START_REF] Liu | SSD: Single shot multibox detector[END_REF][START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF][START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF]. In particular, RCNN-like architectures [START_REF] He | Mask R-CNN[END_REF][START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] operate in a two-stage fashion: first an RPN proposes a set of boxes, and pools features from each box region. Next, separate branches classify the object and refine the box coordinates. [START_REF] Lin | Feature pyramid networks for object detection[END_REF] introduced feature pyramid network (FPN) to aggregate features from several network layers. Finally, Mask R-CNN [START_REF] He | Mask R-CNN[END_REF] extended this model to instance segmentation by adding a mask prediction branch.

In this work, we convert this architecture into an object tracker by introducing a lightweight discriminative template matching block before the RPN. The resulting attention map guides the RPN to propose only boxes corresponding to the object of interest. Disabling the matching component turns the model into a standard object detector.

Method

An ideal model for tracking by detection can be described as a generic object detector that can be efficiently adapted to detect a specific object in a specific scene. In this section, we propose such an approach, shown in Fig- We begin by briefly describing the Mask R-CNN architecture in Section 3.1. We then discuss our strategy of incorporating Siamese-like template matching into this model in a principled way in Section 3.2. Next, we propose our discriminative templates that efficiently integrate information about the distractors in Section 3.4. Finally, we discuss our strategy for training the unified model on object detection, tracking, and video segmentation datasets in Section 3.5.

Preliminaries

A Mask R-CNN detector, shown in Figure 2, consists of a backbone network (often a ResNet), a Region Proposal Network, and bounding box classification, regression and mask prediction heads. The former takes a frame as input and outputs a set of feature maps {C 1 , C 2 , C 3 , C 4 , C 5 }, extracted from the respective blocks of the backbone and encoding the image with different degrees of spatial and semantic granularity. In practice, the output of the first block is discarded, due to memory constraints. The remaining feature maps are then updated via top-down lateral connections to propagate the information for the coarse but semantically rich top layers to the more spatially precise bottom layers, resulting in the final set of feature maps {P 2 , P 3 , P 4 , P 5 } (see [START_REF] Lin | Feature pyramid networks for object detection[END_REF] for details). The feature dimensionality of these maps is fixed to 256, but their spatial dimensions decrease from fine to coarse, thus the resulting architecture is referred to as Feature Pyramid Network (FPN).

An RPN is implemented as a 3 × 3 convolutional layer that is applied to each FPN level in a sliding window fash-ion, outputting an objectness score for each of the anchor boxes centered at the corresponding location. Crucially, the anchor boxes only capture various aspect ratios of the boxes, wheres scale variation is handled by the FPN. That is, a 1 × 1 × D dimensional feature at location (x, y) in P 5 represents the largest possible object centered at that location, whereas a feature of the same dimension at the corresponding location in P 2 represents the smallest possible object in centered in the same region. We use this observation to derive our scale-invariant object template in Section 3.2.

Finally, the top k boxes according to the RPN score are selected, and an ROI-Pool operation is used to convert their feature representations to a fixed size. The resulting features are passed to separate bounding box classification, regression, and mask prediction branches (see [START_REF] He | Mask R-CNN[END_REF] for details). We now describe our approach to efficiently adapting this architecture to the task of object tracking.

Tracking as generalized object detection

Given a bounding box around the object of interest in the first frame, how can we adapt the Mask R-CNN detector to only track that specific instance? We take inspiration from Siamese-based approaches for tracking that store an object template from the first frame and compute a similarity between the template and the test frame representation in a sliding-window fashion. Differently from those methods, instead of localizing the objects directly via template matching, we use the resulting similarity map to reweight the feature representation of an object detector. This allows us to reuse the rest of the detection architecture and train the model jointly on images and videos.

Our key observation is that every object can be represented by a 1 × 1 × D feature T in one FPN layer, corresponding to its scale and center location. Thus, we begin by extracting the corresponding representation for the template box in the first frame. In the standard detection training setup, Mask R-CNN assigns each groundtruth box in an image to a specific level in the feature pyramid, adding a loss that enforces that features at that scale generate a proposal around the groundtruth box. We use this same mapping to map our template box to the corresponding FPN level, and use the feature in that level that corresponds to the center of the template box. At test time, however, the scale of the object might have changed. Conveniently, we do not need to update the template to account for this, since scale variation is already handled by the FPN. Thus, we simply compute the similarity maps at all the levels of the feature pyramid via S i = P i T , where stands for cross-correlation.

Next, instead of directly using the resulting similarities to localize the object, we propose to instead treat them as attention maps to guide the detector. To this end, we update the original FPN representations via P i ← P i • S i , where • stands for the dot product. Notice that this operation simply reweights the original representation, preserving the information used by the RPN in the next stage. Thus, we can naturally capitalize on the strong objectness prior learned by detectors on COCO, as well as learn to produce objects masks for free. This re-weighted feature representation is used to generate and pool features for region proposals. The pooled, re-weighted features are finally passed through class-agnostic bounding box and mask regression heads.

At test time, our model produces multiple detections with confidence scores at every video frame. By default, we select the highest-scoring detection to construct the track, but we can make use of multiple detections by re-ranking them with external cues, such as predictions of an object dynamics model, or temporal smoothness cues (Section 4.2).

Joint Detection and Tracking

The modifications described above convert a standard Mask R-CNN detector into a tracker, which can not directly be used as a standard detector. We present two modifications that allow the tracker to be trained and evaluated as a standard, image-based detection model. First, when applied to a single image, we disable the attention module. Equivalently, this can be thought of as setting the attention to a uniform value of 1 at all pixel locations. Second, in order to output a class-specific bounding box and mask, as in standard Mask R-CNN, we instantiate a separate final layer for the box and mask regression heads for detection. Note that our model shares all parameters for detection and tracking except this final fully connected layer. We show in Section 4.3 that training jointly for detection with tracking improves tracking accuracy, while allowing our model to operate as a powerful single-frame detector, which can be useful for identifying distractor objects during tracking.

Discriminative Templates

Consider the frames from one of the videos in GOT shown in Figure 3 together with the corresponding similarity map S i from the appropriate FPN level of our model. The model is supposed to track the cart in this video, however, the similarity map for the test frame shown in the bottom left is not localized on the object. We now propose a simple and efficient way of learning a discriminative template, increasing the robustness of the tracker.

Recall that in the FPN a feature vector at each location encodes an object centered at that region at the corresponding scale. Thus, sampling a large enough pool of features from all the levels outside of the ground truth bounding box naturally provides us with a training set for learning a linear discriminator for the object of interest in a given video. Moreover, such a discriminator can be found efficiently in a closed form via least squares. In particular, given a template T and a set of negatives N = {n 1 , n 2 , . . . , n q }, we define Figure 3: The effect of our proposed discriminative template on an example of a video from GOT-10k dataset [START_REF] Huang | GOT-10k: A large highdiversity benchmark for generic object tracking in the wild[END_REF]. By simply using the center feature of the bounding box around the cart (top left), the resulting attention map (bottom left) for the test frame (top right) is not focused on the object. In contrast, our discriminative template (bottom right) results in a much better attention map. the data matrix A, and the label vector y as follows: A = T ; n 1 ; n 2 ; . . . ; n q , y = 1; 0; 0; . . . ; 0

We then want to find a vector T d , which we call a discriminative template, that minimizes

AT d -y 2 2
(2) holds. A closed form solution is available via:

T d = (A T A + λI) -1 A T y, (3)
where I is the identity matrix and λ is a regularization hyper-parameter. We then use T d to compute the similarity maps in the same way: S i = P i T d . Note that computing T d requires only a matrix inverse and matrix multiplications, operations which are fully differentiable in the elements of A, and can be implemented in standard deep learning frameworks. Thus, we can backpropagate though this computation. This guides the backbone to learn a feature space where objects can be separated via a linear classifier in an end-to-end manner.

Figure 3 (bottom right) shows that our discriminative template indeed significantly increases the precision of the similarity maps by incorporating the information about the distractors in a principled way. We now describe how we train our unified framework on dataset for object detection, tracking, and video segmentation.

Training

We first train our model on COCO for object detection [START_REF] Lin | Microsoft COCO: Common objects in context[END_REF] following Mask R-CNN training [START_REF] He | Mask R-CNN[END_REF]. We then transfer the learned objectness prior to the tracking task.

To this end we add the discriminative template computation, and attention reweighting components described above, and fine-tune it on the ImageNet VID [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] and YTVOS [START_REF] Xu | YouTube-VOS: A large-scale video object segmentation benchmark[END_REF] datasets. As ImageNet Vid does not provide segmentation groundtruth, we do not use it to update the mask branch. When fine-tuning for tracking, we make three simple modifications: (1) Our training batches consist of pairs of frames: for a batch of size K, we sample K videos, and then sample a template frame and a search frame at random from the video1 ; (2) We re-weight FPN features of the search frame using the feature corresponding to the template frame's bounding box; (3) Only a single, classagnostic groundtruth box is used for training, which is the one corresponding to the tracked object in the search frame. These minor modifications allow us to maximally preserve the objectness priors learned on COCO.

Experiments

We begin with introducing the datasets used to train and evaluate our model, and providing the implementation details. Next we analyze the various choices made while designing our approach in Section 4.3. Finally, we compare our method to the state-of-the-art in Section 4.4.

Datasets and evaluation

We use the COCO [START_REF] Lin | Microsoft COCO: Common objects in context[END_REF] dataset to train our model for object detection, and ImageNet VID and YTVOS [START_REF] Xu | YouTube-VOS: A large-scale video object segmentation benchmark[END_REF] to train the tracking module. We evaluate on two very recent, large scale tracking benchmarks: OxUvA [START_REF] Valmadre | Long-term tracking in the wild: a benchmark[END_REF] for long term tracking and GOT [START_REF] Huang | GOT-10k: A large highdiversity benchmark for generic object tracking in the wild[END_REF] for tracking of diverse objects. In addition, we use the DAVIS'17 [START_REF] Pont-Tuset | The 2017 DAVIS challenge on video object segmentation[END_REF] dataset for video object segmentation to evaluate the quality of the masks produced by our tracker, and the LTB35 videos from the VOT 2018 long term challenge to benchmark [START_REF] Kristan | The sixth visual object tracking vot2018 challenge results[END_REF] against prior submissions to the challenge. We describe each of these datasets in more detail in the supplementary material.

Implementation details

Network architecture and training We use the Mask R-CNN detection framework throughout our experiments. In particular, we use the ResNet-50 FPN backbone, which achieves a useful balance between accuracy and efficiency. Our final model is trained for detection on MS COCO, as described in Sec. 4.3 and for bounding-box tracking on Im-ageNet VID and YTVOS. We will release training and evaluation code along with trained models upon acceptance. As these heuristics can obscure the improvement of the underlying matching approach, we show results using no temporal information in Sec. 4.3 and 4.4, and show state-ofthe-art results without heuristics on OxUvA. For completeness, we implement one simple heuristic which we ablate in Sec. 4.3. At every frame t, our detector outputs a set of candidate detections D t = {d 1,t , . . . , d k,t }, along with a confidence score c i,t for each detection. In our standard implementation, we select the detection d * t with the highest confidence. To incorporate temporal smoothness, we implement a simple heuristic: for each candidate box d i,t , compute the mask intersection-over-union j i,t with the predicted detection d * t-1 at the previous frame, and update the confidence as c i,t ← αc i,t + (1 -α)j i,t . Then, we select d * t as the detection that maximizes this reweighted confidence. We set α = 0.6 for all of our experiments. In order to avoid latching onto distractors, we temporarily disable this smoothness component if the track is broken, i.e. the IoU between the object locations at time t and t + 1 is small (< α low ); we re-enable the smoothness component if we maintain a smooth track for n frames, i.e. a track with consecutive object locations that have IoU > α recover . We set α low = 0.1, α recover = 0.3, and n = 30. We always show results both with and without this component for clarity.

Temporal heuristics

Ablation study

In this section we analyze the influence of different components of our approach on the final performance. We use the dev sets of OxUvA and GOT-10k datasets for analysis, due to their complexity and diversity. Note that OxUvA requires explicitly thresholding confidence scores in order to detect when an object is not present in the video. For ablation, we report the area under the ROC curve (i.e., TPR vs. FPR curve) to better understand the performance of ablated components across score thresholds. GOT-10k does not require setting such a threshold, so we use the standard Average Overlap metric described in Section 4.1.

We start with a baseline variant of our approach, which is trained only on videos labeled for tracking and achieves 63.2 OxUvA AUC and 64.7 GOT AO. Next, we evaluate the importance of object priors by pretraining our model on COCO as a generic object detector. This variant, shown in row 2, results in an 1.7% improvement in OxUvA AUC and a 3.7% improvement in GOT AO. Next, we train our model for detection and tracking jointly. As expected, this multitask training strategy provides a modest bump on both Ox-UvA and GOT, leading to a model that improves in tracking while additionally being able to perform single-image detection. These improvements confirm our intuition that object priors are critical for tracking, and that the universal nature of our model is indeed helpful in transferring information from object detection datasets.

As described in Sec. 3.4, our framework is flexible, admitting various strategies for computing a discriminative template, T d . We analyze a few strategies for computing this template. In particular we compare our proposed linear regression framework (denoted as T d ='lin. reg.') to two simple baselines: a non-discriminative one, that simply averages several features vectors sampled from the ground truth box (denoted with T d = 'mean pos'), and a discriminative one that uses the difference between the means of positive and negative samples as a template (denoted with T d ='mean diff'). Note that these can be seen as special cases of linear regression. First, we observe that all these variants increase the model's performance, but the linear regression approach results in the largest improvement of 5.3% OxUvA AUC and 0.9% GOT AO. Second, the 'mean diff' baseline shows the worst performance, which is counterintuitive. We attribute this result to the fact that simply subtracting the mean of the negative examples from the template leads to unstable behavior during training. In contrast, our principled approach to computing the template simplifies optimization. Finally, we show that incorporating the temporal smoothness (Section 4.2) provides significant improvements, particularly for short-term tracking as in GOT.

Discussion. Performing ablations on two diverse tracking datasets allows understanding the impact of ablated components for different challenges. For example, the use of detection priors seems to be significantly more pronounced in GOT, which requires tracking a diverse set of objects, than for OxUvA. This is to be expected: while video datasets are large enough to learn priors for common objects, imagebased datasets like COCO provide priors for more diverse categories. Meanwhile, our discriminative templates provide a significant improvement on OxUvA, but a more modest improvement on GOT. We attribute this to the fact that our discriminative template is able to avoid latching onto distractors when the object of interest disappears, a phenomenon that is far more common in the long-term OxUvA dataset than on GOT. Approach TPR TNR GM LCT [29] 22.7 43.2 31.3 MDNet [START_REF] Nam | Learning multi-domain convolutional neural networks for visual tracking[END_REF] 42.1 0 32.4 TLD [START_REF] Kalal | Tracking-learningdetection[END_REF] 14 [START_REF] Valmadre | Long-term tracking in the wild: a benchmark[END_REF]. As these methods perform poorly, we first run recent state-of-the-art trackers (DaSiam [START_REF] Zhu | Distractor-aware siamese networks for visual object tracking[END_REF], SiamMask [START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF], and SiamRPN++ [START_REF] Li | Siamrpn++: Evolution of siamese visual tracking with very deep networks[END_REF]). We show that our approach significantly improves over both prior state-of-the-art, as well as these recent works.

Approach AO SR

DaSiam [START_REF] Zhu | Distractor-aware siamese networks for visual object tracking[END_REF] 46.0 54.3 SiamMask [START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF] 66.8 78.3 SiamRPN++ [START_REF] Li | Siamrpn++: Evolution of siamese visual tracking with very deep networks[END_REF]

Comparison to the state-of-the-art

We now compare our full approach to state-of-the art methods in object tracking and video object segmentation.

OxUvA evaluation We begin by presenting comparisons on the dev set of the OxUvA long term tracking benchmark in Table 2. We compare to the state-of-the-art approaches reported in [START_REF] Valmadre | Long-term tracking in the wild: a benchmark[END_REF]. As the approaches reported in [START_REF] Valmadre | Long-term tracking in the wild: a benchmark[END_REF] perform poorly qualitatively and quantitatively, we further evaluate two more recent trackers: DaSiam [START_REF] Zhu | Distractor-aware siamese networks for visual object tracking[END_REF] and SiamMask [START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF] on this dataset. We use their publicly available code.

As shown in Table 2, our evaluation of DaSiam [START_REF] Zhu | Distractor-aware siamese networks for visual object tracking[END_REF], SiamMask [START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF], and SiamRPN++ [START_REF] Li | Siamrpn++: Evolution of siamese visual tracking with very deep networks[END_REF] out-performs the methods reported in [START_REF] Valmadre | Long-term tracking in the wild: a benchmark[END_REF]. Next, we evaluate our approach without using any temporal information in the "w/o temporal" row. This variant is completely stateless, and individually performs matching on each frame of the video. By contrast, almost all prior tracking approaches use heuristic tem-Approach P R F SiamMask [START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF] 64.5 46.8 54.3 DaSiam-LT [START_REF] Zhu | Distractor-aware siamese networks for visual object tracking[END_REF] 62.7 58.8 60.7 MBMD [START_REF] Zhang | Learning regression and verification networks for long-term visual tracking[END_REF] 63.4 58.9 61.0 SiamRPN++ [START_REF] Li | Siamrpn++: Evolution of siamese visual tracking with very deep networks[END_REF] 65.0 61.0 62.9

Ours w/o temporal 61.0 56.9 58.9 Ours 61.2 61.2 61.2 poral smoothing to improve the performance of their models. Despite this lack of temporal information, our approach outperforms all but the very recent SiamRPN++ approach, including the recent work of [START_REF] Zhu | Distractor-aware siamese networks for visual object tracking[END_REF][START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF]. By adding the simple temporal heuristic described in Sec. 4.2, we further improve our results by 0.8%, achieving state-of-the-art results. We report results on the held out test set in supplementary.

GOT evaluation To validate our conclusions above, we further evaluate our approach on GOT-10k. As prior methods evaluated on GOT-10k use only the GOT-10k training set for training, we can not fairly compare our approach to them. Instead, we use [START_REF] Zhu | Distractor-aware siamese networks for visual object tracking[END_REF][START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF][START_REF] Li | Siamrpn++: Evolution of siamese visual tracking with very deep networks[END_REF] as baselines, which are the best method prior to ours on OxUvA. We report results on the validation set in Table 3, and additionally show results on the test set in the supplementary material. As can be see from the table, we outperform all of these works by over 4%, which we attribute to the ability of our method to generalize to diverse object categories.

LTB-35 Evaluation

We compare to state-of-the-art methods on the LTB-35 benchmark, which was used to evaluate long term tracking in the VOT 2018-LT challenge. This dataset focuses on tracking in videos over 2 minutes long on average, where the object of interest can frequently disappear and reappear in the video. We compare to state-of-theart results in Table 4, as well as SiamMask [START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF]. Note that prior approaches, other than [START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF], provide dataset-specific hyperparameters that are tuned for this dataset. By contrast, we use a single model, a single set of hyperparameters, and a simple temporal heuristic across all datasets. Despite this, our method obtains a competitive F-measure of 61.2 on this dataset while outperforming on other datasets.

Mask evaluation on DAVIS'17 Finally, we evaluate our unified approach on the task of video object segmentation.

To this end we use the validation set of DAVIS '17, and results are presented in Table 5. We show qualitative results of our method in the supplementary material. All methods in Table 5, with the exception of SiamMask and our method, require pixel-perfect segmentation in the first video frame and operate at a speed of less than 2 frames-per-second. By contrast, our method adds only a small overhead to the underlying detection model used. For our experiments, we used a ResNet-50 FPN backbone for Mask R-CNN, which led to a speed of approximately 7FPS. Despite using less supervision and computational time, our approach is competitive with dedicated video segmentation methods that use pixel-level masks in the first frame.

Detection evaluation on COCO As discussed in Section 3.3, our model can be used as a detector at test time. Although our focus is on tracking, we find that our model outputs high quality detections, providing a COCO instance segmentation mAP of 30.5, compared to 34.4 for an equivalent standalone detector that cannot track objects. We show qualitative results in Fig. 4, comparing our results with SiamMask [START_REF] Wang | Fast online object tracking and segmentation: A unifying approach[END_REF] and SiamRPN++ [START_REF] Li | Siamrpn++: Evolution of siamese visual tracking with very deep networks[END_REF]. By leveraging objectness information, our method is able to successfully localize the full extent of objects, while providing precise instance segmentation masks for the object of interest. Further, we note an intriguing phenomenon due to our use of objectness: in failure cases, as in the bottom row of Figure 4, our method behaves qualitatively differently from prior work. When the object of interest is not visible, our method will report a mask with low confidence. Even in these cases, our method almost invariably segments objects, rather than drifting onto background regions or parts, as with prior methods.

Conclusion

This paper introduces a novel generic object tracking approach built on top of a state-of-the-art object-detection framework. The resulting model can be trained jointly for the two tasks, effectively incorporating objectness priors into tracking. Additionally, we propose learning discriminative templates in a fully differentiable manner that encode information both about the object of interest and about the distractors, increasing the tracker's robustness. Finally, we extend our method to the related task of video object segmentation by simply adding a mask prediction branch.

Our resulting framework for tracking and video segmentation demonstrates state-of-the-art results on two recent tracking datasets (OxUvA and GOT10k), and also shows competitive performance on the DAVIS'17 benchmark for video object segmentation. We empirically show that these improvements are largely due to the generic objectness prior learned from the COCO dataset.

Figure 1 :

 1 Figure 1: Objects of interest in generic, user-initialized tracking share a common set of objectness traits. Our approach (a) learns a generic objectness prior from imagebased datasets, and (b) adapts it to a specific object of interest (e.g. the bus in the top left) by computing a linear discriminator between the object and its background in closed form. This allows tracking objects through significant deformations without latching onto distractors.

Figure 2 :

 2 Figure2: Overview of our approach. First, we use a state-of-the-art object detector[START_REF] He | Mask R-CNN[END_REF] to extract features for the template image containing the object to be tracked (top left). Next, we compute a discriminative template that separates the features corresponding to the tracked object from the distractors in the first frame using linear regression (top right). Finally, attention masks computed with this template are used to reweight the feature maps of the detector to focus on the object of interest (bottom). Note that unlike standard, category-specific detectors, our box-head and mask-head output a single, categoryagnostic prediction for the tracked object.

  ure 2. Our model leverages advances in standard object detection architectures by progressively incorporating modifications to build a state-of-the-art tracker, while maintaining the model's detection capabilities.

4. 4 . 1

 41 Qualitative results.

Table 1 :

 1 Prior tracking approaches rely heavily on temporal information to simplify the tracking problem. Evaluating the influence of different components of our approach on the OxUvA dev and GOT-10k val sets.See Section 4.3 for details.

	Det Init?	Joint Det Train?	T d	Smooth?	OxUvA AUC	GOT AO
			center		63.2	64.7
			center		64.9	68.4
			center		65.8	68.6
			mean diff		67.6	68.8
			mean pos		69.1	69.1
			lin. reg.		71.1	69.5
			lin. reg.		72.1	73.0

Table 2 :

 2 In the top half, we show the current reported stateof-the-art results on OxUvA, from

		.1 94.9 36.6
	SiamFC + R [6]	35.4 43.8 39.7
	DaSiam [53]	40.0 84.2 58.0
	SiamMask	50.4 88.7 66.9
	SiamRPN++	63.6 79.9 71.3
	Ours w/o temporal 63.2 79.1 70.8
	Ours	65.5 78.2 71.6

Table 4 :

 4 F-measure on LTB35 (VOT 2018-LT challenge). Unlike many prior methods, we use a single model across all our experiments that is competitive on VOT while outperforming on other datasets.

Table 5 :

 5 Qualitative comparison of our method with prior work. Top row: While SiamMask suffers from drift, and SiamRPN++ provides a loose bounding box around the object, our method provides a tight segmentation mask. Bottom row (failure case): All trackers fail to track the car of interest, which is nearly invisible in the distance. Intriguingly, while prior methods drift to background regions or parts (as with SiamMask and SiamRPN++), our method almost invariably latches onto objects. DAVIS '17 validation results with intersectionover-union (J ) and F-measure (F). Most prior methods require a labeled mask in the first frame ('Mask sup') or perform computationally expensive end-to-end fine-tuning per video ('Deep FT' row), our method efficiently and accurately segments objects without mask supervision.

			Init			SiamMask	SiamRPN++	Ours
	Figure 4: Measure	PReMVOS CINM FeelVOS	SiamMask Ours
			[28]	[4]	[41]	[43]
		Mask sup?
		Deep FT?
		Mean	73.9 67.2	69.1	54.3 59.2
	J	Recall	73.1 74.5	79.1	62.8 68.6
		Decay	16.2 24.6	17.5	19.3 8.4
		Mean	81.8 74.0	74.0	58.5 67.8
	F	Recall	88.9 81.6	83.8	67.5 76.1
		Decay	19.5 26.2	20.1	20.9 12.0

compare to the state-of-the-art approaches, including the ones that require finetuning the model on the test sequences. The

While we could limit the template frame to be the first frame of the video (as at test time), this would drastically reduce the diversity of our frame pairs.


MD Scott W Canna
email: scott.canna@chp.edu.

MD Charlotte Girard

BS Louise Malle

MD Adriana De Jesus

MD Neil Romberg

MD Judith Kelsen

MD Lea F Surrey

MD Pierre Russo

PhD Andrew Sleight

MD Eduardo Schiffrin

MD Cem Gabay

MD Raphaela Goldbach-Mansky

MD Edward M Behrens

R G-M

with the correlation of total IL-18 and macrophage activation syndrome (MAS)-prone conditions, they found elevated free IL-18 specifically in patients with Still's disease, and also observed a higher IL-18/IL-18BP affinity than previously described. Further investigation is warranted to better understand discrepancies between calculated and measured free IL-18, and appropriate caution should be taken in the interpretation of calculated free IL-18 until the aforementioned assumptions have been addressed.

This report provides the first clinical clues that the association between free IL-18 and MAS is no mere epiphenomenon. Specifically, the magnitude and chronicity of elevated IL-18 in NLRC4-related MAS (and in most systemic juvenile idiopathic arthritis 5 ) might suggest a ''priming'' effect for exuberant MAS-like responses. However, this patient's rapid improvement also implicates a role for IL-18 in propagating inflammation. In addition, the patient's enterocolitis was associated with substantially more free IL-18 in stool than in serum, suggesting less buffering by IL-18BP at mucosal surfaces. Overall, the important work initiated nearly 2 decades ago in describing the endogenous regulation of IL-18 continues to undergo scientific, and now clinical, refinement as we better understand the homeostatic and pathogenic functions of this cytokine.

Polyfunctionality of bona fide resident lung CD69 1 natural killer cells

To the Editor:

We read with great interest the recent article from Marquardt et al 1 dedicated to the detailed characterization of human lung natural killer (NK) cells. On the basis of the expression of CD69, they nicely distinguished bona fide resident CD69 1 from circulating CD69 2 CD56 dim NK cells in a large cohort of patients undergoing pulmonary lobectomy. They have shown that NK cells in the lung are highly differentiated and predominantly circulating ones, which contrasts with lung CD3 1 T cells.

Although we do appreciate their comprehensive work, we would like to express 2 comments and queries.

In accordance with some of our previous works, the authors reported that NK cells were hypofunctional:

d In patients with lung carcinoma, we analyzed the phenotype and functions of lung NK cells and observed that the intratumoral NK cells were significantly hypofunctional and more commonly CD69 positive as compared with other NK cell populations. 2 In light with Marquardt et al, 1 we could then suggest that most of these intratumoral NK cells are resident. d In patients with antisynthetase syndrome (aSS)-a rare autoimmune connective tissue disease characterized with interstitial lung disease (ILD) in the foreground 3 -peripheral blood NK cells are highly differentiated and hypofunctional as well, irrespective of the IL-15 serum titers. 4,5 Immunohistological studies in the lungs showed a massive infiltration of NK cells. 4 In addition, we recently observed in a 39-year-old woman with active aSS that the proportion of CD69 1 CD3 2 CD56 1 NK cells was higher in bronchoalveolar lavage as compared with blood (37% vs 16%, respectively). In respect with Marquardt et al, 1 this would suggest that NK cell infiltration in aSS-related-ILD mainly consists of resident NK cells.

We thus regret that the authors studied lung NK cell functions as a whole, regardless of their status of circulating or resident lung NK cells. We would be very interesting in knowing whether the authors showed any differences in polyfunction assays, according to the CD69 expression of the lung NK cells. Furthermore, hypofunctionality of lung NK cells was determined after in vitro stimulation with nonspecific stimuli, including K562 target cells and phorbol 12-myristate 13-acetate and ionomycin or IFN-a. We think it would be of interest to evaluate the proliferation and polyfunctional capacities of both CD69 2 and CD69 1 lung NK cells, following IL-15 stimulation. Indeed, IL-15 is locally produced within the lung 1,6 and involved in NK cell regulation (differentiation and functionality). 7 We think that such complementary results might expand our knowledge about the distinct role of circulating and resident lung NK cells. Furthermore, these comprehensive data might help in understanding the involvement of NK cells in different diseases commonly affecting the lung, including cancer and ILD, and thus could lead to the development of new lung-targeted therapeutic strategies. 7

Baptiste

Reply

To the Editor:

We thank Hervier et al 1 for providing comments on our article 2 characterizing natural killer (NK) cells in the human lung. Based on analysis of CD69 expression, our results suggest that most CD56 bright NK cells are tissue-resident, whereas most CD56 dim NK cells in the lung circulate. 2 Our results also indicate that the total lung NK cell population is largely hypofunctional. Hervier et al 1 regret the fact that we studied NK cell functionality only as a whole, that is, regardless of the cells' status as tissue-resident (CD69-expressing) or circulating. 1 Furthermore, they express that it also would have been of interest to study NK cell function following stimulation with IL-15, which is locally produced in lung. We believe that these are important questions worth considering in more detail.

In our study, we showed that both CD56 bright and CD56 dim NK cells in lung tissue are hypofunctional compared with the corresponding subsets in peripheral blood. Sorted CD56 bright lung NK cells produced less TNF and GM-CSF after stimulation with PMA/ionomycin. Similarly, sorted CD56 dim NK cells responded less well in response to stimulation with K562 cells and rituximab-coated 721.221 cells, as measured with assays for both degranulation and production of TNF. Overall, our results showed that lung CD56 dim and CD56 bright NK cells are hyporesponsive to several distinct stimuli and by multiple functional readouts. 2 Given that most CD56 bright NK cells express CD69, our results indicate that the lung-resident NK cells are also hyporesponsive. In preliminary experiments, lung NK cells responded to IL-15 by upregulation of Ki67, a marker of proliferation, and increased expression of granzyme B (unpublished data, Marquardt et al, 2017). However, more extensive comparative studies are required to draw more firm conclusions whether there are distinct effects of IL-15 on tissue-resident NK cells.

Several recent studies of tissue-resident lymphocytes, including NK cells, have revealed that there is considerable phenotypic and functional heterogeneity among tissue-resident lymphocyte subsets. 3 In addition to CD69, tissue-resident lymphocytes can be characterized by the expression of adhesion molecules, for example, CD49a and CD103. These molecules facilitate retention in specific parts of tissues. For example, CD103 binds to E-cadherin, which is predominantly expressed on epithelial cells and, as such, identifies intraepithelial tissue-resident lymphocytes. 4,5 Although the expression of CD69 on NK cells in tissues provides a good indication of tissue-residency, CD69 is also upregulated upon NK cell activation. Therefore, analysis of CD69 in combination with other markers of tissue-residency, for example, CD103 and CD49a, would be desirable to further dissect the characteristics of bona-fide human lung-resident NK cells.

The recent findings by Hervier et al of increased frequencies of NKp46 1 cells in lungs of patients with antisynthetase syndrome, 6 as well as reports on increased CD69 expression on intratumoral NK cells, 7 further highlight the need for more detailed studies of human lung-resident NK cells, in health and various settings of human disease. Indeed, a better understanding of these cells could lead to improved insights into disease pathogenesis, as well as promoting the identification of new drug targets or diagnostic tools in lung-associated conditions.

Nicole Marquardt, PhD

Hans-Gustaf Ljunggren, MD, PhD Jakob Micha€ elsson, PhD

  Hervier, MD, PhD a,b Isabelle Cremer, PhD c Vincent Vieillard, PhD a From a Sorbonne Universit es, Centre d'Immunologie et des Maladies Infectieuses (CIMI-Paris), Paris, France; b AP-HP, Hôpital Piti e-Salpêtri ere, D epartement de M edecine Interne et Immunologie Clinique, Centre National de R ef erence des Maladies Musculaires, Paris, France; and c Universit e Paris Descartes-Paris 5, Centre de Recherche des Cordeliers, Paris, France. E-mail: baptiste.hervier@psl.aphp.fr. Disclosure of potential conflict of interest: The authors declare that they have no relevant conflicts of interest.

were supported by the Intramural Research Program of the National Institute of Arthritis, Musculoskeletal, and Skin Diseases. E.M.B. was supported by the National Institutes of Health (grant no. R01 HL112836-A1), the Nancy Taylor Foundation, and Sean Fischel Connect. C.Ga. is supported by the Swiss National Science Foundation (grant no. 10030_152638), the Rheumasearch Foundation, and the Institute of Arthritis Research. Disclosure of potential conflict of interest: C. Girard, A. Sleight, and E. Schiffrin are employed by AB2Bio. P. Russo is employed by the University of Pennsylvania; has received money from Bob, Ciotoli law firm for expert testimony; received grants DK 62481 and UL1 TR000003 for other works; and received royalties from Springer. C. Gabay's institution received a grant and consulting fees from AB2Bio for this work and grants from Roche for other works. C. Gabay personally received consultancy fees from Roche, Abbvie, Pfizer, BMS Novarits, Sanofi, and Regeneron for other works. R. Goldbach-Mansky's institution received grants from Regeneron, Lilly, and Novartis for this work. E. M. Behrens has personally received board membership from the American Board of Pediatrics and consultancy fees from AB2BIO; is employed by the Children's Hospital of Philadelphia; and received grants from the National Institutes of Health. The rest of the authors declare that they have no