Quoc Cuong Le

Donatello Conte
email: donatello.conte@univ-tours.fr

Moncef Hidane
email: moncef.hidane@insa-cvl.fr

Unbalanced Optimal Transport in Multi-Camera Tracking Applications

Keywords: Multi-object Tracking, Multi-view Tracking, Unbalanced Optimal Transport

Multi-view multi-object tracking algorithms are expected to resolve multi-object tracking persistent issues within a single camera. However, the inconsistency of camera videos in most of the surveillance systems obstructs the ability of re-identifying and jointly tracking targets through different views. As a crucial task in multi-camera tracking, assigning targets from one view to another is considered as an assignment problem. This paper is presenting an alternative approach based on Unbalanced Optimal Transport for the unbalanced assignment problem. On each view, targets' position and appearance are projected on a learned metric space, and then an Unbalanced Optimal Transport algorithm is applied to find the optimal assignment of targets between pairs of views. The experiments on common multi-camera databases show the superiority of our proposal to the heuristic approach on MOT metrics.

Introduction

Multiple Object Tracking (MOT) is still one of the most challenging and vital problems in computer vision. Therein, the goal is to determine the position and identity of a variable number of targets throughout video frames. In the past recent years, the rise of deep learning approaches [START_REF] He | Deep residual learning for image recognition[END_REF] has led to an increase in the performance, robustness and reliability of Single Object Tracking (SOT) algorithms. Implementing these SOT trackers to track multiple objects simultaneously, however, appears challenging for many typical reasons, such as initialization step at every frame, interactions between targets causing frequent mutual occlusions and identity switches.

A popular approach to track multiple objects is to adopt the tracking-bydetections paradigm ( [START_REF] Andriluka | People-tracking-by-detection and peopledetection-by-tracking[END_REF][START_REF] Zamir | Gmcp-tracker: Global multi-object tracking using generalized minimum clique graphs[END_REF]), relying on detections at every frame. This approach has been reinforced in the recent years since many powerful object detection algorithms ([16, 29, 23]), e.g., Faster R-CNN, Mask R-CNN, YOLO, SSD, have emerged and even outperformed humans in the past recent years. In principle, the detection-based MOT methods directly link together detections which belong to targets, on the entire videos, in order to form their final trajectories. Tracking-by-detection MOT methods lead to data association algorithms, often formulated as a global optimization problems in which a graph-based representation of detections with edges weighted by a distance (or similarity) is adopted. The distance between detections mostly includes Euclidean distance, time delay, and appearance affinity [START_REF] Zamir | Gmcp-tracker: Global multi-object tracking using generalized minimum clique graphs[END_REF][START_REF] Berclaz | Multiple object tracking using kshortest paths optimization[END_REF][START_REF] Tang | Subgraph decomposition for multitarget tracking[END_REF][START_REF] Sadeghian | Tracking the untrackable: Learning to track multiple cues with long-term dependencies[END_REF]. The online/offline distinguishes between methods that solely use results from previous detections, and the ones that consider the whole (or batch) time-sequence in order to compute data associations.

Both SOT-based and online association-based methods require an efficient way to control the state of targets in order to prevent missing tracks, occlusions, and identity switches. For SOT-based methods, Markov Decision Processes (MDP) were adopted in the papers [START_REF] Xiang | Learning to track: Online multi-object tracking by decision making[END_REF][START_REF] Zhu | Online multi-object tracking with dual matching attention networks[END_REF] to tackle this issue. This approach has been extended to an overlapping multiple camera setting in [START_REF] Le | Online multiple view tracking: Targets association across cameras[END_REF] and has shown capability in allowing the individual cameras to recapture/re-identify their lost targets.

Within multiple camera systems, Multi-Camera Multi-Object Tracking (MC-MOT) or Multi-Target Multi-Camera Tracking (MTMCT) problem is frequently formulated as an assignment problem or, in many cases, the re-identification/ recognition problems as the object of interest is mainly human or transport vehicle. Since data association approaches are extendable in multi-camera cases, most of MCMOT algorithms of the state-of-the-art are mainly derived from singlecamera association-based MOT approaches. Indeed, the role of multi-camera tracking is to link detections or tracklets across cameras in the network. Generally, associating targets between two views is formulated as assignment problem, or bi-graph matching problem, which is originally resolved by Hungarian or Munkres algorithms. This is not the case of MOT because of the varying target number. As a solution to this issue, the modified version of the Munkres algorithm is used with virtual targets.

In our case study, we address the target association problem between different views within an overlapping camera system for online multi-camera applications. The target matching is well defined by the unbalanced assignment problem, in which the number of targets in one view is not equal to those in another view. In this paper, we propose a novel assignment approach formulated as an unbalanced optimal transport problem for multi-view tracking applications. Our second contribution is to develop a deep distance learning framework for Optimal Transport. Our third contribution is to adopt the target association between two cameras within multiple camera systems. Our multi-camera tracking framework is functional with mere pairs of cameras, which is called as "dual-camera" approach in the multi-camera tracking problem (Fig. 1). Our approach helps elevate the all-camera condition, renders it more flexible to any number of cameras inside the camera system, and essentially adapts well with our proposed assignment problem at the early of this paper.

The structure of this paper is as follows: we first mention the works related to ours; secondly, we describe the formulation of our dual-camera target association problem as unbalanced optimal transport; then introduce our proposed distance learning method of Optimal Transport based on a deep neural network; and finally, we present our experimental results showing the advantage of our method. Since handling multiple different targets is the main work of MOT algorithms, the tracking-by-detection paradigm has evolved as their major approach. This is especially true as the result of the advent of high-performing category detectors.

Tracking-by-detection approaches can be sorted into the following two groups.

Offline approaches Following the tracking-by-detection paradigm [START_REF] Zamir | Gmcp-tracker: Global multi-object tracking using generalized minimum clique graphs[END_REF][START_REF] Berclaz | Multiple object tracking using kshortest paths optimization[END_REF], graph optimization problems are formulated to create links between detections of targets, in successive frames, then the chain of detections through frames determines the full trajectory of a target. These methods have become popular because they simplified the classic issues mentioned above, such as trackers management, interaction, initialization, and update. The data association problem is formulated via a graph whose nodes represent the detections/features and whose edges are weighted by the distance (or similarity) between detections. Association methods usually collect all detections/features over the video, the current position of a target (a node of the graph) being thus determined by adjacent nodes that represent past and future detections. The goal of data association methods is to optimize the cost made by the edges of the graph. There are various methods using global and flow network optimization algorithms [START_REF] Zhang | Global data association for multi-object tracking using network flows[END_REF], and relying on criteria such as Graph Clique [START_REF] Zamir | Gmcp-tracker: Global multi-object tracking using generalized minimum clique graphs[END_REF][START_REF] Tesfaye | Multi-target tracking in multiple non-overlapping cameras using constrained dominant sets[END_REF], Graph Multicut [START_REF] Keuper | A multi-cut formulation for joint segmentation and tracking of multiple objects[END_REF][START_REF] Ristani | Tracking multiple people online and in real time[END_REF], Network Flow [START_REF] Zhang | Global data association for multi-object tracking using network flows[END_REF][START_REF] Pirsiavash | Globally-optimal greedy algorithms for tracking a variable number of objects[END_REF][START_REF] Berclaz | Multiple object tracking using kshortest paths optimization[END_REF][START_REF] Chari | On pairwise costs for network flow multi-object tracking[END_REF], Maximum Weight Independent Set [START_REF] Brendel | Multiobject tracking as maximum weight independent set[END_REF][START_REF] Kim | Multiple hypothesis tracking revisited[END_REF][START_REF] Choi | Near-online multi-target tracking with aggregated local flow descriptor[END_REF] Online approaches To fulfill the need for immediate tracking results in many applications, numerous papers proposed online tracking methods [START_REF] Andriluka | People-tracking-by-detection and peopledetection-by-tracking[END_REF][START_REF] Pirsiavash | Globally-optimal greedy algorithms for tracking a variable number of objects[END_REF][START_REF] Sadeghian | Tracking the untrackable: Learning to track multiple cues with long-term dependencies[END_REF][START_REF] Fagot-Bouquet | Improving multi-frame data association with sparse representations for robust near-online multi-object tracking[END_REF][START_REF] Zhu | Online multi-object tracking with dual matching attention networks[END_REF]. Within the tracking-by-detection paradigm, only detections in the current and previous frames are used to form targets' trajectories. One of the most popular approaches to associate detections is the bipartite matching formulation [START_REF] Sadeghian | Tracking the untrackable: Learning to track multiple cues with long-term dependencies[END_REF][START_REF] Reilly | Detection and tracking of large number of targets in wide area surveillance[END_REF][START_REF] Zhu | Online multi-object tracking with dual matching attention networks[END_REF], usually solved by using the Hungarian algorithm or heuristic approaches. Some offline methods, which can perform online when their optimization process only uses detections from the first to current frames of videos (i.e., causal system), or several methods can be considered as "near-online" methods such as [START_REF] Choi | Near-online multi-target tracking with aggregated local flow descriptor[END_REF][START_REF] Ristani | Tracking multiple people online and in real time[END_REF], as their offline optimization process applies on a window of frames at the time in videos, which causes the delay on tracking results. Alternatively, the tracking-by-detection strategy and multiple SOT algorithms are combined to benefit from the SOT trackers, and the ability to recover lost targets of data association approaches in [START_REF] Xiang | Learning to track: Online multi-object tracking by decision making[END_REF][START_REF] Zhu | Online multi-object tracking with dual matching attention networks[END_REF], classified as SOT-based approaches.

Multi-view multi-object tracking

MOT approaches based on a single camera have recently been extended to multiple cameras. These approaches have been proposed in an attempt to cover the observation of the objects fully. Multiple-camera tracking can solve the problem of occlusion, where the interesting targets are frequently occluded by the environment or by other targets. First attempts in using multiple (non-overlapping) cameras dealt with the re-identification problem, in order to track objects between cameras [START_REF] Wang | Intelligent multi-camera video surveillance: A review[END_REF]. Following this approach, many researchers studied the problem of collaboratively using overlapping cameras for tracking. Almost all authors made the hypothesis that the exact position of each camera is already known, and camera calibration has been done before applying the tracking process. In the tracking phase, the trackers implemented on different cameras usually pool their results with 3-D coordination via projection from the image plane to ground plane in the real world [START_REF] Mikic | Video processing and integration from multiple cameras[END_REF][START_REF] Sankaranarayanan | Object detection, tracking and recognition for multiple smart cameras[END_REF][START_REF] Pflugfelder | Localization and trajectory reconstruction in surveillance cameras with nonoverlapping views[END_REF]. This allows combining the different results, and in particular, reconnecting detections/tracklets to missing targets. Meanwhile, K-shortest path (KSP) [START_REF] Fleuret | Multicamera people tracking with a probabilistic occupancy map[END_REF] only uses detections from all cameras to first detect targets' positions on the ground via a POM (Probabilistic Occupancy Map), then perform tracking later. Besides of the above generic multi-camera tracking approaches, the methods based on the tracking-by-detection arises as an alternative. These methods inherit from most of the global optimization methods of MOT in single view such as graph multicuts [START_REF] Tang | Subgraph decomposition for multitarget tracking[END_REF][START_REF] Ristani | Tracking multiple people online and in real time[END_REF][START_REF] Keuper | A multi-cut formulation for joint segmentation and tracking of multiple objects[END_REF], graph cliques [START_REF] Zamir | Gmcp-tracker: Global multi-object tracking using generalized minimum clique graphs[END_REF][START_REF] Dehghan | Gmmcp tracker: Globally optimal generalized maximum multi clique problem for multiple object tracking[END_REF], network flow [START_REF] Zhang | Global data association for multi-object tracking using network flows[END_REF][START_REF] Pirsiavash | Globally-optimal greedy algorithms for tracking a variable number of objects[END_REF][START_REF] Chari | On pairwise costs for network flow multi-object tracking[END_REF]. Meanwhile, the other data association methods including bipartite matching [START_REF] Andriluka | People-tracking-by-detection and peopledetection-by-tracking[END_REF][START_REF] Sadeghian | Tracking the untrackable: Learning to track multiple cues with long-term dependencies[END_REF] and independent set [START_REF] Brendel | Multiobject tracking as maximum weight independent set[END_REF][START_REF] Kim | Multiple hypothesis tracking revisited[END_REF] do not address multi-camera tracking problem, because the tracklets are formed through the detections in consecutive frames (i.e., a short time window) of a single view, whereas tracking with multiple cameras is to connect trajectories of targets at different times. Some other approaches [START_REF] Tesfaye | Multi-target tracking in multiple non-overlapping cameras using constrained dominant sets[END_REF][START_REF] Khan | Consistent labeling of tracked objects in multiple cameras with overlapping fields of view[END_REF] generalize multi-camera tracking into two main steps: MOT on every single view, then linking the trajectories across cameras. Unfortunately, none of those mentioned methods perform online. Recently, Le et al. [START_REF] Le | Online multiple view tracking: Targets association across cameras[END_REF] introduced an online multi-camera tracking based on data association on each processing frame. In the next section, we introduce our dual-camera tracking approach in a multi-camera setting based on unbalanced optimal transport to handle hard occlusions and prevents identity switches. Our strategy is to assign targets from one to another view with the help of Deep Neural Nets. In literature, there are several approaches that have the same initiative to combine deep neural nets and optimization methods on which gradients are backpropagated such as [START_REF] Xu | How to train your deep multi-object tracker[END_REF][START_REF] Brachmann | Neural-guided ransac: Learning where to sample model hypotheses[END_REF]. However, to the best of our knowledge, our paper is the first one applying this strategy in MOT with the multiple overlapping cameras.

3 Proposed Method

Targets Association Across Cameras as an Unbalanced Optimal

Transport Problem This section describes in detail our approach to solving target association across cameras via optimal transport. Within a frame-synchronized, overlapping camera network, associating targets between different cameras emerges as the main issue for collaborative tracking.

Let us start by considering the case of a network consisting of two cameras C 1 and C 2 . At a given frame index F , we have {v 1 1,F , . . . , v 1 n,F } targets detected in Camera C 1 and {v 2 1,F , . . . , v 2 m,F } targets detected in Camera C 2 . In order to simplify our notations, we drop the F subscript in sequel. In general, n = m, since some targets can be seen by only one camera. This can happen either because a given target occupies a position that does not belong the common field of view or, more crucially, because of occlusion.

Each detection v k i , k ∈ {1, 2} is characterized by an feature vector generally consisting of an appearance vector, extracted from the bounding box provided by the detector, and the target's position. The current practice leverages the capability of recent deep convolutional neural networks to extract useful appearance features from the bounding boxes provided by the detectors, e.g. by using VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] or ResNet [START_REF] He | Deep residual learning for image recognition[END_REF] as a 'backbone'. For the target's position, it is necessary to define a common coordinate system, where the position of targets can be converted into the same measure unit. For pedestrian tracking, this issue is usually resolved by projecting the target's feet point on the image into the ground plane via the homography matrix of the camera. This is the solution we retain in our current setting.

These feature vectors allow to define a cost matrix C ∈ R n×m , whose entry C i,j defines the cost of associating target v 1 i to target v 2 j . The matrix C allows, in turn, to formulate the problem of target association between Cameras C 1 and C 2 , at frame index F , as a, possibly unbalanced, assignment problem. These problems amount to solving integer linear programs, using either combinatorial algorithms such as the Hungarian or the auction algorithm, or, ignoring the integer constraints, continuous linear programming.

For associating targets across different cameras, the definition of an appropriate cost matrix poses two serious problems. The first one is related to the definition of appearance features. These features should incorporate some kind of invariance with respect to the different cameras, that is, the appearance feature of the same target computed through two different cameras should be close. This invariance is not necessarily enforced when using popular convolutional networks such as VGG or ResNet. The second issue is related to the combination of the appearance features and the position, the appearance features being generally in the range [0, 1], while the position extending to the whole field of view.

In order to solve the issues raised by the two previous problems, we propose to adopt a learning-based approach, where the appearance features and their combination with the target's position are learned from a set of examples. The training data in this case are generated from the training sequences of the datasets we consider. More precisely, we extract from each training video frame the provided bounding boxes and the corresponding ground-truth assignments. With this training set at hand, we aim at end-to-end gradient-based learning, that is, the empirical loss that we minimize for learning should be related to the assignment task we consider, and implemented by (automatically computed) gradient descent.

Using combinatorial algorithms such as the Hungarian or auction algorithms rules out the possibility of using automatic differentiation engines for performing gradient descent. Furthermore, even when ignoring integer constraints, linear programming solvers can hardly be differentiated, since their solutions are not unique. To deal with this problem, we follow a recent line of works [START_REF] Cuturi | Differentiable ranks and sorting using optimal transport[END_REF] by considering the natural relaxation of the assignment, namely the optimal transport problem and its entropic regularization [START_REF] Peyré | Computational optimal transport[END_REF].

In our formulation of assignment problem of targets in two views via Optimal Transport, the sets of targets v 1 i n in one view is being matched with those v 2 i m in other view. Those two sets represents two empirical distributions: one Source and one Target, supported on a feature space X .

α := n i=1 a i δ xi , β := m j=1 b j δ yj , (1)
where δ x is the Dirac at x ∈ X and a i and b j are the corresponding weights.

In our setting, we will consider uniform discrete measures, which is, all the components of a weight vector are equal.

The optimal transport between source ans target is represented by an optimal transport plan P, which minimizes the following transportation cost:

L C (a, b) := min P∈U(a,b) C, P = min P∈U(a,b) i,j C ij P ij , (2)
where C is the ground cost matrix, whose elements C ij are the pairwise distance between the Dirac δ xi of the source measure α and those δ yj of the target measure β, and U(a, b) is a coupling from the source a to the target b. The feasible couplings are defined by a set of coupling matrices {P ∈ R n×m + }, where P ij depicts the amount of mass flowing from x i toward y j , under the mass preservation constraint.

U(a, b) = {P ∈ R n×m + : P1 m = a and P T 1 n = b}. (3)
The Optimal Transport (OT) problem with entropic regularization has a dual form following [START_REF] Feydy | Interpolating between optimal transport and mmd using sinkhorn divergences[END_REF]:

min P>0 max (f,g)∈R n ×R m C, P -εH(P) + a -P1 m , f + b -P T 1 n , g (4)
where the set of admissible dual variables (called potentials) (f ∈ R n , g ∈ R m ). The optimal transport plan solved via the dual problem (4) has a closedform [START_REF] Feydy | Interpolating between optimal transport and mmd using sinkhorn divergences[END_REF]:

P = π = exp 1 ε (f ⊕ g -C) . (α ⊗ β) , (5)
where f ⊕ g is denoted as a sum matrix of 2 vectors f and g whose cell {f ⊕ g} ij is equal to f i + g j , in the same manner, α ⊗ β is also denoted as a product matrix of 2 vectors α and β whose cell {α ⊗ β} ij is equal to α i β j .

Ground cost learning for UOT-based targets association across cameras

This section describes our proposed deep distance learning framework, which helps to compute an appropriate distance for the Optimal Transport problem between targets of one camera and those of another. More concretely, on each camera, the appearance feature of each target is extracted from its image patch via a deep convolutional neural network, e.g. VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], ResNet [START_REF] He | Deep residual learning for image recognition[END_REF]. Meanwhile, its position x is determined by projecting the target's feet point on the image into the ground plane via the homography matrix of the camera. Both the appearance and location feature vectors are the input of a deep neural network whose output is an embedding in a feature space X . The collection of all points mapped from all targets of a camera via the deep neural net generates a distribution. As discussed in the previous section, given two distributions originated from the targets of a pair of cameras, target association in a pair of two cameras is an Unbalanced Optimal Transport from a distribution, called source, to the other one, called target. Therefore, the Optimal Transport plan is followed by a thresholding step, to obtain a binary matrix as the association matrix of targets between the two cameras. Fig. 2 displays the pipeline of our distance learning framework, which aims to learn ground cost between targets across cameras so that the optimal transport plan approximates the ground-truth assignment.

Because our deep-learning-based method is a supervised learning approach, it is required a training data with labels. Our training data are directly generated from the training sequences of a dataset. Precisely, for each frame of videos, every pair of cameras gives a single assignment as the label of a sample, while the data of the sample is extracted from the ground-truth bounding boxes via the deep extracting feature net. In the case of N cameras in the network, the combination of camera possible pairs is N (N -1)/2, which is also the number of samples generated in each frame instant.

In our formulation, for each target i, given Φ i ∈ R 2048 (i.e., output of ResNet50 backbone [START_REF] He | Deep residual learning for image recognition[END_REF]) and x i ∈ R 2 (i.e., target coordinate on ground), the embedding function f w , via our deep neural network (see Fig. 3), projects the appearance feature and location of target i into the feature space X ,

f w : (Φ, x) → X ,
where w is the parameters of the deep neural net. As a result of unbalanced optimal transport, the transport plan shows the mass flows from point i of source to point j of target. Based on the properties of optimal transport [START_REF] Peyré | Computational optimal transport[END_REF], any pair of close points distributions source and target results in a significant mass flow on its transport plan compared with others. Fig. 4 (a) is an optimal transport plan in which i th row represents the mass of the i th source point being transferred to all target. Since only consistent mass transfers from one point on source to a unique point in target is sought, the optimal transport plan between source and target is expected to be well "sparse", which means that the matching can be deduced straightforwardly (see Fig. 4 (b)) by thresholding the optimal transport plan. We then can obtain the assignment from source to target.

In terms of optimization, the dissimilarity between the optimal transport plan P ε (α, β) and the ground-truth assignment G(α, β) is measured by a loss function L. The learnable parameters w of our neural net is then determined via a minimization problem: The loss functions in our framework are formulated as following. Given two sets of targets v k1 i n and v k2 j m

w = arg min w L (P ε (.; w) , G(.)) (6)
, each belongs to a single camera, the assignment task is to find the correspondence of common targets in the pair of cameras k 1 and k 2 while excluding the targets which can be seen in only one view. Given P ε ∈ R n×m + the transport plan and G ∈ {0, 1} n×m the ground-truth assignment, we propose our loss which is delivered from the dual problem of regularized optimal transport problem (4). Therefore, the first order condition to reach the optimal solution [START_REF] Peyré | Computational optimal transport[END_REF] yields to:

log(P ij ) = f i + g j -C ij ε (7)
In the training phase of our experiments, by default, both total masses of the measure α and β are equal 1. The constraint of mass conservation in the Bal-anced Optimal Transport problem leads to the sum of all elements of the optimal transport plan P smaller or equal 1. The equality happens in the case of the balanced optimal transport, and the inequality for the Unbalanced Optimal Transport case. As a result, the ground-truth assignment G needs to be normalized to keep the assignment matrix and optimal transport plan comparable. Hence, from the expression of the transport plan [START_REF] Chari | On pairwise costs for network flow multi-object tracking[END_REF], our loss is formulated as follows:

L (P ε , G) = i,j log(G ij ) -max f i + g j -C ij ε , log σ (8)
where the normalized assignment coupling G i,j is defined as

G ij = G ij i,j G ij + γ + σ ≥ σ ( 9
)
with σ is a tiny threshold value, and γ is a normalization constant. This threshold value is added to avoid the logarithm of zero value in the loss function ( 8) and to set a margin for any near-zero transport, which does not contribute to the distance loss if its value is extremely low. Additionally, the parameters of our neural net w are updated iteratively via minimizing the loss L. The derivation of the loss function to the net parameters ∂L/∂w is computed via back-propagation, which occurs after every optimal transport of a source-target pair from two cameras.

Experimental Results

Implementations

In our implementations, we build two versions of distance learning neural nets in order to compute the source-target distance in the Optimal Transport:

(a) The appearance feature of targets obtained from the backbone of ResNet50, in addition to their location, is considered as the inputs of our distance learning network. Our deep network is a series of Full Connected Layers with ReLU layer on the top of each. The outputs of FC layers are respectively 1024, 512, 256 and 128 (see Fig. 3 (a)). (b) The second model is modified from the original one, but instead of using locations in the first layer, it is concatenated with the second last output layer. The intuition behind is to emphasize the location feature of targets, because, in tracking applications, positions of targets are crucial to the performance of tracking algorithm (see Fig. 3 (b)).

In the deployment phase, numerous experiments with different configurations are conducted within the framework of multiple camera tracking of the paper [START_REF] Le | Online multiple view tracking: Targets association across cameras[END_REF]. Therefore, the target assignment or matching is applied on only two cameras, collaborative tracking in our multiple camera approach occurs on pairs of cameras, but one camera can reach all others through the whole tracking process. Precisely, at each frame, each camera consecutively pairs with all other cameras, then within each pair of cameras, an optimal transport plan C is computed in order to link targets from one camera to the other in the pair. As mentioned in the section 3.1, the value of each cell C ij of the optimal transport plan implies how likely element i of source set is matched to element j of target set based on the amount of mass being transferred, named OT value. Therefore, each missing target on one view is associated with its corresponding target on each other view by an OT value obtained from its optimal transport plan. Hence, the tracking result of the missing target is replaced by the "tracked" target with the highest value among its correspondences on all other cameras.

The Optimal Transport algorithm we used in this paper is a public Optimal Transport library [START_REF] Feydy | Interpolating between optimal transport and mmd using sinkhorn divergences[END_REF] on Python with GPU parallelization support, named KeOps-GeomLoss3 . The parameters of the unbalanced optimal transport problem (4) are set as follows:p = 2; "blur" = 0.5 → ε = blur p ; "reach" = 0.1 →

τ 1 = τ 2 = τ = reach p ; D ϕ = KL : soft Kulback-Leibler divergence.
Meanwhile, the other parameters in the expression ( 9) are adjusted for σ = 10 -8 and γ = 10 -4 . The value of threshold to convert transport plan to assignment matrix is set equal to 10 -3 , which is greater than σ in order to reduce the sensibility during testing phrase. The detailed implementation of our deep distance learning method will be available publicly on our project page.

Benchmarking Performance

This section shows our experimental results verifying the efficiency of the multicamera MOT algorithm with various appearance features. As a performance evaluation for MOT algorithms, the benchmark MotChallenge [START_REF] Leal-Taixé | MOTChallenge 2015: Towards a benchmark for multi-target tracking[END_REF] has been released with two datasets (MOT15 and MOT16), which contain many singleview video sequences recorded by static or dynamic cameras, and the evaluation metrics of CLEAR MOT [START_REF] Bernardin | Evaluating multiple object tracking performance: the clear mot metrics[END_REF] and ID measure [START_REF] Ristani | Performance measures and a data set for multi-target, multi-camera tracking[END_REF] are used. Additionally, the MotChallenge also provides multiple video sequences, but most of them are not from multiple camera aspect, which requires overlapping zones, synchronization, calibration. Therefore, these datasets, unfortunately, unfit to this case study that focuses on using multiple overlapping views to tackle the targets missing by occlusions. As the multi-camera method aims to improve identity robustness in single views, we will emphasize ID scores in the sequel.

Datasets. In our experiments we used the well-known PETS2009 [START_REF] Ferryman | Pets2009: Dataset and challenge[END_REF] and EPFL Multi-camera Pedestrian Videos [START_REF] Berclaz | Multiple object tracking using kshortest paths optimization[END_REF] datasets. Among all sequences of PETS2009, the most relevant and suitable for our multiple-camera tracking system is "PETS09-S2L1" with 7 views from 7 synchronized and calibrated cameras. For our experiment, only one main view (from the camera 1) and 4 close-up views (from the cameras 5, 6, 7, and 8) are used. Besides of the sequence "PETS09-S2L1" with 7 cameras, the sequence "PETS09-S2L2" is also available with only 3 cameras. The scenario of surveillance is to track an influx of people moving on the roads with different speed, and this makes it far more crowded that "PETS09-S2L1". Since the lack of cameras in this sequence, we set up dualcamera tracking experiments on View 1 and View 2. View 3 is excluded, due to its small impact on the sequence and the absence of ground-truth data as well. On the other hand, the EPFL dataset provides multiple indoor and outdoor video sequences, recording pedestrians by 4 different cameras. Due to the similarity between sequence scenarios, only the sequence "Terrace1" is selected for the experiments. In terms of camera topology, only about 15 -20% of the observable zones are covered by all cameras in our tracking sequences.

Detection. In all tracking-by-detection approaches, the detector plays an important role in tracking performance. Detections in video frames are generated by the public high-accuracy detectors such as OpenPose [START_REF] Cao | Realtime multi-person 2d pose estimation using part affinity fields[END_REF] and R-CNN [START_REF] He | Mask r-cnn[END_REF].

Evaluation metric. To validate the efficiency of our various settings on the multi-camera MOT approach, we adopt the CLEAR MOT metrics and ID measures and in particular the following scores: MOTA (multiple-object tracking accuracy), MOTP (multiple-object tracking precision), IDs (identity switches), IDF1 (ID F1-score), IDP (ID precision), IDR (ID Recall), False Positive (FP) and False Negative (FN). For further details on the metric, we recommend the MOTChallenge website 1 . In comparison between MOT scores and ID-measures, all multiple camera approaches slightly improves both MOTA and MOTP scores, regarding to ID-measures. Because the CLEAR MOT metric does not focus on re-identification ability of tracking algorithms [START_REF] Ristani | Performance measures and a data set for multi-target, multi-camera tracking[END_REF], while ID-measure scores does. In other words, the significant improvement can be seen on IDF1 and IDP score. As another important indicator for tracking performance in CLEAR MOT metric, IDs score (i.e. identity switches) relates more to ID preservation, which is essential in multiple camera tracking. Therefore, in the following analysis, we measure the impact of methods based on IDF1, IDP and IDs scores rather than MOTA and MOTP.

Performance analysis

The results shown in the following tables are the average values of all views. Concretely, the overall tracking results of the PETS sequence can be seen in the Table 1, Table 2 andTable 3. Each score column has either a ↑ or a ↓ indicating whether better corresponds to higher or lower, respectively. The red color indicates the best score and the blue for the second best.

Primarily, our multi-camera tracking method aims to address hard occlusion problems. It leads to an important reduction of identity switches and a significant improvement of ID measures in comparison with the single-camera method. In the sequence "PETS09-S2L1", the targets have their complex movements and mutual interactions inside the overlapped area of the tracking scene. All the methods using the target trajectory as the features of the affinity measure show the better scores in all categories, in comparison with the approaches, which do not consider historical position record of targets (i.e., trajectories), but only the instant measure including image patch and position of targets. In detail, the method with full camera collaboration (All-cam) [START_REF] Le | Online multiple view tracking: Targets association across cameras[END_REF] shows off its superiority. Meanwhile, our Unbalanced Optimal Transport approach (UOT) is less robust, but still significantly improves tracking scores compared to single-camera approach. Notwithstanding, in the tests with the sequence "EPFL/terrace1", the tracking scene composes 8 identities moving mainly around a relatively small area covered by a smaller camera number, which makes the scene more crowded and targets hardly seen by all cameras. Consequently, the original approach [START_REF] Le | Online multiple view tracking: Targets association across cameras[END_REF] failed to improve tracking results, because, with a smaller camera amount, it is obviously less probable that there are more than 2 or 3 cameras observed the same target at the same time. The results in Tab. 2 show that all other approaches with dual-camera mode perform significantly better than the original ones. The next remark is that in the scenario where there are only short trajectories that can be seen, the trajectory feature is less reliable. In other words, shorter trajectories, less effective the original approach is. Hence, in Tab. 2, our distance learning method based on Optimal Transport outweighs the conventional approaches which only use position or trajectory of target as input feature for affinity measure. Secondly, the KSP method performs poorly on the sequence PETS09-S2L1, but gives a greater score on EPFL/terrace1. We can explain that KSP method was developed on the EPFL Multiple View Pedestrian Dataset. In fact, they assume that the targets being observed by cameras system does not leave the scene during their presence. In other words, the targets have to finish their complete trajectories before leaving the scene. The out/in positions of targets is also fixed on the scene, so we can see the actors walking in and out at the Q. Le et al. same place. Under these conditions, they came up with the K-shortest path problem where K, which is the number of targets in the tracking videos. On EPFL/terrace1, the algorithm has found 8 paths, which exactly corresponds to 8 targets in the video that help them get the highest IDF1 score. Back to the sequence PETS09, the algorithm cannot deal with the targets that usually went out of and returned to the scene. It only found the longest paths and ignored the targets which appeared in a short period of time and regularly got confused by other targets at the boundary. Moreover, in this database, there is no constrain on where people will appear and disappear on the scene. Apparently, this leads to a negative score on MOTA. It indicates that the KSP algorithm cannot handle the enter/exit of targets. Another problem with KSP is that the tracking process occurs on a grid, called Probabilistic Occupancy Map (POM), the discrete unit size directly affects the accuracy of the tracker. Unfortunately, increasing the size of POM required more iterations to make sure the occupancy map converged correctly. Finally, on the tests with dual-camera sequence "PETS09-S2L2", we excluded the methods which require more than 2 cameras to be operational, including all camera [START_REF] Le | Online multiple view tracking: Targets association across cameras[END_REF] and KSP [START_REF] Berclaz | Multiple object tracking using kshortest paths optimization[END_REF]. As single object trackers can generate long trajectories for targets, trajectories are still an important feature to distinguish targets that we can see in Table 3. The approach [START_REF] Le | Online multiple view tracking: Targets association across cameras[END_REF] with dual-camera only using trajectory as target features archived the second-best result on ID-measures and the best on MOT-scores. Meanwhile, our UOT dual-cam approach based on position only obtained the best scores on ID-measures and the second-best on MOTscores. Unfortunately, two of our UOT methods using distance learning could not outperform others in this sequence.

Method

Conclusion

In this paper, we proposed a novel unbalanced assignment method based on optimal transport to address the target assignment problem between two cameras in an online multi-camera tracking application. A deep metric learning method is introduced with an efficient metric loss function. Our experiments showed the effectiveness of our approach to the multiple camera tracking systems.

Fig. 1 .

 1 Fig. 1. (a) Assignment between two distributions as Optimal Transport problem (b) Target assignment across cameras in multiple cameras tracking application

Fig. 2 .

 2 Fig. 2. The pipeline of our distance learning framework. The red arrow indicates the direction during training process, meanwhile the blue lines for testing.

Fig. 3 .Fig. 4 .

 34 Fig. 3. Proposed distance learning neural net. The neural net consists of a CNN backbone (e.g. ResNet50 in our case), which extract appearance features from raw image, and a series of Fully Connected (FC) layers with ReLU layers as activations. Model (a) with locations at the bottom of the deep distance network, meanwhile, model (b) with locations at the second last FC layer.

Table 1 .

 1 Scores on "PETS09-S2L1" multi-camera sequence.

	Method + Feature	IDF1↑ IDP ↑ IDs↓ MOTA↑ MOTP↑
	Single cam [38]+∅	57.49 62.24 333	68.44	68.83
	All cam [21]+path	72.8 78.53 98	73.26	70.69
	KSP [15]+∅	21.51 18.16 812 -29.63	64.27
	Dual-cam [21]+path	67.96 72.72 126	73.4	70.65
	Dual-cam UOT+DL (a) 68.15 73.41 153	73.16	70.81
	Dual-cam UOT+DL (b) 66.71 71.73 174	72.19	70.65
	Dual-cam UOT+pos	66.04 70.66 163	72.76	70.60

Table 2 .

 2 Scores on "terrace1" multi-camera sequence.

	Method + Feature	IDF1↑ IDP ↑ IDs↓ MOTA↑ MOTP↑
	Single cam [38]+∅	21.88 25.66 388	55.98	72.53
	All cam [21]+path	21.32 25.05 461	54.14	72.47
	KSP [15]+∅	25.85 23.51 695	19.57	62.26
	Dual-cam [21]+path	23.23 26.72 382	56.71	72.43
	Dual-cam UOT+DL (a) 24.36 31.40 305	46.86	72.91
	Dual-cam UOT+DL (b) 25.15 28.88 385	56.93	72.63
	Dual-cam UOT+pos	22.00 25.32 381	56.40	72.48

Table 3 .

 3 Scores on "PETS09-S2L2" dual-camera sequence.

	+ Feature	IDF1↑ IDP ↑ IDs↓ MOTA↑ MOTP↑
	Single cam [38]+∅	53.46 55.53 321	63.66	75.14
	Dual-cam [21]+path	55.63 57.67 327	63.79	75.16
	Dual-cam UOT+DL (a) 53.16 55.76 310	62.62	75.16
	Dual-cam UOT+DL (b) 53.75 55.83 329	63.51	75.16
	Dual-cam UOT+pos	57.30 59.38 312	63.74	74.98

https://www.kernel-operations.io/geomloss/