Salwa Baabou
email: baabousalwa@gmail.com

Furqan M Khan

Awatef François Bremond

Ben Frad

Mohamed Amine Farah

Abdennaceur Kachouri
email: abdennaceur.kachouri@enis.rnu.tn

Francois Bremond
email: furqan.khan|francois.bremond@inria.fr

Awatef Ben Fradj
email: benfradjawatef@yahoo.frmed.farah@yahoo.fr

Tracklet and Signature Representation for Multi-Shot Person Re-Identification

come

Abstract-Video surveillance has become more and more important in many domains for their security and safety. Person Re-Identification (Re-ID) is one of the most interesting subjects in this area. The Re-ID system is divided into two main stages: i) extracting feature representations to construct a person's appearance signature and ii) establishing the correspondence/matching by learning similarity metrics or ranking functions. However, appearance based person Re-ID is a challenging task due to similarity of human's appearance and visual ambiguities across different cameras. This paper provides a representation of the appearance descriptors, called signatures, for multi-shot Re-ID. First, we will present the tracklets, i.e trajectories of persons. Then, we compute the signature and represent it based on the approach of Part Appearance Mixture (PAM). An evaluation of the quality of this signature representation is also described in order to essentially solve the problems of high variance in a person's appearance, occlusions, illumination changes and person's orientation/pose. To deal with variance in a person's appearance, we represent it as a set of multi-modal feature distributions modeled by Gaussian Mixture Model (GMM). Experiments and results on two public datasets and on our own dataset show good performance.

Index Terms-Person Re-Identification (Re-ID), Part Appearance Mixture (PAM), tracklet, signature representation I. INTRODUCTION Person Re-Identification (Re-ID) aims to match individuals appearing across non-overlapping camera networks at distinct times and locations. A person's appearance across different cameras is very variable, which makes the recognition of individuals more and more challenging.

A typical Re-ID system may have an image (single shot) or a video (multi-shot) as input for feature extraction and signature generation. Thus, the first step in Re-ID is to learn a person's visual signature or model and then compare the two models to get either a match or a non-match. Extracting a reliable signature depends on the availability of good observations. Besides, faulty trajectory estimation and incorrect detections introduce errors in signature generation and extraction that affect the Re-ID quality. The most obvious and simplest signature of a person is characterized by low-level features like color, texture and shape. However, these features are hardly unique, not descriptive enough and prone to variations. Color/texture varies due to cross view illumination variations, pose variations, view angle or scale changes in multi-camera settings.

A subject may be fully or partially occluded by other subjects or carrying items that lead to errors in matching between tracklets. Furthermore, some works in person Re-ID used body-parts methods (such as SDALF, MPMC) [START_REF] Pala | Multimodal person Re-Identification using RGB-D cameras[END_REF] to solve the issue of signature alignment but this problem is still difficult and not efficient as these methods require real detections and many annotations. All these issues may affect the performance of person Re-ID which is still not robust enough to guarantee high accuracy in practice.

To sum up, the contribution of this paper is: how to pre-process the tracklets to make them good for computing the signature and then represent it for multishot Re-ID based on PAM approach [START_REF] Khan | Multi-shot Person Re-Identification using part apperance mixture[END_REF]. This may cater the high variance in a person's appearance and discriminate between persons with similar appearances. A Mahalanobis based distance is defined to compute similarity between two signatures.

The paper is organized as follows: The Re-ID process which contains person detection and tracking is described in the following section. Section III is the core of the paper: it introduces the Part Appearance Mixture Approach PAM by presenting the signature representation and computing the similarity between these latter using metric learning algorithms. Finally, we evaluate the quality of our signature representation based on the realized experiments and results before concluding.

II. RE-ID PROCESS

The advances in computer vision, as well as machine learning techniques in the recent years, have ameliorated this expedition towards smart surveillance at a fast pace and as a result, a plethora of algorithms for the automatic analysis of the video sequences have been proposed. They include, for instance, person detection, person tracking, activity monitoring, and person Re-Identification. Some survey papers such as [START_REF] Paul | Human detection in surveillance videos and its applications-a review[END_REF][START_REF] Moeslund | A survey of computer visionbased human motion capture[END_REF][START_REF] Yilmaz | Object tracking: A survey[END_REF][START_REF] Vezzani | People reidentification in surveillance and forensics: A survey[END_REF][START_REF] Bedagkar-Gala | A survey of approaches and trends in person re-identification[END_REF][START_REF] Wang | Person Re-Identification: system design and evaluation overview[END_REF][START_REF] Jasher Nisa | A review on different Methods of person Re-Identification[END_REF] have presented them in detail.

Fig. 1 illustrates the Re-ID process, i.e the diagram of person Re-Identification system, containing the different steps that we will follow and explain it later. It starts with automatic person detection. In recent years, most of the existing person re-identification works have ignored this step and assume perfect pedestrian detection. However, perfect detection is impossible in real scenarios and misalignment can seriously affect the person Re-ID performance. Therefore, this factor should be carefully studied in future studies. In order to build a strong visual signature of people appearances, persons have to be accurately detected and tracked, so the step of person tracking should be also taken into consideration. However, person detection and multiple person tracking are difficult problems with their own hurdles. Significant amount of work has gone into the problem of person detection over the years as well as Multiple Object Tracking (MOT) [START_REF] Nguyen | Robust Global Tracker based on an Online Estimation of Tracklet Descriptor Reliability[END_REF][START_REF] Nguyen | Multi-Object Tracking of Pedestrian Driven by Context[END_REF] within a single camera's Field-Of-View (FOV) as well as multiple cameras which has also been widely researched but it remains an open problem. For feature extraction and descriptor generation, the most commonly used features are color, shape, position, texture, and softbiometry. The adopted feature is determined by different factors. On one side, the signature should be unique and discriminative enough which can lead to the selection of biometry or soft-biometry features. On the other side, camera resolution, computational load and other implementation issues can prevent or limit their usage and more generic features are required. It is worth noting, that the Re-ID system; in which our work is focusing; as appearing in the relevant literature, turns out to be divided as we said into two distinctive steps (see Fig. 1): i) extracting distinctive visual features to represent the human appearance and ii) establishing correspondence by learning or discovering an optimal metric that can maximize the distance between samples from different classes whilst minimizing the distance between those belonging to the same class.

All these steps will be described in details in the following section. Subject detection can be considered also as classification process; First intuitive idea is to deal with detection as classification of all possible bounding boxes in the image, and classify them as different subjects. To take this way, we need a sliding window with certain step to span the whole image. In addition, we need the windows to be with different sizes, and scales. At the end, we have the bounding box of the subject, and a score (confidence) of the classification. Theoretically, if we have a very fast classifier, this can work. But in reality, the sliding window is slow, we need too many windows to guarantee that all possible regions are tested. To solve the problem of sliding window, instead of looking at all possible positions, we can have a smarter system that can find some interesting regions, and tell the classifier where to look. Example of region proposals are selective search, and Edge-boxes.

There are many subject detectors that we can cite: R-CNN, fast/er R-CNN [START_REF] Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] which are two-stage detectors; first, they propose regions, then they apply classification and bounding box regression. The modern detectors deal with the whole detection process as bounding box regression, so they are much faster which are YOLO [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF] and SSD [START_REF] Liu | SSD: Single Shot multibox Detector[END_REF].

In this paper, we will use the SSD detector [START_REF] Liu | SSD: Single Shot multibox Detector[END_REF]. Fig. 2 shows a visualization of a sample from CHU Nice dataset of the detection results. In fact, the SSD detector differs from other single shot detectors due to the usage of multiple layers that provide a finer accuracy on subjects with different scales. (Each deeper layer will see bigger subjects). It starts with a VGG pretrained model. Then after the image is passed on the VGG network, some convolutional (conv) layers are added producing feature maps of sizes 19x19, 10x10, 5x5, 3x3, 1x1. These, together with the 38x38 feature map produced by VGG's conv 4-3, are the feature maps which will be used to predict bounding boxes.

B. Multi-Object Tracking

Insprired by Multi-object tracking approach in [START_REF] Nguyen | Multi-Object tracking using Multi-Channel Part Appearance Representation[END_REF], the tracking process is based on the method of a robust online multi-object tracking which combines a local and global tracker. In the local tracking step, we use the frame-to-frame association to generate the tracklets ( object trajectories; which are represented by a set of multi-modal feature distributions modeled by the GMMs. In the global tracking step, the tracklet bipartite association method is used based on learning Mahalanobis metric between GMM components using KISSME [START_REF] Kstinger | Large scale metric learning from equivalence constraints[END_REF] metric learning algorithm. The local tracker's objective is to find correct object trajectories in the past, while, the global tracker tries to find object associations between aggregated tracklets. In the first step, the tracklets are constructed by putting together frame-to-frame tracker's output. For a reliable tracklet, tracklet filtering is applied by splitting spatially disconnected or occluded tracks and filtering out noisy tracklets. In the second step, in every video segment ∆t, the global tracker carries out data association and performs on line tracklet matching. Association and matching process happens based on Mahalanobis metric among representations of tracklets stacked in two previous video segments (2∆t).

Fig. 3 presents a visualization of a sample of consecutive frames from CHU Nice dataset of the tracking results.

III. PART APPEARANCE MIXTURE (PAM) APPROACH

A. Signature Representation

We define a tracklet T r i between two frames m and n as a sequence of tracked subject's bounding-boxes as follows:

Tr i = {N m i , N m+1 i , ..., N n-1 i , N n i } (1)
Where N represents the subject bounding-box and i is the subject ID. Inspired by Part Appearance Mixture PAM approach in [START_REF] Khan | Multi-shot Person Re-Identification using part apperance mixture[END_REF], and to cater the variance in a person's appearance, we model it as a multi-modal probability distribution of descriptors, using GMM to represent this appearance. Thus, the tracklets representation are modeled as a multi-channel appearance mixture (appearance model). The representation divides body into three parts: full, upper and lower. Each channel in the mixture model corresponds to a particular body part.

Given a set of nodes (detection bounding-boxes) belonging to a tracklet T r i , its PAM signature representation Q is defined as a set of appearance models M p i :Q= {M p q |p∈ {f ull, upper, lower}}; one for each part p of person q. Each appearance model in the set is a multivariate GMM distribution of low-level features of part p. Appearance models help to overcome occlusion, pose variation and illumination problems.

To describe an object, we use appearance features that are locally computed on spatial grid of object detection bounding-boxes; the features are computed efficiently to be shared between the parts (upper and lower body regions are defined as 60% of boundingbox of the person) including: HOG [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF], LOMO [START_REF] Liao | Joint dimension reduction and metric learning for person re-identification[END_REF], HSCD [START_REF] Zeng | Efficient person re-identification by hybrid spatiogram and covariance descriptor[END_REF] and Color histogram features. While the framework exploits HOG feature as a shape based feature to overcome difficulties of pose variation, it benefits from other features to cope with illumination and appearance changes happening in long occlusions.

The similarity between two signatures is partially based on computing Mahalanobis distance between means of GMM components. People appearing in a video have different appearance and produce GMMs with variable number of components. Therefore, the number of components are not a priori determined and need to be retrieved. In order to infer the number of GMM components for each appearance model automatically, Akaike Information Criterion (AIC) model selection is used. Knowing fixed number of the components, the parameters of a GMM could be learned conveniently using Expectation-Maximization method.

B. Similarity metric for signature representation

As mentioned before, a signature is a set of part appearance mixtures. Similarity between two signatures G i and G j is defined as the sum of similarities between

Sim(G i , G j ) = p∈P exp(- d(M q p , M g p ) -γ p,g 1
3 (β p,g -γ p,g ) ) (2)
where P = {f ull, upper, lower}, d(M p p , M g p ) is max normalized distance between a query person q and a gallery person g of part p. β p,g and γ p,g are the maximum and minimum normalized distances, respectively, between person g in gallery and any other person q in query set. The factor 1 3 in formula makes Gaussian similarity kernel goes to zero for q that has maximum normalized distance from g. We define the distance between two GMMs as he distance between their components weighted by their prior probabilities:

d(M 1 , M 2 ) = i=1:K,j=1:K π 1i π 2j d(G 1i , G 2j ) (3)
where G nk is the component k of GMM M n∈{1,2} with corresponding prior π nk and d(G i , G j ) = i , G j ) which is defined in the following section.

C. Distance computation between signatures

To define similarity between two signatures, fdivergence based distances; in particular Jeffry's Divergence (JDiv) is used, and since we restrict covari-ance matrices to be diagonal, it can be computed as follows:

JDiv(G i , G j ) = 1 2 (µ i -µ j ) T ψ(µ i -µ j ) + 1 2 tr { -1 i j + -1 j i -2I}(4) where ψ = -1 i + -1 j
The distance between two GMMs is computed based on the Mahalanobis distance, squared Mahalanobis distance of a pair of vectors is defined as follows:

d 2 (x i , x j) = (x i -x j ) T M (x i -x j ) (5)
where M is a positive semi-definite matrix. The parameters of Matrix M are estimated using KISSME [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF]. i.e M = -1

+ - -1
-, where + and -are feature-difference covariance matrices of positive and negative classes, respectively. Given the mean of two Gaussian distributions, µ i and µ j , the positive and negative class covariance matrices are defined as:

+ = yij=+ (µ i -µ j )(µ i -µ j ) T (6)
- = yij=- (µ i -µ j )(µ i -µ j ) T (7)
where y ij ∈ {+, -} is the ground truth similarity label between pairs of Gaussian distributions (G i , G j ). Alternatively, matrix M can be estimated using XQDA [START_REF] Nguyen | Robust Global Tracker based on an Online Estimation of Tracklet Descriptor Reliability[END_REF] in similar spirit.

IV. EXPERIMENTS AND RESULTS

A. Datasets

We have evaluated our work on two challenging benchmark datasets: PRID2011 and iLids-VID and on our own dataset: CHU Nice dataset. These datasets were chosen because they provide multiple images per individual (i.e Multi-shot datasets) collected in realistic visual surveillance settings using two cameras.

B. Performance Evaluation

We use the Part Appearance Mixture approach with two different image descriptors: HOG and LOMO. The image descriptors are computed just from the full body, then we extract the upper and lower descriptors from the full body descriptors. III show the recognition rate (%) at different ranks (rank-1, 5, 10, 20) of a baseline method LOMO+XQDA [START_REF] Liao | Person re-identification by local maximal occurrence representation and metric learning[END_REF] and PAM-LOMO+XQDA on PRID2011, iLIDS-VID and CHU Nice datasets, respectively.

From the above experiments, we notice that PAM-LOMO+KISSME achieves good performance on three datasets; it achieves 92.5%, 79.5% and 81.8% rank-1 recognition rates on PRID2011, iLIDS-VID and CHU Nice datasets, respectively. This shows that our adaptation of feature descriptor LOMO and metric learning KISSME to PAM representation is effective.

C. Evaluation of signature representation quality

As shown in Fig 5, a visualization of a selected sample from CHU Nice dataset of PAM signature representation is presented. Indeed, the first image corresponds to one of the input images used to learn appearance model. Its followed by the composite images, one for each component of the GMM mixture. Optimal number of GMM components for each appearance model varies between persons. GMM components focus on different pose and orientation of the person. Moreover, We visualize each GMM component by constructing a composite image. In fact, given appearance descriptor, we compute the likelihood of an image belonging to a model component and then by summing images of corresponding person weighted by their likelihood we generate the composite image. We can say that our signature representation is able to cater variance in person's pose and orientation as well as illumination, it deals also with occlusions and is able to reduce effect of background. However, we can notice that this PAM signature present some limitations, specially on our own dataset CHU Nice, which can affect the quality of our signature representation (see Fig 5). Among these challenging problems, we can cite:

• Bad detection • Number of frames by pose • Number of GMM components not adequate with the number of person's pose/orientation and depends of the low-level features used.

V. CONCLUSION

Person Re-ID is a challenging task with three aspects: First, is is important to determine which parts should be segmented and compared. Second, there is a need to generate invariant signatures for comparing the corresponding parts. Third, an appropriate matching function (i.e similarity metric or a ranking function) must be applied to compare these signatures.

In most studies, the Re-ID process is designed under the assumption that the appearance of the person is unchanged which doesn't seem reasonable in practise. As future work, we are trying to improve the PAM signature representation, by using the skeleton and extracting the pose machines from our dataset, i.e CHU Nice dataset, which will be soon introduced as a new public dataset in the field of person Re-ID.

Fig. 1 :

 1 Fig. 1: Person Re-ID diagram

Fig. 2 :

 2 Fig. 2: A visualization of a sample from CHU Nice dataset of the Detection Results

Fig 4

 4 shows a representation of some samples of tracklets of a person from CHU Nice dataset.

Fig. 3 :

 3 Fig. 3: A visualization of a sample of frames from CHU Nice dataset of the Tracking Results

Fig. 5 :

 5 Fig. 5: A visualization of selected samples of signature representation from CHU Nice Dataset

TABLE I :

 I Top ranked Recognition rates (%) on PRID2011

	Methods	Rank-1 Rank-5 Rank-10 Rank-20
	LOMO+XQDA	-	-	-	-
	PAM-HOG+KISSME	55.3	80.7	90.2	95.6
	PAM-LOMO+XQDA	-	-	-	-
	PAM-LOMO+KISSME	92.5	99.3	100.0	100.0

TABLE I ,

 I TABLE II and TABLE

TABLE II :

 II Top ranked Recognition rates (%) on iLIDS-VID

	Methods	Rank-1 Rank-5 Rank-10 Rank-20
	LOMO+XQDA	53.0	78.5	86.0	93.4
	PAM-HOG+KISSME	33.9	60.0	70.2	79.1
	PAM-LOMO+XQDA	-	-	-	-
	PAM-LOMO+KISSME	79.5	95.1	97.6	99.1

TABLE III :

 III Top ranked Recognition rates (%) on CHU Nice dataset

	Methods	Rank-1 Rank-5 Rank-10 Rank-20
	LOMO+XQDA	30.7	64.6	80.3	90.3
	PAM-HOG+KISSME	-	-	-	-
	PAM-LOMO+XQDA	38.5	69.2	84.6	100.0
	PAM-LOMO+KISSME	81.8	90.9	100	100