Michael Filhol
email: michael.filhol@limsi.fr

Line Patris

Pierre Guitteny
email: pierre.guitteny@gmail.com

Text-sign parallel corpus study to start designing an automatic translation system

Keywords: Automatic translation, Sign Language, corpus annotation

 for representation of Sign Language rules, the ultimate goal is to build a set of translation mechanisms from annotated text to AZee operations, usable as input to a virtual signer animation system. Prospects are given to enrole theoretical frameworks capable of describing rhetorical/discourse structure representation.

Introduction and context

Automatic translation has been a challenge for Natural Language Processing ever since the field existed, and it is known to be a most challenging scientific issue. Public translation systems like one can find online usually involve a choice in the source and target languages, but only written languages are made available. We now want to consider Sign Language (SL), in particular French Sign Language (LSF), as the target language. An example application would be for a SL user reading a textual webpage to select part of the text and run an applet program that would display an animated virtual signer producing the Sign Language for the selected text.

To translate from a language to another through an automatic process, some form of model is needed for each of the languages. Then an extra module is needed to bridge between the two. The ViSiCAST project about a decade ago did work on such a pipeline from text to sign [18]. It used HPSG, a unification grammar based on syntactic relations in text. Sign Language being far from the linearity of a textual input, we propose to use an approach to it and translation to it that fit its specificities.

This paper presents our start on a new translation project, dedicated to Sign Language as target. We want the system built bottom-up from data to enforce fully-acceptable output with no text-to-text bias. The next section presents two SL-specific properties, which will guide us on our way to it. Then, we introduce the corpus we based our preliminary work on and from which we will derive our first ideas and conclusions. After commenting on the choice for symbolic output of the translation module, the paper describes our methodology for the study of the corpus. We conclude with short-and long-term objectives for the future of the project.

Sign Language specificities

The task we address here is that of a translation system between languages using different channels: the source language is linear text; the target is gestural thus multi-linear. This leads to many differences of properties between the two, and we present two of them here.

Use of signing space

In sign linguistics, it is universally observed that sign languages make a relevant and consistent use of space, assigning locations of the signing space to parts of the signed productions. Lexical units (signs) can be relocated in space for particular reasons, and full clauses can also be signed leaning towards one side or the other.

In terms of implementation, the "lexical agreement" problem has been addressed more than once [18, 20], using syntactic relations and unifications systems. All of these are based on the assumption that a sign production is a sequence of lexical signs in the first place, whose order is syntactically constrained. Therefore, relocation is possible but limited to the cases where the reason for it is syntactic, e.g. to direct a verb.

The point in this paper is to look at spatialisation in the general sense, including the majority of cases where it is not analoguous to lexical agreement.

Discourse-grain observations or you lose

Another property of Sign Language-which is arguably a more or less direct consequence of 2.1-is its unique preferred order for the clauses of a discourse. In Sign Language, it is more acceptable [17]:

• to sign any element of context first (location, time...) and sign anything that takes place in that context afterwards;

• to sign the fixed and immutable things first in a setup (ground, houses...) and sign any animated object moving in it afterwards;

• to sign anything that identifies a topic or a target first, and sign the "hot news" or point at the target last.

For example, "Bob is playing with his sister in the garden" will typically be translated with the garden first (place) and the verb last (action and point of the sentence). This is unless the actual information is that the children are in the garden, as when phrased "It is in the garden that Bob and his sister are playing"; then the sign [garden] will come in last with a marked form. Now this SL property, illustrated with lexical ordering in a sentence here, holds even for discourse-level ordering of sentences/clauses:

• everything participating in the setup is always signed before the action;

• chronological order is preserved when narrating a sequence of events-no lexical before/after mix;

• cause canonically preceeds consequence;

• etc.

While written language authors change orders and use meaningful prepositions to direct the link between two clauses, it is close to illegal or very misleading in Sign Language to do the same. We will see that this is consistent throughout our whole corpus study hereafter.

The reason we bring up this issue here is to emphasise on how much SL discourse structure differs from that of text, and to address the consequent fact that translating from one to the other cannot be carried out in a sequential fashion following the flow of text. Word-to-word translation is generally known to be of very poor quality; it is likewise for sentence-by-sentence translation of a text into Sign Language. In practice for instance, Sign Language interpreters would always rather be given summaries of the talks they interpret, and the overall quality of a resulting interpreted talk does vary according to how much is known of the talk beforehand [17].

Therefore, we claim that an automatic translation system should try and approach a text on that global level to carry the whole meaning and logic of a text into sign, knowing that clause order may change.

Corpus-driven methodology

To design a symbolic translation system from text, we need to parse the input into a symbolic representation of it. But as we have said, we want to place our study on the discourse level first rather than on the sentence level. Also, our approach is corpus-driven, but not based on machine learning techniques. Indeed, they usually need a linear signal and a tremendous amount of data to be trained, two premises which available SL corpora do not yet fulfill. This section presents and justifies the parallel corpus on which we base the rest of the work we report here.

The corpus

The corpus we use for this study is a parallel corpus of 1,000 French news item summaries, translated into LSF by professionals [34]. For each item we have a text comprising a 1-sentence title and a paragraph-long news content, together with their translated equivalents in a sign video (title + content). It is important to note, in relation with section 2.2, that the signed version is never a sentence-by-sentence translation, and that the whole text is always digested into a fully-acceptable SL production.

For our purpose, there are four advantages to this corpus which we explain hereafter.

Corpus genre By design, the corpus holds a series of texts with comparable length and style, which makes the data homogeneous, and its informational purpose is consistent. However, the vocabulary it not controlled or limited to any specific field. These two properties make a good tradeoff between convenience of the data to work with and potential for scalability.

Clean data Journalistic information is always written with the purpose of concision and clarity. None of the news content is bloat or misleading, which is clearly a plus. Besides, it is to note that unlike unverified web content, there are virtually no syntactic or spelling errors, which avoids relying on too much robustness from any text parsing system.

News content Of course, news reports almost systematically mention several people or place names, as well as events and clear relationships between them. This type of situation is known to be prone to spatial relationships in the signing space.

Local business Finally, we also appreciated that much of the recent years' Natural Language Processing research in our lab had produced systems trained on the huge AFP news agency feed. We could work with local tools kindly made available to us, highly trained for our genre of input texts.

AZee as output

During the past few years, LIMSI has been developing abstract sign description models capable of specifying various SL features, including:

• the Zebedee model for lexical signs including their geometric variability, in particular the dependency of their "location" on signing context [START_REF] Rosset | The limsi participation to the qast track[END_REF];

• the Azalee model for linguistic structures and the way they involve and synchronise the different articulators of the body [START_REF] Websourd | L'actualité en bref et en lsf de l'année[END_REF].

AZee is intended to be a general description language to be used as input to virtual signers in the future, combining Azalee and Zebedee, thereby capturing all of these features. In figure 1, AZee is the format of the input represented on the left-hand side, and the gears represent the animation system needed to produce the signed output on the right-hand side. Now if we consider this part of the work available, a good question in view of our full symbolic translation system is whether this input to the avatar animator can be used as output of the translator. In other words, could AZee be the interface between those two processes of the pipeline? See fig. 2.

Factoring out the invariants of a linguistic structure and parameterizing the variables into a dedicated AZee operator builds a production rule which we call "azop". An azop is named with a short explanatory description of the operation performed, and reused for any instance form. For example and as has already been reported in the DictaSign project [15], an "enumeration" operator can be written and applied to a list of items to enumerate, which implements a synchronised head movement as invariant and can be used with any list of things to enumerate.

AZee allows to build production rules on very different levels. A rule can be created to:

• add a facial expression to a single lexical item in order to add some form of adjectival meaning to it, e.g. a rule "adj-big" on a lexical sign that will typically generate a cheek puff on the sign [bear] to sign "a big bear";

• constrain the order of two (or more) full clauses if a structure is found to govern that order, e.g. a rule "setup" applying to a situational context and an event taking place in that context, that will make an avatar sign them in that order as it is preferred in SL syntax;

• even, on a much higher level, add discourse markers to punctuate full outline sections, e.g. a rule to generate automatic transition sentences like "now let's talk about", with the adequate pauses, eye blinks, etc.

AZee rules can represent phonetic details, lexical precedence, semantic and rhetorical discourse operations... There is no separation in these levels; they can all be dealt with using the same formalism. One goal of AZee is actually to build a "semantic grammar", to govern surface productions from the intended meaning, which discards syntax (in the linear ordering sense) from the top position of structuring elements. Also, we have seen that a coarse-grain level of discourse study was the preferred working level when dealing with translation to Sign Language. We are led to hypothesise that AZee is a good candidate to serve in our symbolic translation system. It takes the production rule system closer to the semantics of the discourse as a whole, which is what we want in an ideal output for such a system.

Methodology

Our methodology is based on a separate observation of text and video. This section describes what was done on each side of the corpus, and brings up a comparison task afterwards. Working from the videos first guarantees that we always work on the texts knowing what we are looking for and why, regarding the signed production, we are looking for it. Separating the steps and not studying the two languages side-by-side guarantees that there is no word-toword or sentence-to-sentence bias in our study, i.e. that we do concentrate on a linguistic system alone regardless of the task, and that we keep regarding meaning as the ultimate element to translate.

Video analysis

The analysis of the video part of the corpus, before any kind of comparison with the text, is made in collaboration with a professional Sign Language interpreter. Its process is somewhat empirical as there are no tools able to serve as counterpart to the ones used on text, like syntactic parsers or text scripts able to locate and tag a given sequence or pattern of lexical units. However, it is bound to stabilise with time and we intend to formalise it as we go.

The first step we took is ask the interpreter to write a free-hand description regarding information order in the video and everything regarding the signing space. This study, in relation to section 2.2, takes place on the discourse level as the interpreter describes the signing based on a segmentation in coarse-grain "chunks", which can be:

• a situational context, in which the following signed chunks will take place;

• a time/place setup for an event;

• the description of an event, person or place or any entity or group which will be referred to afterwards;

• the main point of the discourse, usually observed last.

The list is non-exhaustive and built on the fly. The chunking taking place here relies more on the intuition of where a new purpose of a type above starts or ends, but SL being an oral language, and similarly to any vocal-oral corpus, the notion of syntactic sentence becomes blurred. Then we draw graphs to represent links between the chunks, using every chunk as potential node, preferably written top to bottom to keep track of signing order. When they are assigned a specific location they are tagged with the linguistic reason for that spatialisation, and when two chunks are semantically bound, we draw edges between the nodes and tag the edge with identification of the link such as "cause", "opposition", "precedence", "context of". Once again, this is not yet claimed to be a formal framework, but an exploratory means of using a Sign Language professional's intuition to search for regularities and indeed quite a few emerged:

• every pair of chunks linked by an edge tagged "opposition" has two different locations, most the time followed by a verb "agreeing" with one or both locations (see example below);

• probably a particular case of the above, the same applies to comparisons between two entitites;

• every sequence of signs involving several geographical places locates one with reference to the previous-usually up to two or three and typically country, then area/region, and/or finally town;

• every video ends with the clause carrying the actual news, as expected...

Figure 3 shows snapshots of the clear opposition in the news item reading:

With two cowboys, the beautiful country of the American West and a love story, the film Brokeback Mountain combines all the ingredients of the typical western film. However, they are arranged in a way John Wayne would surely not have appreciated.

In 3.a, we see a snapshot of the first half, where everything describing the film is signed to the signer's left. In 3.b, the signer is referring to John Wayne and we see that his body has turned to his right. Finally, in the last two seconds of The point of this was to know where to start with annotating the texts, i.e. parsing and tagging the texts knowing what to look for. It is the purpose of the next section.

Text annotation

Three interesting phenomena were selected in the study of the videos described above, now the next step to take is to examine the text corpus and annotate the text structures related to the semantics of the chosen signed structures. In this section this time, the whole job will be text-only. It specifies the way we annotate the French text corpus, regarding the three following items: opposition, comparison and structures we call "geographic localisation". Also, we describe the tools we used to carry out the annotation task, part of it automatically.

Annotation specification

Opposition is a self-explanatory concept, but can be very subjective thus quite hard to define an annotation guideline for, and it can occur between segments of various lengths, from word to long sentence. This would enable us to analyse various situations but of course complicates the annotation process. To tackle this, we give an objective definition for a subset of opposition constructions, based on syntax: two entities are opposed when they are syntacticly conjoined through one of a set list of lexical opposition conjunctions.

Comparison is also a relation between two distinct entities, but there is a subtle difference in the way they are respectively treated: both opposed entities are at the same level, while in a comparison one is taken for the norm against which the other is measured. Observations also show that the segments involved in a comparison are rather shorter than those opposed, a simple word or syntagm on average. Comparison in French is structured as follows: "a est plus/moins/aussi adj que b" (in English: a is more/less/as adj than/as b). We chose to focus once more on this lexically marked (hence objective) pattern. To locate occurrences of this pattern in the texts, we used a semantic tagger called Wmatch, whose behaviour is described in the "tools" section below.

Geographic localisation is the last phenomenon we chose to annotate in the texts. It is interesting to include it in our study for two reasons, first because it seems always to trigger a use of signing space in a very precise way, and second because it is frequent in journalistic texts. Indeed, it consists in selecting a smaller portion of a given location with the use of cardinal directions, as we see in "au sud de Gaza" (south of Gaza), or "le nord de l'Italie" (the northern part of Italy). It can be extended with the name of the geographical entity being located, which is usually named first, then located with reference to the former. These patterns can also be tagged using Wmatch.

Here is an example of our XML-style annotation for geographic localisation in a text (the tags' names are still subject to modifications): <_loc> <_pt_cardinal> sud </_pt_cardinal> <_prep> de </_prep> <_det> la </_det> <_loc> <_province> <Gaza> bande de Gaza </Gaza> </_province> </_loc> </_loc>

Tools

In the LIMSI-CNRS environment, several software tools are used daily for parsing and editing text. We present three of them in this section, two of which we use in our study, while the third provides interesting extension prospects for our annotation processes.

The latter is named XIP, which stands for Xerox c Incremental Parser, a syntactic parser able to extract dependencies, named entities and semantic roles [START_REF] Filhol | Internal report on zebedee[END_REF]. We had first decided to use the dependencies to annotate oppositions, because the conjunctions were well tagged and considered as connectors, entering a binary dependency with the second verb of the sentence. The semantic parser, Wmatch described below, was not so good at finding them, but this was remedied by a new set of rules added to its grammar for our purposes, so we were able to put XIP aside with no loss of efficiency and a gain in the homogeneity of our set of XML tags. Still, XIP is not definitively removed from our panel of tools, because its output information on syntactic node dependencies will be useful in a later step of our project.

Wmatch [16] is a semantic tagger, and as such functions only at a local level. It is therefore unable to annotate far-reaching relations as XIP does, but local focus makes for greater precision at a local level, which is what we needed in the first steps of our project. It detects perfectly the patterns of comparison, annotating the adverb and the adjective under a distinctive tag, as well as the cardinal directions and locations associated to them.

PASTA, for Poste d'Annotations Syntaxiques Très Agile (very agile syntactic annotation post) is a software for annotating XML data with syntactic relations, originally designed for the Passage project [33]. In Passage, the types were used for syntagm type labels such as NPs, VPs, etc. and the relations for linking syntactic relations, for example a verb and its subject. A screenshot is given in figure 4, showing a dialog box where the user is prompted to fill in two slots of a relation. We chose to use this tool because it enables us to annotate both types and relations in a given text. Moreover, PASTA does not discriminate between a word or a group of words as source and/or target of a relation. With some modifications to its list of types and relations, PASTA enables us to annotate as groups the entities being opposed or compared (we did not specify their nature any more for the moment, but it may be done later as a refinement in our annotations). The relations highlight the connections between the two entities being compared or opposed and their connector, the cardinal direction and the place located with it, or the name of a geographic entity and its precise location. For that matter, our progress is as follows: automated annotations with semantic tagging are completed-and in the next part we will describe how we obtained them-; we are currently carrying out the manual pass of the annotations.

Methodology and tagging conventions

Once we had narrowed our project down to the three phenomenons described in the previous section, and isolated their characteristics, the task was a simple matter of determining the input and output formats, and programming the necessary modules to obtain the desired effects. We made our XML tagging conventions quite simple, and mostly followed Wmatch's tagging style, for the sake of consistancy.

We first tagged the texts with Wmatch to obtain the first version of our XML files, which was done using the native grammar of Wmatch [28]. The next step was to write a grammar extension to include new rules for opposition conjunctions, and adjust the comparison conjunctions to our writing conventions. We then applied this new grammar to the already-tagged files to produce our working version. In this version, we have kept all the tags supplied by Wmatch, as they may be useful when we need to be more precise in our analysis, and our tags highlighting the opposition and comparison conjunctions. We will then use PASTA manually to annotate the relations between the different components of our three structures. At the end of this task, we will fully have annotated the XML files with the desired information.

Comparison

Once we have annotated the text we can compare it to the videos, looking at both following cases:

• a structure is annotated in the text and a spatialisation of the corresponding entities takes place in the video; these are cases to validate the systematic link between the annotated lexical structures and a use of space in SL;

• a structure is annotated but the equivalent sign production falls out of the pattern above; these cases will be interesting to comment on, perhaps to refine some of the categories (e.g. more constraint on the type of comparison: comparing to one vs. comparing to many, etc.).

For example, we already see a very clear rule for geographic localisations. Let us look at the example given in section 4.2.1 again, with the semantically tagged structure for "south of Gaza". There are three important tags in this hierarchical structure: the two loc tags-the first holding the whole structure and the second nested in the first-and the cardinal direction tag pt cardinal. The geographic localisation structure is the whole group: the first encapsulating loc. When translated into Sign Language, every structure using this template consistently raises the same signed structure, i.e. in order:

1. a sign is performed for the second ' loc'-tagged item; 2. a zone is activated in space with a circular movement of a hand on a vertical plane and a quick look of eye gaze;

3. a subpart of that zone is designated (e.g. pointing sign) according to the pt cardinal term. Now for each XML tag structure that we will obtain once the whole corpus is annotated with relations through PASTA, we hope to find such translation rules, and start validating our corpus-driven approach to translation.

Objectives and conclusion

In short, this paper has presented a methodology to derive a first set of symbolic translation rules from text to sign, specifically looking at the spatialisation issue. Using a parallel corpus of text-to-sign translation, a first study was made of the videos (disregarding the texts) to find relevant reasons for sign and sentence relocation. Three first clear categories emerged: comparisons, oppositions, and geographic localisations. Scripts were run and tools used to tag the texts where lexically marked instances of these structures could be found, this time disregarding the videos. Then by looking at the two separate annotations, we will state whether systematic rules can be derived to translate any of those three structures.

A short term objective with this study is to find systematic rules governing use of space in LSF, when translating comparisons, oppositions and geographical localisations, in the case of textual constructs that are possible to pick up automatically. By the date of the conference, we should be able to produce the first findings on this, as the text annotation will be finished and compared to the video descriptions.

In a longer run, if we first succeed in formalising invariants on these sample structures, we will continue with more structures. There are other reasons for spatialising a sign sequence, and there are other features to look at and capture in production rules. Clause and statement order, for which we give a prospect below, is interesting to look at too. It is hoped that with this methodology, a full translation architecture with AZee support can be specified and implemented as the interface between the text and a Sign Language animation module.

To tackle clause/sentence order in Sign Language, we need to look at the rhetorical function of each counterpart in text, as a lot of reordering seems to take place to avoid linking lexical tokens like text equivalents "therefore", "because", etc. We intend to initiate collaborations with experts on rhetorical structure or formal discourse representation theories like RST or SDRT [START_REF] Filhol | A combination of two synchronisation methods to formalise sign language animation[END_REF], which formalises the way in which parts of a discourse link to one another. While it can still not be in reach to annotate this automatically, it will be interesting to work with a manual annotation of the discourse parts with this type of theory and see what emerges from a comparison of those to the sign chunks observed in the videos.

Figure 1 :

 1 Figure 1: Sign Language animation from formal description input

Figure 2 :

 2 Figure 2: AZee input to a virtual signer, and output from the text translation system

Figure 3 :

 3 Figure 3: Example of opposition structure observed in the corpus

Figure 4 :

 4 Figure 4: Annotating a relation with the PASTA program

Acknowledgement

We would like to thank Sophie Rosset for helping with the semantic tagging of our text corpus and use of Wmatch, and Patrick Paroubek for the time he spent adapting the PASTA program to our needs.


Baptiste Fosty

François Bremond

Visual recognition of gait parameters

Visual Recognition of Gait parameters

Baptiste Fosty, François Bremond keywords: RGB-D camera, gait analysis, rehabilitation There is major interest nowadays in Moderate-High Intensity Aerobic Activities for nonpharmacological interventions in elderly suffering from neurodegenerative diseases like Alzheimer's Disease and Related Disorders [START_REF] Kemoun | Effects of a physical training programme on cognitive function and walking efficiency in elderly persons with dementia[END_REF]. Within the context of the development of serious games for this population, we have developed algorithms to interact with the virtual environment through simple gesture recognition and walking speed computation (see Figure 1).

Method

Based on an RGB-D (Red Green Blue + Depth) camera such as Kinect, the walking speed computation is performed as follow. First, people on the scene are detected using a background substraction algorithm [START_REF] Nghiem | Controlling background subtraction algorithms for robust object detection[END_REF] and tracked using a multi-feature algorithm [START_REF] Chau | A multi-feature tracking algorithm enabling adaptation to context variations[END_REF]. Then, the feet positions are estimated based on the 25% of the lowest 3D points of the person. Finally, the walking speed of the person is computed from the successive positions of the feet.

Results and discussion

With the help of a specialist in the domain of physical activities, a protocol has been set up to assess the precision of the computed walking speeds (covering the walking speed spectrum, from 1 1.5 km/h to 5.5 km/h) and to prove the reproducibility of these results. Based on 36 participants who performed this protocol, the system has shown an absolute mean error of 0.17 km/h with a higher accuracy for median speeds (around 4.5 km/h). A paper is actually under reviewing process in Gait and Posture journal.

Following that, we decided to study whether this system would be useful in rehabilitation. Some experts of this domain in a rehabilitation center (Centre Hélio Marin in Vallauris, France) have been interviewed and they were very enthusiasts about using this type of system to get objective gait parameters. In collaboration with them, we started to develop a proper application for rehabilitation, adding new functionnalities around gait parameters computation such as side-byside video comparison, automatic sequencing of video or 3D display (see Figure 2).

Figure 1 :

 1 Figure 1: Gesture recognition and walking speed computation for serious game interactions. The gestures are : turning right (left picture), turning left (right picture) and action (center picture). The yellow dots show the positions of the arm and feet.

Figure 2 :

 2 Figure 2: Screenshot of the inferface for rehabilitation application. Top : patient general information. Left image : color image of the camera. Right image : 3D side view of the cloud of point of the person. Bottom (thumbnails) : sequences detected in the video.


Anh-Tuan Nghiem
email: nghiemtuan@gmail.com

François Bremond

Francois Bremond
email: francois.bremond@inria.fr

Background subtraction in people detection framework for RGB-D cameras

Introduction

In 2010, Microsoft introduced Kinect, the first low cost RGB-D camera. Together with the camera, Microsoft also provided a library for people detection and skeleton detection. As stated in [START_REF] Shotton | Real-time human pose recognition in parts from single depth images[END_REF], this library employs a background subtraction algorithm to detect foreground regions. However, perhaps due to high variance of depth measurement when objects are too far from the camera, this library only detects people when they are in the range of 0.5 to around 4.5 m from the cameras. This is also mentioned in [START_REF] Spinello | People detection in rgb-d data[END_REF]. Therefore, this library is not suitable for a general video monitoring application when monitored people may be outside the working range of the library.

To overcome the shortcoming of the provided library, we construct a new people detection framework consisting of a background subtraction, a people classifier, a tracker and a noise removal component. In this paper, we present a new background subtraction algorithm within the people detection framework specific to handle different type of noise in video from RGB-D cameras.

We evaluate the proposed people detection framework on the videos of -project. In our experiments, the proposed people detection framework based on the new background subtraction algorithm achieves comparable detection results with the library from PrimeSense, the company behind Kinect sensors when people are in the working range of the library (0.5 -4.5m). Beyond this range, the library stops working whereas our framework can still be able to detect people with few errors.

The paper is organised as follows. First we present related background subtraction algorithms, especially for RGB cameras in section 2. We then present the overall structure of the people detection framework in section 3. The details of our background subtraction algorithm are presented in section 4. Section 5 presents how our people detection framework removes noise. In section 6 we compare our algorithm with the library from PrimeSense and with GMM, a popular background subtraction algorithm for RGB cameras. Finally, the conclusion is presented in section 7.

Related works

For RGB cameras, background subtraction algorithms [START_REF] Stauffer | Adaptive background mixture models for real-time tracking[END_REF][START_REF] Elgammal | Non-parametric model for background subtraction[END_REF][START_REF] Barnich | Vibe: A universal background subtraction algorithm for video sequences[END_REF][START_REF] Hofmann | Background segmentation with feedback: The pixel-based adaptive segmenter[END_REF] have been used extensively to automatically detect foreground regions (moving regions) in a video coming from fixed cameras. [START_REF] Piccardi | Background subtraction techniques: a review[END_REF][START_REF] Radke | Image change detection algorithms: a systematic survey[END_REF] are extensive surveys on background subtraction algorithms.

For RGB-D cameras, one can consider depth videos as a special case of gray scale videos and apply the background subtraction algorithms of gray scale images to depth videos. However, due to high variance of depth measurement as illustrated in figure 1, current state-of-art background subtraction algorithms might not have good results in foreground / background classification.

Let's examine how some of these background subtraction algorithms can handle the depth video from RGB-D cameras.

In [START_REF] Stauffer | Adaptive background mixture models for real-time tracking[END_REF], the values at each pixel are modeled by a mixture of weighted Gaussian (GMM). A pixel value is classified as background if it belongs to a Gaussian with a heavy weight. For noisy data from depth videos, it is difficult to select the suitable threshold on the Gaussian weights to correctly classify foreground / background. Moreover, if people stand close to the background, only a small part of their body which is relatively distant from the background are de- Another approach is to model dynamic background using non-parametric methods. In [START_REF] Elgammal | Non-parametric model for background subtraction[END_REF], the values at each pixel is modeled by a history of N most recent values. The model construction and updating become simpler but the parameter selection to distinguish foreground / background is difficult. In [START_REF] Barnich | Vibe: A universal background subtraction algorithm for video sequences[END_REF][START_REF] Hofmann | Background segmentation with feedback: The pixel-based adaptive segmenter[END_REF], background models are also a set of N values taken from input videos but not the most recent values. These N values are taken randomly from incoming pixel values. The more a pixel values appears, the higher chance that it is included in the background model. For noisy data from depth videos, similar to [START_REF] Stauffer | Adaptive background mixture models for real-time tracking[END_REF], these algorithms also suffer the same problems such as selecting appropriate parameters and distinguishing people close to background from noisy depth values of background.

In summary, for depth videos from RGB-D camera, most of current background subtraction algorithms try to solve the problem of foreground / background classification at pixel level. At this level, the information is quite limited to have good classification when the variance of depth measurement is too high.

In this paper, we propose a background subtraction algorithm specific for depth videos from RGB-D cameras. Embedded in a people detection framework, it does not classify foreground / background at pixel level but provides useful information for the framework to remove noise. Noise is only removed when the framework has all the information from background subtraction, classification and object tracking.

Background subtraction in a people detection framework for RGB-D camera

The proposed people detection framework for RGB-D camera consists of a background subtraction algorithm, a HOG-based people classifier (see [START_REF] Nghiem | Head detection using kinect camera and its application to fall detection[END_REF]), a tracker, and a noise removal component. The noise removal component receives information from all the other components to effectively remove noise.

The background subtraction in this people detection framework, unlike other background subtraction algorithms for RGB cameras, assigns not only foreground or background label to each pixel but also labels corresponding to different types of noise specific to depth video.

The background model inside the background subtraction algorithm is updated selectively with the help of the feedback from the framework. Specifically, when the framework detects a person, the region corresponding to that person is not updated so that the background subtraction algorithm can detect that person even when that person stays in the same place for a long time. The Flickering noise contains the remaining background depth values which are highly deviated from the mean value of the background region. Flickering noise can be detected based on the fact that the depth value in this region appears only in a very short period, normally only one or two consecutive frames. On the other hand, when a person stands close to the background, although the depth corresponding to the person is close to this range, the values are more stable and not flickered as in case of the flickering noise.

Background subtraction algorithm 4.1. Modeling high variance of depth measurement

Background model

The purpose of our background model is to gather information to correctly model three types of regions described above: background, flat noise, and flickering noise.

Employing the idea from background subtraction algorithms for RGB cameras, we model depth values at each pixel by a set of codewords. Each code word is described by a tuple (m, n, t s , t l , n c ). In this tuple, m is the mean value of the depth values falling into this codeword. A depth value d is called "falling" into a codeword w if |d -m w | < τ B where τ B is the predefined threshold mentioned in section 4.1. It is fixed and the same for every codeword. n is the number of depth values falling into this codeword. t s is the frame number when this codeword is created. t l is the frame number of the last time this codeword has a depth value falling into it. n c : the number of times that this codeword has depth values falling into it consecutively. For example, if the depth values fall into the current codeword at frame 1, 2, 4, 5, 8, then n c = 3 which corresponds to 3 group of consecutive falling. This is illustrated in figure 3. At a given pixel, for an incoming depth value d, the algorithm will find the corresponding codeword that d can fall into. If there is no such codeword, a new codeword with m = d is created and inserted into the background model. Otherwise, the found codeword will be updated with d.

Updating codewords is straightforward for most of the features of codewords except the mean value m. In our algorithm m is updated as follows:

m new = min(N, n)m old + d min(N, n) + 1 (1)
where N is a predefined value to help d to affect the value of m even when n is big. In our experiment, we set N = 100.

Depth value classification

As described in section 3 about overall structure of the framework, the classification of foreground / background pixels is realised at two levels: pixel level of background subtraction algorithm and object level when the framework has the feedback from the classification and tracking step.

At the pixel level, the background subtraction algorithm assigns one of the following label to each incoming depth value: {BG, F L, F LK, T B, F G}. BG/ F L/ F LK means depth values are in the background / flat / flickering region in figure 2. T B means that depth values belong to an uninterested object newly appearing in the camera view. For example, when a chair is moved or a bag is put on a table, the pixels belonging to those objects should have label T B. F G means that depth values do not belong to one of the above background labels. In other words, these depth values correspond to moving objects.

To explain the classification of depth values, let's assume that we have a set W = {w} of codewords for a given pixel P and we want to classify a depth value d. In W , assuming that w 0 is the codeword with the highest number of depth values falling into it.

To classify d, the algorithm firstly finds the codeword w * whose the mean value m * is close to d (|d -m * | < τ B ). If there is no such codeword, the algorithm assigns label F G to d. Otherwise, the label of d is determined based on the characteristics of w * .

The algorithm assigns the label BG to d if w * is also w 0 . The algorithm assigns the label F L to d if w * is not w 0 but τ B < |m * -m 0 | ≤ τ F τ F is a small threshold mentioned in section 4.1.

The algorithm assigns the label F LK to d if w * satisfies two conditions. Firstly, when several consecutive depth values fall into the same codeword w * , the duration of this consecutiveness is no longer than β frames (βn * c > n * , β = 3 in our experiment). Secondly, this phenomena should occurs once in at least every γ frames ((t -t s )/γ ≤ n * c where t is the current timestamps). γ = 100 in our experiment.

In our algorithm, if a depth value satisfies the constraints of both F LICKERIN G and F LAT , it will carry both labels for later processing.

The algorithm assigns the label T B to d if w * satisfies two constraints of a temporal background like a moved chair. Firstly, the chair should stay in the new place long enough to be considered as background (n * > α, α = 200 in our experiment). Secondly, after the chair has been moved to a new place, at each pixel in the region corresponding to the new place of the chair, the depth value measured by RGB-D camera is always the same except when there is another object moving by ((t * l -t * s )/n * < ψ where ψ = 0.8 in our experiment).

d is assigned the label F G if the corresponding codeword w * does not belong to one of the above conditions.

Eliminating noise

As explained in section 3, at the pixel level, the background subtraction algorithm does not have enough information to remove noise. Therefore, noise is only removed when there is information from the classification and tracking.

Two types of noise regions are considered: foreground regions containing only noise pixels and foreground regions containing both noise pixels and real people. Let's call the first type as pure noise and the second type as mixed noise.

Eliminating pure noise

Assuming that at frame t, a foreground region R t is classified as a human by the classification task with probability P H (R t ). Moreover, R t is linked by the tracker to the foreground regions R t-1 at frame t -1, R t-2 at frame t -2 etc.

We define the probability of being the noise of type l for R t in single frame t as P l (R t ) = N l /N . Here N l is the total number of foreground pixels inside R t with the noise label l assigned by the background subtraction algorithm and N is the total number of pixels in R t .

Then R t is classified as not noise of type l if:

P H (R t-i )(1 -P l (R t-i )) > ϕ (2)
In our experiment, ϕ = 0.8. If in M consecutive frames, R t is not classified as a noise region of type l, future foreground regions linked with R t will also be considered as not noise of type l by the above criteria.

Let's take an example on how to distinguish a person and a foreground region corresponding to flat noise to understand how pure noise is removed.

When a person enters a room, he should be somewhere in the middle of the room before approaching the wall where flat noise may occur. Therefore, at least in some consecutive frames, the probability of being classified as potential flat noise P F L (R t ) is small. Moreover, if the people classifier can recognise the person with high probability (P H (R t )) during these frames, the value on the left hand side of the equation ( 2) should be high. Consequently, in the subsequent frames, the person will not be classified as flat noise even when he stays close to the wall for a long time.

On the other hand, when a foreground region corresponding to flat noise is detected, in all frames during its lifetime, the probability that this foreground region is classified as flat noise should always be high. Beside that, if the framework has a good people classifier, the probability that this foreground region is classified as person should be small. Combining these two conditions, the value on the left hand side of equation ( 2) should be small and this foreground region will be classified as noise by the framework.

The above method fails when the person stays only in regions very close to the wall (background) as soon as he enters the room. However, this case is not common in the reality.

We can apply the same reasoning for flickering and temporal background noise.

Eliminating mixed noise

Mixed noise corresponds to foreground regions containing both people and noise pixels. Because it contains people, mixed noise cannot be detected and removed by the method presented in section 5.1. This type of noise is detected based on the comparison between the size of the foreground region and the normal size of human body. Specifically, the probability P M (R t ) that a foreground region R t is classified as mixed noise is computed as follows:

P M (R t ) = max(P 1 (R t ), P 2 (R t ), P 3 (R t )) (3)
where P 1 (R t ), P 2 (R t ), P 3 (R t ) corresponds to the probability that the height, width, area of R t are bigger than those of a normal person. Because noise can only increase the size of foreground regions corresponding to people, we model those possibilities using half of a normal distribution as follows:

P i (R t ) ∝ 0 if x ≤ T x 1 -N (x|T x , σ x ) otherwise (4)
where x corresponds to height, width, area of the foreground region. T x , σ x are the mean and variance of the normal distribution of corresponding human size.

A foreground region is classified as mixed noise if its probability of mixed noise is larger than a certain threshold.

To remove noise from a foreground region corresponding to mixed noise, we simply remove foreground pixels with label noise assigned by the background subtraction algorithm and keep only the pixels with label F G.

With equation ( 4) a person pulling some objects or a group of people might be classified as mixed noise. However, because the noise removal process only removes pixels with noise labels, those people are often not affected.

PrimeSense GMM Proposed Precision 0.979 0.979 0.998 Sensitivity 0.987 0.987 0.995 F-Score 0.983 0.983 0.996 Table 1. Performance of people detection on close range videos.

Experiments

For RGB cameras, there are standard dataset such as the Change Detection Challenge [START_REF]A video database for testing change detection algorithms[END_REF] to compare the performance of different background subtraction algorithms. However, for RGB-D cameras, there has been no such standard dataset yet. Therefore, to test our background subtraction algorithm, we use some videos from the project Dem@care [START_REF]Dementia ambient care: Multi-sensing monitoring for intelligent remote management and decision support[END_REF] and specific videos expressing several difficult problems of background subtraction algorithm for depth videos. The frame rate of these videos is around 4 frames /s.

The ground truth of these videos are object bounding boxes drawn on selected frames. The frames are selected if they contain moving people. A person is considered as detected (true positive) if the area of the intersection between the bounding box of the ground truth and the bounding box given by the algorithm is larger than 70% of the area of both bounding boxes.

In our experiment, we compare our background subtraction algorithm with GMM [START_REF] Stauffer | Adaptive background mixture models for real-time tracking[END_REF] and with the people detection library of PrimeSense, the constructor of the RGB-D camera Microsoft Kinect.

For GMM, we replace our background subtraction algorithm in the people detection framework with GMM and turn off noise removal algorithms. Therefore, GMM has the ability to update its background selectively according to the results of people classification and tracking. To handle high variance of depth measurement, we set up parameters of GMM so that it favors removing noise than detecting moving objects.

For the library of PrimeSene, it is a full people detection framework and we only take the output of the library to compare with the results of the other two algorithms.

In the first experiment, we compare the performance of PrimeSense and our people detection framework with GMM and with the proposed background subtraction algorithm on close range videos when the distance between the camera and the farthest point in the scene is less than 5m. Therefore, the variance of depth measurement on these videos is small. There are three close range videos consisting of 12446 frames. Among these frames there are 114 ground truth's bounding boxes. As shown in table 1, our algorithm with noise removal can achieve comparable results with those of PrimeSense library and GMM.

In the second experiment, we compare the performance of our background subtraction algorithm with the performance of GMM on long range depth videos in which the distance between the camera and the farthest point is around 7m (figure 5, Long range). From this experiment, we do not compare the performance of PrimeSense library because it can only detect people if they are within 5m away from the camera. From the results of this experiment (table 2, dataset Long), we see that the precision of GMM is not as good as the precision of our background subtraction algorithm with noise removal. This reflects the fact that when the distance increases, the proposed algorithm is better than GMM in handling high variance of depth measurement.

In the third experiment, we compare the performance of our background subtraction algorithm with the performance of GMM on a depth video of a very long corridor (figure 5, Corridor). In this video, the variance of depth measurement at pixels corresponding to the end of the corridor is quite high. From the results of this experiment (table 2, dataset Corridor) we see that, our background subtraction algorithm with noise removal has a higher precision index than the precision index of GMM. This means that it can better handle the high variance of depth measurement when objects are far from the camera. The sensitivity of object detection also shows that both GMM and the proposed people detection framework can detect people even when they are very far from the camera.

In the final experiment, we employ a video in which there is a dark bookcase on the right of the frame (figure 5, Dark bookcase). Because this region does not reflect much the infra-red emitted from the depth-camera, the depth-camera can measure valid depth values intermittently in only few frames. Most of the time, the depth-camera assigns zero depth value to pixels in this region. Consequently, GMM produces many noise in this region which decrease the precision dramatically (table 2, dataset Dark region). For our algorithm, it cannot remove this kind of noise using flat noise removal as the permanent background is zero which is very different from measured depth values when they are available. However, we see that the valid depth values in this region has the characteristic of flickering noise. Therefore, with the help of the temporal filter our algorithm can remove most of flickering noise. The precision of our algorithm in this experiment is not as high as the precision in other experiments because when the treadmill heavily occludes the person, the person region is split into some small regions which are considered as noise.

Conclusion

In this paper, we present a background subtraction algorithm in a people detection framework for RGB-D camera. The proposed background subtraction algorithm differs from other pixel-based background subtraction algorithm in two aspects. Firstly, our algorithm does not try to classify The general idea of a background subtraction algorithm inside an object detection framework and the idea of flat noise and flickering noise can be generalised for videos of RGB camera. Therefore, in the future, we also would like to combine this idea to the case of people detection in videos from RGB camera.

Figure 1 .

 1 Figure 1. Depth value of a background pixel far from the camera. The difference between maximum and minimum value is as large as 40 cm.

Figure 2 .

 2 Figure 2. Range of depth pixel values is divided into three regions: Background, Flat noise, and Flickering noise. See text for more details. To model the high variance of depth measurement, we divide the variance range into three separate regions: background, flat noise, and flickering noise as illustrated in figure 2. In this figure, background region corresponds to the densest region of background depth values. Formally, a depth value d belongs to the background region if |d-d 0 | ≤ τ B where d 0 is the mean of the depth values in the background region and τ B is a small fixed threshold. The Flat noise contains background depth values d within the range τ B < |d -d 0 | ≤ τ F where τ F is around 20 cm in our experiment. To distinguish the background depth values in this range from the depth values due to a person standing close to the background, we employ the classification and tracking information as explained in section 5.1.

Figure 3 .

 3 Figure 3. Illustration of computing nc. (See text for details).

Figure 4 .

 4 Figure 4. The performance of noise removal.

Figure 5 .

 5 Figure 5. The detection results of GMM (upper ) and the proposed algorithm (lower) on some videos from RGB-D cameras.

Table 2 .

 2 Performance of GMM and the proposed background subtraction algorithm on three dataset. (#frames: number of frames, #GT: number of ground truth bounding boxes)

		Data			GMM			Proposed
	Name	#frames #GT Precision Sensitivity F-score Precision Sensitivity F-score
	Long	12355	258	0.89	0.97	0.93	0.95	0.97	0.96
	Corridor	1069	210	0.5	0.9	0.64	0.98	0.97	0.97
	Dark region 610	132	0.44	0.97	0.61	0.85	0.94	0.89


Sławomir Bąk

Filipe Martins

Francois Brémond

Person re-identification by pose priors

Keywords: re-identification, metric learning, pose matching

The person re-identification problem is a well known retrieval task that requires finding a person of interest in a network of cameras. In a real-world scenario, state of the art algorithms are likely to fail due to serious perspective and pose changes as well as variations in lighting conditions across the camera network. The most effective approaches try to cope with all these changes by applying metric learning tools to find a transfer function between a camera pair. Unfortunately, this transfer function is usually dependent on the camera pair and requires labeled training data for each camera. This might be unattainable in a large camera network. In this paper, instead of learning the transfer function that addresses all appearance changes, we propose to learn a generic metric pool that only focuses on pose changes. This pool consists of metrics, each one learned to match a specific pair of poses. Automatically estimated poses determine the proper metric, thus improving matching. We show that metrics learned using a single camera improve the matching across the whole camera network, providing a scalable solution. We validated our approach on a publicly available dataset demonstrating increase in the re-identification performance.

INTRODUCTION

Person re-identification is a well known problem in computer vision community. This task requires finding a target appearance in a network of cameras with non-overlapping fields of view. The changes in person pose together with different camera viewpoints and different color responses make the task of appearance matching extremely difficult.

Current state of the art approaches focus either on feature modeling [START_REF] Bak | Person re-identification using spatial covariance regions of human body parts[END_REF][START_REF] Farenzena | Person re-identification by symmetrydriven accumulation of local features[END_REF][START_REF] Wang | Shape and appearance context modeling[END_REF][START_REF] Bazzani | Multiple-shot person re-identification by hpe signature[END_REF] that designs descriptors invariant to camera changes or on metric learning [START_REF] Koestinger | Large scale metric learning from equivalence constraints[END_REF][START_REF] Davis | Information-theoretic metric learning[END_REF][START_REF] Guillaumin | Is that you? metric learning approaches for face identification[END_REF][START_REF] Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF][START_REF] Dikmen | Pedestrian recognition with a learned metric[END_REF][START_REF] Zheng | Person re-identification by probabilistic relative distance comparison[END_REF][START_REF] Li | Locally aligned feature transforms across views[END_REF] that uses training data to search for matching strategies minimizing the appearance changes (intra-class variations), while highlighting distinctive properties of the target (maximizing inter-class variation). Feature-modeling approaches concentrate on feature representation which should be invariant to pose and camera changes. These approaches usually assume a priori an appearance model, focusing on designing novel features for capturing the most distinctive aspects of an individual. Nevertheless, metric learning approaches are often the one that achieve the best performance in re-identifying people. These approaches learn a distance function that transfer the feature space from one camera to the other such that relevant dimensions are emphasized while irrelevant ones are discarded. Although, this transfer function boosts the recognition accuracy, it is usually camera pair dependent and requires large training data (hundreds of annotated image pairs with the same individual) for each camera pair. This might be unattainable in a large camera network. Moreover, metric learning can lose the performance while directly computing the difference between two feature vectors without aligning them first.

In this paper, instead of learning a single metric that tackles all difficulties related to appearance changes, we propose to divide the person re-identification task into several sub-tasks: (1) target alignment; (2) pose estimation and (3) pose-driven metric pool.

The first two sub-tasks: target alignment and pose estimation are approached using the recently published method [START_REF] Bak | Improving Person Re-identification by Viewpoint Cues[END_REF] (see Section 2.1). From Fig. 1 we can notice that serious perspective changes might significantly distort the appearance, deteriorating the recognition accuracy. Fortunately, a simple but efficient affine transformation using 3D scene information can improve alignment of two images extracted from different cameras. The pose orientation can be automatically estimated using the motion or the trajectory of the target. Given two images with estimated poses of the subject, it is highly desirable to develop the strategy that could exploit pose information for improving the matching accuracy. In the result we define the third task that consists in generating a pool of metrics, each one learned to match a specific pair of poses. In this paper we propose to learn the metric pool using only a single camera. We believe that this avoids over-fitting the metric to the given camera pair. Once the metric pool is learned, it can be applied to any pair of cameras. While matching two images, we select the proper metric based on automatically estimated pose of the target image and of the record image in the gallery (database). The selected metric reflects the transformation of the feature space between two given poses, thus improving matching. The proposed solution showed to be effective and scalable (see Sec. 3).

THE METHOD

Our approach requires two steps. The first stage overcomes perspective changes and estimates the pose of the target. We apply a simple but effective technique [START_REF] Bak | Improving Person Re-identification by Viewpoint Cues[END_REF] that minimizes perspective distortions by an affine transformation of a cropped image containing the target. The pose is estimated using 3D scene information and motion of the target. The second stage is responsible for appearance matching using a metric pool that consists of metrics, each one learned to match a specific pair of poses.

Target alignment and pose estimation

The changing viewpoint in a network of cameras might significantly distort the visual appearance of a person. We minimize perspective distortions by rotating the cropped image with the person by an angle computed by mapping the ground plane living in real world coordinates to the vertical of a given image (see Ref. 12). This mapping can be computed by employing the calibration information of the camera (e.g. Tsai calibration). In the result, the cropped image should contain the person aligned to the vertical axis (see alignment in Fig. 1).

We estimate the pose by using 3D scene information and motion of the target. [START_REF] Bak | Improving Person Re-identification by Viewpoint Cues[END_REF] Pose orientation is computed as a dot product between the viewpoint vector and the motion vector. The view point vector is defined as the difference between the real world location coordinates of the target in the scene and the real world location coordinates of the camera, both projected on the ground plane. The motion vector is determined by the difference between two consecutive real world location coordinates of the target. The dot product stands for the angle between the camera viewpoint and the pose direction of the target. Employing this simple but effective method, we obtain sufficiently accurate information for estimating the pose. This information is used to select the data for training a metric pool for pose changes and then it can be used to select the proper metric while matching different poses.

Given two images with estimated poses of the subject, we develop the strategy that can exploit pose information, thus improving the matching. This strategy consists in generating a pool of metrics, each one learned to match a specific pair of poses. For learning metrics we employ Mahalanobis metrics, which have recently attracted a lot of research interest in computer vision.

Mahalanobis metric

As state of the art we can consider KISS metric, [START_REF] Koestinger | Large scale metric learning from equivalence constraints[END_REF] Large Margin Nearest Neighbor Learning (LMNN), [START_REF] Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF] Information Theoretic Metric Learning (ITML) [START_REF] Davis | Information-theoretic metric learning[END_REF] and Logistic Discriminant Metric Learning (LDML). [START_REF] Guillaumin | Is that you? metric learning approaches for face identification[END_REF] These machine learning algorithms learn a distance or similarity metric based on the class of Mahalanobis distance functions. Mahalanobis-like metric measures the squared distance between two data points x i and x j :

d 2 M (x i , x j ) = (x i -x j ) T M(x i -x j ), (1)
where M ≥ 0 is a positive semi-definite matrix and x i , x j ∈ R d is a pair of samples (i, j). Metric learning methods rely on the idea that similar data points should be closer in the feature space than dissimilar points.

For training we need a similarity label y ij : y ij = 1 for similar pairs, i.e. if the samples share the same class label (y i = y j ) and y ij = 0 otherwise.

Usually metric learning approaches rely on an iterative optimization scheme which can get computationally expensive for large datasets. In contrast to these methods, KISS metric is a non-iterative method that builds on a statistical inference perspective of the space of pairwise differences. [START_REF] Koestinger | Large scale metric learning from equivalence constraints[END_REF] In the result, it is orders of magnitudes faster than comparable metric learning approaches.

Learning a pose-change metric

Let us assume that the pose can be described by the orientation angle between the motion vector of the target and the viewpoint vector of the camera (see details in Sec. 2.1). Thus, for each image we have given the pose as the angle in the range of [0 • , 360 • ). We divide this range into n bins, i.e. pose p ∈ P , where |P | = n. Given n bins of estimated poses, the idea is to learn metrics that stand for transfer functions between pairs of two poses. In the result, the metric pool will consist of n(n+1) 2 metrics, each one learned to match specific pair of poses.

Learning is performed in the following way. For each pair of poses, we automatically select subjects that support a given pose transition taking place in the same camera. We learn a pose-change metric using only a single camera to avoid including any color transfer in the metric. While learning metrics, we follow a well known scheme based on image pairs, containing two desired poses of the same target. Let us assume that we want to learn the metric for the pose change from pose a to pose b. In this case y ij = 1 only if it is the same subject (y i = y j ) and only if it supports the pose change (p i = a ∧ p j = b), y ij = 0 otherwise.

For learning metrics we employ the previously mentioned metric learning tools. [START_REF] Koestinger | Large scale metric learning from equivalence constraints[END_REF][START_REF] Davis | Information-theoretic metric learning[END_REF][START_REF] Guillaumin | Is that you? metric learning approaches for face identification[END_REF][START_REF] Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF] As the learning is performed offline, the time complexity is not the main concern. However, if the reader is interested in training on a large dataset we recommend the KISS metric. [START_REF] Koestinger | Large scale metric learning from equivalence constraints[END_REF] In experiments we compare the performance of all these approaches.

The set of learned metrics stands for the metric pool. This metric pool is not dependent on a camera pair. Thus, once the pool for matching poses is learned, it can be applied to any pair of cameras.

While matching two images given from different (or the same) camera, we first align subjects and estimate their pose orientations. Having two poses, we select the corresponding metric from the metric pool. This metric is used to compute the similarity between given subjects that is used in the final ranking. As the selected metric reflects the transformation of the feature space between two given poses, it improves the recognition accuracy. This section evaluates the re-identification performance, while employing metric learning tools for handling pose changes. The results are analyzed in terms of recognition rate, using the cumulative matching characteristic (CMC) [START_REF] Gray | Evaluating Appearance Models for Recognition, Reacquisition, and Tracking[END_REF] curve. The CMC curve represents the expectation of finding the correct match in the top n matches.

Database and experimental setup

We carry out experiments on the SAIVT-SOFTBIO database. [START_REF] Bialkowski | A database for person re-identification in multi-camera surveillance networks[END_REF] This dataset consists of 152 people moving through a network of 8 cameras. Subjects travel in an uncontrolled manner, thus most of subjects appear only in a subset of the camera network. This provides a highly unconstrained environment reflecting a real-world scenario. In average, each subject is registered by 400 frames spanning up to 8 camera views in challenging surveillance conditions (significant illumination, pose and viewpoint changes). Provided annotations given by coarse bounding boxes indicate locations of the subjects in each frame. The centers of these bounding boxes build trajectories of the subjects. Thanks to trajectories and 3D scene information we align and estimate the pose of subject images. [START_REF] Bak | Improving Person Re-identification by Viewpoint Cues[END_REF]

Appearance model

Every cropped and aligned image is scaled into a fixed size window of 64 × 192 pixels. A set of rectangular sub-regions is produced by shifting 32 × 32 regions with a 16 pixels step. This operation results in 33 overlapping rectangular sub-regions. From each sub-region, we extract RGB color and HOG histograms. We minimize color dissimilarities caused by camera illumination changes by applying histogram equalization to each color channel. By this operation we try to avoid a dependency of our metric on the camera color spectrum.

Learning and testing

For learning a metric we selected the camera 5 as it provides the sufficient number of subjects (37) and images (279) that support a given pose change. The pose orientation was divided into n = 3 bins: the front pose, the back pose and the side pose (either left or right as the image can be flipped). The centers of bins are 0 • , 180 • and 90 • , respectively. The image is classified in one of the poses based on the nearest neighbor strategy.

We learn the transfer from the side pose to the back pose using a single camera. By using a single camera we want to avoid including any color transfer in our metric, thus producing independent to camera pair metric. Training data was processed using the method Ref. 12 that crops, aligns and estimates pose of the subject in each frame.

Learning is performed using KISSME framework 5 that provides several metric learning tools ( KISS metric, [START_REF] Koestinger | Large scale metric learning from equivalence constraints[END_REF] Mahalanobis distance with similar pairs, Large Margin Nearest Neighbor Learning (LMNN), [START_REF] Weinberger | Distance metric learning for large margin nearest neighbor classification[END_REF] Information Theoretic Metric Learning (ITML) [START_REF] Davis | Information-theoretic metric learning[END_REF] and Logistic Discriminant Metric Learning (LDML). [START_REF] Guillaumin | Is that you? metric learning approaches for face identification[END_REF] For testing we randomly select a single image for each subject that supports the given pose change. All camera pairs are evaluated using 50 subjects. The procedure is repeated 10 times to obtain reliable statistics.

Results

Fig. 2 illustrates the results of re-identification on different camera pairs. identity label corresponds to the diagonal metric M that is the Euclidean distance (L 2 metric). The remaining labels correspond to state of the art metric learning tools: kissme, 5 mahal-Mahalanobis for y ij = 1, itml, 6 ldml 7 and lmnn 8 ).

From the results, it is apparent that learning a metric for the pose change improves the recognition for all camera pairs. Each metric learning method shows improvement w.r.t the L 2 metric. This result is very promising, especially in Fig. 2(a,b) where we can notice the improvement even when the camera pair does not contain the training camera 5.

From Fig. 2(c,d) it seems that the improvement in matching is higher when the testing camera pair contains the camera 5. This might mean that our metric show some dependency on the training data (e.g. the selected model is dependent on the features available in the camera 5). In the result, while learning a metric pool we should properly select the training data to obtain sufficiently general metrics. Alternative solution would be to choose such image features for representing the appearance model that are less dependent on the camera. We performed tests using only the HOG descriptor, but unfortunately we obtained decrease in the performance. Designing a new appearance model, we need to look for a trade-off between its discriminative power and invariance through cameras. This task is particularly hard, especially, as this trade-off varies from data to data. [START_REF] Varma | Learning the discriminative power-invariance trade-off[END_REF]

CONCLUSION AND PERSPECTIVES

We proposed to re-identify people using their pose orientations. Our strategy is based on the idea that the appearances of the subject in different poses ought to be matched by different metrics. The pose can be estimated using the motion of the target. Given estimated poses, we learned the metric pool, i.e a set of metrics where each one is learned to match a specific pair of poses. We learned the transfer functions employing Mahalanobis metrics using only a single camera. This allowed us to apply the metric to uncorrelated camera pairs, providing the scalable solution for large camera networks. Experiments on various cameras demonstrated that our method consistently improves the re-identification accuracy. In the future, we will further explore the generalization capability of the pose-driven metric pool. Different training schemes will be tested to obtain more general metrics. Additionally, we plan to analyze the number of metrics in the metric pool w.r.t. the re-identification accuracy and propose a finer mapping from the image to the pose using a depth sensor.

Figure 1 .

 1 Figure 1. Person re-identification using pose priors: the figure illustrates the process of matching the same subject across different camera views; the cropped images are first aligned, then the pose orientations are estimated determining the metric for the given pose change. The selected metric reflects the specific to pose change transformation of the feature space.

Figure 2 .

 2 Figure 2. Person re-identification by pose change metric: CMC curves obtained on the SAIVT-SOFTBIO dataset on different camera pairs. The metric for the change from the side pose to the back pose was learned using images from the camera 5.

ACKNOWLEDGMENTS

This work has been supported by PANORAMA, CENTAUR and MOVEMENT European projects.