1. le calcul des produits point à point pour toutes les primitives, 1024×1024), cette accélération descend à une valeur moyenne de ×1, 6. Lorsqu'ils sont combinés ensemble, l'accélération atteint un maximum de ×5, 11 pour des tailles 128 × 128 et ×3, 35 pour des tailles 1024 × 1024. En ce qui concerne le Nehalem octo-coeurs, les accélérations sont plus faibles, mais le temps d'exécution est plus faible aussi, en raison de ses bus de cache rapides. Le gain apporté par OpenMP est moins important excepté pour Nehalem comparé au gain du codage SIMD. Concernant les temps d'exécution, la transformation SdM→MdS + SIMD est nécessaire pour permettre une exécution temps-réel (40 ms par image) pour des images 1024×1024. Dans le cas de l'implémentation mono-thread, seul le SandyBridge (avec instructions SSE) est temps-réel et peut permettre un gain de ×4.0 après augmentation de la fréquence d'horloge. L'exécution de CT a été testée sur les séquences utilisées dans [79] à la fois en version monochrome et couleur. À cause des performances réduites des versions multi-threadées, l'algorithme est finalement implémenté et évalué sans multithreading. Après les différentes transformations algorithmiques, les tests effectués montrent des temps d'exécution allant de 3 ms à 11ms, rendant donc l'algorithme rapide et compatible avec une exécution temps-réel sur un coeur.

Suivi temps-réel : matrices de covariance couleur-texture et commutation automatique de descripteur/opérateur . .

Synthèse du manuscrit en version française

Introduction

Cette thèse propose un système de vision par ordinateur capable de détecter et suivre plusieurs objets dans les séquences vidéo. L'algorithme proposé de recherche de correspondances s'appuie sur les matrices de covariance obtenues à partir d'un ensemble de caractéristiques extraites des images (couleur et texture principalement). L'idée initiale sur laquelle reposent les bases de notre algorithme a été publiée par Porikli et al. [START_REF] Porikli | Covariance tracking using model update based on lie algebra[END_REF] et son principal avantage est l'utilisation d'un descripteur de l'objet à suivre fusionnant des sources d'information très hétérogènes. Cette représentation, compacte et discriminante est efficace pour le suivi d'objets mais également leur ré-identification.

Quatre contributions sont introduites dans cette thèse. Tout d'abord nous nous intéressons à l'invariance des algorithmes de suivi face aux changements de contexte. Nous proposons ici une méthodologie pour mesurer l'importance de l'information couleur en fonction de ses niveaux d'illumination et de saturation. Ensuite, une deuxième partie se consacre à l'étude des différentes méthodes de suivi, leurs avantages et limitations en fonction du type d'objet à suivre (rigide ou non rigide par exemple) et du contexte (caméra statique ou mobile). La méthode que nous proposons s'adapte automatiquement et utilise un mécanisme de commutation entre différentes méthodes de suivi en considérant leurs forces complémentaires [START_REF] Laguzet | Color tracking with contextual switching: Real-time implementation on CPU[END_REF]. Notre algorithme s'appuie sur un modèle de covariance qui fusionne les informations couleur-texture et le flot optique (KLT) modifié pour le rendre plus robuste face aux changements d'illumination [START_REF] Romero | Feature points tracking adaptive to saturation[END_REF]. Une deuxième approche propose l'analyse des différents espaces de réprésentation couleur afin d'obtenir un descripteur assurant un bon équilibre entre pouvoir discriminant et invariance photométrique. Nous avons proposé une nouvelle représentation par matrice de covariance qui par une nouvelle façon d'intégrer les caractéristiques de texture LBP (pour Local Binary Patterns en anglais) permet de réduire la taille des matrices tout en conservant un très bon pouvoir discriminant [START_REF] Romero | Covariance descriptor multiple object tracking and re-identi cation with colorspace evaluation[END_REF].

Une troisième contribution porte sur le problème de suivi multi-cibles où plusieurs difficultés apparaissent parmi lesquelles la confusion des identités de deux objets, les occultations, la fusion ou la division des trajectoires. Ici, un ensemble de matrices de covariance distribuées spatialement est utilisé pour ré-identifier les cibles. La solution finale que nous proposons utilise une fonction d'énergie discrète qui s'appuie sur le comportement de l'ensemble des trajectoires et sur les modèles d'apparence proposés [START_REF] Romero | Total bregman divergence for multiple object tracking[END_REF]. Finalement, à partir des algorithmes développés, nous avons réalisé une étude sur l'adéquation algorithme-architecture et implémenté les codes sur des architectures multi-coeur en atteignant des gains importants en terme de rapidité d'exécution [START_REF] Laguzet | Color tracking with contextual switching: Real-time implementation on CPU[END_REF][START_REF] Romero | Real-time covariance tracking algorithm for embedded systems[END_REF].

Ainsi la thèse se divise en six parties. Tout d'abord, nous introduisons dans la partie 0.2 les caractéristiques visuelles disponibles au niveau pixel et leur utilisation pour définir le descripteur plus global de l'objet à étudier. Nous abordons ensuite dans 0.3 le problème de la détection d'objet et du suivi. La troisième partie 0.4 détaille les premières contributions de la thèse : la commutation d'opérateurs et de caractéristiques pour le suivi. Ensuite, nos approches proposées en ce qui concerne la mise en correspondance par covariance font l'objet de la partie 0.5. Enfin, l'étude algorithme-architecture est développée dans la partie 0.6.

Information locale et représentation d'objets

La qualité d'un descripteur associé à un objet peut être mesurée suivant trois critères :

• la compacité de la représentation, pour permettre un stockage et une exécution efficaces ;

• le caractère distinctif ou discriminant, c'est-à-dire que chaque descripteur doit être unique pour chaque objet de la vidéo considérée ;

• la répétabilité : la représentation doit être constante au cours du temps malgré les modifications des conditions d'acquisition ; À partir des multiples informations pixelliques disponibles (voir paragraphe 0.2.1), plusieurs descripteurs d'objets sont envisageables (paragraphe 0.2.2).

Information disponible au niveau pixel

La couleur, la luminance. La vision humaine est fondamentalement tri-chromique : elle est basée sur les réponses de trois types différents de cônes de photo-récepteurs situés dans notre rétine (LMS). Pour des raisons historiques, la plupart des caméras fournissent des informations de couleur dans un système RGB mais il existe bien d'autres espaces de représentation (par exemple XY Z, les systèmes de couleurs opposées, L * u * v * , L * a * b * , HSV , HSI, et HSL) [START_REF] Trémeau | Image numérique couleur: De l'acquisition au traitement[END_REF][START_REF] Gertjan | Color in Computer Vision[END_REF]. Remarquons également les modèles de couleurs gaussiens qui correspondent à une nouvelle théorie de la mesure de la couleur basés sur la théorie espace-échelle dans le domaine spatio-spectrale [START_REF] Geusebroek | Color and scale: The spatial structure of color images[END_REF].

Contours et gradients.

Les dérivées de l'image sont des données essentielles pour décrire la structure locale des images et leurs applications en vision par ordinateur sont vastes : détection des contours, extraction de caractéristiques, flux optique, la segmentation d'image et la détection d'objet. Les opérateurs de gradient les plus populaires sont ceux de Sobel [START_REF] Parker | Algorithms for Image Processing and Computer Vision[END_REF] et de Canny [START_REF] Canny | A computational approach to edge detection[END_REF] et il existe plusieurs approches dédiées à la couleur : l'approche vectorielle de Di Zenzo [START_REF] Di | A note on the gradient of a multi-image[END_REF], l'approche de Carron [START_REF] Carron | Color edge detector using jointly hue, saturation and intensity[END_REF] qui fusionne les gradients de teinte, saturation et luminance en accordant une importance plus grande aux couleurs de forte saturation.

Texture. Classiquement, les bancs de filtres tels que celui de Gabor [START_REF] Fogel | Gabor filters as texture discriminator[END_REF] sont efficaces mais difficiles à utiliser dans des applications temps-réel. Récemment des méthodes plus économiques et tout aussi discriminantes telles que les Motifs Binaires Locaux LBP (pour Local Binary Patterns) [START_REF] Ojala | Performance evaluation of texture measures with classification based on kullback discrimination of distributions[END_REF] ont vu le jour. L'opérateur consiste à analyser un voisinage circulaire autour d'un pixel central. La valeur du pixel central est utilisée comme un seuil pour ses voisins. Si un voisin est de valeur supérieure, alors le code qui lui sera associé dans le LBP est 1, et sinon 0. Enfin, chaque bit dispose d'un poids en fonction de sa position dans le voisinage, et l'on peut en déduire une valeur décimale caractéristique de ce motif local. Il s'agit donc d'une représentation à la fois compacte et peu sensible aux changements de contraste. Après une simple normalisation, ils deviennent invariants aux changements de rotation.

Primitives locales et points d'intérêt. Il existe un nombre considérable de méthodes de détection et de description de points d'intérêt : Harris [START_REF] Shi | Good features to track[END_REF], SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF], SURF [START_REF] Bay | Surf: Speeded up robust features[END_REF] et FAST [START_REF] Rosten | Faster and better: A machine learning approach to corner detection[END_REF].

Information de mouvement. Dans le cas d'une caméra fixe, les informations de mouvement peuvent être extraites aisément pas soustraction de fond. Dans un cadre plus général, et en particulier lorsque la caméra est mobile, on s'intéressera plutôt à des approches de calcul de flot optique [START_REF] Horn | Determining optical flow[END_REF][START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF].

Information de profondeur. Enfin, avec l'émergence des capteurs RGB+D de type Kinect, il est désormais devenu aisé d'obtenir des cartes de profondeurs. Les méthodes alternatives, basées sur la stéréo-vision et la lumière structurée offrent par ailleurs des possibilités étendues dans certaines conditions d'acquisition, pour des distances capteur-objet supérieures à 5 mètres ou encore en extérieur.

Représentation d'un objet

À partir des informations disponibles au niveau pixel, différentes représentations plus globales sont possibles. L'objet peut être directement représenté par le bloc de pixels qui lui est associé dans l'image, où chaque pixel porte une information de luminance, de couleur (dans différents espaces de représentation) ou de profondeur par exemple. Ensuite, il peut être décrit par un histogramme, de couleur [START_REF] Gevers | Color-based object recognition[END_REF] ou de gradients orientés HOG [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] ou par son contour [START_REF] Kass | Snakes: Active contour models[END_REF] ou bien encore par une matrice de covariance de caractéristiques pixelliques [START_REF] Tuzel | Region covariance: A fast descriptor for detection and classification[END_REF]. Comparée aux représentations par histogrammes, ces derniers descripteurs sont de dimension plus faible et leur taille ne dépend pas de la taille initiale de l'objet dans l'image. De plus, ils peuvent être calculés rapidement par l'usage d'images intégrales. C'est sur cette représentation que s'appuie une grande partie de nos travaux de thèse.

Détection et mise en correspondance

La reconnaissance visuelle est un problème étroitement lié à l'apprentissage des catégories visuelles à partir d'un ensemble limité d'instances. Typiquement deux approches sont utilisées pour résoudre ce problème: l'apprentissage des catégories génériques et la ré-identification d'instances d'un objet particulier. Dans le dernier cas, il s'agit de reconnaître un objet en particulier: le portrait sur un magazine, une liste de personnes enregistrées sur une base de données, des monuments connus tel que la tour Eiffel. D'autre part la reconnaissance générique consiste à retrouver toutes les instances d'objets qui appartiennent à la même catégorie conceptuelle : toutes les voitures, les piétons, les oiseaux, etc. Pour la reconnaissance spécifique, les algorithmes visent à trouver des correspondances ou à faire du matching en utilisant des descripteurs propres à l'instance observée. La reconnaissance générique utilise des modèles statistiques de l'apparence et la forme des objets, afin de chercher dans de nouvelles images inconnues les zones de forte similarité avec le modèle statistique. Ce type d'apprentissage requiert l'accumulation d'un ensemble d'images d'entraînement pour extraire ou construire le mod-èle de chaque catégorie. Trouver les correspondances existantes entre deux (ou plusieurs) images n'est pas un problème aisé. Un objet peux générer deux instances complètement différentes en fonction des situations non contrôlées telles que l'illumination, le point de vue de la camera, la position de l'objet, les occultations et l'encombrement.

Les approches de classification les plus répandues dans la communauté de la vision par ordinateur sont les approches SVM (pour Support Vector Machine en anglais) et Adaboost. Dans le cadre de notre étude concernant l'utilisation des matrices de covariance, nous nous intéressons également à la classification dans les espaces Riemaniens. Des descriptions plus complètes aux SVM et à leur implémentation peut être trouvée dans [START_REF] Smola | Learning With Kernels: Support Vector Machines, Regularization, Optimization and Beyond[END_REF] et [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]. Dans le domaine de la vision par ordinateur, on se réfère très souvent aux travaux de [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] sur la classification piéton/non piéton par HOG (histogrammes de gradients orientés) associés aux SVM. Concernant Adaboost, les travaux précurseurs concernent la détection de visages en utilisant des ondelettes de Haar [START_REF] Viola | Robust real-time face detection[END_REF].

Enfin, la classification piéton/non piéton a également été effectué dans un espace Riemanien [START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF] en utilisant une représentation par matrices de covariance. L'approche est inspirée par l'algorithme LogitBoost [START_REF] Friedman | Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors)[END_REF] qui opère sur les espaces vectoriels.

Concernant les métriques utilisées, de nombreuses méthodes ont été proposées par la communauté de la vision par odinateur. Quatre approches ont particulièrement inspiré nos travaux.

Appariement de blocs ou Template matching. Il s'agit de comparer directement les images associées aux objets en utilisant de simples mesures, telles que l'erreur quadratique moyenne MSE (pour Mean Squared Error) , la corrélation normalisée croisés N CC (pour Normalized Cross-Correlation) ou encore l'indice de similarité de structure SSIM (pour Structural SIMilarity index). Ces approches s'avèrent plus adaptées pour mettre en correspondance ou suivre temporellement des zones de petite taille et des objets planaires dont le mouvement est rigide.

Le flot optique médian plus populairement connu sous le nom de Flock of Trackers F oT est une approche capable de suivre des objets non-rigides par analyse du flot optique obtenu par l'approche de suivi KLT (Kanade-Lucas-Tommasi) en utilisant l'opérateur médian pour estimer la direction de déplacement de l'objet. Quand une cible est détectée, son rectangle englobant est partitionné en quadrants. Chacun d'entre eux est suivi par KLT, les 50% de points les plus erronés sont écartés, et le reste participe au calcul du déplacement global. Cette méthode apporte une certaine robustesse par rapport aux occultations partielles. Elle est adaptée dans le cas d'objets rigides montrant des points saillants. Mean-Shift. Dans le cas du suivi Mean-Shift [START_REF] Comaniciu | Mean shift: A robust approach toward feature space analysis[END_REF], l'objet est modélisé par une représentation couleur-espace, et le suivi est fait par descente de gradient utilisant la distance de Bhattacharyya. L'histogramme est généralement quantifié afin d'assurer une exécution temps-réel et d'éviter les représentations creuses. Cette approche est particulièrement adaptée aux objets dont les couleurs sont caractéristiques par rapport à celles du reste de l'image. Elle permet une bonne résilience vis-à-vis des déformations de l'objet.

Appariement d'histogrammes et

Le suivi par covariance. L'objet à suivre est représenté par sa boîte englobante et en chaque point (x, y) de l'objet est calculé un vecteur de caractéristiques F (x, y). La combinaison originalement proposée par Tuzel et al. [START_REF] Tuzel | Region covariance: A fast descriptor for detection and classification[END_REF] continent la position des points (x, y), leur couleur (composantes RGB) et la norme des 1ère et 2nde dérivées d'intensité par rapport à x et y. Ainsi, chaque point est converti en un vecteur de dimension 9 :

F (x, y) = [
x y R(x, y) G(x, y) B(x, y) ∂I(x,y) ∂x ∂I(x,y) ∂y

∂ 2 I(x,y) ∂x 2 ∂ 2 I(x,y) ∂y 2
] T , L'objet est ensuite représenté par la matrice de covariance de taille 9 × 9 obtenue à partir de l'ensemble de ces vecteurs. Le suivi consiste alors à retrouver dans la nouvelle image la région dont la matrice de covariance est la plus similaire. Cela peut être fait par une recherche exhaustive [START_REF] Tuzel | Region covariance: A fast descriptor for detection and classification[END_REF] à la fois sur la position et sur l'échelle. Cette approche est performante pour suivre tout type d'objet, particulièrement lorsqu'il n'y a aucune information disponible sur la dynamique de l'objet ou que celui subit des mouvements erratiques entre deux images. Plusieurs approches ont également été proposées pour incorporer des données concernant la dynamique de l'objet et estimer le déplacement le plus probable.

Adaptation au contexte et commutation de méthode

Aucune des méthodes de suivi évoquées dans la partie 0.3 n'est applicable de manière universelle, chaque algorithme ayant ses avantages et ses limites. Les performances qu'ils offrent sont bien souvent dépendantes de l'application et il n'existe pas d'algorithme qui puisse être considéré comme supérieur aux autres. Quelques-uns des aspects à prendre en compte dans la définition d'un algorithme de suivi sont les suivants : la nature de la cible, les changements de conditions d'éclairage, les variations de pose ou d'apparence, les déformations non -rigides et des occlusions totales ou partielles.

Malheureusement, dans de nombreuses applications telles que la vidéo-surveillance, il n'y a aucune information préalable disponible sur la nature de la cible (rigide ou non-rigide) et l'estimation de ces informations à la volée peut poser un problème très difficile, c'est particulièrement vrai lorsque les objets sont détectés en utilisant des algorithmes de soustraction de fond au lieu d'un classificateur d'objets. Cette thèse soutient qu'une bonne méthode de suivi doit résulter de la combinaison de plusieurs approches, l'algorithme qui en résulte devant être capable de gérer une plus grande liste de difficultés que tous les algorithmes originaux tout en étant suffisamment rapide. L'objectif est d'obtenir un bon équilibre entre la robustesse et la précision tout en conservant des besoins en puissance de calcul aussi faibles que possible afin de maintenir l'exécution en temps réel. Ici trois approches sont proposées.

Adaptation aux changements de saturation

Principe. Au cours d'une séquence d'images il est possible de rencontrer de fortes variations de luminosité, qui ne sont pas toujours uniformes dans l'image. Il n'est donc pas aisé de définir au préalable les meilleures caractéristiques à utiliser : couleur ou luminance.

Le traitement couleur est généralement plus coûteux en temps de calcul tandis que la luminance est par essence sensible aux variations photométriques. La couleur permet de définir de précieux invariants couleur [START_REF] Gevers | Color-based object recognition[END_REF] qui sont malheureusement peu fiables dans le cas de faibles saturations. Nous proposons une approche de suivi qui sélectionne automatiquement les caractéristiques en fonction de leur pertinence : la luminance sera utilisée aux pixels pour lesquels la saturation de la couleur est faible et la couleur sera utilisée dans les autres cas. Cette approche a été validée dans le cadre du suivi de points de type KLT [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF][START_REF] Tomasi | Detection and tracking of point features[END_REF]. La composante couleur utilisée a été choisie de manière à être compacte et invariante aux changements d'illumination.

Choix de la composante couleur. Une façon directe de séparer l'information de chrominance (invariante) de celle de luminance est de normaliser (norme L 1 ) chaque composante (R, G, B) en les divisant par la somme des trois, qui correspond à l'intensité de la couleur. On obtient les composantes (r, g, b) qui ne dépendent plus que de l'albedo, propriété intrinsèque de réflectance du matériau.

À partir de ces trois composantes, nous calculons la valeur scalaire L 1 = max(r, g, b) qui représente la composante la plus saturée parmi les trois. Notons que cette information de couleur est plus facile à manipuler que les trois composantes et conserve l'invariance. Elle s'avère toutefois moins discriminante. Dans le cas du suivi KLT où la cohérence temporelle inter-image est pleinement exploitée, cet aspect a un impact relativement faible sur les résultats de suivi.

Mesure de pertinence de la couleur. L'approche est inspirée des travaux de [START_REF] Carron | Segmentation d'images couleur dans la base Teinte Luminance Saturation : approche numérique et symbolique[END_REF] concernant la détection de contours dans l'espace HSV en fusionnant les information de teinte, saturation et luminance à l'aide d'une fonction de pertinence dépendant de la saturation, typiquement une fonction sigmoïde. Dans notre cas, la fonction de coût utilisée dans l'approche de suivi par KLT est modifiée de manière à accorder une importance plus forte à L 1 lorsque la saturation est élevée et à la luminance sinon.

Résultats. Á partir d'expériences menées sur plusieurs séquences et en comparant les résultats de KLT appliqué dans différents espaces couleur, nous avons pu constater un très bon compromis entre le nombre de points correctement suivis et le temps d'exécution. Utilisé seul, l'invariant L 1 échoue lorsque la saturation est faible. La méthode proposant une association entre L 1 et la luminance I est capable de s'adapter à cette situation. Elle permet de maintenir les performances du suivi dans le cas de séquences saturées et améliore les résultats de manières significatives dans le cas de séquences peu saturées. D'autre part, utiliser les trois composantes (r, g, b) ne permet pas toujours de meilleurs résultats que L 1 mais augmente toujours, et de manière significative, les temps de calcul. Enfin, nous avons montré la pertinence du choix de L 1 comparé à la teinte pour l'ensemble des séquences de test. Ceci est probablement du au caractère bruité de cette composante.

Première coopération FoT + CT

Afin de répondre au problème du suivi d'objet sur de longues durées, nous proposons une approche fondée sur le suivi FoT évoqué dans la partie 0.3, qui analyse de manière robuste le flot optique associé aux points de l'objet. D'autre part nous utilisons le descripteur par covariance permettant de modéliser l'apparence globale (couleur et texture).

L'approche FoT estime de manière robuste le mouvement apparent de l'objet et permet théoriquement de résister aux occultations partielles jusqu'à 50% de masquage [START_REF] Wendel | Robustifying the flock of trackers[END_REF]. La restriction principale de cette méthode est que l'objet doit montrer un nombre suffisant de points saillants pour que le suivi par KLT soit fiable. Elle s'avère plus adaptée pour suivre des objets dont le mouvement est spatialement uniforme, autrement dit les objets rigides.

Le suivi par covariance CT apparaît comme une alternative qui de manière compacte modélise la cible par les corrélations au sein d'un ensemble d'attributs préalablement sélectionnés (communément il s'agit des coordonnées spatiales, de la couleur et de la couleur). Contrairement à FoT, de bonnes performances sont atteintes même pour des objets faiblement texturés ou pour des objets au mouvement non-rigide. Sa faiblesse principale est le temps d'exécution et ceci devient plus important lorsque la recherche de la nouvelle position de l'objet se fait de manière exhaustive dans l'image. Notons toutefois que CT peut être accélérée par un couplage avec d'autres méthodes de suivi telles que le filtrage particulaire [START_REF] Liu | Covariance Tracking via Geometric Particle Filtering[END_REF] qui permet de limiter l'espace de recherche.

Le mécanisme proposé est décrit graphiquement par la figure 34567. Tout d'abord, FoT est exécuté et l'analyse de la variance des vecteurs de mouvement permet d'identifier l'objet comme rigide ou non-rigide. S'il est rigide, FoT est exécuté tant que les résidus du suivi KLT (caractéristiques de l'erreur) restent inférieurs à une valeur seuil R max ou que la dissimilarité D t entre deux matrices de covariance successives est inférieure à une valeur seuil D max . Si l'objet est non-rigide ou que FoT a échoué (à cause d'une occultation par exemple), CT est exécuté pour retrouver l'objet pendant une période de temps T . Si l'objet n'est pas retrouvé après cette période il est considéré comme définitivement perdu.

Conclusion

Trois contributions ont été apportées en ce qui concerne le suivi adapté au contexte. Ces approches apportent l'un et/ou l'autre des avantages suivants :

1. La robustesse, soit par rapport aux occultations de l'objet, les mouvements abrupts, les changements de conditions d'acquisition.

2. La rapidité: dans la méthode de suivi adaptée aux changements de saturation, la couleur (qui requiert plus de temps de calcul) n'est utilisée que si nécessaire. Ensuite, les approches MS et FoT sont plus rapides mais moins robustes que CT seul. La combinaison de CT avec l'une des ces approches permet donc de réduire le temps d'exécution sans détériorer la qualité du suivi. Le suivi sera d'autant plus rapide que MS ou FoT seront appelées fréquemment. Cela correspond aux cas où la caméra est statique et que l'objet n'est pas occulté.

Table 1: Analyse de la procédure de coopération MS+CT. Ces résultats ont été obtenus sur un processeur Nehalem. Le tableau montre pour chaque séquence : la taille de l'objet suivi à l'initialisation ; le pourcentage d'images (%im) pour lequel les algorithmes MS et CT algorithmes sont exécutés ; le nombre total de cycles par pixel (cpp) passé par chaque partie de l'algorithme (MS, CT, mise à jour du modèle de covariance, calcul de la similarité) pendant toute la durée de la séquence.

Séquence

Suivi multiple temps-réel par covariance et ré-identification

L'un des objectifs de ce chapitre est de trouver la meilleure combinaison d'opérateurs de texture et de couleur qui améliorent à la fois la robustesse et le caractère distinctif du descripteur de covariance tout en gardant sa compacité et sa rapidité de calcul ainsi qu'une bonne robustesse vis-à vis des changements de couleur et d'illumination.

Ces matrices de covariance sont ensuite utilisées dans le cadre de la ré-identification d'objets et du suivi multiple. À cet effet, une méthode de suivi multi-objet originale est proposée, elle minimise une fonction d'énergie discrète combinant les probabilités d'association des trajectoires aux différentes cibles ainsi que leur concordance avec les modèles d'apparence décrits par des matrices de covariance.

Étude du choix des primitives texture

Plusieurs approches existent pour inclure des informations de texture au sein des matrices de covariance en vue d'améliorer leur pouvoir discriminant, citons GRMC (pour Gabor Region Covariance Matrix en anglais) utilisant les réponses issues des bancs de filtres de Gabor (voir partie 0.2.1) [START_REF] Pang | Gabor-based region covariance matrices for face recognition[END_REF][START_REF] Tou | Gabor filters as feature images for covariance matrix on texture classification problem[END_REF], ce qui mène à une matrice de covariance de taille relativement importante et donc à des temps de mise en correspondance prohibitifs. Certaines publications remplacent les filtres de Gabor par des LBP (voir partie 0.2.1) comme dans la méthode LBCM (pour Local Binary Covariance Matrix en anglais)

de [START_REF] Guo | Facial expression recognition using local binary covariance matrices[END_REF], le GLRCD (pour Gabor-LBP based Region Covariance Descriptor en anglais) [START_REF] Zhang | Gabor-LBP based region covariance descriptor for person re-identification[END_REF]). C'est également l'approche que nous avons privilégiée puisqu'elle répond à nos objectifs d'efficacité et de compacité.

Le choix de la meilleure façon de présenter l'information de texture sous la forme de motifs binaires locaux dans le descripteur de la matrice de covariance n'est pas évident à faire. GLRCD utilise une introduction brute des valeurs décimales des LBPs, valeurs très instables dans le cas des rotations. En outre, les opérations arithmétiques classiques ne sont plus applicables (ajouter ou sommer plusieurs valeurs décimales de LBP n'a pas de signification en termes de texture). LBCM utilise chaque bit dans le modèle de LBP comme un élément indépendant formant une chaîne de P bits. Ceci s'avère plus stable et il devient cohérent d'effectuer des opérations arithmétiques de façon indépendante pour chaque bit de la chaîne.

Le problème est que le nombre de fonctions à l'intérieur de la matrice de covariance augmente avec le nombre P de bits considérés pour le codage des LBPs. Cela a un impact significatif sur la vitesse d'exécution .

Dans ces travaux de thèse, nous proposons une nouvelle façon d'introduire les LBPs dans les matrices de covariance par un nouveau descripteur de covariance appelé ELBCM (pour Enhanced Local Binary Covariance Matrix). Il utilise les angles définis par les motifs LBP uiniformes présentés sur la figure 4-1. L'utilisation des formules trigonométriques est coûteuse en temps de calcul, mais ce problème est complètement résolu en utilisant une table de correspondance associant directement à chaque motif les valeurs de cosinus et sinus des angles θ 0 et θ 1 .

ELBCM présente de multiples avantages par rapport aux précédents descripteurs de texture à base de covariance. Il est plus compact (7 composantes contre 11 pour LBCM) et plus stable car moins affecté par les petites rotations, qui impactent seulement les angles θ 0 et θ 1 alors que pour LBCM les mêmes rotations affectent irrégulièrement la valeur des bits dans le vecteur descripteur LBP en fonction de leur position. Un avantage supplémentaire du descripteur de la ELBCM est que sa taille est complètement indépendante du nombre de P voisins utilisés dans le modèle LBP .

L'apport de ELBCM a été montré sur trois applications : la classification de texture, la classification d'expressions faciales, le suivi d'objets. Les résultats obtenus démontrent la généricité de cette approche et les bonnes performances tant en termes de temps d'exécution qu'au niveau du pouvoir discriminant de la représentation.

Étude des primitives couleur

Après l'étude sur la texture, nous nous intéressons au choix des composantes couleur les plus pertinentes à introduire dans la matrice de covariance en vue d'une mise en correspondance efficace c'est-à-dire précise (qui produit peu d'ambiguïtés d'appariement) et rapide. Quatre représentations couleur sont utilisées : RGB, HSV, les modèles de couleur gaussiens évoques dans la partie 0.2.1 et l'invariant L 1 introduit dans 0.4.1.

Deux applications sont visées : la classification de textures couleur et le suivi de visage dans le cas de conditions d'illumination variables. Les expérimentations réalisées dans le premier cas nous permettent de constater l'apport de l'approche ELBCM par rapport aux approches comparables. Dans la seconde application, seules les combinaisons utilisant la composante invariant L 1 = max(r, g, b) permet de résister aux forts changements d'illumination. Pour des conditions d'acquisition constantes, la méthode ELBCM offre de bons résultats pour la plupart des espaces couleur utilisés.

Ré-identification d'objets

Principes. La mise en correspondance est rendue difficile lorsque les objets doivent être re-détectés et ré-identifiés à partir d'une seconde caméra sous des conditions d'acquisitions différentes en termes d'illumination et de perspective, à des instants très différents. Le descripteur de l'objet doit alors être suffisamment caractéristique de l'objet. À cet effet, Bak et al. ont

proposé dans [START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF] un maillage dense de matrices de covariances [START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF] ainsi qu'une opération de moyenne appelé MRC (pour Mean Riemannian Covariance en anglais), qui permet de fusionner au cours du temps les informations d'apparences issues de ces multiples échantillons. En partant de cette approche, nous proposons de réduire le temps d'exécution par l'usage d'un nouvel arrangement de matrices MRC. Les régions d'intérêt associées aux objets sont redimensionnées (typiquement 96 × 128 pixels), puis des anneaux concentriques de rectangles sont positionnés autour du centre de l'objet, avec des aires qui augmentent de manière exponentielle pour permettre un recouvrement entre deux couches successives. Cela est inspiré des méthodes de FREAK et DAISY [START_REF] Alahi | FREAK: Fast Retina Keypoint[END_REF][START_REF] Tola | DAISY: An efficient dense descriptor applied to wide-baseline stereo[END_REF] mais pour des régions rectangulaires, permettant des calculs rapides grâce à la méthode d'image intégrale.

Résultats. Quelques résultats sont visibles sur la figure 4-19. Afin de valider notre approche de ré-identification nous utilisons les courbes CMC (pour Cumulative Matching Characteristic en anglais) utilisées également dans [START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF] et [START_REF] Gray | Evaluating appearance models for recognition, reacquisition, and tracking[END_REF]. Elles representent en ordonnée le pourcentage de fois où l'identité réelle apparaît parmi les n meilleurs appariements, avec n en abscisse. Ces tests sont réalisés sur les bases de données ETHZ [START_REF] Schwartz | Learning Discriminative Appearance-Based Models Using Partial Least Squares[END_REF] et PETS'09 L1-Walking-Sequence 1 [START_REF] Ferryman | Pets2009: Dataset and challenge[END_REF]. Les taux de ré-identification obtenus sont comparables à ceux rapportés dans [START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF] mais en utilisant 75% de matrices de covariance en moins et des matrices plus compactes.

Suivi multiple

Le but de notre algorithme de suivi multi-objet est de détecter, d'identifier et de tracer les positions des objets. Idéalement le nombre de trajectoires par objet de un exactement, mais les inter-occlusions, les disparitions, l'apparition de faux positifs de la méthode de détection peut provoquer des dérives et des fragmentations des trajectoires. La possibilité d'inclure des informations relatives à l'apparence dans la fonction d'énergie a été laissée ouverte dans [START_REF] Andriyenko | Multi-target tracking by continuous energy minimization[END_REF]. Cette idée est récupérée ici et enrichie en efficacité et en robustesse grâce à la représentation des cibles par les matrices de covariance.

Image par image l'algorithme évalue toutes les correspondances possibles entre les détections d i et les trajectoires p j . De nouvelles pistes sont créées lorsque cela est nécessaire. Finalement, l'algorithme génère une cartographie dynamique contenant l'ensemble couples M T = {(d i , p j )} qui minimise une fonction d'énergie discrète. Cette fonction d'énergie E(d i , textbf p j ) privilégie les configurations plausibles et pénalise celles qui sont incohérentes. Notre fonctionnelle d'énergie comprend la plupart des termes utilisés dans [START_REF] Andriyenko | Multi-target tracking by continuous energy minimization[END_REF] : un terme de détection fondée sur la détection de données

Une de nos contributions est de rajouter le terme d'énergie associé à l'apparence, celle-ci étant modélisée par les matrices de covariance. Ces matrices appartient à un espace Riemanien formé par l'ensemble des matrices définies positives (SPD).

Une variété de métriques et de dissimilarités a été proposée dans la litérature afin de comparer deux matrices dans cet espace.

Parmi elles, la mesure de dissimilarité Jensen-Bregman LogDet Divergence [START_REF] Cherian | Efficient similarity search for covariance matrices via the jensen-bregman logdet divergence[END_REF] s'avère particulièrement intéressante car elle est rapide à calculer et elle jouit de très bonnes propriétés de robustesse. Concernant la minimisation de (4.11), nous utilisons l'algorithm de Munkres [START_REF] Munkres | Algorithms for the assignment and transportation problems[END_REF] (la méthode Hongroise) utilisée par ailleurs dans [START_REF] Jungling | Detection and tracking of objects with direct integration of perception and expectation[END_REF].

Résultats. Nous avons évalué l'algorithme en utilisant les métriques proposées dans [START_REF] Stiefelhagen | The CLEAR 2006 evaluation[END_REF] et [START_REF] Li | Learning to associate: Hybridboosted multi-target tracker for crowded scene[END_REF] et en partant de bases de données très largement utilisées dans la communauté ainsi que des séquences enregistrées dans notre laboratoire en utilisant le premier point de vue : la base PETS 2009-S2L1-V11 , la base TUD2 avec les séquences TUD-Campus, TUD-Crossings et TUD-Stadtmitte. La vérité-terrain est disponible pour toutes ces bases de données. Nos résultats sont visible en-ligne 3 . Sur la base de ces expériences, les résultats montrent que notre approche de suivi n'est pas aussi précise que celles proposées par [START_REF] Milan | Continuous energy minimization for multi-target tracking[END_REF] ou [START_REF] Berclaz | Multiple object tracking using k-shortest paths optimization[END_REF], mais elle a l'avantage important de travailler à la volée, en estimant les trajectoires image par image. Dans la méthode proposée dans [START_REF] Milan | Continuous energy minimization for multi-target tracking[END_REF], le suivi nécessite l'ensemble des images de la vidéo pour estimer le nombre de trajectoires optimal et leur tracé le plus probable.

Bilan

Dans cette partie, une étude a été menée pour proposer un nouveau descripteur par matrice de covariance en proposant une

Implementation temps-réel: optimisations logicielles et transformations algorithmiques

Dans cette partie nous décrivons les transformations et accélérations opérées sur l'algorithme de suivi par covariance.

Notons que ce travail a également été mené dans le cas du suivi Mean-Shift en vue de faire coopérer les deux méthodes [START_REF] Laguzet | Color tracking with contextual switching: Real-time implementation on CPU[END_REF].

Afin d'évaluer les performances des algorithmes et l'impact des optimisations, des comparaisons ont été faites sur trois générations de processeurs : Penryn/Yorfield, Nehalem/Bloomfield et Sandy-Bridge qui sont tous des processeurs multicoeurs, excepté le Nehalem. Le processeur du Sandy-Bridge a la possibilité d'être over-clocké. La mémoire RAM peut être plus over-clockée que le processeur ce qui signifie qu'à une fréquence plus élevée (4.4 GHz) la RAM semble plus rapide ce qui résulte en un nombre réduits de nombre de cycles.

Deux stratégies peuvent être exploitées pour optimiser CT . La première consiste à réaliser du multi-threading en parallélisant la boucle principale de traitement. Cela est effectué avec OpenMP. La seconde nécessite une transformation de la mise en forme des données en mémoire SdM→MdS (Structure de Matrices vers Matrices de structures). Le but de cette manipulation est de transformer un ensemble de matrices indépendantes en une matrice unique, où chaque cellule combine les éléments de toutes les matrices dans une structure de données. La contribution d'une telle transformation est de tirer parti de la performance du cache en exploitant la localité spatiale et temporelle. L'algorithme de suivi par covariance se compose de trois parties :

2. le calcul de l'image intégrale des primitives, 3. le calcul de l'image intégrale des produits.

Le produit de caractéristiques et de sa transformation sont décrits dans les algorithmes 16 et 17. Les notations utilisées sont les suivantes :

• h et w : hauteur et largeur de l'image

• n F : nombre de caractéristiques (primitives) image

• n P : nombre de produits de caractéristiques,

n P = n F (n F + 1)/2,
• F : cube (SdM) ou matrice (MdS) de caractéristiques,

• P : cube (SdM) ou matrice (MdS) de produits de caractéristiques,

• I F et I P : deux cubes (ou matrices) d'images intégrales (à partir de F ou P ).

Grâce à la commutativité de la multiplication, la moitié seulement des produits doivent être calculés (la boucle sur k 2 commence à k 1 , ligne 3). Comme les deux dernières étapes sont similaires, nous ne présentons qu'une version générique du calcul de l'image intégrale (Algo. 18) et sa transformation (Algo. [START_REF] Bouguet | Pyramidal implementation of the lucas kanade feature tracker description of the algorithm[END_REF].

Algorithm 1:

Optimisation ode CT -version SdM du produit de caractéristiques.

1 k ← 0 2 foreach k 1 ∈ [0..n F -1] do 3 foreach k 2 ∈ [k 1 ..n F -1] do 4 [ point-to-point multiplication ] 5 foreach i ∈ [0..h -1] do 6 foreach j ∈ [0..w -1] do 7 P [k][i][j] ← F [k 1 ][i][j] × F [k 2 ][i][j] 8 k ← k + 1
Algorithm 2: Optimisation de CT -version MdS du produit de caractéristiques.

1 foreach i ∈ [0..h -1] do 2 foreach j ∈ [0..w -1] do 3 k ← 0 4 foreach k 1 ∈ [0..n F -1] do 5 foreach k 2 ∈ [k 1 ..n F -1] do 6 P [i][j × n P + k] ← F [i][j × n P + k] × F [i][j × n P + k] 7 k ← k + 1
Une fois que cette transformation est faite, nous utilisons les instructions SIMD dans les différentes parties de l'algorithme.

Pour ce qui concerne le produit, les deux boucles internes à k 1 et k 2 sont totalement déroulée de manière à montrer la liste de toutes les multiplications et la liste des vecteurs à construire par des instructions de permutations 4 . Par exemple, pour Algorithm 3: Optimisation de CT -version SdM de l'image intégrale.

1 foreach k ∈ [0..n -1] do 2 [classical in place integral image] 3 foreach i ∈ [0..h -1] do 4 foreach j ∈ [0..w -1] do 5 I[k][i][j] ← I[k][i][j] + I[k][i][j -1] + I[k][i -1][j] -I[k][i -1][j -1]
Algorithm 4: Optimisation de CT -version MdS de l'image intégrale.

1 foreach i ∈ [0..h -1] do 2 foreach j ∈ [0..w -1] do 3 foreach k ∈ [0..n -1] do 4 I[i][j × n + k] ← I[i][j × n + k] + I[i][(j -1) × n + k] + I[i -1][j × n + k] -I[k][i -1][(j -1) × n + k]
une valeur typique de n F = 7, il y a n P = 28 produits, elles sont effectuées dans l'ordre suivant (les nombres entre crochets sont les indices des caractéristiques) : 5,[START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF][START_REF] Andriyenko | Multi-target tracking by continuous energy minimization[END_REF]4]; [START_REF] Andriyenko | Multi-target tracking by continuous energy minimization[END_REF][START_REF] Andriyenko | Multi-target tracking by continuous energy minimization[END_REF]4,4] × [5,[START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF]4,5]; [4,5,5,[START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF]] × [START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF]5,[START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF][START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF].

[0, 0, 0, 0] × [0, 1, 2, 3]; [0, 0, 0, 1] × [4, 5, 6, 1]; [1, 1, 1, 1] × [2, 3, 4, 5]; [1, 2, 2, 2] × [6, 2, 3, 4]; [2, 2, 3, 3] × [
Dans ce cas, le septième vecteur est rempli à 100 %, mais il deviendra sous-optimal si n P n'est pas divisible par le cardi-

Introduction

With the development of optics, electronics and computers, it became possible for science to capture, store and process great amounts of spatial information in the form of bi-dimensional or higher dimensional arrays. These technologies pushed scientists to think on the possibility of automatizing and emulating the visual perception abilities of animals and human beings. Most computer vision efforts are trying to automatize the ability to recognize objects, scenes and categories. An important variety of potential applications have been imagined e.g., content-based image search, video data mining, object identification for mobile robots. Some of them already exist in a limited form e.g., OCR characters, car plates and finger print recognition.

Visual recognition is the problem of learning visual categories from a limited set of samples and identifying new instances of those categories, the problem is often separated into two types: the specific case and the generic category case.

In the specific case the objective is to identify instances of a particular object, place or person e.g., the portrait of a magazine, people registered in a database and popular monuments such as the Eiffel tower. Whereas in the generic category case we seek to recognize different instances that belong to the same conceptual class e.g., cars, pedestrians, road signs and mugs. Visual object matching and modeling is challenging because the instances of the same object can appear very different due to many often uncontrollable variables such as illumination conditions, camera viewpoint, object pose, partial occlusions and clutter from the background. Generic category recognition can be even more challenging as different object instances from the same category often exhibit significant variations in appearance.

This thesis proposes a computer vision system for detecting and tracking multiple targets in videos.

The covariance matching method, as defined by Porikli et al. [START_REF] Porikli | Covariance tracking using model update based on lie algebra[END_REF], is the guiding thread of our work because it offers a compact representation of the target by embedding heterogeneous features in a elegant way. Therefore, it is efficient both for tracking and recognition.

Four categories of contributions are proposed.

The first one deals with the adaptation to a changing context, following two aspects. A preliminary work consists in the adaptation of color according to lighting variations and relevance of the color. Then, literature shows a wide variety of tracking methods, which have both advantages and limitations, depending on the object to track and the context. Here, a deterministic method is developed to automatically adapt the tracking method to the context through the cooperation of two complementary techniques. A first proposition combines covariance matching for modeling characteristics texturecolor information with optical flow (KLT) of a set of points uniformly distributed on the object [START_REF] Romero | Feature points tracking adaptive to saturation[END_REF]. A second technique associates covariance and Mean-Shift. In both cases, the cooperation allows a good robustness of the tracking whatever the nature of the target, while reducing the global execution times [START_REF] Laguzet | Color tracking with contextual switching: Real-time implementation on CPU[END_REF].

The second contribution is the definition of descriptors both discriminative and compact to be included in the target representation. To improve the ability of visual recognition of descriptors two approaches are proposed. The first is an adaptation operators (LBP to Local Binary Patterns ) for inclusion in the covariance matrices . This method is called ELBCM for Enhanced Local Binary Covariance Matrices [START_REF] Romero | Enhanced Local Binary Covariance Matrices ELBCM for texture analysis and object tracking[END_REF]. The second approach is based on the analysis of different spaces and color invariants to obtain a descriptor which is discriminating and robust to illumination changes. The various experiments implemented in tracking and recognition (texture , faces , pedestrians ) show very promising results.

The third contribution addresses the problem of multi-target tracking, the difficulties of which are the matching ambiguities, the occlusions, the merging and division of trajectories. We also propose the re-identification of targets using a set of spatially adapted covariance descriptors and minimizing a function of discrete energy that takes into account the kinematic behavior of the whole objects and model their appearance [START_REF] Romero | Total bregman divergence for multiple object tracking[END_REF].

Finally to speed algorithms and provide a usable quick solution in embedded applications this thesis proposes a series of optimizations to accelerate the matching using covariance matrices. Data layout transformations, vectorizing the calculations (using SIMD instructions) and some loop transformations had made possible the real-time execution of the algorithm not only on Intel classic but also on embedded platforms (ARM Cortex A9 and Intel U9300) [START_REF] Romero | Real-time covariance tracking algorithm for embedded systems[END_REF].

The first two chapters of this thesis provide an overall panorama of the state of the art techniques, the following two chapters describe all the contributions to the fields of image processing and computer vision. The last chapter is consecrated to the optimization and acceleration of the covariance tracking algorithm.

Chapter 1 covers some of the most common local pixel information operators used to detect and analyze abrupt and transitional changes (e.g., color, gradients, edges, corners, contours) for a variety of image pattern recognition applications such as image segmentation, key-point detection/matching and optical flow. All these techniques are the base of higher-level computer vision algorithms that contemplate the target shape, color and texture patterns to perform object detection,recognition and tracking. This thesis is primarily focused on the improvement of the covariance region descriptor.

Some of the state-of-the-art techniques for the detection and matching of objects in video sequences are introduced in Chapter 2. The detection part of the chapter is composed by very brief introductions to some of the most important classifiers: Support Vector Machines (SVM's), AdaBoost and classification with Riemannian manifolds (the type of topological spaces where covariance matrices reside). The second part of the chapter is devoted to matching, the difference here is that the objective is not to find a generic category of objects but to match particular instances based on their appearance.

Chapter 3 is dedicated to the long-term tracking problem. It explores some of the situations where a particular tracking method fails while others succeed. A robust algorithm could result from the combination of an ensemble of methods if a mechanism is employed to find an agreement between the algorithms so that the successful ones compensate the failures of the others. Different approaches can be used to combine a set of tracking algorithms: a competition scheme selects the best output from the set of tracking method responses; a cooperation scheme structured in the form of a cascade applies first the least expensive and less accurate one which finds a cheap but rough estimate of the target location, the intermediate results are then refined by more precise (but consuming) methods. A different alternative is to analyze the context and select the algorithm that best adapts to the situation. This contextual switching approach is thoroughly explored in this chapter.

To finalize the contributions to the field of computer vision Chapter 4 presents a series of modifications to increase the discriminant power of the covariance descriptor by integrating some textural and color operators. Textural information is incorporated by means of local binary patterns (LBP). Color is processed using invariants for brightness and saturation changes. Evaluations of these improvements for object tracking and other related problems such as texture analysis and facial gesture recognition are provided. Other improvements presented in this chapter are dedicated to the multiple object tracking and description problem. Two methods are proposed to face this problem: a pedestrian re-identification method based on an array (or grid) of covariance matrices, and a heuristic for the minimization of a discrete energy function that measures the complexity in the set of possible trajectories measuring its curvature, speed and appearance. The appearance similarity measure used to compare a pair of covariance matrices is the total Bregman divergence which is very efficient and fast to compute. This method implies a temporal coherency within the frames allowing the evaluation and fitting of curves that trace the target trajectories. The grid of covariance matrices is suitable for recognizing targets captured in a multiple-camera configuration, while the trajectory and appearance modeling of the second approach is best adapted for identifying the target trajectories on a single camera (preferably a fixed one) using a tracking by detection strategy. As mentioned above, Chapter 5 discusses the accelerations techniques proposed to implement the algorithms of this thesis in real-time on Intel desktop processors and other architectures such as the ARM Cortex 9. These methods are discussed in depth, proposing dedicated methods to increment the degree of parallelism of the algorithm mostly on the data level using data layout transformations together with SIMD instructions. At the end some multi-threading techniques are evaluated.

The discussion about the global results and perspectives evoked by this work are discussed in final conclusions chapter.

The first two chapters of this thesis provide an overall panorama of the state of the art techniques, the following two chapters describe all the contributions to the fields of image processing and computer vision. The last chapter is consecrated to the optimization and acceleration of the covariance tracking algorithm.

Chapter 1 covers some of the most common local pixel information operators used to detect and analyze abrupt and transitional changes (e.g., color, gradients, edges, corners, contours) for a variety of image pattern recognition applications such as image segmentation, key-point detection/matching and optical flow. All these techniques are the base of higherlevel computer vision algorithms that contemplate the target shape, color and texture patterns to perform object detection, recognition and tracking.

Some of the state-of-the-art techniques for the detection and matching of objects in video sequences are introduced in Chapter 2. The detection part of the chapter is composed by very brief introductions to some of the most important classifiers: Support Vector Machines (SVM's), AdaBoost and classification with Riemannian manifolds (the type of topological spaces where covariance matrices reside). The second part of the chapter is devoted to matching, the difference here is that the objective is not to find a generic category of objects but to match particular instances based on their appearance.

Chapter 3 is dedicated to the long-term tracking problem. It explores some of the situations where a particular tracking method fails while others succeed. A robust algorithm could result from the combination of an ensemble of methods if a mechanism is employed to find an agreement between the algorithms so that the successful ones compensate the failures of the others. Different approaches can be used to combine a set of tracking algorithms: a competition scheme selects the best output from the set of tracking method responses; a cooperation scheme structured in the form of a cascade applies first the least expensive and less accurate one which finds a cheap but rough estimate of the target location, the intermediate results are then refined by more precise (but consuming) methods. A different alternative is to analyze the context and select the algorithm that best adapts to the situation. This contextual switching approach is thoroughly explored in this chapter.

To finalize the contributions to the field of computer vision Chapter 4 presents a series of modifications to increase the discriminant power of the covariance descriptor by integrating some textural and color operators. Textural information is incorporated by means of local binary patterns (LBP). Color is processed using invariants for brightness and saturation changes. Evaluations of these improvements for object tracking and other related problems such as texture analysis and facial gesture recognition are provided. Other improvements presented in this chapter are dedicated to the multiple object tracking and description problem. Two methods are proposed to face this problem: a pedestrian re-identification method based on an array (or grid) of covariance matrices, and a heuristic for the minimization of a discrete energy function that measures the complexity in the set of possible trajectories measuring its curvature, speed and appearance. The appearance similarity measure used to compare a pair of covariance matrices is the total Bregman divergence which is very efficient and fast to compute. This method implies a temporal coherency within the frames allowing the evaluation and fitting of curves that trace the target trajectories. The grid of covariance matrices is suitable for recognizing targets captured in a multiple-camera configuration, while the trajectory and appearance modeling of the second approach is best adapted for identifying the target trajectories on a single camera (preferably a fixed one) using a tracking by detection strategy

As mentioned above, Chapter 5 discusses the accelerations techniques proposed to implement the algorithms of this thesis in real-time on Intel desktop processors and other architectures such as the ARM Cortex 9. These methods are discussed in depth, proposing dedicated methods to increment the degree of parallelism of the algorithm mostly on the data level using data layout transformations together with SIMD instructions. At the end some multi-threading techniques are evaluated. The discussion about the global results and perspectives evoked by this work are discussed in final conclusions chapter.

Introduction

Computer vision is the field of signal processing that studies the information contained by bi-dimensional (or higherdimensional) arrays of visual information looking for an interpretation expressed by decisions. In its origins in the 1960's, the objective of computer vision was to imitate the human vision and replicate it on the machines so that they could observe the scenes in the same way that humans do. But other domains have emerged since then using sensors of different nature than humans eyes, this is the case of microscopic images, medical imaging such as x-rays and magnetic resonance (MRI) scans, so, it is worth noting that by images we refer not only to the traditional photography formed by light in the human visible spectra (colors from red to violet), but to a broader category of images that includes any type of spatial evolving physical measurement can be treated: electromagnetic waves (light), acoustic waves (ultrasound imaging), magnetic resonance, etc.

At the beginning of the processing chain what computers only know is the image dimensions (width and height) and the type of spatial data stored in the form of pixels (e.g., surface brightness, color, magnetic resonance intensity). Computer vision operates at different levels of abstraction discovering meaning inside these large arrays of numbers by comparing pixels with their neighbors with simple arithmetic operations like numerical differentiation, measuring interesting properties such as the direction of steepest change (i.e., gradient vector) and its strength (contrast). Gradients are helpful to explore interesting local properties such as saliency which defines when some points are unique and deserve more interest.

If these points or regions are stable enough their location on novel images taken at different instants or perspectives should be easy to establish. Following this concept, objects are described as clouds of connected interest points. Color provides a complementary source of information to gradients and texture, it allows us to recognize non-rigid (animated) objects such as animals and people by their characteristic colors appearances (given by their skin, hair, plumages or clothes). Furthermore, a variety of color invariants and constancy methods exist that allow us to handle illumination changes better for color than for brightness images. Because both sources of information are complementary, it is possible to use them together in a variety of applications such as image segmentation where the objective is to subdivide an image into multiple regions of homogeneous properties.

The evolution of images over time is also an essential source of information, pixels (or regions of them) moving in similar directions can be clustered together and separated from the rest. To detect changes over time, some methods do simple image subtraction (supposing a static camera configuration), others construct a statistical distribution explaining each individual pixel (or regions of them) by their distribution of brightness or color (using mixtures of Gaussian models or histograms). In any case, when pixel values are far from the modeled ones this can be an indicator of the presence of movement. This principle is widely used to detect moving objects in tracking and activity recognition applications. Motion images is also used for optical flow analysis which allows to describe patterns of visual motion related to objects, surfaces and edges. Higher-level applications such as shape from motion or camera stabilization.

Pattern analysis is often exploited to identify the presence of an object or an interesting property. Patterns are always associated to a descriptor used for comparing and matching. The quality of a pattern descriptor is measured in terms of three properties: compactness, distinctiveness and repeatability.

This chapter explores some of the very basic image processing techniques frequently used in computer vision to identify interesting points/features on images, how to build descriptors of them and the diversity of object representations currently used in the state of the art. A variety of topics related to the nature of the information contained in pixels (e.g., brightness, color, motion and depth) and their neighbors is explored in the first section:

• Filter operators measuring inter-pixel relations.

• Histograms for representing distributions of pixel values: brightness, colors, gradients, etc.

• Textures, filter-banks, wavelets, local binary patterns and their statistical methods.

• Local features detectors (e.g., Harris corners, SIFT and SURF points) and their descriptors.

Pixel information

Individual pixels store very few information. It is unconceivable to assign an interpretation to large arrays of data (contain-

ing

Brightness and color images

Natural images are formed by the physical interaction of three fundamental processes: illumination, material reflection and detection/observation. Depending on theses three elements a final gray-scale or color image is formed. Color and luminance images both are the result of the interaction of light, materials and the observers sensitivity.

The whole process starts with light illuminating the visual scene 1 . Many different attributes characterize a light source the radiation pattern and the spectra of photons over the wavelengths. When the proportion of short-wavelength photons in the emitted light is bigger than long-wavelength ones light looks bluish. In contrast, when more photons of long wavelengths are emitted, the color is reddish. Candle lights and halogen lamps emit smooth spectra that can be easily and uniquely characterized by a single number referred to as temperature of radiation.

Materials are the second aspect to think of in the process of image formation. Depending on their spectral reflectance, their geometry and their rugosity, materials modify the light beam spectrally (by absorbing some wavelengths) and geometrically by diffusion, specular reflection or refraction for instance.

The third process in image formation is how light is observed or recorded by a camera or the eye integration the photons energy over a certain bandwidth, spatial area and for a period of time. In the eye, the spectral integration is performed grouping the visual spectrum into three spectral broadbands (i.e., short, middle and long-range wavelengths).

The integration time takes around 50 ms and it is more sensible (acute) at the central area of the retina (the fovea) and less sensible in the periphery. Color cameras mimic this temporal and spectral characteristics of the eye, recording three color bands in about the same period of time. In most of the cases spatial resolution is uniform over the whole image, with a resolution in the range of mega-pixels. 1 An electromagnetic radiation of a certain intensity composed by photons containing energy of certain wavelengths and traveling in a certain direction.

Brightness images

Surface brightness depends on how much incident light arrives at that patch and on the fraction of the incident light that gets reflected. A brightness pixel is influenced by three major phenomena: the camera response to light, the fraction of light reflected from the surface to the camera, and the amount of light coming to the surface. If X is a point in the space that projects to the point x in the image, and I patch (X) is the intensity emmitted by the surface at X and I camera (x) is the camera response at x, our model is

I camera (x) = kI patch (X) (1.1)
where k is a constant which in practice can be determined by calibration. Modern cameras have linear responses within a reasonable intensity range but pronounced non-linearities are expected for darker and brighter illumination conditions.

The amount of light received by a patch depends on the intensity of the light source (luminaire) and on the geometry of the scene. The amount of light depends on the angle formed by the surface normal and the source light as those patches facing the light collect more radiation than tilted away surfaces. Surfaces reflect light by a process of diffuse reflection which scatters light evenly across all the directions leaving the surface, as such, the perceived brightness of a surface should not depend on the viewing direction. Some fraction of light is usually absorbed by the surface, this absorption is independent from the incident light direction, a parameter which determines how much incoming light is reflected is the albedo and it represents the fraction of light which is reflected, diffuse surfaces are also known as Lambertian.

As mirrors are not diffuse, what an observer sees depends on the direction at which it looks to the mirror, this is phenomenon is referred as specular reflections.

More formally, the brightness of a pixel in a Lambertian surface (where light is equally reflected in all directions) is modeled by the amount of light a surface collects, this quantity is higher for surfaces facing the source light than for surfaces oriented along the direction in which rays travel. Mathematically, this is expressed by the cosine of the angle θ formed by the illumination ray and the surface normal,

I = ρI 0 cosθ, (1.2)
where I 0 is the intensity of the light source, θ is the angle between the source direction and the surface normal, and ρ is the diffuse albedo. More complete models consider surfaces having diffuse and specular reflections, this is the lamber-tian+specular model, assuming an infinitely distant light source, we write N(x) for the surface normal at x, S is a vector pointing from x towards the source with length I 0 (source intensity), ρ(x) is the albedo at x, and V is(S, x) is a function In reality reflections are not purely Lambertian neither specular, and light is concentrated and diffused in some directions around the specular reflection direction. Image taken from [START_REF] Forsyth | Computer Vision: A Modern Approach[END_REF].

that is 1 when x can see the source (when there is line of view) and zero otherwise. The intensity at x is

I(x) Image intensity = ρ(x)(N • S)V is(S, x) Diffuse term + ρ(x)A Ambient term + M Specular term
.

Interesting properties about the scene can be observed directly from brightness images e.g., specularities are a source of information about the surface shape, the albedo is a inherent property of a surface rather than a property of the picture of it. Surface properties that do not change when image circumstances do are called intrinsic properties. The problem is that different albedo surfaces under different illuminations may report the same pixel intensity. Lightness constancy is a desirable property of a vision system, because it allows it to compensate changes in the intensity of illumination and accurately report the lightness of a surface (whether it is white, gray or dark). In the second row a graph displays the relative spectral sensitivity of the three cone types (figure taken from [START_REF] Gertjan | Color in Computer Vision[END_REF]).

Color images

Human vision is basically trichromatic: it is based on the responses of three different cone photo-receptors types located in our eyes retina. Ganglion cells in our retina combine the responses arising from these cones to form three opponent channels: an achromatic one (black and white) and two chromatic ones (red-green and yellow-blue). These information is sent in the form of pulse-like signals to the visual cortex, where perception of color takes place. However, the most abundant light-sensitive cells are the rods which are not sensitive to color and generate luminance intensity signals only.

Each retina holds about one hundred million of photo-receptors and about 95% of them are rods. Depending on the light intensity level our vision adapts and works in three different states: scotopic, mesopic and photopic. Pure scotopic vision works in the light-dark dimension only, it is activated at very low light levels (< 0.01cd/m 2 ). Mesopic vision corresponds to intermediate light levels (0.01 -1cd/m 2 ) in this state both rods and cones are active but color discrimination is very poor. Truly color discrimination is possible at light levels above (1cd/m 2 ) when our vision becomes photopic. In the retina there are three different photo-receptors: L-cones, M-cones and S-cones. Figure 1-3 represents the spectral sensitivities of each type of cone. It is worth noticing that the L and M-cones wavelength responses are largely overlapping while S-cones wavelength response is somehow isolated. At each wavelength there exists a unique combination of L, M and S sensitivities. The resulting response corresponds to an integral of the wavelength-by-wavelength product of the light spectrum arriving to the eye and the spectral sensitivity around the spectral window, resulting in a tuple formed by the three cone-type integrals. The perceived color is determined relatively to this 3-valued tuple, but not exclusively, as the visual systems also makes spatial comparisons, making the final perceived color dependent on the neighbor colors as well.

Besides modeling the reflections, computer methods need also to quantize the color information and represent it numerically. Any arbitrary numbering scheme could do the job, as long as each number uniquely defines a color. Colors could be represented by ordering sequentially all the colors as they appear in the rainbow, going from the deep red till the deep violet. For historical reasons, most of the cameras yield color information in an RGB scheme. But color information can be mapped to alternative schemes or color-spaces (e.g. XYZ, RGB, oponent color systems, L*u*v*, L*a*b*, HSV, HSI, and HSL) which might appreciate better certain properties of color information than other schemes. For example, the hue-saturation-value model (HSV ) decomposes the RGB values into an orthogonal coloring scheme, decoupling the intensity information from the chromatic information. Many computer vision tasks such as image segmentation and object recognition require stable and repeatable color measurements. To achieve this purpose, color invariant is a basic requirement. The desired color invariance is obtained by color transformations at a pixel level. Many of these transformations are non-linear and tend to intensify the amount of noise i.e. a small perturbation of the RGB values causes a large jump in the transformed values. Thus, it is necessary to study how noise amplification propagates in RGB values to propose stable algorithms adapted to the color information at hand. Many luminance-based algorithms can be ported to the color domain, but color image processing methods should avoid the introductions of false chromaticities, in differential-based algorithms, the derivatives of the separate channels must be combined without loss of derivative information. The simple application of existing luminance-based operators on the separate color channels, and the subsequent combination fail because the influence of undesired artifacts. Computation of color gradients is performed better by color-tensor-based techniques [START_REF] Van De Weijer | Robust photometric invariant features from the color tensor[END_REF] for example. The influence of color in perception and computer vision is huge, but this is still a developing field, and many questions and problems are still waiting a solution.

There are many color representations and models in literature, as in [START_REF] Trémeau | Image numérique couleur: De l'acquisition au traitement[END_REF] and [START_REF] Gertjan | Color in Computer Vision[END_REF]. A different theory of color measurement called Gaussian color models is introduced here.

Gaussian color models

Image formation implies physical measurements over the spectral, spatial and time dimensions. Gaussian color models are a new theory of color measurement, it is an extension of the Gaussian derivative framework (scale-space theory) to the spatio-spectral domain [START_REF] Geusebroek | Color and scale: The spatial structure of color images[END_REF]. One of the most important achievements in scale-space theory is the remark that Gaussian shapes prevent the creation of extra details on higher scale images (i.e. lower resolution). Gaussians offer a general probe for spatio-spectral differential quotients. Let E(λ) be the spectral energy distribution of light, it is a function of λ, which denotes the wavelength. Let now G(λ 0 , σ λ ) be a Gaussian kernel of spectral scale σ λ and positioned at λ 0 . The spectral energy distribution E(λ) may be approximated by the Taylor expansion at λ 0

E(λ) = E λ0 + λE λ0 λ + 1 2 λ 2 E λ0 λλ + • • • (1.3)
Assuming measurements of the spectral energy distribution are made with a weighted integration over the spectrum we have

E σ λ = E λ0,σ λ + λE λ0,σ λ λ + 1 2 λ 2 E λ0,σ λ λλ + • • • (1.4)
where E λ0,σ λ = ∫ E(λ)G(λ; λ 0 , σ λ )dλ measures the spectral intensity. Gaussian color models measure the the coefficients E λ0,σ λ ,E λ0,σ λ λ and E λ0,σ λ λλ in the Taylor expansion of the Gaussian weighted spectral energy distribution at λ 0 with the spectral aperture (scale) σ λ .

Spatial information can be introduced to the model yielding a Taylor expansion at wavelength λ 0 and coordinates 400 450 500 550 600 650 700 400 450 500 550 600 650 700 400 450 500 550 600 650 700 mm mm mm x = (x, y). The result of this is the exploration of an energy density volume in a spatio-spectral space

E(λ, x) = E +   x λ   T   E x E λ   + 1 2   x λ   T   E xx E xλ E λx E λλ     x λ   + • • • , (1.5)
where

E λ m x n (λ, x) = E(λ, x) * G λ m ,x n (λ, x; σ λ , σ x )
is the m-th differentiation with respect to λ (spectral differentiation)

and the n-th with respect to x (spatial differentiation).

It is safe to assume that camera sensitivities approximate Gaussian functions around the red, green, and blue areas of

     Ê Êλ Êλλ      = 1 3      1 1 1 1 1 -2 1 -2 1           R G B      . (1.6)
This color space is used further for color gradient computation and other matching applications in Section 4.2. More details about the Gaussian color models can be found in [START_REF] Gertjan | Color in Computer Vision[END_REF].

Gradients and edges

Image derivatives are essential to describe images local structure and their applications in computer vision are vast: edge and contour detection, feature extraction, optical flow, shape from shading, image segmentation and object detection. First order derivatives, gradients and edge images result from sharpness or contrast operators applied to gray-scale or color images. Sharp brightness changes are important indicators of information and a multiplicity of phenomena can cause those changes (e.g., changes in surface normal direction and albedo, object and background occluding contours). Identifying the nature of this changes is valuable for multiple applications: occluding contours and surface patch orientations provide information about the shape of an object while sharp albedo changes are carriers of important texture information.

A variety of different image processing operators can be applied to detect edges, the strength of the response indicates the brightness contrast i.e. or the sharpness of the change. As light gets brighter or darker, or any other procedure involved in the image formation changes e.g, surface orientation, camera aperture and sensibility, the resulting image gets brighter and the original image I is scaled by some value s as sI, and the magnitude of the gradient passes from ∥∇I∥ to s∥∇I∥.

Under these circumstances gradient strength is affected and undesirable effects for the purpose of edge detection or shape recognition may appear. A simple solution to this problem is to represent gradients by their orientation in the place of their magnitude.

In the rest of this subsection explores some of the most common operators used to compute the spatial and temporal gradients in gray-scale and color images. The theory gray-scale image gradients is pretty well established and some simple operators like the Sobel operator are widely used in many application fields, unfortunately, this is not the case of color images which is a more complicated phenomenon and where a vast gamma of color operators are required to express different color properties.

Gray-scale gradients

For an image I, the gradient operator is expressed as

∇I = ( ∂I ∂x , ∂I ∂y
) (1.7)
which is estimated by

∂I ∂x = lim δx→0 I(x + δx, y) -I(x, y) δx ≈ I i+1,j -I i,j . (1.8)
When images are treated as 2D-discrete functions, their gradients are 2D vectors defined by their horizontal and vertical directional derivatives calculated at each pixel location. The brightness gradient points to the direction of largest possible intensity increase. The strength (magnitude) of the gradient corresponds to the rate of change attained in the gradient vector direction.

If the filters responses are linear, the order in which they are applied is not important (e.g., differentiation after smoothing, or smoothing an image derivative), or integrated into a single operator that applies the smoothing and the differentiation at the same time

∂(G σ * I) ∂x = ( ∂G σ ∂x ) * I, (1.9)
where

( ∂Gσ ∂x
) is the derivative of the Gaussian smoothing kernel. Parameter σ controls the scale of the smoothing, and has a substantial effect on the response of a derivative filter (only image artifacts of a size larger than σ will be still distinguishable from the background rendering a recognizable operator response).

The Sobel operator [START_REF] Parker | Algorithms for Image Processing and Computer Vision[END_REF] is one of the most popular and most basic differential operators in image processing. The operator convolves two 3 × 3 kernels with the original image to calculate the directional derivative approximations (one horizontal and one vertical), these kernels already incorporate some smoothing.

Edges are obtained after refining gradient responses using classical image processing techniques such as Canny's method [START_REF] Canny | A computational approach to edge detection[END_REF] which extracts local maxima in the direction of the gradient vector (thinning the edges to one pixel) and filtering out pixels representing noise by hysteresis thresholding.

Color gradients

In principle, gradients and edge operators for color images (or vectorial images in general) are similar to their gray-scale counterparts, here, trichromatic or spectral images are used to obtain single component images representing gradient's magnitude or edges binary images (indicating if a pixel belongs to an edge or not). Each color channel is processed marginally, but at some point in the processing chain marginal results need to be fused. Different approaches may be followed:

• Marginal gradients fusion: The idea behind these methods is to fuse the results obtained from the individual components. Fusion can be done on the individual binary edges maps using logical operators (AND/OR). Gradient magnitude may represent the confidence given to the decision of the pixel belonging to and edge or not. Some methods normalize these magnitudes obtaining values between 0 and 1 and use diffuse logic operators.

• Method of Di Zenzo: [START_REF] Di | A note on the gradient of a multi-image[END_REF]: Here, color gradient vectors are calculated looking for the direction of maximal change.

This is achieved by maximizing the L 2 norm which evaluates the vectorial distance at the colorimetric space.

• Hue based methods: This approach for fusing marginal gradients is exclusive of the Hue-Saturation-Value (HSV ) color space. The principle is that highly saturated colors are insensible to noise and deserve higher confidence than low saturated ones. Carron and Lambert's method [START_REF] Carron | Color edge detector using jointly hue, saturation and intensity[END_REF] weights the individual gradients and mixes them according to their saturation. Using one of the following alternatives:

G Carron 1 = αG H + G S + G V , G Carron 2 = αG H + (1 -α)G S + (1 -α)G V , (1.10)
where α is a function whose value depends on the saturation α = α(S) (indicating the relevance of the hue), G H , G V and G S are the individual hue, saturation and value gradients. Some precautions are needed to apply this method correctly: hue is a circular type information and the relevance accorded to it must consider the saturation of all the pixels involved during the gradient computation (neighborhood's geometrical mean is proposed to this purpose).

Figure 1-5: Gaussian smoothing and derivative filters in gray-scale and color up to second order in x = (x, y)(spatial differentiation) and λ (spectral differentiation). Image taken from [START_REF] Gertjan | Color in Computer Vision[END_REF].

• Gaussian color model derivatives: Gaussian models are used too to define the spectral derivatives. Supposing a Gaussian centered at a fixed wavelength with a fixed spectral bandwidth (standard deviation) there are three spectral derivatives available: E which is the zero-order derivative, the first order derivative E λ (which compares the yellow and the blue parts of the spectrum), and the second-order derivative E λλ that compares the green middle part of the spectrum to the outer regions on it (these parameters are implemented on common camera devices imitating the properties of human vision). Convolving the spectral Gaussian measurements E, E λ , E λλ with a Gaussian derivative kernel the spatio-spectral derivatives of a color image are estimated (spatial position and scale are controlled by the Gaussian derivative parameters). The total edge strength attributed to color information is expressed as

E W = √ E 2 x + E 2 y + E 2 λx + E 2 λy + E 2 λλx + E 2 λλy , (1.11)
the spatio-spectral exploration (color receptive fields) up to the second order derivatives is depicted in Figure 12345.

As for gray-scale images, marginal gradients are obtained after applying a differential filter/operator such as Sobel or Prewitt to each individual channel, but color image methods must to avoid the introduction of false chromaticities. In addition, color differential must combine separate channel derivatives without loss of information.

Other image derivative operators

Second order derivatives such as the Laplacian filters, are used to find areas of rapid change (edges) in images. The Laplacian operator is defined as

L(x, y) = ∇ 2 I(x, y) = ∂ 2 I(x,

Filter banks

Texture can be modeled as the repetition of certain primitive elements e.g. spots, bars, edges, sometimes referred as textons.

A natural representation of a texture is to describe what textons the texture exhibits and how these elements repeat. Some special types of filters are required to denote the presence of such primitive shapes. Each point in the image is represented by the set of elements found nearby. As the whole texture repeats forming a pattern, primitive elements are repeated as well.

Each filter acts as a sub-element detector at a collection of scales and orientations and each point is mapped to a vector of filter responses calculated at that point. In general, this vector gives a description about the similarity of the neighborhood around our pixel in comparison to each texton sub-element in a diversity of scales and orientations. Orientation invariant filters can be used as well as an alternative to the use of multiple oriented filters [START_REF] Fogel | Gabor filters as texture discriminator[END_REF]. Conforming a complex pattern of concentric spots (see Figure 123456).

LBP Original Form

LBP Ciruclar Forms

Figure 1-7: An example of the LBP original forms appears in the first row: all pixels in the neighborhood are thresholded by the pixel at the center, then a binary code is formed by the dot product of the thresholded values and their weights. Different sized neighborhoods of the form LBP P,R (x c , y c ) appear on the second row. Here P is the number of pixels considered by LBP pattern and R the radius distance from the neighborhood center [START_REF] Pietikäinen | Computer Vision Using Local Binary Patterns[END_REF].

Local binary patterns

Signal processing methods based on filter banks are computationally too complex to meet the real-time requirements of [START_REF] Ojala | Performance evaluation of texture measures with classification based on kullback discrimination of distributions[END_REF] where the authors highlighted two important aspects of a texture: a pattern and its strength. The original and most basic operator consists of a 3 × 3 square neighborhood where its center is used as a threshold, the resulting thresholded values are accumulated considering a weight associated to their position, these weights grow as powers of two: {2 0 , 2 1 , • • • , 2 P -1 }. The first row in Figure 1-7 illustrates the operator concept, the idea is to build a binary code of length P (i.e., for a 8-pixel neighborhood there are 2 8 = 256 possible labels) which depends on the relative gray values of the center and the pixels in the neighborhood.

Several years after the original publication, this basic operator was presented in a more generic form by Ojala et al. [START_REF] Ojala | Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[END_REF].

In its generic form, the operator has no limitations on the neighborhood's size or on the number of sampling points (P ).

The local texture of the monochromatic image I(x, y) is characterized by the joint distribution of P + 1 gray values

T = t(g c , g 0 , g 1 , • • • , g P -1 ) (1.16)
where g c is the gray value of an arbitrary pixel (x, y) (i.e., g c = I(x, y)), and g p = I(x p , y p ) is the gray value of a sampling point p ∈ {0, • • • , P -1} located at

x p = x + Rcos(2πp/P )

y p = y -Rsin(2πp/P ) (1. 17
)
where P is the total number of sampling pixels around (x c , y c ), these samples are taken sampling the circle of radius R at evenly spaced degrees. The T distribution is approximated by the local neighborhood differences

T ≈ t(g 0 -g c , g 1 -g c , • • • , g P -1 -g c ), (1.18)
this makes the operator invariant to changes of the mean gray value. But these values are not invariant to other changes in gray levels, to alleviate this limitation only signs of the differences are considered

t(s(g 0 -g c ), s(g 1 -g c ), • • • , s(g P -1 -g c )) (1.19)
where s(z) is the step function

s(z) =      1 if z ≥ 0 0 if z < 0 . (1.20)
Accumulating the thresholded differences weighted by powers of two, the operator is finally defined as ULBP patterns are also convenient because they reduce the possible number of labels conducting to more concentrated (less sparse) histogram distributions.

LBP P,R (x c , y c ) = P -1 ∑ p=0 s(g p -g c )2 p . ( 1
An excellent source of information about the LBP operators and its applications in computer vision is the book of Pietikäinen [START_REF] Pietikäinen | Computer Vision Using Local Binary Patterns[END_REF]. In Section 4.1.2 it is described how to use ULBP's to estimate the angles in a circular pattern and how this information is embedded inside covariance matrices for texture and gesture recognition and object tracking applications.

Local features (interest points)

Local feature detection and feature description are two linked but individual processes. In many computer vision applications, first a detector is applied to locate the features (e.g. edges, corners, t-junctions), and after selecting the most important features their descriptors are then constructed to represent them.

Many feature based techniques have been used since the early days of computer vision being of particular interest the works of Hannah [START_REF] Hannah | Computer matching of areas in stereo images[END_REF], Moravec [START_REF] Moravec | Rover visual obstacle avoidance[END_REF] and Harris and Stephens [START_REF] Harris | A combined corner and edge detector[END_REF]. There are two main approaches for finding feature points correspondences: 1) searching locally for the highest correlation location (or the least square errors sum) or 2)

independently detecting features in all the images and match based on their local appearance. Local search is preferred when images are taken in nearby points or in a rapid succession (high frame video sequences). On the other hand, matching is preferred for larger motions or long term occlusions.

Local invariant features provide representations that allow to efficiently match local structures between images. Approximately the same set of local features needs to be extracted from two images showing the same object: local features must be located precisely and in a repeatable fashion. A second criteria to meet is that features should be distinctive, allowing the algorithms to identify them in a list. To resist occlusions, a sufficient number of features evenly distributed along the target is typically demanded.

Corner and feature point detectors

A procedure to find the set of key-points that are stable and easy to identify is the initial step for the local feature representation. In the recent years computer vision has seen an explosion on the number of corner of feature points detectors and descriptors: Harris [START_REF] Shi | Good features to track[END_REF], SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF], SURF [START_REF] Bay | Surf: Speeded up robust features[END_REF] and FAST [START_REF] Rosten | Faster and better: A machine learning approach to corner detection[END_REF], etc. To keep brevity, only the most popular ones will be described here: Harris and SIFT.

Harris detection method was designed to find features that are easy to track. They showed how textureless patches patches are nearly impossible to localize, contrary to patches with great contrast changes. Gradients in a patch are not the only condition required, straight lines (ensemble of locally connected points sharing the same gradient orientation) suffer from the aperture problem (see [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF] and [START_REF] Horn | Determining optical flow[END_REF]) which says that: "it is only possible to align features along the direction normal to the edge direction". Locations with at least two directions are easier to localize than straight lines. The examples contained in Figure 1-9 exemplify this problem.

Comparing one image patch against itself while introducing a very small variation in the position ∆u provides an For each key-point, the surrounding region is normalized for rotation and scale changes. Local descriptors are built then for each normalized region. Finally, feature matching is performed by comparing the local descriptors using an appropriate similarity measure. Image taken from [START_REF] Szeliski | Computer Vision: Algorithms and Applications[END_REF].

N pixels N pixels Similarity measure A f B f A 1 A 2 A 3 T f f d B A < ) , (
estimation about the stability of the feature. This is the autocorrelation function of a surface patch

E AC (∆u) = ∑ i w(x i ) [I 0 (x i + ∆u) -I 0 (x i )] , (1.23)
where I 0 and I 1 are just two images being compared and u = (u, v) is the displacement vector between the two, w(x i ) is an spatially varying weighting function such as the Gaussian kernel. The summation is done here for every pixel x i inside the patch, pixels are indexed by i.

When a patch has a compactly localized and strong minimum it means it is stable and easy to find, edges following a straight line can be ambiguous in one direction while smooth textureless patches are nearly impossible to localize because they have no stable minimum.

This concept was formalized by Lucas et al. and Shi and Tomasi in [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF] and [START_REF] Shi | Good features to track[END_REF] respectively. The autocorrelation surface can be approximated by the truncated Taylor series: The aperture problem is illustrated with three relevant cases: (a) a corner-like point is given and a unique and stable correspondence is relatively easy to find, (b) the point is located over an edge, as many points can be positioned over and edge the resulting flow is not unique and poorly defined, (c) a textureless point is given, even more candidate displacements exists and the point could move in any direction.

I 0 (x i + ∆u) ≈ I 0 (x i ) + ∇I 0 (x i ) • ∆u.
Matrix A must be interpreted as a tensor (a multi-band image), the authors applied eigenvalue analysis to it getting two eigenvalues (λ 0 , λ 1 ) and two eigenvector directions. They observed that the degree of uncertainty for the key-point autocorrelation energy concentration depends mostly in the smaller of the eigenvalues min(λ 0 , λ 1 ), so good features to track are those where the minimum eigenvalue is above a certain threshold.

Förstner [START_REF] Förstner | A feature based correspondence algorithm for image matching[END_REF] and Harris and Stephens [START_REF] Harris | A combined corner and edge detector[END_REF] improved this key-point location criteria proposing a rotationally invariant scalar operator applied to weighted autocorrelation matrix A. The method of Harris and Stephens avoids the costly eigenvalue analysis that requires the calculation of square roots using the following mathematical property

det(A) -α trace(A) 2 = λ 0 λ 1 -α(λ 0 + λ 1 ) 2 . (1.26)
In practical terms any coarsity measure which downweights the value of edge-like features where λ max ≫ λ min is useful. The typical steps towards feature points detection outlined in Algorithm 5. Figures taken from [START_REF] Szeliski | Computer Vision: Algorithms and Applications[END_REF] .

of a set of rotationally invariant grayscale descriptors. They found that the smoothed (Gaussian derivative) version of the Harris operator using parameters σ d = 1 (Gaussian derivative scale) and σ i = 2 (scale of the smoothing of A) outperforms the others.

Scale invariant feature transform (SIFT)

In images containing low-frequency detail, fine scale-features may not exist, but when this images are regarded with a lower level of detail (at a lower resolution) some feature points begin to appear. Considering this problem some methods extract features with a variety of scales e.g., using a pyramid of images and then matching features belonging to the same pyramid level. This approach works fine for comparing images taken in similar conditions and without significant scale changes, however, in applications such as object tracking or recognition, the scale of the object in the image is unknown and can be drastically different from one sample to the other.

A different and more applicable approach is to detect features which are invariant to scale changes. Mikolajczyk and ... Schmid [START_REF] Mikolajczyk | Scale & affine invariant interest point detectors[END_REF] and Lowe [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] developed feature detectors following this idea and obtained in result two different methods based on different assumptions.

The works of Lindeberg are considered within the most relevant in computer vision because he established the theory that formally analyzes the effects of scale. In his works [START_REF] Lindeberg | Scale-space theory in computer vision[END_REF] and [START_REF] Lindeberg | Feature detection with automatic scale selection[END_REF] he helped to define the concept of characteristic scale for automatic scale selection using the extrema of the Laplacian of Gaussian (LoG) function.

Based on these works, Lowe developed the Scale Invariant Feature Transform (SIFT) [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] which constructs a spacescale L(x, y, σ) with the repeated convolution of a variable-scale Gaussian G(x, y, σ) with the input image I(x, y). that produces an scale-space stack of images (left-column). Each image in the stack is separated by a factor k in scalespace. SIFT separates this convolutions into octaves (doublings of σ), each octave is subdivided into an integer number of intervals s, then the value of k is k = 2 1/s . For extrema detection, it is necessary to produce s + 3 images in the stack of blurred images for each octave. When a complete octave has been processed, the Gaussian image that has twice the initial value of σ is re-sampled by taking every second pixel in each row and column. The stack of DoG (Figure 1-11 second column) images results from the subtraction of adjacent images in the scale-space representation as defined by equation (1.15).

From the stacks of DoG images local extrema are detected comparing each sample point to its eight neighbors in the current image and width the nine neighbors in the scale above and below. Only values which are larger than all their neighbors or smaller than all of them are selected. The cost of this is not that high because most of the candidate points are discarded with just a few comparisons. Because point extrema can be arbitrary close together, there is no minimum spacing of samples that warranties the detection of all extrema. The solution is a trade off between efficiency and completeness, experiments show that many of the extrema that are close together are quite unstable and disappear with small image perturbations.

As the detection of SIFT points is based on a different concept than the autocorrelation stability of Harris-like operators these methods complement each other and can be used together.

> (r+1) 2 r
where r = 10.

SIFT algorithm is for sure one of the most transcendental algorithms in computer vision in the recent years. Because its robustness, it represents the milestone for many other applications such as 3D object reconstruction and object retrieval.

It biggest problem however is the execution speed, focusing on this problem Bay et al. published in [START_REF] Bay | Surf: Speeded up robust features[END_REF] their Speeded Up Robust Features (SURF), an alternative method which is several times faster than SIFT and claimed by their authors to be more robust against different image transformations. SURF is based on Haar wavelets and integral images some concepts that we will discuss in Chapter 2 when describing Viola and Jones algorithm used for the detection of faces on images and videos.

Motion information

The importance of movement in computer vision cannot be overestimated. It is used in a broad range of applications such as: structure from motion, camera stabilization, object detection by background subtraction and optical flow just to name a few. Background subtraction and optical flow are for sure the more relevant methods in the context of object detection and tracking applications, so our discussion about motion information in computer vision will concentrate on this methods.

Background subtraction is a technique to extract the foreground of an image yielding regions and pixels of interest for the localization of moving objects. Unfortunately, the detection of moving objects with these techniques is somehow limited to static camera configurations. A background images is compared against the current image to detect moving pixels and regions, these background images can simply be the preceding image on the sequence or a more elaborate model that incorporates more information from the past.

Optical flow generally involves the minimization of the brightness or color difference between correspondent pixels.

Lucas et al. [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF] KLT optical flow algorithm computes independent estimates of the motion of each pixel, uses patch-based approach to estimate local image displacements with sub-pixel precision using a Taylor series expansion.

Background subtraction methods

Two relevant techniques are discussed here: the Σ∆ background subtraction algorithm presented by Lacassagne et al. [START_REF] Lacassagne | Motion detection: Fast and robust algorithms for embedded systems[END_REF] and the multi-layer algorithm based on color and texture of Yao and Odobez [START_REF] Yao | Multi-layer background subtraction based on color and texture[END_REF]. The first method is super fast and yields very acceptable results while the second one is slower but gives finer results.

Background subtraction is a widely used approach for detecting moving objects in videos captured from static cameras.

The basic principle is that as objects move they affect the image formation process and pixels related to the objects location in the image change more than those who belong to the static background. Differentiating pixel-by-pixel the current frame and a reference frame in principle should be enough. The absolute frame difference at time t, noted D t shows higher intensities for the pixel locations x = (x, y) which changed in the two frames. The problem with this approach is that it only works for the cases where all foreground pixels are moving and all the rest remain static. The true is that the background is never completely static and lighting changes, waving trees, clouds, shadows and complex phenomena related to the image formation process produce small or even important changes in brightness, texture and color. To reduce the number of false moving pixels detected a threshold can be helpful to filter out insignificant motion but it is never easy to propose a threshold that works for all the cases. A different possibility is to construct a reference background image by computing the average of a series of preceding images (containing only background is naturally preferred). Wren et al. propose in [START_REF] Richard Wren | Pfinder: Real-time tracking of the human body[END_REF] to fit a Gaussian probability density function for the previous n pixels. A running average algorithm is an efficient algorithm that avoids recalculating averages for complete at each iteration

µ t (x) = αI t (x) + (1 -α)µ t-1 (x) (1.27)
where I t (x) is the pixel's x current value and µ t (x) its previous average. Parameter α ∈ R[0, 1] controls the linear combination in (1.27) and defines the flexibility of the background model update (when α → 1 pixel's model is updated faster and when α → 0 it is more conservative). The standard deviation σ t (x) of the Gaussian is calculated in a similar fashion as µ t (x), then a pixel is labeled as foreground if the following inequality holds

|I t (x) -µ t (x)| > kσ t (x).
(1.28)

Stauffer and Grimson assume that a single Gaussian is not enough for modeling the pixels behavior and propose a Gaussian mixture model (a weighted sum of multiple Gaussian models) as a solution [START_REF] Stauffer | Adaptive background mixture models for real-time tracking[END_REF].

a) Σ∆ background subtraction: This background subtraction method estimates the parameters of the background using Σ∆ modulation which is a very popular tool in analog-to-digital conversion. This method requires very few resources in comparison to other background subtraction methods which makes it attractive for real-time implementations in embedded processors. Let f t be a time-varying continuous or discrete 1D signal, a discrete signal d t is estimated at discrete time indexes {t i } i∈N , and at every discrete value t i the following update formulas are executed

d ti =      d ti -ε, if d ti-1 < f ti d ti + ε, if d ti-1 ≥ f ti .
(1.29)

When applied for background subtraction, the input signal is now the value at pixel location x = (x, y) over time

I t (x). A first order Σ∆ background M t (x) estimator is computed from it.
Then, the values of the absolute differences

|I t (x) -M t (x)| are used to compute a second order Σ∆ background estimator V t (x) which is considered a parameter of dispersion.
The basic version of the Σ∆ background subtraction is shown in Algorithm 7, where the Σ∆ background model M t and its Σ∆ variance V t are updated every frame. N is an amplification factor for V t . Motion is detected for every pixel x where O t (x) ≥ V t (x). V min and V max are two parameters used to control possible overflows in V t (x).

This basic algorithm has been enhanced to provide cleaner results using morphology operators and other modifications, more details are found in [START_REF] Lacassagne | Motion detection: Fast and robust algorithms for embedded systems[END_REF] and [START_REF] Manzanera | σ-δ background subtraction and the zipf law[END_REF].

Algorithm 7: Σ∆ background subtraction algorithm. Data: Let I = {I t (x)} t=1,...,N be an image sequence acquired with a static camera. The background model learned up to time

Input image I t Result: Binary image Êt foreach pixel x do if M t-1 (x) < I t (x) then M t (x) ← M t-1 (x) + 1 if M t-1 (x) > I t (x) then M t (x) ← M t-1 (x) -1 otherwise M t (x) ← M t-1 (x) foreach pixel x do O t (x) = |M t (x) -I t (x)| foreach pixel x do if V t-1 (x) < N × O t (x) then V t (x) ← V t-1 (x) + 1 if V t-1 (x) > N × O t (x) then V t (x) ← V t-1 (x) -1 otherwise V t (x) ← max(min(V t (x), V max ), V min ) foreach pixel x do if O t (x) < V t (x) then Êt (x) ← 0 else Êt (x) ← 1 b) Multi-Layer
t is denoted as M = {M t (x)}. At each time t the model M t (x) is a structure containing M t (x) = {K t (x), {m t,k (x)}, B t (x)}, which consists of a list of K t (x) modes {m t,k (x)} k=1,••• ,Kt(x)
where the first B t (x) ≤ K t (x) have been identified as background pixels (at different layers possibly). There is a limit for the number of modes a pixel x can store which is

K max . Each mode m t,k (x) consists of 7 components m k = {I k , Îk , Ǐk , LBP k , w k , ŵk , L k }: • I k is the average RGB vector (I R k , I G k , I B k )
of the mode k. • Îk and Ǐk denote the estimated minimal and maximal RGB vectors (maximums and minimums of each R,G and B channels are calculated independently) associated to the mode k.

• LBP k is the average LBP pattern from pixels assigned to this mode k (average of the 1's or 0's of the pattern are calculated independently).

• w k ∈ [0, 1] denotes the weight assigned to this mode (i.e., the probability that this mode belongs to the background).

• ŵk the maximal value that w k has achieved in the past.

• L k the background model layer to which this model belongs. L k = 0 means that m k is not a reliable background mode and L k = ℓ > 0 indicates a reliable mode in the ℓ-th layer.

Multi-layered background modeling is able to detect foreground objects against all backgrounds learned from the past but covered by long-term stationary objects (stationary occlusions). An approach which is useful to detect abandoned luggage and background scene changes (e.g., graffiti paintings or posters).

At each frame t the algorithm receives a LBP t (x) and a RGB value I t (x) measured at the image location x = (x, y), the algorithm seeking the mode of the background that best match these measurements. The mode which is the closest one

to the measurements is k = arg min k D(m t-1,k ). If this distance is above a certain threshold (i.e., D(m t-1,k ) > T bg ),
a new mode is created with the parameters {I t , LBP t , w start , w start , 0}. Here, w start is the initial weight (confidence in a newer model being in the background should be low). New models are added to the list of modes kept for each pixel x if K t-1 (x) < K max , if not, it replaces the existent mode with lowest weight. In the case the measurements closely match one of the existent modes, its representation needs to be updated as

Ǐt, k = min(I t , (1 + β) Ǐt-1, k) Ît, k = max(I t , (1 -β) Ît-1, k) I t, k = (1 -α)I t-1, k + αI t LBP t, k = (1 -α)LBP t-1, k + αLBP t w t, k = (1 -α i w )w t-1, k + α i w where α i w = α w (1 + τ ŵt-1, k) ŵt, k = max( ŵt-1, k, w t, k) L t, k = 1 + max{L t-1,k } k=1•••Kt-1,k̸ = k if L t,k = 0 and ŵt, k > T bw , (1.30)
all the other modes are just recopied from the previous frame (i.e., m t,k = m t-1,k ), with the exceptions of the weights which are updated (decreasingly) as T bw is used to verify if the updated mode can be considered a reliable background mode. An hysteris scheme is used to update the model weights according to three different parameters: the weight decreasing factor α d w which is proportional to the constant factor α w , on the constant τ and on the maximal weight ŵk . The larger the values of τ and ŵk , the smaller the value of α d w and the slower the weight decreases. Modes observed for a sufficiently amount of time see their chances of being forgotten reduce (background models occluded by stationary objects). Details about these parameters and how color and texture distances are computed during mode comparison against color and texture measures at x are found in [START_REF] Yao | Multi-layer background subtraction based on color and texture[END_REF].

w t,k = (1 -α d w )w t-1,k with α d w = α w 1 + τ ŵt-1,k . (1.31)

Optical flow

Optical flow is the study of the motion field that exists between two image frames taken at two different times (t and t + ∆t). Such motion field is caused by the relative changes in position and perspective between an observer (any type of sensor such as an eye or a camera) and the scene. In 1981 Horn and Schunck proposed a global method which densely estimates the field of motion between two images [START_REF] Horn | Determining optical flow[END_REF]. In the same year, Lucas et al. proposed a method of differences to estimate the motion in a local feature neighborhood (popularly known as KLT) [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF]. KLT is more robust to noise and other types of difficulties than Horn and Schunck's method. Because KLT is sparse it is faster than Horn and Schunck's which is global. As such, KLT is better adapted to address the problem of robust object tracking in real-time that we are dealing in this thesis.

KLT decomposes the problem in two different ones:

1. determining the translation of the window image which surrounds a particular feature point [START_REF] Tomasi | Detection and tracking of point features[END_REF] and 2. modeling the linear image deformation [START_REF] Shi | Good features to track[END_REF][START_REF] Rosten | Faster and better: A machine learning approach to corner detection[END_REF], the solution to both problems is based on a matching criterion such as the sum-of-squared-difference (SSD) and solving a Newton-Raphson-like optimization problem. Initially, feature windows are detected by feature points detection algorithms (as explained in 1.1.4) based on their texturedness or cornerness. These attributes are measured by their standard deviation in the spatial intensity profile, the presence of zero crossings of the Laplacian of the image intensity and the circular pattern around the point as for the FAST corner detector in [START_REF] Rosten | Faster and better: A machine learning approach to corner detection[END_REF].

A sequence of 2D images is represented in general by two spatial coordinates x and y and a temporal coordinate t.

The image signal is then denoted as I(x, y, t). Images taken at near instants are usually strongly related to each other, this correlation is usually expressed as patterns that move in an image stream. The goal of KLT algorithm is to identify how these patterns move. Considering this, the image function I(x, y, t) is not completely arbitrary, up to some point it satisfies the following equality

I(x, y, t + ∆t) = I(x -ξ, y -η, t).
(1.32)

The amount of motion d = (ξ, η) represents the displacement vector of the point x = (x, y) between time instants t and t + ∆t. In most of the cases, different regions in an image are completely unrelated, thus the motion vector d is a function of x, y, t and τ . In figure 1-13 the displacement d is depicted as the template (Mario) moves from t to t + ∆t. For a single pixel to be tracked it is required that it has a very distinctive brightness with respect to all of its neighbors.

This precise value can change due to noise or illumination changes. It is often hard or impossible to determine where a pixel went from one frame to another based only on one local information. KLT method does not track single pixels but windows of pixels. Not every window can be tracked reliably, only windows containing enough texture can be considered.

In addition, different points within a window may behave differently, the window may be along an occluding boundary, so that points belonging to different objects move at different velocities. In the worst case, they may even disappear or reintroduce to the image after suffering occlusion. Monitoring the evolution of the residue (sum of squared errors, SSE)

allows the detection of possible failures.

Optical flow is useful for many applications such as shape from motion and object tracking. The flock of trackers or median flow algorithm (FoT or MF) described in Section 2-7 details how to use the results from the KLT optical flow to robustly estimate the displacement of a target. Appendix A contains more details about how the optical is computed for simple planar translations and other more complex geometric transformations such as affine maps.

Range information

Range images or depth maps can be formed by multiple cameras stereo-vision or by active methods that project some sort of light forming an special pattern.

In stereo-vision a 3D point p being viewed from two cameras whose relative positions is encoded by a rotation and a translation. Fusing the pictures recorded by both cameras and exploiting their disparity allows stereo-vision algorithms to estimate depth. Stereo-vision requires two processes, the first is the fusion of the features observed by the set of cameras and the reconstruction of their three-dimensional image. However, fusing the set of images turns out to be a very expensive process in computational terms since each picture tipically consists of millions of pixels, with thousands of feature points and edges to match.

Active methods avoid these costs using rays of light and their intersection to estimate the depth of each pixel in the image. Methods based on the triangulation of laser reflections on surfaces exists since the early 1970's. A laser and a pair of rotating mirrors are used to perform a surface sequential scan. Other methods, based on the same principle gain accelerate the scanning of the scene forming a plane of light using cylindrical lenses. The main drawbacks of active triangulation are that the technology is relatively slow and that at some points the laser spot may be hidden from the camera causing undefined regions in the resulting image, some inaccuracies are expected caused by specular reflections, because the reflected light A range sensor scanning the surface of an object projecting a laser plane and calculating the triangulations with the aid of a camera (image from [START_REF] Forsyth | Computer Vision: A Modern Approach[END_REF]).

will not arrive the camera unless it happens to lie in the direction of the mirror direction. Even worse, mirror reflections can find secondary paths which arrive to the camera creating false depth measurements.

Depth-maps are advantageous because pertinent information related to the geometry of the objects and the scene is already provided. Classical image processing methods based on brightness and color information need to be mathematically adapted to deal with range images. Here, contiguous patches with similar shape are grouped using special segmentation methods based on differential geometry used to identify surface discontinuities (e.g. step and roof edges). To some extent depth-map features are more robust and invariant to viewpoint changes than images from photographs. They are extremely useful to determine occlusion boundaries and silhouettes and to separate objects from background. Additionally, because range images doesn't have to deal with texture, color or illumination variations it is easier to do realistic simulations in range images than with ordinary photographs, opening the possibility to train accurate classifiers without over-fitting directly from simulations.

Two-dimensional light patterns are now capable of acquiring depth images in real-time and inexpensively. An example of this is the Kinect, a popular video-game device used by Microsoft Xbox 360 and developed by Primesense2 which provides 30 depth-images per second at a very reasonable price (less than 90 USD in 2013). Depth-images will continue to gain importance in the computer vision and digital image processing communities, as this technology will be soon integrated in other popular consumer electronics products such as digital cameras and smart-phones.

This section we provided a panoramic view of the different local interpretations that a pixel and its neighbors can have in different contexts. Most of the presented local feature detectors and descriptors are based on masks and operators that produce a stronger signal when the presence of a particular chosen type of feature is found. Feature detector operators progressively scan the whole image. The resulting responses are then encoded according to their strength to represent a pixel in the response image. These operators are generally small-sized prototype masks representing the features that we are looking for, the output is a measure of correlation and the goodness of the fit of the prototype to the local image neighborhood. In the next section we will focus on the problem of representing objects using some of the low-level features discussed here.

Object representations

Objects representations are based on cues computed at a pixel level, and combine them by structuring them or cumulating them, in order to provide a relevant signature of the object, which would be preferably unique and therefore separable from other object classes. Robust representations need to consider multiple cues with a high degree of redundancy, carrying shape and appearance information at the same time. The best object representation is context dependent: objects which appear very small in the image don't require a detailed (and complicated) appearance models (i.e. a blob's location and size are often enough) while full detailed objects benefit from appearance representations. Single points (centroids) are only suitable for very small-sized objects, bigger ones need to be better represented using their most interesting points

(key-points) or a collection of primitive geometric shapes such as a rectangles, ellipses and active contours that register their position, orientation, scale and non-rigid deformations.

Simple motion models such as translations, affine maps, or projective transformations (i.e. homographies) are used to adapt these primitive forms to handle deformations. In theory, rectangles and ellipses are more suitable for rigid objects, while non-rigid objects are better modeled by deformable complex shapes such as contours, skeletal models and articulated shapes. But in many practical cases, due their simplicity and execution time limitations, rectangles and ellipses are employed too.

In the rest of this section some of the most important and frequently used object appearance representations are discussed. One of the simplest alternatives is to estimate the object appearance probability densities in a parametric way such as a Gaussian model or a mixture of Gaussians, or with nonparametric methods such as histograms and Parzen windows.

Probability densities are estimated considering the pixels inside the shape model (rectangle, ellipse or contour), and assuming a low correlation these representations perform a separation of appearance features and their spatial distribution. It is clear that this supposition is not true, but this assumption is useful considering the possibility of invariant representations.

This invariance comes at the loss of some discriminant power: two completely different objects can be confounded when their probability densities are similar enough. Templates (sub-images) carry both spatial and appearance information, but they only encode the appearance from a single view, limiting the degree of allowed variations on the object's pose. Multiple view appearance models blend together multiple different views of an object using principal component analysis (PCA) and independent component analysis (ICA) used for shape and appearance information. Active contours such as snakes take a local feature map an treat its response as a landscape on which a deformable parametric curve r(s), 0 ≤ s ≤ 1 slithers.

Finally, our revision of possible object appearance representations finishes with the covariance-based models that measure feature maps inter-correlations such as colors, brightness, gradients orientation, optical flow and spatial coordinates within the object area, multiple views can be averaged with this approach too, but it is necessary to respect some geometrical properties (see Section 1.2.5).

Sub-image representations

The simplest way to represent an object is to consider its image, that is the matrix of pixels which represents the object (see

Figure 1-16-(a)).

Here a direct representation of the appearance pattern is obtained by writing down the intensity or color at each pixel, in some defined order relative to fixed point in the image (commonly the top-left corner).

Other features such as image gradients, Gabor responses, LBP codes and color invariants can be considered too. Feature selection is extremely important since simple image intensities are very vulnerable and sensitive to illumination changes.

Unfortunately, there is no image feature suitable for all the variety of problems and for all imaging conditions and this selection as many others in computer vision is context-dependent (see Figure 12345678910111213141516).

Sub-image representations are based on very basic techniques borrowed from digital signal processing and image processing communities. While these representations are simple and straightforward, their application leads to numerous inconveniences e.g., the total number of calculation grows in proportion to the number of pixel inside the object's shape, it is not robust to object occlusions and depth discontinuities, changes in scale and orientation are not considered. One typical inconvenient of representing a full object depending on a single region is that the possibility of finding a depth discontinuity is higher for bigger tracking regions.

Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA)

Turk and Pentland reduced the dimension of the appearance space using a related vector space model for images [START_REF] Turk | Eigenfaces for recognition[END_REF].

With their Eigenface approach Principal Component Analysis (PCA) is used to estimate the principal directions of variation in the space of all face images (or category space being considered), reducing the high-dimensional pixel lists to a much more compact encoding which highlights the key appearance attributes. PCA turns a set of possibly correlated variables into a smaller set of uncorrelated variables, this way, only few dimensions hold most of the information. When PCA analysis is performed, images are decomposed into a linear combination of vectors accounting for the greatest variance in data (principal components), only the eigenvectors with the greatest eigenvalues are kept and the object is represented by the set of coefficients of this linear combination (see For images containing colors, gray levels or angles it is unpractical to map each possible value to an array (the resulting array would be too large and sparse). A more convenient option is an array of B histogram bins (i.e. buckets), where each bin stores the number of pixels with values inside a predefined interval of size h.

Histograms are formally defined as

f (x) = 1 nh (# of pixels in the same bin as x) (1.33)
where n is the total number of pixels in the image, inspired more recent and popular methods such as the SIFT key point descriptor [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] (introduced in 1.1.4) and the histograms of oriented gradients (HOG) developed originally for pedestrian detection [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] (see Figure 123456789101112131415161718). Note that the precise information of the corresponding gradients and/or the edge positions is discarded when calculating gradient distributions. In practice, these methods sub-divide the image window into small regions (cells) where each cell stores a local 1D histogram of the gradient directions or edge orientations corresponding to the pixels inside the cell.

X i for i = 0 • • • n -1
Gradient histograms are of a tremendous importance in computer vision, the popular histogram of oriented gradients (HOG) descriptor concatenates a series of overlapping gradient histograms to construct a vector descriptor, this algorithm presented by Dalal and Triggs [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] is currently one of the most important algorithms for object recognition and other related tasks. To compute the HOG descriptor a sub-window is first divided into small spatial cells and each one produces an histogram based on the gradient values and their orientations for all pixels within the cell. The orientations in the 8 × 8 sized cells are grouped into 9 orientation bins. HOG descriptors are already robust to small shifts and rotations, to provide robustness to illumination effects, local gradient responses are contrast-normalized using a block-based measure. This normalization is done before entering the information into the histograms.

w x,C = ∥∇I x ∥ ∑ u∈C ∥∇I u ∥ (1.34)
which favors gradients that are large when compared to its neighbors. The effect of this operation is an enhancement of the objects outline curves.

The final HOG sub-window descriptor is formed by the normalized histograms from the dense grid of overlapping blocks.

c) Color histograms: Under some circumstances color-based descriptions provide a higher discriminative power helping to recognize objects by their color characteristics rather than merely depending only on brightness and gradients.

However, illumination and recording conditions may affect the observed colors in a complicated way and photometric invariance methods are often required [START_REF] Gevers | Color-based object recognition[END_REF]. Color description methods are commonly based in different multidimensional data classification techniques such as color image segmentation i.e., extracting significant colored regions in an image or object. In the trichromatic representation, each pixel is associated to a point in the 3D color-space. Pixels which belong to the same region usually have similar properties and form compact point-clouds in the color space. In histogram-based methods, spatial pixel distribution doesn't matter and color distributions are considered only. Some operators process colored pixels to obtain photometrically invariant information, the problem is that under some circumstances such as low saturation and illumination, these invariants are very unstable.

Unidimensional histograms are based on the supposition that homogeneous color regions can be easily identified by mode manifestations inside each 1D component histogram. By projecting color information into a sub-space of inferior dimension it is possible to reduce the computational consumption. The problem is the risk of underestimating information loss that occurs when the algorithm ignores the different modes that appear in the original representation.

Multiple techniques exist in the literature to manipulate 1D histograms, their main differences reside in the selection of the appropriate color components, the extraction of modes from the histograms and the determination of the most representative component. A severe inconvenient of 1D color histograms is the difficulty to perform color class separation: two separable classes in the 3D space may become impossible to separate when projected to the 1D space. As a result, different regions are involuntary grouped into a single region.

Color histograms require great amounts of memory, typically RGB colors re encoded using 256 3 values. A simple and naive solution consist in reducing the size of the histogram by considering only the most significant bits of each component. This is equivalent to reducing the number of quantization levels and performing a type of pre-segmentation.

The problem with this method is that it does not consider the original distribution of the color pixel values, other techniques prefer to first analyze the 3D histogram to identify high density cells from the lower density ones. Next, a set of connected high density cells is grouped into a class. This technique requires a threshold which has a significant impact on the performance of the method. Indeed, two different classes can be accidentally grouped into a same class if it is too low, and many classes can be ignored if it is too high.

Two important choices have to be made when constructing and histogram. The first one is the bin-width h and secondly, the precise location of the bin edges. Both decisions can greatly affect the resulting histogram. Using kernels it is possible to avoid histogram sensibility to bin edges positioning. Then equation 1.33 becomes

f (c) = 1 n n ∑ i=1 σ -1 Ci K ( c -C i σ Ci ) , (1.35)
where c represents one of the color channels contained in x, and {C 0 ,

• • • , C n } is the set of i-bins on that channel, σ Ci

Segmentation and active contour representations

In a natural environment, it is very unlikely to find objects of very simple geometric shapes such as circles, rectangles and ellipses. Irregularly curved complex shaped objects are more common in the real world, to avoid model statistics contamination it is often desirable to find the boundary that separates them from the background. This boundary can be found by segmentation-based methods that assign a label depending on pixel properties such as brightness, texture, color, location and motion. Segmentation is one of the most widely studied problems in computer vision, the most important type of segmentation algorithms are k-means [START_REF] Macqueen | Some methods for classification and analysis of multivariate observations[END_REF], region splitting and merging [START_REF] Chen | The lambda-connected segmentation and the optimal algorithm for split-and-merge segmentation[END_REF], mean-shift [START_REF] Cheng | Pattern Analysis and Machine Intelligence[END_REF], active contours (e.g., snakes) Kass et al. and level sets [START_REF] Osher | Geometric level set methods in imaging, vision, and graphics[END_REF], normalized-cuts and binary Markov random fields together with graph cuts [START_REF] Blake | Markov Random Fields for Vision and Image Processing[END_REF].

Region splitting (divisive clustering) is the process of finding finer image regions depending on the concentration of the information. The algorithm starts with a single region, it estimates its histogram and determines a threshold that best separates the larger peaks on it. The process is repeated for each resulting region until all the histograms are uniform enough or below a certain size. Region merging (agglomerative clustering) does the opposite than region splitting, these algorithms start with pixel-sized regions and link them together based on their similarity (e.g., average color difference)

and their region size. Region merging methods can be used as a pre-processing step that obtains pixel groupings (or superpixels) [START_REF] Mori | Recovering human body configurations: Combining segmentation and recognition[END_REF] to robustify and accelerate the execution of higher-level algorithms such as optic-flow, stereo matching and object recognition.

One of the first methods to estimate the parametric curve that surrounds a target is the method of snakes [START_REF] Kass | Snakes: Active contour models[END_REF] coefficients w 1 and w 2 are always positive and control the elasticity and stiffness of the snake.

Patterns of snake models are matched to an image by searching for the value of the parameter vector that minimizes the external energy. The internal energy is used as a regularizer which favors certain shapes.

Active contours can be applied for modeling dynamic images too (temporal sequences) by defining an additional layer of likely object motions and deformations. In this situation, the active contour model is expressed as r(s, t) and the feature maps evolve with time F (t). A method to estimate the temporal evolution of the contour was later proposed by Terzopoulos As alternatives to the traditional methods just mentioned, computer vision and computer graphics are increasingly relying on machine learning and information-theoric methods. On the other side, information-theoric methods include the still developing field of Computational Information Geometry as a novel paradigm to perform high-fidelity data analysis using the language and thinking of geometry.

Among the many the existing types of data representations, symmetric positive definite matrices (SPD's) are of particular interest to this field. SPD matrices are gaining importance because it has been observed that in very different applications and with very heterogeneous sources such as Diffusion tensor imaging (DTI) [START_REF] Vemuri | Total bregman divergence and its applications to dti analysis[END_REF], brain-computer interfaces (BCI) [START_REF] Barachant | Bci signal classification using a riemannian-based kernel[END_REF] and conventional imaging the information can be modeled and with great convenience and with a very similar fashion with covariance matrices which are just a particular case of SPD matrices. In computer vision, Tuzel et al. [START_REF] Tuzel | Region covariance: A fast descriptor for detection and classification[END_REF] proposed to use the covariance matrices obtained from several image statistics computed inside a region of interest as the region descriptor (known in the literature as covariance descriptor). In comparison to histogram representations, covariance descriptors dimensionality is much smaller. Even better, covariances can be calculated very fast using integral images and the cost is almost independent of the size of the region. Covariance matrices (SPD's in general) are not elements of the Euclidean space, they are points in a Riemannian manifold. As such, classical machine learning algorithms for clustering and classification need to be adapted to work in Riemannian geometry.

Covariance descriptors require basically two types of operations: 1) measuring distances between two points in the manifold and 2) barycenter calculation to have a model that represents all the samples coming from the same class. Knowledge on Riemannian geometry is required to understand these concepts. In the rest of the section, it is explained how to compute the covariance descriptor given an image and a region of interest, and then different methods for measuring distances and updating the models are introduced. This section describes the principles of the method, it explains an efficient computation of the descriptor together with some examples of feature combinations frequently used for this type of descriptors.

Covariance matrices as image region descriptors

Let I represent a luminance (grayscale) or a three dimensional color image (other types of images can also be considered: infrared, depth images, etc.), and let F be the

W × H × d dimensional feature image extracted from I F (x, y) = ϕ(I, x, y) (1.37)
where ϕ is any d-dimensional mapping forming a feature combination for each pixel including features such as intensity, color (in any color space), gradients, filter responses, or any possible set of images obtained from I (feature configurations are covered in Section 1.2.5). Let now

{z k } k=1•••n be a set of d-dimensional points inside the rectangular region R ⊂ F (region size is n). The region R is represented with the d × d coavariance matrix C R = 1 n -1 n ∑ k=1 (z k -µ)(z k -µ) T (1.38)
where µ is the mean vector of the points.

The covariance matrix is a d × d square matrix which fuses multiple features naturally measuring their correlations.

Elements in the main diagonal represent the variance of each feature, while elements outside this diagonal represent the Let i be an image of size W × H and let I be its correspondent integral image of the same size. The value of each pixel in I results from the accumulation of all the pixels inside the rectangle defined by the image's upper left corner and the pixel of interest in i. Mathematically the value of I(x, y) is

I(x, y) = ∑ x ′ ≤x y ′ ≤y i(x ′ , y ′ ). (1.39)
Once equipped with the integral image I, any rectangular sum can be computed in constant time with only four references, the method is graphically described in For the sake of simplicity, many details about the covariance descriptor computation are omitted here, a thoughtful discussion is provided in Section B.1 while some implementation aspects about the integral images method (precision and parallelization) will be discussed in Section 5.2.2.

Covariance descriptor feature spaces

The information considered by the covariance descriptor should be adapted to problem at hand. Covariance descriptors have been used in computer vision for object detection [START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF], re-identification [START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF][START_REF] Romero | Covariance descriptor multiple object tracking and re-identi cation with colorspace evaluation[END_REF] and tracking [START_REF] Porikli | Covariance tracking using model update based on lie algebra[END_REF]. The recommended set of features to use depends significantly on the application and the nature of the object: tracking faces is different than tracking pedestrians for example. The set of textures present in faces is completely different than the set of textures found in pedestrians, or vehicles. Color plays an important role when tracking/re-identifying pedestrians or cars because the enormous variability of colors they may have, but color has less significance for faces because they are very limited in the set of colors they usually have.

φ u,v (z) = ∥k u,v ∥ 2 σ 2 e (-∥ku,v∥ 2 ∥z∥ 2 /2σ 2 ) [ e iku,vz -e -σ 2 /2 ] (1.41)
where u and v govern the orientation and scale of the kernels. The wave vector k u,v is defined as

k u,v = k v e iϕu (1.42)
where k v = k max /f v and ϕ u = πu/8. The maximum frequency k m ax is usually set to 5 in the field of face recognition, f v is the spacing between kernels in the frequency domain. The family of Gabor kernels are constructed by two parameters:

orientation u ∈ {0, • • • , 7} and scale v ∈ {0, • • • 4}.
The Gabor features that go into the covariance matrix are

g uv (x, y) = |I(x, y) * φ u,v (x, y)| (1.43)
representing the magnitude of the convolution of φ u,v with I.

G √ G 2 x + G 2 y ]
Pedestrian tracking [START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF][START_REF] Porikli | Covariance tracking using model update based on lie algebra[END_REF][START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF] and [START_REF] Romero | Covariance descriptor multiple object tracking and re-identi cation with colorspace evaluation[END_REF] [

x y R G B |Ix| |Iy| ] [ x y R G B |Ix| |Iy| |Ixx| |Iyy| ] [ x y H S V |Ix| |Iy| ] [ x y R G B Var LBP ]

Conclusions

This chapter has presented the basics of the image formation process and different cues useful for higher level computer vision algorithms such as: luminance, color, spatial gradients, temporal gradients (movements) and range images. The image operators presented in this chapter are just a sample of the immense amount of algorithms developed by the computer vision community to extract and analyze all this information.

Most of the operators and algorithms described on the first section of the chapter are considered low-level because they are applied uniformly and without distinctions to all the pixels in an image searching for interesting textured locations, highlighting zones of high luminance or color contrast, or modeling how pixel values evolve in time denoting the presence of movement.

In the second section some medium-level algorithms and operators used to build object representations were discussed.

While basic sub-image representations that compare patches pixel-by-pixel can be enough for some cases, these representations are often unable to handle image noise and structure variations caused by scale changes or articulate deformations in flexible objects. Some statistical methods such as PCA and LDA were presented after, these representations are useful to capture the directions of variation using the eigenvalue decomposition or optimizing the separability of the data by modeling the differences. Histograms representations build an holistic description using the distribution of the pixel intensities, colors, gradients and other features. Histograms are convenient for describing objects because they are invariant to changes of shape, size and orientation. To embed some degree of structure to the histogram representations the popular HOG descriptor uses a grid of overlapping histograms. Contour representations are useful because simple geometric shapes such as circles, rectangles and ellipses are uncommon in a natural environment and each type of object usually haves a very characteristic shape, the problem with this representations is the elevated complexity of the operations they often require, as one of the objectives of this thesis is to propose an algorithm that is able to run in real-time in present day embedded architectures this representations were not considered.

Finally, the covariance matrix descriptor presented in Section 1.2.5 similarly to the histogram representations it is able to handle nonrigid objects, fusing multiple features without the exponential growth on the descriptor size observed in histogram representations among other advantages makes the covariance matrix descriptor a strong candidate for the real-time multiple object tracking algorithms proposed in this thesis.

Among the pixel cues defined in this chapter, many of them are useful for the continuation of this thesis. Particularly object representations used for detection or matching based on:

• color distributions,

• a collection of feature points with their optical flow,

• covariance matrices of space, color and gradient/texture features,

• histograms of oriented gradientes (HOG) to perform pedestrian detection for mobile or static cameras.

Next chapter is devoted to the following problems:

• Object detection: machine learning techniques are used to analyze the information contained within a set of labeled samples, build generalizations and applying them to classify a set of unlabeled samples.

• Matching and tracking: Matching is used to compare the appearance descriptors or on-line representations of each one of the detected targets in a scene. Tracking algorithms propose some high-level techniques to trace the evolution of a target in a scene and strategies defined to handle problems such as occlusions and target crossings.

Introduction

In tracking applications, the objective is to estimate the state (e.g., location, size, orientation) of a moving target over time.

This process is often subdivided into two other subproblems: detection and matching. Detection deals with the difficulties of generic object recognition i.e., finding instances from a particular object class or semantic category (e.g., humans, faces, vehicles) registered in digital images and videos. Matching methods in turn recognize the identities from objects previously seen in older frames. Generic object recognition requires models that capture the variability of the instances appearances and shapes. Matching algorithms on the other hand analyze particular information and construct discriminative models that help to disambiguate all different instances of the same category to avoid confusions.

Using appearance matching and dynamical constraints it is possible to design tracking algorithms that trace the target trajectories and follow their changes of appearance, location, orientation, scale and shape. A variety of tracking methods have been proposed since the beginnings of computer vision, some of them estimate the target paths following their state evolution using a Bayesian framework (e.g., particle filters, hidden Markov models, conditional random fields), others measure the perceived optical flow to determine object displacements and scale changes (e.g., median flow). Exhaustive appearance-based methods compare a dense set of overlapping candidate locations to detect the one that best fits a model (based on texture, spatial frequency information or any other hint). The number of comparisons is reduced giving some preference to the most likely locations based on the previously observed target dynamics. Local searches based on gradientdescent algorithms can be used too to smoothly detect target displacements but these methods tend to fail if the tracker is attracted by local minimum.

This chapter opens discussing some of the most popular classifier techniques taken from the machine learning literature to the computer vision field. For instance, Support Vector Machines (SVM's) [START_REF] Cortes | Support-vector networks[END_REF] and AdaBoost [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF] are widely used to tackle the problem of object detection (this is discussed in Section 2.1) and activity recognition. Both methods are vectors classifiers that operate on Euclidean spaces, but many mathematical entities in computer vision do not form vector spaces, in turn, they often reside on non-linear manifolds. Symmetric positive definite matrices (SPD) such as the covariance matrix descriptors presented in Section 1.2.5 are just a good example. Some considerations that are detailed in Section 2.1.3.1 are thus required to perform covariance descriptor classification. The most common approach consists in computing the tangent space to the manifold at the mean of the data points and obtain an Euclidean approximation of the manifold, but a newer approach presented in [START_REF] Jayasumana | Kernel methods on the riemannian manifold of symmetric positive definite matrices[END_REF] represents the internal product between the vector mappings obtained from the SPD matrix space into a pair of vectors located in a high dimensional Hilbert space where Euclidean geometry applies correctly. This technique is used for the particular case of pedestrian detection using covariance matrix classification.

Object classifiers are the corner stone of the tracking-by-detection algorithms (such as the one proposed in Section 4.4),

where targets are re-detected every frame (or blocks of frames) and the tracking algorithm is only charged of identifying the targets and tracing their trajectories.

Popular matching operators such as the mean square error (MSE), the sum of absolute differences (SAD) or the Riemannian metric are just some examples of the typical operators, metrics or divergences used to compare a pair of image regions (or their models) and establishing correspondences. These operators are discussed in Section 2.2 together with other high-level algorithms popular for object tracking applications: the Flock of Trackers (FoT) algorithm (Section 2.2.2)

and Covariance Tracking algorithm (CT) (Section 2.2.4) are based on these matching techniques. more classes is still an active subject. An approach to tackle this problem is to map the input patterns on the Riemannian manifold to an Euclidean vector and to apply there any of the well-studied vector classifiers. The performance of this approach can be very low because it fails to identify the inherent structure of the Riemannian manifold. The method of Tuzel et al. [START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF] presented in Section 2.1.3.1 (page 72) considers the geometrical properties of the Riemannian manifold and obtains better results. This method uses a LogitBoost classifier adapted to the Riemannian manifold thanks to the mapping functions that take a covariance matrix to its local tangent space. A novel technique was presented by Jayasumana et al. in [START_REF] Jayasumana | Kernel methods on the riemannian manifold of symmetric positive definite matrices[END_REF] where Kernel SVM's are used on the set of SPD matrices Sym + d for training and testing. The authors argue that the iterative approach of [START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF] which requires a combination of weak learners on different tangent spaces is an extremely expensive gradient procedure that is repeated at every boosting iteration. Being the cause of a very poor scalability of the algorithm in terms of the number of training samples. Furthermore, the method of Tuzel et al. learns classifiers on tangent spaces used as Euclidean approximates of the manifold, while the method of [START_REF] Jayasumana | Kernel methods on the riemannian manifold of symmetric positive definite matrices[END_REF] makes use of a rich high dimensional feature space.

Object detection by descriptor learning

The outline of this section is the following: Support Vector Machines (SVM) and AdaBoost are introduced first to briefly describe how pattern recognition and data analysis work as supervised learning models in machine learning. Once these classifier algorithms and their most important details have been introduced we will continue with an explanation of their application for the problem of object detection in computer vision. At the end of the section, the required considerations for covariance matrix descriptor classification using these classifier techniques are introduced.

Support Vector Machines (SVM)

In this subsection a brief introduction to the problem of data classification with Support Vector Machines (SVM's) is given.

The very basic concepts of the separating hyperplane, how the separating margin is maximized and kernels methods are discussed. The primal and dual optimization problems are presented too. The aim of this subsection is to present the pattern classification problem and to discuss its geometric properties. How these optimization problems are solved is not discussed here. More comprehensive introductions to SVM's and their implementations can be found in [START_REF] Smola | Learning With Kernels: Support Vector Machines, Regularization, Optimization and Beyond[END_REF] and [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF].

One of the fundamental problems of learning theory is data classification. The simplest case of this problem occurs when we are given two classes of objects and then faced with a new object that we are required to assign to the most similar class. The formal definition of this problem is as follows: given the empirical data

(x 1 , y 1 ), • • • , (x m , y m ) ∈ X × {±1},
where X is some non-empty set from which the patterns x i are taken (often referred to as the domain), and where the y i values are often called labels, targets, outputs or observations. Here, only two classes of patterns are presented, thus, this is a binary classification problem. More complex problems can occur where the patterns are assigned to many different labels (or classes). The problem of learning to generalize the information contained by the presented labeled samples (known as training samples) and apply it to unseen data points, this means that when we are presented with some new pattern x ∈ X , the problem consists in determining the corresponding y ∈ {±1} so that in some sense the pair (x, y) is similar to the training examples. To achieve this, a notion of similarity in X and in {±1} is required. Comparing binary labels {±1} is trivial as they can only be identical or different, but the chosen similarity measure for the input space is a deep question that machine learning algorithms try to solve depending on the context.

A similarity measure of the form k : X × X → R is a function that maps two given patterns and returns a real number that characterizes their similarity. It is assumed that the function k is symmetric, this is k(x, x ′ ) = k(x ′ , x) for every pair x, x ′ ∈ X . This type of functions are called kernels in the context of machine learning literature.

A very common and simple similarity measure used in mathematics is the dot product, given two vectors, x, x ′ ∈ R N , the dot product operator is defined as

⟨x, x ′ ⟩ := N ∑ i=1 [x] i [x ′ ] i , (2.1)
which corresponds to the addition of the element-by-element (i-th entries) products. The geometrical interpretation for the dot product operator is that it computes the cosine of the angle formed by the two vectors (if both vectors have been Image taken from [START_REF] Smola | Learning With Kernels: Support Vector Machines, Regularization, Optimization and Beyond[END_REF].

normalized). The self-dot product of a vector ⟨x, x⟩ allows the computation of the length (norm) of the vector x as

∥x∥ = √ ⟨x, x⟩. (2.2)
Similarly, the length of the difference vector represents the distance between the two vectors x and x ′ . Therefore, the dot product operators allows us to compute angles, lengths and distances.

Given a vector space H equipped with the dot product operation and a set of patterns The model on the left (the less complex one) fails to capture the complexity expressed by the training samples misclassifying a large number of them. In contrast, the model on the right assigns to much trust to each point and offers a solution that is extremely complex and consistent with all the training points. The problem with this solution is that it may not work well with new unknown samples. This problem is popularly known as the over-fitting problem. The model in the middle shows a solution that has a good classification rate for the training set and which is not affected by the presence of some atypical points (outliers).

x 1 , x 2 , • • • , x m ∈ H,
The optimal separation hyperplane is constructed solving the following problem

min w∈H,b∈R τ (w) = 1 2 ∥w∥ 2 , subject to y i (⟨x i , w⟩ + b) ≥ 1 ∀i ∈ 1, • • • , m. (2.3) Equation (2.
3) is popularly known in the machine learning community as the primal optimization problem used to estimate the hyperplane parameters during the SVM formulation. Using the Lagrange multipliers 1 , this same problem can be expressed in a more convenient form where the normal vectors w of the decision hyperplanes are represented as general linear combinations (i.e., using non-uniform coefficients) of the training patterns. For this, the following Lagrangian equation is proposed

L(w, b, α) = 1 2 ∥w∥ 2 - m ∑ i=1 α i (y i (⟨x i , w⟩ + b) -1), (2.4)
where the Lagrange multipliers are α i ≥ 0. The Lagrangian equation in (2.4), is maximized with respect to α and minimized with respect to w and b. The derivatives of L with respect to the primal variables at this saddle point are

∂ ∂b L(w, b, α) = 0, ∂ ∂w L(w, b, α) = 0, (2.5)
which leads us to

m ∑ i=1 α i y i = 0 and w = m ∑ i=1 α i y i x i . (2.6)
Equation (2.6) implies that the solution vector is an expansion expressed in terms of the training points. The patterns associated to α i > 0 correspond to the points that strictly satisfy

α i [y i (⟨x i , w⟩ + b) -1] = 0, (2.7)
they are called support vectors, and they are located exactly over the margin. All the other training samples are considered irrelevant, so that the influence of patterns located very far away from the decision boundary is removed.

Pedestrian detection with SVM classifiers

As with faces, pedestrians detectors are trained to identify regular 2D texture patterns that usually appear in upright people.

As with the Viola and Jones dector (Section 2.1.2), this patterns need to be sufficiently captured by an image sub-window representation. The histogram of oriented gradients method of Dalal and Triggs (HOG) [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] extracts a dense gradientbased descriptor from the window of interest. And an SVM (linear support vector machine) tells if it is a person or a non-person.

These descriptors need to be trained to separate positive samples (actual pedestrians) from the negative samples. HOG descriptors can be treated as points in an Euclidean vector space. Thus, SVM classification methods can be used directly

AdaBoost classifier

The AdaBoost algorithm, introduced in 1995 by Freund and Schapire [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF] takes an input as a training set {(x 1 , y 1 ), • • • , (x m , y m )} where each x i belongs to an instance space X, and each label y i belongs to some label set Y and where m is the total number of learned samples. In the simplest case, the label set Y has only two possibilities Y = {-1, +1} but it is possible to extend this algorithm to handle multi-class problems. AdaBoost calls a weak or base learning algorithm repeatedly, following a series of T rounds t = 1, • • • , T and maintaining a distribution of weights over the training set elements i (samples that are harder to classify receive a higher weight). Those weights are updated from round t to round t + 1. The weight assigned to the sample i at round t is denoted as D t (i). When the algorithm starts, the algorithm doesn't know which samples are going to be harder to classify, so, in the first round all weights may have the same value (i.e.

D 0 (i) = 1/m). At each round the accuracy of the classifier is evaluated, the weights D t+1 (i) of incorrectly classified samples are increased forcing the next weak learner to focus on those hard samples that deserve more attention.

AdaBoost uses a linear combination of a collection of weak classifiers to construct a strong classifier of the form

H(x) = sign ( T ∑ t=1 α t h t (x)
) .

(2.8)

Weak classifier 1

Samples weight is uniform in the first round.

Sample weights are updated according to the accuracy of the classifier.

Weak classifier 2 Weak classifier 3 Weak classifier 4

Strong Classifier During each round t the weak hypothesis h t maps the samples in the input set X to the output set Y (i.e. h t : X → {-1, +1}). To measure how well this weak hypothesis fit the training data, the misclassified samples are penalized as follows

ϵ t = Pr i∼Dt [h t (x i ) ̸ = y i ] = ∑ i:ht(xi)̸ =yi D t (i), (2.9)
and h t is the weak classifier that minimizes the classification error ϵ t considering the weight distribution D t .

In linear combination of the equation (2.8) each weak classifier is weighted by α t . The value of α t depends on the fitting error ϵ t associated to the weak classifier h t

α t = 1 2 ln ( 1 -ϵ t ϵ t ) , ( 2

Result:

The final hypothesis:

H(x) = sign ( ∑ T t=1 α t h t (x) ) 1 Initialize D 1 (i) = 1/m. 2 for t = 1, • • • , T do 3
Train the weak classifier using distribution D t .

4

Get the weak hypothesis h t : X → {-1, +1} and calculate the error

ϵ t = Pr i∼Dt [h t (x i ) ̸ = y i ]. 5 Chose α t = 1 2 ln ( 1-ϵt ϵt
) .

6

Update the weights using

D t+1 (i) = D t (i) Z t ×      e -αt if h t (x i ) = y i e αt if h t (x i ) ̸ = y i = D t (i) exp(-α t y i h t (x i )) Z t .
Z t is a normalization factor that makes D t+1 a valid distribution.

Application example: Viola and Jones face detector

As an object category, frontal faces are well suited for window-based representations. Different face instances have a high regularity of 2D texture patterns. Viola and Jones use an sliding window alimenting generic and discriminative machine learning algorithm charged of discovering these regularities [START_REF] Viola | Robust real-time face detection[END_REF].

This face detection method is based on Haar wavelet features, and requires to compute pixel averages at multiple overlapping regions of different sizes both for training and during execution.

A large set of cropped image faces is first required for constructing the classifier, these images represent the positive samples. A large set of negative samples non-faces images is required too. Negative and positive samples are then re-scaled to a fixed resolution size e.g., 24 × 24. A library of rectangular features windows is then constructed, each rectangular feature is a Haar-like filter which is parametrized by its relative location (inside the image sample), scale and type (see Figure 234). Where the output of a given filter applied to an image is the sum of the intensities of the pixels under the gray-shaded regions minus accumulation of pixels intensities under the white-shaded regions. Viola and Jones designate a bank of filters composed by nearly 180,000 rectangular features.

To avoid computing redundant accumulations multiple times, the rectangular features evaluation is accelerated using integral images as it was described in Section 1.2.5, this way, computing each rectangular feature response requires only 6-8 array references.

For an image x, let f t (x) be the rectangular feature response and let θ t be the threshold which minimizes the weighted error on the training data at round t (for that feature). Thus, the weak classifier response is then

h t (x) =      +1 if f t (x) > θ t -1 otherwise, (2.11)
the tuple formed by the rectangular feature and its threshold θ t define the learned feature.

A final strong classifier results from the weigthed addition of the individual weak classifiers after T rounds of boosting,

h(x) =      +1 if ∑ T t=1 α t h t (x) ≥ 1 2 ∑ T t=1 α t -1 otherwise, (2.12)
where α t is the classifier weight at round t. The value of alpha t is adjusted according to the error rate of its associated weak learner.

Given a test image and the ensemble of learned weak classifiers (2.11) together with their corresponding weights α t that define the strong classifier the recognition process is very simple.

An sliding window traverses the whole image evaluating the strong classifier response (2.12) at a variety of locations and scales. During this evaluation, only the selected T feature responses are measured using the integral images method.

Classification on Riemannian Manifolds

Two different methods for classification in Riemannian manifolds are described in this section. The first approach is based on a collection of classifiers using trained at their tangent spaces and combined through boosting. A second option uses Kernel methods presented by Jayasumana et al. [START_REF] Jayasumana | Kernel methods on the riemannian manifold of symmetric positive definite matrices[END_REF] to calculate the inner products in the Hilbert space H where the all the point x in the manifold M are mapped. But to respect the manifold structure, the kernel function must satisfy some properties described in Section 2.1.3.3.

LogitBoost for classification on Riemannian manifolds

Let {(X i , y i )} i=1,••• ,N be the training set with respect to the class labels, X i ∈ M, y i ∈ {0, 1}. The process of classifications maps each point X in the manifold as

F (X) : M → {0, 1} (2.13)
dividing the manifold into two parts based on the training set of labeled samples.

However, the idea of a function which divides the manifold is a complicated notion that is difficult to import from the Euclidean space. A straightforward (and naive) approach would be to map the manifold into a higher dimensional Euclidean space which can be interpreted as the flattening of the manifold. The problem is that there is no general mapping function that preserves the distances between all the points in the manifold which means that the global structure of the manifold is not respected by these mappings. Tuzel et al. deal with this situation using an incremental approach where several weak classifiers are trained at the tangent spaces and at the end they are combined through boosting. The matrix logarithm function at X, log X (•) is employed (see equation (B.39) in the appendix for a definition of this function) to obtain the mapping of the neighborhood around X ∈ M to the tangent space T X . Because this mapping is an homeomorphism, the local structure around the neighborhood of X in the manifold is preserved. And because the tangent space is an ordinary vector space, standard machine learning classifiers can be applied.

The Riemannian distance d R between to points in the Riemannian manifold is approximated by the distance

d 2 R (Y, Z) ≈ ∥vec X (log X (Z)) -vec X (log X (Y))∥ (2.14)
and the approximation error can be expressed in terms of the sum of the pairwise distances

ε = N ∑ i=1 N ∑ j=1 (d R (X i , X j ) -∥vec X (log X (X i )) -vec X (log X (X j ))∥). (2.15)
The best point in the manifold to train the classifier is the one which provides the best approximation or the minimum error in equation (2.15). It has been determined empirically that the mean of the training points is a good minimizer. Weak learners are then trained on the tangent space at the mean of the training points, at each iteration, a new weighted mean is computed (the weights are adjusted by boosting) and the miss-classified samples receive greater weights. This way, the mean tends to move towards these points and more accurate classifiers are then learned.

The classification procedure used in [START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF] is the LogitBoost algorithm [START_REF] Friedman | Additive logistic regression: a statistical view of boosting (with discussion and a rejoinder by the authors)[END_REF] which operates on vector spaces. Here, the probability of x being in class 1 is given by

p(x) = e F (x) e F (x) + e -F (x) , F (x) = 1 2 L ∑ l=1 f l (x) (2.16)
which depends on the set of weak learners regression functions {f l (x)} l=1•••L that minimize the negative binomial log likelihood of the data

- N ∑ i=1 [y i log(p(x i )) + (1 -y i ) log(1 -p(x i )))] , (2.17)
fitting a weighted least squares regression f l (x) of the training points x i ∈ R m to the response values z i with the associated weights w i , where

z i = y i -p(x i ) p(x i )(1 -p(x i ))
, and

w i = p(x i )(1 -p(x i )). (2.18)
The Riemannian version of the Logit-boost classification method is described completely in Algorithm 9.

Algorithm 9: LogitBoost classification on Riemannian manifolds

Data: A labeled training set {(X i , y i } i=1,••• ,N , X i ∈ M, y i ∈ {0, 1} 1 Start with uniform weight values w i = 1 N , i = 1, • • • , N , F (X) = 0 and p(X i ) = 1 2
2 for l = 1 • • • L do 3
Compute response values and weights:

z i = yi-p(Xi) p(Xi)(1-p(Xi)) , w i = p(X i )(1 -p(X i )) 4
Compute the wighted mean of the points (Riemannian Covariance Mean):

µ l = arg min X∈M ∑ N i=1 w i d 2 (X i , X) 5
Map the data point to the tangent space at µ l :

x i = vec µ l (log µ l (X i )) 6
Fit the function g l (x) by the weighted least-squares regression of z i to x i using weights w i .

7

F (X) ← F (X) + 1 2 f l (X), where f l = g l (vec µ l (log µ l (X)) and p(X) ← e F (X) e F (X) +e -F (X)

8 Store F = {µ l , g l } l=1•••L 9 Output the classifier sign (F (X)) = sign ( ∑ L l=1 f l (X)
)

Pedestrian classification on Riemannian manifolds with LogitBoost classifiers

Tuzel et al. presented in [START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF] a method for detecting pedestrians using classification methods in the Riemannian manifold

where MSE matrices such as covariance matrices lie.

The proposed feature vector in [START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF] for pedestrian detection is

ϕ(I, x, y) = [ x y |I x | |I y | √ I 2 x + I 2 y |I xx | |I yy | arctan |Ix| |Iy| ] T , (2.19)
where (x, y) define the pixel locations, I x ,I xx ,• • • are the intensity derivatives and the the arctan |Ix| |Iy| function defines the orientation of the edge. This 8-dimensional feature vector results in a 8 × 8 covariance matrix descriptor which encodes the variances of the features, their correlations and their spatial layout. The algorithm receives an arbitrary sized detection window R where there is a very large number of different and overlapping subregions r 1,2,••• where covariance descriptors can be computed. To explore them, the algorithm starts with the regions of 1 10 the width and height of the detection window R. After exploring each scale, the size is then incremented in steps of 1 10 , until r = R. Tuzel et al. in [START_REF] Tuzel | Pedestrian detection via classification on riemannian manifolds[END_REF] The same procedure is repeated until the specified detection rates are satisfied.

Kernel methods on Riemannian manifolds

Kernel methods in R n are extremely effective in machine learning and computer vision to explore non-linear patterns of data. Kernel methods map the input data into a high dimensional feature space where a richer representation of the data distribution is attainable. With kernel methods, each point x on a non-linear manifold M is mapped to a feature vector ϕ(x) in a Hilbert space2 H. A kernel function k : (H × H) → R is used as the inner product on H. It should be noted that in general, Riemannian manifolds are non-linear and that many algorithms designed for R n are not available for them. The method described in Section 2.1.3.1 maps the points on the manifold to the tangent space at one particular point (usually the However, these benefits are obtained when the selected kernel is positive definite only. [START_REF] Jayasumana | Kernel methods on the riemannian manifold of symmetric positive definite matrices[END_REF] a radial basis function (RBF) that maps the input data to an infinite dimensional Hilbert space in R n . The Gaussian kernel k G (x i , x j ) := exp ( ∥x i -x j ∥ 2 /2σ 2 ) based on the Euclidean distance between to data points x i and x j is used. A kernel on a Riemannian manifold can be obtained by replacing the Euclidean distance by a geodesic on the manifold. However, not every geodesic distance yields a positive definite kernel. The sufficient and necessary conditions for a geodesic metric to generate a valid positive definite kernel are not described here but can be found in [START_REF] Jayasumana | Kernel methods on the riemannian manifold of symmetric positive definite matrices[END_REF].

Jayasumana et al. proposed in

The log-Euclidean distance for Sym + d (SPD matrices) is derived by exploiting the Lie group structure under the group operation X i ⊙ X j := exp (log(X i ) + log(X j )). Here, the exp(•) and log(•) operators denote the usual matrix exponential and logarithm operators and not the point dependent exponential and logarithm maps of the log-Euclidean Riemannian metric. A geodesic which connects the points X i ,

X j ∈ Sym + d is parametrized as γ(t) = exp ((1 -t) log(X i ) + t log(X j )) for t ∈ [0, 1].
The geodesic distance between these points is thus expressed as

d g (X i , X j ) = ∥ log(X i ) -log(X j )∥ F , (2.20)
where ∥ • ∥ F denotes the Frobenius matrix norm 3 . The log-Euclidean distance defines a true geodesic distance measure on Sym + d and produces a valid positive definite kernel. Being able to compute positive definite kernels allows us to use algorithms developed for R n while respecting the manifold's geometry. This way, the problem of sample classification on the Sym + d manifold using Kernel SVM's given the set of training samples {(X i , y i )} m 1 , where X i ∈ Sym + d and the label y i ∈ {±1} turns out in the research of the hyperplane in H that optimally separates the feature vectors ϕ(X i ) that belong to the positive and negative classes with a maximum margin. The relative position of the point ϕ(X i ) ∈ H to the hyperplane is what determines the class to which the sample belongs to. Other kernel-based algorithms can be used similarly on the manifold of Sym + d matrices such as multiple kernel learning (MKL), kernel principal component analysis (PCA) or kernel k-means clustering.

Object matching

Many different matching methods have been proposed by the computer vision community. Four matching methods which inspired this thesis are presented here.

• Template matching: comparing image patches directly using simple measures such as the mean squared error (MSE), normalized cross-correlation (NCC) or the structural similarity index (SSIM). These approaches are more suitable for to track small patches and small rigid planar objects.

• Median KLT optical flow: popularly known as the flock of trackers algorithm, this method is able to track non-rigid objects analyzing the KLT optical flow and using the median operator to estimate the moving direction of a target composed by multiple local patches.

• Mean-Shift: In Mean-Shift M S tracking [START_REF] Comaniciu | Mean shift: A robust approach toward feature space analysis[END_REF], the target is modeled by a color-space representation, and the tracking is achieved by a gradient-based optimization using the Bhattacharyya distance.

• Covariance tracking: Using the covariance descriptor for matching it is possible to perform local and exhaustive searches. Some methods that incorporate information about the target dynamics to estimate the most plausible displacements are mentioned too.

Template matching by correlative methods

The mean squared error (MSE) is a popular measure of fidelity which is also interpreted as the degree of similarity or, conversely, the level of error and distortion between two different signals. One of these two signals is a template representing a target and the other is the candidate location. Consider a template image T (x i ) defined by the N pixels

x i ∀i ∈ {0, 2, • • • , N -1} inside of it.
A patch of the same size P (x i ) is taken from the target image at the candidate's location. The MSE between these two images is

MSE(T, P ) = 1 N N -1 ∑ i=0 (T (x i ) -P (x i )) 2 . (2.21)
We can also refer to the error signal e i = T (x i )-P (x i ), which is the difference between the template and the candidate images. A more general form would be the ℓ p norm

d p (T, P ) = ( N -1 ∑ i=0 |e i | p ) 1/p . (2.22)
MSE is often converted into a peak signal-to-noise ratio measure (PSNR).

MSE has many attractive features: 1)it is simple and inexpensive to compute, 2) all ℓ p norms are valid distance metrics in R satisfying all the conditions of non-negativity, identity, symmetry and triangular inequality, 3) MSE measures the energy of the error signal e i which has a clear physical meaning, and finally, 4) MSE is suitable for optimization problems such as gradient-descent and Hessian matrix (as seen for the KLT optical flow in Section 1.1.5) allowing to search locally for the minimum-MSE. On the other hand, template matching with MSE has some severe inconveniences: 1) all signal samples are equally important, 2) MSE is independent of the signs of the error e i and 3)temporal or spatial relationships between pixels are not considered.

Normalized cross-correlation (NCC) represents a different approach for measuring the degree of similarity between an image and a template. One advantage of NCC over MSE is that it is usable for conditions in which the brightness of the image and the template significantly vary. The equation that defines NCC is

NCC = 1 N N -1 ∑ i=0 (T (x i ) -T )(P (x i ) -P ) σ T σ P , (2.23)
where T and P are the averages of the template and the candidate's patch and σ T and σ P are their standard deviations.

NCC can be interpreted as the dot product of two normalized vectors. Similar to MSE, NCC imposes significant limitations on the signal samples. Both are unable to view the interaction of samples between each other or how the image structures are affected by and additive error. The structural similarity index (SSIM), have proven being useful as an image fidelity measurement. SSIM can be implemented at a single scale, over multiple scales or in the wavelet domain. The approach followed by SSIM is motivated on the observation that natural image signals are highly structured, which means that image signals often have strong neighbor dependencies that carry important information about the structures of the objects.

Supposing that T and P are the template and the local patch we want to compare, SSIM measures the similarity of three different elements of them: the similarity of their local luminances l(T, P ), the similarity of the local patch contrasts c(T, P ), and the similarity of the local patch structures s(T, P ). All these similarities are obtained using simple local statistics as

SSIM (T, P ) = ( 2µ T µ P + C 1 µ 2 T + µ 2 P + C 1 ) l(T,P ) • ( 2σ T σ P + C 2 σ 2 T + σ 2 P + C 2 ) c(T,P ) • ( σ T P + C 3 σ T σ P + C 3 ) s(T,P ) , (2.24)
where µ T and µ P are the sample means of T and P , σ T and σ P represent their standard deviations and σ T P is the sample cross correlation of T and P after removing their means. The terms C 1 , C 2 and C 3 are small positive constants used to stabilize each of the similarity terms so that sample means, variances or correlations do not lead to numerical instabilities when they are close to zero. SSIM has nice properties such as symmetry (S(T, P ) = S(P, T )), it is also bounded: -1 < S(T, P ) ≤ 1 (achieving the maximum value S(T, P ) = 1 when both images are the same). SSIM index can be computed locally with a sliding window that moves pixel-by-pixel across the image mapping them to a SSIM similarity image. A SSIM score can be obtained averaging over this image.

Median flow (Flock of Trackers) object tracking

In Section 1.1.5, the principles Lucas-Tomasi-Kanade algorithm (KLT) [START_REF] Baker | Lucas-kanade 20 years on: Part 1[END_REF] were reviewed. KLT receives a list of key-point locations at time t and obtains their location at time t+1 by estimating the linear transformation that makes the local patches approximately fit. If a target is regarded as a simple collection (a cloud) of feature points, the estimation of the optical flow on the key-points associated to it should give a valuable cue to estimate the target overall state (e.g., pose, scale, location).

The problem in practice is that optical flow algorithms like KLT face significant challenges such as dramatical changes in key-point appearance, key-point disappearances and occlusions resulting in tracking failures. Reliable target tracking algorithms need to be able to estimate which points deserve more confidence than the rest and which points are outliers. A surrounding patch W is commonly assumed, patches taken at times t and t + 1 are compared using the mean squared error (MSE), normalized-cross-correlation (NCC) or the structural similarity SSIM. These measures methods are useful to detect occlusions and rapid movements in planar objects, but they may fail for non-rigid objects and slowly drifting trajectories.

Kalal et al. proposed in [START_REF] Kalal | P-N Learning: Bootstrapping Binary Classifiers by Structural Constraints[END_REF] a forward-backward error used the detect optical-flow failures. The principle behind it is very simple: tracking should be independent of the direction of time-flow. A key-point traces a trajectory forward in time, then at the last frame of the sequence this trajectory is validating tracking the same point backward in time. If both trajectories significantly differ, the point is considered unreliable. Forward-backward is consuming because it needs to track two times a feature (once for each time sense), some alternative methods have been proposed to estimate the consistency of the trajectories. Key-points are usually part of bigger units (cars, pedestrian, hands, faces, etc.) that move in groups with a certain degree of coherence. In [START_REF] Wendel | Robustifying the flock of trackers[END_REF], Wendel et al. use the trackers neighborhood consistency and a temporal predictor (Markov chain) to evaluate how much confidence the algorithm assigns on them.

When a new target is detected, as set of points is initialized forming a rectangular grid within the target bounding box.

Each point is then tracked by KLT generating an sparse motion flow between images I t and I t+1 . Each point flow is then analyzed and evaluated using the aforementioned methods, 50% of the worst (less confident) trackers are then filtered out.

The remaining points are then used to estimate the displacement of the whole target using the median over each spatial dimension, this is the reason why this method is referred as Median Flow (MF) or Flock-of-Trackers (FoT). From now on, the name used in this work to refer to this method will be FoT.

For each pair of points, a ratio is formed using their distance in the current t and previous t -1 frames. The median of all ratios provides an estimation the target's change of size which depends on the optical flow results only.

Thanks to the robustness of the median, FoT can resist up to 50% of outliers in the translation estimation and 100×(1-√ 0.5)% for the estimation of the scale change. Tolerance to outliers can be even higher because outliers do not conspire and in most cases they are distributed evenly above and below the median. Nevertheless, it is still possible to find scenarios where the inlier percentage is extremely low and where the translation and scale change estimation are very difficult. In the original FoT version, all the trackers are placed regularly over the object forming a grid, it is clear that not all these trackers will be placed at good locations suitable for tracking, and these poorly placed trackers tend to drift away from their original position on the grid and behave as outliers. After estimating the global target displacement and scale change, all these targets are reseted to their original place on the grid. Arguing that this approach is suboptimal, Wendel et al. replace the grid by a set of cells. In this version of the FoT method, trackers are allowed to move and find their suitable offset within their cell i.e. their best position for tracking. Cells guarantee an even coverage of the target, forcing the trackers to In both methods all the trackers are initiated at their corresponding positions in the grid, Cell FoT lets the trackers move within their cells, but they are relocated when they get out from their cells.

stay inside inside of them. If a tracker moves out from its cell, it is repositioned at the cell's center.

Mean-Shift (M S tracking)

In M S tracking [START_REF] Comaniciu | Mean shift: A robust approach toward feature space analysis[END_REF], the target is modeled by a color-space representation, and the tracking is achieved by a gradientbased optimization. The similarity between the target model at initial location x 0 and the target candidate at location x t , is computed as the Bhattacharyya distance, noted ρ, and based on the bin-to-bin product of their respective color distributions.

If a color appears on both the object and its vicinity or background, it is not relevant for tracking because it reduces their separability. A lower confidence has to be granted to that color, as in [5], where the background colors are subtracted from the histogram using the log-likelihood ratio of foreground/background.

The histogram is generally quantized in order to allow real-time execution and avoid sparsity. Consequently, after background subtraction and quantization, the histogram can be close to empty, and the similarity measure might vanish even for small changes in the color distribution.

Considering the reference target model q u , the tracking consists in finding in frame t the candidate location x t for which the representation p t u (x t ) maximizes the similarity to the model (u is the index on the bins). The Bhattacharyya distance is expanded in Taylor series as in [START_REF] Comaniciu | Mean shift: A robust approach toward feature space analysis[END_REF] in order to allow gradient-based optimization. Iteratively, each pixel of index i and color c i in the target contributes to the computation of the new location, with a weight w i , defined as follows:

w i = ∑ u √ q u p t u (x t ) δ(c i -u), (2.25)
with δ being the Kronecker function. The success of M S relies on the good description of the target by its color distribution.

Ideally, the color-space has to be chosen so that: 1) the histogram is not empty after background subtraction; 2) the weights (2.25) are non-zero on the target and are numerous enough.

Covariance descriptor tracking

Given an image representing the target, the aim of covariance tracking is to locate the object and its pose even after non-rigid transformations in an arbitrary image. The original combination of features proposed by Tuzel et al. [START_REF] Tuzel | Region covariance: A fast descriptor for detection and classification[END_REF] considers pixel locations (x, y), color values (RGB components) and the norm of the first and second order derivatives of the intensities with respect to x and y. After the initial mapping, each pixel is converted to a nine-dimensional feature vector

F (x, y) = [ x y R(x, y) G(x, y) B(x, y) ∂I(x,y) ∂x ∂I(x,y) ∂y ∂ 2 I(x,y) ∂x 2 ∂ 2 I(x,y) ∂y 2 ] T , (2.26)
thus, the resulting covariance region descriptor is a 9 × 9 matrix.

Tracking algorithms search in the target image for the region that has the most similar covariance matrix, the dissimilarity expresion is presented in this subsection. The simplest tracking algorithm initially proposed by Tuzel et al. scans the target image using a brute force approach, testing different locations and scales. This approach is suitable for single object tracking applications where there is no available information about the target's dynamics and where unpredictable changes on the location and scale can occur from one image to the next. Probabilistic search methods take the set of N previous locations {x t-(N -1) , . . . , x t } and their covariance descriptors to fit the transition functions representing the evolution of the target's dynamics (e.g., translation, rotation, scale, shear).

Distance calculation: Riemannian metric for SPD matrices

Covariance models and instances can be compared and matched using a simple nearest neighbor approach (i.e., finding the covariance descriptors that best resembles a model). The problem is that covariance matrices (SPD matrices in general)

do not lie on the Euclidean space and many common and widely known operations in Euclidean spaces are not applicable or require to be adapted (e.g., a SPD matrix multiplied by a negative scalar is no longer a valid SPD matrix). A n × n SPD matrix only has n × (n + 1)/2 different elements, while it is possible to vectorize them and perform element-byelement subtraction, this approach provides very poor results as it fails to analyze the correlations between variables and the patterns stored in them. A solution to this problem is proposed in [START_REF] Förstner | A metric for covariance matrices[END_REF] where a dissimilarity measure between two covariance matrices is given as

ρ(C 1 , C 2 ) = n ∑ i=1 ln 2 λ i (C 1 , C 2 ) (2.27)
where

{λ i (C 1 , C 2 )} i=1,••• ,n are the generalized eigenvalues of C 1 and C 2 computed from λ i C 1 x i -C 2 x i = 0 i = 1, • • • , d. (2.28)
In (2.28), x i ̸ = 0 are the generalized eigenvectors. Distance measure (2.27) satisfies the metric axioms for SPD matrices

C 1 and C 2 1. ρ(C 1 , C 2 ) ≥ 0 and ρ(C 1 , C 2 ) = 0 only if C 1 = C 2 , 2. ρ(C 1 , C 2 ) = ρ(C 2 , C 1 ) 3. ρ(C 1 , C 2 ) + ρ(C 1 , C 3 ) ≥ ρ(C 2 , C 3 ).
(2.29)

Exhaustive search covariance tracking

Computing the covariance descriptors in the brute force search mechanism is not expensive due to the integral images method discussed in Section 1.2.5. Without this method computing these descriptors for multiple arbitrary sized and overlapping regions over the whole image would be extremely prohibitive for real-time implementations.

The algorithm receives the input image I and applies to it the feature map ϕ(x, y) obtaining the set of N feature images.

Integral images are calculated for these N feature images (primary features) and their N (N +1)/2 crossed-products (second order features). This will allows us to calculate feature averages over arbitrary sized regions very fast. Applying an sliding window and varying its size, the algorithm computes the candidate's covariance descriptors and compares them against the template model using the Riemannian metric

d 2 (M, Y) = tr [ log 2 (M -1 2 YM -1 2 ) ] , (2.30)
the denotes the distance between the sample Y and the covariance model M.

The most likely location for the target in I is the region that has the covariance descriptor which is closest to the template's model. This method works reasonably well for single target tracking algorithms or image retrieval applications but not for multiple target tracking. The reason is that this method imposes no restriction on the targets dynamics and trackers of similar appearance get confused easily which results in identity swaps that may occur if a mechanism to prevent them is absent. ∑ Ns n=1 w n t = 1 and where all the N s particles are sampled from the density q(x n t |x n t-1 , y t ). The weight associated to each particle is

Probabilistic search methods

w n t ∝ p(y t |x n t )p(x n t |x n t-1 ) q(x n t |x n t-1 , y t ) w n t-1 . (2.32)
From time to time, all the particles are re-sampled and their weights are re-started to 1 Ns . Then, the Monte Carlo approximation of the expectation xt = 1 Ns ∑ Ns n=1 w t x n t ≈ E(x t |y 0:t ), is used to estimate the state x t . An example of the Monte Carlo approach for tracking with covariance matrices is shown in Figure 2345678.

The target state x = (d, s, v) stores information about the location d = (x, y), the scale s = (w, h) and its speed v = (v x , v y ). It is common to model state transitions with a first-order (B = 0) or second-order auto-regressive dynamic model but matrices A, B, C and Σ that define the target dynamics are difficult to estimate and [START_REF] Wu | Probabilistic tracking on riemannian manifolds[END_REF] propose a sampling scheme to model the transitions as

x t = Ax t-1 + Bx t-2 + CN (0, Σ), (2.33)
d n t = d t-1 , v n t = 0 u < u 0 d n t = d n t-1 + v n t-1 u ≥ u 0 , (2.34)
where u is a random number from the distribution U (0, 1), u 0 ∈ [0, 1], d n t is the location of the n-th particle in time t, d t-1 is the location of the target at t -1, v n t and v n t-1 are the velocity of the n-th particle in time t and t -1 respectively.

Steepest descent covariance tracking

The search methods discussed in Section 2.2.4.3 are sub-optimal both in terms of accuracy and execution time. The reason is that global searches find the best match by comparing exhaustively in the whole image. This problem is more evident in high-resolution images or big-sized objects. Moreover, the likelihood of finding distractions (objects with similar covariance descriptors) is greater when the whole image is scanned without considering the most probable target locations.

As a workaround, Tyagi et al. propose a gradient descent based method [START_REF] Tyagi | Steepest descent for efficient covariance tracking[END_REF] expecting that the target's location changes gradually (the prior of finding the object is higher nearby the previously known location).

Avoiding the computation of distance metrics in numerous locations, the steepest descent method of [START_REF] Tyagi | Steepest descent for efficient covariance tracking[END_REF] saves computational resources. The gradient descent method for covariance matrices can be formulated as an optimization problem that only requires the calculation of the local gradient at the current location.

Considering the Riemannian metric of equation (2.30), it is possible to state the problem in terms of the current location

x as f (x) = d 2 (M, Y x ) = tr [ log 2 ( M -1 2 Y x M -1 2 )] (2.35)
The gradient of f (x) is given by

∇f (x) = [ ∂ x f (x) ∂ y f (x) ] T (2.36)
The new target location is estimated iteratively, at iteration i + 1 it is obtained as

x i+1 = x i -η i ∇f (x i ) (2.37)
which is the SPD space steepest descent algorithm. The value of η i controls the step of the descent (learning rate), it can follow an annealing schedule expressed as η i = η 0 (1i/N ), where N is determined empirically. Iterations will continue until convergence, which occurs when ∥η i ∇f (x i )∥ < t conv . For the sake of brevity, the complete development of the SPD gradient equations was omitted here. It is provided in Section B.2 for the interested reader.

Conclusions

Here we conclude our panoramic exposition of the state-of-the-art methods for detection, matching and tracking. As we have seen, some of the existing machine learning methods for data classification have been successfully applied to the problem of object detection and classification. Dalal and Triggs HOG pedestrian detector or Viola and Jones face detector are important contributions to the field of computer vision that have influenced many other applications. Object detection by classification is very important in the context of surveillance applications, because contrary to the background subtraction methods presented in Section 1.1.5.1 they allow us to detect objects based on their appearance and not on their motion, this is particularly useful for non-static camera configurations.

The object matching techniques presented in Section 2.2 are just some examples of the enormous variety of algorithms used for finding correspondences between particular object instances based on their appearance. In the rest of this thesis we will constantly refer to these techniques, specially to the FoT algorithm (median flow) and to the tracking methods based on the covariance descriptor.

In the following chapter, we will explore how these methods can be merged to form a cooperation method that adapts to the context and avoids unnecessary calculations while being robust and capable of facing the challenges posed by the long-term tracking problem: occlusions, disappearances, crossings, noise, etc.

CHAPTER 3

Introduction

This chapter proposes a series of context adaptation capabilities to the previously mentioned tracking algorithms. Section 3.1 initiates with a mechanism proposed in this thesis to evaluate the relevance of color in a local neighborhood and master the difficulties posed by brightness and color saturation changes.

Sections 3.2 and 3.3 introduce some higher-level algorithms that result from combining some of the existing stateof-the-art base methods mentioned in the previous chapter. The objective is to obtain a good balance of robustness and accuracy while keeping the required computing power as low as possible to maintain the real-time execution of the base algorithms. This is possible because the idea of method combination not forcibly implies incrementing processing work, the algorithms described in this chapter were designed to analyze the context and automatically select the more appropriate tracking algorithm according to it. None of the base tracking methods discussed this far is universally applicable, each algorithm has its advantages and limitations and the performance they offer is most of the times application dependent and there is no algorithm which can be considered superior to all the others. Some relevant aspects and threats that a tracking algorithm needs to consider are: the nature of the target, lighting conditions changes, pose and appearance variations, nonrigid deformations and partial or total occlusions. Unfortunately, in many applications such as video-surveillance, there is no prior information available about the nature of the target (e.g. rigid vs. non-rigid) and estimating this information on the fly may pose a very challenging problem, this is specially true when objects are detected using background subtraction algorithms instead of an object classifier. This thesis argues that a good tracking method may result from the combination of several others, the resulting algorithm should be able to handle a larger list of difficulties than any of the original algorithms while still being fast enough. An enormous processing effort is needed to meet all these goals as real-time execution is commonly compromised when the number of targets to track grows.

The base algorithms considered for this purpose where: (i) KLT optical flow algorithm (Section 1.1.5), (ii) Mean-Shift (MS) algorithm (Section 2.2.3) and (iii) Covariance tracking (CT) (Section 2.2.4.1).

Optical flow algorithms are of great interest because of their success in determining the displacement of small-sized patches that act very similar to planar surfaces and where it is unlikely for them to suffer non-rigid deformations. The problem with optical flow is that it fails in the presence of occlusions and large and unpredictable displacements. SIFT and SURF feature points (and their associated descriptors) studied in Section 1.1.4 are better for handling large displacements, rotations and scale changes, the problem with SIFT and SURF is that they require the target to be sufficiently textured in order to have a considerable amount of feature points to match. Kernel-based methods such as Mean-Shift (MS) and

Covariance Tracking (CT) construct a global statistical model of the target based on its color and texture, both methods are very good at handling complicated behaviors and tricky non-rigid deformations but they offer a limited capacity to separate the targets from the background, which is possibly their weakest point.

The FoT algorithm performs a robust analysis on the optical flow KLT results to detect outliers and estimate the target translation and scale change. Our discussion initiates with a complete overview of the methods used to detect the optical flow outliers Section 2.2.2. After it, a detailed description of the cooperation algorithm between the FoT and CT (FoT+CT) is provided in Section 3.2 together with some experiments that justify its pertinence.

Finally, in the last section of the chapter a scheme of cooperation between MS and CT (MS+CT) is proposed in Section 3.3. This algorithm was developed in collaboration with Florence Laguzet, a PhD student college from the Laboratoire de Recherche en Informatique (LRI), it integrates my research work to improve the discriminant power and speed of the covariance descriptor and Florence Laguzet's work to ameliorate to the MS algorithm by automatically selecting the most discriminant color-space. It is noteworthy to say that very few details about the MS algorithm are provided here, more information about the color-space switching mechanisms can be found in [START_REF] Laguzet | Color tracking with contextual switching: Real-time implementation on CPU[END_REF] and [START_REF] Laguzet | Automatic color space switching for robust tracking[END_REF].

Adaptability to brightness and saturation changes

Two different pixels may have the same luminance but different chromaticities, color information provides a complementary measure to luminance that provides an additional discriminative power. In applications like motion analysis where the displacements of a large amount of points are required, color features can be particularly helpful. Those features result from the application of the gradient of Dizenzo defined for vectorial images with the Harris operator using a a procedure very similar to the one used for gray-scale images [106]. Color offers us the opportunity to use invariants and color constancy techniques that provide robustness against illumination intensity and geometry changes [START_REF] Gevers | Color-based object recognition[END_REF][START_REF] Geusebroek | The Amsterdam library of object images[END_REF]. Unfortunately, those properties are exclusive of well-saturated colors and since color features does not depend on luminance variations they may lead us to lose some useful textural information. One of the greatest disadvantages of color features is that they tend to be noisy when luminance and/or saturation is low. Under such conditions, it is recommended to favor luminance information over color because it is more reliable and requires less computing power.

Within an image sequence, it is possible to experiment an important decrease of lighting intensity. Even worst, the illumination change is generally not uniform over the whole image. Therefore, it is not straightforward to identify the best feature for tracking, either color or luminance information. Important gains of discriminative performance can be achieved measuring the relevance of color based on the saturation and the intensity. Color invariants can be used in an optical flow KLT-like tracking method [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF][START_REF] Tomasi | Detection and tracking of point features[END_REF] and in covariance matrix descriptors adapting the information to the relevance of color.

L1 color invariant

The most classical invariant color-space is (H, S, V ) where the hue H gives an interpretation of color which is invariant to shadows and specular reflections [START_REF] Gevers | Color-based object recognition[END_REF], V is the luminance and S the saturation. While Hue has interesting invariance properties, its value is not so reliable when color saturation is low. In addition, its calculation is more elaborated than luminance and less easy to accelerate since it is based on trigonometric functions.

A very direct way to separate chrominance and luminance from the original (R, G, B) components is to normalize them, obtaining the components (r, g, b) which depend only on the albedo and therefore have no luminance information.

For the case of the red color component this normalization is expressed as

r = R R + G + B = a R a R + a G + a B (3.1)
representing the albedo.

Starting from rgb, the scalar value L1 = max(r, g, b) represents the more saturated channel while being easier to manipulate compared to a three-dimensional data. As many color invariants, L1 reduces the separability between colors.

Indeed, as r, g or b, all gray colors become indistinct

( r = g = b = 1 3
) . In addition, L1 can reduce the distinction between two colors with same maximum value. The assumption here is that when just small windows of interest are being considered, the probability that two neighbor pixels have same maximum but different colors is low in most natural-image sequences. However, this robustness is reached at the price of a lower separability between colors, especially when their saturation is low. This is noticeable for instance on the eyes of the character drawn on the box of Figure 3-1, which become uniformly gray using L1. This is also true with Hue since the title of the game can not be read. It is also obvious that the hue produces more noise than L1, especially when saturation is low 2 . In a tracking context, this problem can lead to the detection of outliers points and to matching instabilities. The following section introduces the color relevance function, which is used to determine when the color invariant feature can be used or not.

Relevance of color vs. luminance

Carron proposed in [START_REF] Carron | Segmentation d'images couleur dans la base Teinte Luminance Saturation : approche numérique et symbolique[END_REF] a method for color contour detection in the HSV space by fusing the information from hue, saturation and value channels while keeping coherency with the introduction of relevance measure which depends on the saturation channel. Hue can be considered as a complement to value and saturation channels and be exploited only when it is considered relevant. Or hue can be privileged over value and saturation and let consider the later only when hue is not relevant. This approach can be extended to other invariants with similar properties.

For each point p there exists a relevance coefficient β (p) which results from a sigmoid function applied to the saturation S at point p:

β(S) = 1 π [ π 2 + atan (u β (S -S 0 )) ] (3.2)
where S 0 is the inflection point and u β the slope parameter. Under S 0 , the luminance is privileged, otherwise the color invariant feature is considered to be sufficiently well-defined and can be used properly.

Coefficient β is defined at each pixel involved in the tracking, in order to use color when it is meaningful and to rely on luminance otherwise.

Color saturation adaptive KLT feature tracking

Tracking is done considering (D k , I k ) and (D k ′ , I k ′ ), where D is the color invariant for the color-space in consideration (H or L1) and I their corresponding luminance images at their respective times k and k ′ . A physical point P is located at the image in p and p ′ for frames k and k ′ .

Each point p and p ′ has its corresponding coordinates (x, y). For each point p to be tracked, let be a small window of interest W centred around it, and q a point located in W. The motion undergone by W is modelled by a function δ (p, A)

where the vector A describes the deformation of the window from one frame to another. In that manner, the point P at time k ′ is located at p ′ = δ (p, A).

The tracking procedure consists in computing the parameters A that minimize the following error function

ϵ (A) = ∑ q∈W (γ (q, q ′ ) ϵ D (q, A) + (1 -γ (q, q ′ )) ϵ I (q, A)) 2 (3.3)
with:

ϵ D (q, A) = ∥D k (q) -D k ′ (δ (q, A))∥ (3.4) ϵ I (q, A) = ∥I k (q) -I k ′ (δ (q, A))∥ (3.5)
In addition, γ (p, p ′ ) is the geometric mean of the relevance coefficients for each of the points compared:

γ (p, p ′ ) = √ β k (p) β k ′ (p ′ ) (3.6)
After a Taylor expansion for D k ′ (δ (q, A)) and I k ′ (δ (q, A)) and keeping only the first order coefficients, it yields the following approximation:

D k ′ (δ (q, A)) = D k ′ ( δ ( q, Â)) + GD ( δ ( q, Â)) J Â δ ∆A (3.7)
I k ′ (δ (q, A)) = I k ′ ( δ ( q, Â)) + GI ( δ ( q, Â)) J Â δ ∆A (3.8)
where G D and G I are the Jacobian matrices of D k ′ and I k ′ calculated for both directions x and y. Working with equations (3.7),(3.8) and (3.3), it finally results in the following linearized system

( ∑ q∈W VC V T C ) ∆A = ∑ q∈W γ∆ k D VD + (1 -γ) ∆ k I VI
with:

∆ k D = D k (q) -D k ′ ( δ ( q, Â)) (3.9) ∆ k I = I k (q) -I k ′ ( δ ( q, Â)) (3.10)
The vectors V D and V I are defined for an affine motion model as where g D x , g D y and g I x , g I y are respectively the invariant chrominance and the luminance image gradients. Vector V C used for the calculation of the Hessian matrix is defined as

V D = [ g D
V C = [ g x g y xg x xg y yg x yg y ] T
where the gradients g x and g y are defined as a combination of the chrominance and luminance gradients:

g x = [ γg D x + (1 -γ) g I x ]
(3.13)

g y = [ γg D y + (1 -γ) g I y ]
(3.14)

Features tracking evaluation

After color conversion when necessary, the feature points are detected by the Harris operator dedicated to color images, then the color channels are loaded in the tracking pyramid. In order to determine which color-space can track more points feature lists were obtained for each of the tested color-spaces and grouped on a single list. Each of these feature lists is built obtaining their gradients in vertical and horizontal directions (applying a Sobel Filter), measuring for each point the Harris operator response, applying a threshold that depends on the global maxima of this function. Finally, local maximum are selected. DiZenzo gradient [106] is employed in RGB color-space as well as Carron2 [START_REF] Carron | Segmentation d'images couleur dans la base Teinte Luminance Saturation : approche numérique et symbolique[END_REF] gradient for HSV.

Sequences are played forward and then in reverse order to verify if points come back to their initial location. Feature positions on the forward trajectory

T k f = (x t , x t+1 , • • • , x t+k ),
where k is the number of frames, are compared with their corresponding positions on the backward trajectory

T k b = (x t , xt+1 , • • • , xt+k ).
If the calculated distances are less than a defined distance d f b = 1.5px the features are considered as tracked correctly.

For the pyramidal tracking we use the exactly the same parameters (window sizes, number of levels,etc) as the ones described in [START_REF] Bouguet | Pyramidal implementation of the lucas kanade feature tracker description of the algorithm[END_REF]. Five image sequences were considered during experiments, the first two sequences were captured indoors in a controlled illumination environment, 1)Basketball [START_REF] Baker | A database and evaluation methodology for optical flow[END_REF] has stable illumination conditions, it exhibits colourful and textured matte objects that are somehow easy to follow. 2)Cardgame [START_REF] Geusebroek | The Amsterdam library of object images[END_REF] shows a colorful and textured box with a little bit of specular reflections in its borders, it suffers a strong illumination direction change during time. 3)Pedestrian 2 [START_REF] Kalal | P-N Learning: Bootstrapping Binary Classifiers by Structural Constraints[END_REF] is a lowresolution outdoors sequence experimenting a relatively strong camera movement, it displays highly and low-saturated Table 3.1 shows the number of features successfully tracked in each sequence and color-space, the time required to track each feature is also shown. Indeed, the executing times are different from one method to the other because of the color conversion but also on the number of iterations required to converge.

In well-saturated sequences (Basketball and P edestrian2), the luminance I fails to track many of the features that other techniques are able to track employing color information. For example, in the Cardgame sequence which suffers an important change of illumination, Luminance is able to track many of the features applying the photometric normalization, while RGB and HSV drastically fail to handle this, while they behave pretty well in many of the sequences.

The relatively high number of features successfully tracked with L1 in many sequences prove its tracking capability.

As expected, this invariant fails in sequences where color saturation is weak. Tracking with L1P (mixture of max(r, g, b)

and I) we are able to handle this situation thanks to the adaptive capability of this technique. L1P maintains the tracking performance of the L1 invariant in high-saturated sequences and significantly improves the results for the low-saturated ones. Tracking with (r, g, b) does not improve at all the results obtained with L1 while incrementing significantly the calculations.

L1 and L1P behave better than their equivalents for the Hue invariant, being able to track more points for all the tested sequences (except the peculiar case of the Pedestrian 2 sequence), that is probably due to the noisy character of the Hue component, mentioned in Section 3.1.1.

A qualitative analysis of the experimental results can be done with the help of color-maps as those built for the Basketball sequence and displayed in Figure 34. Points are assigned a color from a scale that goes from red to green denoting the frame where features were lost. Red for features that were lost on the first frames and green for features correctly tracked during the whole sequence. L1P keeps the good response of L1 including areas that neither I nor L1 were able to track (like the ear of the man on the right). the separability between colors can be lower due to the reduction of components. However, as written in Table 3.1 the computational costs are reduced with one component and the number of points correctly tracked higher in some difficult sequences, when illuminations changes occur. Mixed approaches L1P and HP provide a better accuracy compared to L1

and H respectively while maintaining the invariance properties needed for challenging tracking situations.

First cooperation (FoT+CT)

An alternative method to the problem of long-term tracking is proposed here, it is based on the FoT algorithm (see Section 2.2.2) that analyzes robustly the target's optical flow and the covariance descriptor that models its global appearance (color and texture). As described in Section 1.1.4, local feature-based algorithms such SIFT and SURF are very efficient to detect and match a set of key-points no matter their position, scale and rotation but they are somehow limited to inflexible objects. FoT robustly estimates the apparent motion of the target and theoritically resisting partial occlusions masking the target for even more than 50% [START_REF] Wendel | Robustifying the flock of trackers[END_REF] of its surface. The main restriction of the FoT method is that the target must be sufficiently unsmooth for the KLT optical flow to be reliable.

Covariance Tracking CT (Section 2.2.4) appears as an alternative choice that compactly models the target by the correlations within a set of attributes (commonly based on its texture and color). High performances are reached even for low textured objects, since they are represented by a global model. One important weakness of the method is the executing time, this becomes more significant when the algorithm is executed using its exhaustive exploring modality. However, CT is accelerated by coupling it with other tracking methods (such as particle filtering [START_REF] Liu | Covariance Tracking via Geometric Particle Filtering[END_REF] for example) that confine the space of research using the information provided by the target state (i.e size, speed, position).

The interest of mobile and real-time video tracking applications is to detect and follow not only a single type of target but a variety of them. The most common types in surveillance applications are: faces, pedestrians, clusters of people and vehicles. In addition, the desired tracking algorithm should be able to deal with very adverse filming conditions: outdoor or indoor, changing angles and scales, with a static or a moving camera, in luminous or dark scenes, etc. In order to have a good adaptivity to the context while still preserving the real-time execution of the algorithm this section presents a new method that combines the complementary advantages of the FoT and CT tracking algorithms by changing adaptively with the context. This algorithm is ideal for dealing with target occlusions without any presumptions about the nature of the target or their type of motion. This cooperation method is denoted from now on as FoT+CT.

FoT outliers detection

The detection of outliers is very important to improve the performance of the FoT algorithm. Some popular methods for this purpose are: a) Forward backward method: Many feature point tracking algorithms detect failures by describing each tracked point by a surrounding patch W which is compared from time t to t + 1 using template matching techniques such as MSE, NCC and SSIM. These error measures allow us to detect failures caused by occlusions or rapid movements because the feature's local patch appearance changes drastically from one frame to the next. But slowly drifting trajectories, where a tracker gets attracted by a nearby location that at this particular instant looks similar to the trackers patch. It is important to emphasize the observation that not all pixels in a local neighborhood move exactly the same. For this problem the affine KLT model was introduced (see Section 1.1.5) to estimate more complex geometrical transformation parameters than simple translations. It is worth considering that the affine KLT method is constrained to planar targets only.

The Forward-Backward (FB) error measures the coherency of a set of feature point trajectories. The sequence S = (I t , I t+1 , • • • , I t+k ) is an image sequence and x t expresses the location of a point at time t. With a feature point tracker such as KLT the point x t is tracked forward for k frames defining the trajectory T k f = (x t , x t+1 , • • • , x t+k ), where f indicates the forward direction and k expresses the length. To estimate the reliability on the trajectory T k f , a validation trajectory is constructed, the point x t+k is tracked backward up to the first frame producing the trajectory

T k b = (x t , xt+1 , • • • , xt+k ).
The Euclidean distance between this two trajectories expresses the FB error as

F B(T k f , T k b ) = ∥x t -xt ∥. (3.15)
Under challenging conditions such as when the motion of the target is faster than what the tracking algorithm can handle, the displacement depends on factors not considered by the tracking model and becomes to some extent random. FB error works pretty well for such conditions, because it is very unlikely that the tracker will follow the same path in the opposite sense.

b) Neighborhood consistency verification: Neighboring feature points are expected to move in a similar way, but as mentioned for the forward-backward case when trackers fail random displacements that have no relation with their neighbors appear more often. Wendel et al. presented in [START_REF] Wendel | Robustifying the flock of trackers[END_REF] their Nh predictor which is implemented as follows. For each feature point i, a neighborhood consistency score S N h i is obtained, based on the behavior of the neighborhood N i which contains the four neighbors of i (edges and corners in the bounding box have only three or two neighbors). S N h i accounts for the number of local trackers that have similar displacements in comparison to i, this value is computed as,

S N h i = ∑ j∈Ni [ ∥∆ j -∆ i ∥ 2 < ε N h ] , (3.16)
where ∆ i and ∆ j are the displacements of the local tracker i and its j-neighbor, while ε N h represents a threshold of the displacement difference. A local tracker is labeled as consistent when S N h i ≥ θ, the suggested value of θ is 1, and the displacement threshold is ε N h = 0.5 pixels.

c) Markov predictor: The Markov predictor (Mp) is used to model the behavior of a local tracker in time. A two-state Markov chain (inlier and outlier) is used to predict the status of a local tracker depending on its state in the previous time instance and a a set of transition probabilities which are calculated incrementally from frame to frame. Each tracker i has its own transition probabilities which are included in the transition matrix T i t structured as

T i t =   p i (s t+1 = 1|s t = 1) p i (s t+1 = 1|s t = 0) p i (s t+1 = 0|s t = 1) p i (s t+1 = 0|s t = 0)   . (3.17)
In equation (3.17), s t represents the current state of the local tracker (inlier=1, outlier=0). Each of the transition probabilities are complementary, and the terms contained in both columns must be equal to 1. The state at t + 1 depends on the probability of the state at the current time t and the transition matrix. This is calculated as product   p i (s t+1 = 1)

p i (s t+1 = 0)   = T i t ×   p i (s t = 1) p i (s t = 0)   , (3.18)
where the left hand of the equation expresses the probabilities of the trackers next state, considering that at the current time the probabilities are p(s t = 1) = 1 and p(s t = 0) = 0 if the current state is inlier (likewise for the opposite case).

The transition probabilities in T i t are updated as follows

p i = (s t+1 = 1|s t = 1) = n i 11 n i 1 p i = (s t+1 = 1|s t = 0) = n i 01 n i 0 , (3.19)
where n i 1 and n i 0 are the relative frequency for the local tracker i is an inlier or an outlier. In turn, n i 11 and n i 01 reflect the relative frequencies of transitions events on tracker i (e.g., n i 01 represents the relative frequency of the tracker i passing from the outlier state to being considered an inlier). The current state of the tracker is verified measuring the error between the trackers displacement and the global target motion and applying a threshold.

FoT+CT algorithm description

FoT+CT is summarized in Algorithm. 10, it starts by analyzing the sequence of images I t (t stands for the time-stamp), targets are appended to an initially empty tracking list denoted as objList. Objects are selected manually or detected by a background subtraction method such as Sigma-Delta (see Section 1.1.5.1) when the camera is static. In moving camera sequences, targets are detected by means of an object classifier (see Section 2.1). Other methods are conceivable such as the disparity map when stereo-vision is available.

In the FoT+CT cooperation method an object detection algorithm is executed each t detect frames (procedure objDetection line 4), unless they are provided by an specific object detector, the nature of the targets provided by background subtraction algorithms is generally unknown. Initially, the detected objects are considered to have a rigid motion and to contain key-points, therefore the tracking is initialized by FoT (see Section 2.2.2). Features points extracted in each rigid object are pushed into a list (line 6), and, each of the n features inside this list is tracked with the KLT pyramidal method (line 6) to compute its motion X k . Each target the global residual flow (X R ) is then calculated, the amount of flow which differs from the median flow (X median ) is

X R = ∑ n k=1 X median -X k n , (3.20)
which represents a measure of the coherence of the optical flow.

A low residual flow indicates that the motion of the points is consistent, i.e., the trackers have correctly converged and agree for a similar motion. In that case, the FoT is well adapted to the object, and the latter is confirmed as rigid (obj rigid in Algorithm. 10). At the same time, the current covariance matrix C t and its dissimilarity D t with the previous model C t-1 are computed, for three reasons:

• preparing a possible switch to CT in case of failure of the FoT,

• confirming the correctness of the tracking or,

• preventing from a drift of the target during the tracking.

While an object remains classified as rigid, the FoT is run.

Abrupt increments of X R can appear for several reasons. The most popular ones are occlusions or object's appearance changes, illumination changes such as shadows or highlights created by luminous sources. Under such situations, the FoT may fail because of the ill-conditioning of the matrices involved in the KLT tracking procedure.

C t-1 ← covUpdateModel(obj)
When a tracker starts losing a target, an increase of the X R is generally observed. The quality of the tracker is monitored as well by constantly comparing the appearance of the target in the current location against the model. If they are unalike (the covariance dissimilarity D t is higher than the threshold D max ), the algorithm activates the covariance exhaustive matching instead of FoT. An object can also be classified as non-rigid when its residual flow X R reaches a limit R max , this means that the CT algorithm should be a better option to track this specific target. This state is verified frequently by testing the performance given by FoT. Indeed, when both FoT and CT behave correctly, (i.e., the residuals are low and the covariance dissimilarity is low), then FoT is logically preferred because it runs faster.

Finally, whatever the current tracking method is (FoT or CT), the covariance model C t-1 is updated considering the appearance at the newest position (line 14), in order to prepare the next frame-to-frame matching.

Additionally to the algorithmic description of Algorithm 10, a description of the switching mechanism for a single target is provided graphically by the flow-chart in Figure 34567. Last frame reached by each method is indicated. Detailed results are also represented for the case of FoT+CT method: percentage of frames CT was preferred over FoT, the average time each method demanded and the speed-up gained using FoT+CT. A list of aspects constitute the evaluation criteria: the length of the sub-sequence where the tracker works successfully , the tracking precision, the stability and the speed.

Sequence

FoT alone is a bad long-term tracker because it does not incorporates a drifting recovery mechanism. This is observable on columns five to seven from Table 3.2. By monitoring the coherence of the flow X r (in equation 3.20) the appearance dissimilarity (the distance to the covariance model D t ) the FoT+CT algorithm is able to inform the tracker when an occlusion or a target drifting is occurring, activating immediately the CT exhaustive search algorithm which should be able to re-detect it if the target is still visible or recollect it when the temporal occlusion ends.

Very frequently, CT alone is enough to follow the target from end to end, however, the trajectories provided by this method tend to be very shaky and in some cases they conduce to a drift of the tracker when it gets distracted by other objects on the background. FoT oscillates less, but is not equipped to deal with total occlusions. FoT+CT in turn inherits the stability of FoT and the robustness of CT and describes smoother trajectories that tend to drift less, not only FoT+CT accomplishes better the task, most of the time it turns faster than executing CT alone.

Figure 3-8 shows the first frame and the selected target for each of the six sequences tested (see Table 3.2). We provide here a careful description of sequence behavior for each of the three methods evaluated (FoT, CT and FoT+CT).

• Sylvester: This sequence displays a plush toy handled by a man, all images were recorded in gray-scale. The target is is easily recognizable because it contains highly contrasted black and white patches. The evolution of the target appearance is shown in Figure 3-9a which contains some snapshots taken using the FoT+CT tracking algorithm. It is easily noticeable that the target behaves pretty much as a rigid object throughout the sequence and that the only difficulty posed to the tracker is to follow the capricious changes of the target's position, scale and perspective. The frames where the tracker is not adequately centered are shaded in red, thanks to the CT algorithms the tracker is able to recover it.

• Auto: Filmed in gray-scale, this sequence displays a vehicle advancing on a road and being followed by a moving camera on the air. So, it is a rigid target that preserves its appearance unchanged from the beginning to the end of the sequence. Even though, this sequence is way more complicated than the previous one, the reason is that the camera moves haphazardly and that the target suffers partial and total occlusions on multiple occasions. Moreover, there are very similar vehicles that appear from time to time near to the target. The evolution of the target captured by the FoT+CT tracker is shown in Figure 3-9b.

• Panda: This video shows a panda walking in its zoo confinement. The video was filmed in color by a still camera that pans and zooms in and out at some points in time. In the images, there are some zones textured pretty much alike to the panda. These zones can cause strong confusions when the tracker looks for the target exhaustively. For example, the text on the bottom is a potential distraction if the target moves around it. Most of the time, FoT is executed except for the moments when the panda turns around or when it passes behind the tree on the top right corner. The use of the CT algorithms becomes indispensable, allowing the algorithm to re-acquire the target once it has been lost. The evolution of the target appearance and the tracker behavior is illustrated in Figure 3-11a.

• Pedestrian 1: This sequence is probably one of the most demanding one of the testing dataset. And it is here where the FoT+CT algorithm probes its utility more clearly. Both FoT and CT are clearly unable to follow consistently the target when executed individually. The real difficulty strives in the unpredictable movements of the camera which cause FoT to lose the target almost immediately. On the other hand, CT is capable of recovering in all the occasions it gets lost, but due to the so spoken unsteadiness of the CT algorithm which is even more severe due to the camera movements the results are not very convincing. The stability of the FoT+CT is remarkable, even the violent shaking of the camera, the tracker is able to retrieve a valid target position for most of the frames correcting immediately all the small drifts of the tracker. • Motocross: This sequence appears to be a recording from a motocross competition. The camera is probably mounted on a bike that is always behind the target. The recording results are very blurred and shaky as the camera advances and jumps constantly. Matching objects under such conditions is a an arduous task. Similar to the previous sequence FoT fails immediately, and while CT is able to retrieve the target it usually takes some frames to glance over the complete set of possible locations and bump into to a promising one. For some frames the optical flow is enough to estimate the displacement of the target, this stabilizes the tracker and reduces the computing cost. When the target is lost because it disappears from the camera field of view or because it moves violently, the CT is acts in response to retrieve it. Most of the frames in this sequence operate in this modality. Figure 3-10 shows the evolution of the target along the sequence, the difficulty posed by this sequence is easily observable.

• PETS2001 Dataset 1: This sequence was captured by a fixed surveillance camera and despite its appearing simplicity some interesting tracking difficulties arise. The first one is attributed to the low-resolution of the target, the other intriguing condition takes place when target traverses from the grass to the pavement, causing the background color to change drastically from green to gray. As the covariance model assigns the same weight to every pixel inside the targets bounding box, this forcibly has an impact on the covariance descriptor. FoT+CT is able to compensate these tracking problems and retrieves the target even when it passes behind the lighting post using the CT algorithm, and thanks to the optical flow analysis of FoT it is able to estimate the displacement of the target even if the color of the background changes. The described phenomenons are noticeable in

Second cooperation (MS+CT)

As for the case of the previous section MS and CT have complementary advantages too. MS represents the target by its color distribution, a model which is almost insensitive to geometric distortions and valuable property when tracking nonrigid targets. MS runs fast but similarly to FoT it is unable to handle large motions and occlusions. CT enjoys a greater separability power and is applicable with many kinds of objects. Contrary to FoT and MS, CT is comfortable with large displacements, but it usually turns out to be less time effective than MS and other tracking methods. In order to reduce the computation times of CT without a robustness loss, this section details a cooperation algorithm between MS and CT (denoted as MS+CT), where the covariance dissimilarity is used as an indication to check the accuracy of MS, then and where the CT algorithm is run to retrieve the target after it is lost or it exits from an occlusion.

MS+CT algorithm description

Kernel-based methods of the type of Mean-shift [START_REF] Comaniciu | Mean shift: A robust approach toward feature space analysis[END_REF] are usually well appropriate to track objects with non-rigid motion and low textural or structural contents, because they rely on a overall statistical distribution that is invariant to some appearance deformations. The price to pay is a decrease of the robustness. Several attempts have improved the method by background subtraction, color space switch [START_REF] Laguzet | Automatic colorspace switching for robust tracking[END_REF] or by using a spatio-colorimetric histogram [START_REF] Gouiffès | Projection-histograms for mean-shift tracking[END_REF]. MS follows a minimization procedure which assumes small inter-frame motions, when MS fails for some reason (like occlusions and false convergence of the MS minimization scheme) CT is asked to retrieve the target. To be able to do so, the exhaustive search CT method (described in Section 2.2.4.2) is selected. Once the target has been retrieved and that the tracking conditions are back to normal, it is possible to give back the control to MS and carry on.

This cooperation scheme is deterministic and no assumption is made on the motion model of the target. Failures of MS and CT can be detected but no explanation about their origin is required as the retrieving mechanism does not depend on it. 1. Initialization:. At time t = 0 a detection algorithm provides the windows of interest where it is assumed the targets are located, here is where the MS is started, at this point the covariance descriptor of the target C t is computed too.

2. Standard tracking:. Since MS is the less expensive algorithm it is preferred over CT when possible. However, the quality of the MS tracking algorithm is verified constantly during the sequence. The appearance of the target j at its current location x t j at time t is represented by the covariance matrix C t j . This descriptor is compared with the target model C j using the Riemannian metric of equation (2.30) to confirm the correctness of the tracking and to prevent draftings of the tracker. If D t represents the dissimilarity of the current target location appearance with respect with respect to the target model, when D t is low the location of the target is confirmed and the tracking continues with the same method. The covariance model of the target C j is updated constantly preparing it to the next frame-to-frame matching. It is very important to highlight that the covariance matching is not run in all frames, i.e. there is no exhaustive search comparing multiple candidate locations. At this point, the target location has already been found by MS, allt that the algorithm needs to do is to compute the covariance descriptor C t j , compare it against the target model C j and update it. All these tasks demand little extra work.

3. MS failure:. Abnormal tracking conditions such as occlusions, illumination changes, large motion or appearance variations. Lighting changes may alter drastically the histogram of the target if the employed MS color-space is not invariant to those changes. When the tracker begins to loose the target, the covariance dissimilarity D t becomes higher than a threshold D max , and the covariance matching (CT) is run in order to re-acquire the target. When the new D t is back to a tolerable low-level i.e. the target is considered re-acquired and controls returns to MS which is less time-consuming. Otherwise, CT executes as long as D t remains higher than D max . If CT is unable to retrieve the target and D t remains high after a long time T , the target is assumed to be definitely lost.

MS+CT evaluation experiments

In this subsection the performance of the MS+CT cooperation algorithm is evaluated for a number of tracking sequences.

• Pedxing: In this sequence, the CT (shown in red) fails at t = 7574, while MS+CT correctly tracks the pedestrian during the whole sequence (see t = 7672). When the target is not occluded, MS is generally more precise than CT, when the pedestrian passes behind the panel at t = 7574, it is correctly handled by switching from MS to CT. For the less-precise feature vector configurations, CT alone is distracted by another pedestrian before the occlusion actually happens.

• Panda: At the beginning of the sequence (from t = 1 to t = 80) MS behaves correctly. But suddenly the panda turns on itself and its appearance changes drastically, this is where the value of D t increases and becomes higher than D max . This value remains high during several successive frames (see t = 190), when the back of the panda is viewed by the camera. CT is charged of tracking during all this period of time, while D t remains high. When the panda turns of itself once again, its appearance is closer to its initial appearance, D t decreases and control returns to MS (t = 300 and t = 400). This type of events occur several times throughout the sequence (t = 800 and t = 900).

The cooperation of both algorithms succeeds in two aspects: it choses the fastest and more precise algorithm when it is reliable (MS) and rapidly switches to the algorithm that is better adapted to deal with its difficulties (CT).

• Motocross: Due to the large motion of the target, the MS fails in frame t = 150 and never retrieves the target. In the cooperation procedure, the dissimilarity D t allows to identify this problem and CT is preferred over MS most of the time. When the target is retrieved, MS is able to track the target for some time. Running CT alone is generally less precise in the fragments of the sequence when the target moves smoothly. For some feature configurations CT alone can even fail.

• Carchase: In this sequence MS loses the target most of the time. For example, at t = 365, lighting conditions change and MS minimization procedure fails. At t = 390 the target is occluded. On the other hand, CT is sometimes distracted by other cars (as shown in frames t = 50, 413, 1250 and t = 2590). With the cooperation of both algorithms several difficulties are handled better: the total occlusion by a bridge between t = 365 and t = 390, the partial occlusion and blur at t = 413, scale change at t = 2430, etc. The association of MS and CT provides more precise results than CT alone for standard situations (i.e. free from occlusions) obtaining an overall more robust algorithm than MS when the target is lost due to occlusions or lighting changes. (which are never executed simultaneously), the model update and the similarity criterion which execute at every frame.

In addition, the percentage of frames for which MS and CT are selected with respect to the sequence length is shown.

In sequences where the camera is more stable, such as Pedxing and Panda, MS is executed most of the time as long as motion is regular and free from occlusions. If those conditions are not met, CT runs and retrieves the target location.

When the camera is moving (as it is the case of Motocross and Carchase sequences), the motion of the target is tricky and unpredictable. Therefore, CT is run more often.

The color tracking by contextual switching has two main advantages:

1. Robustness: CT helps the MS tracking algorithm when it is deficient due to occlusion, tricky motions and appearance changes.

2. Speed: MS is faster the computation time is generally reduced compared to CT alone. The more often MS is run the faster the tracking. Consequently, the cooperation procedure is globally less time-consuming when the camera is still and when the target is not occluded.

Conclusions

The first group of contributions of this thesis to the state-of-the-art has been discussed. The L1 color invariant is our first proposal to improve the quality of the optical flow. The second contribution presented here consist in modifying the equations used by the KLT optical flow algorithm to handle brightness and color saturation changes better. The approach presented here measures the relevance of color using a sigmoid function that depends on the saturation of the pair of pixels being compared. The experiments associated to these contribution demonstrate its utility to increment the number of reliable features to track at the same time that the magnitude of the tracking errors is reduced.

The second set of contributions treated in this chapter is the pair of combination methods FoT+CT and MS+CT. All the three base algorithms: flock of trackers (FoT), mean-shift (MS) and covariance tracking (CT) are known to be capable to handle non-rigid targets, but each one obeys its own limitations: FoT is more suited to objects which have enough salient feature points and which undergo rigid motion, MS is preferably used when the color histogram is discriminant enough, and it doest not require any salient feature. In terms of computation time, FoT and MS are more efficient than CT. In regular situation the FoT algorithm is reliable and describes smooth trajectories for smoothly moving objects but fails when the target is occluded. In turn, the CT algorithm is typically less stable (it describes noisy tracking trajectories) but is able to retrieve the target using the local and exhaustive searches described in Chapter 2. Concerning the MS+CT cooperation scheme, MS in general requires less cycles than CT but is only able to perform local searches. The combination method proposed here is able to automatically detect the sequences (and time intervals of frames along them) where each methods fits better, e.g., in moving camera sequences the CT algorithm is more suitable while in static camera sequences the MS or FoT algorithms can be used as long as the target is not occluded. In future work, it is planned to propose a unified method which would combine the three tracking techniques all together to provide a full adaptation to the target and the context.

So far we have only talked about single object tracking applications in static and moving camera configurations. Next chapter introduces a multiple object tracking method that analyzes the dynamics of the targets and models their appearance using covariance matrices. Beforehand, some improvements to the covariance descriptor color and texture distinctiveness capabilities are proposed.

Introduction

As stated in Sections 1.2.5 and 2.2.4 covariance matrix representations offer a very practical alternative that allows us to build models based on local image correlations between different image cues (e.g., intensity, directional gradients, spatial coordinates, texture representations, optical flow, etc). Covariance matrices are compact and distinctive, these are highly estimated in tracking and matching applications. Even-though covariance matrices are compact, the computation time required to calculate distance between samples tends to be high and grows exponentially with the number of features. The reason is that the Riemannian distance computations require complicated matrix algebra operations. This limitation may have a direct impact on the required time for matching.

One of the objectives of this chapter is to find texture and feature operators and the best set of combinations that enhance the robustness and distinctiveness of the covariance descriptor while keeping its size as small as possible with the intention of minimizing not only the required storing space but the computation time. The new sets of texture and color features proposed are evaluated for three different applications with comparisons to state of the art methods. The set of texture features studied here make use of the popular local binary patterns (LBP) (described in Section 1.1.3) proposing a new feature combination which is suitable to embedding it inside the covariance matrix descriptor and avoiding the problems that other naive similar approaches have found. In turn, the color features studied here are based on the color constancy theory and the Gaussian color models introduced in Section 1.1.1. The objective is to robustify the covariance descriptor by enabling it to handle illumination and color saturation changes using color invariants.

The other objective of the chapter is to introduce an original multiple-object tracking method that minimizes a combinatorial discrete energy function based on the likeliness of many different target trajectories and their congruence with their respective appearance models (based on covariance matrices). This algorithm is discussed at the end of the chapter in Section 4.4.1.

Discriminant texture information in covariance matrices

To improve their texture discriminant power Pang et al. proposed their GRMC (Gabor Region Covariance Matrix) which results from collecting the Gabor filter bank responses (view Section 1.1.3) into the set of features used to build a covariance matrix descriptor [START_REF] Pang | Gabor-based region covariance matrices for face recognition[END_REF] and [START_REF] Tou | Gabor filters as feature images for covariance matrix on texture classification problem[END_REF]. The inconvenience of GRMC is the large number of features it uses (each one for each filter). GRMC is not only expensive to compute but to match. Some publications replace Gabor filters and include local binary patterns: the local binary covariance matrix (LBCM) of Guo and Ruan [START_REF] Guo | Facial expression recognition using local binary covariance matrices[END_REF] and the Gabor-LBP based region covariance descriptor (GLRCD) of Zhang and Li [START_REF] Zhang | Gabor-LBP based region covariance descriptor for person re-identification[END_REF]) are just two examples.

In the rest of this section an enhanced local binary covariance matrix descriptor (ELBCM), after this, it is evaluated for three different applications: texture classification, facial expression classification and object tracking.

Enhanced local binary covariance matrices (ELBCM)

The right way to introduce textural information in the form of local binary patterns into the covariance matrix descriptor is not obvious. GLRCD uses a direct embedding of the LBP's decimal value which can be very unstable for the case of local neighbourhood rotations. Additionally, carrying typical arithmetic operations with such values is meaningless (i.e. adding

or averaging two or more LBP decimal values does not mean anything in terms of texture). LBCM uses each bit in the LBP pattern as an independent feature forming a string of P bits. This is more stable and doing arithmetic independently for each bit in the string is well defined. The problem is that the number of features inside the covariance matrix grows with the number of P bits considered by the LBP P,R operator. This has a significant impact on the execution speed. To improve this situation, an enhanced LBP-based covariance descriptor (ELBCM) is proposed in this section, the ELBCM descriptor uses the angles defined by the uniform LBP patterns presented in Section 1.1.3.

In the case of the LBP 8,R operator there are 58 possible uniform patterns, but the strings formed by all ones and all zeros are ignored because their angle representations are the same, which can can be misleading. Figure 4-1 shows the remaining 56 different patterns used to construct our ELBCM covariance descriptor when this operator is applied. The starting and ending angles θ 0 and θ 1 are marked by the curved arrows inside each pattern. Since computations have to be made on circular quantities, the angles are converted to points on the unit circle, e.g., θ 0 is converted to (cosθ 0 , sinθ 0 ). The LBP (x, y) operator provides the pair of angles (θ 0 , θ 1 ) at (x, y) to v. Local neighbourhoods with non-uniform LBP patterns are ignored and their feature vectors do not contribute to the covariance matrix computation. Because v is a four dimensional vector, the total dimension of ϕ(I, x, y) is d = 7 (the resulting covariance matrices are of size 7 × 7).

The use of trigonometric formulas in (4.2) should make ELBCM computation more time consuming than the other LBP-based methods, but this problem completely overcome using a look-up table that maps directly the 56 ULBP decimal numbers to the sine and cosine values of θ 0 and θ 1 .

ELBCM has multiple advantages in comparison to previous covariance-based texture descriptors. It is more compact (7 components of ELBCM vs. 11 components of LBCM), more stable i.e. it is less affected by small rotations that reflect as changes on the angles θ 0 and θ 1 described by the ULBP pattern while for LBCM the same rotations irregularly affect the value of the bits in LBP(x, y) feature vector depending on their position. An example describing this behavior is presented in Figure 4-2. An additional problem of LBCM is observed in practice where non-positive-definite matrices appear more frequently. A possible explanation for this phenomenon is that bit-strings in the LBP(x, y) operator can be very sparse in smooth regions. An additional advantage of the ELBCM descriptor is that its size is completely independent of the number of P neighbors used in the LBP pattern operator. This property can be very important for many different texture analysis applications such as classification, facial expression recognition and tracking. The ELBCM descriptor is evaluated for a variety of tasks in Section 4.1.2 to confirm its superiority in comparison to other covariance-based configurations and to compare its performance against other state-of-the-art methods. To our knowledge, L 2 ECM T uzel [START_REF] Li | Local log-euclidean covariance matrix (L2ECM) for image representation and its applications[END_REF] remains as the best method for this dataset.

Facial expression classification

To evaluate the performance of our covariance descriptor in the facial expression recognition problem we used the JAFFE dataset 1 which consists of 213 images from Japanese female individuals, covering seven expression categories (neutral, anger, disgust, fear, happiness, sadness ans surprise). To compare with [START_REF] Guo | Facial expression recognition using local binary covariance matrices[END_REF] we followed their same methodology: images were automatically cropped to align the eyes, then they were resized to 100 × 100 pixels and 21 samples of each expression were used for training and only one training sample for testing. The experiments were repeated several times and their averages are displayed in Figure 4567. ELBCM is followed by Gabor GRCM [START_REF] Zhang | Gabor-LBP based region covariance descriptor for person re-identification[END_REF] and LBCM [START_REF] Baker | Lucas-kanade 20 years on: Part 1[END_REF][START_REF] Alahi | FREAK: Fast Retina Keypoint[END_REF] [START_REF] Tou | Gabor filters as feature images for covariance matrix on texture classification problem[END_REF] as the best adapted descriptors for this problem. Tuzel's descriptor [START_REF] Tuzel | Region covariance: A fast descriptor for detection and classification[END_REF] offers very poor performance for this dataset. It was not possible to compare ELBCM to the other methods used in the texture classification experiments as results for this dataset are not publicly available.

Object tracking

Using the covariance matrix as a global region descriptor, we compare ELBCM descriptors with other configurations:

Tuzel's [START_REF] Tuzel | Region covariance: A fast descriptor for detection and classification[END_REF], LBCM [START_REF] Tou | Gabor filters as feature images for covariance matrix on texture classification problem[END_REF] and LRCD [START_REF] Zhang | Gabor-LBP based region covariance descriptor for person re-identification[END_REF] (LBP decimal value without Gabor filters configuration). The initial trackers position is marked in the first frame by hand. We implemented a very simple tracker that is limited to fixed-scale targets and which follows targets locally (i.e. if targets are located far from the trackers positions they are not found) testing different random locations taken from a 2D Gaussian distribution centered at the target's previous location and variances proportional to the target dimensions. Using the similarity of the tested locations against the target model as weights, a weighted average is calculated to estimate the current target location. Three different sequences are discussed here and what interest us is the precision at which the different feature combinations follow the targets and their tracking times.

When ground-truth information is available precision is measured using following weighted error

w error = ∥C gt -C tracked ∥ R (4.3)
where C gt and C tracked are the ground truth and tracked centers respectively, and R is the target size, calculated as the average of the vertical and horizontal bounding box lengths.

• Jumping sequence: This sequence (313 frames, size: 352×288 ) appears in gray-scale, it shows a guy jumping rope in a garden, a building and some trees are behind. The object of interest is the guys face. Due to the jumping speed and random camera movements images appear extremely blurred. After frame t = 41 Tuzel and LRCD confuse the guys face with the background losing the target and never being able to re-catch it again. In contrast, LBCM and ELBCM follow the target until the end. In some frames and due to the shaking of the camera, the guys face appears extremely blurred and trackers drift considerably from the real location, but LBCM and ELBCM are able to catch it again as soon as the target appears reasonably clear again. This sequence comes from the TLD dataset [START_REF] Kalal | Tracking-learning-detection[END_REF]. Results

for this sequence are summarized on the first column of Table 4.1, because LRCD matrices are the smallest (4 × 4)

they are the fastest ones but less precise. The best descriptor in terms of performance is ELBCM which is two times faster than LBCM and more precise in terms of the w error .

• Panda sequence: This sequence (3000 frames, size: 312 × 233 ) was filmed in color and comes from TLD dataset too [START_REF] Kalal | Tracking-learning-detection[END_REF]. It shows a panda enclosed in a park. At the beginning of the sequence the camera zooms in and out while the panda moves. The panda is a non-rigid target and its appearance changes a lot throughout the sequence: it turns on itself and its shape and color distribution drastically change. In addition, the panda passes behind a tree, and the background changes. All methods follow the target correctly until frame t = 1001 when it is no longer visible, then it re-enters to the scene at t = 1182 LRCD, LBCM and ELBCM are able to re-identify it and track it until frame t = 2656 when the camera zooms in and the target size changes significantly affecting the appearance, all these events are observable in Figure 4-11 and 3-11a. Execution time and w error results are shown on the second column of Table 4.1. While LRCD is very fast it oscillates a lot, in turn, ELBCM offers a good balance of precision and speed.

• Pedxing3 sequence: Used in [START_REF] Leibe | Coupled detection and trajectory estimation for multi-object tracking[END_REF], this sequence (188 frames long from t = 7484 to t = 7672, size: 640 × 480)

was registered in color. The target is a pedestrian who is crossing the street, and passes behind a panel. Due to compression noise and other artifacts the quality of the image is very poor. All methods but ELBCM and LBCM fail to catch the pedestrian after he passes behind the panel. Tracking results are given on the third column of Table 4.1. There is no publicly available ground-truth information for this target, so the target was labeled using Vatic2 [START_REF] Vondrick | Efficiently scaling up crowdsourced video annotation[END_REF] to offer a similar comparison to the other sequences, the resulting ground-truth file can be downloaded from my personal website 3 . Tuzel are able to recover it, the target is followed correctly until the camera zooms in (around t = 2450) here only ELBCM is able to recover the target and follow it until the end of the sequence t = 3000.

Discriminant color information in covariance matrices

The relevance of color to improve the discriminant power of the covariance descriptor is discussed and evaluated here.

Many feature combinations that include color inside the covariance descriptor are possible, the most simple approach just replaces the luminance component with the individual color channels (e.g., R, G and B) or by a color invariant. To avoid increasing unnecessarily the number of features used to construct the covariance matrix, spatial-gradients are not computed for each color channel individually, the luminance gradient or texture operator is preferred instead as described in the previous section.

Four different feature combinations are used to represent color information inside the covariance matrices: RGB, HSV, Guassian color models discussed in Section 1-4 and by the L1 invariant presented in Section 3.1.1. Thus the set of colors to test is {RGB, HSV, Gaussian, L1}. Gradient or textural information is given by the luminance gradients (I x and I y ) or by the local binary operator used by ELBCM. Table 4.2 shows the resulting list of feature combinations. As in the previous section, the performance offered by the covariance descriptors on this table was evaluated for texture classification and object tracking applications. Table 4.2: Feature combinations.

Color space

Feature set ϕ(I, x, y)

RGB grads [ x y R G B |I x | |I y | ] HSV grads [ x y H S V |I x | |I y | ] Gauss grads [ x y Ê Êλ Êλλ |I x | |I y | ] L1 grads [ x y L1 |I x | |I y | ] ELBCM RGB [ x y R G B cos(θ 0 ) sin(θ 0 ) cos(θ 1 ) sin(θ 1 ) ] ELBCM HSV [ x y H S V cos(θ 0 ) sin(θ 0 ) cos(θ 1 ) sin(θ 1 ) ] ELBCM Gauss [ x y Ê Êλ Êλλ cos(θ 0 ) sin(θ 0 ) cos(θ 1 ) sin(θ 1 ) ] ELBCM L1 [ x y L1 cos(θ 0 ) sin(θ 0 ) cos(θ 1 ) sin(θ 1 )
]

Texture classificacion using color-texture covariance matrices

Here we repeat our texture classification tests using the color feature combinations of Table 4.2. Unfortunately the Brodatz and the JAFFE datasets used in the previous section contain gray-scale images only. So, only the KTH-Tips dataset was used here. Figure 4-13 shows the classification accuracy that was obtained for this dataset using all the enlisted feature combinations, it also includes the accuracy results for the L 2 ECM T uzel features of [START_REF] Li | Local log-euclidean covariance matrix (L2ECM) for image representation and its applications[END_REF] which was the best feature combination for the non-color tests of the previous section (see Figure 45). The L1 grads feature set is clearly inferior when compared to other color feature combinations for this dataset. However, it still obtains better results that the Tuzel feature combination and it is comparable to the LBP-based ELBCM Lum results of the previous section (see Figure 45). Mixing the LBPbased texture information with the L1 color invariant clearly improves the results as the ELBCM L1 feature set behaves much more better than L1 grads and obtains slightly better results than the RGB grads . Among the feature sets based on the gradients, HSV grads is the one which obtains the best results. In general ELBCM HSV and ELBCM RGB get the best results, they are comparable to the state-of-the-art results offered by L 2 ECM T uzel .

Tracking using color-texture covariance matrices

Two sequences were evaluated here, they analyze two extreme cases, one where the target is highly saturated and where its color is very distinctive in comparison to the environment (i.e. the background and other people around the target), and a second sequence where the target initially appears low-saturated and where lighting changes occur altering significantly the appearance of the target.

• Pets 2009 sequence: The effect of color in the covariance descriptor was verified by selecting a highly colored pedestrian from the PETS2009-S2-L1-View01. The target is a girl wearing a highly saturated red jacket. Color here proves to be valuable as the pedestrian traverses many obstacles, crosses other pedestrians along its path, leaves and later re-enters the scene. 4.2 only L1 grads and ELBCM L1 resist the effects that the illumination changes have on the appearance of the target. L1 grads follows the target throughout the sequence while ELBCM L1 loses it before t = 100 and recovers it many frames after (about t = 520) the reason is that low illumination produces few valid LBP for the ELBCM descriptor. The evolution of the precision error w error for all the feature combinations is plotted by the curves in

Target re-identification with covariance matrices

The multiple target tracking problem and its relatives (e.g., trajectory analysis, activity recognition) are even more difficult when targets have to be detected and re-identified from a second camera and with considerable differences in illumination and perspective and elongated separations in time.

Background clutter, appearance instabilities, illumination and scale changes are just some of the typical difficulties found by the existing algorithms that have tried to solve this problem. In addition, when targets have non rigid motion or low textural and structural contents the gradient or corner-based methods such as the classical KLT [START_REF] Baker | Lucas-kanade 20 years on: Part 1[END_REF] or SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] are not appropriate. Kernel-based methods like Mean-shift [START_REF] Comaniciu | Mean shift: A robust approach toward feature space analysis[END_REF] are usually well adapted to such objects since they rely on global statistical distributions. The price to pay is a decrease of discriminant power, therefore several attempts have enhanced the method by background subtraction [START_REF] Jeyakar | Robust object tracking using local kernels and background information[END_REF], color-space switch [START_REF] Laguzet | Automatic color space switching for robust tracking[END_REF] and by using a spatio-colorimetric histogram [START_REF] Birchfield | Spatiograms versus histograms for region-based tracking[END_REF]. Bak et al. proposed in [START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF] a dense grid of covariance matrices to handle these problems [START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF] together with the so called Mean Riemannian Covariance (MRC) matrices which are used to blend the appearance information from multiple samples.

µ = arg min C∈M N ∑ i=1 ρ 2 (C, C i ), (4.4)
the value of µ is calculated iteratively following a Newton gradient descent method adapted for Riemannian manifolds.

The approximate value of µ at step t + 1 is

µ t+1 = exp µt [ 1 N N ∑ i=1 log µt (C i ) ] , (4.5)
where, exp µt and log µt are specific operators uniquely defined on the Riemannian manifold. Equations (4.6) and (4.7)

express how to calculate them

Y = exp X (W ) = X 1 2 exp(W )X 1 2 , (4.6)
Y = log X (W ) = log(X -1 2 W X -1 2 ). (4.7)
Bak et al. achieve great re-identification rates using a dense grid of of MRC matrices. For the case of human signatures, each image is scaled into a fixed size of 64 × 192 pixels where a grid of overlapping 16 × 16 pixel size cells is constructed.

Neighboring cells are separated by 8 pixel steps. In total, 161 MRC are used to construct the human signature. To reduce the re-identification computational cost we propose a different arrangement of MRC matrices. Images are re-scaled to 96 × 128 pixels, then, rings of concentric rectangles are formed around the image center with exponentially increasing areas allowing some area overlapping. The proposed pattern is inspired in FREAK and DAISY [START_REF] Alahi | FREAK: Fast Retina Keypoint[END_REF][START_REF] Tola | DAISY: An efficient dense descriptor applied to wide-baseline stereo[END_REF] but for rectangular covariance regions, these computations are performed fast thanks to the integral image method.

In our configuration, a total of 43 MRC descriptors were employed, Table B.1 lists of the coordinates used to describe pedestrians (all samples have been resized to 128 × 96). Most of the rectangles are concentrated at the center and have a smaller size, more variability is tolerated at the periphery using bigger rectangles. To further simplify things MRC descriptors are compared one to one in contrast to [START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF] where comparison is maid sliding one grid against the other.

For the case ELBCM-based descriptors a different pattern of descriptors was used, the reason is that some regions are very small and the risk of not finding valid uniform LBP patterns when calculating the ELBCM covariance descriptors gets higher for these regions. To correct this problem, a different pattern of matrices which uses larger image regions was used, it is composed by the complete image (just using a small border to compute the LBP operators) and 4 quadrants. The list of five areas considered is given in Table B.2.

Target re-identification experiments

To validate our re-identification method the cumulative matching characteristic (CMC) of [START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF] and [START_REF] Gray | Evaluating appearance models for recognition, reacquisition, and tracking[END_REF] is used. CMC represents the percentage of times the correct identity match is found on the top n matches. Tests were done on the ETHZ [START_REF] Schwartz | Learning Discriminative Appearance-Based Models Using Partial Least Squares[END_REF] and PETS'09 L1-Walking-Sequence 1 [START_REF] Ferryman | Pets2009: Dataset and challenge[END_REF] datasets.

ETHZ dataset is composed by images from three different sequences (each one formed by 83, 35 and 28 individuals). Each individual, is captured by the same camera which suits just fine to our single-camera tracking objectives. For this experiment, eight different individuals from PETS'09 L1-Walking-Sequence 1 were extracted taking discontinuous samples.

For each individual, 10 images were selected from the beginning ant the end and their MRC matrices (subsection 4.

were calculated. The recognition rate was tested, taking random images and comparing against the registered signatures.

Care was taken to avoid reusing any of the images occupied during signature calculation. Success is declared when the corresponding image identity is found inside the top n list.

To find out which feature combination is more discriminant, experiments were executed using the same feature combinations of the previous section listed in Table 4. Features combinations based on the computation of directional gradients are denoted in continuous lines, dashed lines denote feature combinations based on the ELBCM operator. In most of the cases, features based on the ELBCM operator get a better score. Color information is very important to improve the discriminant capability of the descriptor, features based on luminance or in the color invariant get lower scores. However, in this experiments it is still not clear which color space is more discriminant, the most consistent one is probably the HSV color space that appear very well ranked for all the different dataset sequences.

The obtained re-identification rates are comparable to the ones reported in [START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF] employing a 75% less covariance matrices and fewer components inside them.

Multiple object tracking

Multi-target tracking is considerably more challenging than single object tracking. Due to target inter-occlusions, appearance similarity between targets and other phenomena related to the presence of multiple targets in a scene, a lot more is required to solve this problem than just the mere application of single object tracking methods multiple times.

One of the great difficulties faced by multiple target tracking, is the state-space cardinality the algorithm is forced to deal with. The possible number of trajectories targets can follow is incredibly high (discrete case) or even infinite (continuous case). In addition, when objects suffer considerable changes in scale inside the tracking zone, the possible number of states grows even more. By introducing some dynamical constraints in the linear and angular velocity it is possible to reduce the number of possibilities into a physically plausible set trajectories, locations and scales.

The preponderant role that appearance information plays in the context of object recognition and tracking has been previously discussed. Information in the form of color or texture can be very useful to follow a target or to re-identify it after an occlusion. But taking advantage from this information is not always straightforward as many well known problems: occlusions, scale, illumination, appearance changes and background clutter contamination can drastically reduce the expected performance. Target representations need to be invariant and robust to all such phenomena, but unfortunately most color invariants, although robust against lighting changes, can lead to a reduction in the separability between targets increasing the number of matching ambiguities. To address the multiple object tracking problem a solution based on the tracking-by-detection paradigm is proposed.

Multiple object tracking algorithm description

As a consequence of the continuous increase in the performance and speed offered by current object detection methods (HOG and classification in Riemannian manifolds discussed in Section 2.1), multi-target tracking methods based in the tracking by detection strategy [2] are becoming very popular. Andriyenko and Schindler propose in [START_REF] Andriyenko | Multi-target tracking by continuous energy minimization[END_REF] and [4] the use of discrete/continuous energy functions that evaluate the cost associated to a set of locations based on an overall energy function which considers target's dynamical properties such as linear and angular speed, trajectory persistence (preventing and penalizing unexpected target displacements) and target detections/disappearances occurring far from the tracking area borders.

Total Bregman Divergence (TBD) for appearance modelling

The possibility of including appearance information inside the energy function was left open in [START_REF] Andriyenko | Multi-target tracking by continuous energy minimization[END_REF]. This idea is recovered here and enriched with the efficiency and robustness provided by covariance target representations.

As stated in Section 1.2.5, the set of covariance matrices lies in the Riemannian manifold formed by the set of symmetric positive definite matrices (SPD). A variety of metrics and divergences have been proposed in the literature to perform element by element comparisons on this set. Currently, the most popular metric for SPD matrices is the Riemannian metric of equation (2.30) (see details in B.0.6). Quite recently the Jensen-Bregman LogDet Divergence [START_REF] Cherian | Efficient similarity search for covariance matrices via the jensen-bregman logdet divergence[END_REF] has gained interest because it is cheaper to compute and because it enjoys some interesting robustness properties. The Mean Riemannian Covariance (MRC) discussed in Section 4.3 blends together multiple observations of the same target into a general model. This MRC operator has many disadvantages: it is expensive to compute, there is no closed form definition (it is calculated iteratively) and it is affected by outliers. The geometric median [START_REF] Fletcher | Robust statistics on riemannian manifolds via the geometric median[END_REF] (Section B.0.7.4) is less sensible to outliers but it is expensive to compute and its existence is not completely guaranteed. More recent articles ( [START_REF] Cherian | Efficient similarity search for covariance matrices via the jensen-bregman logdet divergence[END_REF] and [START_REF] Liu | Shape retrieval using hierarchical total bregman soft clustering. Pattern Analysis and Machine Intelligence[END_REF]) propose other dissimilarity measure/divergences that alleviate these difficulties.

The total Bregman divergence (TBD) [START_REF] Liu | Shape retrieval using hierarchical total bregman soft clustering. Pattern Analysis and Machine Intelligence[END_REF]

δ T BD (P, Q) = log (det (P -1 Q)) + tr(Q -1 P ) -d 2 √ c + (log (det Q)) 2 4 -d(1+log 2π) 2 log (det Q) , (4.8)
where

c = 3d 4 + d 2 log 2π 2 + (d log 2π) 2 4
is a robust and efficient dissimilarity measure. The TBD and ℓ 1 -norm based centre Q (known as t-centre) robustly represents a set of covariances {Q i } n i=1 as where

Q = ( n ∑ i=1 w -1 i Q i ) -1 , (4.9)
w i = ( 2
√ c + (log (det Qi)) 2 4 -d(1+log 2π) 2 log (det Q i ) ) -1 ∑ j ( 2
√ c + (log (det Qj )) 2 4 -d(1+log 2π) 2 log (det Q j ) ) -1 . (4.10)
Equations (4.8) and (4.10) may look complicate, but they are much more economical than Riemannian metric based alternatives because they have an analytic expression and only require the calculation of matrix multiplications and determinants (logarithms in these equations are scalar logarithms and not matrix logarithms), in contrast to the Riemannian-metric alternatives which are based on complicated calculations that require the computation of matrix eigenvalues among other expensive calculations.

Discrete energy modelling for multiple object tracking

The aim of our multi-tracking algorithm is to consistently detect, identify and trace object locations trough time. Ideally, the number of trajectories by object is exactly one, but inter-object occlusions, disappearances i.e., background occlusions, and false negatives in the detection method may cause trajectory drifts and fragmentations. At every time frame t the algorithm receives a set of detections

D t = {d 1 , • • • , d m }
and the set of active paths (tracklets)

P t = {p 1 , • • • , p n }.
Each tracklet p j (indexed by j), contains historical information about the object ∀t ∈ {t start , • • • , t end } where t start is the frame where p j was created and t end the last frame where information about p j is available. Tracklet p j 's location at t is denoted as x t j and its appearance description (covariance matrix) as C t j . From these appearance descriptions the appearance model Cj is calculated using the TBD t-center as in equation (4.10). A linear fitting function (i.e. least-squares) applied at time t, provides us the series of predicted locations xt j , ∀t ∈ {t end + 1, • • • , t end + N f } for every active path p j ∈ P t (N f defines the length of the predictions).

Frame by frame the algorithm evaluates all possible matches between detections d i and tracklets p j . New paths are created when it is required. At the end, the algorithm outputs a dynamic mapping set M t containing the final set of tuples M t = {(d i , p j )} that minimize a discrete energy function. This idea is illustrated in Fig. 4-25-a where a set of points is marking locations of the detections (old detection are marked in a lighter red). In Fig. 4-25-b these detections have been grouped into trajectories (tracklets) tracing the evolution of object locations in the scene. The energy function E(d i , p j ) rewards plausible configurations and penalizes unreasonable ones. Our energy function includes most of the terms presented in [START_REF] Andriyenko | Multi-target tracking by continuous energy minimization[END_REF]: a detection term based on the image detection data E det , a physically motivated term which models the object dynamics (linear and angular velocities) E dyn , an appearance term E app that measures the similarity between the appearance of the target at the current location and compares it against tracklet's p j covariance model. The last term considered by this energy functions penalizes path creations (or disappearances) favoring persistent and continuous trajectories E add . All the aforementioned terms are accumulated as it is expressed in the following equation

E(d i , p j ) = E det detection confidence + E dyn dynamic model + E app appearance similarity + E add persistence model . (4.11)
Figure 4-26 shows how each one of this terms collaborates to detect the more plausible configurations.

Detection model: Objects are detected with a sliding window that measures a classifier response using HOG or classification in Riemannian manifolds (Sections 2.1.1.1 and 2.1.3.2). The energy term E det uses the predictions xt j inside p j . Energy is lower when xt j passes trough a region of high pedestrian likelihood:

E det (x t j ) = λ + D(t) ∑ g=1 -c ∥x t j -d g ∥ 2 + c (4.12)
In equation (4.12), D(t) represents the number of detection peaks of frame t and d g is the location of peak g. The value of λ penalizes predictions xt j lacking of image evidence. Dynamic Model:: The motion term E dyn assumes a constant velocity model

E dyn = α∥v t j -vt+1 j ∥, (4.13)
where v t j = ẋt j = x t j -x t-1 j is the current velocity vector of path p j , and vt+1 j is estimated considering the detection location d i as vt+1 j = d i -x t j . Appearance model: Every detection d i ∈ D t has covariance descriptor Ĉi which is compared against each p j ∈ P t through their associated t-center Cj . In the case of strong similarity the energy function is awarded with a negative value -γ, if not the energy functions grows in proportion to the TBD divergence βδ TBD as

E app =      βδ TBD ( Ĉi , Cj ) if δ TBD ( Ĉi , C j ) > δ max -γ if x < δ max . (4.14)
Persistence Model: When the algorithm find correspondences between detections and paths, it also considers the possibility of a new object entering the scene. This hypothesis is penalized in proportion to the Euclidean distance between the detection d i ∈ D t and the tracking area borders. 3. keep a consistent track of each object over time, 4. assign a unique track identity and stay on it throughout the sequence.

The evaluation criteria proposed by CLEAR metrics is a set of mechanisms that enable the judgment of the precision of a tracker to determine the exact location of a target and reflect its ability to track consistently through time. They evaluate how good an algorithm is at tracing each one of the target trajectories and producing as few trajectories as possible per object.

For every frame t, a multiple object tracker outputs a set of hypothesis {h 1 , • • • , h m } for a set of visible objects At each time t the evaluation procedure follows these steps:

1. Establishes the best correspondence between h j and o i .

2. For each correspondence the error in the object's position estimation is computed.

3. Measures all correspondence errors which are accumulated as:

(a) the total number of objects for which there is no output hypothesis (missed objects), (b) the total number of tracker hypothesis for which no real object exists (false positives), (c) the total number of identity changes or mismatch errors caused by swaps as two or more objects pass close to each other, or when an object track is reinitialized with a different track ID after an occlusion.

The performance of a multiple object tracking algorithm is thus expressed with two numbers: (i) the tracking precision which measures how well the exact positions of the objects are estimated, and (ii) the tracking accuracy which counts how # of matching errors at frame t. MT (Mostly tracked) % of ground-truth trajectories covered for more than 80% of the sequence. ML (Mostly lost) % of ground-truth trajectories covered by a less than 20% of the sequence. PT (Partially tracked)

1.0 -M T -M L. MOTP M OT P = ∑ i,t d i t ∑ t ct . MOTA M OT A = 1 - ∑ t (mt+f pt+mmet) ∑ t gt . PF
Track fragmentations. IDS Identity swaps.

many mistakes the algorithm made in terms of false negatives (misses), false positives, mismatches and track recovery failures (swaps). Those metrics are proposed in a very intuitive way as:

1. The multiple object tracking precision (MOTP):

MOTP = ∑ i,t d i t ∑ t c t . (4.15)
which is the total error in estimating the position for object-hypothesis matches pairs over all frames, averaged by the total number of matches made (i.e. or the alignment error between the ground truth and the tracker output).

This measure is independent of the skills of the algorithm to recognize object configurations and keep consistent trajectories, 2. The multiple object tracking accuracy (MOTA):

MOTA = 1 - ∑ t (m t + f p t + mme t ) ∑ t g t (4.16)
where g t is the number of objects while m t , f p t and mme t correspond to the number of misses, false positives and of mismatches, respectively, all of them for time t.

Other evaluation metrics used for testing were taken from [START_REF] Li | Learning to associate: Hybridboosted multi-target tracker for crowded scene[END_REF]. The mostly tracked (MT) value represents the number (or percentage) of targets tracked correctly in more than 80% of the target appearance duration in the sequence, mostly lost (ML) targets are those tracked correctly for less than 20% of their duration in the sequence and partially tracked targets (PT) are those targets not considered by any of the other two categories. Table 4.3 summarizes all the evaluation metrics and terms discussed thus far.

Multiple object tracking evaluation results

To evaluate our approach we tested it four with widely used real world datasets as well as many other sequences of recorded in the laboratory. One of the tested sequences is PETS 2009-S2L1-V1 from the VS-PETS2009 benchmark 4 where only the first viewpoint is used. The other three datasets come from the TUD5 dataset: TUD-Campus, TUD-Crossings and TUD-Stadtmitte. Ground truth is available for all these datasets. All the videos resulting from these experiments are available 6 .

The experimental results are reported in Table 4.4, compared to the state-of-the-art results referenced in Table 4.5, they are not as accurate or precise as Milan et al. [START_REF] Milan | Continuous energy minimization for multi-target tracking[END_REF] or Berclaz et al. [START_REF] Berclaz | Multiple object tracking using k-shortest paths optimization[END_REF], but the advantage of the method proposed here resides on its ability to on run the fly, estimating the object trajectories frame by frame contrasting with [START_REF] Milan | Continuous energy minimization for multi-target tracking[END_REF] that requires the whole bunch of frames to estimate the optimal number paths and their most plausible trajectories. [START_REF] Milan | Continuous energy minimization for multi-target tracking[END_REF], KSP "k-shortest paths" [START_REF] Berclaz | Multiple object tracking using k-shortest paths optimization[END_REF], EKF "Extended Kalman Filter" [START_REF] Milan | Continuous energy minimization for multi-target tracking[END_REF], PF "Particle lter based tracking-by-detection" [START_REF] Michael D Breitenstein | Online multiperson tracking-by-detection from a single, uncalibrated camera[END_REF], HDA "Hierarchical data association" [START_REF] Hofmann | Unified hierarchical multi-object tracking using global data association[END_REF], CO "Continuous energy minimization" [START_REF] Andriyenko | Multi-target tracking by continuous energy minimization[END_REF] and DCO "Discrete-Continuous Optimization" [4].

Sequence

Conclusions

This chapter presented the second and final group of contributions to the computer vision state-of-the-art. The ELBCM texture descriptor [START_REF] Romero | Enhanced Local Binary Covariance Matrices ELBCM for texture analysis and object tracking[END_REF] is the first contribution, it proposes a more convenient way to embed LBP and texture information inside the covariance matrices than other descriptors based on a similar principle (e.g. LBCM [START_REF] Guo | Facial expression recognition using local binary covariance matrices[END_REF] and GLRCD [START_REF] Zhang | Gabor-LBP based region covariance descriptor for person re-identification[END_REF]). The HSV or Gaussian models. The utility of the L1-based descriptors resides on its resistance to brightness and saturation changes. RGB and HSV color spaces worked better than the Gaussian color models and their performance improves when combined with the texture information that ELBCM provides. In both experiments ELBCM RGB performed slightly better than the other feature combinations (except on David sequence where brightness and saturation changes occur).

Section 4.3 presented an object re-identification method [START_REF] Romero | Covariance descriptor multiple object tracking and re-identi cation with colorspace evaluation[END_REF] which is appropriate to assist multiple object tracking applications specially to handing off targets captured on different cameras (i.e. multi-view configurations). The discriminant power of the covariance descriptor was increased by calculating multiple covariance descriptors (i.e. forming an array or pattern) for each sample and averaging this set of covariance matrices for all the samples belonging to the same target.

The multiple object tracking algorithm presented in Section 4.4 represents the last contribution of this thesis related to the computer vision field. The method presented here is not as accurate nor precise as the state-of-the-art methods proposed by Milan et al. but is capable of executing on-the-fly and in real-time.

In addition to the computer vision contributions presented up to this point, this thesis proposes a series of software optimizations and algorithm transformations to accelerate the execution of the covariance matching algorithm and adapt it to different computer architectures all theses aspects are treated in the next chapter.

Introduction

Theoretical advances in mathematics, machine learning and signal and image processing domains together with improvements in electronics and computer architecture technology such as the ever increasing computing power attributed Moore's law (that states that the number of transistors on integrated circuits is doubled approximately every two years), or by trend imposed by parallel architectures are fueling the development of mobile technology applications in different domains such as automotive, health, telecommunications and many others. These applications involve a large diversity of algorithms e.g.

2D and multi-spectral filtering, pattern recognition, segmentation and classification, cryptography and communications.

The development process of all these algorithms is often divided into three sequential steps: the theoretical study of the algorithms, the study of the target architecture and last but not least the implementation.

Algorithms are as much a technology as every other aspect that makes computers work. In many cases, choosing the right algorithm is more relevant than being able to buy the latest and fastest hardware, or as Cormen puts it in [START_REF] Cormen | Algorithms Unlocked[END_REF]:

"Even with the impressive advances that we continually see in computer hardware, total system performance depends on choosing efficient algorithms as much as on choosing fast hardware or efficient operating systems. Just as rapid advances are being made in other computer technologies, they are being made in algorithms as well.". In the same sense, the speed difference between a poorly implemented algorithm algorithm and a nice hardware implementation that considers all the relevant aspects and details about a particular target architecture can be of many orders of magnitude. Algorithmarchitecture adaptations holistically observe the interactions between algorithm analysis aspects (number of arithmetical operations, memory loads and stores, etc) together with the targeted platform execution characteristics (instruction timings, memory capacities, buses speeds) and other particularities related to the chosen architecture such as vectorized instruction sets and multi-threaded execution to obtain sufficient levels of performance. In other words, application developers and software system architects need not only to select the appropriate algorithms to use, but also the means to implement them on hardware eliminating unnecessary operations and stalls and intelligently distributing the computing load between the different sub-elements of the architecture to minimize runtime overhead.

The core algorithm for modeling the appearance of the targets in this thesis is the covariance descriptor, the objective of this chapter is to describe the most crucial challenges and particularities that appeared when implementing and optimizing this algorithm on a variety of desktop processors (e.g., 4-core Intel's Penryn, 8-core Nehalem and 4-core SandyBridge architectures) and on low-power processors suitable for embedded platforms such as Intel's ULV U9300 and ARM Cortex-A9.

This chapter is subdivided in two sections, Section 5.1 opens with a short review of basic computer architecture concepts: the memory hierarchy, memory timings and hardware characteristics, the different levels of parallelism that exist, Flynn's taxonomy and Amdahl's law, the concepts of temporal and spatial data locality and some of the properties that a cache conscious algorithm should meet. Readers familiar with all these concepts can go directly to the second part of the chapter, (Section 5.2) which is dedicated to the analysis and optimization of the computation of the covariance matrix descriptor for parallel architectures. The integral images method used to accelerate the computation of the covariance descriptor (in order to scale the algorithm to an unlimited quantity of descriptors computed on a single image), is evaluated first (Section 5.2.2) to determine the numerical accuracy constraints that we must consider when implementing the covariance computation algorithm. After the numerical correctness of the algorithm is asserted, the baseline implementation of the algorithm is described and benchmarked to determine the most critical bottlenecks for this implementation. Then, the rest of the chapter describes the code and data layout adaptations necessary to vectorize and optimize the algorithm execution.

Short review of parallel computer architectures concepts

Targeted architectures description

This thesis formed part of the ITEA/SPY European project. The aim of this project was to create a new automated, intelligent surveillance and rescue framework adapted for mobile environments. Most of the algorithms developed for this thesis were designed for portable smart camera applications, this constraint pushed the participants to select an embedded multi-core general purpose processor (GPP) member of the ARM Cortex-A series. Because it is very suitable for embedded targets, C programming language was preferred over other high-level languages (C allows to interact so directly with the hardware that it is sometimes classified as a low-level programming language).

The development cycle was the following: algorithms were first implemented and tested on desktop computers (running a Linux environment), once the desired algorithms were validated they were ported to the selected embedded architectures (Intel U9300 and ARM Cortex A9) care was taken to avoid the use of libraries not existing on all the platforms. By respecting this practices, the resulting code is extremely portable, code developed for GPP's remains very easy to design and maintain in contrast to other architectures such as graphic processing units (GPU's), digital signal processors (DSP's) and programmable logic electronics such as programmable logic devices (PLD's) or field-programmable gate arrays (FPGA's).

Intel+SSE and ARM+Neon

There are different ways (an levels) to represent the organization of a computer. The instruction set architecture (ISA) is the model which is closest to the programmer, it enlists the native data types, the instructions, the registers, the addressing modes, the memory architecture, the set of interruptions, how exceptions are handled, the buses to input and output (I/O) peripherals, etc. The micro-architecture (µ-arch) details how the instruction set architecture (ISA) is implemented on a processor, different architectures can implement the same ISA using significantly different semiconductor technologies and different interconnections between the operational elements, variations can go from single gates and registers to complete arithmetic logic units (ALU's) and even more complex or larger elements.

It is not necessary to be a computer architect to develop good quality and efficient code. From the point of view of the programmer, it is only required to have a simple (but clear) mental representation of the organization of the hardware resources being used, most GPP architectures share the concepts of cores, arithmetical and logical units, vector units and cache memories. Figure 5-1a shows an sketch of a typical multi-core processor, every core contains multiple functional units each one designed to perform some type of arithmetic operations, cache memories are used to manage the trade-off that exists between memory speed and capacity.

Each processor has a central clock which controls the pace at which the semiconductors (e.g. gates, functional units) pass from one state to the next. Depending on the properties of the electronics components (i.e. the transistors that form the processor) the maximum allowed operational frequency of the processor clock changes. The frequency of a processor is important as it determines the duration of the instruction cycle. For so many years, the exponential increase on the frequency of a CPU resulted in a free speedup of numeric software. From 1973 to 2003 microprocessor clock rates increased by three orders of magnitude (from 1 MHz to 1 GHz). Legacy code in some sense benefited automatically of the improvements of speed of computer architectures. However, in the last years the perception is that this trend is languishing and that the era free-speedup for legacy code is coming to an end [START_REF] Chellappa | How to write fast numerical code: A small introduction[END_REF]. Since 2005 CPU frequencies have stalled around 2 to 3 GHz and many factors are limiting the growth of achievable performance of single cores: the first important factor is related to the observed non-linear growth of power consumption as the clock rate increases which at the level computers are now turns out to be completely unacceptable (at 130W air cooling systems are no longer practical), clock rates stopped climbing because the power growth needed to be arrested and because of the emergence of mobile and embedded computing where the need for low-power consumption is much more important than it is for desktops and servers. A second problem is posed by the famous discrepancy that exists between the speed of the processor and memories. This problem renders the increase of the clock rates fruitless because many applications are fundamentally bounded by the memory performance and not by computing (i.e. loading data from off-line chips could be more expensive than performing many arithmetic instructions).

To avoid this problem, algorithms need to be coded to avoid memory accesses (loads and stores) as much as possible. Most Nehalem desktop and mobile processors (e.g. Core i3, Core i5, and Core i7 processors) use DMI, but some Core i7 models and other non-consumer processors targeted for workstations and servers such as the Xeon Phi coprocessor1 use a newer technology from Intel known as QuickPath Interconnect (QPI) point-to-point link that similarly to DMI handles the interconnection between the processor and the Platform Controller Hub (PCH). In more complex instances, separated QPI links are used to connect one or more processors and one or more PCH hubs to form a network on the motherboard, this technology allows all its components to access each other and enable a non-uniform memory access architecture (NUMA).

DMI was originally designed to support 10 GB/s in each direction, QPI supports up to 25.6 GP/s.

The Intel ULV 9300 processor (from ultra-low voltage) belongs to the Penryn family which includes an extensive array of micro-architecture elements that boost performance across a broad range of software, this includes the Intel Streaming SIMD Extensions 4 (SSE4) that deliver further performance gains for SIMD (single instruction, multiple data) software (32-bit or 64-bit), the group of applications benefiting from this technology are: graphics, video encoding and processing, 3-D imaging, and gaming, image processing, etc. On the other side, the Cortex-A9 processor offers a good balance of computing power and energy consumption, its floating-point arithmetic unit executes single and double precision scalar operations two times faster than the preceding ARM generation, its Neon Media Processing Engine allows the programmer to accelerate media an signal processing functions using the ARM Neon Advanced SIMD instruction set which supports the execution of rich SIMD operations over 8, 16 and 32 bit floating-point data quantities every cycle.

Registers and cache memories

Due to the enormous disparity that exists between the speed of the processor and the memories, computer architectures are commonly structured in a hierarchical way. Their structure is given by processor registers, multiple cache levels, main memory, network, hard-drives and other peripherals. Figure 5-1a shows a sketch or abstract representation of a typical multicore processor. Inside every core we have the functional units, each one is charged of performing a particular arithmetic operation (they are the basic unit of computation). Registers are the units of memory which are closer to the functional units, they can be used to feed the functional units with operands. CPUs have very few registers and each one offers very few bytes of space. Caches are next type of memories in the hierarchy, they are slightly slower than the core registers but they offer more space. Caches are separated in multiple levels, each level being slower but larger than the one above it. In multi-core systems, only the lowest cache levels are shared between the cores, caches are organized by storage blocks called cache lines. A cache line is typically much larger than the processor's word length and usually larger than a vector register. Common cache line sizes are between 64 bytes and 128 bytes. SIMD registers in SSE are 128-bit wide that is 16 bytes wide, a cache line of 64 bytes length (such as Nehalem) is 4 times longer than the SSE vector register.

Cache lines are used to populate the cache memory every time a data is read from memory, the objective of transferring the entire line instead of just data by data is to amortize the overhead of setting up the transfer. Modern day CPUs include circuitry that analyzes the memory access patterns and prefetches data and instructions to the cache, but for being effective, memory accesses should not jump around indiscriminately in memory so that the extra data read into the cache is fully exploited. Peak memory access performance is obtained only when memory is accessed following a predictable sequence of addresses, so that the program makes full use of the line transfers and prefetching data mechanism are efficient.

Caches are much smaller than main memory, they use SRAM (static random access memory) and they are significantly faster than other types of memory used for main memory such as DRAM (dynamic random access memory). The most important limitations of SRAM is that they have a much smaller capacity per unit area and that they are more costly, large on-chip cache memories currently are on the order of 8 to 20 MB. So, cache memories cannot contain all the code and data of the executing program at a given time, so there is a lookup table in the cache that is used to respond to a memory reference, a separate D-cache is used for fetching data and a I-cache to fetch instructions, if the memory reference is mapped in the cache, then we have a cache hit and the instruction or data can be sent right away to the processor, otherwise we have a cache miss and the next levels are recursively probed until the missing item is found. Cache misses are an important source of delays that usually take between 20 to 100 clock cycles, this is comparable to the magnitude of time required to perform more than 100 floating point operations, ideally, a large number of on-chip compute operations should be performed for every off-chip memory access, thus a good indicator of the performance of an algorithm is obtained by measuring the ratio of computation vs. the number of memory accesses, this ratio is commonly called the arithmetic intensity.

The hit time expresses the time it takes for a cache memory to bring a line in the cache memory to the processor, this time depends on the hierarchy of the cache and for the case of multi-core architectures if the cache memory is shared or not with the other cores. Table 5.1 briefly describes the required time to access a data line depending on the hierarchy level of the memory for the Intel Nehalem processor [START_REF] Doe | Performance analysis guide for intel core i7 processor and intel xeon 5500 processors[END_REF], according to it, the additional time required when a cache miss occurs (depending on its level) can go from 60 ns up to 100 ns. For a remote DRAM access (data that is resident in the DRAM on a remote socket) the number of cycles can go up to 800 cycles instead of just 4 that L1 cache hit takes. The percentage of the memory accesses for instructions or data which are not found in the cache memories is often referred as the algorithm's miss rate (expressed as # misses/# accesses). For the L1 cache memory typical miss rate values range between 3% and 10% depending on the algorithm characteristics, the cache memories size and the memory layout. Similarly, the L2 cache miss rate values should be < 1%. If the greatest part of the memory accesses are covered by the cache memory (cached memory locations) the average latency of the memory accesses will be similar to the cache latency instead of the main memory latency. Table 5.1: Description of the required time to access a data line depending on the hierarchy level of the memory for the Intel Nehalem processor [START_REF] Doe | Performance analysis guide for intel core i7 processor and intel xeon 5500 processors[END_REF].

Memory access

Latency How data is stored in memory has a clear impact on the execution speed of an algorithm, this is due to the principle of locality which analyzes how a value or a set of related storage locations is accessed. What locality measures is how predictable an algorithm or system behaves, algorithms exhibiting strong locality are highly appreciated because they are strong candidates for performance optimization. There are two basic types of reference localities: temporal locality which refers to how often a specific data, address or any other type of resources is accessed within a certain period of time, and secondly spatial locality which refers to the use of elements stored in nearby locations in the near feature. A good locality means that the hardware can win its bet since the supposed prediction conditions are more likely to be true reducing the required communication. Each memory access pulls out an entire cache line around that memory location onto the chip and into the cache memories, using the contents of that line repeatedly while it is still residing in the cache memory is faster than pulling those contents from base memory multiple times. Suboptimal data layouts in memory are a common reason for weak data locality and consequently of poor performances. Thankfully, some simple transformations such as: loop re-arrangements and data layout changes may help to improve the spatial locality, in all cases it is worth to keep in mind that communication is very expensive and computation very cheap that sometimes it is better to increase the required amount of work in exchange of reducing communication.

Instruction sequencing is improved using some of the following code adaptations: loop interchange, loop fusion, loop unrolling and loop unwinding. Examples of these transformations are provided in the Appendix C for consulting. A different nature alternative is to use a technique called buffering, which consists in copying data into contiguous temporary buffers to overcome cache misses due to cache associativity, when working on multi-dimensional data structures such as matrices and images, logically close elements can be far from each other when mapped to a linearized memory. Cache

• Data level parallelism: Data parallelism consists in distributing the data across different computing nodes and computing the same operations on multiple datum simultaneously, there should not exist any dependency between the input data and the results obtained for other data elements. Data parallelism allows to modify multiple values simultaneously using the same modification function. Some implementations of this type of processing are obtained by vector processing using SIMD instructions. Data parallelism is more oriented around data rather than the execution of multiple tasks concurrently, it is closely related to the width of the hardware registers and the number of data elements that can be processed simultaneously following a single control flow. In data parallelism the same flow of control is applied on multiple data elements.

• Thread level parallelism (TLP): This type of parallelism occurs when a program is separated into independent tasks and each task can be performed in parallel without dependencies between each other. In single threaded cores each thread is executed in a separate core and each one keeps its own state (e.g. program counter, state registers) necessary to execute independently their own stream of instructions. Using multiple instruction streams improves the execution times of multi-threaded programs.

For long, programming was done by modeling computers as if they were serial machines, meanwhile, computer architects and compiler designers worked hard to find ways to automatically extract ILP parallelism, this helped to keep things simple for the programmers and allowed them to ignore the true parallelism that existed in hardware but most computer architects believe that these techniques have reached a limit. New processor generations need to avoid their dependency on the rise of clock rates or on automatic IPL mechanisms.

Flynn's taxonomy

The way processors combine control flow and data management define the type of parallelism available on different architectures, Flynn presented in [START_REF] Michael | Some computer organizations and their effectiveness[END_REF] a classic categorization of processor architectures depending on whether they have multiple streams of data, multiple flows of control or both.

• Single Instruction, Single Data (SISD): The standard and typical non-parallel uniprocessor.

• Single Instruction, Multiple Data streams (SIMD): A single operation is executed simultaneously on multiple elements of data by broadcasting the operations on multiple data paths. SIMD processors are also known as array processors.

• Multiple Instruction, Single Data (MISD): no such machine exists.

• Multiple Instruction, Multiple Data (MIMD): This processors execute separate instruction streams, each with its own flow of control and operating on different data. This model characterizes the use of multiple cores integrated in a single processor, or multiple processors in a single computer and multiple computers forming a cluster. When different processors of different architectures are present in the same computer, we have a heterogeneous computer.

Vector processors appeared in the 1970's and were the basis of most supercomputers through the 1980's and the 1990's.

Thanks to the growth of the registers data width today's most commodity CPU's implement architectures featuring vector

Amdahl's law

In regular situations a sequential algorithm need to be redesigned from scratch due to the effects of Amdahl's law which says "... the effort expended on achieving high parallel processing rates is wasted unless it is accompanied by achievements in sequential processing rates of very nearly the same magnitude". This law is useful to measure the maximum expected improvement to an overall system when only part of the system is improved. When applied to parallel computing, this law allows us to estimate the theoretical maximum speedup to be attained when using multiple workers. Improvement is limited by the time needed to execute the strictly serial fraction of the algorithm which cannot be parallelized (t seq ∈ [0, 1]).

The time required by the sequential and scalar version of the algorithm t 1 on a single processor is t 1 = t par + t seq where t par represents the parallelizable fraction of the algorithm. Given p ∈ N processors to do a parallelizable work we have that: where the acceleration (speedup) S p for a parallelism degree of p is:

t p = t
S p ≤ t 1 t p ≤ 1 τ seq + 1-τseq p ≤ 1 τ seq , (5.2)
the reason to use the less than or equal to symbol is that the maximum speedup is only reached when the parallilizable work can be perfectly parallelized.

Considering now what happens when p → ∞ where the maximum speedup is bounded by τ seq as S ∞ = 1 τseq . Hence, the best parallel algorithm might not be the one which minimizes the total amount of computational work, but the one which reduces to the minimum the time it takes to perform the longest chain of tasks that must be performed sequentially or the span of the algorithm. Figure 5-4 shows one of the first implications of Amdahl's law, as the degree of parallelism p grows the parallelizable work is distributed and the span of the algorithm converges to the size of the serial work that cannot be distributed. Figure 5-5 plots the achievable speedups that can be obtained depending on the parallelizable portion of the algorithm.

Figure 5-4: The expected speedup is limited by the non-parallelizable serial portion of the work (deep blue). Image taken from [START_REF] Mccool | Structured Parallel Programming: Patterns for Efficient Computation[END_REF].

GPUs and other offload devices

An important recent trend is the development of attached processing units such as graphic accelerators and co-processors specialized in highly parallel workloads. Graphic Processing Units (GPUs) are promoted by their manufactures as Single Instruction, Multiple Threads (SIMT) architectures (a term coined by Nvidia), a category which corresponds to a tiled SIMD architecture formed by multiple SIMD processors. On a vector machine such as Cray1 or the modern MMX, SSE and AVX incarnations, there is a single processor or individual cores sequencing the instructions and some instructions are purely scalar executing on 8 to 64 bit data, but other instructions are vector and operate on multiple values (e.g. 512 bits of data can be interpreted as 16 single precision floating point numbers). In SIMT processors, each lane of vector computations can be interpreted as a separate thread, and a single hardware sequencer operates on a group (or block) of such threads allowing the parallel execution of thousands of them. A single instruction fetch is broadcast to all the individual processing elements, if a thread branches to a different directions, they are marked as diverged. The sequencer uses a mask to identify the threads that have diverged and then it fetches instructions and distributes it to all the processing elements enabled by this mask. Coherent code needs to be run on those blocks to maintain the efficiency of the entire block high.

GPUs where originally designed for graphic applications particularly those with large amounts of fine-grained parallelism and exhibiting high coherence (a divergent control flow can significantly reduce the efficiency within a block) and as they have relatively small on-chip memories, latency is hided using an extremely large number of active threads.

Running computations on a co-processor or an accelerator is referred as offload processing, GPUs are not the only type of offload device available, other types of devices based on different technologies such as field programmable gate arrays (FPGAs) or co-processors formed by many-core processors exist. A nice example of the last type is given by the Intel MIC (Many Integrated Cores) architecture, which contains more cores than traditional CPUs (over 50) and allows the developers to create platforms running at trillions of calculations per second using the fast and familiar Intel Xeon processor and the Intel Xeon Phi coprocessor, each core has wide vector units able that are able to execute 16 simultaneous single-precision floating point operations, the only condition is to have good data locality on the implementation algorithm because the cache size by core is very small (L1 32K instruction and data caches and L2 512K cache per core). Other semiconductor companies are now commercializing many-core processors systems e.g. Tilera with its Tile GX processors family 3 and Kalray with their MPPA products which can be used either as a stand-alone embedded solution or as an acceleration co-processor of a host CPU.

Offload devices (such as GPUs) are usually located on the PCIe bus they do not share memory with the host and data must be transferred across the PCIe bus to a memory local to the offload device before it can be processed. To alleviate this problem, some manufacturers have begun to integrate those units into the same die as the main processor cores, this is the case of AMD and Intel. NVIDIA (the most important GPU manufacturer) also makes integrated CPU/GPUs using ARM cores for the embedded and mobile markets. The advantage is that memory is physically shared by the GPU and CPU processors.

OpenMP

A single-threaded program can be converted easily into a multi-threading one without worrying about the API's environment variables and other small details using OpenMP. OpenMP is an API specification created in 1998 by a committee of vendors to allow and easy specification of shared-memory concurrency in Fortran, C and C++ programs. Todays most important compilers support OpenMP that includes GNU GCC compiler, Intel C/C++, Microsoft Visual C/C++, Clang and LLVM.

So it works on almost every modern operating system such as Windows, Linux and OSx. Different versions the API specification exists so programmers are suggested to verify first which versions are supported by each compiler.

OpenMP consists of API's pragmas and several other settings to control OpenMP-specific environment variables.

Pragma directives offer a way for each compiler to offer machine and operating system-specific features while retaining compatibility with the Fortran, C and C++ languages. According to the ANSI C and C++ standards, if a compiler does not recognize a given pragma, it must ignore it, this makes completely safe to place OpenMP pragmas in code without worrying if the code will compile on a different toolset. When a master thread arrives to a declared parallel region (by an OpeMP pragma), it forks a specified number of slave threads to divide a task among them (using a fork-join model).

Parallel regions are declared to control how code is assigned to threads, the default behavior makes all threads within the team wait at the end of the parallel region until all other threads have finished running before being joined back again so that the main thread can resume the execution of the code in sequential mode. This behavior can be modified but needs to be explicitly stated by the programmer.

Initially, OpenMP does not guarantee how many threads will be created, usually, it choses a number equivalent to available number of execution pipelines. On standard multiprocessor system this number would be the number of cores available, but systems equipped with Hyper-Threading Technology, the number of pipelines is twice the number of processors.

All OpenMP pragmas are of the form #pragma omp. This directive is adapted by more optional clauses that modify the meaning of the construct, for example, a parallel region is defined using the parallel construct to form a single construct (i.e. #pragma omp parallel).

When the master thread encounters this pragma, a team of threads is forked and the single statement or the block OpenMP offers many other pragmas used to identify code blocks to threads, to control the scope of the variables to be shared across threads or to keep local inside the individual threads, to specify barriers and force the synchronization of threads, how to schedule the tasks (for task-level parallelism) or loop iterations to threads and so forth.

Memory layouts and data reorganizations to enforce spatial locality

Code transformations are not the only possibility to improve performance, a different alternative is to modify the memory layout of the referenced data. Changes in data layout affect how data structures are declared and addressed, this consideration is even more important for parallel computing because a reorganization of the data may help obtain the optimal performance in the vectorization.

An array containing a contiguous collections of data items is one of the most common and well-known data structures.

The usual approach to data abstraction is to declare structures to represent the properties and states of an object. Instances of an object are created in runtime and organized in collections of that structure. These type of collections are popularly known as Array of Structures (AoS) layout. The problem with this layout is that it does not align well for vectorization or caching.

An alternative layout is to create a collection of each element or state in the structure, this layout is known as Structure of Arrays (SoA). Vectorization in this layout is normally easier since the same function is replicated over every element (or groups of elements) in the elemental array. This type of processing pattern is popularly known as map, the function being replicated is called the elemental function since it is applied to all the elements of a collection, usually producing a Figure 5-6: Map pattern, a function is applied to all the elements of a collection, usually producing an output with the same shape as the input. Image taken from [START_REF] Mccool | Structured Parallel Programming: Patterns for Efficient Computation[END_REF].

R 0 R 1 R 2 R 3 G 0 G 1 G 2 G 3 B 0 B 1 B 2 B 3 R 1 R 2 R 3 G 1 G 2 G 3 R 0 G 0 B 0 B 1 B 2 B 3
Figure 5-7: Array of structures (AoS) versus structure of arrays (SoA). The SoA form is typically better for vectorization. However, in some circumstances such as when data is accessed incoherently, AoS may lead to better cache utilization.

new collection with the same shape as the input (see Figure 56).

To illustrate the differences between both data structure organizations an example is proposed: suppose that we can compare the AoS and SoA arrangements for storing the r,g and b components of a set of points in an RGB image, AoS requires the declaration of an array formed by multiple RGB elements (see Listing 1). In contrast, the SoA implementation is composed by a set of arrays each one containing an individual component and example is provided in Listing 2. A loop visiting all the components of an RGB point before moving to the next point using the AoS arrangement exhibits a good data locality of reference because all elements in the lines fetched to the cache memory are utilized. In contrast, a loop that just visits one component of all points (say the red component in our example) has a less satisfying locality of reference because many of the elements in the fetched cache lines are not used. Furthermore, a common disadvantage of the AoS arrangement is that each individual reference in a loop exhibits a non-unit stride access pattern, additionally, this type of arrangements may not be very vectorization friendly due to the non-sequential access patterns that may produce long instruction latencies.

The selected type of arrangement may have an important impact on the performance of the application. An hybrid arrangement can be created as an alternative to AoS and SoA, this approach is known as Hybrid Structure of Arrays (HSoA).

For the RGB images example, the HSoA arrangement is shown in Listing 3. Listing 3: HSoA data layout suitable for SIMD manipulation, here the vector cardinality is four.

R 0 R 1 R 2 R 3 G 0 G 1 G 2 G 3 B 0 B 1 B 2 B 3 R 1 R 2 R 3 G 1 G 2 G 3 R 0 G 0 B 0 B 1 B 2 B 3 R 5 R 6 R 7 G 5 G 6 G 7 R 4 G 4 B 4 B 5 B 6 B 7 R 4 R 5 R 6 R 7 G 4 G 5 G 6 G 7 B 4 B 5 B 6 B 7 R 0 R 1 R 2 R 3 R 4 R 5 R 6 R 7 G 0 G 1 G 2 G 3 G 4 G 5 G 6 G 7 B 0 B 1 B 2 B 3 B 4 B 5 B 6 B 7
In the HSoA arrangement, the SoA arrays are divided forming an array of structures containing n/m-elements, each instance of this structures contains a set of sub-arrays of width m for each of the items in the AoS arrangement, the value of m is often referred as the cardinality of the SIMD register.

The differences between all three approaches is easily visualized in Hand optimization of the data structures is often required, the baseline or out of the box version of an application is often developed first and only after the application executes the algorithm correctly and produces the expected results profiling is run to detect bottlenecks. Respecting this procedure is very important to save us from the arduous and fruitless job of optimizing irrelevant structures and sections of code by hand. Some of the most popular and robust profiling tools available are VTune, Gprof and Oprofile. It is worth emphasizing that the SoA layout is not ideal in all circumstances, specially when accesses are random or incoherent, unneeded data might be brought into the cache.

Covariance tracking implementation

Much of the research work of this thesis formed part of the European project ITEA/Spy which demanded a robust real-time object tracking algorithm targetted to run on embedded low-power consumption systems. This section explores all the different techniques that were evaluated in this work to accelerate the computation of the covariance descriptor in order to achieve this objective.

Image processing algorithmic and architectural optimizations

Computer vision and image processing algorithms are often divided into three different abstraction levels:

• Low-level: They execute regularly and independently of the data stored inside the data arrays. For the case of image processing algorithms, the same operator is applied for all the picture elements or image locations irrespectively of their values. Algorithms of this type receive an input image and create and output a new one (or a set of them).

Examples: filtering algorithms (e.g. linear, no-linear, recursive, non-recursive) such as smoothing and gradient filters.

• Intermediate-level: Algorithms of this type are applied to analyze the information contained on the input data arrays (e.g. pixel values and their neighborhoods for the case of image processing), to extract semantic information: feature point, contours, background subtraction and object detectors. This type of image processing algorithms usually receive one or more input images and produce high-level (or structure) information.

• High-level: These algorithms are charged of providing an interpretation and taking decisions based on the information provided by low-level and intermediate-level algorithms. Typical computer vision algorithms of this type are: object tracking, activity recognition, scene understanding, etc.

Most of the research on algorithm-architecture adaptation for image processing applications is concentrated on the low and intermediate levels.

Integral images accuracy considerations

Before describing the proposed optimizations for the parallelization and vectorization of the covariance descriptor algorithm, it is necessary to study its correctness limits related to the data-type precision used during the implementation of the algorithm. The fewer the bits used to represent the data, the more operations it is possible to perform in parallel using SIMD instructions, but there is a trade-off decision here because a reduction on the precision bits may cause a reduction of performance in the best case or meaningless descriptors in the worst scenario. Floating point data types facilitate the programming effort, but it is extremely important to be aware of its pitfalls.

In the covariance descriptor computation data-flow in The finite number of bits used to real numbers requires to accept just an approximation of the true value. The precision of a number is measured in terms of the number of digits that contain meaningful data (referred as the significant digits).

Depending on the type of data being used our algorithms are able to accumulate a different number of elements without incurring in accuracy errors:

• Large summations are prone to run out of bits very fast, if a naive serial algorithm is used the partial sum can grow very large in comparison to the new values to be added to it and if the new values to add are less than the smallest representable increment the summation will not have any effect on the accumulator and the increments are rounded to zero. This is equivalent to ignoring some pixels during the integration process which obviously conduces the algorithm to wrong results.

Line 8 in Listing 4 shows how all the information from one summand is lost because the summand a is much more larger than b. This phenomenon might be difficult to see when summing large sets of floating-point numbers because usually such pronounced differences are no seen between the individual terms, but errors can still grow from the accumulation of many small errors. Listing 4 is an eloquent example of how precision is quickly lost when the magnitude of the two numbers to be added is significantly different. In summation area tables (such as the integral images are), one of the operands represents the accumulated term so far and in general it can be much larger than the terms being added to it. If the data spans over an extreme range of floating point values say from 2 -100 to 2 100 for example, storing the data with fixed point precision would require 200 bits per pixel. Neither a single nor a double precision valued integral image is sufficient to hold these values without error. This problem is usually disregarded because usual data rarely span such a large ranges of precisions.

Integral images are applicable in most cases, but single precision integral images need to be handled with care and avoided unless we are very sure of not incurring in precision errors. The first stage is by far the most consuming one, here, the set of image features

F (fmat[k][i][j] in code) is accessed
one-by-one for a particular spatial location (i, j). As the set of features are located different images, there is no spatial locality, nearby locations in a particular feature image are loaded to fill the cache lines, but these values are immediately discarded and prefetching hardware is underutilized. This obviously traduces into large amounts of cache misses that cause several CPU stalls that increase the application running time.

Three different methods are projected to optimize the covariance tracking CT running time using desktop and embedded processors with parallel architectures. In rough terms they are based on:

1. a data layout transformation (SoA→AoS), 2. architectural optimizations such as:

• multi-threading the SoA version with OpenMP middle-ware,

• using SIMD instructions (SSE/AVX for Intel, Neon for ARM), 3. a loop-fusion and serialization transformations code transformation.

SoA→AoS transformation

The goal of SoA→AoS data layout transform (Structure of Arrays to Array of Structures) consists in transforming a set of independent arrays into one single array where each cell is a structure combining the elements of each independent array.

The contribution of such a transform is to leverage the cache performance by enforcing spatial and temporal cache locality.

The first aspect we want to optimize is the locality of the features for a given point of coordinates (i, j). In the SoA version we have two cubes: one that stores all the pixel features F SoA (fmat in code) which size is n F × h × w and a different cube P SoA (prmat in code) of size n P × h × w) that stores the feature crossed-products. In the AoS data layout these cubes are transformed into two 2D-arrays F AoS and P AoS of size h × (w • n F ) and h × (w • n P ). Listing 6 is useful to illustrate the differences between both layouts by displaying the C macros used to navigate through these arrays in both implementations, it is worth to mention that the spatial coordinates x and y are handled in code by the indexes j and i and that the index k is used to select a particular feature.

The SoA→AoS transform consists in swapping the loop nests and changing the addressing computations from a 3D-

form cube[k][i][j] into a 2D-form like matrix[i][j × n + k],
where n is the structure cardinal (here n F or n P ). The lack of spatial locality within the features in the SoA representation is illustrated in

P [k][i][j] ← F [k1][i][j] × F [k2][i][j] 7 k ← k + 1
Algorithm 17: Products of features -AoS version

1 foreach i ∈ [0..h -1] do 2 foreach j ∈ [0..w -1] do 3 k ← 0 4 foreach k1 ∈ [0..nF -1] do 5 foreach k2 ∈ [k1..nF -1] do 6 P [i][j × nP + k] ← F [i][j × nF + k1] × F [i][j × nF + k2]
process is depicted in Figure 5-13. Two 128-width registers allows us to represent eight 32-bit integers or single precision values, for the case of a feature vector composed by n F = 7 features (composed by F

0 , F 1 , • • • , F 6 ) an eight feature (F 7 )
is appended as a padding element but it is not included in the feature combinations used to calculate the crossed products. Seven vectors contain the 28 feature products and the 7th vector is 100% filled, but it will become sub-optimal if n P is not divisible by the cardinal of the SIMD register (4 with SSE and Neon). For SSE some of the factor permutations can be achieved using only one instruction, the others need a maximum of two instructions, and because some permutations can be re-used to perform other permutations, it is possible to achieve a factorization over all the required permutations. In the case of our example (n F = 7), only fifteen shuffles are required. In Neon things are more complex, and while some of the permutations can be done using 128-bits registers (giving us a degree of parallelism of 4

F 0 F 0 F 0 F 0 F 0 F 1 F 2 F 3 X X X X P 0 P 1 P 2 P 3 F 0 F 0 F 0 F 1 F 4 F 5 F 6 F 1 X X X X P 4 P 5 P 6 P 7 F 1 F 1 F 1 F 1 F 2 F 3 F 4 F 5 X X X X P 8 P 9 P 10 P 11 F 1 F 2 F 2 F 2 F 6 F 2 F 3 F 4 X X X X P 12 P 13 P 14 P 15 F 2 F 2 F 3 F 3 F 5 F 6 F 3 F 4 X X

Multi-thread implementation

Concerning our problem, two different strategies are possible to parallelize the covariance descriptor and tracking algorithm. One scheme would be to implement a task-level parallelism using OpenMP or Pthreads (POSIX Threads) (to create a pool of threads and distribute the set of targets among them). The second strategy uses loop-level parallelism: the operations on the images are divided into balanced groups of rows per thread. This approach is the more straightforward to apply using OpenMP since (as we will see next) it only demands to add some pragma lines just before the most consuming loops in our source code, this last approach is evaluated here. From this experiments we can conclude that although the effect of cache overflow is less dramatic for the multi-threaded SoA+OpenMP version (which can be attributed to the additional cache memory spaces are used when the data is distributed among the unshared core cache memories), the real improvement to the covariance descriptor computation algorithm was obtained by the SoA→AoS transformation combined with the code vectorization. For the rest of the chapter, the vectorization of code is the only architectural optimization considered as realistic for this algorithm.

In the next subsection, the last optimization to the covariance descriptor computation algorithm is given. It is a loop fusion transformation combined with other scalarization techniques to keep the intermediate and re-usable calculations in local registers to reduce the number of memory accesses loads and stores and increase the arithmetic intensity.

Loop fusion and scalarization

loop-fusion and serialization transformations is used here to increase the arithmetic intensity (AI) ratio by reducing the total amount of communication expressed in the form of loads and stores. Three AoS algorithm blocks are fused together into a single one:

• the first block is an instance of Algorithm 17 which computes from the n F first-order features the set of n P crossedfeature products (second order features), • the second algorithm is the first order instantiation of Algorithm 19 which computes the first order n F feature integral images,

• and finally, a second instance of Algorithm 19 is used to compute the n P second-order integral images.

Three different versions of the loop-fusion together with serialization transformations were tested, the first one is a scalar parametric version (for variable n F ) that fuses the external i-loops and keeps the three j-loops unchanged. The second one is a specialized version for the particular case of n F = 7 where the three internal loops are fused together too. The third one is the SIMDized version of the second one. The internal loop fusion allows to save in LOAD/STORE instructions in order to avoid writing the products of features into memory and to read them afterwards to compute the integral image of products.

In this particular case, loop-fusion and serialization transformations has been done by hand, but some tools like PIPS [START_REF]PIPS[END_REF] can do such kind of transformations automatically [START_REF] Irigoin | Semantical interprocedural parallelization: an overview of the pips project[END_REF]. The expected complexity of the transformations is reported in Table 5.10 for the scalar version, Table 5.11 reports the expected complexity for the vectorized SSE and Neon versions.

The transformed versions of the AoS and AoS+SIMD are from now on referred as AoS and AoS+T+SIMD respectively.

These series of transformation are the last optimization provided in this thesis to accelerate the execution of the covariance descriptor computation algorithm. As these versions of the algorithm were ported to the embedded platforms Intel U9300 After the transitions ends (before 400 × 400), the speedup drops to ×1.68 as the algorithm demands for this region ×1.69 more cpp's.

and ARM Cortex-A9 benchmark results for this series of transformation are provided in the following subsection.

Implementation on embedded systems

In this subsection the implementation results on embedded processors like the Intel ULV (Ultra Low Voltage) Penryn U9300 (that belongs to the Penryn family) and the ARM Cortex-A9 are provided. The average power consumption (TDP) of these processors is very low: about 10 W and 1 W respectively. Both run at very close frequencies (1.2 GHz and 1.0 Ghz) but there are significant differences. For example, the Intel ULV U9300 uses out of order execution to reduce the impact of costly delays while ARM Cortex-A9 does in order execution which means that it follows the strict order of the program assembly instructions and does not tries to execute independent instructions while some data is retrieved. Another important difference is that the SSE4.1 offers a throughput of one SIMD operation per cycle while all Neon instructions on the Cortex-A9 take two cycles. Concerning the acceleration, we can see (Tables 5. [START_REF] Bhatia | Positive Definite Matrices[END_REF] and 5.14) that the optimization of the kernel provides a speedup of ×2.9 for Intel and ×2.2 for ARM that assets the need of all optimizations. As both processors have two cores, all the processing parts can be done either on one core (the execution time is the sum of all parts) or on two cores (the biggest part is on one core and the two other parts are on the second core). With such a coarse grain thread distribution, the Intel U9300 can track targets in real-time for 640 × 480 images, while the ARM Cortex-A9 can do it for image sizes up to 320 × 240.

As said previously, there is a factor 10 for power consumption, and only a factor 4 in execution time. So for small images (up to 320 × 240) the Cortex-A9 is the best choice as it is real-time and more power efficient than the Intel. While for bigger sizes (up to 640 × 480), the Intel is the only choice for real-time implementation.

Before the optimizations, the kernel function of the covariance descriptor computation represented about 80% of the execution time on the Intel ULV 9300 processor. In the final AoS+T+SIMD version it represents 35% and the features computation algorithm represents now about 60% of the execution time on this platform. With respect to the ARM Cortex-A9, before the optimizations, the kernel represented about 73% of the total execution time, after the AoS+T+SIMD optimizations it only represents 45%. In both platforms, the most consuming part after the optimizations corresponds to the features computation algorithm, it is very likely that there is still plenty of room for more optimizations in this part, particularly in the integration of the features computation and the data structures used by the covariance descriptor computation algorithm.

Conclusions

This chapter has presented a real-time implementation of the robust ELBCM-based covariance tracking algorithm developed in the previous chapters. The covariance descriptor algorithm has a parameterizable complexity that can be adapted to the number and nature of features for trade-off between robustness and execution time. Classical software and hardware optimizations have been applied: SIMDization and loop-fusion and serialization transformations combined with the AoS→SoA transform to accelerate the kernel of the algorithm. These optimizations allow a real-time execution on heavy embedded systems like Intel U9300 and on light ones like the ARM Cortex-A9. The loop-fusion technique accompanied with other scalarization techniques represents provides the more important gains of performance, it is possible to use SIMD instructions together with this technique, the AoS+T+SIMD technique requires gets a speedup of about ×2 mixing these techniques, unfortunately, this doesn't work so well on the ARM Cortex-9 platform where the improvement on the performance is minimal. This algorithm will be ported soon to other platforms such as the ARM Cortex-15 and more other more energy-efficient Intel processors. As far as I know, this implementation of the covariance tracking algorithm is the first real-time implementation for embedded systems that keeps all its qualities and improves its robustness.

General conclusions and perspectives

This thesis has described our work in image processing and computer architecture. The aim was to propose methodologies for object re-identification and tracking with their implementation on multi-core embedded systems. This work has lead to several publications but also to a a real integration in the European project Surveillance imProved sYstem of ITEA2, coordinated by Cassidian (EADS).

Starting from a general overview on different features available at a pixel-level, the first chapter has introduced some of the possible object representations. Due to the large amount of methods, it was difficult to make a decision. Indeed, the representation has to be invariant against distortions such as illumination changes, noise or geometric changes while being well separated from the other objects of the scene and from the background.

The second chapter has presented a few algorithms commonly used for object detection and matching for re-identification and tracking. When only the object family to be detected is known (pedestrian, car, plane...), learning-based methods are commonly used. For re-identification and tracking, it is assumed that some information are available on a specific object of interest and that this object has to be retrieved in another context or in another frame of the same video. Concerning visual tracking, it is complicated to find an unified method which could be efficient whatever the nature of the object and whatever the context. In our work, we were more interested in three different methods:

• Flock of Trackers (FoT) which represents the object as a set of local areas which are tracked by KLT-like methods and robust statistics on the computed optical flow are used to provide the global motion vector. This method performs well but only for objects of rigid motion, which have enough texture.

• Mean-Shift tracking (MS) which represents the object by its color distribution and finds its new location in the video by optimization. This method is interesting because the distribution can be computed for any object, contrary to feature points.

• Covariance tracking (CT) which represents the object by a covariance matrix of feature vectors.

Our choice has been to study some color and texture features and to evaluate them in terms of quality, i.e. color invariance and precision, and computation times. Then, as an object representation, we have finally chosen covariancebased techniques for several reasons:

• it provides an unified representation for object detection and matching;

• it is compact with a fix size whatever the object size;

• it mixes heterogeneous information such as spatial, color and texture what makes it highly discriminant.

with a higher degree of confidence than with small windows. However, smaller windows are preferable for displacement estimations as it is less probable for them to cover regions with depths discontinuities.

The problem of determining the motion parameters is that of finding A and d that minimize the dissimilarity (matching (A.17)

Because of the Taylor series truncation (linearization), the linear system in (A.14) can only be approximately satisfied.

addition ax + by / ∈ M even when x, y ∈ M and a, b ∈ R. Here, the usual Euclidean calculus does not apply. The same happens for the case of conventional statistics. Analysis on nonlinear manifolds is required when certain natural constraints make the underlying space nonlinear. For example elements of R n with unit norm constraint, or matrices with orthogonality constraints. Using differential geometry of the underlying manifold, one can usually derive most of the corresponding results from Euclidean spaces.

Manifolds are sets that are locally Euclidean, there exists manifolds in different types of spaces: R, R n×n , the general linear group GL(n, R) (formed by n×n real invertible matrices together with their multiplication) or the set of symmetrical n × n matrices. One space of particular interest in this chapter is the space of positive matrices i.e., ⟨x, Ax⟩ >= 0 for all x ̸ = 0, (B.5)

where ⟨x, y⟩ represents the inner product between two vectors x and y.

For a manifold M and a point p in M , there is a tangent space associated to it. This space is formed by the set of velocity vectors of all curves passing through p on M . It is known that T p M is a vector space of the same dimension as M . Many problems in computer science and vision are solved first in T p M and these solutions are then projected back to M using a mapping technique known as the exponential map: exp p : T p M -→ M (see section B.0.5). There is an inverse to this mapping function that points from the manifold to the tangent space: exp -1 p : M -→ T p M . For every point p on the manifold M , and for every vector v 1 , v 2 in T p M , there is an inner product ⟨v 1 , v 2 ⟩ p . This metric varies smoothly across p, and the usual Euclidean metrics are considered according to the tangent space type e.g., ∥q -p∥ for R n , tr(A * B) for R n×n .

B.0.1 A quick Tour of Basic Differential Geometry

In 1827, Carl Friedrich Gauss published the mathematical results he obtained while measuring the distance between cities and other landmarks. Gauss showed [START_REF] Gauss | General Investigations of Curved Surfaces[END_REF] that surfaces can be analyzed extrinsically: relating them to their embedding in the Euclidean space (E 3 for the earth's surface surface case) or intrinsically: analyzing their properties which depend solely on the distance measured along curves on the surface. In the particular case of the Earth, this arises from the fact that the surface of a globe does not have the same geometry as the Euclidean plane. Today, a whole field of mathematics know as Differential geometry has evolved around this original idea. Differential geometry is the theory of curved surfaces, where a surface is described by Cartesian coordinates and then analyzed employing differential calculus. This theory is the cornerstone of many interesting applications like 3D Computer graphics and Computer stereo vision.

B.0.2 Differential geometry of surfaces

Differential geometry of surfaces revolves around the fact that a curved surface can be studied without reference to a higher dimensional Euclidean space. The mathematical concept which helps to analyze intrinsically the geometry of surfaces is called the Gaussian curvature of a surface, or often simply the curvature of a surface. Care must be taken to avoid confusion with the curvature of a curve. The curvature of a curve is an extrinsic geometric property, telling how it is bent in the plane, or in space. The surface of an sphere is considered to curve positively at every point in it because it bulges out in all directions. If we take any point on the surface of an sphere, there is a tangent plane touching the surface at that point only so that the rest of the surface lies all on one side except at that point. This property does not hold when considering a flat plane or the surface of a cone: for any point in them there are many other others which intersect on the tangent plane that corresponds to that point (the curvature is zero). Surfaces have negative curvature when given any point on the surface and its respective tangent plane, the surface is separated in two by the tangent plane at that point (there are points of the surface which lie on both sides of the plane).

B.0.3 Riemannian geometry

Riemannian geometry started as a very broad and abstract generalization of the differential geometry of surfaces embedded in E 3 . To each point on a curved surface corresponds a local coordinate patch on which a two-dimensional Euclidean coordinate system is defined: the tangent space in E 2 . Carrying on this idea, we arrive to the concept of the n-dimensional Manifold (denoted as M n ). In mathematics, manifolds are n-dimensional topological spaces where the neighborhood of each point resembles to E n (an space modeled on Euclidean spaces). A Riemannian manifold is a real smooth manifold M equipped with an inner product g p on each tangent space T p M that varies smoothly from point to point. A Riemannian metric (tensor) groups the family of g p inner products, providing the tools required to define other geometric notions on a Riemannian manifold such as lengths or curves, angles, areas, curvature, gradients of functions and divergence of vector fields.

B.0.4 Riemannian Metrics

According to [START_REF] Gallier | Lectures on geometric methods for computer science[END_REF], if (M, g) is a Riemannian manifold, then the concept of length makes sense for any piecewise smooth curve on M . It is possible to define the structure of a metric space on M , where d(p, q) is the greatest lower bound of the length of all curves joining p and q. Curves on M yielding the shortest distance between two points are called geodesics, they play an important role in many geometrical applications.

Given any point p ∈ M , for every vector v in its tangent space (v ∈ T p M ), the norm of v is defined by

∥v| = √ g p (v, v). (B.6)
The Riemannian inner product, g p (u, v), of two tangent vectors, u, v ∈ T p M is denoted as ⟨u, v⟩ p . Now, given a smooth parametric curve on M , which is a map from a closed interval [a, b] in R so that γ : [a, b] -→ M defines a smooth curve in M . This smooth curve can be regarded as a piecewise smooth curve from p to q iff there is a sequence a = t 0 < t 1 < • • • < t k-1 < t k = b of numbers t i ∈ R, so that each map, γ i = γ[t i , t i+1 ] defines a curve segment.

The set of all piecewise smooth curves that go from p to q is denoted by Ω(M ; p, q) or Ω(p, q), this set is an important object know as the path space of M from p to q. At any junction point, γ i-1 (t i ) = γ i (t i ) there may be a jump in the velocity vector of γ.

Given any curve, γ ∈ Ω(M ; p, q), the length, L(γ), of γ is

L(γ) = k-1 ∑ i=0 ∫ ti+1 ti ∥γ ′ (t)∥dt (B.7)
For every curve, γ, on M , let D dt be the associated covariant derivative along γ, also denoted ∇ γ ′ . A curve, γ : I -→ M is a geodesic if γ ′ (t) is parallel along γ, that is, iff This map takes a given tangent vector to the manifold, runs along the geodesic starting at that point and going in that direction, for a unit time. Since v corresponds to the velocity vector of the geodesic, the actual distance (in Riemannian terms) traveled will be dependent on that. A variation on the tangent vector v will get us different points on M resulting of the mapping exp p .

Dγ ′ dt = ∇ γ ′ γ ′ = 0 (B.

B.0.6 Riemannian Metrics on Positive Matrices

Let M n be a Hilbert space with product ⟨A, B⟩ = trA * B and the associated norm ∥A∥ 2 = (trA * A)

1/2
. The set of Hermitian matrices constitutes a real vector space H n in M n . The subset P n consisting of strictly positive matrices is an open subset in H n . P n is a differentiable manifold. The tangent space to P n at any point A ∈ P n is denoted as

T A P n = {A} × H n .
The inner product in H n leads to a Riemannian metric on the manifold P n . This metric is expressed by the differential The infimum in (B. [START_REF] Bhatia | Positive Definite Matrices[END_REF]) is attained at a unique path joining A to B. This path is the geodesic from A to B.

ds = ∥A -1/2 dAA -1/2 ∥ 2 = [ tr(A -1 dA) 2 ]
Let H(t), a ≤ t ≤ b be any path in H n and let γ(t) = e H(t) , then but this mean does not satisfy some expected properties and gives poor results. As we will see, all the desired properties are satisfied by the geometric mean, justifying the need of defining a suitable geometric mean of SPD matrices.

L(γ) ≥ ∫ b a ∥H ′ (
A mechanism to update the covariance model was presented in 2006 by Porikli et al. in [START_REF] Porikli | Covariance tracking using model update based on lie algebra[END_REF], but there are other options at our disposal. Most of these options are based on the estimation of the geometric mean. In this section a novel method based on the geometric median is introduced to protect the algorithm against outliers presence.

B.0.7.2 Binary Geometric Mean

Let R + be the set of positive numbers. A mean is a function m : R + × R + → R + that satisfies the following conditions Several problems arise for the list of conditions in (B.33). For example, matrix multiplication is non commutative, the order relation A ≤ B defined for the case when the difference B -A is positive definite does not necessarily imply

A 2 ≤ B 2 .
The arithmetic and harmonic means A + B 2 , While these matrices are positive, they do not obey properties (iii)' and (iv)' because the exponential map is not orderpreserving, and A → A t is order preserving if and only if 0 ≤ t ≤ 1.

( A -1 + B -1 2
) -1 (B.

The operation

A#B := A 1/2 ( A -1/2 BA -1/2
) 1/2 A 1/2 (B.37) satisfies all the desired properties (i)'-(v)'. These operation was presented in 1975 by Pusz and Woronowicz [START_REF] Pusz | Functional calculus for sesquilinear forms and the purification map[END_REF] and is valid for the binary case (two matrices problem) only. The extension for more than two positive matrices turned out to be a tricky problem which resisted solution for more than 25 years. The solution to this problem has been found recently, the approach (introduced in the next subsection) involves some differential geometry.

B.0.7.3 Riemannian Mean

The Riemannian barycenter center of mass of m elements A 1 , A 2 , • • • , A m is defined as provides us the value of this point.

G(A 1 , A 2 , • • • , A m ) =
In a Euclidean space the averages s j of a vector a 1 , • • • , a m are defined as

s 1 = a 1 s 2 = 1 2 (a 1 + a 2 ) s 3 = 2 3 s 2 + 1 3 a 3 = 1 3 (a 1 + a 2 + a 3 ) s k = k -1 k s k-1 + 1 k a k .
(B.40)

A procedure inspired in the Euclidean case can be ported to calculate positive matrices averages

S 1 = A 1 S 2 = (A 1 # 1/2 A 2 ) S 3 = S 2 # 1/3 A 3 S k = S k-1 # 1/k A k (B.41)
It is not possible to expect that S m would be the Riemannian mean G(A 1 , • • • , A m ). However, there is an adaptation of this idea which provides a sequence that converges to G.

Given a sequence of independent trials in which an integer is chosen from the set {1, 2, • • • , m} with equal probability.

Thus, a sequence of integers represented by I = {i 1 , i 2 , • • • }. Now, let {S k (I, A)} be the sequence of binary means

S 1 = A i1 , S 2 = S 1 # 1/2 A i2 , • • • , S k = S k-1 # 1/k A i k . (B.42)
In [START_REF] Sturm | Probability measures on metric spaces of nonpositive curvature[END_REF] it is shown that for almost all I its corresponding sequence {S k (I, A)} converges to G(A 1 , • • • , A m ). The Riemannian mean G(A 1 , • • • , A m ) is the limit of a sequence constructed from A 1 , • • • , A m by taking at each step a binary geometric mean.

One problem with this approach is that it is not deterministic, Holbrook solved this issue in [START_REF] Holbrook | No dice: a determine approach to th cartan centroid[END_REF], he has shown that it is possible to reach the barycenter G(A 1 , • • • , A m ) as a limit of a deterministic walk, for any X ∈ (P )(n) let φ(X) = As an example, choosing X = A 1 and r = 1, the sequence in (B.44) develops as

((((A 1 # 1/2 A 2 )# 1/3 A 3 ) • • • # 1/m A m )# 1/m+1 A 1 )# 1/m+2 A 2 • • • which converges to G(A 1 , • • • , A m )
. This is true, because the distance of φ r,n (X) from G is reduced after every m steps.

B.0.7.4 Riemannian Geometric Median

If M is Riemannian manifold and given some samples this gradient is not defined for x is any of the data points x i .

An iterative steepest descent method is used to find the Riemannian geometric median when it exists where n = x ′ • y ′ . Similarly and after some algebraic manipulations, the covariance of the region R((x ′ , y ′ ); (x ′′ , y ′′ )) where, Once all the terms required by equation (B.57) to obtain the gradient have been given, the target location at iteration i + 1 is obtained as

m k+1 = Exp m k (αv k ), v k = ∑ i∈I k w i Log m k (x i ) d(m k , x i ) • ( ∑ i∈I k w i d(m k , x i )
202 is C R((x ′ ,y ′ );(x ′′ ,y ′′ )) = 1 n -1 [ Q x ′′ ,y ′′ + Q x ′ ,y ′ -Q x ′′ ,y ′ -Q x ′ ,y ′′
∂ x P (x) = ∂ x (M -1 2 Y x M -1 2 ) = ( M -1 2 ⊗ M -1 2 ) vec(∂ x Y x ) = M -1 2 (∂ x Y x )M -
x i+1 = x i -η i ∇f (x i ) (B.63)
which is the SPD space steepest descent algorithm. The value of η i controls the step of the descent (learning rate), it can follow an annealing schedule expressed as η i = η 0 (1i/N ), where N is determined empirically. Iterations will continue until convergence, which occurs when ∥η i ∇f (x i )∥ < t conv .

Figure 0 - 1 :Figure 0 - 2 :

 0102 Figure 0-1: Mécanisme de coopération FoT+CT.

Figure 0 - 3 :

 03 Figure 0-3: (a) Ensemble des 56 motifs LBP uniformes dans un voisinage (8, R). Pour chaque motif, une flèche circulaire indique le début et la fin de l'angle (θ 0 and θ 1 ). (b) Dans la méthode ELBCM, un motif LBP défini par deux angles θ 0 et θ 1 est décrit par v(θ 0 , θ 1 ) = [cos(θ 0 ) sin(θ 0 ) cos(θ 1 ) sin(θ 1 )]. Le motif LBP associé à la valeur décimale 63 prend la valeur v(π/8, 5π/8) = [cos(π/8) sin(π/8) cos(5π/8) sin(5π/8)].

Figure 0 - 4 :

 04 Figure 0-4: Exemples de résultats obtenus sur la base de données PETS'09, montrant une ré-identification à différents instants et à différents points de vue.

  nal du vecteur (4 avec SSE, 8 avec AVX). Les permutations sont réalisées en une seule instruction pour certains d'entre eux, et en deux instructions dans le pire des cas. Parce que certaines permutations peuvent être réutilisées pour réaliser d'autres permutations, une factorisation est faite sur toutes les permutations nécessaires. Par exemple, avec n F = 7, quinze remaniements sont nécessaires. La quantité de mémoire requise par CT est égale à (n F + n P ) × sizeof(float)) × h × w octets. En supposant h = w = 1024 , n F = 7 et n P = 28 , l'algorithme a besoin de 140 Mo, beaucoup plus que la taille de la plus grande mémoire cache disponible ! Afin d'évaluer l'impact des optimisations, les trois versions de l'algorithme (SdM , M dS, M dS + SIM D) ont été comparées pour des tailles variant de 128 × 12 à 1024 × 1024. Les temps d'exécution obtenus sont reportés dans la tableau 2.

  Specific object recognition works by matching and geometric verification. In contrast, generic object categorization often includes a statistical model of their appearance and/or shape. Learning visual objects requires to gather a set of training images and to extract a model able to identify the presence and/or localization of instances in novel images. The type of training data that is required depends on the desired detail-level of recognition i.e., categorize objects present in the image, detect them with a coarse spatial localization, or to segment the pixels belonging each of the objects and the clutter from background.

  perhaps millions of values) without considering their spatial and time relations and constraints. In addition, individual pixel values are unstable and vulnerable to noise. Image processing and computer vision methods analyze pixel neighborhoods to detect changes and their distributions in space and time to study their statistical properties and identify how local patterns are composed. Furthermore, a group of pixels related in time and/or space deserves more confidence than isolated pixels because in most of the cases noise doesn't conspire and can be easily filtered out by local linear and non-linear operators. Depending on the type of sensor used to capture the image pixel values can represent a) illumination intensity (brightness), b) a wavelength in the electromagnetic spectra and c) the distance from the focal point to the object object in the 3D space popularly known as range information or depth. Local arithmetic operations between pixel neighbors (in time and space) generate other types of images such as gradient, edges and motion images. Examples from all these kinds of images are shown in Figure 1-1. They are classified here into five distinct categories: lightness and color images, gradient and textural information, feature points and motion.

Figure 1 - 1 :

 11 Figure 1-1: Pixel interpretation is context-dependent: (a) illumination intensity, (b) color components (RGB), (c) spatial gradient (Sobel), (d) motion: color denotes angle while represents speed and (e) the distance from the camera focal point (depth).

Figure 1 - 2 :

 12 Figure1-2: Diffuse (or Lambertian) and specular models are exemplified here. In the Lambertian model, the incoming light is diffused equally in all the observing directions, while in the specular reflection model light is reflected with the same angle θ formed by the direction of the incident ray of light and the normal to the surface at the point of incidence. In reality reflections are not purely Lambertian neither specular, and light is concentrated and diffused in some directions around the specular reflection direction. Image taken from[START_REF] Forsyth | Computer Vision: A Modern Approach[END_REF].

Figure 1 - 3 :

 13 Figure 1-3: In the first row mosaics showing the cone mosaic at the central fovea of L-cones, M-cones and S-cones respectively. In the second row a graph displays the relative spectral sensitivity of the three cone types (figure taken from [21]).

Figure 1 - 4 :

 14 Figure 1-4: Gaussian color model spectral derivative responses E, E λ and E λλ . Taken from [21].

1 - 4 .

 14 the visible spectrum. An approximation of the Gaussian color model is given by the opponent color spaces. Here, the brightness or intensity channel I = R + G + B can be regarded as a Gaussian-weighted spectral response, the yellow-blue channel Y B = R + G -2B approximates the first-order spectral derivative because it results from comparing one half of the spectrum (around the blue color) with the other half (around yellow), the red-green channel RG = R -2G + B can be interpreted as a second-order derivative which compares the center of the spectrum with the extremes (in the visual bandwidth). The responses of the E, E λ and E λλ responses are shown in Figure The opponent color interpretation of the Gaussian color model is then expressed as

Figure 1 - 6 :

 16 Figure 1-6: The scheme on the first row show how to build local texture representations by filtering an image with a set of filters of various scales and summarizing their individual responses. An example of a possible set of filters is shown on the second row: (a) 48 oriented filters, and (b) a set of orientation invariant filters. Image taken from [45].

Figure 1 - 8 :

 18 Figure 1-8: Object recognition with local features requires the algorithm to extract the sets of key-points in both images.For each key-point, the surrounding region is normalized for rotation and scale changes. Local descriptors are built then for each normalized region. Finally, feature matching is performed by comparing the local descriptors using an appropriate similarity measure. Image taken from[START_REF] Szeliski | Computer Vision: Algorithms and Applications[END_REF].

Figure 1 - 9 :

 19 Figure 1-9:The aperture problem is illustrated with three relevant cases: (a) a corner-like point is given and a unique and stable correspondence is relatively easy to find, (b) the point is located over an edge, as many points can be positioned over and edge the resulting flow is not unique and poorly defined, (c) a textureless point is given, even more candidate displacements exists and the point could move in any direction.

Algorithm 5 :12

 5 Basic feature point detection algorithm Data: Input image I Result: Coarsity image K The input image I is convolved with the derivatives of a Gaussian to get the horizontal and vertical derivatives I x and I y . From I x and I y three new images are computed corresponding to the outer products of these gradients: I 2 x , I x I y and I 2 y . 3 Images I 2x , I x I y and I 2 y are convolved with a larger Gaussian (the resulting images define the autocorrelation matrix A for each location).

Figure 1 - 10 :

 110 Figure 1-10: Autocorrelation function for three different types of patches take from the image in (a). Patch in (b) is taken from the cloud, no distinguishable peaks are found in its autocorrelation neighborhood, the patch in (c) comes from the roof edge, this patch suffers from the 1D aperture problem (path location can be confused along the edge) and finally, patch in (d) comes from the flower bed and a good unique minimum of the auto-correlation function allows easy localization.Figures taken from[START_REF] Szeliski | Computer Vision: Algorithms and Applications[END_REF] .

Figure 1 - 11 :

 111 Figure 1-11: Stack of smoothed images L(x, y, σ) calculated by SIFT and the DoG computed from them. Image taken from [89].

Figure 1 -

 1 Figure 1-11 shows how SIFT starts by incrementally convolving the image with a set of Gaussian kernels G(x, y, σ),

Figure 1 -

 1 [START_REF] Berclaz | Multiple object tracking using k-shortest paths optimization[END_REF] illustrates how foreground objects are detected using background subtraction.

  Background Subtraction Based on Color and Texture: Yao and Odobez propose in [152] a multilayer background subtraction technique which takes the advantages of local texture features represented LBP's and photometric invariant color measurements in the RGB color space. The idea is to complement the advantages of each type of measurement. On its hand, LBP's are very stable and robust to light variations in rich texture regions but weak on uniform regions. Color information overcomes LBP's limitation on those regions. Both the LBP feature and the selected color feature are able to handle local illumination changes such as shadows from moving objects.

Figure 1 - 12 :

 112 Figure 1-12: Background subtraction methods detect foreground objects by comparing the current image with the background models. The original image is the top-left image, current background representation is shown in top-right image, in the bottom-left we have the distance between the current image and the background image and in the bottom right the foreground image is extracted. Images taken from [152].

Figure 1 - 13 :

 113 Figure 1-13: Translation optical flow. Point correspondence is found easily for corner points but difficult for edges or homogeneous regions.

Figure 1 - 14 :

 114 Figure 1-14: 2D images sequences from real world 3D objects generally present occlusions and reflectivity phenomena such as specular reflections violating equation (1.32) assumptions.

Figure 1 -

 1 Figure1-15: A range sensor scanning the surface of an object projecting a laser plane and calculating the triangulations with the aid of a camera (image from[START_REF] Forsyth | Computer Vision: A Modern Approach[END_REF]).

Figure 1 -

 1 16-(b)).Some years later,Belhumeur et al. proposed to optimize class separability by working instead in a subspace obtained by Fisher's Linear Discriminant Analysis (LDA). Because LDA explicitly attempts to model the difference between the classes of data, the resulting method known as Fisherfaces[START_REF] Belhumeur | Eigenfaces vs. fisherfaces: Recognition using class specific linear projection[END_REF] improves the discrimination capabilities of PCA-based methods and is able to deal with specific discrimination tasks such as distinguishing people with mustaches, or people wearing glasses from the rest.As with sub-image (template) representations, vector space models require closely aligned 2D patterns. Template and sub-image alignment is demanded to ensure a fair comparison of the object attributes e.g. noses and eyes should be positioned in the right place when comparing images of frontal faces.

Figure 1 -

 1 Figure 1-16: (a) In the sub-image (template) representation, an image of size (N = W × H) is represented by a list of N gray-values or colors. Each sub-image is a point in the N -dimensional space. (b) Using PCA or LDA analysis, the original N -dimensional space is reduced to a vector space where images are represented by linear combinations of the vectors of greatest variability. (c) Using histograms, an holistic description is formed by measuring the distribution of pixel intensities, colors, gradients, etc. Images taken from [130].

Figure 1 - 17 :

 117 Figure 1-17: Pixels are severely affected by changes in illumination, gradient orientations are less sensitive and more robust to those changes.

Figure 1 - 18 :

 118 Figure 1-18: Popular methods use gradient-based histograms as descriptors, e.g., (a)SIFT local keypoint descriptor and (b) R-HOG descriptor.

  is the standard deviation of each bin. Unstable pixel values in channel c are associated to very smooth (widespread) kernels, while well-defined and stable pixel values are associated to a very narrow one (see Figure1-19).

Figure 1 - 19 :

 119 Figure 1-19: Robust histograms are constructed using color uncertainty to derive the parametrization of a variable kernel. Kernel sizes are steered by the amount of uncertainty of the color invariant values.

2 )Figure 1 - 20 :

 2120 Figure 1-20: The geometrical transformations experience by the whole object can be modeled and simplified by determining the transformation of the active contours parametrization model instead (see examples on the left). Parametric spline curves are attracted toward edges operator responses (image on the right). Image from [130].

and 1 . 2 . 5

 125 [START_REF] Terzopoulos | Tracking with kalman snakes[END_REF] based on the Kalman filter to model the uncertainty of the changes using a Gaussian model. After this, Isard and Blake proposed in 1998 the method of condensation[START_REF] Isard | Condensation-conditional density propagation for visual tracking[END_REF] that introduced particle filters to the computer vision community for modeling multi-modal distributions. Using level sets it is possible to avoid the estimation of the parametric curve, instead, the zero-crossings of a characteristic function are used to define it. Level sets evolve to fit and track objects of interest by modifying the embedding function. Some level-set methods are based on image gradients and others re-cast the problem into a segmentation framework, where an energy function measures the consistency of the image statistics (e.g., color, texture, motion) inside and outside the segmented regions. Matrix based descriptors For many years, the raw pixel values of several image statistics such as color, gradient and filter responses were used because they are the simplest choice for image features. The Lucas-Kanade (KLT) optical flow algorithm (Section 1.1.5.2) developed in the 1980's and 1990's belongs to this type of methods. Raw pixel matching is problematic because pixel values are not robust in the presence of illumination changes and nonrigid motion. Furthermore, because of the high dimensional representation (each pixel acts as a dimension), matching time grows with the size of the region and the number of features considered. Histogram methods (Section 1.2.3) provide a natural solution to these problems because a region is represented as a nonparametric estimation of the joint distribution. Histograms are widely used for nonrigid object tracking, texture representation, matching and other problems in the field of computer vision. Their great disadvantage is that histogram size grows exponentially with the number of features considered.

Figure 1 -

 1 21. ∑ A(x)<x ′ ≤C(x) A(y)<y ′ ≤C(y) i(x ′ , y ′ ) = I(C) + I(A) -I(B) -I(D).(1.40)

Figure 1 - 21 :

 121 Figure 1-21: Integral images concept.

Table 1 . 1

 11 displays a summary of the different feature combinations that could be found when using covariance descriptors in computer vision. Covariances obtained from pixel locations (x, y) are the same for all regions of the same size, but still they are important because they give a hint of the other features spatial relationship. Observing that the original features of Tuzel et al. in[START_REF] Tuzel | Region covariance: A fast descriptor for detection and classification[END_REF] behaved poorly when applied in face recognition problemsPang et al. proposed in[START_REF] Pang | Gabor-based region covariance matrices for face recognition[END_REF] to use Gabor wavelets to describe faces in a database. These filters have similar responses to the mammalian simple cortical cells, exhibiting strong characteristics of spatial locality, scale and orientation selectivity. Features based on these filters are expected to carry more important information than the first and second order gradients boosting the discriminative capacity of the covariance descriptor. The 2-D Gabor kernel is a product of an elliptical Gaussian and a complex plane wave

Figure 2 - 1 :

 21 Figure 2-1: A simple example of 2D binary classification. Three models increasing in complexity from left to right are applied. The model on the left miss-classifies a large number of training samples, the more complex one on the right trusts the training samples and proposes a solution that is consistent with the whole training set but that may not work well on new points. The model in the middle allows some training samples to be miss-classified looking for a more general solution. Image taken from [88].

  an hyperplane in H is defined as {x ∈ H|⟨w, x⟩ + b = 0} where w ∈ H and b ∈ R, where vector w is orthogonal to the hyperplane. A canonical formulation of the hyperplane is given by the pair (w, b), for the set of samples x 1 , x 2 , • • • , x m ∈ H the closest training sample to it is min i=1,••• ,m |⟨w, x i ⟩ + b| = 1, which has a distance equivalent to 1/∥w∥.The hyperplane separation margin and the magnitude of its orthogonal vector w play an important role on the algorithms based on support vectors. If all labelled training points are easily separated with a big margin there are reasons to believe that the same will happen while testing new patterns. A representative training set warranties the proximity of the testing samples to at least one of the trained samples. If all of them have a distance to the hyperplane bigger than ρ, small changes on the hyperplane parameters will not drastically modify the training classification results.The pattern recognition problem using classifiers reduces to the problem of determining the function f w,b : H → {±1} that maps or classifies correctly a labelled set (x i , y i ) ∈ H × {±1}. This means that f w,b = y i for every input pattern x i (or a significant part of them). The objective here is not to obtain a classifier that makes the training errors disappear, it is much more important to estimate the parameters that generalize the information contained in the training set. This idea is illustrated in Figure2-1, where three different margins are displayed for a simple 2D example with varying complexity.

Figure 2 - 2 :

 22 Figure 2-2: Linear SVM classifiers can help us to visualize the aspects of the HOG descriptor representation are distinctive.From a typical pedestrian window (extreme left) a HOG descriptor is computed (center left). Each of the orientation buckets in each window is a feature, a linear SVM assigns a weight to each feature. HOG features weighted by the positive weights are visualized in the center right image (important feature appear lighter). On the extreme right, weighted by the absolute value of negative weights is shown, highlighting the zones that strongly suggest a person is absent.

Figure 2 - 3 :

 23 Figure 2-3: The linear combination of multiple weak classifiers responses forms a strong classifier.

. 10 )Algorithm 8 :

 108 the value of α t is bigger as the fitting error ϵ t decreases. The process stops if for any of the weak classifiers the error probability is ϵ t ≥ 0.5 (more than 50% of the training samples are misclassified).The completeAdaBoost training procedure is summarized in Algorithm 8. As the training advances, the values the next round (t + 1) on the weight distribution D t+1 (i) are updated. Weights associated to the misclassified samples increase and those associated to the successfully classified samples go down (see the update equation in Algorithm 8 line 6). AdaBoost classification pseudo-code. Data: List of labeled points (x 1 , y 1 ), • • • , (x m , y m ) where x i ∈ X, y i ∈ Y = {-1, +1}.

Figure 2 - 4 :

 24 Figure 2-4: Some Haar-like rectangular features examples.

Figure 2 - 5 :

 25 Figure 2-5: Classification in Riemannian manifolds example.Here the manifold M ∈ M is depicted as the surface of the sphere and the tangent spaces are represented by the planes. The training samples are projected to the tangent space via log µ l , weak classifiers are learned on the tangent spaces T µ l . Sample X 3 is misclassified for the weak learner on the left g 1 , its weight increases pulling the mean µ 2 in the second iteration towards it.

  used a cascade of K = 30 LogitBoot classifiers operating on 8 × 8 SPD matrices. The linear regression functions (learners) g k,l are located in a 36-dimensional vector space (the upper triangle of the 8 × 8 covariance matrix). During classification, the algorithm receives a set of labeled pairs {(R i , y, i)} i=1•••N , where R i are the image windows containing background and pedestrians and y i ∈ 0, 1 are the labels. As explained before, for each detection window R there is a very large number of covariance descriptors which can be computed, so, each weak classifier is associated to one of these sub-windows in R. The sub-window associated to the l-th classifier of cascade level k is denoted by r k,l . While training in the k-level, the algorithm classifies all the negative examples {R - i } i=1,••• ,Nn (N n is the total number of negative samples) with the previous (k -1) LogitBoost levels. All the correctly classified samples are removed from the training set (all sub-windows from negative images are negative). Each one of the K LogitBoost classifiers can have a different number of learners L k . The idea is to optimize each cascade level until 99.8% of the positive samples are detected and at least 35% of the negative samples are rejected. At each iteration l of the k-th LogitBoost cascade level, 200 sub-windows are sampled among all the possible sub-windows. Weak classifiers are learned representing each subwindow and the classifier that best minimizes the log likelihood function (2.17) is appended to list of classifiers of level k.

Figure 2 - 6 :

 26 Figure 2-6: Cascade of LogitBoost Riemannian classifiers for human detection, each classifier k in the cascade has a variable number of classifiers L k a detection is declared when the a sample passes through all the classifiers.

Figure 2 - 7 :

 27 Figure 2-7: Grid FoT and Cell FoT comparison.In both methods all the trackers are initiated at their corresponding positions in the grid, Cell FoT lets the trackers move within their cells, but they are relocated when they get out from their cells.

Figure 2 - 8 :

 28 Figure 2-8: Covariance tracking with a Monte Carlo framework, the state of the target at time t -1 is shown with a red rectangle, a set of N particles is sampled near tho this location (shown in white) and are used to estimate the target transition from state x t-1 to x t .

Figure 3 - 1

 31 Figure 3-1 shows a few images from the sequence Cardgame from the ALOI database 1 . The first row displays the classical RGB images. Obviously, such features suffer from the illumination changes. The invariant features L1 and H are shown on the second and third rows respectively. As expected, the photometric variations are no more visible in these color-spaces.

Figure 3 - 1 :

 31 Figure 3-1: a)Original RGB images. b) L1 invariant images. c) Hue invariant images.

Figure 3 - 2 :

 32 Figure 3-2: Optical flow detected with L1P at Basketball sequence second frame. Displacement vectors in yellow are displayed magnified.

Figure 3 - 5

 35 Figure3-5 display the color-maps built for the Pedestrian 2 sequence, Luminance I succeeds to track the features over the car on the left corner, mostly because this is the most textured region of the sequence. On the other side, it fails to track the people walking at the center of the image that can be regarded as color blobs. Almost any color-space (except Hue) succeeds to track correctly both the car (luminance region) and the people walking at the center.It is also noticeable inFigure 3-5 that L1P tracked less color features than L1 for this sequence. The influence of the Luminance component, the same that improves the robustness to track low-saturated regions, reduces at some proportion the ability to track color features. It is possible to reduce this phenomenon by finding more appropriate sigmoid function parameters: S 0 and u β . Execution times scale-up proportionally to the number of components exploited during tracking. Single component tracking techniques like I, Hue and L1 require less time than multiple component techniques. Despite its simplicity in terms of calculations, Luminance I requires almost the same execution time as Hue and L1, because of the photometric normalization. Multiple component techniques require more execution time, a price to pay for the robustness and capability to track complementary features.

Figure 3 - 3

 33 Figure 3-3 displays the mean forward-backward euclidean distances d E (x t , xt ) computed from all the correctly tracked points in each frame of the Pedestrian 2 sequence. The error increases with the number of frames between x t and xt in the forward and backward trajectories, but remains lower than one pixel for each method. Note that the use of a single color component (L1, H or L1) instead of the whole color-space (RGB, HSV or rgb) usually leads to a lower accuracy. Indeed

Figure 3 - 3 :

 33 Figure 3-3: Mean forward-backward euclidean distances d E (x t , xt ).

Figure 3 - 4 :Figure 3 - 5 :

 3435 Figure 3-4: Tracking colormaps with Luminance I, L1 (max(r, g, b)) and L1P (mixed I and L1) for the Basketball sequence. Points are painted in a color-scale that goes from red to green representing the time the feature was lost

2 1Figure 3 - 6 :

 236 Figure 3-6: Inconsistent trajectories are penalized by the Forward-Backward error. Points that are visible throughout the sequence are consistently tracked forward and backward in time (point 1), while points that are occluded have inconsistent trajectories (point 2).

Figure 3 - 8 :

 38 Figure 3-8: Targets location at first frame for each sequence.

Figure 3 -

 3 11b shows the sub-images captured by the trackers bounding box at each frame of the sequence.

Figure 3 Figure 3 - 9 : 101 Figure 3 Figure 3 - 11 :

 3391013311 Figure 3-9: Sylvester and Auto sequences.

Figure 3 -

 3 Figure 3-12 represents the flow chart of the MS+CT cooperation algorithm. This figure can be easily decomposed in the following steps:

Figure 3 - 12 :

 312 Figure 3-12: Flow chart MS+CT

Figure 3 -

 3 Figure 3-13(a)-(d) shows the initial and the final target position for the set of tested sequences. MS alone is marked in pink, CT in red and the MS+CT method in blue or green depending on which method has been used on that frame.

Figures 3 -

 3 Figures 3-14, 3-15 and 3-16 are helpful to see where the MS (in pink) fails and where each one of the base methods MS (blue) or CT (green) are being used.

Figure 3 - 13 :

 313 Figure 3-13: Sequences used in the experiments: first and last frames with the different tracking results displayed with different colors: M S with colorspace switching in pink, CT in red, and CT /M S in blue /green.

Figure 3 - 14 :Figure 3 - 15 :Figure 3 - 16 :

 314315316 Figure 3-14: Tracking results of sequence Pedxing

Figure 4 - 1 :

 41 Figure 4-1: (a)The set of 56 different uniform LBP patterns in a (8, R) neighborhood. For each pattern a curved arrow indicates the start and end angles (θ 0 and θ 1 ). (b)In ELBCM, the uniform LBP pattern defined by two angles θ 0 and θ 1 is described by v(θ 0 , θ 1 ) = [cos(θ 0 ) sin(θ 0 ) cos(θ 1 ) sin(θ 1 )]. The LBP pattern represented by decimal value 63 maps to v(π/8, 5π/8) = [cos(π/8) sin(π/8) cos(5π/8) sin(5π/8)].

Figure 4 - 2 :

 42 Figure 4-2: Example of the type of problems observed when LBP patterns are embedded inside covariance matrices. When a template is transformed by a linear transformation such as a small rotation, many of its 3×3 neighborhoods may be altered and the LBP pattern strings or decimal values are severely affected. In contrast, the proposed angular representation used to build the ELBCM descriptor is more stable.

Figure 4 - 3 :Figure 4 - 4 :Figure 4 - 5 :

 434445 Figure 4-3: Some texture examples taken from the Brodatz dataset.

Figure 4 - 6 :Figure 4 - 7 :

 4647 Figure 4-6: Some texture examples taken from the KTH-TIPS dataset.

Figure 4 -8 and 4 - 9

 449 Figure 4-8 and 4-9 show some samples taken each five frames testing the four vector configurations, Figure 4-8corresponds to the Jumping sequence, here, the shakings of the camera, the nonrigid deformations on the face and the blurring conditions are very evident. While the tracker at some points seems to start losing the target, the distinctiveness of the covariance model allows our method to recover it. The same for the Panda sequence in Figure4-9a where the target shows noticeable changes of size, perspective and other non rigid transformations, nevertheless the ELBCM covariance tracker is able to recover it even when it gets out and re-enters the scene. The third tested sequence was Pedxing3, the set of cropped bounding boxes in Figure4-9b denote the non-rigid deformations experienced by the target and the occlusion that occurs when the target passes behind the panel, the tracker was able to recover the target and follow it until he gets out of the. Figures 4-10, 4-11 and 4-12 show some frames taken from each one of the tested sequences, bounding boxes in yellow correspond to the ELBCM Lum feature vector configuration, LBCM is denoted in red, Tuzel in blue and the LRCD in green. Plots of the w error for each of the tracking sequences are provided too. In the jumping sequence, the performance offered by LBCM and ELBCM Lum are comparable, Tuzel and LRCD both fail (the w error grows and never recovers for them after t = 50). For the Panda sequence, all the feature configurations behave more or less the same before the target gets out of the scene around frame t = 1000. When the target re-enters around t = 1300, all feature configurations but

Figure 4 - 8 :Figure 4 - 11 :Figure 4 - 12 :

 48411412 Figure 4-8: Bounding boxes cropped from the Jumping sequence.

Figure 4 - 13 :

 413 Figure 4-13: Texture classification accuracy for the KTH-TIPS dataset using color-texture covariance matrices.

Figure 4 -

 4 [START_REF] Bichsel | Strategies of robust object recognition for the automatic identification of human faces[END_REF] shows the evolution of the target throughout the sequence.

Figure 4 - 15 -

 415 (a) shows the locations of the trackers at t = 466, at this point all the feature combinations have succeeded to follow target throughout its first crossing.

Figure 4 -

 4 15-(b) depicts the positions of the trackers at t = 750, most of the trackers have already drifted from the target, only ELBCM Gauss , ELBCM RGB and RGB grads have succeeded to track the target consistently throughout the whole sequence. An increased level of invariance typically leads to a reduction of the distinctiveness of the descriptor, this may explain why the L1-based descriptors rapidly fail (around t = 450), then HSV-based descriptors (ELBCM HSV and HSV grads ) fail around t = 710 when the target crosses a different pedestrian wearing a blue jacket with some reddish bands. Only the RGB grads , ELBCM RGB and ELBCM Gauss succeed in tracking the target until the sequence end, the ELBCM-based descriptors are slightly more precise than the gradients based RGB grads .

Figure 4 - 14 :

 414 Figure 4-14: Evolution on the appearance of the selected target on the PETS2009-S2-L1-View001 sequence.

Figure 4 -

 4 18.

Figure 4 -Figure 4 -Figure 4 - 16 :

 44416 Figure 4-15: (a) and (b) show the location of the target at t = 469 and t = 750 respectively.

Figure 4 - 18 :

 418 Figure 4-18: Tracking precision error w error obtained using all the color-based feature set combinations for David sequence. Due to the illumination changes, only the L1-based color feature sets (L1 grads and ELBCM L1 ) succeed to track the target.

Figure 4 - 19 :

 419 Figure 4-19: Some frames of PETS'09 showing the re-identification at different times and points of view.

Figure 4 - 20 :

 420 Figure 4-20: Proposed MRC pattern and its resemblance to FREAK

2 . 4 - 22 , 4 - 23 and 4 -

 24224234 The results of the experiments are shown in Figures 4-21, 24 where the cumulative matching characteristic for each dataset is reported. What these curves represent is the percentage of times the correct target class was ranked among the first K possible matches (e.g. rank score 1 measures the percentage of times the correct target class was ranked first).

Figure 4 - 21 :Figure 4 - 22 :

 421422 Figure 4-21: Cumulative Matching Characteristic for ETHZ sequence 1.

Figure 4 - 23 :Figure 4 - 24 :

 423424 Figure 4-23: Cumulative Matching Characteristic for ETHZ sequence 3.

Figure 4 - 25 :

 425 Figure 4-25: In every frame t our multi-target tracking receives a series of detections {d 1 • • • d n } and consistently traces the object trajectories {p 1 • • • p m }.

Figure 4 - 26 :

 426 Figure 4-26: Decisions are made considering the addition of multiple energy terms. Top row shows the configurations that render higher energies for each individual term, lower energies are exemplified in the lower row. Detection peaks are denoted in gray.

Figure 4 - 27 :

 427 Figure 4-27: CLEAR matching error examples. In (a) the distance between o 1 and h 1 is computed, if it exceeds a certain threshold T , there is no valid correspondence o 1 is declared missing and h 1 becomes a false positive. Matching error examples are shown in (b), here h 2 is initially assigned to o 2 , but after a few frames o 1 and o 2 cross paths and their trackers follow the wrong objects. Some frames later, o 2 swaps again with o 3 . Image taken from[START_REF] Stiefelhagen | The CLEAR 2006 evaluation[END_REF]

{o 1 ,

 1 • • • , o n }. Different kinds of errors can occur at this point, Figure 4-27-(a) illustrates the precision error where the distance between the hypothesis and the actual target position is measured, if it exceeds a certain threshold T , the correspondence is invalidated (the object is declared missing and the hypothesis becomes a false positive). Examples of matching errors or identity swaps are given in Figure 4-27-(b).

Figure 4 - 28 :

 428 Figure 4-28: Target recovered correctly after occlusions in TUD-campus sequence.

experiments shown in Section 4 . 1 2 . 2 .

 4122 improve the discrimination of the descriptor for different applications such as texture recognition, gesture recognition and tracking.The influence of color inside the covariance descriptor was verified next using a variety of color representations. The set of classification experiments performed in Section 4.2.1 shows how color can help to improve the discriminant power of the ELBCM descriptor. The robustness of the L1 color invariant[START_REF] Romero | Feature points tracking adaptive to saturation[END_REF] was verified too in Section 4.Based on both experiments it is possible to conclude that the ELBCM Inv and Inv grads descriptors are as discriminative as the original ELBCM Lum descriptor (based purely on luminance) but less discriminative than other color descriptors based on RGB,

Figure 4 - 29 :

 429 Figure 4-29: Frames taken from the PETS2009-S2-L1-Time_12-34-View_001 sequence showing targets crossing each other and being occluded. The discrete energy minimization algorithm based on the togal Bremgan divergence is able to trace their trajectories and handle their identities.

Figures 5 -

 5 Figures 5-2a and 5-2b show similar sketches for the Intel Nehalem and ARM Cortex A9. Both processors are able to execute almost the same type of arithmetical operations. But each processor design involves a very complex set of technologies (implemented as circuitry) that vary significantly the inner functioning of a processor: mechanisms that vary the operational frequency of the processor to economize in energy depending on the load, to pre-fetch data and/or instructions, fetch and branch prediction mechanisms, to change the order of execution of the stream of instructions received depending on the data availability and their dependencies, etc. For the case of the Nehalem, Figure 5-2a shows the communication link to the main memory (e.g. DDR3 SDRAM) and to other peripheral using the Direct Media Interface (DMI) or Quick-Path Interconnect (QPI) technology to implement what it is known as the point-to-point link that connects to the Platform Controller Hub (PCH). This circuit, handles the communication to different ports such as serial ATA (to communicate with the hard-drive), PCI Express, USB 2.0 ports, the network interface, etc. This last aspect is illustrated in Figure 5-1b.

Figure 5 - 1 :

 51 Figure5-1: (a) A multi-core processor with hierarchical cache. Each core has its own registers and functional units, typically instruction and data caches are separated in the highest level. Larger and slower caches are shared with other cores forming a hierarchical structure of memory. (b) An schema showing how Intel Core processors are connected to the main memory (DDR3 SDRAM blocks) and using the DMI point-to-point interconnect to access other peripheral. Figures taken from[START_REF] Mccool | Structured Parallel Programming: Patterns for Efficient Computation[END_REF] and from Intel H57 and H55 documentation.2

  statements (enclosed within curly braces) within the parallel region are executed on each thread, threads are joined after the last statement in the region. The following code double k1 = alpha; double k2 = (1 -alpha); #pragma omp parallel for for(int i = 0; i<n; i++) { C[i] = k1 * A[i] + k2 * B[i]; } calculates the linear combination of two vectors A and B component by component and stores the results in C. Instead of executing each iteration sequentially, the iterations of the loop (the values from i = 0 to i = n -1) are divided amongst the threads of the team initiated in line 3.

Figure 5 - 8 : 1 /Listing 1 : 1 /Listing 2 : 1 /

 5811121 Figure 5-8: Three alternative arrangements for storing RGB images can be considered: the Array of Structures (AoS) (top), the Structure of Arrays (SoA) arrangement (middle) and the Hybrid Structure of Arrays (HSoA) (bottom).

Figure 5 - 8 .

 58 For the case of out RGB example, HSoA is well adapted for the vectorization of a number of algorithms such as color gradient and feature computation, color-space conversions, etc. Elements should be aligned and arranged according to their data type sizes and the correspondent vector register widths: architectures equipped with SSE4a technology use 128-bit registers which allow to allocate sixteen 8-bit fixed point values, four 32-bit single precision floating point values and just two 64-bit double precision floating point values.

Figure 5 - 9

 59 shows the data flow followed by the covariance descriptor computations and matching algorithm.

Figure 5 - 9 :

 59 Figure 5-9: Covariance data flow: features computation

Figure 5 - 9 ,

 59 features are obtained from the original luminance (gray-scaled) or colored images (RGB). Each feature represents a different aspect of the image and each one has its own dynamic range. Integral images are computed from them and from their crossed products. To prevent us from computing spurious or meaningless descriptors it is important to assure that the data type used during the accumulation all those values is able to calculate the accumulation accurately. This section analyzes different implementations for the integral image calculation and their implications.Integral images are a type of two dimensional data reduction or more precisely a scan collective operation which produces a set of partial reductions of the input image forming a new output image of the same size.

Listing 5 :

 5 Integer pixel values: The simplest case corresponds to the accumulation of b-bit integers in m-bit valued integral images. In this scenario it is possible to accumulate at most 2 m-b values without accuracy errors. Example: If the image is composed by 8-bit integers and they are accumulated on integral images represented by 32-bit floating point (IEEE754) values which have 24-bits 4 to store the significand, then we can store 2 24-8 = 65536 pixels of type byte without accuracy errors. The double precision 64-bit floating point format with 53-bit to store the significand allows us to accumulate much more pixel values (2 53-8 = 35.1843 × 10 12 ).• Fixed point values: In the covariance descriptor algorithm we require to take the product of the features, data with pixed point values are of the order of the square of feature image pixels (e.g. the square of a 16-bit fixed point values has a range of 32 bits). A double precision integral image can hold up to 253-32 = 2097152 ≈ 2 million of such 32-bit values without loss of precision. This corresponds to a 1400 × 1400 square image.• Floating point values: Because of their flexibility, floating points values are preferred to accelerate development time. Here we are interested in verifying the correctness and the applicability limits of floating point feature images in the covariance descriptor algorithm. What matters in the integral image is the fixed point precision of its data type, this is equivalent to the number of possible values data can take within a certain range. The addition of floatingpoint numbers is different from binary integer addition (or fixed-point addition). Fixed point addition is more or the less analogous to how additions are performed by hand: numbers are lined up one over the other and additions are performed one digit at a time starting from the least significant bits. Floating point addition runs a similar procedure too, but first floating point numbers are shifted so that the corresponding digits line up. If the two numbers have significantly different exponents then some of the least significant digits are discarded.Floating point arithmetic has many pitfalls: a real value such as (0.1) 10 in base ten has no exact finite representation and its (approximated) binary form is: (0.1) 10 = (0.000110011001100 • • • ) ≈ 0 : 100000001490116119384765625. Moreover, floating point arithmetic does not respect neither the distributive nor associate operators (-10000001 + 10 7 ) + 0.5 ̸ = -10000001 + (10 7 + 0.5),(5.3) this attributed to absorption and cancellation phenomena exemplified by Listings 4 and 5. Cancellation phenomena.

Figure 5 - 12 :

 512 Figure 5-12: Layout of SoA cube and the AoS feature matrix.

Figure 5 - 12 .

 512 In the AoS representation features features are gathered together in contiguous memory addresses.//Feature imagesFMAT(i,j,k) fmat[k][i][j] //Crossed feature-products PRMAT(i,j,k) prmat[k][i][j] //Feature images FMAT(i,j,k) fmat[i][(j)*NF+k] //Crossed feature-products PRMAT(i,j,k) prmat[i][(j)*NP+k]Listing 6: C macros used to address the SoA cubes (on the left) or the AoS 2D-arrays (on the right) that store the set of features (and their crossed-products) used to compute the covariance matrix, k denotes the index of the feature (or featureproduct) while i and j denote the image coordinates. NF and NP correspond to n F and n P respectively.The product of features and its transformation are described in Algorithms 16 and 17. Due to commutativity of the multiplication, only half of the products have to be computed so, the loop on k 2 starts at k 1 (Algorithm 16 line 3). As the two last stages are similar, only the generic versions of integral image computation are presented (the same for features and their crossed-products). The SoA version is shown in Algorithm 18 while its AoS equivalent in Algorithm 19.

Figures 5 -

 5 Figures 5-14, 5-15 and 5-16 show the evaluation results of three different parallelized versions of the covariance

Figure 5 - 14 :

 514 Figure 5-14: Performance in cpp of a 1×4-core Penryn for image sizes ∈ [128..1024]. Before the cache overflow transition starts (close to 250 × 250), the AoS+SIMD is ×2 faster than the SoA version, once the transition ends (at 500 × 500), the performance drops significantly and the algorithm demands ×2.2 more cpp's than before the cache overflow, the speedup in this zone is just ×1.3.

Figure 5 - 15 :

 515 Figure 5-15: Performance in cpp of a 2×4-core Nehalem for image sizes ∈ [128..1024]. Contrary to the Penryn architecture, the SoA+OpenMP is faster on the Nehalem processor than the AoS+SIMD and the AoS versions thanks to the larger number of threads (8 vs. 4). Before the cache overflow starts at 250×250, the speedup provided by OpenMP is ×2.8 faster than the scalar and single threaded AoS. After the overflow transition ends at 650 × 650 for the SoA+OpenMP version, the speedup obtained drops to ×2.3 because the algorithm in this regions demands ×1.4 more cpp's.

Figure 5 - 16 :

 516 Figure 5-16: Performance in cpp of a 1 × 4-core SandyBridge4 for image sizes ∈ [128..1024]. As the available number of threads for this processor is only 4, the AoS+SIMD is again faster than SoA+OpenMP 4 (similarly to the case of the Penryn architecture in Figure 5-14). Before the cache overflow transition begins (at 225 × 225), the speedup obtained is ×2.3.After the transitions ends (before 400 × 400), the speedup drops to ×1.68 as the algorithm demands for this region ×1.69 more cpp's.

Figures 5 -Figure 5 - 18 :

 5518 Figures 5-17 and 5-18 provide the measured cpp's for the SoA, AoS, AoS+SIMD, AoS+T and AoS+T+SIMD versions of the algorithm implemented on the Intel U9300 and ARM Cortex-A9. In both processors, the SoA version is very inefficient specially when it is compared with the more efficient one (i.e. AoS+T+SIMD). But there are two big differences between them, the first one is impact of the optimizations: for the Cortex-A9 when images larger than 100×100 are applied the only possibility for optimization is to apply the loop-fusion and serialization transformations, the impact of the Neon instructions is insignificant and the number of cpp's required by the AoS+SIMD version is very similar to those required by the AoS version. The same occurs for the case of the AoS+T+SIMD and AoS+T versions, where it is not clear if Neon reduces the execution time. The problem comes from the memory hierarchy of the Cortex-A9 and from the number of extra instructions required by Neon to perform the permutations (48 for Neon versus only 15 for SSE). The second difference is the cpp range of values between both architectures: the Intel ULV 9300 demands about ×4.5 less cpp's than ARM Cortex-9 an important

x 2 g 2 x x 2 g x g y xyg 2 x xyg x g y x 2 g x g y x 2 g 2 y xyg x g y xyg 2 y xyg 2 x xyg x g y y 2 g 2 x y 2 g x g y xyg x g y xyg 2 y y 2 g x g y y 2 g 2 y 2 x xg x g y yg 2 x yg x g y xg x g y xg 2 y yg x g y yg 2 yg 2 x g x g y g x g y g 2 y

 2222222222222 Ax + d) -I(x) ] 2 w(x)dx.(A.12)Now, to minimize (A.12) it is necessary to differentiate it with respect the unknown entries of matrix D and vector d and set the result to zero. The resulting system is also linearized by the truncated Taylor expansionJ(Ax + d) = J(x) + g T (u), (A.13)resulting in the 6 × 6 linear systemT z = a (A.14)wherez T = [ d xx d yx d xy d yy d x d y ]is the concatenation of the entries of the deformation matrix D and the displacement vector d. difference between the two images I (x -J(x)), and the 6 × 6 matrix T , which is computed from one image only and is written as

8 )B.0. 5

 85 The Exponential MapAn exponential map helps to parametrize a Riemannian manifold, M , locally near any p ∈ M in terms of a map from the tangent space T p M to the manifold.Let (M, g) be a Riemannian manifold, for every p ∈ M , let D(p) be the open subset of T p M given byD = {v ∈ T p M |γ v (1) is defined}, (B.9)where γ v is the unique maximal geodesic with initial conditions γ v (0) = p and γ ′ v (0) = v. The exponential map is the map from exp p : D(p) -→ M , so that exp p (v) = γ v (1).(B.10)

1 / 2 .

 12 (B.11) Equation (B.11) is useful to compute the length of a differentiable path in P n . The expression γ : [a, b] → P n parametrically defines the path, and it is possible to get its length asL(γ) = ∫ b a ∥γ -1/2 (t)γ ′ (t)γ -1/2 ∥ 2 dt (B.12)For any two points A and B in P n letδ 2 (A, B) = inf{L(γ):γ is a path from A to B} (B.13)This metric in P n respects the triangle inequalityδ 2 (A, B) ≤ δ 2 (A, C) + δ 2 (C, B) (B.14)because the path γ 1 from A to C can be adjoined to the path γ 2 from C to B, forming the path A to B of length L(γ 1 ) + L(γ 2 ).

δ 2 (

 2 A, B) = ∥ log A -log B∥ 2 . (B.18) represents the geodesic from A to B.It is possible to get the length of any subinterval [0, a] of [0, 1] thanks to the parametrizationH(t) = (1t) log A + t log B. (B.19)The update the model, the straightforward choice would be the use of the arithmetic mean of A 1 , • • • , A m

1 0a 1 -( a p + b p 2 )

 112 (i) m(a, b) = m(b, a) (ii) min(a, b) ≤ m(a, b) ≤ max(a, b) (iii) m(αa, αb) ≤ αm(a, b) for all α > 0.(iv) a ≤ a ′ ⇒ m(a, b) ≤ m(a ′ , b) (v) m is continuous. (B.26) The arithmetic, geometric and harmonic means defined as the most familiar examples of means satisfying the conditions enlisted in (B.26). But there are several others such as the logarithmic mean L(a, b) = ab log a -log b = ∫ t b t dt, (B.28) and the binomial means B p (a, b) = a, b) = max(a, b), (B.31) and lim p→-∞ B p (a, b) = min(a, b). (B.32)As stated before, the set of n × n covariance matrices is a subset of the positive matrices P(n) set. The five conditions of (B.26) can be imitated to the define the concept of matrix mean which is a map M :P(n) × P(n) → P(n) satisfying (i)' M (A, B) = M (B, A). (ii)' If A ≤ B, then A ≤ M (A, B) ≤ B. (iii)' M (X * AX, X * BX) = X * M (A, B)X,for all nonsingular matrices X (iv)' A ≤ A ′ ⇒ M (A, B) ≤ M (A ′ , B) (v)' M is continuous. (B.33)

( A p + B p 2 )

 2 [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] do meet the five conditions in (B.33), but the geometric mean have some peculiarities. The matrix A 1/2 B 1/2 is not positive, not even Hermitian unless A and B commute.The analogy of (B.30) for the case of positive matrices is lim p→0

arg min m ∑ j=1 δ 2 2 (

 2 X, A j ) (B.[START_REF] Facciolo | [END_REF] where arg min f (X) returns the minimum point X 0 in the function f (X). It can be shown that the solution of the matrix equation

X# 1

 1 /r A k where k = r( mod m). Let the sequence φ r,n = φ r+n-1 • • • φ r+1 • φ r . (B.43)For all X, and for all positive integers r,lim n→∞ φ r,n (X) = G(A 1 , • • • , A m ) (B.44)

x 1 , x 2 ,i w i = 1 .

 121 • • • , x N ∈ M and their associated weights w 1 , w 2 , • • • , w N where ∑ The weighted sum of distances isf (x) = ∑ i w i d(x, x i ) (B.45)where d is the Riemannian distance function in M . Assuming that x i lies in a convex set U ⊂ M , then any pair of points in U is connected by a unique shortest geodesic contained entirely in U .The weighted geometrical median is the minimizer of f (x) (equation B.45) m = arg min x∈M N ∑ i w i d(x, x i ), (B.46) when all the weights are equal (w i = 1 N ), equation (B.46) represents the geometric median. It is important to remark the difference between the weighted geometrical median defined in equation (B.46) from the Fréchet/Karcher mean which is a generalization of the Euclidean minimum squares principle into Riemannian geometry as µ = arg min x∈M N ∑ i w i d 2 (x, x i ) (B.47) For a Riemannian manifold M , the gradient of the Riemannian sum-of-distances function is given by ∇f (x) = -N ∑ i=1 w i Log x (x i )/d(x, x i ), (B.48)

) - 1 .From 1 . 51 )- 1 [

 11511 [START_REF] Facciolo | [END_REF], the (i, j)-th element of the covariance matrix is C R (i, j) (i)µ(i))(z k (j)µ(j)).(B.50)Expanding the means and rearranging the terms we haveC R (i, j)The covariance in a given region depends on the sum of each feature dimension z(i) i=1•••n , as well as the sum of the multiplication of any pair of features z(i)z(j) i,j=1•••n , requiring in total d + d 2 integral images, one for each feature dimension z(i) and one for the multiplication of any pair of feature dimensions z(i)z(j).Let P be a W × H × d tensor of the integral images of each feature dimensionP (x ′ , y ′ , i) = ∑ x<x ′ ,y<y ′ F (x, y, i) for i = i • • • d (B.52)and Q be the W × H × d × d tensor containing the feature product-pair integral imagesQ(x ′ , y ′ , i, j) = ∑ x<x ′ ,y<y ′ F (x, y, i)F (x, y, j) for i, j = i • • • d. (B.53)Now, let p x,y be a d dimensional vector and Q x,y a d × d dimensional matrix such asp x,y = [P (x, y, 1) • • • P (x, y, d)] , y, 1, 1) • • • Q(x, y, 1, d) . . . Q(x, y, d, 1) • • • Q(x,y, d, d) R((x ′ , y ′ ); (x ′′ , y ′′ )) be the rectangular region defined by the top-left point (x ′ , y ′ ) and the right-bottom point (x ′′ , y ′′ ). The covariance of the region bounded by (1,1) and (x ′ , y ′ ) is C R ((1, 1); (x ′ , y ′ )) = 1 n Q x ′ ,y ′ -1 n p x ′ ,y ′ p T x ′ ,y ′ ] (B.55)

- 1 n

 1 ( p x ′′ ,y ′′ + p x ′ ,y ′ -p x ′ ,y ′′ -p x ′′ ,y ′ ) ( p x ′′ ,y ′′ + p x ′ ,y ′ -p x ′ ,y ′′ -p x ′′ ,y ′ ) T ] (B.56)where n = (x ′′x ′ ) • (y ′′y ′ ). After constructing the integral images the covariance of any rectangular region can be computed in O(d 2 ) time regardless of the size of the region R((x ′ , y ′ ); (x ′′ , y ′′ )). The complete process is represented graphically in Figure B-1.

Figure B- 1 :B. 2 -1 2 Y x M -1 2 )∂

 1222 Figure B-1: Covariance Description Calculation

1 2(B. 60 )

 160 where the operator ⊗ denotes the Kronecker matrix product. Partial derivatives (∂ x Y x ) and (∂ y Y x ) can be approximated from discrete data as∂ x Y x ≈ (Y x|x+dx,y -Y x|x-dx,y )/2dx ∂ y Y x ≈ (Y x|x,y+dy -Y x|x,y-dy )/2dy (B.61)where dx = dy = 1.The subtraction operator in the partial equations (B.61) needs to be replaced by its equivalent in the SPD space∂ x Y x ≈ 0.5 log Yx|x-dx,y (Y x|x+dx,y )/dx ∂ y Y x ≈ 0.5 log Yx|x,y-dy (Y x|x,y+dy )/dy (B.62)

  combinaison de composantes couleur et de texture, dans le but d'assurer un bon pouvoir discriminant tout en réduisant la taille de la représentation. Concernant la texture, le descripteur ELBCM est proposé. Les résultats obtenus en classifica-

tion de texture, suivi et ré-identification d'objets sont très prometteurs. Ce descripteur est également utilisé dans le cadre du suivi multi-cibles, où nous avons proposé une nouvelle fonction de coût intégrant un modèle d'apparence basé sur la divergence de Jensen-Bregman.

Table 2 :

 2 Optimisation de CT : temps d'exécution (ms) pour une image 1024×1024 avec 7 caractéristiques.

	mono-threading MdS MdS+SIMD SdM 219.6 144.5 SdM 117.7 137.3 140.0 multi-threading MdS MdS+SIMD 110.7 219.3 118.2 65.5 91.0 116.5 69.9 103.6 52.5 30.9 43.5 51.1 31.0 SandyBridge+OC 77.7 Processeur Penryn Bloomfield SandyBridge 41.0 25.7 33.6 40.5 25.9 Nehalem 103.8 79.7 48.6 29.8 63.6 38.0

La transformation SdM → MdS est très efficace avec une accélération de près de ×2. La version SIMD permet une accélération de ×2.5 pour des images 128 × 128. Lorsque les données ne rentrent pas dans le cache (pour des images

  It has been proven experimentally that ULBP patterns account for the majority of texture patterns in natural images, and they are indications that uniform patterns are more stable or less prone to noise.

		.21)
	Equation (1.21) interprets the sign of the differences in a neighborhood as a P -bit binary number, which results in 2 P
	different labels. The local gray-scale texture is approximately described with the 2 P -bin distribution of LBP labels
	T ≈ t(LBP P,R (x c , y c )).	(1.22)
	Pixels located at the diagonals are interpolated. While this may seem costly, this generic form is advantageous as it
	is adaptable for different scales sizes, and rotation invariant descriptors are easily derived from this circular neighborhood
	indexing.
	Uniform patterns (ULBP) [102] are the result of another modifications made to the original definition of the LBP
	operator. Here, a uniformity measure that counts the number of bitwise transitions from 0 to 1 or vice versa is employed. A
	pattern is considered uniform if its uniformity measure is ≤ 2. For example, 00000000 and 01110000 are uniform patterns but 11001001 and 10010011 are not.

  To robustify the Harris operator, Mikolajczyk and Schmid introduced in [94] a scale selection mechanism evaluating the LoG function at each one of the Harris points found in a multiscale pyramid and to keep the extreme points only. The complete algorithm for the detection of SIFT points is summarized where σ ∈ {σ 0 , kσ 0 , k 2 σ 0 , • • • , 2σ 0 } and k = 2 1/s . For each point x ∈ p estimate the precise sub-pixel and sub-scale offset x by quadratic fitting: x = ∂ 2 D For each point x ∈ p calculate D(x) and remove from p if |D(x)| < D thresh .

	in Algorithm 6.
	Algorithm 6: Scale Invariant Feature Transform (SIFT) point detection algorithm
	Data: Input image I
	Result: List of SIFT points p
	1 For each octave, construct a scale-space stack of s + 3 blurred images L(x, y, σ) by applying the Gaussian kernel
	G(x, y, σ) 2 For each octave stack obtain the stack of differentials of Gaussians (DoG) doing the subtraction
	D(x, y, σ) = L(x, y, kσ) -L(x, y, σ). 3 Compare each point in D(x, y, σ) to its 26 scale-space neighbors and keep insert in a list p those which are larger
	than all or smaller than all.
	∂x 2	-1 ∂D ∂x and
	update x as x ←-x + x.

4 5 6 For each point x ∈ p calculate the Hessian matrix H =   D xx D xy D xy D yy   and remove from p if trace(H) 2

det(H)

  Thanks to the averaging in the covariance computation, noisy pixels are largely filtered out which contrasts with raw-pixel methods. Covariance matrices are low-dimensional compared to other descriptors, and due to symmetry C R has only (d 2 + d)/2 different values. The covariance descriptor C R does not use any information regarding the ordering or the number of the points implying a certain scale and rotation invariance. Nevertheless the covariance descriptor ceases to be rotationally invariant when orientation information is introduced in the feature vector such as the norm of gradients with respect to x and y directions.Matching and tracking applications are forced to calculate multiple covariance descriptors in overlapping regions where a high redundancy of operations is expected. Viola and Jones and Tuzel et al. use a very clever trick that allows to compute overlapping region averages supper fast. This method is popularly known as the integral images or the summed area tables method. Integral images are popluar as they are used in many computer vision methods in different contexts (e.g., Viola and Jones face detector[START_REF] Viola | Robust real-time face detection[END_REF] and Bay et al.'s Speeded Up Robust Features SURF[START_REF] Bay | Surf: Speeded up robust features[END_REF]). When constructing the integral images accumulations are performed first over the whole image or over a region of interest and the integral images which result from this accumulation are re-utilized to calculate the feature averages required by equation(1.38).

58

correlations.

Table 1 .

 1 1: Features considered by the covariance descriptor depending on the application.

	Application	Feature set φ(I, x, y)
		[	x y |Ix| |Iy| |Ixx| |Iyy|	]
	Face tracking and recognition [104]	[	x y I |Ix| |Iy| |Ixx| |Iyy| θ(x, y)	]
	Pedestrian detection [140, 151]	[ [	x y I g 00 (x, y) g 01 (x, y) • • • g 74 (x, y) x y |Ix| |Iy| √ I 2 x + I 2 y |Ixx| |Iyy| arctan ]	|Ix| |Iy |	]
		[	x y |Ix| |Iy|	√	I 2 x + I 2 y	arctan	|Ix| |Iy |

Adaptability to brightness and saturation changes

	Context adaptability and method
	switching
	Contents
	3.1

. . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 3.1.1 L1 color invariant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 3.1.2 Relevance of color vs. luminance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 3.2 First cooperation (FoT+CT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 3.2.1 FoT outliers detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 3.2.2 FoT+CT algorithm description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3.2.3 FoT+CT evaluation experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3.3 Second cooperation (MS+CT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 3.3.1 MS+CT algorithm description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 3.3.2 MS+CT evaluation experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 3.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108

Table 3 .

 3

	1: Tracking results

Algorithm 10 :

 10 Tracking methods cooperation Input: Image Sequence I t ∀t

	7 8 9 10	// Analyze the optical flow: foreach obj ∈ objList do if obj rigid = TRUE then FoT(obj) // Classify Object as Rigid/Non-rigid: obj resF low ← residualFlow(obj)

1 objList ← objDetection(I 0 ) 2 for t > 0 do 3 // New detections and object verifying each t detect frames: if t multiple of t detect then 4 objList ← objDetection(I t ) 5 // Track each object features by KLT: foreach obj ∈ objList do 6 pyramidalKLT(I x , I y , I t-1 , I t , featuresList) 11 // Calculate CT model dissimilarity: C t ← getCovarianceMatrix(obj,I t ,V) D t ← covDissimilarity(C t , C t-1 ) 12 if obj rigid = FALSE OR D t > D max OR residualFlow(obj)> R max then 13 // Run Covariance Tracking: obj← covTracking(obj, I t , V) 14 // Update model C t-1 for the next frame:

Table 3 . 2

 32

		# frames	Dataset	FoT	Last Frame CT FoT+CT % CT FoT[ms] CT[ms] Speed-up FoT+CT
	Sylvester Auto Motocross Panda Pedestrian1 PETS 2001 Dataset 1	1344 945 2665 3000 140 338	[118] [73] [73] [73] [73] PETS 2001 5	620 1344 220 945 65 958 95 1015 95 140 42 338	1344 945 958 1015 140 338	40% 34% 36% 40% 71% 60%	2.8 2.9 4 2.2 2.7 2.9	4.8 4.8 6.3 3.7 4.8 4.5	25% 26.5% 24% 24.3% 12.7% 14%

: Tracking results.

Table 3 . 3 :

 33 Analysis of the MS+CT cooperation procedure. Results were obtained on a Nehalem processor. The table shows for each sequence: the target size at the initialization point; the percentage of frames (%fr) for which the MS and CT algorithms are run; the total number of cycles per pixel (cpp) spent by each section of the algorithm (MS, CT, Covariance model Update, computation of the similarity) during the whole sequence.

	Sequence	Target	# frames	MS	CT	Model Update Similarity	Time
	size 93 × 35 23 × 28 Motocross 64 × 47 Pedxing Panda Carchase 45 × 97	189 940 2665 2999	cpp (×10 6 ) %fr cpp (×10 6 ) %fr 7.20 70 60.59 30 2.48 88 8.74 12 5.07 12 30.04 88 0.99 13 19.19 87	cpp (×10 6 ) 1.07 0.57 0.005 1.03	(cpp ×10 6 ) [ms]/fr 2.45 10.1 0.33 1.6 1.06 10.6 1.44 7.3

Table 3 .

 3 3 analyses for each sequence the number of cycles per second (cpp) spent by each section of the MS+CT switching algorithm throughout the whole sequence. The algorithm can be decomposed in several sections, MS and CT

Table 4 .

 4 1: Tracking time and accuracy for three different sequences and four different feature combinations. [ms] w error time/frame [ms] w error time/frame [ms] w error

	Features	Jumping		Panda		Pedxing3
	time/frame Tuzel 4.83	1.83	3.65	1.93	20.5	-
	LBCM	10.3	0.36	7.57	0.28	40.6	-
	ELBCM	4.82	0.26	3.65	0.27	20.5	-
	LRCD	2.25	2.35	1.71	0.38	6.69	-

Table 4 .

 4 3: Multiple object tracking evaluation metrics summary. distance between the hypothesis h i and the object o i at frame t.

	Term	Definition
	{h 0 , • • • , h t } {o 0 , • • • , o t } d i t g t c t m t f p t	hypothesis set. visible objects set. # of ground-truth objects at frame t. # of matches at frame t. # of false negatives (misses) at frame t. # of false positives at frame t.
	mme t

Table 4 .

 4 4: Multiple object tracking evaluation results.

	Sequence	MOTA (%) MOTP (%) MT PT ML IDS
	Pets 2009 S2-L1-V1 70.084	75.9	16	3	0	5
	TUD Campus	74.92	63.4	5	2	0	0
	TUD Satdtmitte	69.73	69	5	1	1	1
	TUD Crossings	74.63	76.34	8	5	0	4

Table 4 .

 4 5: Multiple object tracking state-of-the-art results, OM (Occlusion management plus appearance modeling) "Continuous Energy Minimization for Multi-Target Tracking"

  Amdahl's law (equation (5.2)) expresses how to calculate the theoretical maximum speedup of a program using multiple processors in parallel computing which is limited by the sequential fraction of the program. For instance, if 95% of the program can be parallelized, the theoretical maximum speedup would be 20x no matter how many processors are used. Image taken from Wikipedia

									Amdahl's Law
		20.00
		18.00
													Parallel Portion
		16.00												50%
														75%
		14.00												90%
														95%
		12.00
	Speedup	10.00
		8.00
		6.00
		4.00
		2.00
		0.00
		1	2	4	8	16	32	64	128	256	512	1024	2048	4096	8192	16384	32768	65536
								Number of Processors
	Figure 5-5:

Table 5 .

 5 Each pixel on the initial image is used to calculate a set of n F feature images (primary features), this step involves the computations of gradients, color transformations, bilinear interpolations and/or LBP texture operators. The For a one megapixel image of size 1024 × 1024 and using double precision floating point arithmetic (which needs 8 bytes per value) to compute the integral images the algorithm would require 1024 × 1024 × 35 × 8 ≈ 293 MB of memory!. All this transformations apply low level algorithms where the input image is transformed to create a set a new image (or a set of images), a set of algorithm optimizations are proposed in this section to improve the algorithms runtime performance, the set of notations common to all the different versions of the algorithm is given here first. The input image I whose height and width is represented by h and w, is first decomposed by the ϕ(I, x, y) operator into a collection of n F feature images stored in F . From them n P images of the crossed-feature products (i.e. n P = n F (n F + 1)/2) are obtained and stored in P . Table5.2 summarizes all these notations. 2: Covariance descriptor algorithm notations.

	Notation	Meaning
	I h and w n F	Input image Height and width of I Number of features used to build the descriptor
	n

next steps consist in calculating the n P feature crossed products (or second order features) and the integral images from all the primary and second order features, that makes n F + n P integral images in total. Lets calculate now the amount of memory space required by the covariance matching algorithm, as n P = n F (n F + 1)/2, for the case of the ELBCM Lum descriptor we have n F = 7 primary features and n P = 28 secondary features that makes 35 integral images in total. P Number of crossed-products of features n P = n F (n F + 1)/2 F A data structure that contains all the features images P A data structure containing all the feature image products I F and I P The summed area tables (integral images) computed from F or P pixel features product of features

Table 5 .

 5 3: Baseline scalar SoA algorithm profiling results for ROI and full image results.

	Function name	Description	ROI CPU Time	%	Full-image CPU Time	%
	lbp_r1u8_operator_f32 sumMatrices_SOA rgb8matrix_get_L_RGB_F32 Gets L,R,G and B features LBP operator I F and I P computation bilinearInterpol Bilinear interpolation	3.235s 2.034s 0.492s 0.462s	51.98% 32.69% 7.91% 7.42%	2.910s 25.616s 0.504s 0.243s	9.94% 87.51% 1.72% 0.83 %
	the full-image results it appears that the execution speed of the algorithm is considerably affected by the number of pixels
	being treated, here, function sumMatrices_SOA consumes about 87.50% of the application running time becoming the
	most critical function to optimize.
	The kernel function (sumMatrices_SOA) can be decomposed into three different stages:
	1. point-to-point features crossed products,
	2. the integral image computation of features,
	3. the integral image computation of products.

Table 5 .

 5 4 compares the profiling results obtained for both versions, the gain of the SoA→AoS transformation is a speedup of about 1.5x. In turn, Table5.5 enlists some of the hardware events reported by Intel VTune Amplifier XE 2013 related to memory accesses (for instructions and data) in both SoA and AoS versions, the number cache misses and stalls is

significantly reduced, for example the number of MEM_LOAD_RETIRED.LLC_MISS events which corresponds to last level cache misses and accesses to the local DRAM (signaled by the MEM_UNCORE_RETIRED.LOCAL_DRAM) is reduced almost by a half.

Table 5 . 4 :

 54 SoA vs AoS profiling (full image version).

	Function name	Description	SoA full-image	AoS full-image
			CPU Time %	CPU Time %
	sumMatrices_SOA/AOS	Computes I F and I P	25.616s	87.51% 16.665s	81.89%
	lbp_r1u8_operator_f32	LBP operator	2.910s	9.94%	2.688s	13.21%
	rgb8matrix_get_L_RGB_F32 Gets L,R,G and B features	0.504s	1.72%	0.469s	2.31%
	bilinearInterpol	Bilinear interpolation	0.243s	0.83%	0.528s	2.59%

Table 5 .

 5 5: SoA vs AoS hardware event count

	Hardware Event Type	SoA		AoS	AoS+SIMD
		Count	Count	Ratio (vs. SoA) Count	Ratio (vs. SoA)
	MEM_LOAD_RETIRED.L1D_HIT	41.418×10 9 35.828×10 9 0.865	12.272×10 9 0.296
	MEM_LOAD_RETIRED.L2_HIT	10.402×10 6 5.214×10 6	0.501	33.821×10 6 3.251
	MEM_LOAD_RETIRED.LLC_MISS 13.267×10 6 7.284×10 6	0.549	8.453×10 6	0.637

Algorithm 16: Products of features SoA version 1 k ← 0 2 foreach k1 ∈ [0..nF -1] do 3 foreach k2 ∈ [k1..nF -1] do 4 foreach i ∈ [0..h -1] do 5 foreach j ∈ [0..w -1] do 6

  X X P 16 P 17 P 18 P 19F 3 F 3 F 4 F 4 F 5 F 6 F 4 F 5Seven vector products are enough to calculate the 28 crossed products for the n F = 7 case with a degree of parallelism of 4 (four 32-bit integer values or single precision floating point values) with 128-bit width SIMD registers.

	a0	a1	a2
	b0	b1	b2
	a3	a4	a5			a6	F 4 F 5 F 5 F 6
			X	X	X	X	X	X	X	X
	b3	b4	b5			b6	F 6 F 5 F 6 F 6
			P 20 P 21 P 22 P 23	P 24 P 25 P 26 P 27
	Figure 5-13:

Table 5 .

 5 Tables 5.[START_REF] Bak | Multiple-shot human re-identification by mean riemannian covariance grid[END_REF] and 5.7 provide an estimation of the algorithmic complexity and the amount of memory accesses for both scalar and SIMD (SSE and Neon) versions. They also provide the arithmetic intensity (AI) (a measure popularized by Nvidia) that compares the number of arithmetic operations (including the number of permutations for SIMD version) and the number of memory accesses. Note that the scalar version has a low AI of 0.5 as the number of memory accesses is twice the number of operations. It can also be noticed that the SIMD version has ×2.7 less operations for SSE (respectively 1.6 for Neon) and ×4.8 less memory accesses, thus the AI ratios reach 0.9 for SSE and 1.5 for Neon.A comparison of the profiling results obtained using AoS and AoS+SIMD versions of the algorithm is shown in Table 5.8. The part of the algorithm which was vectorized is represented in code by the function sumMatrices_AOS_SIMD and has a speedup of 2.61x with respect to its scalar version making it about four times faster than the SoA baseline version. 6: Complexity and arithmetic intensity of scalar AoS version.

			AoS scalar version with 3 loops
	Instructions	Arithmetic instructions MUL ADD	Memory accesses LOAD STORE	Arithmetic Intensity
	Product of features Integral of features Integral of products Total	n P 0 0 n P	0 3n F 3n P 3(n P + n F )	2n P 4n F 4n P 6n P + 4n F 2n P + n F n P n F n P	----
	With n P = n F (n F + 1)/2 and n F = 7		2n 2 F + 5n F 133	4n 2 F + 9n F 259	-0.5

), other permutations require instructions only available with 64-bit registers, like the look-up table instruction named vtbl. So in Neon the process is more complicated and 128-bit float registers should be: 1) split into 64-bit registers, 2) type-casted into 64-bit integer registers, 3) permuted with vtbl instructions 4) type-casted into 64-bit float registers and 5) combined into 128-bit float registers. In total 48 SIMD Neon instructions are required to create the seven pairs of products. For the scalar SoA baseline version, this function represented about 84.07% of the running time execution, in the scalar AoS version it was reduced to about 81.89%. With the AoS+SIMD version it represents now only 64.45%. The total execution

Table 5 .

 5 7: Complexity and arithmetic intensity of the SIMD AoS version.

		AoS SIMD (with n F = 7) version with 3 loops
	Instructions	Arithmetic instructions Memory accesses Arithmetic Intensity MUL ADD LOAD STORE
	Product of features Integral of features Integral of products	7 0 0	0 21 6	2 28 2	7 7 2	---
	Total SSE (+ 15 PERM) Total Neon (+ 48 PERM)	49 82		54 54		0.9 1.5

time of AoS+SIMD for the Pedxing3 sequence on a Nehalem architecture (Intel Core i5) was about 11.525 the AoS+SIMD version is 2.78x faster than the SoA one and 1.82x faster than the scalar AoS.

Table 5 . 8 :

 58 AoS and AoS+SIMD profiling results (full image version).

	Function name	Description	AoS full-image	AoS+SIMD full-image
			CPU Time %	CPU Time %
	sumMatrices_AOS/AOS_SIMD Computes I F and I P	16.665s	84.07% 6.382s	64.46%
	lbp_r1u8_operator_f32	LBP operator	2.688s	13.56% 3.001	30.31%
	rgb8matrix_get_L_RGB_F32 Gets L,R,G and B features	0.469s	2.37%	0.518s	5.23%

Table 5 .

 5 10: Complexity and arithmetic intensity for the loop fusion + scalarization version.

		AoS scalar version + Loop Fusion
	Instructions	Arithmetic instructions MUL ADD	Memory accesses LOAD STORE	Arithmetic Intensity
	Integral of features Integral product of features Total	0 n P n P	2n F 2n P 2(n P + n F )	2n F n P n P + 2n F n P + n F n F n P	---
	Total with n P = n F (n F + 1)/2 Total with n F = 7		1.5n 2 F + 3.5n F 98	n 2 F + 4n F 77	-1.3

Table 5 .

 5 11: Complexity and arithmetic intensity of scalar and SIMD versions with loop-fusion and serialization transformations and scalarization. AoS SIMD (with n F = 7) version + (loop fusion + scalarizations)

	Instructions	Arithmetic instructions Memory accesses Arithmetic Intensity MUL ADD LOAD STORE
	Integral of features Integral product of features	0 7	4 14	4 7	2 7	--
	Total SSE (+ 15 PERM) Total Neon (+ 48 PERM)	40 73		20 20		2.0 3.7

Table 5 .

 5 [START_REF] Bhatia | Positive Definite Matrices[END_REF]: cpp and execution time for Intel U9300

	Sequence	Panda (312 × 233) SoA AoS++ SoA AoS++ Pedxing (640 × 480)
	Features computation (cpp) 128 Kernel computation (cpp) 599 Tracking (cpp) 23 Total (cpp) 738 Kernel / total 81 %	150 87 23 248 35 %	128 618 11 769 80 %	150 91 11 264 34 %
	Total speed-up 1-C execution time (ms) 2-C execution time (ms)	45 36	×2.9	15 9	197 158	×2.8	68 38

Table 5 .

 5 14: cpp and execution time for ARM Cortex-A9

	Sequence	Panda (312 × 233) SoA AoS++ SoA AoS++ Pedxing (640 × 480)
	Features computation (cpp) 461 Kernel computation (cpp) 1491 Tracking (cpp) 96 Total (cpp) 2048 Kernel / total 73 %	461 395 96 952 42 %	486 1600 19 2106 73 %	486 415 19 921 45 %
	Total speedup 1-C execution time (ms) 2-C execution time (ms)	149 108	×2.2	69 36	647 492	×2.2 283 149

  t)∥ 2 dt. (B.15) If γ(t) is any path joining A and B in P n , then H(t) = log γ(t) is a path joining log A and log B in H n (the length of the path in Euclidean space). The right-hand side of equation (B.15) imposes a lower bound: the length of the straight line segment joining log A and log B. This means that L(γ) ≥ ∥ log A -log B∥ 2 .For any two matrices H and K in H n δ 2 (e H , e K ) ≥ ∥H -K∥ 2 , so the map(H n , ∥ • ∥ 2 ) (B.17)which joins H and K in H n . In the particular case where A and B are commuting matrices in P n the distance

exp

--→ (P n , δ 2 ) (B.

[START_REF] Birchfield | Spatiograms versus histograms for region-based tracking[END_REF]

) is metric increasing.

However, it can be proved that when A and B commute, the exponential map carries the line segment joining log A and log B in H n to the geodesic joining A and B in P n .

Parametrically, we write [H, K] for the line segment

H(t) = (1t)H + tK, 0 ≤ t ≤ 1

http://www.cvg.rdg.ac.uk/PETS2009

http://www.mis.tu-darmstadt.de/node/428

http://andresromeromier.wikispaces.com/

Cela est fait par _mm_shuffle_ps en SSE

At each location, compute any of the scalar coarsity (interest) measures (e.g. Harris corners (1.26)).Schmid et al. performed tests to evaluate many of the different interest point detectors that have been proposed in computer vision[START_REF] Schmid | Evaluation of interest point detectors[END_REF]. They first evaluated the detection repeatability, that is how many times a feature is kept under varying conditions such as rotations, scale and illumination changes and the presence of noise. Secondly they evaluated the information content available at each of the detected feature points. They defined the information content as the entropy

http://www.primesense.com/

The method of Lagrange multipliers (named after Joseph Louis Lagrange) is a strategy for finding the local maxima and minima of a function subject to equality constraints.

A Hilbert space H is a real or complex inner product space that is also a complete metric space with respect to the distance function induced by the inner product.

Defined in various ways such as ∥A∥F = √ ∑ m i=1 ∑ n j=1 |a ij | 2 = √ trace(A * A) = √ ∑ min{m, n} i=1 σ 2 i .

http://staff.science.uva.nl/~aloi/

Hue is artificially represented in range [0-255] although it is an angular value. Therefore some of the noise is due to that representation, and some of the noise is due to ill definition of Hue.

available on http://vasc.ri.cmu.edu/idb/html/jisct/index.html

available on http://i21www.ira.uka.de/image_sequences/#taxi

http://www.kasrl.org/jaffe.html

Video Annotation Tool from Irvine, California (VATIC) http://web.mit.edu/vondrick/vatic/.

http://andresromero.github.io/

http://www.cvg.rdg.ac.uk/PETS2009

http://www.mis.tu-darmstadt.de/node/428

http://andresromeromier.wikispaces.com/

Intel Xeon Phi coprocessors provide up to 61 cores,

threads, and 1.2 teraFLOPS of performance.

http://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/h57-and-h55-express-chipset-brief. pdf

KB, 8-way, Access: 4 cycles.

http://www.tilera.com/products/processors/TILE-Gx_Family

Only 23 bit are explicitly stored but a 24-th bit is implied by the format.

The ratio between the arithmetic complexity and the number of memory accesses.

k ← k + 1

FoT+CT evaluation experiments

To evaluate the performance of the FoT+CT cooperation algorithm a set of very challenging sequences was selected. Most of them include partial and complete occlusions, targets moving very fast or very adverse imaging conditions where the objects appear ill-defined and/or blurred. Table 3.2 summarizes the set of sequences included in this dataset. The first three columns denote the sequence name, the total number of frames considered for the tests and their citation or URL for download. The three sub-columns on the table (under Last Frame sub-heading) indicate the overall results obtained by each method: FoT, CT and FoT+CT represented by the frame number at which each method stopped. The last group of sub-columns (under the FoT+CT heading) writes down the behavior of the switching mechanism: the percentage of frames the algorithm spends using the CT exhaustive search algorithm (%CT), the average number of milliseconds demanded by each method (FoT and CT). The rightest column indicates the total speed-up obtained by using the switching mechanism in comparison to executing CT alone for the entire sequence.

The descriptive abilities of out ELBCM descriptor are evaluated for three different applications: texture and facial expression classification and tracking. Two different datasets are used for texture classification: the Brodatz and KTH-TIPS. The classification accuracy provided by ELBCM for these datasets is compared against results obtained by the following state-of-the-art methods: VZ-join, Lazebnik, Hayman, Tuzel, HLSS, Jalba, L 2 ECM and LBCM.

ELBCM evaluation

In this subsection, we exhibit some possible applications of our ELBCM descriptor such as texture and facial expression classification and object tracking.

Texture classification

For performance evaluation we used two widely used datasets in texture classification. The Brodatz dataset and KTH-TIPS dataset [START_REF] Hayman | On the significance of real-world conditions for material classification[END_REF].

Brodatz Dataset: The Brodatz dataset contains 111 different textures each one represented by a single 640 × 640 image. Scale, viewpoint or illumination changes are not a problem in this dataset, the main difficulty of this dataset in terms of classification is that it includes non-homogeneous textures, Figure 4-3 displays some texture samples taken from this dataset. To have comparable results, we have followed the same methodology of [START_REF] Li | Local log-euclidean covariance matrix (L2ECM) for image representation and its applications[END_REF] where each image is subdivided into 49 overlapping blocks of size 160 × 160 using a pixel block stride of 80. Each class is trained using twenty four randomly selected images from their respective 49 images set (the remaining ones are used for testing). During training, each image is now divided into four 80 × 80 patches and their ELBCM covariances are then computed. During testing, each covariance matrix of the testing image is compared to all the covariance matrices inside the training set using the Riemannian metric proposed in 2.30. A label is assigned to each test image using a KNN algorithm (k = 5) which counts the number of votes gathered by the four test image covariance matrices for each class. In Figure 4-4 the classification accuracy results for the Brodatz dataset are reported. Different feature map configurations in the covariance matrices were tested: Tuzel [START_REF] Tuzel | Region covariance: A fast descriptor for detection and classification[END_REF], LBCM [START_REF] Guo | Facial expression recognition using local binary covariance matrices[END_REF] and ELBCM, other state-of-the-art results are included too (taken from [START_REF] Li | Local log-euclidean covariance matrix (L2ECM) for image representation and its applications[END_REF]). The reported results are the average of twenty different experiment runs.

Energy minimization strategy

Our energy function is certainly nor linear neither convex, this problem is tackled evaluating every pair of combinations resulting from the crossing between the set of detections D t with the tracklets set P t . This way our problem reduces to a classical combinatorial optimization problem that can be efficiently solved by Munkres algorithm [START_REF] Munkres | Algorithms for the assignment and transportation problems[END_REF] (the Hungarian method) as in [START_REF] Jungling | Detection and tracking of objects with direct integration of perception and expectation[END_REF]. For each detection d i ∈ D t the minimization algorithm builds a list of candidate paths measuring the distance between d i and the predicted location xt j of the active path p j ∈ P t . The TBD between the d i 's covariance descriptor Ĉi and p j 's model c j may also be considered. Paths whose predictors at frame t are outside a circle of radius r from the d i and whose covariance model is extremely dissimilar are considered impossible combinations and an infinite energy is assigned accordingly. In the case of few or no paths close to any of detections, the null candidate is added indicating the creation of new path. Once the the algorithm has a complete list of candidates with their respective energies, it creates a new list of non-exclusive combinations {M k } k=0,••• ,K-1 . Munkres algorithm evaluates the cost of all these non-exclusive combinations and selects the less expensive one 2 From all the different p ℓ 's build the list of K possible candidate combinations:

3 For each combination in {M k } k=0,••• ,K-1 , apply equation (4.11) to evaluate the energy associated to it.

4 Select the optimal combination M t = min({M k } k=0,••• ,K-1 ) having the minimal energy (using Munkres method).

5 Update the paths position, predictions and appearance model.

6 Create new paths as M t indicates.

Multiple object tracking evaluation

The current best practice to evaluate multiple object tracking algorithms is to use of the CLEAR-metrics [START_REF] Stiefelhagen | The CLEAR 2006 evaluation[END_REF] which concretely defines what an ideal multiple object tracking algorithm should be able to do and proposes a set of measures to evaluate how good an algorithm is at this task. An explanation about these metrics is provided before passing to the analysis of the results obtained using the multiple tracking method proposed by this thesis.

CLEAR Multiple object tracking evaluation metrics

Following the definition proposed by the CLEAR metrics a multiple object tracking algorithm should be able to:

1. determine the correct number of objects present in the image at all points in time, 2. estimate the position of each object as precisely as possible,

Block size:

64 bytes for all caches. Figure 5-2: Targeted architectures associativity and cache line size are in conflict if the programmer wants to hold, for instance, a small rectangular section of an image or a matrix in cache. One simple solution is to copy the desired block into a contiguous temporary buffer, this may have an initially high cost but alleviates cache thrashing if the data section is accessed frequently.

Main memory

Main memory is the next element in the memory hierarchy of a computer architecture. extracted by increasing the dispatch logic and the number of functional units available, using SIMD instructions sets only the width of the functional units is increased. The problem with SIMD extensions is that the programmer and the compiler are responsible of extracting the instruction level parallelism (ILP) that previously was extracted by the out of order core of the microprocessor. As many multimedia and digital signal processing (DSP) algorithms work on data vectors, SIMD programming is well adapted to these classes of applications.

For an image X its integral image I at the pixel (x, y) is the sum of all the pixels values of X above and to the left of it. This is expressed mathematically as:

(5.4) Equation (5.4) is implemented by Algorithm 12 using two loops for iterating within the image pixel elements. A different type of nation is used here, the pair of images X(x, y) and I(x, y) is now denoted by X[i][j] and

where the spatial location (x, y) is now denoted by [i][j], where i represents the index of the lines and j indexes the columns. These is the type of notations used consistently throughout the rest of the chapter to explain other image processing algorithms making them compatible with popular notations used for linear algebra and image processing algorithms in C (refer to Numerical Recipes in C [START_REF] Press | Numerical Recipes 3rd Edition: The Art of Scientific Computing[END_REF]). The relation

One pitfall of Algorithm 12 is that as it progresses in its walk across the image, the magnitude of the values accumulated on I may rapidly grow in comparison to the values on X, if the partial sums stored in I grow large enough, new values to add from X will be less than the smallest representable increment and their summation will have no effect. A two-pass algorithm is thus proposed to alleviate this situation, all the accumulations are now separated into two different passes:

one vertical and one horizontal. If the accumulations are executed first in the vertical direction, S y [i][j] accumulates all the pixels in the j-th column of X from line 0 to line i. Conversely, the accumulator S x [i][j] can be used in the place of S y if the accumulations are executed in the horizontal direction first. Following this approach, the magnitudes differences between the operands to add and the running accumulator are reduced together with the chances of incurring in accuracy errors (see Figure 5678910). The S y accumulator of the two-pass integral image algorithm (passing in the vertical direction first) is represented mathematically as:

Where S y (x) calculates the vertical accumulations of all the terms in column x from line zero up to line y. The final accumulations are performed in the second pass as: Two different version of the two pass algorithm exists depending on the initial direction of accumulation (i.e. vertical or horizontal), Algorithms 13 and 14 show both versions. As the number of for loops is doubled, the two-pass integral image is slower than the single pass (speed-optimized method) and each pass inside the loop demands one addition (ADD), two load operations (LOAD) and one store (STORE). In consequence, the arithmetic intensity goes down to AI = (2ADD)/(4LOAD + 2STORE) = 0.333. An this analysis without considering the lack of spatial locality of the data when computing the vertical accumulations, in summary, the two-pass accumulation method sacrifices the execution speed in favor of the accuracy of the accumulations.

Algorithm 13: Vertical accumulation 2-pass method. Data: An image X of size h × w.

Result:

The integral image I of size h × w.

Algorithm 14: Horizontal accumulation 2-pass method. Data: An image X of size h × w.

Result:

The integral image I of size h × w.

A variation of the two-pass accumulation method for the integral images calculation can be obtained using the following recurrence relation

where each pixel This variation of the 2-pass accumulation algorithm is equivalent (i.e the elements in X[i][j] and the intermediate values are accumulated in exactly the same way) the difference is that this method runs in a single pass, requires less memory (only two intermediate arrays that are swapped) and only requires two additions (ADD), three loads (LOAD) and two stores (STORE). So, its arithmetic intensity is AI = 2/5 = 0.4. In conclusion, this method offers a good compromise between the advantages of the speed-optimized method and the precision-optimized two-pass method.

Algorithm 15: 1-pass integral image computation algorithm optimized for speed and accuracy. Data: An image X of size h × w.

Result:

The integral image I of size h × w.

For the case of the feature images used to compute the covariance descriptor, we can assume pixels to be in the range 0 to 255 which requires 8-bit to represent them. The single precision floating point data type (float 32-bit) has 24 bits for the significand, thus a total of:

values can be summed without accuracy problems using the single pass version of the algorithm. Equivalently for the double precision floating point values (double 64-bit), there are 53 bits for the significand which can store a total of:

values.

These values are enough for calculating integral images of maximum sizes 256 × 256 for the single pass method and 65, 536 × 65, 536 for 2-pass method using single floating point precision. For the case of double precision data types, its possible to accumulate a totality of (5.931641 × 10 6 ) × (5.931641 × 10 6 ) = 35.1843 × 10 12 using the single pass method.

The real limitation appears when dealing with the integral images of second-order degree features (feature crossedproducts). For these images the range of values goes from 0 to 2 16 = 65, 535 (supposing 8-bit fixed point feature images), the maximum number of pixels we can accumulate using single precision floating point integral images for these crossedproducts is:

which limits us to patches of size 16×16 for the single pass accumulation method, and 256×256 for the 2-pass accumulation method. For the case of double floating point precision the algorithm can handle:

values of this type, which means that images of size 370, 727 × 370, 727 can be represented adequately enough by double precision data images using the single pass accumulation method. For the type of feature images being used, single precision floating point integral images are clearly not enough to guarantee the correctness of the algorithm when handling images bigger than 256 × 256 due to insufficient number of bits used to handle precision in the significant.

Concerning the implementation of the covariance descriptor algorithm, the necessity of recurring to double precision is an undesired constraint. Not only do 64 bit operations usually have a bigger latency, but more importantly, the degree of SIMD parallelism is reduced by two (without considering the overhead of handling 64-bit data instead of 32-bit).

Furthermore, the set of Neon SIMD instructions does not include 64-bit instructions.

Some alternatives exist that might be useful to overcome or reduce the impact of this problem:

• Considering the possibility of not to use integral images. This option surely has an strong impact on the speed of the algorithm when comparing multiple covariance descriptors (as it is the case for tracking applications).

• If all the inputs are about the same size, the parallel inclusive scan pattern implementation represented in arrangement (detailed in [START_REF] Mccool | Structured Parallel Programming: Patterns for Efficient Computation[END_REF]) is advantageous as it allows us to achieve some degree of parallelism, but it is not perfect either, indeed it is quite easy to invent input values that break any specific ordering of the operations.

• Using compensated summations [START_REF] Higham | Accuracy and Stability of Numerical Algorithms[END_REF], for instance, the Kahan summation algorithm which significantly reduces the numerical error by keeping a separate running compensation, which is a variable used to accumulate the small errors.

Figure 5-11: Serial and parallel implementations of the inclusive scan pattern. Image taken from [START_REF] Mccool | Structured Parallel Programming: Patterns for Efficient Computation[END_REF].

Covariance tracking algorithm baseline analysis

In the baseline version of the algorithm the complete set of feature images F is stored separately using a cube data structure

) which can be regarded as an instance of a structure of arrays (SoA) data structure.

The index k is used here to select the desired feature image and the pair (i,j) to select the spatial coordinates. Image cubes are straightforward to implement, the required arithmetic to compute the memory a address using a table of 3D pointers only demands three integer additions, still, the latency time of a memory access is extremely dependent on the data access pattern.

Two different versions of the algorithm were benchmarked:

1. Region of interest (ROI): the algorithm uses previous target position information to estimate a confined tracking zone where the algorithm is convinced that target is located.

2.

Full image: all feature computations and integral images are calculated uniformly for every pixel location on the input image I.

The ROI version of the algorithm is suggested for single target tracking applications, as it reduces significantly the computing effort, but as the number of concurrent targets grows it is wiser to operate uniformly over the complete image.

Both versions were benched using VTune Amplifier XE 2013 using a Nehalem architecture (Intel Core i5) with 4 GB of RAM and running Ubuntu Linux 13.04. The profiling results for these baseline implementations are shown in Table 5.3

using the Panda and Pedxing3 sequences already presented in the previous chapters.

In the ROI implementation, function lbp_r1u8_operator_f32 (responsible of applying the LBP operator to compute the pattern angles to introduce in the ELBCM feature vector) takes about 51.98% of the execution time, the second most consuming function (32.68% of the execution time) is sumMatrices_SOA, this function computes the kernel of the covariance descriptor algorithm. Based on these results the priority should be given to lbp_r1u8_operator_f32, however, in Result: An integral image cube I of size n × h × w.

Algorithm 19: Integral image -AoS version, n ∈ {n F , n P } Data: An image X of size h × (n • w).

Result: An integral image I of size h × (n • w).

Code vectorization

Now that the features are laid out continuously in memory using the proposed AoS form, it is easier to extract parallelism using SIMD instructions. The final objective is to reduce the total number of memory accesses and arithmetic instructions.

The most consuming part in the scalar implementation corresponds to the computation of the crossed feature products.

The two internal loops in Algorithm 17 k 1 and k 2 are fully unrolled and the complete set of multiplications is obtained by first constructing the list of factors (in vector form) through permutation instructions (e.g., _mm_shuffle_ps in SSE). For example, for a typical value of n F = 7 features there are n P = 28 final products, thus, seven vector products are enough (using four-element vectors). The associated vectors to obtain all these products in our example are (the numbers are the feature indexes):

In the mentioned case, all the seven features (contained in a pair of 128-bit SIMD registers) are distributed into fourteen vector combinations that are necessary to calculate the seven vector products. A collection of permutation intrinsics (SSE or Neon) is employed to get them. The initial 4-element vector pair x0 and x1, they contain all the n F = 7 features that are distributed to create the product factors (a0 to a6 and b0 to b6) used to calculate the complete set of products. After this, seven SIMD instructions are then used to perform all the 28 multiplications of the original scalar algorithms. where N 2 corresponds to an image of size N × N . The cpp results are plotted versus the number of pixels (image size).

Detecting the cache overflow (when data do not fit in the cache) is possible using the cpp metric, it occurs when the curve increases significantly. This metric reflects the influence of memory transfers on computation using a fair metric which is independent from the processors clock and which offers a valid methodology to compare different architectures.

Three different versions were tested (SoA+OpenMP, AoS, AoS+SIMD), they were benchmarked on three generations of Intel processors: Penryn4, Nehalem and SandyBridge and varying the images sizes from 128 × 128 up to 1024 × 1024. factor here is the latency of the instructions. Table 5.12: Impact of loop-fusion and serialization transformations, speedups obtained for Intel's ULV U9300 and Cortex-A9.

As it appears in

Algorithm version

Let us now focus on the impact of the loop-fusion and serialization transformations, (Table 5.12) lists the speedups Concerning power efficiency, there is a factor 4 of speed between the Intel U9300 and the Cortex-A9. However, as the

Contributions

The first group of contributions presented in Chapter 3 reinforce the adaptability of the tracking algorithm to the context.

First, the context can change in terms of illumination. To tackle this problem, we proposed a method which selects color features automatically according to their relevance. The color is used only when the saturation is high enough. In addition, a simple and compact color invariant has been proposed. This contribution has been published in [START_REF] Romero | Feature points tracking adaptive to saturation[END_REF] and has won the Best Student Paper award. The resulting invariant is used to improve the results of the FoT algorithm. It can also be used to form an ELBCM-based texture and color-invariant covariance descriptor (see the tests in Section 4.2.2).

Second, the chosen tracking algorithm can fail for some reason, for instance a loss of discriminant power or an occlusion.

Additionally, the tracking algorithm can fail because it is not well chosen for a given object, for example, when FoT is used on an object which has no salient feature points. The FoT+CT and MS+CT switching mechanisms increase the robustness of the final tracking algorithm by compensating their individual failures this research work has lead to a journal publication [START_REF] Laguzet | Color tracking with contextual switching: Real-time implementation on CPU[END_REF].

Finally the aim of the last group of contributions to the field of computer vision was to improve the distinctiveness of the covariance descriptor by mixing texture (in the form of LBP angles) and color information. The resulting ELBCM descriptor was published in [START_REF] Romero | Enhanced Local Binary Covariance Matrices ELBCM for texture analysis and object tracking[END_REF], it is presented in Chapter 4. ELBCM can be used for many applications such as facial expression and texture recognition, object matching and tracking, target re-identification, etc. Our work in target re-identification using an array of covariance matrices and evaluating different color-spaces led to a conference publication [START_REF] Romero | Covariance descriptor multiple object tracking and re-identi cation with colorspace evaluation[END_REF]. Mixing the target appearance and their dynamics it was possible to propose a multi-target tracking procedure based on the tracking-by-detection approach which is suitable for on-the-fly operation [START_REF] Romero | Total bregman divergence for multiple object tracking[END_REF].

Chapter 5 analyzed the kernel of the covariance descriptor computation algorithm and proposed some simple techniques to accelerate its implementation on embedded GPP processors such as the ARM Cortex-A9. The set of acceleration techniques prosed were published in [START_REF] Romero | Real-time covariance tracking algorithm for embedded systems[END_REF] and they are completely scalable for future processor architectures which provide a higher degree of parallelism (in the form of wider SIMD registers or a higher number of cores). This algorithm has already been ported to a faster embedded platform (ARM Cortex-A15) and our team at the LRI is now discussing with representatives of Intel and Kalray to create collaboration projects to port this algorithm to their many-core platforms: the Intel Xeon Phi co-processor and the MPPA-Manycore of Karlay.

Future work

There are still a lot of topics to explore in the domain of detection, matching and tracking using covariance matrices.

The evaluation of our algorithms, particularly the proposed ELBCM method, can be continued. It could be interesting to test the training of an ELBCM-based pedestrian detection using Kernel methods on Riemannian manifolds (as described in Section 2.1.3.3). We would also like to test the ELBCM descriptor using the L 2 ECM approach proposed in [START_REF] Li | Local log-euclidean covariance matrix (L2ECM) for image representation and its applications[END_REF].

For the multi-target tracking algorithm, the plans for the future are based on the exploration of new techniques to use the appearance information (modeled by covariance matrices) and the dynamics of the targets using integer and linear programming as in [START_REF] Milan | Continuous energy minimization for multi-target tracking[END_REF]. To optimize the algorithm there are two main points to treat: to tackle precision for integral image computation for wider images (typically HD 1920 × 1080 pixels) (and how to parallelize it if it makes sense with the required precision); and the efficient parallelization of the multi-target tracking on multi-core and many-core systems like Intel XeonPhi for high-performance systems and Kalray MPPA for embedded systems.

APPENDIX A KLT optical flow models

In this appendix two models used in the KLT optical flow are detailed, the translation model which is valid only when the surface is locally planar around the point and the motion is fronto-parallel. And the affine transformation model which handles more complex transformations rather than simple translations. Affine maps and homographies may associate different velocities to different points inside the window. However, the size of the window area has to be chosen as a trade-off, it has to be long enough to have enough information to estimate the motion model and small enough to avoid being time consuming.

A.1 KLT with a translational motion model

Dropping the time variable we have that J(x) = I(x, y, t + ∆t) and I(x -d) = I(xξ, yη, t), the local image model is

where d is the displacement vector which minimizes the double integral over the neighborhood window W:

In equation (A.2), w assigns a weight to each point inside W. Usually, the value of w is determined with a Gaussianlike function centered at W so that pixels close to the borders of the region are considered less important than pixels close to the center. In the simplest case, a uniform value such as w = 1 is assigned to every point.

In the general case, there is no linear relation between pixel intensities. When the displacement vector d is small, a good linear approximation is obtained considering the Taylor expansion up to the first order term. The result of this linearization is The correct affine model is found by iterating in a Newton-Raphson style.

While the affine model considers both the deformations and translations, Lucas and Kanade [START_REF] Shi | Good features to track[END_REF], warn against modeling the two problems at the same time. This is because the deformation matrix D and displacement vector d interact through V , and errors in the estimation of D affect d and vice-versa. So, attempting to determine deformation parameters is not only useless but can lead to poor displacement solutions. Frame by frame, the linear deformation of the feature window (modeled by D) is likely to be small (D should be close to the zero matrix), in contrast, the magnitude of d can be significant even from one frame to the next, the affine model is necessary to adapt the feature window, deforming it when perspective changes occur. It is not necessary to estimate D at every frame because window deformations are only important once enough changes have accumulated.

Matrix Information Geometry

All the concepts introduced in this chapter were taken from mathematical texts specifically designed for computer and data scientists [START_REF] Gallier | Lectures on geometric methods for computer science[END_REF], [START_REF] Bhatia | Positive Definite Matrices[END_REF] and [START_REF] Porikli | Tutorial on differential geometry in computer vision and pattern recognition[END_REF]. The work of Nielsen and Bhatia [START_REF] Nielsen | Matrix Information Geometry[END_REF] deserving special interest.

Differential Geometric Methods

Differential Geometric Methods are being used for many different applications such as Shape Analysis and Activity Recognition. In these areas, we often need to do some comparatives between object instances and registered models, perform classification between two or more classes or to calculate the average of some samples to get a single model which somehow arrives to represent them all (to estimate a barycenter for example). For simplicity reasons, and maybe because of the academic origins of the majority of the computer vision community, the convenience of carrying out these calculations in their own spaces instead of forcing the Euclidean metrics to fit into the problems has been disregarded.

Here, we have some examples of some typical problems we often want to solve:

5. Assume M is some kind of geometric space. For a, b ∈ M , find a shortest path from a to b.

If M is a vector space, there are good and very popular methods for solving such problems. However, if M is a curved space it may be very difficult. Nonlinear manifolds are spaces that are not vector spaces, for example cases where the Similarly, the natural parametrization of the geodesic [A, B] when A and B commute is

which means that δ 2 (A, γ(t)) = tδ 2 (A, B) for each t.

In the general case (when A and B can be any two elements of P n ), there exists a unique geodesic [A, B] joining A and B. This geodesic has a parametrization

for each t.

From this parametrization, it can be proved that the geodesic length is

Equation (2.27) is the Riemannian metric on the manifold P n . Applying the definition of the norm ∥ • ∥ 2 we have

where λ i are the eigenvalues of the matrix A -1 B.

B.0.7 Model Update Techniques

B.0.7.1 Symmetric Positive Definite Matrices (SPD) Averages

Because of occlusions and appearance changes, tracking objects for the long-term is a difficult problem. To identify occlusions and follow appearance changes, a robust algorithm must adapt its object description. Such updating mechanism needs to recognize the relevant information available from all the description samples. It must be insensible to abrupt changes, and should consider all the information at hand to improve object discrimination.

Covariance matrices are a subset of the symmetric positive definite matrices (SPD) set. The averaging of this type of matrices is an important and open problem which arises when one has to represent as closely as possible(and through a single matrix) the information contained in several n × n positive matrices A 1 , • • • A m . Matrix Information Geometry is the field of mathematics dedicated to these type of problems. Good sources of information from this area of mathematics are [START_REF] Nielsen | Matrix Information Geometry[END_REF] and [START_REF] Bhatia | Positive Definite Matrices[END_REF]. for (j = 0; j < 100; j = j + 1)

B.3 Pedestrian re-identification tables

for (i = 0; i < 5000; i = i + 1)

x