Alexandre

Stéphane

Laurent, Thibault, Philippe, Youssouf, Eric, Suresh, Christophe, . . . Karim

rection of this thesis and having guided me throughout the whole work. I would like to acknowledge my three thesis supervisors, Martyna POREBA, Erwan PIRIOU and Maroun OJAIL

The aim of this study is to analyze, compare and retain the most relevant tracking methods likely to respect the constraints of embedded systems, such as Micro Aerial Vehicles (MAVs), Unmanned Aerial Vehicles (UAVs) and intelligent glasses, in order to find a new robust embedded tracking system. A typical VINS consists of a monocular camera that provides visual data (frames), and a low-cost Inertial Measurement Unit (IMU), a Micro-Electro-Mechanical System (MEMS) that measures inertial data. This combination is very successful in the system navigation field thanks to the advantages that these sensors provide, mainly in terms of accuracy, cost and reactivity. Over the last decade, various sufficiently accurate tracking algorithms and Visual Inertial Navigation Systems (VINS) have been developed, however, they require greater computational resources. In contrast, embedded systems are characterized by their high integration constraints and limited resources. Thus,in this thesis, a solution for embedded architecture, relaying on efficient algorithms and providing less computational load, is proposed.

First, relevant tracking algorithms are studied focusing on their accuracy, robustness, and computational complexity. In parallel, numerous recent embedded tracking computation architectures are also discussed. Then, our robust visual-inertial tracking approach, called: "Context Adaptive Visual Inertial SLAM", is introduced. It alternates between visual KLT-ORB and EKF Visual-Inertial tracking, according to the navigation context, thanks to the proposed execution control module. The latter uses several parameters concerning the scene's appearance, the motion types, etc. Consequently, tracking continuity, robustness and accuracy are improved, even in difficult conditions. Moreover, our proposal is suited to embedded systems integration, given the low algorithms computational complexity and the implemented PoIs management leading to decrease the number of PoIs as well as the occurrences of their detection. All our experiments and tests was performed using the different EuRoC dataset sequences. Au cours de cette dernière décennie, différents algorithmes performants de suivi ont été développés. En revanche, ils nécessitent des ressources calculatoires conséquentes, compte tenu des différentes formes d'utilisation possibles. Or, les systèmes embarqués imposent de fortes contraintes d'intégration, ce qui réduit leurs ressources, particulièrement en termes de capacité calculatoire. Ainsi, ce type de système nécessite de recourir à des approches efficaces avec moins de charge et de complexité calculatoire. L'enjeu de cette thèse réside dans cette problématique. L'objectif est d'apporter une solution embarquée de suivi qui permettrait d'assurer un fonctionnement robuste dans différents environnements de navigation. Une analyse des algorithmes pertinents de suivi, visuel et visuel-inertiel et des environnements de navigation ainsi qu'une étude de différentes architectures embarquées de calcul sont menées, afin de proposer notre solution nommée « système de suivi inertiel-visuel adaptatif à l'environnement de navigation~». Cette dernière consiste à alterner entre deux approches de suivi : KLT-ORB et EKF VI Tracking, selon les conditions de navigation du système, grâce au module de contrôle, tout en assurant la cohérence du système global en gérant le nombre de PoIs et l'occurrence de leur détection et en respectant les contraintes des systèmes embarqués. Tous nos expérimentations et tests ont été réalisées en utilisant le jeux de données EuRoC.

Mots-Clés. Suivi visuel, Suivi visuel-inertiel, Suivi robuste, Localisation et Cartographie Simultanée (SLAM), Odométrie, Navigation visuelle-inertielle, Systèmes multi-capteurs, Systèmes embarqués, Co-Conception, Fusion, Couplage centrale inertielle/caméra, Couplage lâche, Couplage serré. 1.9 Methodologies and example methods chronological overview (inspired from [START_REF] Csurka | From handcrafted to deep local invariant features[END_REF]) . . . . . . . . . . . . 1.10 Trade-off between the different characteristics of the cited algorithms (KLT here represent Shi Tomasi PoI detection). 1.11 Global workflow of MatchNet system [START_REF] Han | MatchNet: Unifying feature and metric learning for patch-based matching[END_REF] 1.16 An overview of the 3D-to-3D algorithm [START_REF] Scaramuzza | Visual Odometry [Tutorial][END_REF] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.17 An overview of the 3D-to-2D algorithm [START_REF] Scaramuzza | Visual Odometry [Tutorial][END_REF] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.18 KLT feature tracking algorithm [START_REF] Kyu | Kanade-Lucas-Tomasi (KLT) feature tracker[END_REF]

List of Figures

3.8

The smallest drone available on the market that uses VIO to estimate its own position announced by Qualcomm [START_REF] Suleiman | Navion: A 2-mW Fully Integrated Real-Time Visual-Inertial Odometry Accelerator for Autonomous Navigation of Nano Drones[END_REF]. (Figure captured from [START_REF] Loianno | Estimation, Control, and Planning for Aggressive Flight With a Small Quadrotor With a Single Camera and IMU[END_REF]) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.9 Co-design method overview (algorithm-hardware trade-off) (inspired from [START_REF] Zhang | Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach[END_REF]) . . . . . . . . . . . . 3.10 Overview of the ISC algorithm operating diagram (inspired from [START_REF] Zhang | Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach[END_REF]) . . . . . . . . . . . . . . . . . [START_REF] Suleiman | Navion: A 2-mW Fully Integrated Real-Time Visual-Inertial Odometry Accelerator for Autonomous Navigation of Nano Drones[END_REF] VIO pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.12 Photo of Navion chip (die) [START_REF] Suleiman | Navion: A 2-mW Fully Integrated Real-Time Visual-Inertial Odometry Accelerator for Autonomous Navigation of Nano Drones[END_REF] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . [START_REF] Ullah | EMoVI-SLAM: Embedded Monocular Visual Inertial SLAM with Scale Update for Large Scale Mapping and Localization[END_REF]) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Navion

EMoVI-SLAM diagram (from

3.14

Wearable EMoVI-SLAM (a head-gear and a processing unit tied on waist) (from [START_REF] Ullah | EMoVI-SLAM: Embedded Monocular Visual Inertial SLAM with Scale Update for Large Scale Mapping and Localization[END_REF]) . . . . . . . . . . 3.15 The embedded VI SLAM PIRVS package [START_REF] Zhang | PIRVS: An Advanced Visual-Inertial SLAM System with Flexible Sensor Fusion and Hardware Co-Design[END_REF] . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.16

Loose coupling of PIRVS/GPS (PIRVS Loosely Coupled Sensor Fusion Example) [START_REF] Zhang | PIRVS: An Advanced Visual-Inertial SLAM System with Flexible Sensor Fusion and Hardware Co-Design[END_REF] . . . . . . . . . .

3.17

4.5

The ETH Machine Hall environment (MHxx) [START_REF] Burri | The EuRoC Micro Aerial Vehicle Datasets[END_REF] . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 The Vicon environment (Vxx) [START_REF] Burri | The EuRoC Micro Aerial Vehicle Datasets[END_REF] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.7

The Asctec Firefly hex-rotor helicopter for data acquisition [START_REF] Burri | The EuRoC Micro Aerial Vehicle Datasets[END_REF] . . . . . . . . . . . . . . . . . . . . .

5.2

The comparison of numerous runs of ORB SLAM (m) (horizontal axis shows the IDs of the processed frames) 5.3 KLT-based Visual Odometry (with and without scale) vs. KITTI 02 ground truth trajectories (m) . . . . . . 5.4 KLT-based Visual Odometry (with and without scale) vs. KITTI 01 ground truth trajectories (m) . . . . . . xvii 5.5 On the left: EKF VI system without any scale estimation vs. EuRoC dataset (V101) ground truth; On the right: EKF VI system with scale estimation, based on the distance between two images, vs. EuRoC (V101 & MH01) dataset ground truth (m) (horizontal axis shows the IDs of the processed frames) . . . . . . . . . 5.6 On the left: EKF VI system without any scale estimation vs. EuRoC dataset (V101) ground truth; On the right: EKF VI system with scale estimation, based on the IMU pose computed between two images, vs.

EuRoC dataset (V101) ground truth (m) (horizontal axis shows the IDs of the processed frames) . . . . . . 5.7 On the left: EKF VI system without any scale estimation vs. EuRoC dataset (V101) ground truth; on the right: EKF VI system with scale estimation, based on the rate between inertial and fused poses, vs. EuRoC Récemment, les systèmes de navigation ont évolué et sont devenus plus intégrés, mobiles et autonomes grâce à l'utilisation d'architectures de calcul de plus en plus embarquées. Plusieurs implémentations embarquées de différents algorithmes de suivi ont été réalisées dans ce sens. Elles permettent d'assurer un suivi suffisamment précis mais peuvent impliquer une complexité calculatoire importante. Par ailleurs, la forte densité d'intégration des systèmes électronique impose des limitations en termes d'énergie, d'encombrement et de capacité calculatoire. La problématique majeure des systèmes embarqués de suivi réside dans l'adéquation des différents algorithmes efficaces aux architectures de calculs embarqués, en respectant les contraintes imposées, en particulier en termes de capacité calculatoire. Le sujet de cette thèse, intitulée "Caméra intelligente embarquée pour le suivi robuste d'objets sur plateforme mobile", s'intéresse à cette problématique et vise à proposer un système de suivi visuel-inertiel adaptatif à l'environnement de navigation respectant les contraintes des systèmes embarqués. Afin de mener à bien ce travail de recherche, une étude bibliographique des divers algorithmes et techniques de suivi a été mene. Ensuite, les différentes architectures et contraintes d'un système embarqué de navigation visuelle-inertielle, ainsi que les méthodes d'adéquation algorithmearchitecture, connues sous le nom de « co-design », sont décrites et discutées. Dans le même temps, une étude sur les conditions et les environnements de navigation a été réalisée en se fondant principalement sur le jeu de données public EuRoC. Le choix de ce jeu de données repose sur la disponibilité des informations inertielles et visuelles qui illustrent des mouvements de drones dans des environnements variés. Ces environnements incluent des scènes avec différentes textures et luminosités et avec des vitesses linéaires et angulaires et des types des déplacements et des rotations divers. Les algorithmes et techniques exploités dans la proposition qui émane de cette thèse prennent en considération les différentes catégories d'environnement de navigation identifiées suite à ces analyses et études de ce jeu de données. En effet, la solution proposée se base sur un module de contrôle d'exécution qui permet d'analyser le contexte de navigation et d'alterner par la suite, entre deux différentes techniques de suivi. La première, purement visuelle, elle est fondée sur l'algorithme Kanade-Lucas-Thomasi (KLT) et destinée à la navigation dans des environnements faciles, caractérisés par des mouvements lents et stables et des scènes texturées et illuminées. La deuxième, visuelle-inertielle, elle est basée sur l'algorithme Extended Kalman Filter (EKF). Elle est adaptée à la navigation dans des environnements complexes, qui se caractérisent principalement par des mouvements rapides et des problèmes de texture et de luminosité. En s'adaptant à l'environnement de navigation tout en réduisant le nombre de points à suivre et en basculant entre ces deux méthodes de suivi, le système proposé peut produire des résultats efficaces, robustes et adaptés à des implémentations embarquées pour des systèmes comme les Micro-Aerial Vehicles (MAVs).

• Contributions

Pour traiter la problématique posée dans cette thèse, présentée et discutée auparavant, nous avons adopté une orientation de recherche qui se focalise, tout d'abord, sur l'analyse, la comparaison et l'utilisation des algorithmes les plus pertinents pour le suivi, tout en respectant les contraintes des systèmes embarqués, et qui implique, ensuite, de mettre en place :

• Une étude des différents environnements de navigation , ainsi qu'une identification et une description des caractéristiques des environnements faciles et difficiles de navigation,

• Un système de suivi adaptatif alternant deux méthodes de suivi grâce à un module de contrôle d'exécution et une méthodologie de sélection de PoIs. Il est constitué de trois parties :

1. la méthode de suivi visuel basée sur le KLT-ORB "KLT-ORB Tracking", dédiée aux environnements faciles de navigation (sans vibration, sans mouvement rapide, sans flou, suffisamment de texture.), 2. la méthode de suivi visuel-inertiel basée sur EKF "EKF VI Tracking", destinée aux environnements difficiles de navigation (mouvements rapides, zones noires, scènes non texturées.), 3. le module de contrôle d'exécution, basé sur de nombreuses métriques afin de décrire l'état de l'environnement de navigation et la qualité du suivi actuel.

Système embarqué pour un suivi robuste

Cette thèse s'organise en cinq chapitres. Les deux premiers chapitres s'intéressent aux systèmes de navigation visuels et visuels-inertiels, respectivement. Ensuite, le troisième chapitre traite les différents systèmes et architectures embarquées utilisés spécifiquement dans des systèmes de suivi. Enfin, les chapitres quatre et cinq se focalisent sur la présentation et la discussion de notre proposition et de ses résultats, respectivement. Dans ce qui suit, nous résumons les différents points traités dans chaque chapitre de cette thèse afin de fournir un aperçu sur son contenu global.

• Etat de l'art

De nombreux travaux de recherches traitant la thématique de la navigation, particulièrement le suivi, ont été proposés. Les méthodes de navigation peuvent être classées selon différents critères. La catégorisation la plus répandue dans la littérature est celle qui divise ces méthodes en "Odométrie" et "Localisation et Cartographie Simultanées (Simultaneous Localization And Mapping (SLAM))". Tout d'abord, l'odométrie est le processus d'estimation du mouvement d'un robot (translation et rotation) en observant une séquence d'images de son environnement. Ce processus de traitement peut impliquer des accumulations d'erreur odométriques au fil de temps. L'odométrie visuelle (Visual Odometry (VO)), plus précisément, représente un cas particulier d'une technique nommée "Structure From Motion (SFM)" [START_REF] Fraundorfer | Visual odometry: Part i: The first 30 years and fundamentals[END_REF]. Cette dernière traite la problématique de la reconstruction 3D de la structure de l'environnement en utilisant un ensemble d'images séquentiellement ordonnées ou non.

Or, le SLAM [56][11] représente un moyen pour qu'un robot arrive à se localiser dans un environnement inconnu. En plus, contrairement à l'odométrie, le SLAM autorise la construction progressive d'une carte de son environnement et assure un retour qui permet de réduire l'erreur accumulée au fil de temps. Le SLAM a été largement étudié au cours des deux dernières décennies [START_REF] Debeunne | A Review of Visual-LiDAR Fusion based Simultaneous Localization and Mapping[END_REF], ce qui a donné lieu à de nombreuses solutions [108][153][175] [START_REF] Ullah | EMoVI-SLAM: Embedded Monocular Visual Inertial SLAM with Scale Update for Large Scale Mapping and Localization[END_REF] basées sur différents capteurs, notamment des caméras et des IMUs.

Systèmes visuels

La majorité des systèmes de suivi se décompose en deux étapes de calcul. La première est destinée au prétraitement des images. Pendant cette étape, il est possible d'utiliser des techniques basées sur le flux optique ou sur l'extraction des caractéristiques de l'image, telles que les Points d'Intérêt (PoIs). En effet, les PoIs représentent des caractéristiques d'image connus pour leur précision et leur aspect générique. Notre étude de différentes approches de la littérature a conduit à identifier les algorithmes de détection/description des PoIs comme le plus approprié pour un système de suivi à ressources limitées. En effet, cet algorithme permet de détecter un nombre suffisant de PoIs tout en étant invariant à la majorité des transformations requises, notamment la rotation. Quant aux méthodes basées sur le flux optique, l'algorithme KLT présente également un intérêt particulier. C'est une méthode différentielle qui prend comme hypothèse que le flux est constant dans un voisinage local du pixel considéré, et résout l'équation du flux optique pour tous les pixels dans ce voisinage par la méthode des moindres carrés. En effet, l'algorithme KLT offre un suivi visuel précis particulièrement en cas des déplacements courts entre deux images successives dans une séquence d'images. Il est caractérisé également par sa légère complexité calculatoire par rapport aux algorithmes basés sur la détection/description des PoIs.

Différents systèmes visuels de navigation et de suivi (VO et SLAM) sont proposés dans la littérature et fournissent des résultats précis et fiables. Cependant, ils sont influencés par différentes contraintes liées à l'environnement de navigation, telles que les environnements sombres, peu texturés ou des mouvements qui brouillent l'image, ce qui réduit considérablement leur robustesse et peut provoquer l'échec du suivi. De plus, il faut garder à l'esprit qu'un système visuel monoculaire souffre de la dérive d'échelle dans le temps due à l'utilisation d'une seule caméra. Par conséquent, afin d'améliorer les performances de suivi, il est nécessaire soit de multiplier le nombre de caméras (stéréovision), ce qui imposera un calcul supplémentaire, soit d'utiliser d'autres types de capteurs afin d'assurer des données complémentaires à celles de la caméra, comme l'intégration des mesures inertielles issues de l'IMU.

Systèmes visuel-inertiels

Les systèmes multi-capteurs de suivi revêtent un grand intérêt dans la recherche académique et industrielle. La combinaison caméra/IMU est parmi les couplages les plus répandus. Ces deux capteurs se caractérisent par leur complémentarité. En effet, l'IMU fournit des mesures d'accélération et de vitesse angulaire à une fréquence importante tandis que la caméra produit des informations visuelles précises mais à une fréquence réduite. Les systèmes de navigation multi-capteurs, visuels-inertiels, peuvent être divisés, en fonction du type de couplage, en deux catégories : couplage serré ou couplage lâche. Cette catégorisation est orthogonale au découpage des méthodes de navigation en se basant sur la distingtion VO et SLAM. De même que les systèmes visuels purs, les systèmes de suivi visuel-inertiel ont besoin de passer par le calcul de la pose. Bien évidemment, cette étape est différente de celle utilisée pour les systèmes visuels purs à cause de l'introduction des mesures inertielles. Ainsi, pour ce type de système, un algorithme de fusion de données est recommandé, notamment l'algorithme EKF vu son adéquation avec nos besoins en terme de précision et du respect de la capacité calculatoire des systèmes à ressources limitées.

Architecture embarquées pour le suivi Actuellement, il existe différentes architectures de calculs dédiées et d'accélération. Afin de choisir l'architecture la plus adaptée pour l'implémentation d'une application donnée, il est nécessaire de trouver un compromis entre la qualité des résultats et les contraintes imposées par un système embarqué. La limitation de la capacité calculatoire est considérée parmi les principales contraintes à prendre en compte lors de la conception d'un systèmes embarqué. Or, l'utilisation de plusieurs capteurs engendre un nombre important de données à traiter et à fusionner, ce qui a un effet sur la complexité calculatoire du système. Par conséquent, concevoir un système embarqué de suivi visuel-inertiel nécessite des architectures de calcul embarqué plus spécialisées et plus raffinées selon les objectifs du système.

• Suivi visuel-inertiel adaptatif à l'environnement de navigation

Dans cette thèse, nous proposons un système basé sur les données visuelles-inertielles qui peut assurer le suivi dans différentes conditions de navigation, grâce à un module de contrôle qui alterne entre deux approches de suivi, visuelle et visuelle-inertielle. Pour cela, tout d'abord, une analyse de l'environnement de navigation est réalisée pour permettre, d'une part, d'identifier les environnements de navigation problématiques qui nécessitent un plus grand effort en termes de temps et de capacité de calcul, d'autre part, d'identifier les cas où la navigation peut être plus facilement assurée et les contraintes sont assouplies. Ainsi, un suivi robuste basé sur la combinaison de différentes méthodes algorithmiques est proposé et est assuré par un module de contrôle déié. Ce dernier est utilisé pour analyser les données inertielles, la qualité de l'image, le nombre de PoIs et le retour d'information sur la qualité du suivi, entre les images actuelles et précédentes, afin d'identifier le type de mouvement (rotation, translation) et la qualité des données visuelles. Il vise également à choisir la méthode de suivi la plus adaptée à l'environnement de navigation actuel. Dans ce travail, un choix motivé est fait entre l'approche visuelle KLT-ORB Tracking, basée sur le flux optique via l'algorithme KLT, et l'approche visuelle-inertielle EKF VI Tracking, basée sur la fusion de données en utilisant l'algorithme EKF.

Le suivi visuel KLT-ORB Tracking permet un suivi plus rapide, en particulier dans un contexte de navigation facile (scènes à faible mouvement et texturées), ce qui est avantageux pour les systèmes embarqués mobiles. En effet, dans la plupart des travaux, un suivi visuel dans un système SLAM est effectué en détectant les PoIs et en les décrivant pour chaque image. Cette approche nécessite un temps d'exécution et un coût de calcul importants (e.g. au moins 10ms pour ORB SLAM). Cela explique les avantages de l'utilisation de la méthode KLT-ORB Tracking basée sur le flux optique. Elle permet de calculer rapidement la pose relative entre l'image actuelle et l'image précédente, en diminuant la fréquence de détection/description des PoIs. En particulier, dans le cas de mouvements de translation et de rotation relativement faibles. Aussi si la cartographie et la création de KeyFrame ("image clé") ne sont pas nécessaires, la redétection d'un nombre élevé de PoIs est dispensable ce qui réduit le temps de traitement. Ainsi, le suivi visuel KLT-ORB Tracking limite la complexité du calcul des PoIs et de la localisation des poses pour le SLAM. Néanmoins, lorsque l'environnement de navigation est difficile, le suivi est effectué en fusionnant les données inertielles et visuelles à l'aide de l'algorithme EKF. C'est une technique de suivi précise et efficace malgré les coûts de calcul élevés par rapport à la méthode visuelle pure. Il est possible de réduire ce temps et d'en tirer parti pour une mise en oeuvre solide du SLAM sur les systèmes mobiles embarqués. Ceci est réalisé en alternant entre les deux méthodes, grâce au module de contrôle d'exécution qui vise à analyser l'environnement de navigation et, en fonction du type de mouvement et des conditions de la scène, l'approche de suivi appropriée est sélectionnée et exécutée.

• Expérimentations

Nous avons réalisé des différentes expérimentations afin de pouvoir évaluer la qualité globale du système proposé. Dès nos premières expérimentations et tests des approches de la littérature, tel que ORB SLAM, nous avons pu soulever trois aspects problématiques, liés principalement à l'évaluation des trajectoires estimées, qui sont le non-déterminisme, l'estimation de l'échelle et l'alignement des trajectoires. Afin d'évaluer correctement notre proposition et de la comparer par rapport aux autres travaux de l'état de l'art nous avons adopté la méthodologie décrite ci-après. Tout d'abord, pour remédier au non-déterminisme, nous avons utilisé dans nos calculs la trajectoire moyenne calculée suite à dix exécutions de chaque approche testée. Ensuite, nous avons analysé plusieurs techniques d'estimation d'échelle parmi lesquelles nous avons retenu la méthode qui consiste à calculer le ratio entre la pose inertielle et la pose fusionnée (visuelle-inertielle), cette estimation est inspirée de la proposition présentée dans [START_REF] Spaenlehauer | A Loosely-Coupled Approach for Metric Scale Estimation in Monocular Vision-Inertial Systems[END_REF] Finalement, le système de suivi adaptatif a été évalué dans sa globalité. Nous avons ainsi pu vérifier l'intérêt de l'utilisation du module de contrôle en étudiant la qualité du suivi, le temps de calcul mais aussi l'impact de la phase de transition entre les deux méthodes. La transition entre les deux modes impose une erreur qui peut aller jusqu'à 0.30 mètres et un temps de calcul entre 0.25 et 0.27 secondes, en fonction de la méthode qui prendra le relais. En effet, chacune des deux méthodes de suivi a une propre phase d'initialisation dont la différence majeure réside dans le nombre d'images exploitées. La qualité globale du suivi adaptatif représente une erreur ATE de 0.08 mètres sur les séquences faciles et de 0.19 mètres sur les séquences difficiles, ainsi qu'un temps de calcul estimé par 32 f ps pour les séquences difficile et par 29 f ps pour les séquences faciles. La solution proposée fournit des résultats de précision comparable avec la plupart des résultats des travaux connus de la littérature. Elle permet aussi de garantir la robustesse du suivi dans différents environnements de navigation, même les plus contraignants. Enfin, cette méthode permet d'avoir des résultats en terme de temps de calcul acceptables et rend envisageable une implémentation sur une architecture embarquée.

Conclusions et perspectives

Cette thèse présente notre proposition d'une méthode de suivi visuelle-inertielle robuste adaptée à la mise en oeuvre des systèmes embarqués mobiles tels que les MAVs. D'une part, les algorithmes de suivi sont contraints par les conditions du contexte de navigation, en particulier, ceux basés sur la vision. Ces derniers sont sensibles aux scènes noires à faible texture et aux mouvements rapides de la caméra qui peuvent provoquer des flous dans l'image. Par conséquent, les mesures inertielles sont utilisées dans l'objectif d'améliorer ces systèmes visuels, leur intégration impose une complexité calculatoire plus importante à cause des pré-traitements des mesures brutes de l'IMU et de la fusion avec les données visuelles. D'autre part, les systèmes embarqués sont contraints en termes de ressources. Ils sont limités par la puissance de calcul, la mémoire embarquée et la charge utile des capteurs. Ainsi, ils ne peuvent pas gérer facilement et directement la mise en oeuvre d'algorithmes de suivi à haute complexité de calcul. Cette thèse décrit et évalue notre proposition dédié aux systèmes embarqués tout en assurant l'adéquation algorithme-architecture. Elle consiste à alterner entre deux méthodes de suivi : le suivi visuel KLT-ORB Tracking et le suivi EKF VI Tracking, grâce au module de contrôle d'exécution qui analyse le contexte de navigation (mouvement et scène) et permet de passer à la méthode de suivi appropriée. L'évaluation du suivi visuel-inertiel adaptatif proposé assure un fonctionnement robuste dans les différents environnements de navigation, faciles et difficiles, fournissant une précision, exprimée en termes de RMSE d'ATE de 0.08 mètres dans les environnements faciles et de 0.19 mètres dans les environnements difficiles. Ces résultats sont obtenus grâce à l'efficacité des algorithmes de suivi choisis et au module de contrôle d'exécution.

Les recherches présentées dans cette thèse offrent des prolongements et perspectives potentiels pour la suite des travaux. Ces derniers visent à étendre les contributions fournies et à améliorer la praticabilité de la solution proposée dans les applications réelles. Pour les futurs travaux, nous avons comme prévisions les sujets suivants :

-Amélioration le processus d'estimation de l'échelle : l'échelle est un problème de

Introduction Problem Background

Motion tracking is a challenging field of research in computer vision (CV), robotics and more generally artificial intelligence (AI). This topic is presented, especially in our context, as a measure of the positioning of an object moving in a predefined space. Thus, body position and/or orientation are estimated. Actually, it is possible to measure only position (x, y, z) so this is called the 3 Degrees of Freedom (3 DoF or 3D) tracking, or to measure position (3 coordinates) and orientation (3 independent angular coordinates), simultaneously, then this is known as the 6 DoF or 6D tracking.

Nowadays, motion tracking is becoming an increasingly important task in a wide range of application and intelligent devices, for example in military, entertainment, sports, medical applications and also in validation of CV and robotics. The following paragraphs describe some of the relevant applications examples of motion tracking: Health Care and Person Aid for example, in [START_REF] Khanh | A Virtual Blind Cane Using a Line Laser-Based Vision System and an Inertial Measurement Unit[END_REF] [START_REF] Kun | Hardware implementation of a virtual blind cane on FPGA[END_REF], an application based on motion tracking and designed specifically for visually impaired people (figure 1) is proposed. It is a real-time virtual blind cane system, that combine three sensors in a loosely coupled way : a camera, an IMU and a laser.

The distance of the obstacles, regarding the camera, is calculated using the triangulation based on the laser stripe center position and the camera parameters. Inertial data are integrated at this step, applying Kalman Filter algorithm, in order to perform the motion tracking and compute the system pose. The latter is used to determine the distance to obstacles with respect to the user's body.

FIGURE 1: Overview of the real-time virtual blind cane system proposed in [START_REF] Inspex H | INSPEX-Integrated Smart Spatial Exploration System[END_REF] .

Motion Tracking and Environmental Understanding Google ARCore [START_REF] Google | ARCore Fundamental Concepts[END_REF] is an example of works on Android devices, and focuses on three functionalities which are motion tracking, environmental understanding and light estimation. Motion tracking uses the phone's camera to track Points of Interest (PoIs) (figure 2), in the room coupled to IMU sensor data, in order to determine the 6D pose of the phone as it moves. Environmental understanding detects horizontal surfaces using the same PoIs computed for motion tracking. Light estimation leverages the ambient light sensors in order to and SLAM (Simultaneous Localization and Mapping), which concerns a combination of the motion tracking used for the localization, and the map building task that is done simultaneously with camera motion tracking.

Generally, tele-operated and autonomous navigation systems (odometry-or SLAM-based) are able to capture, process, and actively make sense of the information provided by their sensors. The acquired information depends mainly on the system purpose and its integrated sensors. For example, the landmarks, which are detected on the images captured by the system-mounted cameras, and used in visual tracking and localization tasks, are sufficient to ensure a navigation environment mapping, obstacle avoidance and path planning, however, an additional dense occupancy map can be recommended. Actually, motion tracking is tied to localization and environment mapping. An effective motion tracking process provides accurate navigation environment features, thus ensures accurate localization and environmental mapping. Meanwhile, the miniaturization of electronic systems enabled small-scale, mobile and wearable smart communicating objects (e.g. Internet of Things (IoT)) to become increasingly widespread. This miniaturization aims to perform various complex systems in different fields, such as mobile multisensor embedded systems for navigation and tracking, intelligent glasses, smart cameras, UAVs and Micro Air Vehicles (MAVs). The latest examples are among the highly constrained embedded devices. Therefore, the design of such systems imposes serious challenges in terms of computing capacity, system integration (form factor) and resource constraints, like battery life and energy consumption. The main object of this thesis is to find an adequacy between the algorithms quality and the embedded architecture constraints in order to provide a robust and accurate embedded tracking system. Thus, different tracking approaches are discussed, including visual and visual-inertial ones.

In the following, an overview of these discussions is introduced.

• Visual Motion Tracking

There are numerous methods to process visual data for different tracking tasks. Recently, deep learning techniques have been largely applied to CV and robotics technology. These techniques are used in various applications, such as facial recognition, object recognition, camera pose improvement and motion tracking. However, they require a high computing capacity and an interconnection between the system and external data resources. This involves an additional cost, in terms of energy consumption and computational complexity, and makes the system dependent instead of autonomous. Besides deep learning techniques, there are various other handcrafted visual data processing techniques that are employed in different applications. For example, in PoIs detection, description and matching tasks. These image processing techniques do not require any external network connectivity for data exchange, and save energy with regard to the deep learning techniques. Compared to other systems, only-vision based ones provide an accurate results. This is not only thanks to the algorithms and processing methods used, but also to the precision of the visual data coming from the camera. However, this sensor provides data at a slower frequency, compared to other sensors (cadence). It can also suffer from occultation and less textured environments that cause loss of information, thus loss of system accuracy and robustness. To overcome these problems, it is required to combine several homogeneous/heterogeneous sensors like IMU and camera coupling.

• Visual-Inertial Motion Tracking

Many literature papers are written about visual-inertial data fusion [START_REF] Qin | VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator[END_REF][93] [START_REF] Qin | VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator[END_REF], obtained from a camera/IMU combination. Cameras and IMUs are particularly attractive for providing navigation methods as they are characterized by their low power consumption, light weight and wide availability in the market at attractive prices. An IMU may contain one, two (gyroscope and accelerometer (6 axis)), or three inertial sensors (gyroscope, accelerometer and magnetometer, etc.), that provide raw inertial measurements. Integrating inertial data provides a pose information that enables to overcome the problems of only vision-based navigation methods. The camera/IMU combination comes in different configuration categories, which can depend on different fusion topologies: centralized, decentralized or hierarchical fusion, as well as on different sensors coupling: tightly coupled or loosely coupled. The fusion topology and/or coupling type choice relies on the system's resources and application field. A loosely coupled system requires less computational complexity than a tightly coupled one, but it does not provide the same accuracy as the latter. Furthermore, the visual-inertial fusion requires the use of a data fusion algorithm. There are a wide variety of these algorithms, the most commonly ones in case of tracking systems are: Kalman Filter (KF), Extended Kalman Filter (EKF), Unscented Kalman Filter (UKF) and Particle Filter (PF). Selecting the appropriate algorithm for a given navigation system computation is supported by its nature. For example, Kalman Filter algorithm cannot deal with the non-linearity problem, however, the other three algorithms are designed for this fusion case. The decision depends also on the algorithm computational complexity, and its result accuracy. Moreover, camera/IMU combination requires a hardware integration thought. It is necessary to take into account the processing cost and the management of all this data when making processing algorithmic choices. Computational complexity and overall performance are among the main criteria to be considered when choosing the most appropriate algorithm.

Research Objectives & Contributions

Based on all the previous discussions, our aim in this thesis is to design and develop a new tracking system that ensures a robust and accurate motion tracking and respect the embedded systems constraints, simultaneously. To do so, first, our work's positioning within the relevant literature and its main issues are presented below. Afterwards, the main researches objectives are listed. This is before ending with a summary of the main contributions of this work, the list of publications and the thesis structure.

• Positioning

Further to the previous analysis, this work proposes an embedded multi-sensor tracking system based on the camera/IMU combination. It aims at tracking the system's location continuously at any moment. To achieve this, the proposed solution is a visual-inertial tracking based on PoIs. These latter are image features that provide a generic and accurate representation of the environment. Inertial data derived from the IMU, often fused with visual data, is also used for tracking processes in a specific computation framework. Using visual or visual-inertial data for tracking involves two different computation approaches which are presented and discussed in details in this thesis. In addition, this work focuses on various embedded computing architectures and their main constraints regarding the processing architecture implementation. Computational architectures can be programmed with a programming language such as C, C++, etc., for instance GPP, GPU, DSP and ASIP, or described by a descriptive language like VHDL, for example ASIC and FPGA. Choosing the right target architecture becomes more obvious when a compromise is found between the computational complexity of the used algorithms and the different embedded system constraints, such as its performance, space

• PoIs management methodology performed between different processing functions, that provides the system consistency and saves memory and computational time by limiting the number of the PoIs at 500 or 5 PoIs.

• List of Publications

The papers published during this thesis are provided herein:

Imane Salhi,

Thesis Structure

The thesis is organised in five main chapters.

⋄ Chapter1, provides an overview of state-of-the-art on visual navigation systems. Starting by introducing the main visual systems classification. Then, the main components of visual tracking techniques are discussed. Herein we focus on detailing the different PoIs detection and matching methods, as well as on the explanation of various visual pose estimation. We also introduce some of relevant vision-based tracking and navigation systems.

⋄ Chapter2, deals with visual-inertial navigation systems. Different Visual Inertial Navigation System (VINS) categories and methods are introduced. In addition, a thorough overview of some well-known examples in literature is provided in order to illustrate techniques explained before in this chapter.

⋄ Chapter 3 presents a discussion of multi-sensor embedded navigation systems. In fact, in this chapter, we present the embedded systems classification and constraints. Subsequently, we focus on the different computational architectures, which are used mainly in embedded tracking systems. Lastly, an overview of the current tracking and navigation embedded systems, based on visual-inertial combination, is discussed.

⋄ After all previous discussions and state-of-the-art overviews, the proposed approach is described in chapter 4. The latter provides the navigation environments analysis, performed in order to identify the problematic environments and the main metric impacted by these latter.

Next, the overall workflow of our Context Adaptive Visual-Inertial SLAM is presented. Further to this, the Context Adaptive Visual-Inertial Tracking is explained in depth details. First the proposed execution control module metrics and workflow are detailed, then visual KLT-ORB and EKF Visual-Inertial tracking are addressed.

⋄ Chapter 5 deals the previously mentioned Context Adaptive Visual-Inertial Tracking experi- mental assessment. First, the experimental and assessment environment is described and the main trajectory evaluation issues, including the non-determinism, the scale estimation and the trajectory alignment, are highlighted and addressed. Then, the trajectory evaluation is discussed based on exhaustive analysis and experiments. Afterwards, results of the different proposition components assessments are reported and explained, and the results of the overall solution appraisal is presented and compared to other relevant tracking methods.

⋄ Finally, Conclusion summarizes the thesis by presenting an overview of the proposed approach and discussing their experimental results.In addition, research perspectives and final thoughts are also discussed.

Chapter 1

Visual Navigation System Navigation systems can be roughly seen as a process that has the ability to determine an appropriate and safe path between a starting point and a destination. A lot of navigation methods have been proposed but each often respond to different constraints. Therefore, the required navigation method is highly depending on the field of application. In this chapter, we are particularly interested in visual navigation and its various classes and components.

In the early 2000s, vision-based navigation has proven to be a promising and primary research topic for navigation systems. This is, especially, due to the rapid development of computer vision domain field, which is extending their application area. These navigation systems include obstacle detection, autonomous navigation, localization, etc. Firstly, the visual sensors used for vision-based navigation systems provide complete information about the navigation environment. Also, they are very suitable for dynamic environment perception thanks to their anti-interference capability.

Moreover, such sensors allow to acquire more precise data. Generally, the main steps in the navigation system process are: image acquisition, features detection and matching, pose estimation, tracking and map building/updating if it is required (depending on the navigation system category).

Feature extraction involves identifying robust features in the image, as edges, points of interest (PoIs), etc. It is an essential step as it is relied upon during the rest of all computations. Therefore, in this chapter this topic is addressed with a focus on PoIs as being the image features most suitable for this thesis project. This is because they are a generic and precise image features that facilitate many navigation and tracking process in different environment conditions. Also, the main algorithms for detecting and matching these PoIs are discussed. In addition, the localization and estimation of the pose (position, orientation) impact significantly a navigation system's quality and performance. Therefore, pose estimation is considered one of the most important challenges in a navigation system.

For this reason, in our review we are particularly interested in different pose estimation methods applied and developed for visual navigation and tracking systems.

In this chapter, firstly, the visual navigation system classification, as well as its two main methods which are Visual Odometry (VO) and Simultaneous Localization And Mapping (SLAM) are discussed in section 1.1. Then, in section 1.2, the upstream image processing techniques that are usually involved in a visual navigation system are addressed. The focus is on PoIs detection, description and matching algorithms, because most of the relevant literature systems rely on these image features as they are more generic and accurate for vision-based localization and tracking processing. Afterwards, the relevant methods and processes used in the literature for estimating visual pose are presented in section 1.3. Finally, section 1.4 gives a detailed description of widely used works of the state-of-the-art (EKF MonoSLAM [START_REF] Andrew J Davison | MonoSLAM: real-time single camera SLAM[END_REF], PTAM [START_REF] Klein | Parallel Tracking and Mapping for Small AR Workspaces[END_REF], and ORB SLAM [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF]) to illustrate the discussions of the previous sections.

Systems Classification

Several vision-based navigation classifications are proposed in the literature [START_REF] Desouza | Vision for Mobile Robot Navigation: A Survey[END_REF] [22] [START_REF] Vincent Gay Bellile | Review and classification of vision-based localisation techniques in unknown environments[END_REF]. First of all, vision-based navigation was categorized into two main types: indoor navigation and outdoor navigation [START_REF] Desouza | Vision for Mobile Robot Navigation: A Survey[END_REF]. Additionally, each of these two categories is further classified into different subcategories.

Outdoor navigation can be subdivided in structured and unstructured environments. For the first type, the system navigates on the basis of the road following technique, where the system (robot) detects the road lines and followed them in order to navigate consistently. While for the second one, the system has no properties to follow for navigation, and other alternative solutions are used rate by detecting visual variations in the "visual event" images. However, as they are insufficiently mature, their performance for SLAM applications cannot yet be concluded.

In general, SLAM and VO are two important navigation approaches, however there are interesting differences between them. Table 1.1 summarizes a comparison between the main features of each approach. The visual SLAM provides a consistent global system trajectory estimation based on the

VO Visual SLAM Definition

A method of estimating the ego-motion (camera's motion relative to a rigid scene) of a system using mainly input of camera(s) attached (frames)

V-SLAM is a process that allows system building a map of an environment and using it to know its pose from camera(s) Goal Aims to recover the trajectory incrementally, pose after pose

Aims to obtain a global, consistent estimate of the robot trajectory and keeping a track of environment map Optimization

Optimizing only over the last poses Uncertainty will be optimized, when loop closure is detected Consistency Only local consistency of the trajectory and the local map is used to obtain estimate of the local trajectory

Whereas SLAM is concerned with the global map consistency TABLE 1.1: Comparison between VO versus visual SLAM [START_REF] Scaramuzza | Visual Odometry [Tutorial][END_REF] environmental map record. In fact, the latter is used to close the loop when the robot returns to an area already visited, thus allowing to reduce the drift of the map and the camera trajectory, respectively. While the VO allows, first of all, to obtain the trajectory in a progressive way, pose after pose.

As well, it enables optimization by using only the last trajectory positions (this is also called window BA) without using loop closure. In fact, the main objective of the VO is to ensure consistency of the local trajectory which gives a more accurate estimation of the local trajectory (e.g. using the BA).

Actually, visual SLAM is in general more accurate than VO. Thanks to to its ability to solve multiconstraints problem for trajectory estimation, although this does not necessarily mean it is more robust than VO. Commonly, choosing between VO and visual SLAM relies on the application's need and the possible trade-off between performance, consistency and implementation simplicity. Visual SLAM may sometimes seem more appropriate, especially, when overall consistency of the camera trajectory is required. However, VO gives real-time performance without the need to keep a complete record of the camera's track history, which may be a constraint of the targeted system.

Back-end Optimization Types

Visual navigation systems can also be divided into optimization-based or filter-based systems. In this section each of these two categories is presented. Although in this chapter the interest is in purely visual methods, so this categorization will be largely confined to the visual SLAM approach. In the next chapter, dealing with inertial visual navigation systems, this categorization will be more general and odometry will be even more concerned by this categorization.

Optimization-based methods are implemented by solving the graph from scratch each time it evolves, but sparsifying it by removing everything except a small subset of the past poses T i , as illustrated in figure 1.5. In some cases, the poses used are stored in a sliding window of the camera's most recent poses. Usually, it is a set of heuristically selected keyframes. While the other poses T i and all their measurements X i are rejected because they do not contribute to the estimates (gray elements on figure 1.5). Therefore, in visual ORB SLAM [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF], this approach is also known as keyframe-based SLAM. It aims to calculate the camera pose according to the pose of the points of the 3D map already reconstructed. The goal is to reconstruct new 3D points if necessary, and to refine jointly the reconstructed 3D points and the camera pose for some selected keyframes in the sequence.

Keyframe-based or optimization-based visual SLAM maps the features detected in the current frame

Actually, the use of learned methods is sometimes exaggerated, because often handcrafted techniques can solve a problem much more efficiently and with fewer lines of code than learned ones.

Handcrafted methods are based on a generic techniques and algorithms that work in the same way for any image, unlike learning-based ones. These latter are specific to the training dataset. In addition, the image resolution has an impact on the learned methods-based applications output. Indeed, high resolution images/videos must be used in order to obtain adequate performance, so this causes an increase in the amount of data to be processed, stored and transferred, which is already large.

Handcrafted Methods

The state-of-the-art proposes different handcrafted PoIs detection/description methods. Since 1976, many handcrafted methods of PoIs detection and description, as opposed to recent methods based on the deep learning, have been developed. Some of them are based on the local intensity change(s) [START_REF] Kitchen | Gray-level corner detection[END_REF], others rely on the use of a Hessian matrix [START_REF] Hans P Moravec | Towards automatic visual bbstacle avoidance[END_REF] [START_REF] Harris | A combined corner and edge detector[END_REF]. However, the detection algorithm choice depends mainly on its use case. In addition, the PoIs description, after they are detected, is a required step. It enables them to be used, as well as the salient visual information contained therein, in various PoI-based computer vision applications. Indeed, descriptor is a characteristic representation (color, intensity, scales, gradients, neighbourhood information, etc.) of the PoIs obtained by the detector. It is computed on an image area defined by the detector, this area is called the support of the description.

A PoIs descriptor must be robust to different constraints (motions, luminosity, etc), distinctive, effective, compact, and representative. There is a variety of PoIs detectors and descriptors. In this section, some examples of a local detector-descriptor algorithms, commonly used in the recent years for object tracking and pose estimation, are presented.

Harris is one of the widely used detection algorithms. It was developed in 1988 [START_REF] Harris | A combined corner and edge detector[END_REF], and was then improved to Harris-Laplace in 2001 [START_REF] Mikolajczyk | Indexing based on scale invariant interest points[END_REF], and to Harris-affine in 2004 [START_REF] Mikolajczyk | Scale & affine invariant interest point detectors[END_REF]. Features from Accelerated Segment Test (FAST) is a competing algorithm of Harris. It is efficient for the PoIs detection but it has a high computational complexity. Also, it can be used for PoIs description. Other algorithms exist, which combine detection and description, such as the Scale Invariant Feature Transform (SIFT) [START_REF] David G Lowe | Object recognition from local scale-invariant features[END_REF], the Speeded-up Robust Feature (SURF) partly inspired by the SIFT [START_REF] Bay | Speeded-up robust features (SURF)[END_REF] or the Oriented FAST and Rotated BRIEF (ORB) [START_REF] Rublee | ORB: An efficient alternative to SIFT or SURF[END_REF]. This latter is based on FAST PoIs detection and Binary Robust Independent Elementary Features (BRIEF) description [START_REF] Rublee | ORB: An efficient alternative to SIFT or SURF[END_REF]. ORB provides fast detection and good algorithmic performance [START_REF] Fang | A motion tracking method by combining the IMU and camera in mobile devices[END_REF]. These algorithms are presented with more details in annexe A.

In addition, a large variety of PoIs description methods have been also proposed such as DAISY descriptor. It is inspired by SIFT descriptor and aims to accelerate the computation time and to better deal with several types of invariance [START_REF] Tola | A fast local descriptor for dense matching[END_REF]. Also, for the same objectives as DAISY, there is Cheng descriptor which is presented in [START_REF] Cheng | A deformable local image descriptor[END_REF] and is based on a multi-size support regions centered on the PoIs. As well as the affine photometric model used in the inertial-aided KLT algorithm in [START_REF] Hwangbo | Inertial-aided KLT feature tracking for a moving camera[END_REF].

Actually, to detect, describe and match PoIs, there are different algorithms as listed above. The choice of the good algorithm depends on the application requirements, as well as the different algorithms characteristics, such as their change invariances and computational complexity. For example, in embedded systems, the most important constraints for choosing a PoIs detection/description algorithm are the computational complexity, computation time and memory. To evaluate the different detection/description algorithms, different comparative surveys were conducted [START_REF] Lee | Recent advances in feature detectors and descriptors[END_REF] [112] [START_REF] Işık | A comparative evaluation of well-known feature detectors and descriptors[END_REF].

For this purpose, they used specific comparison metrics such as the number of detected PoIs, detection/description computation time, detector repeatability, point matching rate, etc. The choice of the relevant algorithm in terms of invariance or robustness is related to the application's requirements. For instance, in the case of embedded mobile robotics used for SLAM systems, robustness to rapid translation and rotation, and scale changes is assumed to be important. Indeed, these systems are vulnerable to blur caused by vibrations and rapid motions caused by fast rotation and scale changes. However, it is important to keep in mind that they have limited processing resources, which also have to be considered when choosing appropriate algorithm. These issues are addressed in different papers in terms of runtime, as well as in terms of descriptor size which allows predicting the memory required for each algorithm [START_REF] Tareen | A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK[END_REF][105] [START_REF] Balntas | HPatches: A benchmark and evaluation of handcrafted and learned local descriptors[END_REF]. Actually, both SIFT and ORB algorithms remain among the potential candidates for this type of systems, thanks to their large number of detected PoIs and their large invariances to these movements. Although, these two algorithms are different in terms of their computation properties. More precisely, ORB algorithm is less complex and faster than SIFT [START_REF] Balntas | HPatches: A benchmark and evaluation of handcrafted and learned local descriptors[END_REF]. Thus, the most relevant detection and description algorithm for embedded mobile robotics applications is ORB algorithm. It has good robustness and good performance against such applications requirements, as well as a less complex and faster processing. Indeed, it is possible to have different combinations between detection/description algorithms to make the system more robust and improve its accuracy [START_REF] Ertu Grul Bayraktar | Analysis of feature detector and descriptor combinations with a localization experiment for various performance metrics[END_REF] [START_REF] Heinly | Comparative evaluation of binary features[END_REF]. For example, coupling the Harris detector with the ORB descriptor (rBRIEF) in the case of pure rotations, as well as the Harris detector with the BRIEF descriptor to have an invariance against viewpoint changes [START_REF] Heinly | Comparative evaluation of binary features[END_REF]. Figure 1.10 generally summarizes the trade-off between computational complexity, expressed in terms of computing time, and the storage required by each algorithm (descriptor) [85] [121]. This storage depends on the number of PoIs covered by each algorithm and the size of the descriptor. and/or global geometric constraints. In fact, it is possible to have mismatches which are an incorrect matches due to background clutter or because the PoI is not detected in the other image. This will ultimately result in an inaccurate construction of reality. So, to avoid this, there are robust methods that are used to eliminate the outliers, for example the methods based on filtering of geometric constraints (for example RANdom SAmple Consensus (RANSAC) [START_REF] Martin | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF] method presented in appendix C), using the fundamental matrix and its alternatives.

Similar to PoIs detection/description, it is possible to perform matching using learning processes. Indeed, matching using training can be roughly classified into image-based training and point-based training. The first one is based directly on tasks without attempting to detect a salient image structure (e.g. PoIs, etc.) first. Generally, this learning focuses on image registration, stereoscopic matching, and camera localization. Whereas for point-based learning, it is done by applying operations on the sets of detected PoIs. Generally, it is used for classification, segmentation [START_REF] Charles | PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation[END_REF][174], and registration [START_REF] Simonovsky | A Deep Metric for Multimodal Registration[END_REF] [START_REF] Liao | An Artificial Agent for Robust Image Registration[END_REF].

Visual Tracking

Pose estimation is an important task in different navigation methods. In particular, in FT-based methods, this task is performed immediately after feature detection/description and feature matching. In order to address this topic, firstly, Structure from Movement (SfM) is presented in section 1.3.1 since it is a visual technique for retrieving the camera's 3D poses from successive motions and a known set of 2D images. It is a general imaging technique, on which other more recent techniques have been based or inspired (VO and visual SLAM). Then, basic methods used to compute the pose and the trajectory are described. In particular, feature correspondences-based methods (2D-to-2D, 3D-to-3D and 3D-to-2D feature correspondences) are discussed in section 1.3.2.

Structure from Motion

SfM can be calculated in different ways. This depends on different factors, such as the number and type of cameras used, and whether the images are captured under controlled conditions. In the case of a single calibrated camera, the 3D structure and camera motion can only be recovered up to scale. Up to scale means that it is possible to resize the structure and amplitude of the camera motion while maintaining the observations. To compute the real scale of the structure and motion in global units, it is necessary to have additional information such as: the size of an object in the scene, and the information from another sensor, for example, an odometer or an inertial sensor. So, the process generally consists in finding the correspondences (points) between the scene and using multiview geometry to recover the scene and the 3D pose. Additional BA steps [START_REF] Triggs | Bundle Adjustment -A Modern Synthesis[END_REF] are used to refine the SfM by minimizing the reprojection error.

In the simple case where two fixed cameras or a moving camera (the first and second images are considered as camera 1 and camera 2, respectively), the SfM algorithm assumes that camera 1 is at the origin and its optical axis is located along the z axis (figure 1.13).

Firstly, the SfM requires points matching between images either by using feature matching or by using point tracking from image 1 to image 2. For example, the KLT is used effectively for points tracking in the case of small camera movements.

Then, to compute the current pose relative to the previous one, it is necessary to compute the fundamental matrix, using the corresponding points found in the previous step of the computation. This matrix is used to describe the epipolar geometry of the two poses (current and previous) and it

A u v = -b (1.17)
To obtain the components u and v of the pixel displacement velocity, the least squares solution of the over-determined linear equation is calculated as follows:

u v * = -(A T A) -1 A T b (1.18)
So the pixel motion vector can be identified, as well as the position of the tracking point in the image can be computed. Generally, optical flow tracking is fast, especially, KLT algorithm due to the limited pixel number used as well as the image points number.

Vision-based Localization Systems Overview

Visual SLAM and VO are expected to run in real-time on an ordered sequence of images acquired from a fixed camera configuration (i.e. one or two particular cameras), whereas SfM approaches frequently use an unordered set of images often calculated in the cloud with little or no time constraints and can use different cameras.

Since in this thesis we are interested in monocular systems, we mention below the most known monocular competitive localization methods [START_REF] Taketomi | Visual SLAM algorithms: a survey from 2010 to 2016[END_REF] [START_REF] Kasar | Benchmarking and Comparing Popular Visual SLAM Algorithms[END_REF], including tracking, in the literature, i.e. EKF MonoSLAM [START_REF] Andrew J Davison | MonoSLAM: real-time single camera SLAM[END_REF], PTAM [START_REF] Klein | Parallel Tracking and Mapping for Small AR Workspaces[END_REF] and ORB-SLAM [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF].

EKF MonoSLAM

In the early 2000s, A. Davison conducted historical work [START_REF] Andrew J Davison | MonoSLAM: real-time single camera SLAM[END_REF] [START_REF] Davison | Real-Time Simultaneous Localisation and Mapping with a Single Camera[END_REF] aimed at introducing vision into the SLAM named MonoSLAM. It uses image features to represent landmarks on the map. It iteratively updates the probability density of the feature depth by frame-by-frame matching to recover their 3D positions, thus it initializes a sparse feature-based map, and updates the complete state vector (robot pose plus 3D features pose) in an EKF.

Basically, as shown on figure 1.19 EKF monoslam consists of four main steps:

• Initialization: first of all, the initialization phase is carried out. During this step the camera is calibrated to ensure and improve the computation accuracy, as well as initial landmarks are computed to allow to begin the next computation process.

• Prediction: next is the prediction step. In this step the system state vector is described by: position, orientation (using a quaternion), linear velocities, and angular velocities. These parameters are used to calculate the kinematic model of the camera movement and consequently to predict its trajectory.

• Measurement & Tracking: once the model has been computed, the PoIs management is performed. This is based on the detection and processing of new landmarks. These are then tracked.

• Updates & Correction: in fact the difference between camera observation and prediction is called Innovation. The latter is the basis for this last step. The correction process updates the system status vector with the final position, as well as increases its accuracy by selecting the appropriate matched PoIs to retain according to their quality.

• Tracking thread: where mainly the ORB PoIs extraction and tracking from one image to another is handled. This is accomplished by estimating the camera pose either by re-localization in the eventuality that the system is lost, or by using the previous frame in the case of regular system functioning.

• Local Mapping thread: in the first thread, a decision is made on the need to insert KeyFrame (KF). According to this, for each KF, the local mapping inserts the selected KF and starts the optimization of the system map (map points, KFs, visibility graphs, etc) and the implementation of the local BA.

• Loop Closing thread: this thread takes the last KF processed by the local mapping, and tries to detect and close loops. This is achieved by performing the following main functions: loop detection, similarity transformation analysis, loop fusion and essential graph optimization.

Its main properties are summed up in: it operates in real-time, in small and large indoor and outdoor environments. The system generates a compact and trackable long-term map and it is robust to severe motion clutter. Also, it allows wide baseline loop closing and relocalization based on indexing approaches, and it includes full automatic initialization. It has been recently optimized to deal with stereoscopic and RGB-D contents [START_REF] Mur | ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras[END_REF]. However, despite the algorithmic quality of ORB-SLAM and FIGURE 1.21: ORB-SLAM overview [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF] its robustness in different navigation environment scenarios, it still suffers from time consuming and computational complexity. Moreover, as with any other vision-based navigation and pose estimation method, its quality decreases dramatically in the case of difficult vision where the images to be processed are not useful. New improvements are then proposed to overcome these problems, including the various data fusion, especially visual and inertial data fusion [START_REF] Mur | ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras[END_REF][155], presented in section 2.2.1 of chapter 2.

Conclusions

This chapter describes the different approaches and methods available for visual navigation. Furthermore, the majority of visual navigation systems requires two main computation steps, firstly image pre-processing, then pose estimation for tracking and localization. In this chapter, the different visual navigation classifications was presented with a focus on the most used ones which include odometry and SLAM approaches classification, and optimization and filter-based approaches classification. Subsequently, as PoIs are among the most generic and accurate image features, the focus was on introducing and analysing relevant image processing and PoIs extraction methods. Also, the distinction of the ORB detector-descriptor as the most suitable algorithm for resource-limited systems was presented, since it can detect a sufficient number of PoIs for navigation application with the required invariances and interesting performances. In addition, this study helps to identify the KLT algorithm among the accurate visual tracking algorithm in case of short movements between two successive images; as well as the suitable visual tracking algorithm for embedded systems with limited resources. In fact, this algorithm based on optical flow process is lightweight because it does not require a heavy processing as for PoIs description. Next, various pose estimation methods was presented, focusing on the most commonly used methods in visual navigation and tracking systems including VO and VSLAM, as well as those based on feature correspondences. Finally, a brief overview of the most widespread literature methods was presented. It illustrates the different previous vision-based algorithms and methods, and it discusses the different advantages and limitations of visual systems based on VO and VSLAM. Actually, these visual navigation, tracking, and localization systems provide accurate and reliable results. However, they are influenced by different environmental constraints, such as dark environments or motions that blur the image, which significantly reduces their robustness and causes the system to stall. In addition, the vision-based tracking and localization techniques suffer from the accumulation of errors due to the use of the dead-reckoning concept. Since VO is mainly based on this latter, it has a higher drift rate than VSLAM which combines dead-reckoning and BA to improve the localization accuracy but increases the computing charge and the optimization step complexity. Moreover, it should not be forgotten that a monocular visual system suffers from scale drift over time due to the use of a single camera. More precisely, VO systems are the more influenced ones by this issue than VSLAM systems.

It is difficult to propagate the scale factor throughout the process because, with only one camera, this scale is not observable during the pose estimation process. Therefore, the scale factor becomes the direct reason for errors and drifts accumulated over time, especially when many PoIs are suddenly lost between two successive images. Thus, it is necessary either to use more than one camera (e.g. stereovision) which increases the computation complexity, or to combine the vision with other complementary data obtained from other sensors to improve the performance of the pose estimation and tracking.

Next chapter focuses on the joint use of inertial and visual data using the IMU/Camera coupling.

Especially, it introduces and discusses the visual inertial navigation systems (VINS), including pose estimation for tracking and localization in odometry and/or SLAM methods. Different VINS categories will be defined, as well as the most relevant works of the state-of-the-art will be presented and discussed.

Chapter 2

Visual Inertial Navigation Systems [START_REF] Zhang | PIRVS: An Advanced Visual-Inertial SLAM System with Flexible Sensor Fusion and Hardware Co-Design[END_REF] follows the same principle as VIO. However in these cases the models are non-linear in the majority of cases. These filter-based processing steps are defined as follows:

-Dynamic model(in case of non-linear system):

x I = f (x t-1 , u t ) + w t (2.1)
where: u t : the control vector w t : the process noise, w t ∼ N(0, Q t ) with Q t is the variance.

-IMU status is expressed within a 16-size vector:

x I = I W q T W p T I W v T I b T g b T a T (2.2)
where:

I W q T : the quaternion rotated from the world frame to the IMU frame W p T I : the position of the world coordinate system W v T I : the speed of the world coordinate system b T

g & b T a : the gyroscope bias and the accelerometer bias, respectively -System prediction x t|t-1 and measurement/observation z t :

z t = h(x t ) + n t (2.
3)

x t|t-1 = f (x t-1 , u t ) (2.4)
And the propagated covariance matrix is expressed as:

P t|t-1 = F t P t-1 F t + Q t (2.5)
where:

F t = ∂ f ∂x | x t ,u t (2.6)
-System update equations:

y t = z t h(x t|t-1 ) (2.7) S t = H t P t|t-1 H T t + R t (2.8) H t = ∂h ∂x | x t (2.9)
Even if filter-based methods allow to perform system state estimation with high accuracy, they suffer from potential introduction of significant errors due to the linearization of non-linear measurements reducing system performance. On the other hand, optimization-based methods allow the system to perform the pose estimation task using both visual and inertial measurements in a joint optimization process. For this purpose, b a τ , b g τ : the accelerometer and gyroscope bias, respectively Thus, the R-VIO method updates the state only with landmarks that are moved out of the field of view of the current frame, or with visual measurements taken on frames that are removed from the state vector. This means that not all currently available measurements are used to update the state. In addition, the linearization errors make the filter inconsistent. To solve this problem, research has been ongoing for a long-time. In 2009, the idea of the first Jacobian estimates (FEJ) [START_REF] Huang | A First-Estimates Jacobian EKF for Improving SLAM Consistency[END_REF] was adopted to improve the consistency of the MSCKF [126] [125]. After a few years Guoquan P. Huang, Anastasios I. Mourikis, and Stergios I. Roumeliotis proposed the OC methodology [START_REF] Li | Improving the accuracy of EKF-based visual-inertial odometry[END_REF] which was intended for OC-VINS development [START_REF] Hesch | Towards consistent vision-aided inertial navigation[END_REF] [88] [START_REF] Kottas | On the Consistency of Vision-Aided Inertial Navigation[END_REF]. Up to now and even with the current work of R-VIO, this problem persists despite conserving the appropriate observability properties independently of the linearization points. Furthermore, since the R-VIO is an odometry algorithm, it does not allow loop closure to be performed either by mapping or place recognition. However, this is an important feature to ensure a limited error VINS performance for an accurate long-term pose estimation. As an alternative, VI SLAM system can produce more accurate and more precise results thanks to using mapping and loop closing.

Optimization-based vs Filter-based Approaches

Filter-based approaches provide pose estimation, using a filter that relies on inertial measurements and image features. To improve the computational efficiency in this kind of systems, interference processes are delayed until the last stage of the system, and the filter's past states are marginalized.

The latter has a certain number of drawbacks, mainly it leads to problems of consistency. The EKF algorithm is one of the most common filters used for data fusion. It consists mainly of two major steps: prediction and update. EKF performs a first-order linearization around the current mean and covariance at each step. Therefore, it is reliable only for systems that have a Gaussian model with limited non-linearity. Also, its computational complexity increases quadratically with the number of tracked characteristics in the state vector. Hence, its accuracy is affected or limited. In order to overcome these problems and to obtain better accuracy, the UKF algorithm can be used instead of EKF, specifically for highly non-linear systems. It does not compute Jacobian matrices. However, despite the above advantage, UKF is still costly for computing power, so it is difficult to implement it on embedded systems, such as UAVs. Another algorithm is used for filter-based visual-inertial systems, it is the MSCKF. The latter proceeds by constraining the measurements by a stochastically cloned pose in a sliding window. This algorithm suffers from a noise gain in the unobservable subspace direction, that impacts the consistency of the system state estimation.

Furthermore, optimization-based approaches use a non-linear optimization to reduce directly any errors between the motion obtained from the integrated inertial measurements, and the camera motion obtained using standard reprojection error reduction. These approaches can outperform filterbased ones in terms of accuracy, due to their ability to linearize current and past states. However, optimization-based approaches can suffer from different problems, depending on the optimization method used. These problems are mainly related to computational complexity, which complicates the implementation of these approaches on systems with limited resources. Indeed, these computational costs can be reduced in different ways. Among the most well known and used techniques are the possibility of processing only keyframes, the use of sliding windows or the use of incremental smoothing (updating only small variables subsets).

Most of relevant filter-or optimization-based literature methods are evaluated and compared using EuroC dataset [START_REF] Burri | The EuRoC Micro Aerial Vehicle Datasets[END_REF]; we will come back in the analyze of these evaluations in section in section 4. To conclude, optimization-based methods outperform filter-based methods mainly with excellent localization accuracy, while the main advantages of filter-based methods are in terms of computing resources. Choosing the right method type to use for a given system/application can be difficult and involves making trade-offs between accuracy and computational resources.

Challenges Associated to VINS and Development Trends

Recently, VINS have made significant improvements in their quality and performance. However, this is still not enough to solve all the challenges that this field may face. Among the main of these challenges are: firstly, tracking problems related to difficult navigation environments, such as poor lighting, vibrations and difficult motions. Secondly, localization and semantic mapping issues linked to the outdoor navigation environment constraints, etc. The progress achieved in deep learning is assumed to handle these various challenges, and to enhance the navigation efficiency of the majority of current VINS, especially those based on traditional methods that use geometric features like PoIs, lines, etc. Moreover, it is not easy to detect, represent and track different objects coexisting in the same space, in real time, using a mobile navigation system. Another VINS challenge is due to the used sensors type, this is part of their improvement requirements. While cameras and IMUs represent an interesting combination for VINS, other sensors exist and provide more effective aid, according to system and environment requirements. For example, lightweight and low-cost LiDARs may work better in environments with poor lighting conditions [START_REF] Geneva | LIPS: LiDAR-Inertial 3D Plane SLAM[END_REF][87], event cameras [START_REF] Lichtsteiner | A 128× 128 120 dB 15 µs Latency Asynchronous Temporal Contrast Vision Sensor[END_REF][129] could provide more effective support for dynamic motion, etc. [238][151].

Consequently, today's research is tending to be focused specifically on addressing these challenges and developing more efficient solutions in terms of computational complexity and resources. However, it should not be forgotten that using deep learning continues to have the drawback of its huge computing cost and overhead. Thus, the integration and use of different sensors are one of the current research topics concerning VINS. Lightness and miniaturization have a great importance for new VINS. They allow them to operate on small devices, which have limited computing resources, such as embedded systems, telephones, drones or smart glasses, etc. Indeed, significant and motivating results have been obtained with Microsoft Hololens [START_REF] Furlan | The future of augmented reality: Hololens -Microsoft's AR headset shines despite rough edges [Resources T oolsandToys[END_REF], Intel RealSense and Google Tango [START_REF] Huang | 3Dlite: Towards Commodity 3D Scanning for Content Creation[END_REF].

Consequently, future applications and systems will tend to be as embedded as possible. Moreover, the use of several sensors is much more beneficial than the use of a single one. One sensor cannot correctly detect environmental information, making state estimation subject to great uncertainty. A multi-sensor system improves the accuracy and completeness of a system since the sensors work together to correctly detect the necessary processing information.

Conclusions

In this chapter, methods of VINS, as well as their classification were presented. The focus was on the two main classifications: filter-and optimization-based methods and the loosely-and tightlycoupled methods. Following the optimization-and filter-based approaches, different pose estimation techniques were discussed and illustrated using the most known literature works. In addition, a comparison between the two categories is done in order to explain the advantages and inconveniences of each one.

While there has been significant progress in recent years concerning VINS, there are still many challenges to be addressed and which are currently open for discussion:

• Robust localization: even if current VINS are able to provide accurate motion tracking, they are not robust enough for long-term, large-scale and safety-critical deployments, such as autonomous driving, in part because of resource constraints. Therefore, even by effectively integrating loop closures or building and using new cards, it remains difficult to obtain persistent VINS in environments with difficult conditions such as poor lighting and unfavourable movements.

• Upgrades with various aid sensors: although optical cameras are considered the perfect aid source for the IMU in many applications, other aid sensors may be more appropriate for certain environments and motions. For example, acoustic sonar can be used instead in underwater environments; low-cost lightweight LiDARs can work better in environments, such as environments with poor light conditions; and event cameras for a more accurate capturing of dynamic motions. In this context, it is essential to study in detail the VINS extensions that can use different aid sources for handheld applications.

As inertial and visual sensors increasingly become ubiquitous, VINS have undergone significant research efforts and progress over the past decade, encouraging an increasing number of innovative applications in practice. Moreover, due to the specific sensor properties and application type, it is not easy to develop VINS algorithms from scratch without understanding the advantages and disadvantages of the existing literature approaches. Especially, each method has its own objective and does not necessarily proceed in the same way to carry out required processing at different stages of a VINS. After having presented and compared the main methods dedicated to VINS, the latter architecture and hardware integration is also an interesting topic. Therefore, in the next chapter, the implementation part and hardware integration of these VINS will be discussed with a main focus on embedded systems.

Chapter 3

Embedded Tracking Systems Therefore, it is critical that all components of the system perform as fast and accurately as possible in order to operate together.

As embedded systems are one of the major link of a CPS, this chapter focuses the constraints met by designers when developing a device such as a VINS. Firstly, we briefly define what is an embedded system and trade-offs that have to be made in terms of power efficiency. Then, we focus on the diversity of recent architectures and components used to built an efficient system. In particular, the section 3.1 illustrates this latter by providing metrics. Then, in section 3.2, different existing embedded computing architectures, especially those used for visual-inertial navigation and tracking computation, are presented. Subsequently, in section 3.3, the difficulties of designing an embedded tracking system are explained, including the co-design approach, and the cutting edge VINS and their implementation are described.

Definition and Features

An embedded system is a special purpose computer system, programmed and controlled using a Real-Time Operating System (RTOS). It must meet specific requirements relying on its dedicated tasks. In this section, we introduce the different embedded systems classification, then we explain their main design challenges and performance metrics.

Embedded System Classification

Embedded systems can be defined according to their features and functionality requirements. Obviously, performances and power consumption may differ depending on targeted application:

Autonomy The autonomy discussed here deals with processing and computations made by the system. In fact, autonomous embedded systems are devices that receive inputs (analog or digital), process incoming data and perform several dedicated tasks, and produce the final results. All of this is done autonomously without the need to give part or all of the processing to one or more external processors. The following are some examples of autonomous embedded systems: video game consoles, digital cameras, Digital Audio players, etc.

Real-time This refers to systems able to deal with bounded task execution time. As time is a critical constraint, output accuracy and precision are affected. There are two types of real-time embedded systems: the hard real-time systems, where there is no flexibility or tolerance in time constraints. i.e. the smallest error caused by the non-respect of the deadline or the inaccuracy of the result can cause catastrophic consequences. Soft real-time systems tolerates precise tolerance delays. The response must be given on time but with a tolerance within acceptable limits.

Network-Connected This embedded system category is connected to the Local Area Network (LAN), Wide Area Network (WAN), Internet, cellular communication network, etc., in order to access and use resources. For instance, 5G is a recent network connection used to ensure this communication between embedded systems. In fact, 5G or fifth-generation cellular communications technology is an ongoing topic that affects all industries, especially embedded systems and specifically the IoT. The latter covers the description, development, and use of connected devices.

7nm in 2018 and will reach 5nm in 2020 [106]. However, Complementary Metal Oxide Semiconductor (CMOS) scaling no longer provides efficiency gains proportional to the increase in transistor density.

Nowadays, industrial and academic works and studies are more and more focusing on the design of specialized and optimized accelerators and the use of computing resources heterogeneity. System architects have to find a trade-off between these three criteria: performance, power consumption and energy efficiency. Since those systems still have to perform general tasks [START_REF] Hadi Esmaeilzadeh | Neural Acceleration for General-Purpose Approximate Programs[END_REF], devices such as General Purpose Graphics Processing Unit (GPGPU) or even last-generation Field Programmable Gate Arrays (FPGA) are becoming more and more attractive to designers. Although they are greedy in terms of power consumption, they offer a combination of general purpose System-On-Chip (SoC)

and dedicated accelerators for image processing and highly parallel re-configurable processing (GPU res. programmable logic resources). Today SoC such as those smartphone built-in represents best trade-offs between versatility versus power consumption. All sensors are connected to the chip in a centralized manner.

Thus, the designer faces three major constraints, ensuring the architecture versatility between general purpose and specialized tasks, limited global power consumption and sustaining high computing capability on dedicated times-lot. Firstly, the general-purpose/specialized capability of an embedded system is one of the early issues raised at the beginning of its hardware integration and its implementation. Actually, sensors can be connected together with the same processing part support centrally, as well as they can be distributed to different parts of the system, according to the processing requirements. In fact, the SoC architecture, presented below in section 3.2.1, is the perfect candidate to deal with.

In terms of power consumption, generally a SoC consumes between a few milliWatts and a dozen Watts. For example, a cutting edge smartphone such as the Nokia 7 from HMD Global, the company claims a two-day autonomy (with an integrated battery of 3800mAh). This figures covers the execution of a range of various application involving signal processing, screen display, camera acquisition and associated processing, modem data exchange and so on, etc. Thus, the addition of all powers consumed by dedicated resources. In the case of VI SLAM/VIO, a research team recently design and implement a chip dedicated to VIO Navigation system [START_REF] Suleiman | Navion: A 2-mW Fully Integrated Real-Time Visual-Inertial Odometry Accelerator for Autonomous Navigation of Nano Drones[END_REF]. Indeed, among the main challenges of designing such chip is to perform these tasks with the lowest possible power consumption. The Nano Drones autonomous navigation proposed in [START_REF] Suleiman | Navion: A 2-mW Fully Integrated Real-Time Visual-Inertial Odometry Accelerator for Autonomous Navigation of Nano Drones[END_REF] consumes between 2mW and 24mW, depending on its operating conditions, making it the first odometry system based on visual-inertial fusion that requires such low power consumption. This SoC is called Navion and is presented later in this chapter. Furthermore, traditionally IC provider always gives typical performance metrics to illustrate computing capabilities of their chip. For generalist tasks, millions of Instructions Per Second (MIPS) These metrics are sometimes controversial, because of their dependency to the processing kernel and architecture implementation. They represent a kind of theoretical peak performance not achievable for real-world applications. Thus, some benchmark providers propose typical kernel implementation that can be implemented on candidate architecture and sort them by performance and efficiency in terms of energy (BDti [103], Spec2006 [42], etc). In fact, power and energy aspects consider Watt and Joule metrics, respectively. More realistic measures allow the efficiency to be computed using, for instance, operation per second in power consumption and/or energy delay product (e.g. TOPS/Watt), time processing × power consumed depending on executed kernels or applications, as well as a new mJoule/inference or Frame per Second ( f ps) metrics, which has been introduced to measure efficiency of CNN.

According to the previous discussion, the embedded systems constraints are mainly related to the system's specialization level, power consumption and performance. These constraints are interconnected. Particularly in tracking embedded systems, which is the main subject in this thesis, the challenge is to design an efficient system using a minimum power consumption, using different components (specialized/generalist architectures). Thereafter, the different computing architectures, classified into three types: SoCs, IP Modules, and ASICs, are presented.

Embedded Computing Architectures

The choice of a well-fitted architecture relies on the respect of application requirements. Designers translate these latter into throughput, latency metrics under power constraint. The targeted host architecture ensures the execution of the application on monolithic or heteregeneous ressources. As there is a wide range of embedded devices and components, this section focuses on the description of mainstream available architectures and the trends in IC chip for embedded VINS. First SoCs are presented and illustrated using the examples commonly used in embedded navigation systems, IP modules and ASIC architectures.

System on Chip (SoC)

This type of architecture is able to execute many applications. By design, a SoC is commonly a general purpose platform with small dedicated accelerators. Ressources are interconnected together via dedicated communication media (e.g. bus, crossbar, Network-On-Chip) and data are shared through a specific memory architecture. A software programmer is able to developp applications and/or kernels by using typical compiling toolchain and can also use accelerators with dedicated instructions. The purpose of the SoC is then to meet the performance, the power consumption, and the surface area constraints (in the case of circuit design) set at the time of specification. SoCs are used in different products and in different ways, such as MultiProcessor SoC (MPSoC). The latter platform is capable of managing high-level applications and taking advantage of tasks, processes and thread parallelism.

Over the past ten years, ARM has become the leader as embedded processors provider and has achieved a 95% mobile market share since Intel left in 2016. They offer the most energy-efficient solutions, from microcontroller units (for a wide range of IoT markets worldwide), portable devices and ADAS to high-end industrial computing [START_REF]Products processors[END_REF]. Supplied as a core license, ARM products are integrated into many leading SoCs (TI, Qualcomm, ATMEL, etc.). Today, "big.Little" ARM architecture is the reference for supporting different types of workloads while maintaining low energy consumption.

It consists of a combination of small low-energy flexible cores and large high-performance flexible cores (e.g. 4 cortex A55 + 4 cortex A75). It should also be noted that the RISC-V architecture has appeared in recent years. RISC-V is a free and open-source ISA from U.C. Berkeley [START_REF] Waterman | The RISC-V Instruction Set Manual, Volume I: Base User-Level ISA[END_REF]. More than 100 companies now support this initiative. The keywords of this new ISA are longevity, portability and reliability since a frozen ISA (less than 50 instructions) is provided and the extensions are well classified (multiply and divide, atomic, single precision floating point, double precision floating point, compressed instructions) for a total of 200 instructions.

Among the well-known architectures, the following ones can be listed: NVIDIA Tegra Xavier SoC, this architecture has the leading role in bringing GPGPU devices to the embedded world. Proposed in 2018, NVIDIA Tegra device called Xavier [START_REF] Demler | [END_REF], shown in figure 3.3, is composed of 9 billion transistors. This kind of SoC consists of 8-core CPU Carmel ARM64 10-wide superscalar with functional safety features plus parity and ECC suitable for autonomous driving. In addition, there exists a Packet Video Audio (PVA), this unit is used for processing computer vision tasks as filtering and detection algorithms. Two identical instances are implemented that can works independently or in lockstep mode. Each unit contains an ARM cortex R5 processor, a DMA unit, two memory unit plus two vector processing units (7 ways VLIW). Customizable logic is also available. Operations can be performed on 32 × 8, 16 × 16 or 8 × 32 bit data vector. Furthermore, this NVIDIA device includes a new 512-core Volta GPU. This architecture has been designed for machine learning market and is optimized for inference over training process. It includes 8 stream multiprocessors with individual 128KB L1 cache and a shared 512MB L2 cache. There is also 512 CUDA tensor cores. Besides, there exist a 8K HDR video processors, as well as a Drive Letter Access (DLA) which is able to achieve 5, 7DLTOPS(FP16) with a configuration/block, an input activation and filters weigh, a convolutional core, a post processing unit, and an interface with memories (SDRAM and internal RAM).

Optimized Implementation IC for Sensors Integration and Processing

An ASIC is an electronic device that integrates, on the same chip, all the active elements required for an electronic function or combination to be performed. Indeed, it is exclusively dedicated to a certain application and to a specific user. ASIC devices are optimized in terms of performance, power consumption and occupied area (energy efficiency). Several chips focusing on sensors integration for tracking, localization and navigation have been proposed and designed in academic or industrial works. Firstly, the Holographic Chip: Intel HPU [START_REF] Furlan | The future of augmented reality: Hololens -Microsoft's AR headset shines despite rough edges [Resources T oolsandToys[END_REF] In closing, the offer in terms of embedded computing architecture is large and diverse. This is a competitive industry that is evolving rapidly and continuously. However, these evolutions are data integration and delaying are also required to merge relevant information (e. g movement, pose) in the process pipeline.

Response time. This feature may vary according to the considered approach. In the case of VIO, the system ouputs a pose at the frequency of process tick even if inaccurate or false. In the case of VI SLAM, even if a fused pose is available, a failure can occur due to the closing loop process.

Therefore, both VIO and VI SLAM, in case of KeyFrame(KF)-based methods, compute the final pose just for KFs. Thus, the computation time is reduced, and the response time is also decreased.

Frequency constraint.

Compared to a complete VI SLAM system [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF][155], VIO does not have a loop closure, which occurs at a lower rate and can be off-loaded into the cloud. Thus, a VIO system can provide motion estimation at a higher frequency, which is an essential requirement for autonomous navigation robots, drones or high-speed vehicles, as well as for AR/VR devices.

Highly constrained power consumption budget and form factor. The real-time implementation of these methods requires relatively powerful computing architectures. However, high-embeddability systems such as nano/pico drones or UAVs have high embedded constraints. In fact, these miniature systems have limitations in terms of both, power budget and form factor. For example, in the quadrotor presented in [START_REF] Loianno | Estimation, Control, and Planning for Aggressive Flight With a Small Quadrotor With a Single Camera and IMU[END_REF] (figure 3.8) where the processor used is Qualcomm Snapdragon 801, the same one is used for smartphones, and it consumes about 3W of power [START_REF] Kottas | On the Consistency of Vision-Aided Inertial Navigation[END_REF].

Overview of Existing VIO/VI SLAM Systems

There are various works that propose different solutions for embedded localization and navigation systems, including tracking, based on IMU/camera coupling. Table 3.1 presents the relevant literature embedded system works. In the following, the multi-sensor embedded systems design constraints and co-design methodology are discussed. Then, a brief overview of the remarkable state-ofthe-art contributions is presented in order to illustrate the co-design concept and its various stages. based on Nvidia SoC, Intrinsyc based on Qualcomm SoC). However, their use as platforms that support different sensors is difficult and requires time and expertise in this field. However, nowadays different specialized development solutions exist for SLAM, VI SLAM and VIO methods, and which simplify the algorithmic concept evaluation and validation tasks. For example, Google Tango [START_REF] Marder-Eppstein | Project Tango[END_REF] which is a monocular system, Microsoft HoloLens [START_REF] Furlan | The future of augmented reality: Hololens -Microsoft's AR headset shines despite rough edges [Resources T oolsandToys[END_REF] which has HPU access limitations and is not easy to use as a computing platform, and it is a Windows platform only, as well as Apple ARKit [5] which is limited to Apple hardware and the iOS platform only. Recently, Google proposed Google ARCore [START_REF] Google | ARCore Fundamental Concepts[END_REF] characterized by its potential to enable SLAM on hundreds of millions of Android devices, but which may have a limitation regarding the form factor of the device.

Examples of systems based on the Co-Design approach

Following the previous discussion, we will present herein some of the relevant works that illustrate the embedded VINS implementation and the Co-Design application regarding to the two approaches: VIO and VI SLAM.

VIO Embedded Implementation exp. 1 : Zhengdong's work [START_REF] Zhang | Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach[END_REF] describes resources by two power consumption [START_REF] Zhang | Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach[END_REF] and form factor criterion. This later represents the weight and size available for the computation unit. The form factor depends on energy consumption, for instance, in the case of high energy consumption system, a large battery is required. Or, when the system's energy consumption does not exceed 1W, then it is not mandatory to have a fan. So, it is useful to minimize the size and therefore to optimize the form factor.

For the second step, they specify a performance feature (table 3.2). Firstly, they rely on the accuracy which describes the VIO drift and error estimation. Secondly, they proceed via the rate which measures the potential rate for processing sensor data and computing a state estimation. In [START_REF] Zhang | Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach[END_REF] this rate is split in two parts: the front-end throughput which must be high enough to adapt to the camera's frame rate. And the back-end throughput which should be higher than the keyframe (KF) rate, and high enough to ensure that the landmarks are tracked on consecutive KF. In addition, a specific terminology was used in this step to describe the defined design space D = H x A x I x P, it is presented as follow: D for design space, H for hardware, A for algorithm, I for implementation choices, and P for parameter choices. In [START_REF] Zhang | Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach[END_REF], the hardware choices are limited to embedded CPU and FPGA, because of the very limited power budget of the desired application, which is a nano aerial vehicle.

Then the third step, which is the exploration of the design space, the authors of this paper propose a new strategy called Iterative Splitting Co-design (ISC). This technique allows finding the right trade-off between resources and desired performance. It consists in splitting this research into two steps. The first attempts focus on minimizing resources by conserving the desired estimation error, that is using a selection of well-adapted algorithms and parameter choices. Then, the second ones FPGA, and proposes a new ASIC solution (considered the first) fully integrated, called Navion. This chipboard is manufactured in 65nm CMOS technology. The configuration of Navion relies on several programmable parameters, such as the KF rate, horizon size and number of feature tracks, which can significantly influence throughput, accuracy and energy efficiency. In fact, Navion is characterized by its capacity to give interesting results using visual-inertial data fusion. This is thanks to its suitability for different types of environment. In addition , this VIO system provides efficient and real-time processing. For example, in the case of EuRoC dataset where the frame rate is 20 f ps and the system power consumption is in an average of 2mW at 1V. While for the most problematic navigation cases, where stereo-vision images (size of 752x480) are acquired at a frame-rate up to 171 f p and the IMU measurements are generated at a frequency up to 52kHz, the power consumption is estimated in an average of 24mW at 1V. These results makes it suitable for critical applications such as autonomous navigation, mapping and portable AR/VR.

VI SLAM Embedded Implementation exp. 1 :

In addition, as explained before (chapters 1 and 2), SLAM remains a well-known navigation method for ego-motion tracking, that add to odometry the loop closing step. Until today there is still no enough works on embedded systems that integrates a complete VI SLAM. In the following, EMoVI SLAM [START_REF] Ullah | EMoVI-SLAM: Embedded Monocular Visual Inertial SLAM with Scale Update for Large Scale Mapping and Localization[END_REF] proposes a solution based on ORB-SLAM that considers a loosely coupling between a camera and an IMU. The pose generated by visual ORB SLAM and the pose computed from the IMU inertial measurements are combined using UKF filter to estimate the final pose. Also, in this work the authors proposed a method of computing scale and updating it, which is similar to the method used in VI ORB SLAM (figure 3.13). Finally, an embedded portability of EMoVI-SLAM was proposed (figure 3.14). results are obtained for VINS-Mono [START_REF] Qin | VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator[END_REF], with a loop closure. It gives an average of 0.12m in absolute translation error when considering Up Board (64 bit Intel Atom CPU). This error reaches 0.16m for VINS-Mono on ODROID XU4 (Samsung Exynos 5422 system in an ARM big.Little architecture 4 A7 at 1.5GHz and 4 A15 at 2.0GHz).

To conclude, the literature contains different implementations of VIO or VI SLAM systems which confirms the interest of using a heterogeneous embedded computing architecture. This enables specific functions to be accelerated using dedicated architectures and computing accelerators. Consequently, enabling the implementation by targeting selectively kernels/tasks requires a precise and detailed knowledge of the application and is always a tedious work. Optimizing synchronization of sensors, data exchange, refine the process while fitting the hardware resources utilization comes at the price of performance degradation. These latter must remain acceptable in the targeted application domain.

Conclusion

This chapter addressed the navigation topic for embedded systems, including tracking and VIO/VI SLAM. Firstly, definition and features of embedded systems were discussed. Then, architecture and functionality of components which composes integrated circuits have been described. In fact, the focus was particularly on devices used for processing VINS. Depending on the system aims and use, the computing architecture choice is done to respect its specific constraints and design challenges, and to ensure its outputs accuracy and robustness. Furthermore, increasing the number of sensors leads to an increase in the processing complexity required to perform the data integration and fusion steps. Embedded energy efficient architecture are optimized and specialized but can lose accuracy whereas mainstream architecture are able to run various tasks on heterogeneous resources with accuracy at the cost of power.

To illustrate these issues, first, the co-design was explained, then relevant literature works was described and discussed. Actually, these works are mainly based on inertial-visual fusion, thus they are based on IMU/camera coupling. Their application's fields and use-cases are different, but all of them perform the visual-inertial pose estimation step for localization, tracking and navigation, using either VIO or VI SLAM methods. Figure 3.17 presents a trade-off between the main VI navigation embedded systems constraints, which are power consumption, integration density and processing accuracy and complexity, for different embedded systems implementations levels as smartphone, headset mounted, drone devices and autonomous vehicle. In fact, data fusion must produce an accurate and reliable navigation results (position, orientation, etc.), even the fact that the sensors are

EuRoC MAV Dataset Analysis

Considering the datasets presented in the former section and according to this thesis proposal's purposes, the focus will be, through this work, on the EuRoC MAV dataset [START_REF] Burri | The EuRoC Micro Aerial Vehicle Datasets[END_REF], because it provides various sequences of inertial and visual data, captured by drone, as well as the ground truth. Thus, this allows to evaluate the proposed system and to compare it with different literature works, since EuRoC is the database widely used by the computer vision community.

As explained above, EuRoC MAV dataset provides two types of dataset sequences:

• The first one (MHxx) is specifically meant to evaluate VI SLAM algorithms in a real industrial context. It was acquired in a low textured environment, characterized by reflective surfaces and many black areas. MHxx data is composed by a 3D position ground truth using the Leica multistation in a machine hall at ETH Zurich (figure 4.5). • The second one (Vxxx) was captured in a room where different objects and obstacles had been placed to increase the texture and make the environment more challenging. It contains a 6D

pose ground truth acquired using the Vicon motion capture system, and additional accurate 3D point cloud of the environment (figure 4.6), taken by the Leica 3D laser scanner. All datasets were recorded using an AscTec Firefly MAV equipped with visual inertial sensor as shown in figure 4.7. This MAV consists of a stereo camera Wide Video Graphics Array (WVGA) with global shutter, at 20 f ps, IMU sensor MEMS with sampling rate 200Hz.

In order to identify the most constraining motions and scene type, each sequence is analyzed and characterized according to the two factors that influence the dataset levels. In the following part, the synthesis of each factor for the eleven dataset sequences is presented. The scene can be described The Asctec Firefly hex-rotor helicopter for data acquisition [START_REF] Burri | The EuRoC Micro Aerial Vehicle Datasets[END_REF] by its luminosity, brightness, texture, exposure, etc., whereas, motion is characterized by its type (rotation or/and translation), its velocity (linear or/and angular) and intensity. easy levels are mainly characterized by good luminosity and texture conditions, and without any exposition problem. Medium level is relatively more complicated than easy level: that's is because of exposition and lightning problems. Finally, difficult dataset level encompasses the majority of the scene problems. In particular, this level suffers from a lot of dark spots as well as a luminosity problem arising from lightning conditions and under/over-exposed frames.

In addition, table 4.4 shows the motion characteristics of various levels of different EuRoC dataset sequences. In this analysis, the system motion is divided into six motion type: roll, pitch, yaw, strafing, surging and elevation. A motion is considered difficult, when it is composed of several types of movements, when it is fast, and especially when it is a combination of these two criteria (fast and composite motion). The latter cause a high speed and abrupt motion as well as vibrations that lead to image blur.

To resume, following the different dataset sequences analyses presented above, we propose to simplify the EuRoC dataset sequences categorization into two major difficulty levels: easy and difficult navigation environment, as there is no great difference between the medium and difficult levels.

In fact, for both levels the problems are mainly related to the scene, such as lighting, and to the motion type, such as vibrations. Table 4.5 shows the compendium of all the the main characteristics of these categories.

Context Adaptive Visual-Inertial SLAM Workflow

The proposal is developed within the framework of the SLAM method, in particular ORB SLAM. It focuses on tracking thread, where we provide two different approaches, independently implemented each one in a thread and both based on ORB PoIs. However, the Local Mapping and Loop Closing threads are the same as in ORB SLAM. Actually, as depicted in figure 4.1, our proposed system first analyzes the environment in order to choose the appropriate tracking method. Next, according to the chosen approach, an initialization step is carried out before starting the chosen tracking approach. Afterwards, the system interfaces with other threads following the same process as in ORB SLAM. The transition from tracking to Local Mapping and Loop Closing is performed through keyframe. Therefore, whenever a keyframe is selected, the following threads are executed.

In this section, the overall solution workflow, the transitions and the connections between its different components are discussed. To start, the initialization and tracking steps are presented. Then, local mapping and loop closure processes are described.

System Initialization

The system starts in visual-inertial mode, in order to ensure tracking at system launch, simultaneously with the first analysis of the motion and the surrounding visual scene. Afterwards, depending on the control module's decision, one of the two proposed tracking approaches is executed: either remain using visual-inertial tracking, or move into visual tracking. In both cases it is necessary to proceed through the initialization step. This is a required process for SLAM: it consists in preparing a first set of map points for non-linear optimization and loop closure. Firstly, the initialization process starts by extracting the ORB PoIs in the current image k and matching them with the reference image r. Depending on the number of matches, either the next step is executed or the current frame becomes the reference frame and the same process is repeated. Then, the second phase is to check order to provide real-time performance, based on the visual EKF SLAM [START_REF] Russo | A ROS Implementation of the Mono-Slam Algorithm[END_REF] and 1-point RANSAC [START_REF] Civera | 1-point RANSAC for EKF-based Structure from Motion[END_REF] methods, it is a good idea to consider only the features visible in the current image. In fact, at the beginning of the EKF VIO process, the number of PoIs is limited to an average of 10. Subsequently, during the tracking, these points are sorted from one image to another in order to keep only the points that are still in the camera's FoV. When a PoI is not reliable enough, according to the score threshold computed during the sorting process, or when it is deleted, the system detects and reintegrates up to 5 new PoIs. Hence, based on the experiments (chapter 5), the maximum number of PoIs in a X k state vector is up to 25 PoIs and the average number is 9 PoIs.

Keyframe Selection Decision

This phase is the last tracking task, it comes after the map point creation and the map update. This phase consists in testing each image that has successfully completed the tracking process against three main criteria, inspired from the research of [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF], [START_REF] Mur | Visual-inertial monocular SLAM with map reuse[END_REF] and [START_REF] Quan | Accurate Monocular Visual-Inertial SLAM Using a Map-Assisted EKF Approach[END_REF]:

1. the time gap between two keyframes must be greater than a certain threshold. Indeed, an IMU provides precise and valuable measurements only when it comes to the short term, otherwise the measurements are not accurate enough. This criteria provides the accuracy and reliability of the system.

2. non-linear back-end optimization processing must be fully done. This criteria gives the possibility to have as many keyframes as possible, consequently it improves the motion tracking accuracy.

3. the rotation angle between the last keyframe and the current frame is also a critera for selecting keyframes, i.e. if the rotation angle is above a certain threshold so that image is considered as a keyframe. This criterion has the advantage of insuring the reconstruction of a globally coherent map.

By satisfying one of these criteria, the current image is selected as the keyframe and can then be used for local mapping and loop closure. The purpose of this step is not to use all the images in the SLAM, but to be limited to the keyframes.

Local Mapping & Loop Closure

Once a keyframe is selected and inserted, the system searches for new matches with the local map, then updates the covisibility graph. Subsequently, depending on the tracking information, the map points that are improperly triangulated are eliminated. Moreover, the current keyframe's ORB PoIs that are not matched, are made unable (prevented from finding) to find new matches with the rest of the connected keyframes in the covisibility graph. This is done in order to build new map points.

Subsequently, non-linear optimization based on the local BA is performed to optimize the local map.

Once completed, redundant keyframes are eliminated in order to make the factor graph more concise.

Loop closure performs place recognition via the DBoW2 [START_REF] Galvez-Lopez | Bags of Binary Words for Fast Place Recognition in Image Sequences[END_REF] functions using it in the same way as [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF] and [START_REF] Mur | Visual-inertial monocular SLAM with map reuse[END_REF]. So when a new loop is detected, a Sim(3) optimization and a full BA are performed to eliminate the accumulated drift. In the thesis's proposal, the interest is not to improve these two processes, but to ensure the consistency of the overall system workflow and its functioning.

for the appropriate motions and external environment. Therefore, in order to produce an optimal correct pose, two control modes are proposed: global and local control, using various control metrics and combinations.

Global Control Metrics

The global control, represented by green box on figure 4.9, enables the system status to be checked before choosing the tracking and pose computation methods to be performed, as well as the final pose accuracy and the overall trajectory consistency to be verified. The main metrics employed for this execution control module are:

-Moving Control Metric The controller analyzes the system moving state (mobile or immobile) in order to avoid the pose re-computation. Or, the IMU velocity information (the linear velocity v and the angular velocity w) are used, and system is considered mobile as soon as its velocity is non-zero (v = 0 and/or w = 0). In fact, recalculating the pose when the system is not moving (immobile system) causes a loss of processing resources, and can also leads to errors, due primarily to accumulated noises, in the pose computation for the next step (when the system starts moving again). So, when system is immobile, neither of the two tracking methods is executed and the state of the system remains the same as after the last pose computation.

-Image Quality Metric The image quality is evaluated before starting the tracking process. This evaluation is carried out by computing the histogram of the pixel intensity values for each frame.

Thus we can determine whether the image has a correctly exposed histogram so it can be used as the input to extract new PoIs and execute the purely visual tracking, or it has an underexposed or overexposed histogram then this frame is not mineable for extracting correct PoIs. Consequently, it is necessary to use inertial data and therefore go through visual-inertial tracking. In fact, based on pixel intensity histogram, once more than 80% of the frame is dark, or more than 83% of it is white, the frame is considered as underexposed or overexposed, respectively.

-Field of View Limit Metric After the inertial data (measured between two frames) preintegration, explained latter, the IMU can give a 6D pose for each frame: a 3D translation plus a 3D rotation (figure 4.10), which allow to quantify the system's motion between two frames (two final poses), in terms of rotational angles as well as angular and linear velocities. According to the detected motion

Local Control Metric

In addition to the measurements and tests presented above, there is a local control. This metric is inspired from [START_REF] Ivanov | Evaluating the consistency of estimation[END_REF] and [START_REF] Solà | Consistency of the monocular EKF-SLAM algorithm for 3 different landmark parametrizations[END_REF] and called EKF Consistency Metric. As presented in figure 4.9 (blue box), the local control is applied to EKF VI tracking method. That will facilitate the earlier detection of the pose computation problems, thus avoiding expensive computations. For visual-inertial EKF tracking, the consistency of the current EKF fusion filter is evaluated. The metric used is Normalized Deviation Squared (NDS), which deals with the measurement prediction [START_REF] Ivanov | Evaluating the consistency of estimation[END_REF]. It evaluates the consistency of the measurement prediction (made by the EKF filter) on a previous measurements sample. At this level, if the filter produces non-coherent measurement predictions, the final pose is not calculated and the system moves to the next image [197][107] .

Control Module Functioning

The control module provides the previously described metrics in a sequenced testing process (green block in figure 4.9). These tests ascertain whether or not each metric's thresholds are satisfied in order to generate a Boolean response (e.g. 0 or 1) that finally enables the most appropriate tracking mode to be performed.

At the start, the control module analyzes the present state of the system using the moving control metric. Generally, if the system is in stationary mode, it keeps its last estimated pose, otherwise it can go on the other measuring tests. This allows to optimize the processing time by avoiding recalculating the same pose several times. Secondly, if the system is moving then the image quality is checked. Actually, a histogram analysis is performed, at each frame received, in order to evaluate the frame quality predominantly based on its luminosity. Once the frame is identified as potentially problematic, the system switches directly to visual-inertial tracking while waiting for the next frame.

Using the frame quality metric ensures the robustness of the system and avoids tracking failure due to dark or overexposed locations, for example. Following these two previous tests, which mainly optimize computing performance, the control module checks the FoV, that above all enables to improve the tracking robustness and quality. As it is explained before, the FoV metric depends mainly on the rotational angle, angular velocity and/or linear velocity in order to maintain the tracking continuity in problematic cases. In fact, if the system is moving, the current frame is correct and the FoV thresholds are satisfied, the system is considered navigating in a difficult context and the EKF VI tracking is activated. Otherwise the system switch to visual KLT-ORB Tracking assuming that the navigation context is easy. Next control metric, which is the tracking quality metric, is only applied at the last step of the control module, when we have already verified that the system is moving, the current frame is correctly exposed and the FoV is not changed. It is primarily used to verify the previous tracking success rate in order to identify whether ORB PoIs need to be re-detected before executing the KLT-ORB visual tracking or not. In contrast to the foregoing tests, the pose estimate consistency can be tested in an offline mode using the local control metric inspired from [START_REF] Ivanov | Evaluating the consistency of estimation[END_REF] and [START_REF] Solà | Consistency of the monocular EKF-SLAM algorithm for 3 different landmark parametrizations[END_REF]. Actually, this test is only applied in order to improve the EKF VI pose quality relying especially on the EKF prediction.

Visual KLT-ORB Tracking

As presented on figure 4.12, the proposed visual tracking method is based on the KLT algorithm [START_REF] Kyu | Kanade-Lucas-Tomasi (KLT) feature tracker[END_REF], triangulation and pose estimation, according to the section 1.1, it is considered as a sparse f W l = [x l y l z l θ l φ l ρ l ] T : the l th landmark inverse depth coordinates [147]. It is composed of the camera position (x l , y l , z l ) T where the l th landmark was firstly observed, the azimuth θ l and the elevation φ l angle that define the unit radius (expressed in the global frame) from the camera center (x l , y l , z l ) T to the l th landmark, and ρ l is its inverse depth within the unit ray.

X B k = q W T B k p W T B k v W T B k b T g k b T a k T (4.8) X L k = f W T
Therefore, following 4.7, the EKF error state vector is expressed as:

X k = δq W T B k δp W T B k δv W T B k δb T g k δb T a k δ f W T 1 . . . δ f W T m T (4.10)
where the standard additive error: x = x + δx is used for the 3D position, velocity, biases, and landmarks.

Prediction -IMU Model IMU measures a system's angular velocity ŵ and acceleration â with respect to the inertial frame {B}. These measurements are assumed to be affected by zero mean Gaussian white noise η and bias b that varies slowly over time:

ω = ω + b g + η g (4.11) â = a + b a + η a (4.12)
In practice, the IMU provides measurements at defined times. To facilitate EKF prediction, the IMU propagation model is given directly in discrete time, which provides the necessary derivatives when calculating in close form. Subsequently, the discrete-time IMU state propagation model

X B k|k-1 = f k (X B k-1
) is expressed using the measured acceleration âk-1 and the angular velocity ωk-1 obtained by IMU:

qW B k|k-1 = qW B k-1 ⊗ 1 1 2 ( ωk-1 -bg k-1 )∆t (4.13) pW B k|k-1 = pW B k-1 + ṽW B k|k-1 ∆t + 1 2 g W ∆t 2 + 1 2 RW B k-1 ( âk-1 -ba k-1 )∆t 2 (4.14) ṽW B k|k-1 = ṽW B k-1 + 1 2 g W ∆t + RW B k-1 ( âk-1 -ba k-1 )∆t (4.15) bg k|k-1 = bg k-1 (4.16) ba k|k-1 = ba k-1 (4.17)
where g W the gravity vector expressed in the global frame W.

In this work, the slow random walk of the inertial biases is ignored, therefore the biases b a and b g are considered fixed and are estimated as part of the system state.

The first-order discrete-time linearized IMU error state propagation model is:

δX B k|k-1 = Φ k δX B k-1 + G k η B (4.18)
where η B = [η T a η T g ] T the system noise, its associated diagonal covariance Q is expressed as follows:

Q = σ a ∆t 0 3×3 0 3×3 σ g ∆t (4.19)
where:

σ a : accelerometer covariance.

σ g : gyroscope covariance.

Φ k and G k : the f k (.) Jacobian matrix with regard to the IMU state and the system noise Jacobian matrix, respectively.

Therefore, the covariance matrix is propagated as follows:

P k|k-1 = P B k|k-1 P BL k|k-1 P BL k|k-1 P L k|k-1 = Φ k P B k-1 Φ T k + G k QG T k Φ k P BL k-1 P BL k-1 Φ T k P L k-1 (4.20)
where P L k-1 is the landmarks covariance matrix (it depends on the camera covariance parameters).

Measurement Model

The inverse depth representation [START_REF] Montiel | Unified Inverse Depth Parametrization for Monocular SLAM[END_REF] is applied here to represent features in order to immediately use the new ones. This representation improves the linearity of the measurement equations, and the management of low parallax features. Thus, it enhances the accuracy of the system. The inverse depth representation f W l for the l th landmark can be transformed into corresponding EuclideanXYZ coordinates y W l ( f W l ) as :

y W l ( f W l ) =    x l y l z l    + 1 ρ l m(θ l , φ l ) (4.21)
where m(θ l , φ l ) is defined as:

m(θ l , φ l ) =    cosφ l sinθ l -sinφ l cosφ l cosθ l    (4.22)
Therefore, the measurement model that describes the projection of the l th landmark, wich is represented in inverse depth coordinates, to the k th image is:

z kl = h kl (X B k|k-1 , f W l ) + σ kl (4.23) z kl = π((R W B k R B C ) T (y W l ( f W l ) -p W B k -R W b k p B C )) + σ kl (4.24)
where σ kl is the measurement noise having the covariance Σ kl , and π is the projection function determined by the camera's intrinsic parameters, previously known from the calibration. According to this measurement model, the reprojection error is calculated as follows:

r kl = z kl -h kl ( XB k|k-1 , f W l ) (4.25)
and its linearized approximation is:

r kl ≃ H B lk δX B k + H f lk δ f W l + σ kl = H kl δX k + σ kl (4.26)
where the H B kl and H f kl matrices are derived from the measurement model h kl (X B k , f W l ), with respect to the IMU state estimation and the position of the l th landmark respectively. Thus, the Jacobian measurement matrix H kl is defined as:

H kl = H B kl 0 . . . H f kl 0 . . . ( 4
K k = P k|k-1 H T k (H k P k|k-1 H T k + Σ σ k ) -1 (4.29) Xk|k = Xk|k-1 • K k r k (4. 30
)

P k|k = (I 16+6m -K k H k )P k|k-1 (4.31)
with:

Σ σk (equation 4.29): the stacked covariance matrix of 2m × 2m of visual measurements

• (operator in equation 4.30): equivalent to the operator ⊕ for orientation and vector addition for other state.

According to system state prediction based on IMU measurement, the features used for the update are first detected, by the ORB detector, then matched. Subsequently, the Mahalanobis distance d = r T kl (H kl P k|k-1 H T kl + Σ σ kl ) -1 r kl is calculated to select the matched features that will be employed for the update. In fact, only features whose d is below the given threshold are considered inliers and are therefore used for the EKF update. In addition, the 1-point RANSAC method [START_REF] Civera | 1-point RANSAC for EKF-based Structure from Motion[END_REF] is applied here to find reliable inliers.

Conclusion

In this chapter, the proposed Context Adaptive VI SLAM for mobile embedded systems is detailed. First of all, the navigation environment analysis is a step that has proven to be crucial for a robust tracking and SLAM development. This analysis allows, on the one hand, to identify problematic navigation environments that require a greater effort in terms of computing time and capacity, on the other hand, to find the cases where navigation may be more easily insured and constraints relaxed. Thus, this analysis promotes robust tracking based on combining different algorithmic methods.

After discussing the navigation environments, the overall workflow of the Context Adaptive VI SLAM is explained, outlining the way in which the two proposed tracking methods, EKF VI Tracking and Visual KLT-ORB, interface with the rest of the SLAM processes. As a reminder, this work relies on the ORB SLAM and is only focused on the first tracking thread, where PoIs tracking and pose estimation are performed. Whereas, the ORB SLAM Keyframe selection, as well as its last two threads, Local Mapping and Loop Closing, are used without any modification. So, subsequently, the main Context Adaptive VI Tracking components are explained. First, the execution control module is presented. This module is used to analyze inertial data, image quality, PoI number, and tracking feedback, between the current and previous frames, in order to identify the motion type (rotation, translation) and the visual data quality. It also aims to choose the most adapted tracking method for the current navigation environment. In this work the choice is made between the visual approach based on the optical flow via the KLT algorithm, and the visual-inertial approach based on the data fusion using the EKF algorithm.

Once the navigation environment is analyzed, the two tracking methods used are presented. Thus, the implemented visual tracking is described. This method allows faster real-time tracking, especially in an easy navigation context (low motion and textured scenes), which is beneficial for mobile embedded systems. In fact, in most works, a visual tracking in SLAM system is performed by detecting PoIs and describing them for each frame. This approach requires a significant execution time and computation cost. For instance in ORB SLAM, computing the detected PoIs description takes at least 10ms. This explains the benefits of using our proposed optical flow-based VO. It allows to quickly calculate the relative pose between the current frame and the previous frame, thanks to the decrease in the frequency of PoI detection/description processing. Especially, in the case of relatively low translation and rotation motions, and also if mapping and keyframe creation are not required, there is no need to re-detect a high number of PoIs so the processing runtime is reduced. Thus, the visual KLT-ORB tracking reduces the complexity of pose computation and localization for SLAM. In addition, since KLT algorithm employs 2D points, in our proposal we go through the triangulation process using 2D-3D reprojection in order to provide the map, update it and maintain its coherence across frames and from one approach to another. Simultaneously to this process, the pose estimation task performed herein is characterized by the implementation of the motion model with respect to the previous image calculation, only. This is an advantageous method, in term of computation complexity, compared to other visual methods involving the epipolar geometry computation.

Nevertheless, when navigation environment is difficult, the tracking is performed by fusing inertial and visual data using EKF algorithm. This accurate and efficient tracking technique is explained. In our proposal, the state vector used in the EKF VI system contains, besides position, rotation, speed and bias, landmarks that help to improve the accuracy of the pose estimation. In order to reduce these landmarks accumulation and the system computational complexity, we limit the number of landmarks integrated in the state vector to only 5 ones, in total. In addition, these landmarks are updated every frame processed (removal of lost landmarks and integration of the new selected landmarks). Despite of the expensive overhead of computation as compared to pure visual method, it is possible to minimize this time and take advantage of it for a robust SLAM implementation on embedded mobile systems. This is achieved by alternating between the two methods, thanks to the execution control module. The latter aims at analyzing the navigation environment and, depending on the motion type and scene conditions, the suitable tracking approach is selected and executed.

In order to assess the performances and efficiency of the proposed solution, various tests are carried out, within defined experimental settings and using dedicated evaluation methods and metrics. They are presented and discussed in the next chapter (chapter 5).

The experiments were performed using various sequences of KITTI's [START_REF] Mattyus | Enhancing Road Maps by Parsing Aerial Images Around the World[END_REF] and EuRoC's [START_REF] Burri | The EuRoC Micro Aerial Vehicle Datasets[END_REF] datasets (subsection 4.1 of chapter 4). The KITTI dataset only provides visual data (images), which is characterized by simple and non-abrupt linear motions over a longer distance (mainly important 2D motions with less vibration). Indeed, this visual data is about car navigation motions in an outdoor context, with different dynamic elements such as vehicles, cyclists and pedestrians, as well as different light and shadow conditions. The use of the KITTI dataset is primarily intended to analyze the initial behaviour of visual SLAM, and reach a conclusion regarding the main problems encountered during the different experiments, including the scale estimation and the non-determinism issues.

EuRoC dataset provides inertial data (IMU measurements), in addition to visual data (images of size : [752 x 480]). This is especially useful for evaluating visual-inertial and visual tracking, and SLAM systems in the same conditions. Moreover, EuRoC dataset is widely used in literature for benchmarking different SLAM systems. For these reasons, the EuRoC dataset is the focus of the presented experiments.

Trajectory Evaluation

The accuracy of tracking methods is quantified by evaluating the estimate trajectory with respect to the ground truth. However, this is not straightforward and easy task. In fact, the estimated trajectory and the ground truth are often expressed in different reference frames, therefore they cannot be compared directly. Subsequently, a pre-process of trajectory alignment must be carried out. Furthermore, a trajectory contains different poses values (position and rotation) at different times making it a large collection of data. Consequently, the way to concisely summarize whole trajectory information into precise and accurate metrics is not trivial. There are many processes used to evaluate the estimated trajectories. In this section, the commonly trajectory evaluation techniques and error metrics are discussed. In order to be able to address this issue, non-determinism of the system and scale estimation problem are first discussed.

Non-Determinism

The non-determinism issue relates to the behaviors of the multi-threading system implementation and the heavy use of RANSAC [154][111]. As an example, figure 5.2 illustrates the non-deterministic nature of the ORB SLAM system on each trajectory axis (x, y, z), the discontinuity of some trajectories, such as in run 6 (green trajectory), reflects poses that were not computed during these runs for these frames. Based on ORB SLAM analysis, the non-determinism can generate an additional error margin estimated at ≃ 1m (expressed in RMSE). To overcome this problem, several works [19][44][111] [START_REF] Mur | ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras[END_REF] point out the need to run the non-deterministic system five to ten times, and to calculate its mean, maximum and minimum error. In our evaluations the system is running ten times.

Scale Estimation

The KLT-based VO trajectories, generated using the KITTI dataset, are plotted to illustrate the scaling effect. The issue with scale estimation remains a problematic subject that is not well discussed in literature, in particular the methods used in visual-inertial based systems such as R-VIO [START_REF] Huai | Robocentric Visual-Inertial Odometry[END_REF], VI Mono-SLAM [START_REF] Qin | VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator[END_REF] and Mono Visual EKF SLAM [START_REF] Russo | A ROS Implementation of the Mono-Slam Algorithm[END_REF]. Each proposed scale estimation method is adapted to ground truth measurements. In the second case, the computations give an error estimation that increases over time. This technique aims to compute the rigid body transformation or the yaw-only transformation, for example, using only a single state, which can be the first state. Moreover, both techniques assumed that the trajectory evaluation can be based only on the translational components "trans" (position (x, y, z)) of the estimated trajectory and the ground truth poses [START_REF] Salas | Trajectory alignment and evaluation in SLAM: Horn's method vs alignment on the manifold[END_REF] [START_REF] Zhang | Rethinking Trajectory Evaluation for SLAM: a Probabilistic, Continuous-Time Approach[END_REF].

Various relevant VO/ VIO/ VSLAM/ VI SLAM state-of-the-art works [93][231][55] [START_REF] Quan | Accurate Monocular Visual-Inertial SLAM Using a Map-Assisted EKF Approach[END_REF] apply the [START_REF] Zhang | A Tutorial on Quantitative Trajectory Evaluation for Visual(-Inertial) Odometry[END_REF] tutorial to align and assess their results. In order to be comparable with these systems, we also carried out our evaluations relying on the [230][234] pipeline. This is performed by focusing on the use of all the states to perform the yaw-only rigid body, applying a fixed scale s (presented later in equation 5.2) at 1, for visual-inertial systems and similarity alignment, estimating a scale according to the Umeyama method [START_REF] Umeyama | Least-Squares Estimation of Transformation Parameters Between Two Point Patterns[END_REF], for monocular visual systems, in addition to computing errors on the basis of the translation component considering all the three axis (x, y, z). This approach requires two main steps: time-alignment and geometrical-alignment.

Time-Alignment

Because the ground truth and estimated trajectories do not usually have the same timestamps, these trajectories are aligned temporally in order to ensure a correct temporal association. There is currently no method that addresses the temporal faulty association. To find the ground truth that corresponds to the estimated pose at time t, most tools take a naive matching strategy and simply use the closest ground truth [181][213]. Thanks to this step, the used data will be temporally consistent for the remaining steps.

Geometrical-Alignment

After the time-alignment, the geometrical-alignment is performed as in [START_REF] Zhang | A Tutorial on Quantitative Trajectory Evaluation for Visual(-Inertial) Odometry[END_REF] and is available as an open source tool in [START_REF]Rpg trajectory evaluation online tool[END_REF]. This processing is carried out joining the estimated positions { pi } N-1 i=0 and the ground truth {p GT i } N-1 i=0 , considering that all states have the same uncertainty since the covariance is not determined. The geometrical-alignment aims to compute the transformation S ′ = {s ′ , R ′ , t ′ } that satisfies the differentiation below:

S ′ = argmin S={s,R,t} N-1 ∑ s=0 p GT i -sR pi -t (5.2)
where s is a scalar (s = 1 if the scale is known, like in stereo and inertial setup), R ∈ SO(3) and t ∈ R 3 are the system rotation matrix and translation, respectively.

The equation 5.2 is solved using the Umeyama approach [START_REF] Umeyama | Least-Squares Estimation of Transformation Parameters Between Two Point Patterns[END_REF] discussed in [START_REF] Zhang | A Tutorial on Quantitative Trajectory Evaluation for Visual(-Inertial) Odometry[END_REF] and based on the SVD method to obtain s ′ , R ′ and t ′ = mean N (p GT )s ′ R ′ mean N ( p). Subsequently, the aligned estimated trajectory (position { pi } N-1 i=0 and rotation { Ri } N-1 i=0 components) is calculated according to the computed transformation elements S ′ = {s ′ , R ′ , t ′ }:

p′ i = s ′ R ′ pi + t ′ , R′ i = R ′ Ri , (5.3)
Furthermore, in case where the yaw-only rigid body transformation is used, the Umeyama rotation computation needs to be adapted. Thus, according to equation 5.2 R ′ is expressed as:

R ′ = argmin R∈SO(3) P -R P 2 F
(5.4)

where: P = [r 0 , r 1 , ..., r N-1 ], r i = p GT iµ P and µ

P = 1 N ∑ N-1 i=0 p GT i = mean N (p GT ) P = [r 0 , r1 , ..., rN-1 ], ri = pi -µ P and µ P = 1 N ∑ N-1 i=0 pi = mean N ( p)
.

F

is the Frobenius norm.

The Frobinus expression computed in equation 5.4 can be written also as follows:

P -R P 2 F
= trace(PP T + P PT -2R PP T ) (5.5) therefore, equation 5.4 is similar to:

R ′ = argmax R∈SO(3) trace(R PP T ) (5.6)
Hence, if at this step we are only interested in the yaw-only rigid body transformation, then we will only need to identify one parameter θ ′ that allow the aligned rotation R ′ z computation (equation 5.1):

θ ′ = argmax θ (p 12 -p 21 )sinθ + (p 11 -p 22 )cosθ (5.7)
where p ij is the coefficient of PP T . Actually, such an approach is inherently problematic. In fact, as stated in [START_REF] Zhang | Rethinking Trajectory Evaluation for SLAM: a Probabilistic, Continuous-Time Approach[END_REF], the aligned trajectory { p′ i } N-1 i=0 is computed using the estimate { pi } N-1 i=0 and the ground truth {p GT } N-1 i=0 , and then used once again when evaluating the estimates. Therefore, the Absolute Trajectory Error (ATE) and Relative Pose Error (RPE) metrics (defined in section 5.2.4) are affected. Despite these inconsistency, this alignment technique is commonly used to evaluate several prominent studies in the literature. Likewise, in this thesis we were led to respect this consensus in order to evaluate our solution and to compare it with other literature approaches [START_REF] Huai | Robocentric Visual-Inertial Odometry[END_REF][231] [START_REF] Quan | Accurate Monocular Visual-Inertial SLAM Using a Map-Assisted EKF Approach[END_REF].

Evaluation Metrics

The accuracy evaluation is achieved considering the most common error metrics employed for VO/ VIO/ VSLAM/ VI SLAM and positioning systems appraisal: ATE and RPE [START_REF] Zhang | A Tutorial on Quantitative Trajectory Evaluation for Visual(-Inertial) Odometry[END_REF][228] [START_REF] Sturm | A benchmark for the evaluation of RGB-D SLAM systems[END_REF]. Let { p′ i } N-1 i=0 ∈ SE(3) is a sequence of poses from the aligned estimated trajectory and {p GT } N-1 i=0 ∈ SE(3) is a sequence of the ground truth trajectory. These sequences are assumed to be time-synchronized, equally sampled, and equally long N.

ATE: computes the distance between the estimated aligned poses p′

i of the camera and its ground truth p GT , expressed in meters (figure 5.8). This metric is more suitable for SLAM evaluation since it allows an assessment of the overall consistency of the estimated trajectory with respect to the ground truth. Here, the ATE is computed focusing on the translational component trans (i.e. the 3D position considering all axis (x, y, z)) and expressed as a mean, median and RMSE:

∆ trans( p′ i ) = trans(p GT i ) -∆R i (trans( p′ i ) T ) (5.8)
where ∆R i = R GT i ( R′ i ) T is the rotational error between the aligned trajectory and the ground truth, R GT i and trans(p GT i ) are the rotation matrix and the 3D position of a given ground truth pose i, respectively, R′ i and trans( p′ i ) are the rotation matrix and the translational component of the estimate trajectory i, respectively.

(RMSE), estimated at an average of ≃ 0.2m for the majority of dataset sequences used (table 5.4). Hence, the robustness of this approach is estimated by ≃ 25% to ≃ 30% of the global dataset (≃ 800 used frames out of the total of ≃ 2714 dataset frames). Table 5.4 represents the RPE of the tracked sections for various dataset sequences. According to these measurements, the KLT-ORB tracking behaves correctly on the travelled distances, since the KLT algorithm is adapted to the movement type performed during this period, reaching at maximum a RPE of 0.25m, in term of RMSE, when tracking in a difficult environment such as MH03. However, the KLT-ORB technique is not robust enough to track across all sequences in the dataset, thus ATE cannot be computed.

Statistics on the processing time by the main functionalities of the proposed solution, are given for the KLT-ORB tracking profiling. Table 5.5 presents the tracking runtime results expressed in seconds.

The KLT-ORB tracking thread works at an average frequency of ≃ 32Hz. This is mainly due to the use of the keypoint management and the KLT algorithm. ORB PoIs detection occurrence is reduced, which also decreases the total execution time compared to ORB SLAM.

EKF Visual-Inertial Tracking

In this section, EKF VI tracking is evaluated. First, the quality of the results is studied, then the profiling process is described. As for ORB SLAM odometry, this method is evaluated on different EuRoC sequences and using every time a mean curve from a tenth of runs. Figures 5. As presented in table 5.6, RMSE of the EKF VI tracking ATE is in an average of 0.2m for easy dataset sequences and 0.6m for the most difficult ones (see section 4.1.2 of chapter 4) thanks to the EKF implementation method used for visual and inertial data fusion. However, EKF VI tracking may suffer from some discrepancies and an increase in the number of errors mainly in the middle and at the end of the navigation. These discrepancies are more visible over long distances (> 70 meters, sequences in grey cells on the table 5.6), they lead to ATE estimated at a minimum of 0.39m, in term of RMSE. However, over medium or short distances (between 30 meters and 70 meters, sequences in white cells on the table 5.6), these deviations decrease and provide an estimated error of 0.17m maximum. Based on the results shown in table 5.6, we can conclude that the proposed EKF VI approach has an efficient accuracy over short to medium distances. Furthermore, it also provides strong performance even during fast motions and abrupt rotations (difficult sequences).

In addition, figure 5.17 shows the behavior of the EKF VI' RPE as a function of different subtrajectory lengths. In an easy environment, the RPE values are small and their evolution is more stable over the different distances, for example in V201 the error increases from ≃ 0.07 for a distance traveled of 3m to 0.1m for a distance traveled of 14m. However, for difficult environments, these values start to increase more quickly (e.g. from 0.09m to 2m) but are still less important when compared with those of ORB SLAM odometry.

As previously described, EKF VI tracking is made up of three basic parts: Prediction, in which inertial measurements are processed and pre-integrated in order to predict the system state; Measurement, in which frames are processed to compute the measured state and how many new PoIs need to be detected (if any) and added to the system status computation is decided; and Update, in which the final state and pose estimation is performed. Since our thesis system is a tight coupling, the parts involving Measurement and Update intersect at different computing levels. For profiling evaluation, these parts are thus performed as a single large function that represents about 70% of the overall EKF VI tracking computation complexity. Whereas the Prediction part depends mainly on the IMU and the initial configuration as well as the feedback of the previous system state. The scene characteristics and the motion behavior have a direct impact on the number of detected PoIs, this number varies between 5 and 25 PoIs at maximum. Any lost PoI requires additional processing and therefore additional complexity and computing time. Indeed, this issue leads to a selective renewal of PoIs, or even a total re-detection of new ORB PoIs in order to integrate them instead of the lost ones. As a consequence, a computational load is required to model these new points in a way that makes them suitable for use in the rest of the EKF processing. Loosing PoIs is frequent occurrence in difficult navigation environments (e.g. EuRoC MH03 and V203 dataset sequences, etc.).

The additional computational load, caused by the re-detection and modelling of PoIs, is estimated to be at least 3ms more than the total computational load for tracking in an easy environment (e.g. 35ms in V101 and 39ms in MH04). Summarizing, as presented in table 5.12 and 5.13, the computation complexity, especially that related to PoIs processing, and execution time of the Context Adaptive VI Tracking is significantly improved. Using the EKF VI algorithm on 80% of the used dataset reduces the complexity of the system's computation compared to ORB SLAM [227][169]. This is especially true in regard to the computational load where the approach has a gain of 25% on average, thanks to the reduction of the

Discussion & Conclusion

This chapter aimed to describe experiments performed to demonstrate the relevancy of the proposed Context Adaptive VI Tracking. The experimental environment was introduced as well as the main issues met in the different experiments, were discussed. The non-determinism and the scale estimation problems were explained. To address the non-determinism problem, we relied on the state-of-the-art technique executing the systems ten times, then use the average of the results for evaluation purposes. Scale estimation is an issue little addressed in the present literature, it poses many difficulties regarding the pose estimation. An adapted scale estimation approach was implemented in this work to compute the final camera poses, based on the rate between inertial and fused visual-inertial poses. In addition, a section of this work focuses on the trajectory alignment process and discusses the evaluation error metrics [START_REF] Zhang | A Tutorial on Quantitative Trajectory Evaluation for Visual(-Inertial) Odometry[END_REF] [181] [START_REF] Zhang | Rethinking Trajectory Evaluation for SLAM: a Probabilistic, Continuous-Time Approach[END_REF]. In our thesis, we performed the trajectory alignment applying the yaw-only rigid body and similarity transformations that are suitable for visual-inertial and monocular systems, respectively, and we quantified the system quality relying on the system trajectory accuracy and using the two most common metrics: ATE and RPE. Actually, trajectory alignment is a trajectory pre-processing used to enable the system quality evaluation and comparison with different state-of-the-art tracking methods: ORB SLAM [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF], R-VIO [START_REF] Huai | Robocentric Visual-Inertial Odometry[END_REF], VINS [START_REF] Qin | VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator[END_REF], OKVIS [START_REF] Leutenegger | Keyframe-based visual-inertial odometry using nonlinear optimization[END_REF], VI-EKF SLAM [START_REF] Quan | Map-Based Visual-Inertial Monocular SLAM using Inertial assisted Kalman Filter[END_REF], AVIO [START_REF] Piao | Adaptive Monocular Visual-Inertial SLAM for Real-Time Augmented Reality Applications in Mobile Devices[END_REF], EKF VIO [START_REF] Quan | Accurate Monocular Visual-Inertial SLAM Using a Map-Assisted EKF Approach[END_REF] and ROVIO [START_REF] Bloesch | Robust visual inertial odometry using a direct EKF-based approach[END_REF]. After the previous discussions, our solution and its different components was evaluated. But, since ORB SLAM is considered to be a reference in the domain, it has been taken as a starting point for this work and evaluated in a first step. Our proposal was appraised by evaluating each component separately: the KLT-ORB Tracking and the EKF VI Tracking, and then the whole Context Adaptive Tracking system.

The results of our performance assessment show, that the KLT-ORB tracking provides a short execution time and gives a sufficient accuracy, but also it lacks of robustness in difficult environments (with long distances and/or characterized by abrupt and rapid motions). The KLT-ORB tracking unquestionably provides a significant computational cost reduction thanks to the decrease of ORB PoIs redetection frequency and the use of the simple KLT visual algorithm. Furthermore, EKF VI tracking has been proven to be a robust method in most navigation conditions. However, it requires an additionnal computing workload due to the integration of IMU measurements. We proposed a leverage that consists in reducing the number of PoIs detected (500 or 5 PoIs) and used for pose estimation. The performance results of the proposed system (the Context Adaptive VI Tracking) was presented, analyzed in terms of trajectory accuracy, computational load and computational time, and compared to other state-of-the-art works' results. Actually, Context Adaptive Tracking provides, an average ATE of 0.08m in easy navigation environments and of 0.19m in difficult environments. It provides a continuous and robust tracking in different navigation environments, contrary to some known literature approaches such as ORB SLAM [START_REF] Mur-Artal | ORB-SLAM: a versatile and accurate monocular SLAM system[END_REF] or the adaptive visual-inertial approach proposed in [START_REF] Piao | Adaptive Monocular Visual-Inertial SLAM for Real-Time Augmented Reality Applications in Mobile Devices[END_REF]. In addition, the proposed system accuracy is similar to most inertial visual approaches, however, it better exceeds them in the case of difficult navigation by at least a 0.02m error improvement. It can achieve an average frame-rate processing between 32 f ps and 29 f ps, depending on the navigation environment (easy or difficult), this represents an execution time gain estimated by 1ms and 1.8ms, when navigating in easy context and difficult context, respectively, compared to R-VIO [START_REF] Huai | Robocentric Visual-Inertial Odometry[END_REF]. Therefore, our proposal differs from other works by ensuring the system accuracy and even improving it and also reducing the computational load and execution time. This performance is due to our different contributions, mainly the algorithms used and their adequacy, the control module metrics and its operation, and the overall system implementation process. Moreover, the Context Adaptive SLAM was also assessed in order to verify its behavior regarding the rest of ORB SLAM components (Local Mapping and Loop Closing). Therefore, the Context Adaptive SLAM provides an accuracy loss estimated only by less than 0.01m comparing to ORB SLAM accuracy, especially for easy navigation environments. After all the previous evaluations and discussion, the interest of the proposed execution control module in the proposed context adaptive tracking, using two different approaches, is confirmed regarding the system robustness, accuracy and respect of the embedded system constraints.

Conclusion and Perspectives

Summary of the contributions

This research study proposes a robust visual-inertial tracking method suitable for mobile embedded systems implementation such as MAVs [175][93]. On one hand, tracking algorithms employed in different navigation systems are constrained by the navigation context conditions, especially those used in vision based navigation systems. For example, these systems are sensitive to black scenes with low texture, fast camera motion causing blur images, etc. Therefore, inertial measurement are used to improve visual tracking systems. This requires more computational complexity to perform visual-inertial fusion for tracking. On the other hand, embedded systems are resources constrained. They are limited by computational power, on-board memory and sensor payload. Thus, they cannot handle the straightforward implementation of high computational complexity tracking algorithms. Current thesis develops and evaluates a Context Adaptive VI Tracking for embedded integration. It switches between two tracking methods: visual KLT-ORB tracking and EKF VI tracking, thanks to the execution control module that analyses the navigation context (motion, scene, previous tracking quality, and tracking consistency) and enables the switch to the appropriate tracking method.

First, a study of the state-of-the-art of navigation and tracking techniques based on computer vision and visual-inertial fusion models was put forward. Then, a discussion on the several embedded computing architectures has been also carried out. For designing adequate embedded systems, the hardware/software co-design methodology was explained as well. In addition, concerning tracking and localization, SLAM and odometry methods were discussed with emphasis on embedded implementations based on monocular vision and visual-inertial fusion. Besides this, an analysis of the navigation environments was performed. The focus was on the identification of the most problematic issues that impact the tracking robustness. This study allowed to choose the required control metrics and to configure the proposed execution control module.

After the previous discussion, our solution was presented and detailed. It consists of three main components, which are two different tracking methods: KLT-ORB tracking and EKF VI tracking, and an execution control module. The proposed system switches between these two tracking techniques according to the navigation environment analysis performed using the execution control module.

The latter uses different metrics to alternate between visual and visual-inertial tracking such as system motion (linear and angular velocity), system FoV (traveled distances and rotational angles) and previous tracking quality (number of tracked PoI from last frame). Therefore, our proposition is named: "The Context Adaptive VI Tracking". Subsequently, our proposed system is evaluated. Throughout our appraisals, several challenging issues were highlighted, in particular when comparing the estimated trajectories to the ground truth and computing estimating accuracy errors. These aspects include primarily the non-determinism, the scale estimation and the trajectory alignment. Firstly, to deal with the non-determinism issue, each approach is run ten times and the estimation errors are computed on the average of these runs.

Next, for scale estimation, we analyzed several methods proposed in the literature and we took inspiration from an approach devoted especially to visual-inertial systems [START_REF] Spaenlehauer | A Loosely-Coupled Approach for Metric Scale Estimation in Monocular Vision-Inertial Systems[END_REF]. The scale estimation solution, proposed in this work, consists in computing a ratio between the pure inertial pose and the final fused pose. Subsequently, to evaluate our estimate trajectories regarding the ground truth and compare them with other literature solutions, we have to get the estimation and the ground truth into the same reference frame using an alignment technique commonly used in the state-of-the-art systems, in particular those based on visual-inertial fusion. Therefore, in this thesis, we applied the Zhang and Scaramuzza trajectory alignment [START_REF] Zhang | A Tutorial on Quantitative Trajectory Evaluation for Visual(-Inertial) Odometry[END_REF] based on the Umeyama method [START_REF] Umeyama | Least-Squares Estimation of Transformation Parameters Between Two Point Patterns[END_REF] and available as an open source tool [START_REF]Rpg trajectory evaluation online tool[END_REF].

The Context Adaptive VI Tracking assessment is performed taking into account all the issues addressed above. Our proposal ensures a robust tracking even in the difficult and easy navigation environment. Relying on different EuRoC dataset sequences levels, as one of the most relevant (visual-inertial) dataset used in VINS literature giving a diversity of navigation environments configurations, our proposed system provides an accuracy, expressed in terms of ATE's RMSE, between 0.063 and 0.098meters at maximum in easy environments (V101 and MH01, respectively), and between 0.14 and 0.29meters in difficult environments (MH03 and MH04, respectively). This is reached by maintaining an average frame-rate processing of 32 f ps in easy environments and 29 f ps in difficult ones. These results are obtained thanks to the efficiency of the chosen tracking algorithms, the execution control module and the PoIs management that significantly reduces the occurrence of PoIs detection and also limits the number of the detected PoIs to either 5 or 500 PoIs. Our primary contributions to achieve these results were:

(i) analysis of different types of navigation context, and identification of the more complex and problematic navigation conditions;

(ii) development of an execution control module allowing a switch between two tracking methods, according to various parameters: the system's navigation environment type (image quality), in motion, FoV and the current tracking operating status (PoI cheking);

(iii) PoIs management methodology performed between different processing functions, that saves memory and computational time by limiting the number of the PoIs at 500 or 5 PoIs;

(iv) development of two tracking approaches, appropriate to the embedded implementation, as well as their adaptation to the remaining ORB SLAM processing parts: KLT-ORB and EKF VI tracking.

However, integrating the multi-method solution on an embedded architecture might require difficult engineering and implementation, particularly, in terms of latency required for the switching management between the two tracking methods. For now, this latency can reach 6 seconds in case of medium-difficult tracking environment (> 70 meters).

Perspectives

The research presented in this thesis offers potential extensions and directions for further work. These future developments aim to extend the contributions provided in this thesis and to improve the practicability of the proposed solution in real world applications. In the following paragraphs, our outlook perspectives are introduced.

Improving the scale estimation process Accurate scale estimation is a challenging research problem in visual and visual-inertial tracking. Experimental results have illustrated the benefit of using visual and inertial data for the scale estimation. In particular, the scale estimation method based on IMU and vision-inertial fused poses. As mentioned in chapter 5, this estimation technique varies among the time, because it relies on the final estimated pose which can be erroneous. Future research can be undertaken to develop better scale estimation strategies using visual and inertial data independently of final pose processing, even with tightly coupled sensors. This is intended to avoid the large variations of scale, as well as the additional noise and errors inherent to this.

Improving the trajectory consistency metrics Future studies can also expand the metrics employed for the execution control module. Experimental results have demonstrated the importance of the execution control module and its controlling metrics used to alternate between the two proposed tracking approaches. Following these results, future studies can also improve the existing metrics, especially, trajectory consistency metric. They already can define a new global metric, instead of local NDS metric, that refines the final pose estimation independently of the chosen tracking method.

Therefore, the tracking accuracy can be enhanced.

Hardware-constrained integration of the Context Adaptive Visual-Inertial Tracking

The Context Adaptive Visual-Inertial Tracking can be employed in larger navigation systems. In particular, micro robotic vehicles, which have limitation of computation complexity, memory, space and power resources. The proposed method has been developed to be implemented on such embedded systems and to respect the embedded systems requirements and constraints. Therefore, the future work could focus on porting the solution on an embedded Multi-Processor SoC (MPSoC) architecture of the Xilinx family, Zynq UltraScale+ MPSoC for example, including accelerators and dedicated components, such as GPU, video CODEC, etc. In addition, the flexible I/Os and processing power of the Zynq UltraScale+ MPSoC requires very little hardware beyond the MPSoC itself, other than the sensors and the external memory. The performance/watt measurement of this device is about 3 times better compared to a CPU-based system using silicon from a leading competitor [START_REF]2 Performance characteristics: table of the different algorithms treated and the storage size of each descriptor vector[END_REF]. Subsequently, this embedded implementation will enable to assess the switching latency between the two tracking methods and analyze its behavior, in order to improve it in a constrained context. Also it will allow to evaluate the accuracy and the robustness of the proposed tracking in a real embedded context, and its efficiency in terms of the computation complexity and the required execution time.

  and Communication Sciences and Technologies (MSTIC) Doctor of Philosophy Intelligent Embedded Camera for Robust Object Tracking on Mobile Platform by Imane SALHI

Keywords.

  Visual Tracking, Visual-Inertial Tracking, Robust Tracking, Simultaneous Localization And Mapping (SLAM), Odometry, Visual-Inertial Navigation, Multi-sensor Systems, Embedded Systems, Co-design, Fusion, IMU/Camera coupling, Loose Coupling, Tight Coupling. v UNIVERSITE GUSTAVE EIFFEL Résumé Mathématiques et Sciences et Technologies de l'Information et de la Communication (MSTIC) Docteur en Philosophie Caméra Intelligente Embarquée pour le Suivi Robuste d'Objet sur Plateforme Mobile par Imane SALHI Le suivi visuel-inertiel est une thématique d'actualité, difficile à traiter, notamment lorsqu'il s'agit de respecter les contraintes des systèmes embarqués, comme dans les drones autonomes (Unmanned Aerial Vehicles (UAVs)). Les questions relatives à la miniaturisation, la portabilité et la communication des systèmes électroniques s'inscrivent dans des problématiques actuelles en matière d'avancée technologique. Pour répondre de manière efficace à ces problématiques, il est nécessaire d'envisager des traitements complexes et des implémentations sur des supports contraignants en termes d'intégration et de consommation d'énergie, tels que les microvéhicules aériens (MAVs), les lunettes et les caméras intelligentes.

( 1 )

 1 L junction, (2) V junction, (3) T junction, (4) Y junction, (5) X junction and (6) point of strong texture variations (checkerboard junction) . . . . . . . . . 1.8 Examples of sequences subject to different image transformations : (a) scale and rotation change, (b) viewpoint change, (c) light change , (d) pure rotation, (e) JPEG compression and (f) blur . . . . . . . . . . . .

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.12 Global workflow of DELF system [160] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.13 Structure from Motion from two views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.14 An overview of the 2D-to-2D algorithm [190] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.15 Motion and relative scale estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.19 EKF MonoSLAM overview [46] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.20 Brief overview of the main PTAM features [117] . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.21 ORB-SLAM overview [153] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Conventional IMU (six dimensions) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 The micro-structure of the IMU MEMS's devices : Accelerometer & Gyroscope [140] . . . . . . . . . . . 2.3 Inertial Measurement Unit examples : (a) Bosch IMU chip BNO055, (b) ADIS IMU (ADIS16490BMLZ) . . . 2.4 Overview of common VINS categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Loosely-coupled IMU/camera overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Tightly-coupled IMU/camera overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Functional diagram of VI ORB SLAM [155], focusing on the exploitation of inertial information in the global visual chain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 Evolution of the optimization in the Tracking thread of VI ORB SLAM [155] . . . . . . . . . . . . . . . xvi 2.9 Comparison of Local BA between original ORB SLAM (left) and Visual-Inertial ORB SLAM (right). The local window in Visual-Inertial ORB SLAM is retrieved by temporal order of keyframes, while in ORB SLAM is retrieved using the covisibility graph [155] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.10 VINS Mono System framework, reproduced from [175] . . . . . . . . . . . . . . . . . . . . . . . . 2.11 An illustration of the flowchart of the Monocular VIO algorithm proposed in [92] . . . . . . . . . . . . 2.12 a) EKF pose estimation, b) MSCKF pose estimation . . . . . . . . . . . . . . . . . . . . . . . . . . 2.13 Block Diagram of MSCKF's workflow process in R-VIO . . . . . . . . . . . . . . . . . . . . . . . . 3.1 CPS standard scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Mobile embedded systems examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 NVIDIA Tegra DRIVE Xavier SoC [161] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Xilinx Zynq UltraScale+ MPSoC Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 a) Tensilica L106 DSP, b) Snapdragon 845's Hexagon 685 DSP . . . . . . . . . . . . . . . . . . . . . 3.6 Functional block diagram of ARM Mali C71 ISP [8] . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Navion chip architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  A classification of embedded tracking systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Context Adaptive VI SLAM workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Sequence example of the Oxford Robotcar dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Sequence example of the 7 scenes dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Sequence example of the KITTI dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4. 8

 8 Context Adaptive VI SLAM overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.9 The adaptive execution control flowchart (v: velocity, w: angular velocity, t: time) . . . . . . . . . . . . 4.10 The six DoF movement composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.11 Rotation thresholds (estimated from head and drone motion analysis) . . . . . . . . . . . . . . . . . . 4.12 Visual KLT-ORB Tracking workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.13 EKF VI Tracking workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.14 IMU preintegration diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Experimental environment structural architecture . . . . . . . . . . . . . . . . . . . . . . . . . . .

  dataset (V101) ground truth (m) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8 Illustrations of ATE and RPE (inspired from [230]) . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.9 ORB SLAM (Odometry) aligned trajectory vs. EuRoC dataset ground truth (V201) (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . . . . . . . . 5.10 ORB SLAM (Odometry) aligned trajectory vs. EuRoC dataset ground truth (MH03) (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . . . . . . . . 5.11 ORB SLAM (Odometry) aligned trajectory vs. EuRoC dataset ground truth (V203) (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . . . . . . . . 5.12 RPE translation of ORB SLAM (odometry), played on different EuRoC dataset levels, for different subtrajectory lengths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.13 KLT-ORB tracking aligned trajectory vs. EuRoC dataset ground truth (V201) (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.14 EKF VI tracking odometry (aligned trajectory) vs. EuRoC dataset (V201) ground truth (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . . . . . . 5.15 EKF VI tracking odometry (aligned trajectory) vs. EuRoC dataset (MH03) ground truth (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . . . 5.16 EKF VI tracking odometry (aligned trajectory) vs. EuRoC dataset (V203) ground truth (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . . . . . . 5.17 RPE translation of ORB SLAM (odometry), played on different EuRoC dataset levels, for different subtrajectory lengths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.18 Context Adaptive VI Tracking (before and after alignment) vs. EuRoC dataset (V101) ground truth (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . 5.19 Context Adaptive VI Tracking (before and after alignment) vs. EuRoC dataset (MH03) ground truth (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . 5.20 Context Adaptive VI Tracking (before and after alignment) vs. EuRoC dataset (V203) ground truth (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . 5.21 Context Adaptive VI Tracking vs EuRoC dataset ground truth (V201)(m). The circles mark the switching areas between the two modes: the KLT-ORB method and the EKF VI method. The red rectangles denote the trajectory parts that are from the KLT-ORB method. The others are the trajectory parts that are from EKF VI method (horizontal axis shows the IDs of the processed frames) . . . . . . . . . . . . . . . . . . . . . 5.22 RPE translation of ORB SLAM (odometry), played on different EuRoC dataset levels, for different subtrajectory lengths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.23 Trajectories comparison of different systems (ORB SLAM (Odometry), R-VIO, EKF VI tracking and the proposed system: Context Adaptive VI Tracking) on an easy-medium EuRoC dataset (MH03) (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . xviii 5.24 Trajectories comparison of different systems (ORB SLAM (Odometry), R-VIO, EKF VI tracking and the proposed system: Context Adaptive VI Tracking) on a difficult EuRoC dataset (MH03) (m) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . . . . . . . . 5.25 ORB SLAM odometry, ORB SLAM and Context Adaptive VI SLAM comparison (MH03) (gray lines represent the states correspondences with a step of ten poses) . . . . . . . . . . . . . . . . . . . . . . . . A.1 Illustration of the detection based on a DoG of the SIFT algorithm . . . . . . . . . . . . . . . . . . . . A.2 Histogram construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.3 Illustration of the pixel to be tested (central pixel) and the 16 pixels of the circle [182] . . . . . . . . . . . B.1 Epipolar Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Résumé étendu Introduction Le sujet de cette thèse traite la problématique du suivi pour la localisation. Cette problématique représente un sujet stimulant dans des domaines d'application divers tels que la vision par ordinateur, la robotique et plus généralement l'Intelligence Artificielle (IA). Dans notre travail, nous nous intéressons au suivi dans les différents systèmes de navigation visuels et visuels-inertiels (VINS), pendant lequel les images acquises par une caméra mobile sont traitées et la position ainsi que les mouvements sont estimés afin de se localiser tout au long du déplacement du sysème. En effet, un VINS typique se compose d'une caméra monoculaire chargée de capter des images (informations visuelles) et une unité de mesure inertielle (IMU), à faible coût, capable d'intégrer les mouvements d'un mobile (accélération et vitesse angulaire) pour estimer son orientation (angles de roulis, de tangage, et de cap), sa position et sa vitesse linéaire. En effet, les deux types de capteurs produisent des données complémentaires. Malgré leur fréquence (cadence) moins rapide, les données visuelles peuvent générer des informations plus précises comparées aux données inertielles. En revanche, cette précision est très dépendante des conditions de texture, de luminosité de la scène concernée et des phénomènes d'occultations. L'intégration des données inertielles permet de remédier à ces problèmes et d'améliorer la précision. Cette combinaison est particulièrement efficace dans le domaine de la navigation et du suivi, grâce à la prise en compte des avantages conjoints de ces capteurs notamment en terme de précision, de coût et de réactivité.

1. 3 . 1

 31 Structure from Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.2 Motion Estimation Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.3 KLT Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Vision-based Localization Systems Overview . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 EKF MonoSLAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.2 Parallel Tracking And Mapping (PTAM) . . . . . . . . . . . . . . . . . . . . . . 1.4.3 ORB-SLAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

or

  Instructions Per Cycle (IPC) are ones of them. But these metrics are directly linked to the Instruction Set-Architecture (ISA) thus they are discriminant. Nowadays, specific metrics are promoted to characterize accelerator efficiency. Trillion of Floating Point Operation per Second (TFLOPS) for a given data width, Giga Pixel per second for video, etc. are also used. In 2016, NVIDIA introduces a new performance measurement, the Deep Learning Tera Operations Per Second (DL FLOPS) computing figure, it refers to the ability of processing an image classification (AlexNet) using deep Convolutional Neural Network (CNN).

FIGURE 3 . 3 :

 33 FIGURE 3.3: NVIDIA Tegra DRIVE Xavier SoC[START_REF] Nvidia | [END_REF]

FIGURE 3 . 8 :

 38 FIGURE 3.8: The smallest drone available on the market that uses VIO to estimate its own position announced by Qualcomm [203]. (Figure captured from [130])

FIGURE 3 .

 3 FIGURE 3.13: EMoVI-SLAM diagram (from [211])

FIGURE 3 . 16 :

 316 FIGURE 3.16: Loose coupling of PIRVS/GPS (PIRVS Loosely Coupled Sensor Fusion Example)[START_REF] Zhang | PIRVS: An Advanced Visual-Inertial SLAM System with Flexible Sensor Fusion and Hardware Co-Design[END_REF]

FIGURE 4 . 5 :

 45 FIGURE 4.5: The ETH Machine Hall environment (MHxx)[START_REF] Burri | The EuRoC Micro Aerial Vehicle Datasets[END_REF]

FIGURE 4 . 6 :

 46 FIGURE 4.6: The Vicon environment (Vxx)[START_REF] Burri | The EuRoC Micro Aerial Vehicle Datasets[END_REF]

FIGURE 4 . 7 :

 47 FIGURE 4.7: The Asctec Firefly hex-rotor helicopter for data acquisition[START_REF] Burri | The EuRoC Micro Aerial Vehicle Datasets[END_REF]

FIGURE 4 . 10 :

 410 FIGURE 4.10: The six DoF movement composition

1 .:

 1 the unit quaternion that represents the rotation of the reference frame B k to the word frame W; p W B k ∈ R 3 : the 3D position in the reference B k with respect to W; v W B k ∈ R 3 : the 3D speed of the reference B k with respect to W; b g and b a : the additive bias of the gyroscope and accelerometer, respectively;

Figures 5 .

 5 3 and 5.4 demonstrate the scale estimation impact (graph errors and overlaps) on the magnitude of the estimated poses.

1

  Overview of the real-time virtual blind cane system proposed in[START_REF] Inspex H | INSPEX-Integrated Smart Spatial Exploration System[END_REF] . . . . . . . . . . . . . . . . . . 2 PoIs matching with the real world and used for motion tracking (figure from [143]) . . . . . . . . . . . 3 Donecle aircraft inspection drone system [54]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Proposed System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Example of a common visual navigation systems classification . . . . . . . . . . . . . . . . . . . . . 1.2 Visual navigation systems classification discussed in this section . . . . . . . . . . . . . . . . . . . .

1.3 Visual Odometry steps overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Visual SLAM steps overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Optimization/Keyframe-based approach (T: camera pose, X: feature/landmark) . . . . . . . . . . . . . 1.6 Filtering-based approach (T: camera pose, X: feature/landmark) . . . . . . . . . . . . . . . . . . . . 1.7 Several classical types of classical points of interest.

et dédiée aux systèmes de navigation visuelle-inertielle. Par ailleurs, la trajectoire estimée et la vérité terrain ne sont pas représentées dans le même référentiel. Par conséquent, le calcul des erreurs entre les deux trajectoires n'est pas concluant. Pour cette rai-

  , ainsi que sur le temps d'exécution nécessaire pour les différents calculs. Ces évaluations commencent par décrire la partie suivi de la méthode ORB SLAM puisqu'il représente un point de départ pour notre proposition. ORB SLAM fournit un suivi robuste et précis sur des séquences faciles de la navigation. Or, dans les environnements difficiles, il rencontre des difficultés pour pouvoir assurer le suivi continu. En effet, la qualité du suivi peut se voir détériorée voire le système se perdre et par conséquent le suivi échoue sur le reste de la séquence. De plus, le suivi ORB SLAM impose une charge de calcul importante en détectant 1000 PoIs à chaque image et sur toutes les images de la séquence. En évaluant notre implémentation du suivi KLT-ORB Tracking nous remarquons une réduction importante en terme de temps de calcul. Cela revient à la réduction de la fréquence de détection des PoIs et de leur nombre. En effet, ce temps de calcul est estimé en moyenne à 32 f ps. Comme attendu, le KLT-ORB n'est fonctionnel que sur des séquences faciles et des distances courtes. Pour y remédier le suivi EKF VI Tracking est utilisé pour pouvoir se compléter avec le suivi KLT-ORB Tracking. Lors de l'évaluation du le suivi EKF VI Tracking, nous avons pu confirmer sa robustesse à des environnements difficiles (grandes rotations, vibrations causant le flou dans l'image,etc). Sa précision dans les séquances dites "faciles" est intéressante et est estimée en moyenne à 0.15 mètres alors que dans les cas difficiles cette précision se dégrade et l'erreur peut augmenter jusqu'à 0.69 mètres. Puisque le suivi EKF VI Tracking se base sur la fusion des données visuelles et inertielles, la charge de calcul et le temps d'execution sont affectés. Ainsi, la cadence de traitement est de 35 f ps en moyenne pour les environnements faciles et 37 f ps pour ceux difficiles. Afin de conserver des temps de traitement correctes nous avons proposé de réduire drastiquement le nombre de PoIs à savoir jusqu'à 5 nouveaux PoIs au maximum par image.

	son, un pré-traitement est effectué afin d'appliquer les transformations nécessaires pour pouvoir
	comparer l'estimation et la vérité terrain et pouvoir calculer les erreurs et quantifier la précision du
	système étudié, ce pré-traitement est nommée : "alignement de trajectoire". Dans cette thèse, nous
	avons adopté la méthodologie proposée dans [230], basée sur la technique Umeyama [212].
	Par la suite, notre système est évalué en tenant compte des différents aspects discutés aupara-
	vant. Nous avons commencé tout d'abord par évaluer le fil de traitement de suivi d'ORB SLAM,
	ensuite chaque composante du système proposé est testée indépendamment : KLT-ORB Tracking et
	EKF VI Tracking. Finalement la qualité de l'ensemble de la solution est quantifiée. Pour cela, nous
	avons choisi le jeu de données EuRoC comme base pour nos tests. Dans ces expérimentations nous
	nous sommes principalement basés sur l'exactitude et la robustesse du suivi dans différents types
	d'environnements

les mesures de cohérence des trajectoires : des

	recherche
	difficile dans le domaine du suivi visuel et visuel-inertiel. Des recherches futures peuvent être
	entreprises pour développer de meilleures stratégies d'estimation à l'échelle en utilisant des
	données visuelles et inertielles indépendamment du traitement de la pose fusionnée, même
	avec des capteurs étroitement couplés. Cela devrait permettre d'éviter les grandes variations
	d'échelle, ainsi que le bruit et les erreurs supplémentaires.
	études futures peuvent égale-
	ment élargir les mesures utilisées pour le module de contrôle d'exécution. Les résultats expéri-
	mentaux ont démontré l'importance du module de contrôle d'exécution et de ses métriques
	de contrôle utilisées pour alterner entre les deux approches de suivi proposées. Suite à ces
	résultats, des études futures peuvent également améliorer les métriques existantes. Ainsi, la
	précision du suivi pourrait donc être améliorée.
	-

-Amélioration

Intégration du suivi inertiel et visuel adaptatif en fonction du contexte, sous contrainte matérielle : le suivi inertiel visuel adaptatif en fonction du contexte de navigation peut

	être
	utilisé dans différents systèmes de navigation. En particulier, les MAVs, qui ont des limites en
	termes de complexité de calcul, de mémoire, d'espace et de ressources énergétiques. La méth-
	ode proposée a été développée pour être implémentée sur de tels systèmes embarqués tout
	en respectant leurs exigences et leurs contraintes. Par conséquent, les travaux futurs pourront
	se concentrer sur le portage de la solution sur une architecture SoC multiprocesseur (MPSoC)
	embarquée de la famille Xilinx, et/ou de type GPGPU de chez NVIDIA par exemple Jetson NX
	ou nano.

  Martyna Poreba, Erwan Piriou, Valérie Gouet-Brunet, and Maroun Ojail (Jan. 2019). "Multi-Modal Localization for Embedded Systems." Book chapter in Multi-Modal Scene Understanding, M. Ying Yang (eds.), Elsevier (80 pages: 199-278). 1 Imane Salhi, Erwan Piriou, Martyna Poreba, Maroun Ojail, and Valérie Gouet-Brunet (June 2018). "Suivi d'objet par capteurs visuels et inertiels sur systèmes embarqués." Conference paper in Conférence Française de Photogrammétrie et de Télédétection (CFPT) (8 pages) 2

  1.1 Systems Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Visual Odometry and SLAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Back-end Optimization Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1.2 Points of Interest Detection and Matching . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Handcrafted Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Learned Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 Points-of-Interest Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Visual Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table 1

 1 .2 summarizes the analysis of the experiments conducted in[START_REF] Lee | Recent advances in feature detectors and descriptors[END_REF] [112][START_REF] Işık | A comparative evaluation of well-known feature detectors and descriptors[END_REF]. The results of these works assert that the SIFT descriptor is the most robust to affine transformations, followed by the SURF, BRIEF, DAISY and ORB descriptors. Moreover, Fast, Harris, Harris-Laplace, and Harris-affine algorithms are not robust to this type of transformation. For rotations, the ORB, SIFT, SURF, Fast, Harris, Harris-Laplace, and Harris-affine are robust to such changes. But SIFT remains the most robust followed by ORB, SURF, Harris-affine, and DAISY.

	Algorithm	Invariance	Descriptor Size (byte)
		Detector
	Harris	Rotation, Translation	NP
	Harris-Laplace	Rotation, Translation, Scale	NP
	Harris-affine	Rotation, Translation, Scale, Affine transformation	NP
	Fast	Rotation, Translation, Scale	NP
		Descriptor
	DAISY	Affine transformation, Translation	13 [219]
	BRIEF	Affine transformation, Translation	32 [157]
		Detector & Descriptor
	SIFT	Rotation, Translation, Scale, Affine transformation	128 [157]
	SURF	Rotation, Translation, Scale, Affine transformation	64 [157]
	ORB	Rotation, Translation, Scale, Affine transformation	32 [157]

TABLE 1 .

 1

2: Performance characteristics: table of the different algorithms treated and the storage size of each descriptor vector. (NP: Non Pertinent)

  2.1 Systems Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Back-end Optimization Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Loose and Tight Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. used to compute the dynamic model of the system. The latter are considered as an input for the prediction step that predicts the vehicle motion. For visual measurements, features such as PoIs or others, are used for the measurement/observational model computation. This model is used to update the predictions in the update step. Filter-based VI SLAM[START_REF] Quan | Accurate Monocular Visual-Inertial SLAM Using a Map-Assisted EKF Approach[END_REF]

2.2 Overview of VINS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Optimization-based Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Filter-based Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.3 Optimization-based vs Filter-based Approaches . . . . . . . . . . . . . . . . . . 2.2.4 Challenges Associated to VINS and Development Trends . . . . . . . . . . . . . 2.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . are

  1 of chapter 1. This dataset gives different sequences with different difficulty levels of the navigation environment. Table2.2 illustrates the comparison of the estimated trajectory accuracy claimed by the relevant literature works. In the most of state-of-the-art works, the method accuracy is analyzed using the criterion of the Root Mean Square Error (RMSE) compared to the ground truth. According to evaluation reported in table 2.2, the most accurate method in most difficult cases is VI EKF SLAM, followed by VI ORB SLAM and Adaptive VI SLAM. However, these last two algorithms are the most vulnerable, since they fail in the most difficult conditions, such as V103 and V203 EuRoC sequences (section 4.1), which reduces their reliability and robustness. Other methods listed in table 2.2 are all robust in difficult cases but they are less accurate (high RMSE) compared to VI EKF SLAM even in easy cases (R-VIO is the least accurate while VINS Mono is the most accurate in the remaining algorithms). This reduces the benefit of their use if the application's purpose is to have good accuracy (small RMSE).

	Sequ.	OKVIS	ROVIO	VINS Mono	VI ORB SLAM	Methods VI EKF SLAM	Adaptive VI SLAM	Trifo-VIO	R-VIO
	V101	0.084	0.15	0.048	0.027	0.047	0.096	0.06	0.08
	V102	0.16	0.19	0.048	0.028	0.041	0.059	0.07	0.16
	V103	0.21	0.17	0.17	fail	0.081	0.068	0.13	0.14
	V201	0.13	0.28	0.054	0.032	0.029	0.066	0.065	0.22
	V202	0.17	0.60	0.10	0.041	0.044	0.073	0.12	0.31
	V203	0.26	0.18	0.15	0.074	0.058	fail	0.15	0.44
	MH03	0.26	0.40	0.07	0.87	0.046	0.039	0.24	0.36
	MH04	0.34	0.88	0.09	0.22	0.088	0.092	0.12	1.04
	MH05	0.44	1.26	0.14	0.082	0.064	0.086	0.18	0.86

TABLE 2 .

 2

2: Estimation accuracy (RMSE) of different approaches on the EuRoC MAV dataset

  3.1 Definition and Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Embedded System Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Design Challenges and Performance Metrics . . . . . . . . . . . . . . . . . . . .

3.2 Embedded Computing Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 System on Chip (SoC) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Intellectual Property (IP) Modules . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Optimized Implementation IC for Sensors Integration and Processing . . . . . 3.3 VI Navigation Embedded Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Specific Constraints to VIO/VI SLAM Implementation . . . . . . . . . . . . . . 3.3.2 Overview of Existing VIO/VI SLAM Systems . . . . . . . . . . . . . . . . . . . 3.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . order to carry out its mission, the CPS must supervise and control physical processes in real time.

  efficient hardware components and make easier data exchange with more generalist processing such as multiprocessor platform in the same device. Nowadays, in Xilinx latest product, a.k.a. Adaptative Compute Acceleration Platform (ACAP) called Versal device includes a dual-core ARM A72, dual-core Cortex ARM R5, DSP, Interfaces IPs settled next to a logic part. Furthermore, Intel company recently acquires Altera FPGA Manufacturer[START_REF]Intel. Intel SoCs FPGA[END_REF] and proposes the Arria 10 device.Others manufacturers such as Microsemi or Lattice Semiconductor only provides FPGA with logic cells. is primarily an artificial intelligence accelerator. It is a specialized circuit that implements all the control logic and arithmetic required to execute machine learning algorithms. It is often multi-core designs that run on predictive models such as Artificial Neural Networks (ANNs) or Random Forests (RFs), focusing on low-precision arithmetic, new data flow architectures or in-memory computing capacity. Actually, to perform learning and neural processing tasks, general purpose CPUs are the least suitable because they are not designed for massively parallelized execution. GPUs and DSPs are much better choices, but even then there is plenty of room for improvements.Since the term NPU has been recently introduced, manufacturors decline their own version.HiSilicon/Huawei created the term NPU while Apple publicly uses the term NE/neural engine.

	Neural Processing Unit (NPU)

Xilinx Zynq UltraScale+ MPSoC, this device offers 64-bit processor scalability, and combine realtime control with software and hardware drivers for graphics, video, waveshaping and package processing. Based on a common real-time processor and programmable logic platform, design high Other IP providers such as Cadence/Tensilica have chosen to call their processor a Neural Network DSP (Vision C5) and Imagination Technologies (2NX Series) uses the term Neural Network Accelerator (NNA).

  has 65M logic gates and occupies 144mm2 . It consists of 24 Tensilica DSP cores. It has around 8MB of SRAM, and an additional layer of 1GB of low-power DDR3 RAM. HPU offers a trillion of calculations per second. Claimed as low-power, it consumes 10W for handle gesture and environment sensing.A. Suleiman et al. present in[START_REF] Suleiman | Navion: A Fully Integrated Energy-Efficient Visual-Inertial Odometry Accelerator for Autonomous Navigation of Nano Drones[END_REF] a new compact and dedicated navigation chip, named Navion chip (figure 3.7). It is a customized IC targeting VIO and is intented to fit in a more compact system such as nano/micro aerial vehicle and VR/AR on portable devices. The chip uses inertial measurements and mono/stereo images to estimate the drone's trajectory and a 3D map of the environment. Several optimizations are performed to minimize chip power and footprint for low-power applications, while maintaining accuracy. Authors announced an average power budget of only 24mW while processing from 28 to 171 f ps. Die process is 65nm CMOS, the chip occupies 20mm 2 and presents a fully integrated VIO implementation.

is a custom multiprocessor (used as a coprocessor) called the Holographic Processing Unit, or HPU. It is in charge of the integration of all embedded sensors (IMU, custom ToF depth sensor, Head-tracking cameras, IR camera, etc.) through several interfaces: MIPI, CSI/DSI, I2C, PCI. This Taiwan Semiconductor Manufacturing Company (TSMC) has made a 28nm co-processor

TABLE 3 .

 3

1: State of the Art of competitive embedded VINS methods

TABLE 3 .

 3

2: Performance-Resources Trade-off Specification

[START_REF] Zhang | Visual-Inertial Odometry on Chip: An Algorithm-and-Hardware Co-design Approach[END_REF]

TABLE 4 . 1 :

 41 Navigation Environments Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Collection of odometry and SLAM datasets with individual data and sensor configuration details (Veh = Vehicule, Mob = Mobile) (1/2)

	Chapter 4
	Context Adaptive Visual-Inertial
	SLAM
	4.1

4.1.1 Main Datasets in Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 EuRoC MAV Dataset Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Context Adaptive Visual-Inertial SLAM Workflow . . . . . . . . . . . . . . . . . . . . . 4.2.1 System Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 Local Mapping & Loop Closure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Context Adaptive Visual-Inertial Tracking Components . . . . . . . . . . . . . . . . . . 4.3.1 Execution Control Module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.2 Visual KLT-ORB Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.3 EKF Visual-Inertial Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

TABLE 4 .

 4

2: Collection of odometry and SLAM datasets with individual data and sensor configuration details (Veh = Vehicule, Mob = Mobile) (2/2)

Table 4 .

 4 3 lists the main scene characteristics of each dataset sequence. According to the analysis shown in this table,

	Seq.	Level	Luminosity			Texture	Exposition
	MHxx	Easy Medium Difficult	Good without dark spots Good without dark spots Artificial lighting with	Many objects, Various forms	N/A N/A N/A
			many dark spots
		Easy	Good without dark spots	Many	N/A
	V1xx	Medium	Good without dark spots	geometric	Low
		Difficult	Backlit scenes		objects, Poor	Under/Over-exposed
						texture	pictures
		Easy	The best lighting without	Many	N/A
	V2xx		dark spots			geometric
		Medium	Artificial lighting with a few	objects, Good	N/A
			dark spots			texture (with
		Difficult	Artificial	lighting	with	some	A few under/over -
			many dark spots		exceptions)	exposed pictures

TABLE 4 . 3 :

 43 Scene characteristics in the different EuRoC sequences

TABLE 4 .

 4

5: EuRoC dataset environment categorization

  .27) Update The estimated state update is performed by stacking the m individual residual measurements r kl at time step k together to form a single residual vector r k = [r T k1 . . . r T kl . . . r T km ] T of 2m × 1 expressed as: r k = H k δX k + σ kl (4.28) Similarly, the measurement Jacobians are also combined into a single measurement matrix of 2m × n as: H k = [H T k1 . . . H T kl . . . H T km ] T . Afterwards, the full EKF state and covariance matrix are updated as follows:

TABLE 5 .

 5

5: KLT-ORB tracking execution time evaluation using EuRoC dataset (ms/ f rame)

  14, 5.15 and 5.16 illustrate a top view of different examples of the EKF VI tracking process results, in different dataset levels, compared to ground truth.

			ATE
		Mean	Median	RMSE
	V101	0.109	0.096	0.15
	V201	0.12	0.104	0.15
	V203	0.54	0.53	0.69
	MH01	0.098	0.071	0.17
	MH03	0.28	0.23	0.39
	MH04	0.29	0.30	0.65

TABLE 5 .

 5

6: EKF VI tracking evaluation using RPE and ATE (m)

Table 5 .

 5 [START_REF]Products processors[END_REF] shows the different execution times required for various EuRoC sequences. Moreover, the number of PoIs (min, max and mean) considered during the system computation are also given.

	Dataset	Prediction	Steps Measurement & Update	Global System	Number of PoIs Mean Max Min
	V101	2.6	25	35	9	25	6
	V201	2.4	24	35	12	24	8
	V202	2.5	24	36	10	25	5
	MH01	3.5	27	38	8	22	5
	MH03	4.5	29	40	9	24	7
	MH04	4.5	29	39	9	24	5

TABLE 5 .

 5 7: EKF VI tracking execution time evaluation (ms/ f rame) & PoIs number statistic

TABLE 5 .

 5 12: Comparison of PoIs management in the various methods analysed needed for the visual data computation, such as PoIs detection and optical flow computing or the Inverse Depth representation and state vector insertion processing, takes 24.31ms/ f rame in easy navigation environment and 24.07ms/ f rame in difficult navigation environment. This represents a gain of 2.93ms/ f rame minimum compared to ORB SLAM, notably thanks to reducing the number of detected PoIs. In addition, Pose Estimation & Tracking, which handles all necessary steps for the pose computation and tracking tasks, according to each method (either from visual measurements and following the motion model process, or by using the EKF VI process), had approximately similar execution time, compared to the EKF VIO, and a higher gain compared to ORB SLAM. This is estimated at 5.6ms/ f rame in easy sequences and 10.41ms/ f rame in difficult sequences.

	Dataset	Evaluated system	functions's mean execution time Keypoint Management Pose Estimation & Tracking
		ORB SLAM	27.24	12.72
		R-VIO	21.38	12.90
	Easy	proposed KLT-ORB VO proposed EKF VIO proposed method	24.00 24.60 24.31	6.20 8.00 7.12
		AVIO [168]	NA	19.9
		EKF VIO [177]	NA	12.00
		ORB SLAM	30.02 or lost (failure)	22.31 or lost (failure)
	Difficult	R-VIO proposed KLT-ORB VO proposed EKF VIO proposed method	23.86 lost (failure) 26.30 24.07	13.41 lost (failure) 10.11 11.40
		AVIO [168]	NA	failure
		EKF VIO [177]	NA	13.00

TABLE 5 .

 5 13: Execution time evaluation and comparison using EuRoC dataset (ms/ f rame) (NA:

	Not Available)

  ORB PoIs detection frequency and thus the number of ORB PoIs. The use of the KLT-ORB tracking saved about 20% in execution time over all of the dataset sequences.Before concluding, it is wise to introduce an analysis of the Context Adaptive VI SLAM, in order

	Dataset	proposed SLAM ATE	ORB SLAM ATE
	V101	0.09	0.08
	V201	0.08	0.10
	V203	0.62	0.65
	MH01	0.12	0.09
	MH03	0.18	0.14
	MH04	0.55	0.59

TABLE 5 .

 5 14: ORB SLAM and Context Adaptive VI SLAM evaluation illustrated using the RMSE of the ATE (m) in the trajectory. These deviations are more important, compared to the ORB SLAM, due to the map optimization that requires a sufficient number of PoIs. Whereas, in our proposal, this number is reduced by more than 50% of the required amount. Despite the above challenge, Context Adaptive VI SLAM provides results consistent with those of ORB SLAM. As shown in table 5.14, our proposal SLAM represents accuracy estimated in terms of the average RMSEs of the easy sequences by 0.09m and by 0.45m for difficult navigation sequences; this means a loss in RMSE accuracy estimated in average of 0.01m, approximately, compared to ORB SLAM, mainly in easy navigation environments.

https://www.sciencedirect.com/science/article/pii/B9780128173589000147

https://hal.archives-ouvertes.fr/hal-02338377

Acknowledgements

This PhD thesis work was carried out in the Laboratoire L3A of the Institut d'Intégration des Systèmes et des Technologies in the Commissariat à l'énergie atomique et aux énergies alternatives (CEA) in collaboration with the Laboratoire LaSTIG of the Institut National de l'Information Géographique et Forestière (IGN).

I would like to take this opportunity to thank these two institutions for their support and hosting.

I would like to express my sincere gratitude to Valérie Gouet-Brunet, for having accepted the di-

navigation systems development, and especially for our proposed embedded tracking system proposed in this thesis. In fact, FAST is a corner detection algorithm proposed in 2006 by Edward Rosten and Tom Drummond [START_REF] Rosten | Machine learning for high-speed corner detection[END_REF] [START_REF] Schmidt | An evaluation of image feature detectors and descriptors for robot navigation[END_REF]. It was developed to satisfy the requirements of real-time applications as the moving SLAM robot with limited computer resources. The FAST detector is based on the use of a circle around the central pixel (pixel studied) (figure A.3). FIGURE A.3: Illustration of the pixel to be tested (central pixel) and the 16 pixels of the circle [START_REF] Rosten | Machine learning for high-speed corner detection[END_REF] First of all, the pixel to be tested p must be selected and the 16 pixels circle around it must be taken into consideration. Then, a threshold t is set to perform the test. A pixel p is considered a PoI (corner) if there is a set of n pixels adjacent to the 16 pixel circle (the white dotted circle in figure A.3) brighter than I p + t ("bright" pixels) or darker than I pt ("dark" pixels) (I p : the intensity of the pixel to test). n is chosen equal to 12 pixels, this amounts to the use of the high-speed test which excludes a significant number of non-corners and consequently it only examines the four pixels 1, 5, 9 and 13 (1 and 9 are first examined if they are brighter or darker. If this is the case, pixels 5 and 3 are examined). So to have p as a corner, three of the four pixels must be brighter than I p+t or darker than I p-t . If not, then p can't be a corner. This procedure is applied to all pixels of the image. This algorithm is not safe from any limitations. First of all, the algorithm does not work very well when n < 12, because the number of detected PoI is very high. Then, the speed of the algorithm depends on the sequence in which the pixel test is performed [START_REF] Rosten | Machine learning for high-speed corner detection[END_REF]. In terms of invariances, the FAST detector is non-invariant in terms of scale and rotation changes. To ensure invariance to scale changes, Ethan Rublee and al. [START_REF] Rublee | ORB: An efficient alternative to SIFT or SURF[END_REF] proposed producing a scale pyramid and extracting the PoI from it. These are found by applying the FAST detector, then filtered by calling Harris [START_REF] Harris | A combined corner and edge detector[END_REF] at each scale.

However, for rotation invariance Ethan Rublee et al. [START_REF] Rublee | ORB: An efficient alternative to SIFT or SURF[END_REF] proposed to use the intensity centroid.

Its value allows assigning an orientation to a corner by assuming that the intensity of the corner is shifted from its center. Following these improvements, the new detection algorithm on which the ORB is based is called oFAST (oriented FAST) [START_REF] Rublee | ORB: An efficient alternative to SIFT or SURF[END_REF]. The latter is known for its lightness in terms of computational complexity as well as for its rapid computation time for PoI detection as in [START_REF] Tareen | A comparative analysis of SIFT, SURF, KAZE, AKAZE, ORB, and BRISK[END_REF] and PoI description and matching which is illustrated in [START_REF] Işık | A comparative evaluation of well-known feature detectors and descriptors[END_REF] and explained in follow, especially when compared to the other algorithms mentioned above.

PoI Description

For description ORB is based on BRIEF descriptor. In fact BRIEF [START_REF] Calonder | BRIEF: Binary robust independent elementary features[END_REF] is a binary PoI descriptor. It is based on fixing a set of N point pairs and calculating the difference between their intensity values two by two to allow them to be compared using the following test τ(p, x, y):

where p(x) is the intensity over the entire pixel x(u, v) T .

Appendix A. Detector/Descriptors Examples

The BRIEF descriptor vector size is N 8 (N = 128, 256 ou 512). This descriptor is known for its rapid implementation. As a result, it is effective in real-time applications. However, it fails to have the invariance to rotations. The rotated BRIEF (rBRIEF) is the extension of the BRIEF descriptor that solves this problem of rotation changes. It was proposed as part of the ORB detection/description algorithm. Thus, it is one of the most appropriate and widely used description algorithms for embedded mobile navigation systems.

Appendix B. Epipolar Geometry

This matrix is useful because it allows to calculate the cameras' projection P i matrices. So, in the case of estimating the metric displacement between two images (two cameras), it is possible to use the essential matrix directly without computing the fundamental matrix. In addition, if the cameras are calibrated, it is possible to estimate the essential matrix from the fundamental matrix.

B.1.2 Essential Matrix

In the same way as the fundamental matrix, the essential matrix links two homologous points d1 and d2 , expressed this time in the main plane of each camera (and not in the image frame): di = K -1 i qi is the camera's calibration matrix c i .

The essential matrix provides the following bilinear relationship:

where E ∼ K T 2 FK 1 is the essential matrix. It has only five degrees of freedom that correspond to the extrinsic parameters of inter-image displacement (six degrees of freedom less one due to the uncertainty of the scale factor).

Appendix C RANSAC C.1 Filtering for robust estimation: RANSAC

In practice, when matching, some correspondences are very different from reality and are considered as erroneous or outliers. These data should not be taken into account for the displacement estimate, which is later explained, as they would significantly reduce this estimate's quality. To solve this problem, different approaches have been proposed such as M-estimators [START_REF] Peter | Robust Statistics[END_REF] and RANdom SAmple Consensus (RANSAC) [START_REF] Fischler | Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography[END_REF]. Here we are interested in the latter method.

The RANSAC algorithm proposed by Fischler and Bolles [START_REF] Fischler | Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography[END_REF] is a general parameter estimation approach that is developed to deal with a large proportion of outliers in the input data. Unlike other common robust estimation techniques such as M-estimators and least-median squares that have been adopted by the computer vision community from the statistical literature, RANSAC was developed by the computer vision community. It is a paradigm based on a simple principle: searching for the largest set of data compatible with a parametric model. Moreover, in order to ensure that the system to be solved is sufficiently constrained, a minimum number of data is used to calculate the model parameters, instead of using all available data, and this is depending on the number of degrees of freedom of the model. Lastly, the research performed is not exhaustive but is often done on a sampling approach in order to reduce its computational cost.

RANSAC is an iterative algorithm based on two steps: a) selecting a minimum random sample of matches to estimate the parameters of the geometric model, and b) constructing a subset of matches that is compatible with that geometry. This set is called a consensus set. The consistency is measured by the size of the consensus set (when the original version of RANSAC is performed) or by using a more sophisticated model fit measure (M SAC, M LESAC). The search is stopped when a consensus set of sufficient size is found, or when a sufficient number of runs have been performed to ensure the reliability of the solution.

N represents the number of RANSAC iterations, it is selected to be sufficiently high to ensure that the probability p (usually set at 0.99) of at least one of the random sample sets not having an outlier. Let u be the probability that a selected data point is an outlier and v = 1u the probability of observing an outlier. N iterations of the minimum number of points indicated m are required, when 1p = (1u m ) N (C.1) then after doing some computational manipulations, N is