Keywords: détection d'anomalies, apprentissage profond, modèle génératif, application au transport anomaly detection, deep learning, generative model, transportation application anomaly detection, deep learning, generative model, transportation application

Today, automatic solving transportation problem becomes active subject. In our PhD project, we aim to address a specific challenge in this domain: anomaly detection and tracking. Our ultimate goal is constructing a flexible and effective framework producing high performance on various public datasets. The context of our research is applying and improving previous successful approaches to achieve better results. We deal with two scenarios leading to two methods mentioned in following parts: (1) vehicles and road users segmentation and tracking by future predictions using classical hand-crafted generative methods based on optical flow estimation; (2) anomaly detection by future predictions using multi-channels deep generative frameworks and supervised learning.

Our first research is evaluating the performance of the classical hand-crafted generative approach in future prediction and its capability for improving segmentation and tracking. Recently, there existed various strong deep learning detectors e.g. Mask R-CNN lead to an effective approach for tracking problem: tracking-by-detection. This very fast type of tracker considers only the Intersection-Over-Union (IOU) between bounding boxes to match objects without any other visual information. In contrast, the lack of visual information of IOU tracker combined with the failure detections of CNNs detectors create fragmented trajectories. We propose an enhanced tracker based on tracking by-detection and optical flow estimation in vehicle tracking scenarios. Our solution generates new detections or segmentations based on translating backward and forward results of CNNs detectors by optical flow vectors. This task can fill in the gaps of trajectories. The qualitative results show that our solution achieved stable performance with different types of flow estimation methods. Then we match generated results with fragmented trajectories by SURF features. DAVIS dataset is used for evaluating the best way to generate new detections. Finally, the entire process is tested on DETRAC dataset. The qualitative results show that our methods significantly improve the fragmented trajectories. For future work, we plan to apply CGANs streams of second work for the first task to propose a new competitive process of future prediction for segmentation and tracking.

Despite the moderate success of the first work, there is significant limitations of classical approaches to deal with our main task: anomaly detection. The lower frequency of abnormal events leads to an unbalanced scenario and the features of abnormal events usually do not follow any spatial or temporal relationship. It is also difficult to pre-define the structure or class of abnormal events. Facing to those challenge, most of state-of-the-art (SOTA) anomaly detection methods are based on apparent motion and appearance reconstruction networks and use error estimation between generated and real information as detection features. These approaches achieve promising results by only using normal samples for training steps. In this thesis, our contributions are two-fold. On the one hand, we propose a flexible multichannel framework to generate multi-type frame-level features. On the other hand, we study how it is possible to improve the detection performance by supervised learning. The multi-channel framework is based on four Conditional GANs (CGANs) taking various types of appearance and motion information as input and producing prediction information as output. These CGANs provide a better feature space to represent the distinction between normal and abnormal events. Then, the difference between those generative and ground-truth pieces of information is encoded by Peak Signal-to Noise Ratio (PSNR). We propose to classify those features in a classical supervised scenario by building a small training set with some abnormal samples of the original test set of the dataset. The binary Support Vector Machine (SVM) is applied for frame-level anomaly detection. Finally, we use Mask R-CNN as a detector to perform object-centric anomaly localization. Our solution is largely evaluated on Avenue, Ped1, Ped2 and ShanghaiTech datasets. Our experiment results demonstrate that PSNR features combined with supervised SVM are better than error maps computed by previous methods. We achieve SOTA performance for frame-level AUC on Avenue, Ped1 and ShanghaiTech. Especially, for the most challenging Shanghaitech dataset, a supervised training model outperforms up to 9% the SOTA on unsupervised strategy. Furthermore, we keep in progress several promising ways: building a new dataset for semi-supervised anomaly detection containing both normal and abnormal samples in its training set and applying one-class SVM to propose an end-to-end framework.

Résumé xiii ANOMALY DETECTION AND OBJECT TRACKING BY FUTURE PREDICTION USING GENERA-TIVE METHODS FOR TRANSPORTATION Résumé

Actuellement, le traitement automatiquement des problèmes de transport devient un sujet actif. Dans le cadre de ce travail, nous visons à relever un défi spécifique dans ce domaine : la détection et le suivi des anomalies. Notre objectif est de construire un système flexible et efficace produisant des performances élevées sur diverses bases de données publiques. Le contexte de notre recherche est l'amélioration des approches précédentes pour obtenir de meilleurs résultats. Nous traitons deux scénarios conduisant à deux méthodes mentionnées dans les parties suivantes : (1) la segmentation et le suivi des véhicules et des piétons par des prédictions utilisant des méthodes génératives classiques basées sur des descripteurs a priori (hand crafted) et sur l'estimation des flux optiques ; (2) la détection des anomalies par des prédictions utilisant des systèmes génératifs multicanaux profonds et l'apprentissage supervisé. Notre première recherche vise à l'évaluation des performances de l'approche générative classique pour les prévisions et la détermination des ses capacités à améliorer la segmentation et le suivi d'objets. Récemment, divers détecteurs d'apprentissage profond ont été proposés e.g. Mask R-CNN qui permettent une approche efficace du problème de suivi : le suivi par détection. A l'exception de tout autre information visuelle, ce type de tracker rapide ne prend en compte que l'intersection-sur-union (IOU) entre les boîtes de délimitation pour apparier les objets. Ainsi, l'absence d'informations visuelles du tracker IOU combinée avec les possibles défaillances des détecteurs créent des trajectoires fragmentées. Nous proposons alors un tracker amélioré basé sur la détection par suivi et sur l'estimation du flux optique. Notre solution génère de nouvelles détections ou segmentations basées sur une translation temporelle en avant et en arrière des résultats des détecteurs CNNs en utilisant les vecteurs de flot optique. Cette étape permet de combler une première partie des lacunes des trajectoires. Les résultats qualitatifs montrent alors que notre solution a obtenu des performances stables avec différentes méthodes d'estimation du flot optique. Les lacunes résiduelles au sein des trajectoires sont traitées en utilisant des caractéristiques SURF. La base de données DAVIS est utilisée pour évaluer la meilleure façon de générer de nouvelles détections. Enfin, le tracker résultant est testé sur la base de données DETRAC. Les résultats qualitatifs montrent que notre approche diminue très significativement la fragmentation des trajectoires. Pour les travaux futurs associés à ce tracker, nous prévoyons d'appliquer les réseaux CGAN développés dans le cadre de la seconde partie de notre travail afin de proposer un système compétitif de suivi d'objet basé prévision. Malgré les résultats tangibles de cette première approche, les méthodes classiques présentent des limitations importantes concernant la détection d'anomalies qui est l'un de nos objectifs principaux. La fréquence plus faible des événements anormaux donne un scénario déséquilibré et leurs caractéristiques ne suivent généralement aucune relation spatiale ou temporelle. Face à ces défis, la plupart des méthodes de l'état-del'art se basent sur des réseaux prédictifs et utilisent les erreurs entre informations générées et réelles comme caractéristiques de [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF] 2.17 Object-centric framework for anomaly detection proposed by Ionescu et al. [START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF].

Introduction

This chapter introduces a general overview about our research project. First, we present the scientific context and scenario of our works. This part shows the definition of abnormality and how we consider the tasks of anomaly detection. Then we present the objective of this project in term of scientific researches and its application in real-world context. In order to achieve those goals, various approaches will be largely taken into account for both classical hand-crafted features methods and modern deep learning approaches. On the one hand, we analyse and compare them to show the advantage of recent deep learning models that inspired us to apply. On the other hand, we find the limitations of those state of the art models. Our solutions to improve those drawbacks will be introduced in the next part. Then we highlight our contributions along this project. Finally, we end this chapter by presenting the structure of our PhD thesis. 1

Context and Scenario

In this work, we study the problem of discovering and modelling the interactions between users and transport infrastructures by the analysis of video surveillance streams. This challenging work leads to the ability to recognise particular and abnormal activities in traffic. That can facilitate various applications targeted to improve the safety of vulnerable road users, the security of transport networks or supporting the autonomous car driving. We cast our works as a visual action recognition and understanding problem and we apply different solutions that have been proved being effective in many computer vision tasks. We plan to benefit from the promising results of deep learning in detection and segmentation tasks to detect, segment and track all the elements participating in traffic networks. Thus, we entirely extract all their information for the purpose of constructing a new model for unlearned behaviours or interactions.

Our works address the problem of analysing the information obtained from video surveillance streams. This is an interesting problem because behaviours modelling of individuals and their interactions thanks to the analysis of audio and video streams acquired from surveillance systems is a very active research topic in the scientific community. It is directly related to "Big data" problem which is a common aspect to many application areas such as monitoring the health of elderly people while enforcing their safety and security. Within the area of this research project, we focus on the transportation field. The objects we are searching for are all the ones and all the things that participate to the traffic network : pedestrians, bicyclists, motorbikes, trucks, traffic lights and signs, etc. We particularly concentrate on vulnerable road users (VRU) and their interactions with other actors inside the transportation network.

Actually, there are a lot of successful researches in transportation domain [START_REF] Yang | Exploit all the layers : Fast and accurate cnn object detector with scale dependent pooling and cascaded rejection classifiers[END_REF][START_REF] Chabot | Deep manta : A coarseto-fine many-task network for joint 2d and 3d vehicle analysis from monocular image[END_REF][START_REF] He | Mask r-cnn[END_REF][START_REF] Badrinarayanan | Segnet : A deep convolutional encoderdecoder architecture for image segmentation[END_REF][START_REF] Behl | Bounding boxes, segmentations and object coordinates : How important is recognition for 3d scene flow estimation in autonomous driving scenarios[END_REF][START_REF] Chen | Encoder-decoder with atrous separable convolution for semantic image segmentation[END_REF][START_REF] Xiang | Learning to track : Online multi-object tracking by decision making[END_REF], especially for autonomous driving cars. But they are almost only addressing the first step of video information analysis for answering the question about detection and localisation of all objects inside a video signal. In this thesis, we want to continue the next step that is using the information extracted from the first step to model the interactions between them. In a naive way, it can be considered as a visual action recognition and understanding task. But in this case, the set of actions needed to be classified is very challenging due to the very occlusive scenarios attached to traffic networks. Additionally, the occurrence of abnormal and randomly performed actions also raises the difficulty of our context because those actions are totally unlearned by traditional methods.

In details, we particularly work with two scenarios : (1) vehicles and road users segmentation and tracking by future predictions using classical hand-crated generative methods based on optical flow estimation ; (2) anomaly detection by future predictions using multi-channels deep generative frameworks and supervised learning. Both research scenarios are evaluated on various public benchmarks in the off-line working mode. To evaluate the performances of our approaches, Version intermédiaire en date du 26 novembre 2022 1.2. Objective we consider both qualitative and quantitative measurements. The experimental setup for these two scenarios as well as our proposed methods for each one is precisely presented in the next chapters.

Objective

In term of scientific objectives, we aim to address a specific challenge in transportation domain : anomaly detection and tracking. Our ultimate goal is constructing a flexible and effective framework producing high performances on various public datasets. As a context of this research project, we focus on applying and improving previous successful approaches to achieve better results rather than proposing brand-new methods or innovative network architectures.

For the first scenario, our purpose is to evaluate the capability of classical hand-crafted generative methods for improving segmentation and tracking tasks as well as its potential to deal with further challenging e.g. future prediction and anomaly detection. Naturally, hand-crafted methods without training a complex deep learning models are always the first priority for each system due to its simple pipeline. Supposing that classical hand-crafted methods can treat well almost all tasks, we benefit from the reduction of time consumption and, particularly, we also avoid the preparation of a large dataset for deep learning. In contrast, it is necessary to take the deep learning approaches into consideration when classical ones can not provide sufficient performances to us.

For the second scenario, we cast this challenging problem as our main task. Our purpose is to build a flexible and strong deep learning model for anomaly detection at frame-level. Then we develop a suitable pipeline for abnormality localisation at object-centric level. On the one hand, the most important request for our work is obtaining high performances. We expect our models to surpass state-of-the-art performance in various public datasets. On the other hand, the flexibility of our approach is based on the fact we would like to build a model adapted for various types of input information (i.e. RGB, grayscale, optical flow). As a consequence, it could be easily extended or lightened for further developments. We also demand our model to create distinct features for representing the characteristics of abnormal activities that can be comfortably fed into various classification models.

Definition of Anomaly Detection

As previously defined in [START_REF] Saligrama | Video anomaly identification[END_REF] and retrieved in [START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF], we define an abnormal event as "the occurrence of unusual appearance or motion attributes or the occurrence of usual appearance or motion attributes in unusual locations or times". It clearly appears that the description of an abnormal event can be different from one scene to another because of the environment context Version intermédiaire en date du 26 novembre 2022 or because of the intrinsic and extrinsic camera parameters (location, pose, focal length etc.).

Moreover, different types of anomalous events may occur. An appearance based event can be illustrated by a car or a pedestrian moving in a forbidden area. Short or long-term motion event can be respectively illustrated by a pedestrian jumping over a barrier or by a bicycle zigzagging around pedestrians. For these reasons, we easily imagine that it will be difficult to compute a unique model with high performance for all the situations and for all the environments. Abnormal events occur (fortunately) rarely in the real-life, and it is unrealistic to video caption all anomalous events we could encounter. Even if it is always possible to come across anomalous events in video surveillance, huge resources are required (in terms of time and human work load) to manually detect and annotate each one. The common approach used in most of the state-of-the-art methods is to model the normal activities because they are easy to collect from the video camera. Moreover, the annotation task is reduced to making sure that the "normal events" dataset is not corrupted by anomalous events. We follow this assumption, and we propose to analyse the gain of adding a supervised learning step regarding abnormal events, the final objective being to reach the trade-off between performance and required resources linked to dataset annotation.

Classical approaches

Generally, there are relatively large differences between classical approaches and modern ones in terms of considering the anomaly detection problem. Before the existing of specific datasets for anomaly detection mentioned in some works [START_REF] Lu | Abnormal event detection at 150 fps in matlab[END_REF][START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF][START_REF] Mahadevan | Anomaly detection in crowded scenes[END_REF], most of the researches considered abnormal actions as a particular set of normal activities. All the public datasets [START_REF] Kuehne | HMDB : a large video database for human motion recognition[END_REF][START_REF] Soomro | Ucf101 : A dataset of 101 human actions classes from videos in the wild[END_REF] during this period was constructed for the classical action recognition problems by combining all types of actions without highlighting some abnormalities. By this way, the abnormality was similarly considered as normality. To solve this classical problem in computer vision, we can apply various techniques for the discriminative task such as : image classification, video detection, segmentation and tracking, etc. Following classical hand-crafted methods [START_REF] Wang | Dense trajectories and motion boundary descriptors for action recognition[END_REF][START_REF] Wang | Action recognition with improved trajectories[END_REF][START_REF] Oneata | Efficient action localization with approximately normalized fisher vectors[END_REF][START_REF] Harchaoui | Temporal localization of actions with actoms[END_REF], first, activities are densely tracked to create smooth and continuous trajectories. Then, points of interest are detected around objects and activities. Various types of feature (SIFT [START_REF] David | Distinctive image features from scale-invariant keypoints[END_REF], HOG [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF], HOF [START_REF] Brox | Large displacement optical flow : Descriptor matching in variational motion estimation[END_REF], MBH [START_REF] Wang | Dense trajectories and motion boundary descriptors for action recognition[END_REF], etc.) are extracted at the points of interest's locations along trajectories to help representing the distinction between activities. Finally, they construct suitable inference models to classify the corresponding features. Applying this technique to the transportation domain leads our problem to be simply treated as a detection and tracking task. Intuitively, if we can detect all the users and vehicles involved in a scenario, then we can smoothly and continuously track them frame by frame. In turn, we can extract features around those objects such as velocity, motion vector, etc to model the interactions between them. The first part of our research is for the purpose of doing this work.

Version intermédiaire en date du 26 novembre 2022

Classical approaches

Thanks to the recent successful CNNs model for object detection and segmentation [START_REF] Shaoqing Ren | Faster r-cnn : Towards real-time object detection with region proposal networks[END_REF][START_REF] Wang | Evolving boxes for fast vehicle detection[END_REF][START_REF] He | Mask r-cnn[END_REF][START_REF] Redmon | Yolo9000 : Better, faster, stronger[END_REF] we can almost retrieve the full spatial information about all users and objects involved in a traffic networks related scene captured by a camera. Those strong CNN models outperform all the previous traditional methods that use hand-crafted features in combination with learning representation. Among them, Mask R-CNN [START_REF] He | Mask r-cnn[END_REF] is our best candidate because we can benefit from various types of information (object class, localisation and segmentation at instance level) using only one network. We observe that object tracking is an important proxy task towards action recognition. Most of the effective methods for normal daily action recognition [START_REF] Wang | Action recognition with improved trajectories[END_REF][START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF][START_REF] Wang | Action recognition with trajectory-pooled deepconvolutional descriptors[END_REF] starts with a tracking step for obtaining objects' trajectories. Intuitively, the more information we get from objects, the more effective solution we have for the tracking step. Thanks to the strong performances of Mask R-CNN, we have a simple and effective approach for object tracking : tracking-by-detection. This type of tracker is an active topic in object tracking and achieves many successful works from early researches [START_REF] Breitenstein | Robust trackingby-detection using a detector confidence particle filter[END_REF][START_REF] Andriluka | People-tracking-by-detection and people-detectionby-tracking[END_REF][START_REF] João | Exploiting the circulant structure of tracking-by-detection with kernels[END_REF]. Tracking-by-detection method first applies an object detector to each video frame then associates these detections to tracks. One of the most representative and popular tracker of that kind is IOU Tracker [START_REF] Bochinski | High-speed tracking-by-detection without using image information[END_REF]. It is the core tracker of winning solutions for multi-objects tracking challenge in AVSS 2018 [START_REF] Lyu | Ua-detrac 2017 : Report of avss2017 & iwt4s challenge on advanced traffic monitoring[END_REF].

Limitations

Instead of using visual information to match the objects locations, IOU Tracker takes into account the Intersection Over Union (IOU) between bounding boxes to associate them together. This feature first makes the performance of the tracker completely dependent from the performance of the detector itself. Second, the lack of visual information can lead to confusion between objects in overlapping cases. On the other hand, Mask R-CNN shows its drawback in the case of false negative error on usual objects (see Figure 1.1) although it yields strong performances on COCO dataset [START_REF] Lin | Microsoft COCO : common objects in context[END_REF] and particularly on transportation related domain datasets such as the Cityscapes dataset [START_REF] Cordts | The cityscapes dataset for semantic urban scene understanding[END_REF] (see Obviously, confusing detections between object's classes from Mask R-CNN can also produce wrong tracks, for instance when creating new tracking processes instead of maintaining existing ones.

IOU = Area of Intersection

Solution and contributions

Our work aims to build an improved tracker that can limit the drawback of fragmented trajectories by Mask R-CNN detector on IOU Tracker while keeping the advantage of being fast as a tracker. The first step is filling in the gaps of fragmented detections using a generative approach. The next step consists in combining the generated results with the current results from the detectors. Then, by applying the idea of IOU trackers, IOUs between the bounding boxes are used to eliminate the overlapping results. The final step is performed by associating the trajectories with SURF feature [START_REF] Bay | Speeded-up robust features (surf)[END_REF], which is a high-performance feature for image matching.

Our contributions are the following :

-Introduction of a fast and efficient generative approach using optical flow for filling in the discontinuities of object tracking. Our solution is stable with different types of flow estimation.

-An enhanced tracker integrating Mask R-CNN based IOU tracker with visual information.

Version intermédiaire en date du 26 novembre 2022 1.5. Abnormality centric approaches

Abnormality centric approaches

Naturally, in human behavior analysis, we might consider anomaly detection as an action recognition problem. But this classical point of view leads us to an unbalanced situation where the number of samples for each class is significantly different. Beside, it is difficult to predefine the structure of abnormal events because there is usually not any spatial and temporal relations between those events. Hence, we should tackle this challenge in a specific way. Generally, from the first successful works until now, they proposed three solutions : one-class classification based [START_REF] Wang | Histograms of optical flow orientation for visual abnormal events detection[END_REF][START_REF] Wang | Gods : Generalized one-class discriminative subspaces for anomaly detection[END_REF], changing detection based [START_REF] Del Giorno | A discriminative framework for anomaly detection in large videos[END_REF][START_REF] Hasan | Learning temporal regularity in video sequences[END_REF][START_REF] Radu | Unmasking the abnormal events in video[END_REF] and future prediction based [START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF].

One-class learning first constructs the representation for events then fit a model to data for which annotations are available only for a single class. In anomaly detection, those are labels for abnormal samples. This solution is only appropriated for binary classification, and it has limitations when we need further information as type and localisation. Changing detection is a classical way where each event is compared with its neighbours to find the most different ones. By this way, we could get trouble when abnormal event always or never happens in a sequence. The future prediction based techniques casts abnormal events as unpredicted events.

A generative model to produce future information from previous frames is computed and a model is trained from normal frames and noisy ones ; usually, noisy frames are more blurred than the ground truth.

Limitations and Solutions

Most of recently successful researches [START_REF] Hasan | Learning temporal regularity in video sequences[END_REF][START_REF] Luo | Remembering history with convolutional lstm for anomaly detection[END_REF][START_REF] Luo | A revisit of sparse coding based anomaly detection in stacked rnn framework[END_REF][START_REF] Radu | Unmasking the abnormal events in video[END_REF][START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF][START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF] have tackled this challenge in specific unsupervised ways. They only use normal samples from training set to generate the standards for normal actions then try to enlarge the deviation between abnormal samples in test set and theirs standards. Using unsupervised strategy naturally follows the structure of popular benchmarks datasets containing only normal samples that do not require annotation task.

The state of the art proposes two approaches for defining the anomalous feature that are respectively based on changing detection [START_REF] Del Giorno | A discriminative framework for anomaly detection in large videos[END_REF][START_REF] Hasan | Learning temporal regularity in video sequences[END_REF][START_REF] Radu | Unmasking the abnormal events in video[END_REF] and reconstruction/prediction errors [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF][START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF]. The first solution is a natural approach where each event is compared with its neighbours to find the most different ones. The weakness of this solution appears when abnormal event always or never happens in a sequence. Besides, this solution is mainly appropriated for binary classification, and it has the limitations when we need further information as type and localisation. The second approach deals with these limitations and achieves state-of-the-art performance. Technically, they generate the future prediction or reconstruct current information for each action by GANs models and Convolutional Auto Encoder-decoder (CAE) models. Intuitively, the models trained by only normal samples from the training set will reconstruct better images Version intermédiaire en date du 26 novembre 2022 for normal frames than abnormal ones in test set. Inspired by the promising results following this approach and the benefits from the rise of CGANs and CAE models, we continue to extend the architecture of reconstructing models by integrating four pix-to-pix Image Translation CGAN models [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] as four parallel channels. This 4-channels framework processes both appearance and motion information in grayscale or color format.

In term of features extractions and descriptors encoding, previous anomaly detection frameworks proposed various reasoning approaches related to the features types : at object-centric or local level [START_REF] Mahadevan | Anomaly detection in crowded scenes[END_REF][START_REF] Lu | Abnormal event detection at 150 fps in matlab[END_REF][START_REF] Luo | Remembering history with convolutional lstm for anomaly detection[END_REF][START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF], at frame level [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF][START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF] or at both [START_REF] Hasan | Learning temporal regularity in video sequences[END_REF]. On the one hand, some methods encode their features to calculate abnormality scores ; the decision is done thanks to a threshold or a peak estimation [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF][START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF]. In fact, by integrating all features into one score value, spatial and temporal information are discarded and a simple comparing decision model is learned. On the other hand, some keep their large scale features [START_REF] Radu | Unmasking the abnormal events in video[END_REF][START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF] to train a classifier. If these solutions maintain the wealth of information of the features, the learning process is more complex and takes a long time to converge. To solve this problem, we propose to integrate the complete wealth of the features produced by each of the channels of our framework into a 10-dimensions vector by PSNR technique. The implementation details will be introduced in subsection 4.1.5. Our descriptors bring specific information of each channel but are light enough for fast learning models.

Contributions

In this work we first have modified the state-of-the-art unsupervised GAN-based abnormality detection approach by increasing the feature space. Second, we have added a supervised final step to improve the detection rate. In summary, our contributions are as follows :

-We introduce a flexible and powerful framework containing a multi-channel CGANs (4 streams with 9 output channels in our case) to generate multi-type future appearance and motion information. Our architecture considers more consecutive frames forward translation from t to (t+2) than previous methods with only encode-decode reconstruction [START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF] or translation from t to (t + 1) [START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF]. The number of channels can be freely inserted or removed for multiple purposes.

-We experimentally prove the effectiveness of PSNR in image comparison task. Based on PSNR method, a useful descriptor of output from our generative framework and ground-truth is proposed. The size of the descriptor is also flexible, as well as the number of channels and quite small to adapt to fast classifiers.

-We demonstrate the improvement we get on anomaly detection by adding a supervised stage exploiting the feature extractor proposed by our unsupervised architecture. We add a SVM stage that we train on a subset of abnormal samples from the test dataset.

We achieve at least competitive performances on all benchmarks : Avenue, Ped1, Ped2 in term of frame-level AUC and a huge improvement on the most challenging datasets such Version intermédiaire en date du 26 novembre 2022 1.6. Manuscript Organization as Shanghaitech.

Manuscript Organization

Our thesis manuscript is organized by following a classical structure. It contains five chapters : Introduction, Related Works, Proposed Methods, Experiments and Conclusion.

After the first part introducing general information of our thesis, we present our results of searching for related work. We begin with the background of recent state-of-the-art deep learning models for all classical tasks of computer vision applying in transportation domain. The basis of the convolutional neural network following with various effective models for action recognitions, object detection, segmentations, optical flow estimation is introduced in both supervised and unsupervised contexts. Besides the presentation of existing discriminative models, we focus on the development of recent deep learning generative models to search for their capability of solving our problems. To improve segmentation and tracking framework, we mention two state-of-theart models : Mask R-CNN and IOU object tracker as baseline methods. Then, we propose some useful add-on modifications to minimise its drawbacks. Then we continue with state-of-the-art solutions for our main task : anomaly detection. As explained before, we follow recent successful generative models for future prediction. We highlight their strengths and limitations. We end this chapter with a brief conclusion on the state-of-the-art models and the solutions that we apply.

The third chapter presents the essentials of our first work in detail. In the proposed methods section, we sequentially introduce two frameworks corresponding to two stages : (i) Generating object segmentation by Mask R-CNN and Optical flow estimation and (ii) improving IOU Tracker with generated information and SURF features. We begin with theoretical presentation of Mask R-CNN [START_REF] He | Mask r-cnn[END_REF], Optical flow estimation by LDOF [START_REF] Brox | Large displacement optical flow : Descriptor matching in variational motion estimation[END_REF], Full Flow [START_REF] Chen | Full flow : Optical flow estimation by global optimization over regular grids[END_REF] and SURF features [START_REF] Bay | Speeded-up robust features (surf)[END_REF]. Then we introduce our methods to generate future detection and segmentation. We apply this classical hand-crafted generative information to improve detection and tracking. In the experiments section, we present our implementation of generating object segmentation and the quantitative results on DAVIS dataset we get using it. Then we show qualitative improvements of enhanced trackers with generated information and optical flow on UA-DETRAC dataset. We end this chapter with some analysis about strengths and limitations of our first work.

The fourth chapter introduces the second research : anomaly detection. This is our main task and our most important achievements are included. We begin this chapter with the basis of GAN [START_REF] Goodfellow | Generative adversarial nets[END_REF], Conditional-GAN [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] and U-Net models [START_REF] Ronneberger | U-net : Convolutional networks for biomedical image segmentation[END_REF] . Then, we precisely describe our multi-channels network architecture to learn abnormality centric features as well as supervised inference models by SVM binary classifier. We end the first section with some conclusion and analysis on strengths and limitations of our proposed methods. In the second section, we begin by Version intermédiaire en date du 26 novembre 2022 briefly introducing 4 of the public datasets and some evaluation metrics for anomaly detection.

Then, we provide some tutorials of our network implementations in a step by step manner.

We precisely show our results in terms of both quantitative and qualitative performances to illustrate the effectiveness of our approach for all of 4 datasets. We end this chapter with some discussion related to experimental strengths and weaknesses of our frameworks.

The last chapter provides the global conclusion of our thesis and gives tracks for our future work. We mention the significant contributions of our two frameworks and give some final evaluation regarding its advantages and limitations. We end this chapter as well as this thesis with some promising idea to continue improving this research in the future.

Version intermédiaire en date du 26 novembre 2022

Chapitre 2

Related work

This chapter largely presents the successful state-of-the-art models related to our problems :

(1) Improving segmentation and tracking with classical generative methods and (2) Anomaly detection with deep learning generative methods and supervised inference models.

We begin with the background of successful works in both classical and deep learning methods. We introduce the principle of Convolutional Neural Network (CNN) then all classical hand-crafted methods and modern deep learning models for popular computer vision tasks in transportation domain are mentioned in the next sections.

After that, we focus on the baseline methods that provide us promising solution to deal with our challenging problem : future prediction. There are two approaches we are dealing with : (i) classical hand-crafted future generating based on optical flow and (ii) deep learning multi-channels Conditional Generative Adversarial Network (C-GAN) for future prediction. We analyse their successful performances as well as their drawbacks to find suitable candidates and to propose some improvements to go beyond its limitations. In VGG-Net [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], Simonyan et al. investigated the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting. Their main contribution was a thorough evaluation of networks of increasing depth using an architecture with very small (3 × 3) convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers. These findings were the basis of their ImageNet 2014 Challenge submission, where their team secured the first and the second places in the localisation and classification tasks respectively. They also shown that their representations generalises well to other datasets, where they achieve state-of-the-art results. They have made their two best-performing ConvNet models publicly available to facilitate further researches on the use of deep visual representations in computer vision.

Continuing to extend the depth of layers, Szegedy et al. proposed GoogLeNet [START_REF] Szegedy | Going deeper with convolutions[END_REF]. They introduced a deep convolutional neural network architecture called Inception that achieved the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition 2014 Challenge (ILSVRC14). The main hallmark of this architecture is the improved use of the computing resources inside the network. By a carefully crafted design, they increased the depth and width of the network while keeping the computational budget constant. To optimise quality, the architectural decisions were based on the Hebbian principle and the intuition of multi-scale processing. One particular incarnation used in their submission for ILSVRC14 is called GoogLeNet, a 22 layers deep network, the quality of which is assessed in the context of classification and detection.

Recently, He et al. introduced a very deep residual learning architecture called ResNet [START_REF] He | Deep residual learning for image recognition[END_REF].

Because deeper neural networks are more difficult to train, they presented a residual learning framework to ease the training of networks that are substantially deeper than those used previously. They explicitly reformulated the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. They provided comprehensive empirical evidence showing that these residual networks are easier to optimise and can gain accuracy from considerably increased depth. On the ImageNet dataset they evaluated residual nets with a depth of up to 152 layers, about 8× deeper than VGG-Nets, while still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. What is the reason of such very strong performances of CNNs models ? Zeiler and Fergus [START_REF] Zeiler | Visualizing and understanding convolutional networks[END_REF] in their work tried to visualise and understand convolutional networks activations (Figure 2.2).

They studied the characteristics of the represented information in each layer. Investigating 7layers CNNs models inspired by AlexNET, they use the information in each layer as features then use non-CNNs learning models such as Support Vector Machine (SVM) to classify as a normal visual recognition task. According to their results, the first layer relates to simple low-level characteristics such as simple edges or color. The second layer provides more complex low level information as corner or center-surround. The semantic information exists from the third layer. It roughly corresponds to various objects' parts. In the fourth and fifth layers, we can find selective units for entire objects or large parts of them. Generally speaking, the deeper the layer, the more semantic the information. Comparing with low-level hand-crafted features whose filters are fixed before the learning tasks, the CNNs features are more adaptive to the learning tasks because all filters parameters are updated at each iteration during the training task.

Action understanding

In terms of naive approach, we can consider our problem as a traditional visual action recognition task. A classical solution scheme begins with features extraction, continues with information aggregation to have spatio-temporal representation then finishes with model learning.

From first success of Space Time Interest Point -STIP [START_REF] Laptev | On space-time interest points[END_REF] to the state-of-the-art low-level dense trajectories descriptors method [START_REF] Wang | Dense trajectories and motion boundary descriptors for action recognition[END_REF], they proposed various types of hand-crafted features : SIFT [START_REF] David | Distinctive image features from scale-invariant keypoints[END_REF], HOG [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF], HOF [START_REF] Brox | Large displacement optical flow : Descriptor matching in variational motion estimation[END_REF], MBH [START_REF] Wang | Dense trajectories and motion boundary descriptors for action recognition[END_REF], MpegFlow [START_REF] Kantorov | Efficient feature extraction, encoding and classification for action recognition[END_REF], SURF [START_REF] Bay | Speeded-up robust features (surf)[END_REF], HOG3D [START_REF] Klaser | A Spatio-Temporal Descriptor Based on 3D-Gradients[END_REF], etc. Especially, by adding some techniques to improve dense trajectories by explicit camera motion estimation, detecting human to remove outliers matches for homography estimation, and stabilising optical flow to eliminate camera motion, Wang et al. [START_REF] Wang | Action recognition with improved trajectories[END_REF] achieved state-of-the-art performances with a low-level descriptor on Hollywood2 [START_REF] Marszałek | Actions in context[END_REF], HMDB51 [START_REF] Kuehne | HMDB : a large video database for human motion recognition[END_REF] and UCF101 [START_REF] Soomro | Ucf101 : A dataset of 101 human actions classes from videos in the wild[END_REF].

Recently, there existed some successful models for action recognition using CNNs architecture (Figure 2.3). Simonyan et al. [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF] proposed a two-streams CNNs models combining spatial stream as single image input and temporal stream as multi-frame optical flow input. Tran et al. [START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF] constructed a 3D convolutional network to learn spatio-temporal features. Wang et al. [START_REF] Wang | Action recognition with trajectory-pooled deepconvolutional descriptors[END_REF] continued to improve the dense trajectories method by trajectories pooled convolutional descriptors.

In the two-streams CNNs model [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF], they investigated architectures of discriminatively Version intermédiaire en date du 26 novembre 2022 Version intermédiaire en date du 26 novembre 2022 trained deep Convolutional Networks (ConvNets) for action recognition in video. The challenge is to capture the complementary information on appearance from still frames and motion between frames. They also aimed to generalise the best performing hand-crafted features within a data-driven learning framework. Their contribution was three-fold. First, they propose a two-stream ConvNet architecture that incorporates spatial and temporal networks. Second, they demonstrated that a ConvNet trained on multi-frame dense optical flow is able to achieve very good performance in spite of limited training data. Finally, they proved that multitasks learning, applied to two different action classification datasets, could be used to increase the amount of training data and improve the performance on both. Their architecture is trained and evaluated on the standard video actions benchmarks of UCF-101 and HMDB-51, where it is competitive with the state of the art. It also exceeds by a large margin previous attempts to use deep nets for video classification.

Similar to the idea of integrating CNNs features to action recognition, Tran et al. [START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF] learned spatiotemporal features with 3D Convolutional Networks. They proposed a simple, yet effective approach for the spatio-temporal features learning using deep 3-dimensional convolutional networks (3D ConvNets) trained on a large-scale supervised video dataset. Their findings were three-fold : first, 3D ConvNets were more suitable for the spatio-temporal features learning compared to 2D ConvNets ; second, a homogeneous architecture with small 3 × 3 × 3 convolution kernels in all layers was among the best-performing architectures for 3D ConvNets ; and third, their learned features, namely C3D (Convolutional 3D), with a simple linear classifier outperformed state-of-the-art methods on 4 different benchmarks and are comparable with current best methods on the other 2 benchmarks. In addition, the features were compact : achieving 52.8% accuracy on UCF101 dataset with only 10 dimensions and also very efficient to compute due to the fast inference of ConvNets. Finally, they were conceptually very simple and easy to train and use.

To improve previous effective approaches of dense trajectories, Wang et al. [START_REF] Wang | Action recognition with trajectory-pooled deepconvolutional descriptors[END_REF] introduced Trajectory-Pooled Deep-Convolutional Descriptors. Because visual features are of vital importance for human action understanding in videos, they investigated a new video representation, called trajectory-pooled deep convolutional descriptor (TDD), which shares the merits of both hand-crafted features and deep-learned features. Specifically, they used deep architectures to learn discriminative convolutional feature maps and conducted trajectory-constrained pooling to aggregate these convolutional features into effective descriptors. To enhance the robustness of TDDs, they designed two normalisation methods to transform convolutional feature maps, namely spatiotemporal normalisation and channel normalisation. The advantages of their features came from : first, TDDs are automatically learned and contain high discriminative capacity compared with those hand-crafted features ; second, TDDs take into account the intrinsic characteristics of the temporal dimension and introduce the strategies of trajectories constrained Version intermédiaire en date du 26 novembre 2022 Although all these methods achieved promising results, they focused only on common single activities with simple backgrounds and few participants. In contrast, our context is more challenging because many simultaneous actions appear in traffic networks with a disorderly scenario, and the activities involve multiple phases and multiple objects. A better solution is detection, segmentation and tracking of all the relevant elements in the video then using other specific techniques to model the interaction and to define the class of actions.

Object detection

Recent state-of-the-art object detectors are based on CNNs. R-CNN [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] casts the object detection task as a region-proposal classification problem based on features extracted from AlexNET pre-trained model and SVM classifiers. SPP-Net [START_REF] He | Spatial pyramid pooling in deep convolutional networks for visual recognition[END_REF] provide a flexible input image size due to pooling features techniques. Fast R-CNN [START_REF] Girshick | Fast R-CNN[END_REF] extended previous works on R-CNN and SPP-Net to efficiently classify object proposals using an end-to-end architecture with several innovations to improve training and testing speed while also increasing detection accuracy. Fast Version intermédiaire en date du 26 novembre 2022 R-CNN trained the very deep VGG16 network 9x faster than R-CNN, is 213x faster at test-time, and achieves a higher mAP (that stands for mean Average Precision) on PASCAL VOC 2012.

Faster R-CNN [START_REF] Shaoqing Ren | Faster r-cnn : Towards real-time object detection with region proposal networks[END_REF] extends this approach by generating bounding box proposals with a fully convolutional Region Proposal Network (RPN). RPN considers a set of densely sampled anchor boxes, that are scored and regressed. Moreover, it shares convolutional features with proposal classification and regression branches. These branches operate on fixed-size features obtained using a Region-of-Interest (RoI) pooling layer. In a similar spirit, YOLO [START_REF] Redmon | You only look once : Unified, real-time object detection[END_REF] and SSD [START_REF] Liu | SSD : single shot multibox detector[END_REF] also use a set of anchor boxes, which are directly classified and regressed without a RoI pooling layers.

In YOLO, all scores and regressions are computed from the last convolutional feature maps, whereas SSD adapts the features to the size of the boxes. Features for predicting small-sized boxes come from early layers, and features for big boxes come from the latter layers, with larger receptive fields. All these object detectors rely on anchor boxes. In more details, the first successful model is R-CNN, proposed by Ross Girshick et al. The idea is quite natural by integrating CNNs features into interesting regions extracted by previous region proposal methods (Figure 2.4). Those CNNs features were easy to outperform all previous handcrafted features and show the promising possibilities of CNNs features when applied to the object detection task. Object detection performances, as measured on the canonical PASCAL VOC dataset, have plateaued in the last few years. The best-performing methods are complex ensemble systems that typically combine multiple low-level image features with high-level context. In their works, they proposed a simple and scalable detection algorithm that improves the mean average precision (mAP) by more than 30% relative to the previous best result on VOC 2012 achieving a 53.3% mAP. Their approach combines two key insights : first, one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localise and segment objects and second, when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant Version intermédiaire en date du 26 novembre 2022 performance boost. Since they combined region proposals with CNNs, they call their method R-CNN : Regions with CNN features. They also compared R-CNN to OverFeat, a previous proposed sliding-window detector based on a similar CNN architecture. They found that R-CNN outperforms OverFeat by a large margin on the 200-class ILSVRC2013 detection dataset. An alternative approach using anchor box and regression problem is YOLO [START_REF] Redmon | You only look once : Unified, real-time object detection[END_REF]. Redmon et al. proposed YOLO, a new approach to object detection. Prior work on object detection repurposes classifiers to perform detection. Instead, they framed object detection as a regression problem to spatially separated bounding boxes and associated class probabilities. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation.

Version intermédiaire en date du 26 novembre 2022

Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. Their unified architecture was extremely fast. Their base YOLO model processes images in real-time at 45 frames per second. A smaller version of the network, Fast YOLO, processes an outstanding 155 frames per second rate while still achieving twice the mAP of other real-time detectors. Compared to state-of-the-art detection systems, YOLO made more localisation errors but is less likely to predict false positives in the background. Finally, YOLO learned very general representations of objects. It outperformed other detection methods, including DPM and R-CNN, when generalising natural images to other domains like artwork.

Object segmentation

Object detection is the tasks that lists the objects appearing in an image or a video sequence and that locates them. Intuitively, the first step of discovering the interactions between users and transports infrastructures is detecting and localising all users and vehicles existing in videos.

Further steps, as modelling or classifying those interactions, need more information at pixel-level understanding. Assigning each pixel to an object or background category is the aim of object segmentation. Recent improved techniques of object segmentation proposed efficient approaches for semantic and instance segmentation of moving object, that is the priority input for learning model.

CNNs are driving advances in recognition, not only improving for whole-image classification [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF][START_REF] Szegedy | Going deeper with convolutions[END_REF][START_REF] He | Deep residual learning for image recognition[END_REF] but also making progress on local tasks with structured output. These include advances in bounding box object detection [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF][START_REF] Girshick | Fast R-CNN[END_REF][START_REF] Shaoqing Ren | Faster r-cnn : Towards real-time object detection with region proposal networks[END_REF][START_REF] Redmon | You only look once : Unified, real-time object detection[END_REF][START_REF] Liu | SSD : single shot multibox detector[END_REF], part and keypoint prediction, and local correspondence. The natural next step in the progression from coarse to fine inference is to make a prediction at every pixel. FCNs [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] proposed by J.Long et al. in 2015 is the first end-to-end CNNs model achieving state-of-the-art performances without further machinery.

Fully convolutional versions of existing networks predict dense outputs from arbitrary-sized inputs. Both learning and inference are performed whole-image-at-a-time by dense feedforward computation and backpropagation. In-network upsampling layers enable pixel-wise prediction and learning in nets with subsampled pooling.

Following the idea of making fully convolutional architectures, an improvement of the FCNs model was proposed by DeepLab [START_REF] Chen | Deeplab : Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs[END_REF]. A Deep Convolutional Neural Network such as VGG-16

or ResNet-101 was employed in a fully convolutional fashion, using atrous convolution to reduce the degree of the signal downsampling (from 32x down 8x). A bilinear interpolation stage enlarged the feature maps to the original image resolution. A fully connected CRF was then applied to refine the segmentation result and better capture the object boundaries.

However, repeated subsampling operations like pooling or convolution striding in deep

CNNs lead to a significant decrease in the initial image resolution. Lin et al. proposed RefineNet Version intermédiaire en date du 26 novembre 2022 [START_REF] Lin | Refinenet : Multi-path refinement networks for high-resolution semantic segmentation[END_REF], a generic multi-path refinement network that explicitly exploits all the information available along the down-sampling process to enable high-resolution prediction using long-range residual connections. In this way, the deeper layers that capture high-level semantic features could be directly refined using fine-grained features from earlier convolutions. The individual components of RefineNet employed residual connections following the identity mapping mindset, which allows for effective end-to-end training. Further, they introduced chained residual pooling, which captures rich background context in an efficient manner.

In terms of instance object segmentation, Mask R-CNN [START_REF] He | Mask r-cnn[END_REF] achieved state-of-the-art performance in a general context. This method efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. It is extended from Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition (Figure 2.6). The technique for providing object masks can be considered as semantic segmentation that shared the same spirit with FCNs [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] inside a bounding box with binary classes : object or background. Despite significant progressing steps in object segmentation in a static image, the problem of determining whether an object is in motion, irrespective of camera motion, is far from being solved. MP-Net [START_REF] Tokmakov | Learning Motion Patterns in Videos[END_REF] proposed by Tomakov et al. in 2017 addressed this challenging task by learning motion patterns in videos. The core of their approach is a fully convolutional network, which was learned entirely from synthetic video sequences, and their ground-truth optical flow and motion segmentation. This encoder-decoder style architecture first learned a coarse representation of the optical flow field features and then refined it iteratively to produce motion labels at the original high resolution. They further improve this labeling with an objectness map and a conditional random field, to account for errors in optical flow, and also to focus on moving "things" rather than "stuff". The output label of each pixel denotes whether it has undergone independent motion, i.e. irrespective of camera motion.

Optical flow

Optical flow is the apparent motion of brightness patterns in the image. Because the motion field is the projection of the 3D scene motion into the image, the optical flow would be ideally the same as the motion field. As a promising result of MP-Net, we consider using an optical flow map as an adding input channel for CNNs model in order to enhance information.

The first efficient way to solve this equation with two unknowns was proposed by B.Lucas and T.Kanade in 1981 [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF]. Another algorithm was proposed by Horn and Schuck in 1981 [START_REF] Horn | Determining optical flow[END_REF] by adding smoothness constraint. In 2011, Brox and Malik proposed Large Displacement Optical Flow (LDOF) [START_REF] Brox | Large displacement optical flow : Descriptor matching in variational motion estimation[END_REF] estimation method by adding into the energy function a matching term that penalises the difference between flows and HOG matches. MDP-Flow 2 [START_REF] Jia | Motion detail preserving optical flow estimation[END_REF] proposed by Xu et Version intermédiaire en date du 26 novembre 2022 with a variational approach. Feature information is aggregated from fine to coarse using sparse convolutions and max-pooling. However, it did not perform any learning and all parameters were set manually. The successive work of [START_REF] Revaud | Epicflow : Edge-preserving interpolation of correspondences for optical flow[END_REF] termed EpicFlow had put even more emphasis on the quality of sparse matching as the matches from DeepFlow were merely interpolated to dense flow fields while respecting image boundaries.

Recently, FlowNet [START_REF] Dosovitskiy | Flownet : Learning optical flow with convolutional networks[END_REF] only used a variational approach for the optional refinement of the flow field predicted by the convolutional net and did not require any handcrafted methods for aggregation matching and interpolation. P. version of FlowNet 2.0 is PwCNet [START_REF] Sun | Pwc-net : Cnns for optical flow using pyramid, warping, and cost volume[END_REF] with an encode-decode architecture.

Usually, optical flow is a basement for action recognition. On the other hand, Lucet al. [START_REF] Luc | Predicting Future Instance Segmentation by Forecasting Convolutional Features[END_REF] introduced a new utilisation of optical flow for predicting future segmentation. This first work suggests to us a promising way to exploit optical flow for generating the missing video information.

Supervised context Summary

Method Description Performance

AlexNET 2012 [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] Image classification, single Accuracy (ACC) is the fraction between true prediction (both true positive and true negative) and total prediction.

For searching or detection problems, precision is the fraction of retrieved elements that are relevant to the searching query. Recall is the fraction of the elements that are relevant to the searching problem that is successfully retrieved. By computing precision and recall at every position, one can plot a precision-recall curve, plotting precision p(r) as a function of recall r. Average precision (AP) computes the average value of p(r) over the interval from r = 0 to r = 1. Mean average precision (mAP) for a set of searching elements is the mean of the average precision scores for each searching element.

For image segmentation, Intersection over Union (IOU) score for each class is the fraction between true positive pixels and the sum of true positive, false negative and false positive pixels.

The mean IoU (mIOU) is the mean of IOU for all classes. Especially, for instance-level segmentation, performance on this task is measured by the COCO-style mask AP (average precision on region level) and AP 50 (average precision when overlap at region level is at least 50%).

For optical flow estimation, Endpoint Error (EE) is defined as the scalar length of different vector ||V est -V gt || between estimated optical flow vector V est and ground-truth optical flow vector V gt . Average Endpoint Error (AEE) is average of EE for all optical flow vector.

Version intermédiaire en date du 26 novembre 2022

Unsupervised context

Despite the promising results in supervised context for action recognition and localization, the researches in unsupervised or weakly-supervised context are still challenging. Oneata et al. [START_REF] Oneata | Spatio-Temporal Object Detection Proposals[END_REF] extend 2D object proposal method to produce spatio-temporal proposals by mean of a randomized supervoxel merging process. Kwak et al. [START_REF] Kwak | Unsupervised Object Discovery and Tracking in Video Collections[END_REF] tackle the discovery and localization problem using a part-based region matching approach [START_REF] Cho | Unsupervised Object Discovery and Localization in the Wild : Part-based Matching with Bottom-up Region Proposals[END_REF] then extended to spatio-temporal tubes by tracking. Both methods used hand-crafted features with several complementary process so take long time compared with CNNs approaches. Besides, the context was simple with a single object or co-localization and the action was only localized by tracking.

Recently, we can benefit from a promising CNNs architecture adapted for unsupervised context : Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF]. GAN is an unsupervised generative model, that learns to generate the true data distribution by implicit density estimation. GAN is composed of a generator and a discriminator. In training, the generator is trained in a way the discriminator cannot distinguish fake images produced by the generator from the real ones.

Meanwhile, the discriminator learns to distinguish fake images from real images. Through this adversarial competition, the generated images from GAN become harder to distinguish from the reals. The successful results of GANs leads to various application in generative model : learning unsupervised representation [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF], image translation [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] and synthesis [START_REF] Reed | Generative adversarial text to image synthesis[END_REF], video gererating [START_REF] Vondrick | Generating videos with scene dynamics[END_REF]. In spite of promising achievements in generative tasks, GANs have not been adapted to discriminative task such as the one of our context yet. LSTM for pixel-level prediction, which independently capture the spatial layout of an image and the corresponding temporal dynamics. In [START_REF] Oliu | Folded recurrent neural networks for future video prediction[END_REF], the authors introduced an architecture based on recurrent Conv-AEs to deal with the network capacity and error propagation problems for future video prediction. It consisted of a series of bijective Gate Recurrent Unit (GRU) layers, which allowed for a bidirectional flow of information between input and output : they considered the input as a recurrent state and update it using an extra set of gates. [START_REF] Gao | Im2flow : Motion hallucination from static images for action recognition[END_REF] proposed an approach using Conv-AE that hallucinated the unobserved future motion implied by a single snapshot to help static-image action recognition. The key idea was to learn prior over short-term dynamics from thousands of unlabeled videos, infer the anticipated optical flow on novel static images, and then train discriminative models that exploit both streams of information. Obviously, most recent researchers build their model upon a Conv-AE model to reconstruct the future informations.

Future prediction 2.2.1 State of the art

Generative modelling of future RGB video frames has recently been studied using a variety of techniques : prediction of future human pose [START_REF] Villegas | Learning to generate long-term future via hierarchical prediction[END_REF], generative adversarial training model [START_REF] Mathieu | Deep multi-scale video prediction beyond mean square error[END_REF], forecasting convolutional features [START_REF] Luc | Predicting Future Instance Segmentation by Forecasting Convolutional Features[END_REF]. All their works focus on predicting future information which they never can not achieve. This context is a bit different than our work where we address the problem of generating only the missing information the detector failed to produce. We can freely generate backward and forward segmentation or bounding boxes based on some current results from the detectors. Despite the difference between those contexts, we apply the simple baseline approach from Luc et al. [START_REF] Luc | Predicting Future Instance Segmentation by Forecasting Convolutional Features[END_REF] that translated a segment on the basis of flow vectors.

Generative Adversarial Network

Generative Adversarial Network (GAN) proposed by Ian Goodfellow et al. [START_REF] Goodfellow | Generative adversarial nets[END_REF] is a framework estimating generative models via an adversarial process. By now, GANs are built as deep neural network models and they are used to generate synthetic images. Generally, the architecture comprises two deep neural networks, a generator and a discriminator, which work against each other (thus, "adversarial"). The generator generates new data instances, while the discriminator evaluates the data for authenticity and decides whether each instance of data is "real" from the training dataset, or "fake" from the generator (Figure 2.8). GANs models focus to solve the difficulty of approximating many intractable probabilistic computations that arise in maximum likelihood estimation and related strategies, and difficulty of leveraging the benefits of piecewise linear units in the generative context. of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and the probability of D making a mistake being 1 2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. In the proposed adversarial nets framework, the generative model is pitted against an adversary : a discriminative model that learns to determine whether a sample is from the model distribution or the data distribution. The generative model can be thought of as analogous to a team of counterfeiters, trying to produce fake currency and use it without detection, while the discriminative model is analogous to the police, trying to detect the counterfeit currency. Competition in this game drives both teams to improve their methods until the counterfeits are inseparable from the genuine articles.

Improving Detection and Tracking

Mask R-CNN

Mask R-CNN [START_REF] He | Mask r-cnn[END_REF] is a conceptually simple, flexible, and general framework for objects instances segmentation. In principle, Mask R-CNN is an extension of Faster R-CNN [START_REF] Shaoqing Ren | Faster r-cnn : Towards real-time object detection with region proposal networks[END_REF] that constructs a third branch as FCN [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] for segmentation (Figure 2.9). Therefore, we find Mask R-CNN is an effective combination of elements from the classical computer vision tasks for Version intermédiaire en date du 26 novembre 2022 object detection and semantic segmentation. Mask R-CNN surpasses all previous state-of-the-art single-model results on the COCO dataset for both instance segmentation and bounding box detection tasks. Though, it has a limitation in implementation for other datasets. In particular, when applied to video sequences the detection can get unstable when the object is rapidly changing its appearance. Such events frequently occur in transportation based video sequences, for instance when a vehicle is turning right or left or when its apparent size increase or decrease owing to its relative move with respect to the camera.

IOU object tracker

Before the rise of CNNs models, almost all successful trackers performed object location based on hand-crafted features : IHTLS [START_REF] Dicle | The way they move : Tracking multiple targets with similar appearance[END_REF], H2T [START_REF] Wen | Multiple target tracking based on undirected hierarchical relation hypergraph[END_REF], CMOT [START_REF] Bae | Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning[END_REF], etc. Because of the use of (so called) too simple concepts, these trackers show their limited performances when facing difficult scenarios from recent multi-object tracking challenges [START_REF] Lyu | Ua-detrac 2017 : Report of avss2017 & iwt4s challenge on advanced traffic monitoring[END_REF]. Recently, IOU Tracker [START_REF] Bochinski | High-speed tracking-by-detection without using image information[END_REF] based on the strong performances of CNN based detector surpassed all previous methods in both easy and difficult scenarios. Bochinski et al. considered only the overlaps between bounding boxes obtained from detector to associate them (Figure 2.10). It benefits from the fast and strong performance of CNNs-based detector without any other visual information. As a consequence, this can be seen as a bit risky process due to the complete dependence of the tracking task to the accuracy of the detector : as discussed above, both false positive and negative error of the detector can cause fragmented trajectories. Equal error rate (EER) is a popular measurement metric in biometric security system that used to predetermine the threshold values for its false acceptance rate (FAR) and its false rejection rate (FRR). When the rates are equal, the common value is called the equal error rate.

Anomaly Detection

The value represents the point where the proportion of false acceptances is equal to the proportion of false rejections. The lower the equal error rate value, the higher the accuracy of the system. Area Under Curve (AUC) of Receiving Operation Characteristic (ROC) curve is more popular than EER. ROC curve is plotted with True Positive Rate (TPR) against the False Positive Rate (FPR) where TPR is on the y-axis and FPR is on the x-axis. AUC is the area under this curve.

Higher the AUC, the better the model is. AUC is precisely presented in section 4.1.6.

From frame-level to object-centric level, we can also apply the AUC and EER as potential evaluation metric by adding some bounding box level conditions. Following the definition of [START_REF] Vu | Robust anomaly detection in videos using multilevel representations[END_REF][START_REF] Lu | Abnormal event detection at 150 fps in matlab[END_REF], a candidate frame will be considered as abnormal frame if the intersection between its detected boxes and the ground-truth boxes is greater than 40%. To have a fair comparison, we apply this evaluation metric. In contrast, we find that the bounding box AUC and EER is not enough good for evaluating object-centric level performance. It is still similar to frame-level rather than purely bounding box level. Hence, we also define ourself some alternative evaluation Version intermédiaire en date du 26 novembre 2022 metrics that more adaptive to object-centric level : IOU 50 Rate, IOU 75 Rate and mIOU . Those metric are presented in detail in section 4.1.6.

Early works with hand-crafted features

Before Convolutional Neural Networks (CNNs) became popular, most of the early methods were based on the extraction of hand-crafted features to estimate the models of normal and abnormal events. Motion trajectories were used as the principal features [START_REF] Medioni | Event detection and analysis from video streams[END_REF][START_REF] Zhang | Learning semantic scene models by object classification and trajectory clustering[END_REF] because of their fast extraction and simple implementation. However, the single motion information was not sufficient to represent all the spectrum of abnormal events and the motion estimator was easily confused in crowed and complex scenes.

To improve these limitations, both appearance and motion were extracted along the trajectories. Kim et al. [START_REF] Kim | Observe locally, infer globally : A space-time mrf for detecting abnormal activities with incremental updates[END_REF] used Histogram of optical flow to build space-time Markov Random Generally, most of those methods achieved good performance on some simple datasets without changing the camera orientation, illumination or dealing with complex activities. For more challenging datasets, they just yielded moderate performance due to the limitation of hand-crafted features in case of large datasets and complex scenarios.

Recent successful models with Discriminative Deep learning model

Recently, the existing of powerful deep learning models leads to many successful approaches in anomaly detection. Hinami et al. [START_REF] Hinami | Joint detection and recounting of abnormal events by learning deep generic knowledge[END_REF] At the same time, Liu et al. [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF] introduced a first work of future prediction based anomaly detection. They adopted CGAN techniques with U-Net model as generator to predict the next frame (Figure 2.16). To generate high-quality image, they made the constraints in terms of appearance (intensity loss and gradient loss) and motion (optical flow loss). Then the difference between a predicted future frame and its ground truth was used to detect an abnormal event.

They achieved AU C = 0.85 on Avenue, 0.95 on Ped2, 0.83 on Ped1 and 0.73 on ShanghaiTech datasets.

Continuing of this approach, Ravanbakhsh et al. [START_REF] Ravanbakhsh | Abnormal event detection in videos using generative adversarial nets[END_REF] proposed a GAN architecture for anomaly detection in particular crowd scenes. Their model is trained using normal frames and Version intermédiaire en date du 26 novembre 2022 Matching a single autoencoder network with classification sub-network to build a hybrid network, Nguyen et al. [START_REF] Nguyen | Hybrid deep network for anomaly detection[END_REF] proposed a model adapted from a typical auto-encoder working on video patches under the perspective of sparse combination learning. Their model focused on unsupervised learning common characteristics of normal events with the emphasis of their spatial locations (by supervised losses). This was the first work that directly adapts the patch position as the target of a classification subnetwork. The model is capable to provide a score of anomaly assessment for each video frame. They achieved AU C = 0.83 on Avenue and 0.84 on Ped2 dataset.

Lee et al. [START_REF] Lee | Spatio-temporal adversarial networks for abnormal event detection[END_REF] used ConvLSTM to build spatio-temporal adversarial networks (STAN). They installed a spatio-temporal generator which synthesized an inter-frame by considering spatiotemporal characteristics with bidirectional ConvLSTM. A proposed spatio-temporal discriminator determined whether an input sequence was real-normal or not with 3D convolutional layers.

Then they trained these two networks in an adversarial way to effectively encode spatio-temporal features of normal patterns. After the learning, they independently use the generator and the Version intermédiaire en date du 26 novembre 2022 discriminator as detectors, and deviations from the learned normal patterns were detected as abnormalities. They achieved AU C = 0.87 on Avenue, 0.965 on Ped2, 0.82 on Ped1 datasets.

Expecting to build an end-to-end generative model for anomaly detection, Sabokrou et al. [START_REF] Sabokrou | Adversarially learned oneclass classifier for novelty detection[END_REF] proposed a mix architecture of GAN and CNN for one-class classification. Their architecture contained two deep networks, each of them trained by competing with each other while collaborating to understand the underlying concept in the target class, and then classified the testing samples. One network worked as the novelty detector, while the other supported it by enhancing the inlier samples and distorting the outliers. The intuition was that the separability of the enhanced inliers and distorted outliers were much better than deciding on the original samples.

They achieved EER = 0.16 on Ped2 dataset.

Figure 2.17 -Object-centric framework for anomaly detection proposed by Ionescu et al. [START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF].

Recently, Ionescu et al. [START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF] achieved state-of-the-art performance on various popular benchmarks [START_REF] Mahadevan | Anomaly detection in crowded scenes[END_REF][START_REF] Lu | Abnormal event detection at 150 fps in matlab[END_REF][START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF] by building a reconstruction error model that learned both appearance and temporal gradient feature at object-centric levels then combined K-means clustering with SVMs techniques to produce abnormality scores (Figure 2.17 an interesting alternative supervised scenario [START_REF] Liu | Margin learning embedded prediction for video anomaly detection with a few anomalies[END_REF]. Classical semi-supervised video anomaly detection assumes that only normal data are available in the training set because of the rare and unbounded nature of anomalies. It is obviously, however, these infrequently observed abnormal events can actually help with the detection of identical or similar abnormal events, a line of thinking that motivates us to study open-set supervised anomaly detection with only a few types of abnormal observed events and many normal events available. Under the assumption that normal events can be well predicted, they propose a Margin Learning Embedded Prediction (MLEP) framework. There are three features in MLEP-based open-set supervised video anomaly detection : i) they customize a video prediction framework that favors the prediction of normal events and distorts the prediction of abnormal events ; ii) The margin learning framework learns a more compact normal data distribution and enlarges the margin between normal and abnormal events. Since abnormal events are unbounded, their framework consequently helps with the detection of abnormal events, even for anomalies that have never been previously observed.

Therefore, our framework is suitable for the open-set supervised anomaly detection setting ; iii) their framework can readily handle both frame-level and video-level anomaly annotations.

Considering that video-level anomaly detection is more easily annotated in practice and that anomaly detection with a few anomalies is a more practical setting, their work thus pushes the application of anomaly detection towards real scenarios.

Inspired by the idea of adding supervised learning classifiers into the unsupervised features extraction framework of Liu et al. [START_REF] Liu | Margin learning embedded prediction for video anomaly detection with a few anomalies[END_REF], we would like to go further by three aspects. Firstly, we propose a strong and flexible framework that provide more distinctive features for the following classifiers. Secondly, we investigate the effect of supervised classifiers corresponding to the number of abnormal samples transfered from original test set into training set. Thirdly, instead of going from frame-level to video-level, we have an local approach from frame-level to object-centric level. All of those aspects have their own significant improvements for our performances.

Conclusion

In this chapter, we largely present the state-of-the-art methods targeted to solve our two main tasks : (1)Improving segmentation and tracking with classical generative methods and (2)

Anomaly detection with deep learning generative methods and supervised inference models. We Version intermédiaire en date du 26 novembre 2022 also provide the necessary knowledge of fundamental convolutional neural network models for various computer vision tasks in transportations domain. For the first problem, we focus on two baseline methods : Mask R-CNN and IOU Tracker. For the second problem, we are inspired by the modern approaches of future prediction for anomaly detection.

In the first part, although achieving impressive performance, both baseline methods have limitations in case of broken detection leading to fragmented trajectories. To deal with this drawback, we propose some interesting add-on modification based on simple hand-crafted generative methods. In the second part, we continue to develop the successful deep generative models of future prediction for anomaly prediction by proposing a flexible and strong framework to adapt with various challenging problems : the integration of both appearance and motion information in terms of RGB and grayscale at the input side ; the simplification of feature encoding dealing with supervised abnormality inference model, the capability of going from frame-level detection to object-centric level detection.

Version intermédiaire en date du 26 novembre 2022 Then we present experimental results for both qualitative and quantitative evaluation.

Our first research is an initial work for evaluating the performances of the classical handcrafted generative approach in future prediction and its usability for improving segmentation and tracking. Based on the two baseline methods of Luc et al. [START_REF] Luc | Predicting Future Instance Segmentation by Forecasting Convolutional Features[END_REF] to generate future segmentation by using optical flow vectors, we propose an extension beyond the original transforms. We investigate the generated information not only for both backward and forward directions but also for longer sequences of frames. This simple but useful generative framework leads to the improvement of the objects tracking task. Applying generated information with adding SURF features allow us to enhance the performances of IOU Tracker [START_REF] Bochinski | High-speed tracking-by-detection without using image information[END_REF] by connecting fragmented pieces of trajectories.

The experiments are installed for evaluating two scenarios corresponding to two stages of the methods we proposed in previous parts. For each scenario, we begin with the introduction of the public datasets and evaluation metric where we setup the experiments on. Then we describe our experimental implementations in details : the basic framework for installing our models, the execution environment, the parameters setting, the evaluation metric, etc. Next, we introduce our experimental results in both qualitative and quantitative evaluation. Finally, we analyze our strengths and our limitations regarding to our results.

Version intermédiaire en date du 26 novembre 2022 Then the RoI Align network outputs multiple bounding boxes rather than a single definite one and warps them into a fixed dimension. Warped features vectors for each region are then fed into fully connected layers to make classification using softmax. At the same time, boundary box localization is further refined using the regression model. Warped features vectors are also fed into Mask Generator. It is another Fully Convolutional part acting like FCN [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] for segmentation, which consists of two CNN's to output a binary mask for each RoI. Mask generator allows the network to generate masks for every class without competition among classes.

Optical flow estimation by LDOF

Large Displacement Optical Flow (LDOF) [START_REF] Brox | Large displacement optical flow : Descriptor matching in variational motion estimation[END_REF] is an optical flow extraction method proposed by Brox et al. This method combines the advantages of two possible ways for establishing point correspondences between images with moving objects : one side, energy minimization methods that yield very accurate, dense flow fields, but fail as displacements get too large. Other side, there is descriptor matching that allows for large displacements, but correspondences are very sparse, have limited accuracy, and experience many outliers due to missing regularity constraints . LDOF establishes a region hierarchy for both images. Descriptor matching on these regions provides a sparse set of hypotheses for correspondences. These are integrated into a variational approach and guide the local optimization to large displacement solutions. The variational optimization selects among the hypotheses and provides dense and subpixel accurate estimates, making use of geometric constraints and all available image information.

Region segmentation : The first stage of LDOF is region computation. To create the image region, Brox et al. relied on a segmentation methods based on the boundary detector instead of simple edge detector. The advantage of using boundary detector over simple edge detection is that it takes texture into account. Boundaries due to repetitive structures are damped whereas strong changes in texture create additional boundaries. Consequently, boundaries are more likely to correspond to objects or parts of objects. Then, it increases the stability of the regions to be matched.

The segmentation method returns a boundary map g(x) as shown in Figure 3.2. Strong edges correspond to more likely object boundaries. It further returns a hierarchy of regions created from this map. Regions with weak edges are merged first, while separations due to strong edges persist for many levels in the hierarchy. They generally take the regions from all the levels in the Version intermédiaire en date du 26 novembre 2022 hierarchy into account. From the regions of the first image, however, they only keep the most stable ones, i.e. those which exist in at least 5 levels of the hierarchy. Unstable regions are usually arbitrary subparts of large regions. They also ignore extremely small regions (with less than 50 pixels) from both images. These regions are usually too small to build a discriminative enough descriptor for reliable matching. Region descriptor and matching : To each region they fit an ellipse and normalize the area around the centroid of each region to a 32 × 32 patch. The normalized patch then serves as the basis for a descriptor. They build two descriptors S and C in each region. S consists of 16 orientation histograms with 8 bins, like in SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]. C comprises the mean RGB color of the same 16 subparts as the SIFT descriptor. While the orientation histograms consider the whole patch to take also the shape of the region into account, the color descriptor is computed only from parts of the patch that belong to the region.

Correspondences between regions are computed by nearest neighbours matching. We compute the Euclidean distances of both descriptors separately and normalize them by the sum over Version intermédiaire en date du 26 novembre 2022 all distances :

d 2 (S i , S j ) = S i -S j 2 2 1 N k,l S k -S l 2 2 (3.1) d 2 (C i , C j ) = C i -C j 2 2 1 N k,l C k -C l 2 2 (3.2)
where N is the total number of combinations (i, j). This normalization allows to combine the distances such that both parts in average have equal influence :

d 2 (i, j) = 1 2 (d 2 (C i , C j ) + d 2 (S i , S j )) (3.3)
They can exclude potential pairs by adding high costs to their distance. They do this for correspondences with a displacement larger than 15% of the image size or with a change in scale that is larger than factor 3. Depending on the needs of the application, these numbers can be adapted. Smaller values obviously produce fewer false matches, but restrict the allowed image transformations.

Hypotheses refinement by deformed patches : Rather than deciding on a fixed correspondence at each keypoint, which could possibly be an outlier (Figure 3.3), they propose to integrate several potential correspondences into the variational approach. For this purpose, a good confidence measure is of great importance. They found that the distance between patches globally separates good and bad matches much better than the above descriptors. The main problem with direct patch comparison (classical block matching) is its sensitivity to small shifts or deformations. Once the deformation's corrected, the Euclidean distance between patches is very informative, particularly when considering only pixels from within the region.

The optimum shift and deformation needed to match two patches can be estimated by minimizing the following cost function :

E(u, v) = (P 2 (x + u, y + v) -P 1 (x, y)) 2 dxdy + α (|∇u| 2 + |∇v| 2 )dxdy (3.4)
where P 1 and P 2 are the two patches, u(x, y), v(x, y) denotes the deformation field to be estimated, and α = 10000 is a tuning parameter that steers the relative importance of the deformation smoothness. The energy is a non-linearized, large displacement version of the Horn and Schunck energy and is sufficient for this purpose. The regularizer gets a very high weight in this case, as without regularization every patch can be made sufficiently similar to any other.

As the patches are very small and a simple quadratic regularizer is applied, the estimation is quite efficient. Nevertheless, it would be a computational burden to estimate the deformation for Version intermédiaire en date du 26 novembre 2022 each region pair. To this end, they preselect the 10 nearest neighbours for each patch using the distance from the previous section and compute the deformation only for these candidates. The five nearest neighbours according to the patch distance are then integrated into the variational approach described in the next part. Each potential match j = 1, ..., 5 of a region i comes with a confidence :

c j (i) =            d 2 (i) -d 2 (i, j) d 2 (i, j) if d 2 (i) > 0 0 else (3.5)
where d 2 (i, j) is the Euclidean distance between the two patches after deformation correction and d 2 (i) is the average Euclidean distance among the 10 nearest neighbors. This measure takes the absolute fit as well as the descriptiveness into account. They restrict the distance to be computed only at patch positions within the region. Hence the changing background of a moving object part would not destroy similarity of a correct match.

Variational flow : Although most of the correspondences are correct, the flow field derived Version intermédiaire en date du 26 novembre 2022 from these by interpolation, is far from being accurate. This is because they have a hard decision to make when selecting the nearest neighbour. Moreover, a lot of image information is neglected and substituted by a smoothness prior. In order to obtain a more accurate, dense flow field, we integrate the matching hypotheses into a variational approach, which combines them with local information from the raw image data and a smoothness prior.

The energy following function is optimised with an additional data constraint that integrates the correspondence information :

E(w(X)) = Ψ (|I 2 (X + w(X)) -I 1 (X)| 2 )dX + γ Ψ (|∇I 2 (X + w(X)) -∇I 1 (X)| 2 )dX (3.6) + β 5 j=1 ρ j (X)Ψ ((u(X) -u j (X)) 2 + (v(X) -v j (X)) 2 )dX + α Ψ (|∇u(X)| 2 + |∇v(X)| 2 + g(X) 2 )dX
Here, I 1 and I 2 are the two input images, w = (u, v) is the sought optical flow field, and X = (x, y) denotes a point in the image. (u j , v j )(X) is one of the motion vectors derived at position X by region matching (j indexing the 5 nearest neighbours). If there is no correspondence at this position, ρ j (X) = 0. Otherwise, ρ j (X) = c j , where c j is the distance based confidence in previous formula. α = 100, β = 25, and γ = 5 are tuning parameters, which steer the importance of smoothness, region correspondences, and gradient constancy, respectively. They use the robust function Ψ (s 2 ) = √ s 2 + 10 -6 in order to deal with outliers in the data as well as in the smoothness assumption. They also integrate the boundary map g(X) in order to avoid smoothing across strong region boundaries. Rather than a straightforward three step procedure with (i) interpolation of the region correspondences, (ii) removal of outliers not fitting the interpolated flow field (iii) optical flow estimation initialized by the interpolated inlier correspondences, the above energy combines all three steps in a single optimization problem. The energy is non-convex and can only be optimized locally. They can compute the Euler-Lagrange equations, which state a necessary condition for a local optimum.

Optical flow estimation by Full Flow

Full Flow [START_REF] Chen | Full flow : Optical flow estimation by global optimization over regular grids[END_REF] proposed by Chen et al. is a global optimization approach to optical flow estimation. The approach optimizes a classical optical flow objective over the full space of mappings between discrete grids. No descriptor matching is used. The highly regular structure of the space of mappings enables optimizations that reduce the computational complexity of the algorithm's inner loop from quadratic to linear and support efficient matching of tens of thousands of nodes to tens of thousands of displacements.

Version intermédiaire en date du 26 novembre 2022 Figure 3.4 -Optical flow over regular grid in Full Flow method [START_REF] Chen | Full flow : Optical flow estimation by global optimization over regular grids[END_REF]. Each pixel p in I 1 is spatially connected to its four neighbors in I 1 and temporally connected to (2ζ + 1) 2 pixels in I 2 .

Model : Let I 1 , I 2 : Ω → R 3 be two color images, where Ω ⊂ Z 2 is the image domain. Let

f = (f 1 ; f 2 ) : Ω → [-ζ; ζ]

Version intermédiaire en date du 26 novembre 2022

The objective function is :

E(f ) = p∈I 1 (ρ D (p, f p , I 1 , I 2 )) + λ p,q∈N w p,q ρ S (f p -f q ) (3.7)
where N ⊂ Ω 2 is the 4-connected pixel grid illustrated in Figure 3.4. The data term (ρ D (p, f p , I 1 , I 2 ))

penalizes flow fields that connect dissimilar pixels p and (p + f p ). They use truncated normalized cross-correlation :

ρ D (p, f p , I 1 , I 2 ) = 1 -max(N CC, 0) (3.8)
where N CC is the normalized cross-correlation between two patches, one centered at p in I 1 and one centered at (p + f p ) in I 2 , computed in each color channel and averaged. The truncation at zero prevents penalization of negatively correlated patches. If (p + f p ) is in the buffer zone Ω Ω, the data term is set to a constant penalty ζ.

Their optimization approach assumes that the regularization term has the following form :

ρ S (f ) = min(ρ(f 1 ) + ρ(f 2 ), τ) (3.9)
where f 1 , f 2 are the two components of vector f and ρ(•) is a penalty function, such as the L1 norm or the Charbonnier penalty. Their formulation and the general solution strategy can accommodate non-convex functions ρ, such as the Lorentzian and the generalized Charbonnier penalties. They apply the reduction of message passing complexity from quadratic to linear. The highly efficient min-convolution algorithm will assume that the function ρ is convex. Note that the regularization term couples the horizontal and vertical components of the flow. They apply a Laplace weight to attenuate the regularization along color discontinuities :

w p,q = exp(- I 1 (p) -I 2 (q) β ) (3.10)
Optimization : Objective function 3.7 is a discrete Markov random field with a two-dimensional label space. The label space of the model is [-ζ; ζ] 2 ∩ Z 2 . To optimize the model, they use TRW-S, which optimizes the dual of a natural linear programming relaxation of the problem. They choose TRW-S due to its effectiveness in optimizing models with large label spaces. Note that TRW-S optimizes the dual objective and will generally not yield the optimal solution to the primal problem.

Generating object segmentation by Mask R-CNN and Optical flow

Luc et al. [START_REF] Luc | Predicting Future Instance Segmentation by Forecasting Convolutional Features[END_REF] proposed two baseline methods to generate future segmentation based on flow vector F t-1→t from frame t -1 to t and the current instance segmentation I t at frame t. They were Version intermédiaire en date du 26 novembre 2022 called Shif t and W arp (Figure 3.5a).

-Warp approach translates each pixel of instance mask independently using the flow vector at the corresponding position inside this mask. To yield the new mask, the object class and the confident score is copied from the previous one. The predicted mask and flow field are used to make the next prediction, and so on. This approach is suitable for longtime prediction because of the ability of rescaling objects based on flow vector. In contrast, predicted mask suffers from an accumulated error phenomenon and has many holes inside its boundaries. To fill in those gaps, a post-processing step with morphological operator is necessary.

-Shift approach, in brief, entirely shifts instance mask using the average flow vector calculated inside the mask. Then the object class and the confident score are also copied from the previous mask. While this approach can avoid accumulating errors and generating holes inside the mask, it is not really suitable for longtime prediction due to the non-rescaling of the mask.

We propose an extension beyond the two original baselines. Instead of only predicting future segmentation (forward), we also generate past segmentation (backward) by projecting the flow vector in opposite direction (Figure 3.5b). Intuitively, the missing detection of Mask R-CNN does not last along many frames so we can generate new segmentation in both forward and backward directions. where p i is the pixel of instance I i , F i→l is a translation by flow vectors from frame i to frame l. To avoid the accumulated errors of optical flow estimation methods, we minimize N = 2, j = 1.

This condition maintain the continuity of trajectories along the flow vectors when we generate new masks. We apply this extension in both forward and backward directions with both Shif t and W arp approaches. Therefore, we have 8 ways to generate object segmentation to fill in the gaps of trajectories.

Extracting SURF feature

Speed Up Robust Feature (SURF) [START_REF] Bay | Speeded-up robust features (surf)[END_REF] Version intermédiaire en date du 26 novembre 2022

Fast Hessian detector : Given a point p = (x, y) in an image I, the Hessian matrix H(p, σ ) in p at scale σ is defined as follows :

H(p, σ ) =       L xx (p, σ ) L xy (p, σ ) L xy (p, σ ) L yy (p, σ )       (3.12)
where L xx (p, σ ) is the convolution of the Gaussian second order derivative ∂ 2 ∂x 2 g(σ ) with the image I in point p, and similarly for L xy (p, σ ) and L yy (p, σ ).

Generally, SURF is a speed up version of SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] proposed by David Lowe. Gaussians are optimal for scale-space analysis. In practice, however, the Gaussian needs to be discretized and cropped (Figure 3.6 left half) and even with Gaussian filters aliasing still occurs as soon as the resulting images are sub-sampled. As Gaussian filters are non-ideal in any case, and given Lowe's success with LoG approximations [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF] in extracting SIFT, Bay et al. pushed the approximation even further with box filters (Figure 3.6 right half). These approximate second order Gaussian derivatives, and can be evaluated very fast using integral images, independently of size. One big advantage of this approximation is that, convolution with box filter can be easily calculated with the help of integral images. And it can be done in parallel for different scales. Also the SURF rely on determinant of Hessian matrix for both scale and location. In practice, the 9 × 9 box filters in Figure 3.6 are approximations for Gaussian second order derivatives with σ = 1.2 and represent highest spatial resolution). Denote the approximations by D xx , D yy , D xy , the weights applied to the rectangular regions are kept simple for computational efficiency, but they need to further balance the relative weights in the expression for the Hessian's determinant with coefficient :

|L xy (1.2)| F |D xx (9)| F |L xx (1.2)| F |D xy (9)| F ≈ 0.9 (3.13)
where |x| F is the Frobenius norm. Finally, they achieved :

det(H approx ) = D xx D yy -(0.9D xy ) 2 (3.14)
Version intermédiaire en date du 26 novembre 2022

Furthermore, the filter responses are normalized with respect to the mask size. This guarantees a constant Frobenius norm for any filter size. Scale spaces are usually implemented as image pyramids. The images are repeatedly smoothed with a Gaussian and subsequently sub-sampled in order to achieve a higher level of the pyramid.

In order to localize interest points in the image and over scales, a non-maximum suppression in a 3×3×3 neighborhood is applied. The maximum of the determinant of the Hessian matrix are then interpolated in scale and in image space. Scale space interpolation is especially important in this case, as the difference in scale between the first layers of every octave is relatively large.

Orientation assignment : The proposed SURF descriptor is based on similar properties as SIFT, with a complexity stripped down even further. The first step consists in fixing a reproducible orientation based on information from a circular region around the interest point. SURF uses wavelet responses in horizontal and vertical direction (Figure 3.7)for a neighbourhood of size 6s, with s the scale at which the interest point was detected. Also the sampling step is scale dependent and chosen to be s. In keeping with the rest, also the wavelet responses are computed at that current scale s. Accordingly, at high scales the size of the wavelets is big. Therefore, they use again integral images for fast filtering. Only six operations are needed to compute the response in x or y direction at any scale. The side length of the wavelets is 4s. Once the wavelet responses are calculated and weighted with a Gaussian (σ = 2.5s) centered at the interest point, the responses are represented as vectors in a space with the horizontal response strength along the abscissa and the vertical response strength along the ordinate. The dominant orientation is estimated by calculating the sum of all responses within a sliding orientation window of angle 60 degrees. Interesting thing is that, wavelet response can be found out using integral images very easily at any scale. For many applications, rotation invariance is not required, so no need of finding this orientation, which speeds up the process. SURF provides such a functionality called Upright-SURF or U-SURF. It improves speed and is robust Version intermédiaire en date du 26 novembre 2022 up to ±15°. If it is 0, orientation is calculated. If it is 1, orientation is not calculated and it is faster.

Feature descriptor components : the first step consists in constructing a square region centered around the interest point, and oriented along the orientation selected in the previous section.

For the upright version, this transformation is not necessary. The size of this window is 20s.

Examples of such square regions are illustrated in Figure 3.7.

In detail, a neighbourhood of size 20s × 20s is taken around the key point where s is the size.

It is divided into 4 × 4 subregions. For each subregion, horizontal and vertical wavelet responses are taken and a vector is formed like this, v = ( d x , d y , |d x |, |d y |). This when represented as a vector gives SURF feature descriptor with a total of 64 dimensions. Lower the dimension, higher the speed of computation and matching, but provide lower distinctiveness of features. Another important improvement is the use of the sign of the Laplacian (trace of Hessian Matrix) for underlying interest point. It adds no computation cost since it is already computed during detection. The sign of the Laplacian distinguishes bright blobs on dark backgrounds from the reverse situation. In the matching stage, they only compare features if they have the same type of contrast. This minimal information allows for faster matching, without reducing the descriptor's performance.

Improving IOU Tracker with generated informations and SURF features

Intuitively, the more boxes we have, the more accurate the IOU Tracker is. Despite the strong performances of Mask R-CNN, the missing detections in some frames are inevitable. The technique of generating new segment can be similarly applied for generating new bounding boxes.

We proposed 2 methods of translation (Shift and Warp) with 2 directions (backward, forward) and 2 optical flow methods (LDOF, FullFlow), so we have 8 methods to generate bounding box.

Each box is generated by one of the 8 methods. We consider each box as a special mask which contains only four pixel corresponding to four vertices of box and then the process is repeated.

After applying generative approach, we get a situation where we have more overlapping boxes.

To eliminate those redundant boxes, we apply the idea of IOU tracker that takes into account the IOU between boxes. Let B tG denote the set of generated boxes b i tG and B tM denote the set of instance boxes b i tM directly detected by Mask R-CNN at frame I t (Figure 3.8). To combine the two sets, we compare the overlapping region with a threshold σ IOU by running the algorithm 1 :

Version intermédiaire en date du 26 novembre 2022 Algorithm 1 Eliminate overlapping boxes

for i = 1 → ||B tG || do count ← 0 for j = 1 → ||B tM || do if IOU (b i tG , b i tM ) ≥ σ IOU and class(b i tG ) = class(b i tG ) then count ← count + 1 end if end for if count = 0 then add b i tG to B tM else discard b i tG end if end for
It means that we trust more in the results provided by the Mask R-CNN detector (this assertion is based on the analysis of the experimental results described in section 5.1). After this step, we have only one set B t containing the box b i t of frame I t .

The next stage consists in applying the IOU Tracker. Each box b i t is compared with all the boxes from the L previous frames, where L is the length of the tracker. We associate a box to the previous box which obtains the same class and the maximum overlapping region. If we can not find any IOU value greater than the threshold σ IOU along all L frames, the current tracking process is terminated and we start a new track.

To enhance our tracker, we propose to use SURF [START_REF] Bay | Speeded-up robust features (surf)[END_REF] to verify the matching boxes. This feature have shown its efficiency on image matching problem. By applying SURF to match the similar boxes, we can avoid the fragmented trajectories. We extract SURF points for each bounding boxes.

Each box b i

t is compared with all boxes of L previous frames. If we find two boxes associated to two different classes but with their IOU value and the number of matching SURF points greater than threshold σ IOU and σ SU RF , respectively, we associate them together. To avoid the negative effect caused by extracting SURF features from a too small box, we set another threshold σ S to select the suitable boxes which exhibit a value greater than this threshold. For all the boxes with a value smaller than σ S , we do not extract SURF and we only take into account IOU value for the association step. The whole process is illustrated by Figure 3.8.

Version intermédiaire en date du 26 novembre 2022 t is compared with all boxes of L previous frames. If we find two boxes associated to two different classes but with their IOU value and the number of matching SURF points greater than threshold σ IOU and σ SU RF , respectively, we associate them together.

Experiments

We separately evaluate each stage using different datasets. First, we compare the techniques for generating objects segmentation stages using the DAVIS dataset [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF]. Then, we chose the most suitable methods to apply for the tracking stage. The qualitative evaluation regarding the improvement of the IOU Tracker is performed using the UA-DETRAC dataset [START_REF] Lyu | Ua-detrac 2017 : Report of avss2017 & iwt4s challenge on advanced traffic monitoring[END_REF]. Some samples of those datasets is illustrated in Figure 3.9. First, Mask R-CNN is applied to each frames. Next, we use LDOF [START_REF] Brox | Large displacement optical flow : Descriptor matching in variational motion estimation[END_REF] and Full Flow [START_REF] Chen | Full flow : Optical flow estimation by global optimization over regular grids[END_REF] to extract the optical flow vectors from pairs of frames. Then, we do the Shift and Warp translation in both backward and forward directions. Morphological operators are only applied as a postprocessing step with the Warp method. The extension beyond more than one step time that we call "combined results" in Figure 3.5c is also evaluated in both directions. Here, we choose the simplest case with N = 2 and j = 1. Our results with LDOF optical flow is illustrated in Table 3 sed. While Shift works stably for all of the methods, this is not the case with the Warp based approach. In a simple case, where we only consider optical flow within two consecutive frames in both direction, Warp significantly outperforms Shift. Conversely, when we take into account the development of optical flow along many frames, Shift gives us better results. This difference can be explained by the accumulated error of optical flow. The more frames we process, the more optical flow error we integrate. Shift is chosen to reduce this issue. Furthermore, we did not see any significant differences for each method when we performed LDOF or Full Flow optical flow estimation. Most of the average results in Table 3.3 are similar (Full Flow is slightly better) despite Full Flow significantly outperforms LDOF in the original task of optical flow estimation for both MPI Sintel and KITTI Dataset. We draw that the performance of generating new segmentation depends on how we use the flow vector, not on the type of flow. Therefore, our methods allow us to get a benefit regardless of the optical flow estimation methods, then fast methods are highly prioritized. On the other hand, we find that the original masks created by Mask R-CNN always give us better accuracy than the optical flow generated masks. This analysis suggests us to put more confidence in the discriminative results from Mask R-CNN than in the generative results from optical flow when we combine those results for the next tracking stages.

Qualitative evaluation :

Conclusion

In this chapter, we presented our proposed methods for improving vehicle tracking in detail.

The first section focuses on classical hand-crafted generative methods based on optical flow estimation while the second section presents anomaly detection based on future prediction framework. Generally, our first work is an extension of previous state-of-the-art segmentation and tracking frameworks for going beyond their limitations. In this part, our interesting add-on almost help them improving the qualitative performance aspect. We try to keep the source algorithms being stable then our extensions can be easily integrated into original algorithms.

The strengths of this first work is to provide several simple but interesting techniques to enhance the qualitative performances in some aspect of two classical tasks in computer vision. The computational complexity is quite low due to the hand-crafted solutions obtained without training or fine-tuning any deep network. In contrast, this work has limitations in term of proposing innovative framework. Besides, the possibility to reapply this methods as it is to specific task such as anomaly detection is not significant enough.

Version intermédiaire en date du 26 novembre 2022

Sequence 1a

Sequence 1b Sequence 2a Sequence 2b In this thesis, our contributions are two-fold. On the one hand, we propose a flexible multichannel framework to generate multi-type frame-level features. Our method is based on the prediction of the motion and the appearance of image streams and exploits errors of prediction to detect the abnormalities. We define a multi-channel framework based on Conditional GAN (CGAN) to produce feature maps for better-representing appearance and motion. On the other hand, we study how it is possible to improve the detection performance by supervised learning i.e. by adding some labeled data and a SVM based classifier.

In the first part of this chapter, we present the generalities about GAN and CGAN and introduce the pix2pix CGAN we implement in our architecture. In the second part, we present 67 how we add supervised strategy in the last step to improve the detection rate and how the objects related to the abnormality are localized in each frame. The last part is dedicated to the results of our experiments. We begin with the introduction of the public datasets and the evaluation metric we use. Next, we introduce our experimental results in both qualitative and quantitative evaluations with regards to the training parameters. Finally, we analyze our strengths and our limitations regarding to our results.

Version intermédiaire en date du 26 novembre 2022 We assume that a situation can be qualified as a normal or abnormal situation by analyzing the objects types moving in the scene and their dynamics. Both informations are carried by optical flow and appearance : thus, our architecture is predicting optical flow and image appearance. To detect abnormality in a sequence, we make the assumption that the prediction (output of the GAN) will sufficiently differ from the real image data for the abnormal situations. As introduced in figure 4.1, our architecture is based on conditional GAN that is a type of GAN that involves the conditional generation of images by a generator model [START_REF] Mirza | Conditional generative adversarial nets[END_REF]. CGAN is better adapted to our application because it uses additional information (annotated input data w.r.t. random noise input of GAN)to generate new predicted data. By training a CGAN on images acquired from "normal" situations, the generative model is able to predict the images and the optical flow at time t + 1 by inputting images and optical flow acquired at time t. In the first experimental section, we will show that our CGAN based backbone yields state-of-the-art performance.

From GAN to Conditional GAN

Generative Adversarial Nets [START_REF] Goodfellow | Generative adversarial nets[END_REF] were recently presented as an effective way to train generative models. For the purpose of taking advantage of labels during the training process, they feed the conditional data to both the generator and the discriminator. The term label here is not the ultimate abnormal label but the meaning of pixel of training images. This model is called

Conditional GAN (CGAN) [START_REF] Mirza | Conditional generative adversarial nets[END_REF], a conditional version of generative adversarial nets. min

GAN consists of two adversarial models

G max D V (D, G) = E x∼p g (x) [log D(x)] + E z∼p z (z) [log(1 -D(G(z)))] (4.1)
Generative adversarial nets can be extended to a conditional model by letting both the generator and the discriminator be conditioned on some extra information y. The extra y could be any kind of auxiliary information, such as class labels. They can perform the conditioning by feeding y into both the discriminator and the generator as additional input layer. In the generator the prior input noise p z (z) and y are combined in joint hidden representation, and the adversarial training framework allows for considerable flexibility in how this hidden representation is composed. In the discriminator x and y are presented as inputs and to a discriminative function.

In this conditional context, the objective function of a two-player minimax game would be as

Version intermédiaire en date du 26 novembre 2022 following :

min G max D V (D, G) = E x∼p g (x) [log D(x|y)] + E z∼p z (z) [log(1 -D(G(z|y)))] (4.2)
The Conditional GAN allows us not only to reconstruct from input noise to output image but also to extend to image-to-image translations. It means that we are capable of generating a target image which is not only expected to be similar to a training image but also expected to be similar to the transformation of a training image. This ability of CGAN is the key feature to help us constructing a framework of future predictions. We need the generated image to be similar to the future predicted image, not to the current one. For this reason, we choose CGAN as a suitable model.

Owing to its capacity to generate image, we a take U-Net architecture into account to construct the Generator G of the CGAN model. The U-Net architecture is shortly presented in the next section. The description of our network structure is presented in detail in section 4.1.3.

U-Net architecture

U-Net [START_REF] Ronneberger | U-net : Convolutional networks for biomedical image segmentation[END_REF] is the convolutional neural network that was developed by Ronneberger et al. for biomedical image segmentation. U-Net is based on the fully convolutional network (FCN) [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF] and its architecture was developed to work with less training samples and to obtain a higher segmentation quality.

By replacing pooling layers of usual CNN by upsampling operators, the main idea of U-Net is to supplement a usual contracting network by successive layers. Hence, these layers increase the resolution of the output. Furthermore, a successive convolutional layer can then learn to assemble a finer output based on this information.

There are a large number of feature channels in the upsampling part. This modification allows the network to propagate context information to higher-resolution layers. As a consequence, the expansive path is more or less symmetric to the contracting part and yields a u-shaped architecture (Figure 4.2). The network does not apply any fully connected layers but only uses the valid part of each convolution. The missing context is extrapolated by mirroring the input image to predict the pixels in the border region of the image. This tiling strategy is important to apply the network to large images, since otherwise the resolution would be limited by the GPU memory.

The U-Net architecture contains a contracting path and an expansive path which give it the U-shaped architecture. The contracting path is a typical convolutional neural network with successive typical blocks containing convolutions operations, each followed by a rectified linear unit (ReLU) and a max pooling operation. During the contraction, the spatial information is reduced while feature information is increased. In contrast, the expansive pathway joins the Version intermédiaire en date du 26 novembre 2022 feature and spatial information through a sequence of upsampling operations and concatenations with high-resolution features from the contracting path.

Fundamental of Pix2pix CGAN framework

Image-to-Image Translation with Conditional Adversarial Networks (pix2pix CGANs) is a generative model proposed by Philip Isola et al. [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF]. They investigated conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations.

For the network architecture of the generator, many previous solutions to the problems in this area have used an encoder-decoder network (Figure4.2). In such a network, the input was passed through a series of layers that progressively downsample, until a bottleneck layer, at which point the process is reversed. Such a network required that all information flow passes through all the layers, including the bottleneck. For many image translation problems, there is a great deal of low-level information shared between the input and output, and it would be desirable to shuttle this information directly across the net. For example, in the case of image colorization, the input and output share the location of prominent edges. To give the generator a mean to circumvent the bottleneck for information like this, they add skip connections, following the general shape of a "U-Net". Specifically, they add skip connections between each layer i and layer ni, where n is the total number of layers. Each skip connection simply concatenates all channels at layer i with those at layer ni.

Theoretically, GANs are generative models that learn a mapping from random noise vector x Version intermédiaire en date du 26 novembre 2022 to output image z, G : x → z. In contrast, conditional GANs learn a mapping from observed image y and random noise vector x, to z, G : x|y → z. The generator G is trained to produce outputs that cannot be distinguished from "real" images by an adversarially trained discriminator, D, which is trained to do as well as possible at detecting the generator's "fakes" (Figure 4.3).

The objective of a conditional GAN can be expressed as :

L cGAN (G, D) = E y,x [log D(x|y)] + E y,z [log(1 -D(x, G(z|y)))] (4.3)
where G tries to minimize this objective against an adversarial D that tries to maximize it, i.e.

G * = arg min G max D L cGAN (G, D).
To test the importance of conditioning the discriminator, we also compare to an unconditional variant in which the discriminator does not observe y :

L GAN (G, D) = E z [log D(z)] + E y,z [log(1 -D(G(z|y))] (4.4)
Previous approaches have found it beneficial to mix the GAN objective with a more traditional loss, such as L2 distance . The discriminator's job remains unchanged, but the generator is tasked to not only fool the discriminator but also to be near the ground truth output in an L2 sense. We

Version intermédiaire en date du 26 novembre 2022 also explore this option, using L1 distance rather than L2 as L1 encourages less blurring :

L(G) = E x,y,z [ z -G(x|y) 1 ] (4.5)
The final objective is The last CE512 block only has no batch normalization layer. All the convolutional layers of Encoder are based on 4 × 4 kernel filters with stride 2 to downsample the input source image to the bottleneck layer. Then, Decoder uses transpose convolutional layers for upsampling from bottleneck output size to the predicted output size. We also add skip connections between the layers of Encode-Decode corresponding to the same size of feature maps. The source image is considered as the input of the Generator and it is concatenated with the target image to produce the first input for the Discriminator. The output image of the Generator concatenated with the source image is fed to the Discriminator as second input. We apply L1 loss to measure the distance between target image I t and generated image I g from source image I :

G * = arg min G max D L cGAN (G, D) + λL L1 (G) ( 4
L G = E g I t -I g 1 (4.7)
The adversarial loss of Discriminator D is calculated using the Conditional GAN strategy (L cGAN (G, D)) : The final loss is the sum of both loss with ponderation factors λ G and λ D :

L D = E t log
L = λ D L D + λ G L G (4.9)
In our systems, we apply the default weights for λ G and λ D as in the original architecture of [START_REF] Pinkney | Pix2pix, matlab[END_REF].

Multi-channel pix2pix-CGAN : In order to achieve richer and more sensitive features of appearance and motion to the anomaly to detect, we propose 4 parallel pix2pix-CGANs with different input and output configuration as mentioned in Table 4.1. We separately investigate the temporal evolution of motion and appearance (in both grayscale and RGB format) by CGAN-1, CGAN-3 and CGAN-4 while CGAN-2 explores the relation between appearance and motion between two consecutive time stamps t and t + 1. About temporal length modeling, CGAN-1 Version intermédiaire en date du 26 novembre 2022

Channel

Input (I)

N i Output I g & Target I t N o = N t CGAN-1 G t , G t+1 2 F x,y t→t+1 2 CGAN-2 G t , F x,y t→t+1 3 G t+1 , F x,y t+1→t+2 3 CGAN-3 RGB t 3 RGB t+1 3 CGAN-4 G t , G t+1 2 G t+2 1
Tableau 4.1 -Configuration of input I, output (i.e. predicted images I g ) and target images I t for each pix2pix-CGAN stream. N i , N o , N t denote the dimension of the input, output and target image channels. G t , F

x,y t→t+1 and RGB t equal to grayscale image at frame t, optical flow from frame t to t + 1 along axis x, y and RGB color frame at frame t. All channels are taken into account for calculating loss functions. and CGAN-3 learn from the current frame to the next frame while CGAN-2 and CGAN-4 study the evolution from t to t + 2.

Before passing into CGAN streams, all source image channels are resized to 256 × 256 resolution. We also normalise optical flow maps along both x and y axis to range [0, 1] by the following equation : optimizer [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] with learning rate of 0.0002 and momentum parameter β 1 = 0.5, β 2 = 0.999 for both the Generator and the Discriminator. We test with different activation functions (LeakyReLU, ClippedReLU, eLU) and other hyper-parameters such as mini-batch size to find the most suitable parameters values for each stream and dataset.

F norm m,n =            0.5 -F m,n × 0.5 F min , for F m,n 0 F m,n × 0.5 F max + 0.

Feature extraction with PSNR

Each CGAN predicts the outputs for which it has been trained as described in the table 4.1.

Then it is possible to measure the difference between the predicted output and the target data to decide if the situation is different of what it should be. Generally, in order to measure the distance between two images, we can use two metrics : Mean Square Error (MSE) and Peak Signal to Noise Ratio (PSNR). In signal processing, PSNR is an expression defined by the ratio between the maximum possible power of a signal and the power of distorting noise that affects the quality of its representation. In statistics, the mean squared error (MSE) or mean squared deviation (MSD) of a predictor or an estimator equals to the average of the squares of the errors i.e. the average squared difference between the predicted/estimated values and the real value.

Version intermédiaire en date du 26 novembre 2022

Given the original image I t containing N pixels without noise and its noisy version I g , MSE and PSNR are calculated as :

MSE(I, I s ) = 1 N N i=0 (I i t -I i g ) 2 (4.11) P SN R(I t , I g ) = [I i max ] 2 MSE(I t , I g ) (4.12)
where we denote I i max as the maximum value of a pixel of original image I. The work of Mathieu et al. [START_REF] Mathieu | Deep multi-scale video prediction beyond mean square error[END_REF] shows that PSNR is a promising way to compare the quality of a target image I t with a generated I g , a higher PSNR value indicating a generated image conformed to the target image we reach. The authors define the PSNR between I t and I g from equation 4.12 but in db as : dimensions in all videos sequences. By considering all the training sequences, we obtain a feature map that defines a sub-space that corresponds to the 9-dimensional vectors related to normal situations. Consequently, each new features vector of this sub-space (i.e. a vector obtained from an unseen image sequence), could be linked to a normal event.

P SN R(I t , I g )(dB) = 10 log 10 [max(I t )] 2 1 N N i=0 (I i t -I i g ) 2

CGANs backbone loss evaluation

In this section, we analyse the quality of the training step through 4 popular datasets used for anomaly detection in the state-of-the-art : CUHK Avenue [START_REF] Lu | Abnormal event detection at 150 fps in matlab[END_REF], USCD Pedestrian 1 and 2 [START_REF] Mahadevan | Anomaly detection in crowded scenes[END_REF] and ShanghaiTech [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF]. Some samples of those datasets are illustrated in Several outlier frames exist in Avenue training set as presented in [START_REF] Lu | Abnormal event detection at 150 fps in matlab[END_REF]. All the state-of-the-art works that use this dataset [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF][START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF] have manually removed these outliers to avoid training the CGANs for normal cases with abnormal input data. We apply this task to be fair for comparison.

USCD Ped1 & Ped2 : Ped1 has 34 training and 36 testing videos with 40 abnormal events. Ped2 is smaller with 16 training and 12 testing videos. Almost abnormal events are related to moving vehicles. Ped1 seems to be more challenging than Ped2 due to the different camera angles used. Both have more abnormal events than normal ones in their test sets.

ShanghaiTech : This is a very large benchmark containing 13 scenes integrating complex lightning conditions and camera angles. There are 130 abnormal events and over 270000 training frames. Moreover, pixel level ground truth of abnormal events is also annotated. Normal samples are more numerous than abnormal ones in the test set. Loss evaluation during the CGAN training : For each CGAN stream, we investigate the effect of the Mini-batch size (M), the activation function (A) and the number of training epochs (E). We start from E = 20 for Avenue, Ped1 and Ped2 with M = {32, 64, 128, 256} and A = {leakyRELU (α = 0.2), clippedReLU (α = 0.5), eLU (α = 1)} . Then we increase E up to 30, 40 and observe the convergence of each loss function to choose the most suitable parameters according to the balance between time consumption and loss convergences. An example is illustrated on Outliers removing on Avenue : As previously presented, some outlier frames have been manually removed from the training set to avoid degrading loss function convergence. As a first feasibility evaluation to detect abnormality, we use the trained CGAN-2 to automatically detect these outliers. For that, we accumulate MSE values calculated between the predicted output of Version intermédiaire en date du 26 novembre 2022 the CGAN-2 and the target data for the 3 channels (i.e. the 2-dimensional optical flow and the gray image) and we draw the values as presented on figure Figure 4.11. We can easily observe some peaks occurring exactly at the abnormal frames appearing on the training videos and that have been manually removed. This result shows clearly that CGAN-2 is able to highlight parts of a sequence that differ from the normal cases for which the CGAN-2 has been trained. But if the CGAN-2 has been sufficient for detecting the outliers it is because, it has been trained on the same sequence in which we aim at detecting the outliers. In an operational context, the goal is to detect the abnormalities in an unknown image sequence. To propose a better classification model, we have to analyse the distribution of features vectors for normal and abnormal frames. This analysis is presented in the next section. Ped1 and Ped2 datasets contain only grayscale images. Thus, CGAN-3 that is based on RGB images is reduced to gray level content and we observe a straight line distribution. All other points plots respect the dimension given in table 4.1.

For all plots we observe that two clusters appear in the distributions. Unfortunately, it is clear that both of these clusters are not linearly separable in this raw space : normal and abnormal points distribution partly occupied the same part of the space. The multi-channel CGANs backbone defines a feature space in which the images content is projected and in which abnormality can be detected. Generally, the SOTA proposed two natural approaches for assigning abnormal labels for image samples as described in works applying future prediction approach [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF][START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Liu | Margin learning embedded prediction for video anomaly detection with a few anomalies[END_REF][START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF][START_REF] Vu | Robust anomaly detection in videos using multilevel representations[END_REF]. On the one hand, an abnormal score is computed by thresholding the error map [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF][START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Vu | Robust anomaly detection in videos using multilevel representations[END_REF]. On the other hand, authors propose to use learning model to classify the input into normal and abnormal classes [START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF][START_REF] Liu | Margin learning embedded prediction for video anomaly detection with a few anomalies[END_REF]. Table 4.2 shows the performance for both kind of methods. It clearly appears that the methods based on learning classifiers yield better results even on challenging dataset such as Avenue and ShanghaiTech.

In our work, we propose to train a Support Vector Machine to separate the normal and abnormal features vectors. From this point, we leave the unsupervised learning to adopt a semi-supervised strategy. The next section describes this part of our pipeline and the evaluation results we obtained for all the selected datasets.

Version intermédiaire en date du 26 novembre 2022

Abnormality detection by Support Vector Machine

In this second part of the work, we describe how we can perform abnormality detection from the PSNR based feature map inferred from our multi-CGANs backbone. The analysis of the distribution proposed in the previous section clearly shows that both class cannot be linearly separated. Thus, we proposed to use a kernel based technique to discriminate the two classes. In this work, we propose to use Support Vector Machine (SVM) by following a supervised strategy.

A binary SVM is used as the final layer to design the supervised classifier. It takes the feature maps provided by our network backbone as input after a PSNR transformation. In this section we show that a SVM is able to accurately detect the abnormal frames of a video. We go further and our experimental results will show how the accuracy is improved with regards to the size of Version intermédiaire en date du 26 novembre 2022 the dataset used to train the SVM. We finally propose a framework to localise the objects of the scene that generate the detected anomaly.

The next two sub-sections are respectively dedicated to the description of this SVM layer (section 4.2.1) and of the abnormal object localisation ( (section 4.2.2)). In the last sub-section (section 4.2.3) are presented the quantitative evaluation of the SVM based abnormality detection and the quantitative evaluation of abnormal object localisation task. 0.87 0.96 Vu et al. [START_REF] Vu | Robust anomaly detection in videos using multilevel representations[END_REF] 0.72 0.82 0.99

SVM based frame-level anomaly detection

Learning classifiers methods

Ionescu et al. [START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF] 0.90 0.98 0.85 Liu et al. [START_REF] Liu | Margin learning embedded prediction for video anomaly detection with a few anomalies[END_REF] 0.93 0.77 Tableau 4.2 -Comparison of Frame level AUC on 4 datasets between simple thresholding inference models and complex learning inference models. All reported methods constructed their features space by future prediction networks. In detail, by extracting a part of the test set from one dataset, we have two typical classes for each frame : one frame can be normal or abnormal. Each frame is represented by its features vector i.e. the PSNR feature vector extracted from the output of the multi-CGANs backbone.

The full size of features vector is 9-dimensional corresponding to the 9-streams of CGANs when all the streams are considered. During the experiments, we analyse the performance by using each stream separately and by combining them. In practice, the feature vector can be from 1-dimensional to 9-dimensional vector depending on which combinations are evaluated. Before training the binary SVM classifier, all features vectors are normalised.

Support Vector Machine description

Theoretically, the SVM based binary classification algorithm allows to categorise an unseen object into two separate groups. This classification is based on the properties of the objects and of a set of known samples, which are already classified and that has been used to construct the SVM model. The properties are defined by a n-dimensional features vector, and the model is achieved by creating a linear partition of the feature space into the two classes. An unseen sample is classified by calculating the position of the features vector relatively to the linear model. SVM is not only restricted to linear the discriminative problem. Thanks to a technique known as the kernel trick, it is possible to estimate various types of non-linear decision boundaries. This trick performs a mapping of the original feature space to a higher-dimensional space, in which the problem of binary classification becomes near a linearly separable one. However, even if a kernel is applied, the boundary is always defined in the original space by the training set of feature vectors.

In a n-dimensional space the SVM algorithm constructs an optimal hyperplane that discriminates the samples from two classes, and the optimal hyperplane equation is given by :

xβ + b = 0 (4.14) where :
x is a sample vector β contains the coefficients that define an orthogonal vector to the hyperplane.

b is the bias term This hyperplane splits the feature space into two parts for which the sign of the function f (x) = xβ + b is negative or positive. Once the model is obtained, classifying a new sample consists in determining the part of the space where its feature vector is located.

The SVM model, defined by parameters β and b, is estimated by applying a training algorithm on a set of positive and negative training samples ( x i , y i ) where x i is the feature vector of the i th sample. y i is equal to 1 or -1 with regards to the class to which the sample belongs.

Version intermédiaire en date du 26 novembre 2022

The training algorithm finds and sets up the maximum margin length while keeping samples in the positive (y = 1) or negative (y = -1) part of the space. The optimal hyperplane is separating the two sets of samples such as it is the farthest from any training observations. The smallest perpendicular distance to a training observation from the hyperplane is known as the margin.

The optimal hyperplane depends on the margin defined by a subset of the positive and negative training samples known as support vectors. It appears clearly that the optimal hyperplane and thus the classification accuracy depends on the support vectors.

The SVM training is based on an optimization and relies on a primal and a dual formalization of the SVM problem.

-For separable samples, the objective function is defined by solving arg min β with respect to the β and b such that y j f (x j ) ≥ 1, for all j = 1, .., n.

-For non-separable samples, the objective function is defined by solving arg min(0.5 β 2 + C θ j ) with respect to the β, b, and θ j such that y j f (x j ) ≥ 1θ j and θ j ≥ 0 for all j = 1, .., n and for a positive scalar box constraint C. The algorithm applies slack variables (θ j ) to penalize the objective function for samples that cross the margin boundary for their class. θ j = 0 for samples that do not cross the margin boundary for their class, otherwise θ j ≥ 0.

The algorithm applies the Lagrange multipliers to optimize the objective function, which presents n coefficients α 1 , .., α n . For both separable and non-separable cases, the dual formalization for a linear SVM is as follows :

-For separable samples, L = arg min(0.5

n j=1 n k=1 α j α k y j y k x j x k - n j=1 α j ) (4.15)
w.r.t. α 1 , .., α n subject to α j y j = 0, α j ≥ 0 for all j = 1, .., n and Karush-Kuhn-Tucker (KKT) Complementarity Conditions.

-For non-separable samples, the objective function is similar to separable classes, except for the additional condition 0 ≤ α j ≤ C for all j = 1, .., n.

The final resulting score function is :

f (x) = n j=1 αj y j xx j + b (4.16)
where b is the estimate of the bias and αj is the j th estimate of the vector α for j = 1, ..., n. Written this way, the score function is free of the estimate of β as a result of the primal formalization.

When classes discrimination requires the non-linear boundary, SVM is looking for an optimal separating hyperplane but in a transformed predictor space obtained by applying a kernel trick.

Version intermédiaire en date du 26 novembre 2022

This extension to the non-linear case results in increasing the feature space dimension through the use of functions known as kernels φ. For nonlinear SVM, the algorithm constructs a Gram matrix using the rows of the predictor data X. The dual formalization replaces the inner product of the observations in X with corresponding elements of the resulting Gram matrix. The Gram matrix of a set of n vectors x 1 , .., x n ; x j ∈ R p is an nbyn matrix with element (j, k) defined as G(x j , x k ) =< φ(x j ), φ(x k ) >, an inner product of the transformed predictors using the kernel function φ. Various kernel functions exist like the following :

-Gaussian kernel : G(x j , x k ) = exp(-

x j -x k 2 2σ 2 ) -Radial basis function for γ > 0 : G(x j , x k ) = exp(-γ x j -x k 2 )
-Linear kernel :

G(x j , x k ) = x j x k + c -Polynomial kernel for integer q > 0 : G(x j , x k ) = (αx j x k + c) q .
-Sigmoid kernel : G(x j , y k ) = tanh(αx j x k + c)

In this case, the dual formalization for nonlinear SVM is :

L = arg min(0.5 n j=1 n k=1 α j α k y j y k G(x j , x k ) - n j=1 α j ) (4.17)
w.r.t. α 1 , .., α n subject to α j y j = 0, 0 ≤ α j ≤ C for all j = 1, .., n and Karush-Kuhn-Tucker (KKT)

Complementarity Conditions. G(x j , x k ) are elements of Gram matrix. By this way, the final resulting score function is :

f (x) = n j=1 αj y j G(x, x j ) + b (4.18)
The KKT complementarity conditions are optimization constraints required for optimal nonlinear programming solutions. In SVM, the KKT complementarity conditions are :

       α j [y j f (x j ) -1 + θ j ] = 0 θ j (C -α j ) = 0 (4.19)
for all j = 1, .., n where f (x j ) = φ(x j )β + b with φ(x j ) is a kernel function and θ j is a slack variable. If the classes are fully distinguishable, then θ j = 0 for all j = 1, .., n.

As it will be presented in the next section about evaluation results, we have evaluated different kernels to reach the best results. To enhance performance of SVM classifiers, we also apply kf old cross-validation strategy. This strategy proposed to divide the training dataset into k subsets and to apply the following procedure for each of the k subsets :

-A model is trained using k -1 of the folds as training data ;

-The model performance is evaluated on the remaining subset (i.e. the one that has not been used for training)

The performance measure reported by k-fold cross-validation is then the average of the values Version intermédiaire en date du 26 novembre 2022 computed in the loop. In our experiment, we set k = 5.

Abnormality object localisation

The objective of this last task is to localise the object related to the abnormal event detected at the frame-level. For each abnormal frame, we run a fast detector to compute bounding boxes (BB) for each object in the sequence. Each BB is considered as a new input and a PSNR score is computed for each one by taking the normal sum of 9 PSNR scores corresponding to 9 channels. To decide if one BB is or not one object that is producing the anomaly, we assume that heat region appears at the position of the BB in the error map. Thus, the BBs yielding the minimum values for PSNR scores or values smaller than a threshold refer to the abnormality objects.

After calculating PSNR score for each CGAN stream to encode feature vectors, we learn binary SVM classifier using the Classification Learner Toolbox on Matlab.

To reduce the computational complexity in practical experiments, we do not re-pass each BB throughout the whole framework but we apply directly each BB to the error maps. The error map of each frame is the subtraction of the generated images (optical flow and gray or color images) and the corresponding target frames as illustrated in Figure 4.17.

We compare our method with the state-of-the-art object-centric model (Figure 2.17 candidates BBox obtaining minimum PSNR scores. It is based on the fact that the key object in our real-world scenario is abnormal one, so the false positive decision is more acceptable than false negative one. We could also go beyond this limitation by searching for a soft threshold of PSNR score.

During evaluation, we apply the Mask R-CNN [START_REF] He | Mask r-cnn[END_REF] as the object detector. This model is based on a Resnet101 architecture trained on Imagenet dataset. All steps are implemented on Matlab with Nvidia GeForce GTX 1080.

Evaluation results

Evaluation metrics

As in the literature, we use frame-level Area-Under-Curve (AUC) metric as main measurement for quantitative evaluation and fair comparison with state-of-the-art methods. The type of AUC that we applied is AUC of Receiver Operating Characteristic (ROC) curve. AUC of ROC curve is an evaluation measurement for the classification problems at various threshold settings.

ROC curve is a probability curve and AUC represents the degree or measure of the classifier Version intermédiaire en date du 26 novembre 2022 performance. AUC introduces how much the model is capable of separating between classes.

The higher the AUC, the better the model at predicting negative sample as negative and positive sample as positive. The ROC curve is plotted with True Positive Rate (TPR) against the False Positive Rate (FPR) where TPR is on the y-axis and FPR is on the x-axis ( Besides, we propose some alternative evaluation metrics that are more adaptive to object- where B 50 and B 75 denote the boxes that have the IOU with ground-truth boxes greater than 50% and 75%. For mIOU , there are two possible ways to treat the true negative frames (i.e.

detected box = ground-truth box = 0). One way, we remove true negative frames and report only true positive cases. Other way, we set IOU = 1 for each true negative frame.

Evaluation of SVM based abnormality detection

In this section, we present the evaluation of the supervised part of our solution. We recall that the proposed solution is based on a multi CGANs backbone whose outputs are used to decide if the input frame is containing an anomaly or not. The network learning is based on a semi-supervised strategy i.e. the network weights and the PSNR based decision is trained with only normal image sequences.

To train the supervised SVM of our architecture, the training set has to contain anomaly samples. Only the test parts of the used datasets contain the required positive samples. To be fair during evaluation, we split the test set of each of the datasets into two subsets respectively called SVM-train and SVM-test set for respectively the training and the inference phase. To explore the effect of the supervised scenario on the performance, SVM-train is built by extracting from 10% up to 80% of the samples of the test set, the SVM-test being the remaining samples.

As proposed at the end of the section 4.2.1, we optimise the classifier with regards to the used kernel function i.e. we run the optimisation process on the SVM learner with the following kernel functions : polynomial, gaussian and linear functions. Thus, the SVM-train is splitted into a train and a validation set for 5-fold cross-validation. Similarly to multi-CGANs backbone evaluation, for ShanghaiTech dataset, we run the optimisation process only once when SVM-train equal 80%. Then we apply optimised parameters for the rest of the evaluation (i.e. from 10% to 80%).

Version intermédiaire en date du 26 novembre 2022

As for multi-CGANs backbone, the effectiveness of the supervised layer is measured by the value of AUC. By sequentially increasing the SVM-train set size from 10% of the original test set size up to 80%, we report the evolution of performance on Figure 4.19. We clearly observe that the larger the size of the SVM-train set we create, the better the performance we obtain. We compare our best results for each benchmark with recent state-of-the-art methods in Thanks to the optimisation, we sharply reduce the minimum classification errors after a certain number of iterations. We also achieve best results for each dataset using various set of parameters.

After having the optimised model, we draw the ROC curve and calculate AUC performance.

Results are illustrated in Figure 4. [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] Almost all normal samples situate in higher values than abnormal ones. This distribution corresponds to the fact that PSNR techniques produce high scores if predicted images tend to be similar to the source images.

MSE and PSNR comparison :

We do the first experiments to evaluate the performance of feature encoding using PSNR and MSE. We choose Avenue benchmark as the reference dataset because it involves color images with acceptable dataset size (Pedestrian dataset have only grayscale frames and ShanghaiTech is too large). Results are illustrated in Table 4 4.4. On two grayscale datasets Ped1 and Ped2, CGAN-3 and CGAN-4 produce almost the same performance. The longer temporal horizon of prediction of CGAN-4 brings us a small improvement 1∼2% with respect to CGAN-3. On the other RGB datasets, CGAN-3 surpasses CGAN-4 with a huge difference of 10%. Generally, CGAN-2 achieves best performance among the 4 streams. It shows that learning both appearance and motion evolution can help us generating better features. Obviously, the combination of CGAN-1 (flow) and CGAN-4 (grayscale) produces similar or slightly better performance than CGAN-2 (combines flow with grayscale). This result also proves that taking into account longer temporal horizon of prediction, as CGAN-4, improves performance. Besides, when several streams are combined, better performance is achieved. A four streams combination always produces the best results : that is a strong experimental proof of the relevance of our proposed idea about multi-channel framework.

Evaluation of abnormality objects localization

Quantitative evaluation : We evaluate the quantitative performance of our abnormality object localization models on Avenue dataset. We use AUC and ERR metrics at pixel-level. The obtained results are reported in Table 4.6. Most of the state-of-the-art researches [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF][START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF] have not reported quantitative performance for abnormality localization task. These authors only propose a qualitative analysis to show that the error maps are relevant i.e. heat scores are located where abnormality objects are localised. Hence, we compare our quantitative results with recent methods of Vu et al. [START_REF] Vu | Energybased localized anomaly detection in video surveillance[END_REF][START_REF] Vu | Robust anomaly detection in videos using multilevel representations[END_REF] that applied the same evaluation metrics.

We achieve significant improvements on both metrics. We also report our performance on Single abnormal object localization : Our method achieves promising performance when a single object is generating the anomaly. We investigate the bounding box for which the PSNR score is minimum. Figure 4.36 shows the qualitative results of single abnormal object localization on Avenue dataset. Each row represents a type of abnormality : running, dancing, throwing object, moving to wrong direction. We can exactly detect the abnormal object in every frame.

IOU
Due to the inaccuracy of Mask R-CNN, the bounding boxes are not perfect in some cases, e.g. 4 th image of the first and the second row, and the 2 nd image of the third row. Generally, we can exactly detect and localise the abnormal object in various types of abnormality : running, jumping, dancing, throwing object, moving to wrong direction, abnormal vehicles, etc. Interestingly, our method performs pretty well when occlusion occurs and in overlapped scenarios. In contrast, there are also some flaws related to the size and position of bounding boxes. Sometimes, they are too large or too small. In several other cases, the boxes only belong to parts of abnormal object, e.g. the umbrella existing in figure 4 Multiple abnormal objects localization : In the case of existing multiple abnormal objects, the maximum number of objects to detect is 2. It means that we report the minimum and second minimum PSNR objects. existing drawback of Mask R-CNN (i.e. bounding boxes are too large or too small), the impact of motion score can be spread out and causes the effect on neighbour objects. By this way, the boxes near the first object tend to obtain lower PSNR score than the second real abnormality object.

On the other hand, overlapped objects are detected by Mask R-CNN at a same location, e.g. bicycle and person in pedestrian samples. Obviously, both objects usually obtain lower PSNR score than the rest. Therefore, our algorithm chooses the two objects at the same location instead of searching for a new position. appear when abnormality objects are too close to the other normal boxes : the PSNR score of those boxes is affected by the features of the near abnormality object. Other common errors are directly caused by false negative abnormality of the frame-level detection. Generally, those problems come from two reasons. On the one side, SVM cannot classify frames as abnormal and produces false negatives : for those frames the object detector is not applied. On the other side, some frames are true negatives (i.e. SVM is correct) but the PSNR score is not relevant enough for localising the abnormality objects. A promising solution is proposing an adaptive thresholding rather than the hard-decision strategy.

Conclusions and discussions

In this chapter, a CGAN based SVM anomaly detector is proposed. The multi-CGANs backbone predicts the future information : it is based on four CGANs taking various types of appearance and motion information as input and producing prediction of similar information as Version intermédiaire en date du 26 novembre 2022 output. These CGANs combined to PSNR metric provide a relevant feature space to discriminate normal and abnormal events. The discriminative model is based on a Support Vector Machine trained following a classical supervised strategy. While the binary SVM is applied for frame-level anomaly detection, we propose an anomaly object localisation based on a Mask R-CNN detector.

Our evaluations highlight our good achievements for anomaly detection on 4 reference datasets. Our experiments show that we achieve very promising results although we have not optimised all of the steps. For example, Full-Flow [START_REF] Chen | Full flow : Optical flow estimation by global optimization over regular grids[END_REF] and Lucas-Kanade [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF] could be replaced by recent SOTA methods to reach better performance. dataset. Almost all of the normal samples situate in higher values than abnormal ones. This distribution corresponds to the fact that PSNR techniques produce high scores if predicted images tend to be similar to the source images. computational complexity, our CGANs stage has the same complexity as the SOTA. The inference stage has moderate complexity : higher than [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF] using simple threshold but more simple than [START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF] using very high-dimension features classified by K-means with multi-class SVM.

Limitations : On the one hand, the mix of unsupervised CGAN streams with supervised SVM helps us obtain impressive performances on off-line cases where all test data are prepared and well pre-processed. For online works, we will get troubles when brand-new abnormal actions suddenly occur without existing in the previous learned databases. On the other hand, the anomaly object localisation is significantly affected by the detector we use (Figure 4.41).

Sometimes, bounding boxes is imperfectly generated regarding its size and its labels. We can go beyond this limitation by not applying a high-level detector such as Mask R-CNN but only a low-level region proposal network. Moreover, the proposed framework is not an end-to-end network structure at the moment. An end-to-end approach will be more globally optimal by avoiding several optimisation process done locally.

Version intermédiaire en date du 26 novembre 2022 Version intermédiaire en date du 26 novembre 2022

Chapitre 5

Conclusion and Future works

This chapter summarises the essentials of my PhD thesis as well as presents some potential ideas for future works. The first section is dedicated to conclude the thesis. We briefly remind the general context and objectives of this project and synthesize our remarkable achievements for our proposed methods. The second section introduces what we plan to do to improve some limitations of our current works and to develop our frameworks to real-world applications.

Conclusion

This PhD thesis addresses two challenging problems in computer vision for the transportation applications : (1) vehicles and road users segmentation and tracking and (2) anomaly detection.

By largely reviewing state-of-the-art methods for each task, we decide to follow the future prediction approaches through two possible scenarios. The first one deals with the classical hand-crafted generative methods based directly on optical flow estimation while the second one applies deep learning CGANs model to build a multi-channel generative framework that learned competitive features from both appearance and motion information.

For hand-crafted strategy, our purpose is to evaluate the capability of classical hand-crafted generative methods for improving segmentation and tracking. About the second strategy, we aim at making the most with the available deep generative model to improve SOTA performances obtained for various public datasets. The innovation relies on the flexibility of our model that is based on a combination of several conditional GAN, accepting various types of input information (i.e. RGB, grayscale and optical flow images). By producing features suitable for various classification techniques like Support Vector Machine, our model can be easily extended or lightened for further developments. 121

Improving segmentation and tracking by classical hand-crafted methods

Our first research is an initial work for evaluating the performance of the classical handcrafted generative approach in future prediction and its capacity to improve segmentation and tracking of moving objects. By searching for a strong baseline among state-of-the-art methods, Mask R-CNN [START_REF] He | Mask r-cnn[END_REF] has been chosen because of the promising performance for detection and instance segmentation of the objects. Tracking-by-detection approach has been adopted for tracking step. We focus on the impressive speeding-up performance of the IoU tracker that takes into account only the Intersection-Over-Union (IOU) between bounding boxes to match objects within consecutive frames. On the contrary, because this method does not use other visual information [START_REF] Bochinski | High-speed tracking-by-detection without using image information[END_REF] and because of the failure detection of Mask R-CNN detectors, fragmented objects trajectories appear.

We propose to fill broken trajectories by predicting instance segmentation using Optical flow and then propose an enhanced tracker based on connecting fragmented trajectories by SURF feature [START_REF] Bay | Speeded-up robust features (surf)[END_REF].

Our solution first generates new detections or instance masks by translating backward and forward current information using optical flow vectors. We extend two methods proposed by [START_REF] Luc | Predicting Future Instance Segmentation by Forecasting Convolutional Features[END_REF] : Shift and Warp. The predicted/generated masks obtained by Warp method are then denoised by morphological operations. After this stage, the gaps of trajectories can be filled in. We use DAVIS dataset [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF] with two popular classes of vehicles (bus and car) for evaluating the quantitative and qualitative performance of generating new detections. The quantitative results show that Shift-translation outperforms Warp-translation for long trajectories while

Warp-translation integrated with morphological is more adaptive with short tracking. It also confirms the necessity of morphological operators for post-processing. The qualitative results

show that our solution achieved stable performance with different types of flow estimation methods.

Generated segmentation and bounding boxes are matched along all the sequence by using SURF

features. An IOU-matching algorithm is proposed to automatically fill fragmented parts of the trajectories. The entire process is appliued on DETRAC dataset [START_REF] Lyu | Ua-detrac 2017 : Report of avss2017 & iwt4s challenge on advanced traffic monitoring[END_REF]. We only realize a qualitative evaluation. At this stage of the work, quantitative evaluation was not required. The qualitative results show that our methods significantly improve the fragmented trajectories in particular sequences.

Anomaly detection by multi-channel deep generative neural networks

Although the results obtained for filling fragmented trajectories seems to be usable for easy tasks, for anomaly detection, the obtained performance is not sufficient. Obviously, there are important differences between anomaly detection and general action recognition problem. Firstly, the abnormal events rarely appear unlike normal event : it leads to an unbalanced scenario where Version intermédiaire en date du 26 novembre 2022

Conclusion 123

the number of samples in each class is significantly different. Secondly, the features of abnormal events usually do not necessarily follow any spatial or temporal relationship so that it raises the difficulty of pre-defining the structure or class of abnormal events. In order to tackle those challenges, we propose to train a generative model based on the observation of the apparent motion and appearance for the normal scenes. This model computes predicted images from past images. The abnormal situations are then detected by using error estimation between the generated output of the model and both appearance and optical flow provided by the sequence at the same time.

The proposed network is based on a multi pix-2-pix Conditional GAN (CGAN) [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] architecture with U-Net model. The Generator is an Encode-Decode network for generating future information based on appearance and optical flow images at previous time. The Discriminator tries to classify whether the generated samples are fake. We evaluate a four CGAN streams : each CGAN takes various types of appearance and motion information as input for which it produces a future prediction.

This multi-CGAN backbone provides a better feature space to project the specificities of normal and abnormal events. The SOTA proposed to infer the nature of event by unsupervised function. We decide to do that by using a supervised Support Vector Machine trained on the multi-outputs of the network encoded by Peak Signal-to Noise Ratio. The binary Support Vector Machine (SVM) is applied for frame-level anomaly detection. Finally, we jointly use a Mask R-CNN detector and the mutli-CGAN backbone to localize the obnormality object in each abnormal frame.

Our methods are largely evaluated on CUHK Avenue [START_REF] Lu | Abnormal event detection at 150 fps in matlab[END_REF], USCD Pedestrians [START_REF] Mahadevan | Anomaly detection in crowded scenes[END_REF] and Shan-ghaiTech [START_REF] Liu | Future frame prediction for anomaly detection -a new baseline[END_REF] datasets. Our experiment results demonstrate that PSNR features are better than mean square error maps computed by previous methods. For the strong performance objective, we achieve state-of-the-art frame-level AUC on Avenue, Ped1 and ShanghaiTech. Especially, for the most challenging Shanghaitech dataset, a supervised training model outperforms up to 9% the state-of-the-art on unsupervised strategy. We achieve those impressive results without optimization all steps in the pipeline, particularly optical flow extraction. We apply a very simple flow estimator pre-installed from OpenCV and classical Full-Flow [START_REF] Chen | Full flow : Optical flow estimation by global optimization over regular grids[END_REF] estimator. For the flexibility objective, we confirm the efficiency of integrating more channels because the performances of the unified network are significantly surpassing every single channel's result. Our model is also free to extend or lighten without any exigency due to the independent architecture of each channel. Each combination of various channels can be taken into account for different purposes.

Our features are quite simple and suitable for many classification models in the inference phase.

Version intermédiaire en date du 26 novembre 2022

Future works

We always keep our work in progress for improving both problems : vehicle segmentationtracking and anomaly detection. There are many promising tasks to continue in perspectives :

replacing classical optical flow estimators with recent state-of-the-art models, constructing a new dataset for anomaly detection, re-applying CGANs for the first problem, installing an end-to-end network for the second problem, etc. We discuss these prospects in the following parts.

Optimization of Optical flow extraction : Although we have evaluated LDOF [START_REF] Brox | Large displacement optical flow : Descriptor matching in variational motion estimation[END_REF] and Full-Flow [START_REF] Chen | Full flow : Optical flow estimation by global optimization over regular grids[END_REF] for the first problem to confirm the independence of our methods regarding the type of flow, the second problem is still waiting for a better optical flow estimator. In both cases, all optical flow methods are classical and hand-crafted. Actually, there are many state-of-the-art models using deep learning techniques e.g. FlowNet-2 [START_REF] Ilg | Flownet 2.0 : Evolution of optical flow estimation with deep networks[END_REF], PwC-Net [START_REF] Sun | Pwc-net : Cnns for optical flow using pyramid, warping, and cost volume[END_REF], etc. Intuitively, the better the flow vectors are, the higher accuracy we could finally achieve. The first improvement could be to use one of these optical flow networks to infer the motion content for the multi-CGAN network. We will have to define if the output of the optical flow net is sufficiently accurate for anomaly detection.

Semi-supervised Dataset for Anomaly Detection : Actually, we evaluated our anomaly detection method for the unsupervised scenarios. Multi-CGAN backbone is trained with only normal samples while the SVM classifier is trained on a dataset containing both types obtained by splitting the test dataset. For unsupervised anomaly benchmark, this dataset is only used for the final evaluation. Even if we don't use the training part of the test dataset to blindly evaluate the model, to be completely fair we plan to build a new dataset adapted for semi-supervised scenarios. Moreover, one next step for evaluation will be to apply our methods on more datasets [START_REF] Sultani | Real-world anomaly detection in surveillance videos[END_REF][START_REF] Hassner | Violent flows : Real-time detection of violent crowd behavior[END_REF][START_REF] Leyva | The lv dataset : A realistic surveillance video dataset for abnormal event detection[END_REF] that are more suitable for the supervised scenarios.

CGANs for Improving Segmentation and Tracking : A natural prospect for the first work is re-apply CGANs model of the second work for generating missing segmentations or bounding boxes in broken trajectories. There is always a trade-off here. On the one side, the combining of two strong deep learning models (i.e. Mask R-CNN and CGANs) can sharply increase qualitative and quantitative performance. On other side, we might consume much more time running for those deep networks. We have to deal with the balance between speed and accuracy.

From binary SVM to one-class SVM : For the second work, we also plan to evaluate our framework in the unsupervised scenario. Obviously, we could almost keep all parts of our model except the inference phase based on supervised binary SVM. By replacing binary SVM with one-class SVM, our classifiers will learn the boundary of normal samples and therefore be able to classify any points that lie outside the boundary as outliers or abnormal samples.

Version intermédiaire en date du 26 novembre 2022

From hybrid model to end-to-end model : The ambitious goal for extending our second framework to the end-to-end model is replacing SVM inference phase by a sub neural network for detection/segmentation. To my personal knowledge, there have not existed successful models connecting CGANs to a discriminative segmentation/detection model yet. Recently, Nguyen et al. proposed a promising way to add a small brand of CNN models for classification into encode-decode model [START_REF] Nguyen | Hybrid deep network for anomaly detection[END_REF]. We might follow this approach in the future.

Version intermédiaire en date du 26 novembre 2022

Annexe A

BONUS EXPERIMENTAL RESULTS ILLUSTRATIONS

A BONUS EXPERIMENTAL RESULTS ILLUSTRATIONS

Abstract

Today, automatic solving transportation problem becomes active subject. In our PhD project, we aim to address a specific challenge in this domain: anomaly detection and tracking. Our ultimate goal is constructing a flexible and effective framework producing high performance on various public datasets. The context of our research is applying and improving previous successful approaches to achieve better results. We deal with two scenarios leading to two methods mentioned in following parts: [START_REF] Harchaoui | Temporal localization of actions with actoms[END_REF] vehicles and road users segmentation and tracking by future predictions using classical hand-crafted generative methods based on optical flow estimation; (2) anomaly detection by future predictions using multi-channels deep generative frameworks and supervised learning.

Our first research is evaluating the performance of the classical hand-crafted generative approach in future prediction and its capability for improving segmentation and tracking. Recently, there existed various strong deep learning detectors e.g. Mask R-CNN lead to an effective approach for tracking problem: tracking-by-detection. This very fast type of tracker considers only the Intersection-Over-Union (IOU) between bounding boxes to match objects without any other visual information. In contrast, the lack of visual information of IOU tracker combined with the failure detections of CNNs detectors create fragmented trajectories. We propose an enhanced tracker based on tracking by-detection and optical flow estimation in vehicle tracking scenarios. Our solution generates new detections or segmentations based on translating backward and forward results of CNNs detectors by optical flow vectors. This task can fill in the gaps of trajectories. The qualitative results show that our solution achieved stable performance with different types of flow estimation methods. Then we match generated results with fragmented trajectories by SURF features. DAVIS dataset is used for evaluating the best way to generate new detections. Finally, the entire process is tested on DETRAC dataset. The qualitative results show that our methods significantly improve the fragmented trajectories. For future work, we plan to apply CGANs streams of second work for the first task to propose a new competitive process of future prediction for segmentation and tracking.

Despite the moderate success of the first work, there is significant limitations of classical approaches to deal with our main task: anomaly detection. The lower frequency of abnormal events leads to an unbalanced scenario and the features of abnormal events usually do not follow any spatial or temporal relationship. It is also difficult to pre-define the structure or class of abnormal events. Facing to those challenge, most of state-of-the-art (SOTA) anomaly detection methods are based on apparent motion and appearance reconstruction networks and use error estimation between generated and real information as detection features. These approaches achieve promising results by only using normal samples for training steps. In this thesis, our contributions are two-fold. On the one hand, we propose a flexible multichannel framework to generate multi-type frame-level features. On the other hand, we study how it is possible to improve the detection performance by supervised learning. The multi-channel framework is based on four Conditional GANs (CGANs) taking various types of appearance and motion information as input and producing prediction information as output. These CGANs provide a better feature space to represent the distinction between normal and abnormal events. Then, the difference between those generative and ground-truth pieces of information is encoded by Peak Signal-to Noise Ratio (PSNR). We propose to classify those features in a classical supervised scenario by building a small training set with some abnormal samples of the original test set of the dataset. The binary Support Vector Machine (SVM) is applied for frame-level anomaly detection. Finally, we use Mask R-CNN as a detector to perform object-centric anomaly localization. Our solution is largely evaluated on Avenue, Ped1, Ped2 and ShanghaiTech datasets. Our experiment results demonstrate that PSNR features combined with supervised SVM are better than error maps computed by previous methods. We achieve SOTA performance for frame-level AUC on Avenue, Ped1 and ShanghaiTech. Especially, for the most challenging Shanghaitech dataset, a supervised training model outperforms up to 9% the SOTA on unsupervised strategy. Furthermore, we keep in progress several promising ways: building a new dataset for semi-supervised anomaly detection containing both normal and abnormal samples in its training set and applying one-class SVM to propose an end-to-end framework.

  ) and target images I t for each pix2pix-CGAN stream. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Comparison of Frame level AUC on 4 datasets between simple thresholding inference model and complex learning inference model. . . . . . . . . . . . . . . 4.3 Frame-level AUC performance on Avenue dataset and the comparison between two methods of prediction error encoding : PSNR and MSE. . . . . . . . . . . . . 4.4 Frame-level AUC performance on all 4 benchmarks using PSNR encoding. . . . 4.5 Comparison of Frame level AUC on the 4 datasets between ours solution and recent state-of-the-art methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Pixel-level AUC and EER performance of abnormal object localization on Avenue dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Pixel-level performance of abnormal object localization on Avenue dataset. . . . xxiii

3. 1 1

 11 Mask R-CNN pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Region segmentation in LDOF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Region matching with outliers existing in LDOF . . . . . . . . . . . . . . . . . . 3.4 Optical flow over regular grid in Full Flow method . . . . . . . . . . . . . . . . . 3.5 Generating object segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 SURF detector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 SURF descriptor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Improving IOU-Tracker with generated information and SURF features . . . . . 3.9 Illustration of several samples in DAVIS and UA-DETRAC datasets. . . . . . . . 3.10 Comparison of optical flow estimation performance between LDOF and Full-Flow for bus class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.11 Comparison of optical flow estimation performance between LDOF and Full-Flow for car class. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.12 Qualitative performance of generating object segmentation for bus class. . . . . 3.13 Qualitative performance of generating object segmentation for car class. . . . . . 3.14 Qualitative results of our solution for improving IOU Tracker based Mask R-CNN detector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Our multi-channels pix2pix-CGANs framework for anomaly detection . . . . . 4.2 Two possible network architecture of CGAN generator : Encode-Decode and U-Net with skip connection [39]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Conditional-GAN principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Our pix2pix-CGAN architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Illustration of several samples in Avenue, Pedestrian and ShanghaiTech datasets. 4.6 Comparison of loss convergence for different parameters for CGAN-2 on Avenue dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Illustration of training CGAN-1 on Avenue dataset. . . . . . . . . . . . . . . . . . 4.8 Illustration of training CGAN-2 on Avenue dataset. . . . . . . . . . . . . . . . . . 4.9 Illustration of training CGAN-3 on Avenue dataset. . . . . . . . . . . . . . . . . . 4.10 Illustration of training CGAN-4 on Avenue dataset. . . . . . . . . . . . . . . . . . 4.11 Example of outliers removing on Avenue dataset . . . . . . . . . . . . . . . . . . 4.12 Distribution of normal and abnormal samples in Avenue dataset . . . . . . . . . 4.13 Distribution of normal and abnormal samples in Ped1 dataset . . . . . . . . . . . 4.14 Distribution of normal and abnormal samples in ped2 dataset . . . . . . . . . . 4.15 Distribution of normal and abnormal samples in ShanghaiTech dataset . . . . . 4.16 Distribution of testing samples in Avenue dataset corresponding to our CGANs feature space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.17 The pipeline of abnormality object localisation framework . . . . . . . . . . . . . 4.18 An illustration of AUC-ROC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.19 Evolution of AUC performance according to the size of the SVM-train set . . . . 4.20 Illustration of optimization process when we train SVM binary classifier on Avenue dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.21 Illustration of optimization process when we train SVM binary classifier on Ped1 dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.22 Illustration of optimization process when we train SVM binary classifier on Ped2 dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.23 Illustration of optimization process when we train SVM binary classifier on Shan-ghaiTech dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.24 Illustration of AUC performance achieved at optimized SVM model on Avenue dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.25 Illustration of confusion matrix achieved with optimized SVM model on Avenue dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.26 Illustration of samples distribution achieved at optimized SVM model on Avenue dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.27 Illustration of AUC performance achieved with optimized SVM model on ped1 dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.28 Illustration of confusion matrix achieved with optimized SVM model on ped1 dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.29 Illustration of samples distribution achieved with optimized SVM model on ped1 dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.30 Illustration of AUC performance achieved with optimized SVM model on ped2 dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.31 Illustration of confusion matrix achieved with optimized SVM model on ped2 dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.32 Illustration of samples distribution achieved with optimized SVM model on ped2 dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Version intermédiaire en date du 26 novembre 2022

4 . 37

 437 Single abnormal object localization on Pedestrian datasets . . . . . . . . . . . . . 4.38 Single abnormal object localization on ShanghaiTech dataset . . . . . . . . . . . 4.39 Multiple abnormal objects localization . . . . . . . . . . . . . . . . . . . . . . . . 4.40 Qualitative results of abnormal objects localization on Avenue, Ped1, Ped2 and ShanghaiTech dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.41 Several imperfect cases of anomaly localisations affected by Mask R-CNN detector A.1 Sequence 1a -Larger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 Sequence 1b -Larger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.3 Sequence 2a -Larger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.4 Sequence 2b -Larger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.5 Illustration of training CGAN-1 on Ped1 dataset. . . . . . . . . . . . . . . . . . . A.6 Illustration of training CGAN-2 on Ped1 dataset. . . . . . . . . . . . . . . . . . . A.7 Illustration of training CGAN-3 on Ped1 dataset. . . . . . . . . . . . . . . . . . . A.8 Illustration of training CGAN-4 on Ped1 dataset. . . . . . . . . . . . . . . . . . . A.9 Illustration of training CGAN-1 on Ped2 dataset. . . . . . . . . . . . . . . . . . . A.10 Illustration of training CGAN-2 on Ped2 dataset. . . . . . . . . . . . . . . . . . . A.11 Illustration of training CGAN-3 on Ped2 dataset. . . . . . . . . . . . . . . . . . . A.12 Illustration of training CGAN-4 on Ped2 dataset. . . . . . . . . . . . . . . . . . . A.13 Illustration of training CGAN-1 on ShanghaiTech dataset. . . . . . . . . . . . . . A.14 Illustration of training CGAN-2 on ShanghaiTech dataset. . . . . . . . . . . . . . A.15 Illustration of training CGAN-3 on ShanghaiTech dataset. . . . . . . . . . . . . . A.16 Illustration of training CGAN-4 on ShanghaiTech dataset. . . . . . . . . . . . . . Version intermédiaire en date du 26 novembre 2022

Figure 1 . 1 -

 11 Figure 1.1 -Common missing detection of Mask R-CNN in consecutive frames from (a) to (d) : (a) detection ; (b) non-detection ; (c) detection ; (d) non-detection

Figure 2 . 1 -

 21 Figure 2.1 -AlexNet architecture [43] -The first successful CNN model with 7 layers. This architecture is a basic model for most of the recent innovative CNN architectures.

This result won the 1

 1 st place on the ILSVRC 2015 classification task. They also presented analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to their extremely deep representations, they obtained Version intermédiaire en date du 26 novembre 2022 a 28% relative improvement on the COCO object detection dataset. Deep residual nets are the foundations of their submissions to ILSVRC and COCO 2015 competitions, where they also won the 1 st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation.

Figure 2 . 2 -

 22 Figure 2.2 -Visualization of features in fully trained AlexNet model [114].

Figure 2 . 3 -

 23 Figure 2.3 -Two types of features in action recognitions : Hand-crafted and Deep features [108].

Figure 2 . 4 -

 24 Figure 2.4 -The pipeline of R-CNN [24]. This is the first successful deep learning model for object detection.

  To reduce time consumption in R-CNN, He et al. proposed Spatial Pyramid Pooling architecture called SPP-Net [30]. Existing deep convolutional neural networks (CNNs) required a fixed-size (e.g., 224 × 224) input image. This requirement was artificial and may reduce the recognition accuracy for the images or sub-images of an arbitrary size/scale. In this work, they equipped the networks with another pooling strategy, called "spatial pyramid pooling", to eliminate the above requirement. The new network structure, called SPP-net, can generate a fixed-length representation regardless of image size/scale. The pyramid pooling is also robust to object deformations. With these advantages, SPP-net should in general improve all CNN-based image classification methods. On the ImageNet 2012 dataset, they demonstrated that SPP-net boosts the accuracy of a variety of CNN architectures, despite their different designs. On the Pascal VOC 2007 and Caltech101 datasets, SPP-net achieved state-of-the-art classification results using a single full-image representation and no fine-tuning. The power of SPP-net is also significant in object detection. Using SPP-net, they computed the feature maps from the entire image only once, and then pool features in arbitrary regions (sub-images) to generate fixed-length representations for training the detectors. Their method was up to a hundred times faster than the R-CNN method while achieving a better or comparable accuracy on Pascal VOC 2007. In ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2014, their method ranks 2 in object detection and 3 in image classification among all the 38 teams. This manuscript also introduces the improvement made for this competition. Improving the depth and speed of R-CNN, Girshick proposed the Fast R-CNN architecture [23] with many significant contributions. This paper proposed a Fast Region-based Convolutional Network method (Fast R-CNN) for object detection. Fast R-CNN was built on previous work to efficiently classify object proposals using deep convolutional networks. Compared to previous work, Fast R-CNN employed several innovations to improve training and testing speed while also increasing detection accuracy. As we stated, Fast R-CNN trained the very deep VGG16 network 9× faster than R-CNN, is 213× faster at test-time, and achieves a higher mAP on PASCAL VOC 2012. Compared to SPPnet, Fast R-CNN trains VGG16 3× faster, tests 10× faster, and is more accurate. The advantages of Faster R-CNN were : Higher detection quality (mAP) than R-CNN, SPPnet ; Training was single-stage, using a multi-task loss ; Training can update all network layers ; No disk storage was required for the feature caching. A huge improvement towards Real-Time Object Detection was introduced by Ren et al. in Faster R-CNN [85] using Region Proposal Networks. In their work, they proposed a Region Pro-Version intermédiaire en date du 26 novembre 2022

Figure 2 . 5 -

 25 Figure 2.5 -Network architecture of Faster R-CNN [85]. RPN is a key module to construct an unified end-to-end model for object detection.

Figure 2 . 6 -

 26 Figure 2.6 -The Mask R-CNN framework for instance segmentation [29].

  Fischer et al. constructed appropriate CNNs which were capable of solving the optical flow estimation problem as a supervised learning task. They proposed and compare two architectures : a generic architecture and another one including a layer that correlates feature vectors at different image locations (Figure 2.7). However, FlowNet did not outperform traditional methods. The state of the art with regard to the quality of the flow had still been defined by traditional methods. Particularly on small displacements and real-world data, FlowNet cannot compete with variational methods. FlowNet 2.0 [35] proposed by Eddy Igg et al. in 2017 advanced the concept of end-to-end learning of optical flow and made it work really well. The large improvements in quality and speed are caused by three major contributions : first, they focused on the training data and show that the schedule of presenting data during training was very important. Second, they developed a stacked architecture that includes the warping of the second image with intermediate optical flow. Third, they elaborated on small displacements by introducing a subnetwork specializing in small motions. The lighter Version intermédiaire en date du 26 novembre 2022

Figure 2 . 7 -

 27 Figure 2.7 -Network architecture of Flow-Net [19]. This is the first successful deep learning model for optical flow estimation.

Future

  video information prediction recently has became an active topic due to significant progresses in deep learning, especially in generative adversarial networks (GANs) and Convolutional Auto-Encode (Conv-AE) models. They predicted various types of future information for specific applications.[START_REF] Mathieu | Deep multi-scale video prediction beyond mean square error[END_REF] trained a classical 7-layers CNN to generate future frames given an input sequence. To deal with the inherently blurred predictions obtained from the standard Mean Squared Error (MSE) loss function, they proposed three different and complementary features learning strategies : a multi-scale architecture, an adversarial training method, and an image gradient difference loss function.[START_REF] Walker | Dense optical flow prediction from a static image[END_REF] built a 7-layers CNN for predicting the future motion of each and every pixel in the image in terms of optical flow given a static image.[START_REF] Finn | Unsupervised learning for physical interaction through video prediction[END_REF] developed a Long-Short Term Memory (LTSM) based action-conditioned video prediction model that explicitly models pixel motion to learn about physical object motion without labels, by predicting a distribution over pixel motion from previous frames. Inspired by the same idea,[START_REF] Lotter | Deep predictive coding networks for video prediction and unsupervised learning[END_REF] constructed the LTSM based PredNet network which learned to predict future frames in a video Version intermédiaire en date du 26 novembre 2022 sequence, with each layer in the network making local predictions and only forwarding deviations from those predictions to subsequent network layers.[START_REF] Villegas | Decomposing motion and content for natural video sequence prediction[END_REF] built a deep neural network for the prediction of future frames in natural video sequences upon the Conv-AE and Convolutional

Figure 2 . 8 -

 28 Figure 2.8 -GAN principle : The generator generates new data instances, while the discriminator evaluates the data for authenticity. The loss is applied to updated both generator and discriminator.

Figure 2 . 9 -

 29 Figure 2.9 -Network architecture of Mask R-CNN based on very deep ResNet [29].

2. 4 . 1

 41 Evaluation metrics for Anomaly Detection Generally, one usually uses Equal Error Rate (EER) and Area Under Curve (AUC) to evaluate quantitative performance of an anomaly detection model. Version intermédiaire en date du 26 novembre 2022

Figure 2 .

 2 Figure 2.10 -IOU Tracker principle : Associating detections by their spatial overlap between time steps based on high accuracy detections at high frame rates [7].

  Fields (MRF) graph for detecting abnormal activities in video. The nodes in the MRF graph corresponded to a grid of local regions in the video frames, and neighbouring nodes in both space and time were associated with links. They captured the distribution of typical optical flow with a mixture of probabilistic principal component analyzers to learn normal patterns of activity at each local node. For any new optical flow patterns detected in incoming video clips, they applied the learned model and MRF graph to compute a maximum a posteriori estimate of the degree of normality at each local node. Further, they proposed the incremental update of the current model's parameters as new video observations stream in, so that their model could efficiently adapt to visual context changes over a long period of time. Qualitative performances illustrated that their space-time MRF model robustly detected abnormal activities both in a local and global sense. Mahadevan et al. [66] learned the Mixture of Dynamic Textures (MDT) during training then computed negative log-likelihood of the spatio-temporal patch at each region at test phase (Figure 2.11). They proposed three properties for designing their models : (1) joint modelling of appearance and dynamics of the scene, and the abilities to detect (2) temporal and (3) spatial abnormalities. The model for normal crowd behaviour was based on mixtures of dynamic textures, and outliers under this model were labelled as anomalies. Temporal anomalies were equated to events of low-probability, while spatial anomalies are handled using discriminant saliency. To evaluate their models, they presented new datasets : USCD Pedestrians Ped1 & Ped2. They achieved state-of-the-art performances on their dataset at the moment of publication. Developing the capability of motion representations, Histograms of optical flow orientation (HOFO) was extracted by Wang et al. [109] to classify abnormal events by one-class SVM or kernel PCA. The details of the histogram of the optical flow orientation descriptor were illustrated for describing movement information of the global video frame or foreground frame. Version intermédiaire en date du 26 novembre 2022

Figure 2 .

 2 Figure 2.11 -Learning MDTs for temporal abnormality detection by Mahadevan et al. [66].

Figure 2 . 12 -

 212 Figure 2.12 -Discriminative framework combining HOG, HOF, MBH for anomaly detection proposed by Giorno et al. [22].

  solved the problem of environment-dependent nature by integrating a generic Fast R-CNN model and environment-dependent anomaly detectors. They learned CNN with multiple visual tasks to exploit semantic information that is useful for detecting and recounting abnormal events and then appropriately plugged the model into anomaly detectors (Figure 2.13). They achieved AU C = 0.898 on Avenue and AU C = 0.922 on Ped2 dataset.

Figure 2 .

 2 Figure 2.13 -Environment-dependent anomaly detectors proposed by Hinami et al. [33]. The left part illustrates learning procedures of two types of models : generic and environment-specific models, and right part shows testing procedure of joint detection and recounting abnormal events.

Figure 2 .

 2 Figure 2.14 -Stack RCNN framework for anomaly detection proposed by Luo et al. [63]. The blue boxes represent the input of stacked RNNs. The green and orange boxes represent coding vectors. The yellow circles are similarities between neighbouring frames.

Figure 2 .

 2 Figure 2.15 -Generative framework combining motion features or learned features with autoencoder to reconstruct the scene for anomaly detection proposed by Hasan et al. [27].

Figure 2 .

 2 Figure 2.16 -Anomaly detection based on future frame prediction proposed by Liu et al. [53].

  ). First, they installed an unsupervised feature learning framework based on object-centric convolutional auto-encoders to encode both motion and appearance information. Then, they proposed a supervised classification approach based on clustering the training samples into normality clusters. They applied one-versus-rest abnormal event classifier to separate each normality cluster from the rest. For the purpose of training the classifier, the other clusters is treated as dummy anomalies. During inference, they Version intermédiaire en date du 26 novembre 2022 decided an object as abnormal if the highest classification score assigned by the one-versus-rest classifiers was negative. Continuing the approach of using an additional classifier layer to inference abnormality, instead of training SVMs with only normal samples from training dataset, Liu et al. proposed

Chapitre 3

 3 Improving detection and tracking using future prediction based on optical flow In this chapter, we introduce the essential of our first contribution in details. The structure contains two parts : (1) Improving detection using future generated object segmentation based on optical flow, and (2) Enhanced Tracker with IOU-Tracker, Mask R-CNN and Optical flow.

3. 1

 1 Proposed methods 3.1.1 Instance segmentation by Mask RCNN Mask R-CNN [29] proposed by He et al. is a state-of-the-art deep neural network for solving instance segmentation problem in computer vision. Given an input image, Mask R-CNN infers the object bounding boxes, classes and masks. Generally, there are two stages in Mask R-CNN (Figure 3.1). First, it generates region proposals where there might be an object based on the input image. Second, it predicts the object classes, refines the bounding box and generates a mask at the pixel level of the object based on the first stage proposal. Both stages are connected to the backbone structure.

Figure 3 . 1 -

 31 Figure 3.1 -General pipeline of Mask R-CNN. It contains two stages : First, it generates region proposals where there might be an object based on the input image. Second, it predicts the object classes, refines the bounding box and generates a mask at the pixel level of the object based on the first stage proposal.

Figure 3 . 2 -

 32 Figure 3.2 -Region segmentation in LDOF [9]. Left : Segmentation of an image. A region hierarchy is obtained by successively splitting regions at an edge of certain relevance. Dark edges are inserted first. Right : Zoom into the hand region of two successive images.

Figure 3 . 3 -

 33 Figure 3.3 -Region matching with outliers existing in LDOF[START_REF] Brox | Large displacement optical flow : Descriptor matching in variational motion estimation[END_REF]. Displacement vectors of the matched regions drawn at their centroids. Many matches are good, but there are also outliers from regions that are not descriptive enough or their counterpart in the other image is missing.

  2 be a flow field that maps each pixel p in I 1 to (p + f p ) in an augmented domain Ω ⊃ Ω, which contains Ω and a large surrounding buffer zone. The buffer zone absorbs pixels that flow out of the visual field. The augmented domain Ω ⊃ Z 2 is the Minkowski sum of Ω and [-ζ; ζ] 2 ∩ Z 2 , where ζ is the maximal empirical displacement magnitude. The maximal empirical displacement magnitude is measured by taking the maximal displacement observed in a training set. For example, the maximal displacement magnitude on the KITTI training set is 242 pixels. They perform the optimization on1/3-resolution images, so ζ = 81 for the KITTI dataset.

Furthermore, we also

  consider the development of flow vector beyond one time step. Given instance results I t , I t+N of frame t and t + N achieved from Mask R-CNN detector, we have (Figure 3.5c) : I t+j;0<j<N = p t+j w.r.t F t→t+j (p t ) = p t+j and F t+j→t+N (p t+j ) = p t+N (3.11)

Figure 3 . 5 -

 35 Figure 3.5 -Generating object segmentation (a) Shift and Warp translation ; (b) Forward and backward translation ; (c) Combined results beyond one time step

Figure 3 . 6 -

 36 Figure 3.6 -Basic of Laplacian of Gaussian with Box Filter in SURF detector with Hessian matrix[START_REF] Bay | Speeded-up robust features (surf)[END_REF]. Left to right : The (discretized and cropped) Gaussian second order partial derivatives in y-direction and xy-direction, and approximations thereof using box filters. The grey regions are equal to zero.

Figure 3 . 7 -

 37 Figure 3.7 -Localizing interest points and using Haar wavelet for orientations assignments and descriptor extraction in [5]. Left : Detected interest points for a Sunflower field. This kind of scenes shows clearly the nature of the features from Hessian-based detectors. Middle : Haar wavelet types used for SURF. Right : Detail of the Graffiti scene showing the size of the descriptor window at different scales.

  Then, for more distinctiveness, SURF feature descriptor has an extended 128 dimensions version. The sums of d x and |d x | are computed separately for d y < 0 and d y ≥ 0. Similarly, the sums of d y and |d y | are split up according to the sign of d x , thereby doubling the number of features. It does not add much computation complexity.

Figure 3 . 8 -

 38 Figure 3.8 -Improving IOU-Tracker with generated information and SURF features. To combine the two sets : generated set B tG and Mask R-CNN instances set B tM , we compare the overlapping region with a threshold σ IOU . After this step, we have only one set B t containing the box b i t of frame I t . Then Each box b i t is compared with all boxes of L previous frames. If we find two boxes associated to two different classes but with their IOU value and the number of matching SURF points greater than threshold σ IOU and σ SU RF , respectively, we associate them together.

Figure 3 . 9 -

 39 Figure 3.9 -Illustration of several samples in DAVIS dataset [77] and UA-DETRAC dataset [65]. First row : DAVIS dataset. Second row : UA-DETRAC dataset.

3. 2 . 1

 21 Improving detection using future generated object segmentation based on optical flow DAVIS 2016 consists of fifty high quality, Full HD video sequences, spanning multiple occurrences of common video object segmentation challenges such as occlusions, motion blur and appearance changes. Each video is accompanied by densely annotated, pixel-accurate and per-frame ground truth segmentation. Because the UA-DETRAC only takes into account three classes (bus, van and car) we restrict the use of the DAVIS sequences to the ones that deal with the same types of vehicles. Unfortunately, only a few sequences within the DAVIS dataset satisfy this constraint. Worst, those sequences are not challenging enough to generate the missing detections from Mask R-CNN we want to cope with. Thus, in order to prepare the situation of false negative errors, we run Mask R-CNN for each frames, then randomly discard some of the generated detection. The original annotation of discarded frames are utilized as ground-truth to compare with the generated results. The performance is measured with the mean Intersection-over-Union (mIOU) metric, which first computes the IOU for each semantic class and then computes the average over classes. IOU is defined as follows : IOU = T rueP ositive(T P ) T rueP ositve(T P ) + FalseN egative(FN ) + FalseP ositive(FP ) (3.15)

Figure 3 .Figure 3 . 10 -Figure 3 . 11 -Figure 3 . 12 -Figure 3 . 13 -

 3310311312313 Figure 3.10 -Comparison of optical flow estimation performance between LDOF and Full-Flow.Results are reported on bus class of DAVIS dataset. Obviously, the difference of quality between two methods is significant, not only for flow localization but also for flow intensity and direction. Best viewed in color

Figure 3 . 14 - 4

 3144 Figure 3.14 -Qualitative results of our solution for improving IOU Tracker based Mask R-CNN detector ; the trajectories of the car inside the red circle are improved : (1a)(2a) Original IOU Tracker based Mask R-CNN detector, for sequence 1a we can track only one frame over 8 frames and sequence 2a achieves 3 frames over 8 frames ; (1b)(2b) Improved IOU Tracker based Mask R-CNN detector with Shift generator and SURF matching, both sequences now achieve 6 frames over 8 frames. Those sequences are illustrated in large size in Annexe A.

  : a generative model G that represents the data distribution, and a discriminative model D that calculates the probability when a sample came from the training data (i.e. real sample) rather than G (i.e. fake sample). Both G and D could be a non-linear mapping function, such as a multi-layer convolutional neural networks. To learn the distribution p g over data x, the generator builds a mapping function from a prior noise distribution p z (z) to data space as G(z). And the discriminator, D(x), produces a single scalar representing the probability when x is real samples rather than p g . G and D are both trained simultaneously : adjusting parameters for G to minimize log(1 -D(G(z)) and adjusting parameters for D to minimize log D(x). They are following the two-player min-max game with value function V (G; D) : Version intermédiaire en date du 26 novembre 2022

Figure 4 . 1 -

 41 Figure 4.1 -Our multi-channels pix2pix-CGANs framework for anomaly detection. In each CGAN stream, the number of channels and the type of each channel for input image (i.e. source image I) and output image (i.e. generated or predicted image I g ) are different. CGAN1 takes 2 grayscale channels as input and generates 2-dimensional optical flow channels as output.CGAN2 takes 1 grayscale channel and a 2-dimensional optical flow channel as input and generates 1 grayscale channel and 2-dimensional optical flow channel as output. CGAN3 takes a 3-dimensional RGB channels as input and 3-dimensional RGB channel as output. CGAN4 takes 2-dimensional grayscale channels as input and generates 1-dimensional grayscale channel as output. The configuration is described in detail in Table4.1. The channels of generated images are similar to the channels of ground-truth images (i.e. target images I t ). Best viewed in color.

Figure 4 . 2 -

 42 Figure 4.2 -Two possible network architecture of CGAN generator : Encode-Decode and U-Net with skip connection [39].

Figure 4 . 3 -

 43 Figure 4.3 -Conditional-GAN principle : The generator -Given a label and random noise as input, this model generates data with the same structure as the training data observations corresponding to the same label. The discriminator -Given batches of labeled data containing observations from both the training data and the generated data from the generator, this network attempts to classify the observations as "real" or "fake".

. 6 ) 4 . 1 . 4

 6414 Multi-channel pix2pix-CGAN frameworkOur framework combines 4 parallel streams that represent 9 image channels as output. Each CGAN stream is based on a pix2pix-CGANs[START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] architecture described in Figure4.4.

Figure 4 . 4 -

 44 Figure 4.4 -Our pix2pix-CGAN architecture. Encoder blocks CEk are from first block 256 × 256 × N i to bottleneck block 1 × 1 × 512 of Generator. Decoder blocks CDk are from bottleneck block 1 × 1 × 512 to last block 256 × 256 × N o of Generator. N i , N o denote the number of input and output image channels of Generator. Discriminator blocks Ck are from block 256×256×(N i +N o ) to block 16 × 16 × 1. Because of the skip connection, the total channels pass to Discriminator is N i + N o . Best viewed in color.

  5, for F m,n > 0 (4.10) where F m,n is the flow value at pixel (m, n), F max and F min are the maximum and the minimum value of optical flow over all videos. Due to the optical flow normalization, negative flow values are mapped to [0, 0.5] while positive values are mapped to [0.5, 1]. By this way, we can maintain the difference between motion directions. All pix2pix-CGAN streams are then trained by Adam

(4. 13 )

 13 We follow the conclusions of Mathieu et al. and we apply PSNR metric to evaluate the difference between the generated output and the target one for each CGAN. The generated output are the predicted optical flows and the images predicted by each of the CGAN. The target data are the available measures i.e. the acquired frames and the optical flow calculated from two consecutive frames. Obviously, if we accumulate all PSNR values along all channels and all streams to learn a threshold, we might lose the benefits carried by each stream. So we apply a late fusion strategy by separately calculating PSNR for each one and we decide by analysing the obtained equivalent PSNR vector.Our 4-streams architecture provides 9 output channels. Thus, we obtain 9 PSNR values encoded as 9-dimensional features vector. The vectors are normalised to range [0, 1] for all 9

Figure 4 . 5 .

 45 First of all, we briefly introduce those benchmarks. Then we present our step by step implementation of the multi-CGANs backbone. Version intermédiaire en date du 26 novembre 2022 Datasets and evaluation metric CUHK Avenue : It contains 16 training sequences with some outliers and 21 testing videos containing 47 irregular events as throwing objects, loitering and running. The size of the people is changing because of the camera position and angle. The normal samples of the test set are more numerous than abnormal ones.

Figure 4 . 5 -

 45 Figure 4.5 -Illustration of several samples in CUHK-Avenue dataset [58], USCD-Pedestrian datasets [66] and ShanghaiTech dataset [53]. First row : Avenue dataset. Second row : Pedestrian dataset. Third row : ShanghaiTech dataset. ShanghaiTech is more challenging than other due to multiple captured views and flexible abnormal samples.

Figure 4 . 6 -

 46 Figure 4.6 -Comparison of loss convergence between different parameters for CGAN-2 on Avenue dataset. Figure (a) shows the case where the loss are not convergent after 40 epochs, Minibatch size 64 and leakyReLU activation function while Figure (b) illustrates a good set of parameters E = 20, M = 128, A = eLU corresponding to a good loss convergence. We find that the total loss is stable from the 1000 th iteration and each loss is almost convergent. Best viewed in color.

Figure 4 . 6 .

 46 Figure 4.6. Particularly, because ShanghaiTech has a very large training set, we train at only 1 epoch and optimize parameters for CGAN-2 then adapt those parameters for the other CGANs to reduce processing time. We choose CGAN-2 as the reference stream for all optimization processes because CGAN-2 can learn the relation between both appearance and motion evolution. The final training results for all CGAN-streams are illustrated in Figures 4.7, 4.8, 4.9 and 4.10. Due

  , 4.8, 4.9 and 4.10. Due Version intermédiaire en date du 26 novembre 2022 to the balance of layout for illustration of each part and for the purpose of clarity, we need to keep the figures as near as possible to their corresponding parts, so for the other datasets, the training convergences are shown in Annexe A.

Figure 4 . 7 -

 47 Figure 4.7 -Illustration of training CGAN-1 on Avenue dataset. We achieve good loss convergences for epoch E = 20, mini batch size M = 128 and activation function A = eLU . The total loss gets rapidly small and each loss is convergent from early iterations.

Figure 4 . 8 -

 48 Figure 4.8 -Illustration of training CGAN-2 on Avenue dataset.We achieve good loss convergences at E = 20, M = 128 and A = eLU . We find that the total loss is stable from the 1700 th iteration while each loss is almost convergent from early iterations.

Figure 4 . 9 -

 49 Figure 4.9 -Illustration of training CGAN-3 on Avenue dataset. We achieve good loss convergences at E = 20, M = 128 and A = eLU . We find that the total loss is stable and each loss is almost convergent from the 1000 th iteration.

Figure 4 . 10 -

 410 Figure 4.10 -Illustration of training CGAN-4 on Avenue dataset. We achieve good loss convergences at E = 20, M = 128 and A = eLU . We find that the total loss is stable and each loss is almost convergent from the 3000 th iteration.

Figure 4 . 11 -

 411 Figure 4.11 -Example of outliers removing on Avenue dataset. In order not to do it manually, we train CGAN-2 on training set to produce output images. Then we accumulate MSE values between output and ground-truth images for all of the 3 channels and we draw the corresponding values. We can easily observe some peaks occurring at the abnormal frames on training videos. We remove the outlier images corresponding to the peaks to obtain a new clean Avenue training set. Best viewed in color.

  , 4.13, 4.14 and 4.15 show the distribution of test vectors infer for respectively Avenue, Ped1, Ped2 and ShanghaiTech dataset. The axes of each figures denote the value of PSNR before normalization. Because we use PSNR based feature vector, by definition normal and abnormal samples are respectively associated to high and low values.

Figure 4 .

 4 [START_REF] Oneata | Efficient action localization with approximately normalized fisher vectors[END_REF] is another illustration of this non-linearly separation problem for each CGAN.

Figure 4 . 12 -

 412 Figure 4.12 -Distribution of normal and abnormal samples in Avenue dataset. The axes of each figure denote the value of PSNR before normalization. For CGAN-2 and CGAN-3, normal points are blue and abnormal points are black. For CGAN-1 and CGAN4, normal = light blue and abnormal = red.

Figure 4 . 13 -

 413 Figure 4.13 -Distribution of normal and abnormal samples in Ped1 dataset. The axes of each figure denote the value of PSNR before normalization. For CGAN-2 and CGAN-3, normal points are blue and abnormal points are black. For CGAN-1 and CGAN4, normal = light blue and abnormal = red.

Figure 4 . 14 -

 414 Figure 4.14 -Distribution of normal and abnormal samples in Ped2 dataset. The axes of each figures denote the value of PSNR before normalization. For CGAN-2 and CGAN-3, normal points are blue and abnormal points are black. For CGAN-1 and CGAN4, normal = light blue and abnormal = red.

Figure 4 . 15 -

 415 Figure 4.15 -Distribution of normal and abnormal samples in ShanghaiTech dataset. The axes of each figure denote the value of PSNR before normalization. For CGAN-2 and CGAN-3, normal points are blue and abnormal points are black. For CGAN-1 and CGAN4, normal = light blue and abnormal = red.

In [ 36 ]

 36 , Ionescu et al. "...believe that including any form of supervision is an important step towards obtaining better performance...". In this paper, the authors propose to train a supervised one-versus-rest classifier on feature maps representing different kinds of normality defined by different clusters obtained on the training samples. On the contrary, instead of training SVMs with only normal samples from training dataset, Liu et al. proposed an interesting alternativesupervised scenario[START_REF] Liu | Margin learning embedded prediction for video anomaly detection with a few anomalies[END_REF]. Classical semi-supervised video anomaly detection assumes that only Version intermédiaire en date du 26 novembre 2022 et al.[START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF]

Figure 4 . 16 -

 416 Figure 4.16 -Distribution of testing samples in Avenue dataset corresponding to our CGANs feature space. The first row illustrates the distribution of all samples by green point. The second row separates the distribution of abnormal and normal samples by different color : normal = blue ; abnormal = black or red.

  ) proposed by Ionescu et al.[START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF]. Ionescu et al. propose to extract object-centric features at the first step and then use those features for the rest of their pipeline. Their work helps them in neutralising the effect of background information which is usually stable throughout the sequence and highlights the distinctions around the objects. In a trade-off, their method increases the complexity of feature space and time consumption for every task. In our side, we apply the object detector after inferring the CGAN streams and after classifying normal/abnormal frames by SVM. Obviously, normal frames appear more often than abnormal ones. Hence, we avoid object detection task in normal frames and thus significantly reduce time processing. Second, we apply a simple PSNR score for both frame-level and object localization model instead of applying K-means clustering to generating the normal classes for unsupervised SVM as their method[START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF]. By reusing the error maps at frame-level classification stage, we accelerate our process by directly calculating PSNR score of bounding boxes on the error maps. Obviously, our methods look more natural because we apply global features for the global task (i.e. frame-level classification) and object-centric metrics for abnormality object localization.Generally, this framework applies a hard-decision strategy to localize the abnormal objects. It leads to the issue that the final performance highly depends on the accuracy of the bounding box detector. The missing detections or confusing detections can affect the final decision. Another problem is how to pre-define the number of abnormal objects in cases of hard-decision. In popular datasets, this number should be one or two objects. Then we could search for two Version intermédiaire en date du 26 novembre 2022

Figure 4 . 17 -

 417 Figure4.17 -The pipeline of abnormality object localisation framework. After the frame-level anomaly detection stage, we obtain the labels for each frame. For each abnormal frame, we run a fast detector to compute bounding boxes (BB) for each object in the sequence and we directly apply each BB to the error maps. PSNR scores are computed for each one by taking the normal sum of 9 PSNR scores and BBs yielding minimum values for PSNR scores refer to the abnormality objects.

Figure 4 .

 4 18). T P R = T rue P ositive (T P ) T rue P ositive (T P ) + False N egative (FN ) (4.20) FP R = T rue N egative (T N ) T rue P ositive (T P ) + False N egative (FN ) (4.21) A strong classification model has its AUC near to 1 which means it has a good measure of separability (Figure 4.18). A weak model has its AUC near to 0 which means it has the worst measure of separability. In case of AU C = 0, the model is predicting all positive samples as negative and vice versa. Particularly, when AU C = 0.5, it means the model has not the ability of distinguishing between class or the model is achieving the performance at randomize level. For the EER, the best system achieves the lowest EER. For quantitative evaluation of abnormality localization, we use the same bounding box AUC and EER metrics reported in [101]. If the intersection between a detected box and the ground-truth box is smaller than 40% of the area of ground-truth box, the detected box is removed.

Figure 4 . 18 -

 418 Figure 4.18 -An illustration of AUC-ROC. This performance belong to a good model based on AUC metric. The ROC curve is rapidly asymptotic to 1 while FPR is still small.

Figures 4. 20

 20 Figures 4.20,4.21,4.22 and 4.23 show the minimum classification error curve during the

  Figures 4.20,4.21,4.22 and 4.23 show the minimum classification error curve during the optimization process of the SVM parameters for each dataset. The optimization process is time consuming and depends on the size of the dataset, from 30 minutes for Pedestrian dataset to 6 hours for ShanghaiTech. As we previously mentioned, optimisation if done once. Based on those figures, we confirm the necessity and effective performance of SVM optimisation process.

Figure 4 . 19 -

 419 Figure 4.19 -Evolution of AUC performance according to the size of the SVM-train set.The vertical axis shows the AUC performances and the horizontal axis shows the size of the SVMtrain set corresponding to how many samples of original test set are taken. By sequentially increasing the SVM-train set size from 10% of the original test set size up to 80%, we report the evolution of performance. Obviously, the larger the size of the SVM-train set we create, the better performance we obtain. Considering the effect of the supervised scenario, we show that from 50% of the original test set size, we surpass state-of-the-art performance on Avenue, Ped1 and ShanghaiTech. Best viewed in color.

Figure 4 . 20 -

 420 Figure 4.20 -Illustration of optimization process when we train SVM binary classifier on Avenue dataset. The minimum classification error is sharply decreased after 6 iterations then becomes stable. We achieve optimized parameters at box constraint level 0.05 with the cubic kernel.

Figure 4 . 21 -

 421 Figure 4.21 -Illustration of optimization process when we train SVM binary classifier on Ped1 dataset. The minimum classification error is sharply decreased after 11 iterations then becomes stable. We achieve optimized parameters at box constraint level 262.3 with Gaussian kernel.

Figure 4 . 22 -

 422 Figure 4.22 -Illustration of optimization process when we train SVM binary classifier on Ped2 dataset. The minimum classification error is sharply decreased after 24 iterations then becomes stable. We achieve optimized parameters at box constraint level 994.76 with the cubic kernel.

Figure 4 .

 4 Figure 4.37 shows the qualitative results of single abnormal object localization on Pedestrian datasets. Those datasets are more challenging than Avenue dataset because the camera view is quite far, most of the objects are small and overlapped, and only grayscale images are avai-

Figure 4 . 23 -

 423 Figure 4.23 -Illustration of optimization process when we train SVM binary classifier on ShanghaiTech dataset. The minimum classification error is sharply decreased after 15 iterations then becomes stable. We achieve optimized parameters at box constraint level 92.50 with the Gaussian kernel.

Figure 4 .

 4 Figure 4.38 illustrates the qualitative results of single abnormal object localization on Shan-ghaiTech dataset. There are two types of abnormality illustrated in this figure : jumping (first row) and wrong vehicles (all other rows). All abnormal objects are detected, but there are still some flaws due to detection/classification errors of Mask R-CNN, especially in the last row : instead of localising all the persons with a bicycle, our algorithm reports only the umbrella as

Figure 4 . 24 -

 424 Figure 4.24 -Illustration of AUC performance achieved at optimized SVM model on Avenue dataset. The ROC is rapidly asymptotic to 1 while False positive rate is still very small. It means that our model achieves good performance.

. 38 .

 38 Most of the imperfect cases come from the fact that our anomaly object localisation depends on the accuracy of the Mask R-CNN detector. Sometimes, overlapped objects are detected by Mask R-CNN at the same location, e.g. bicycle, person and umbrella in ShanghaiTech samples. When one of those objects achieves minimum PSNR score, our algorithm chooses only this box but not the total location. Version intermédiaire en date du 26 novembre 2022

Figure 4 . 25 -

 425 Figure 4.25 -Illustration of confusion matrix achieved with optimized SVM model on Avenue dataset. We find that most of the wrong decisions are False Positive while False Negative Rate is much lower. It can explain the high performance on AUC metric of our models. In real word transportation scenario, if we consider the abnormal activities are the dangerous activities then the False positive is acceptable and we must reduce the False Negative Rate. It means that our model is very adaptive to real-world applications.

Figure 4 .

 4 [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] illustrates the qualitative results of multiple abnormal objects localizations. Generally, our methods achieve moderate performance. We successfully detect the first object (i.e. the minimum PSNR score) while the second object are sometimes false positives. Those errors often happens when the false positive object are near the main abnormal object. We can explain this problem by investigating the motion features and the inaccuracy of Mask R-CNN.On the one hand, the optical flow estimation is not perfect. The motion errors are accumulated throughout our pipeline, from flow estimation to flow reconstruction. Obviously, the difference of optical flow is always an important part of the PSNR score. Combining with the Version intermédiaire en date du 26 novembre 2022

Figure 4 . 26 -

 426 Figure 4.26 -Illustration of sample distribution achieved at optimized SVM model on Avenue dataset. Almost all of the normal samples situate in higher values than abnormal ones. This distribution corresponds to the fact that PSNR techniques produce high scores if predicted images tend to be similar to the source images.

Figure 4 .

 4 Figure 4.40 shows us several examples selected from Avenue, Ped1, Ped2 and ShanghaiTech to demonstrate true positive and failure cases of anomaly object localization. Obviously, we can strongly detect various types of abnormal events in the crowed and cluttered background, for different camera angles and involving not only single but also multiple objects. Failure cases

Figure 4 . 27 -

 427 Figure 4.27 -Illustration of ped1 performance achieved with optimized SVM model on Avenue dataset. The ROC is rapidly asymptotic to 1 while False positive rate is still very small. It means that our model achieves good performance.

Figure 4 . 28 -

 428 Figure 4.28 -Illustration of confusion matrix achieved with optimized SVM model on ped1 dataset. We find that almost all of the wrong decisions are False Positive while False Negative Rate is much lower. It can explain the high performance on AUC metric of our models. In real word transportation scenario, if we consider the abnormal activities are the dangerous activities then the False positive is acceptable and we must reduce the False Negative Rate. It means that our models is very adaptive to real-world applications.

Strengths:

  Our anomaly detection framework is very flexible. It can be easily extended by adding, removing or replacing any of the CGAN streams and inference methods. About Version intermédiaire en date du 26 novembre 2022

Figure 4 . 29 -

 429 Figure 4.29 -Illustration of sample distribution achieved with optimized SVM model on ped1dataset. Almost all of the normal samples situate in higher values than abnormal ones. This distribution corresponds to the fact that PSNR techniques produce high scores if predicted images tend to be similar to the source images.

Figure 4 . 30 -

 430 Figure 4.30 -Illustration of optimization performance achieved with optimized SVM model on ped2 dataset. The ROC is rapidly asymptotic to 1 while False positive rate is still very small. It means that our model achieves good performance.

Figure 4 . 35 -

 435 Figure 4.35 -Illustration of sample distribution achieved with optimized SVM model on ShanghaiTech dataset. Almost all of the normal samples situate in higher values than abnormal ones. This distribution corresponds to the fact that PSNR techniques produce high scores if predicted images tend to be similar to the source images.

Figure 4 . 36 -

 436 Figure 4.36 -Qualitative results of single abnormal object localization on Avenue dataset. Each row represents a type of abnormality. From the top row to the bottom row : running, dancing, throwing object, moving to wrong direction. We can exactly detect the abnormal object in every frame. Due to the drawback of Mask R-CNN, the bounding boxes are not perfect in some cases, e.g. 4 th image of the first and second rows, 2 nd image of the third row.

Figure 4 . 37 -

 437 Figure 4.37 -Qualitative results of single abnormal object localization on Ped1 and Ped2 datasets. There are two typical types of abnormality in Pedestrian datasets : wrong vehicles (car, bicycle,etc.) and moving to wrong direction. Generally, the abnormal vehicles are easier (to detect) than the wrong moving direction. While all abnormal vehicles are localized in the first, second and fourth row, there are some false positive errors in the third row where the abnormal persons are crossing the road. The same drawback of Mask R-CNN bounding boxes still happens as in Avenue dataset.

Figure 4 . 38 -

 438 Figure 4.38 -Qualitative results of single abnormal object localization on ShanghaiTech dataset. There are two types of abnormality illustrated in this figure : jumping (first row) and wrong vehicles (all other rows). All abnormal objects are detected, but there are still some flaws due to the drawback of Mask R-CNN, especially in the last row. Instead of localizing all the persons with a bicycle, our algorithm chooses only the umbrella.

Figure 4 . 39 -

 439 Figure 4.39 -Qualitative results of multiple abnormal objects localization on Pedestrian and ShanghaiTech datasets. We set the number of abnormal object to 2. While the first object (i.e. obtaining minimum PSNR score) is almost always true, the second object (i.e. obtaining the second minimum PSNR score) is false positive when its is near the first object. This drawback comes from the accumulated errors of the optical flow in estimating and generating steps. Besides, overlapped objects are detected by Mask R-CNN at the same location, e.g. bicycle and person in pedestrian samples. Obviously, both objects usually obtain lower PSNR score than the rest. Therefore, our algorithm chooses the two objects at the same location instead of searching for a new position.

Figure 4 . 40 -

 440 Figure 4.40 -Qualitative results of abnormal objects localization on Avenue, Ped1, Ped2 and ShanghaiTech dataset. The left side contains correct localizations and the right side shows the failure cases with two main errors : false negative frames and detection of wrong objects. The true objects are marked by green boxes. Best viewed in color.

Figure 4 . 41 -

 441 Figure 4.41 -Several imperfect cases of anomaly localisations affected by Mask R-CNN detector. Left : Too large bounding box ; Middle : Too small box ; Right : Wrong object. Best viewed in color.

Figure A. 1 -

 1 Figure A.1 -Larger illustration of Sequence 1a in Figure 3.14

Figure A. 2 -

 2 Figure A.2 -Larger illustration of Sequence 1b in Figure 3.14

Figure A. 3 -

 3 Figure A.3 -Larger illustration of Sequence 2a in Figure 3.14

Figure A. 4 -

 4 Figure A.4 -Larger illustration of Sequence 2b in Figure 3.14

Figure A. 5 -

 5 Figure A.5 -Illustration of training CGAN-1 on Ped1 dataset. We achieve good loss convergences at E = 40, M = 128 and A = eLU .

Figure A. 6 -

 6 Figure A.6 -Illustration of training CGAN-2 on Ped1 dataset. We achieve good loss convergences at E = 40, M = 128 and A = eLU .

Figure A. 8 -

 8 Figure A.8 -Illustration of training CGAN-4 on Ped1 dataset. We achieve good loss convergences at E = 40, M = 128 and A = eLU .

Figure A. 10 -

 10 Figure A.10 -Illustration of training CGAN-2 on Ped2 dataset. We achieve good loss convergences at E = 40, M = 128 and A = eLU .

Figure A. 12 -

 12 Figure A.12 -Illustration of training CGAN-4 on Ped2 dataset. We achieve good loss convergences at E = 40, M = 128 and A = eLU .

Figure A. 14 -

 14 Figure A.14 -Illustration of training CGAN-2 on ShanghaiTech dataset. We achieve good loss convergences at E = 1, M = 128 and A = eLU .

Figure A. 16 -

 16 Figure A.16 -Illustration of training CGAN-4 on ShanghaiTech dataset. We achieve good loss convergences at E = 1, M = 128 and A = eLU .

1 Introduction 1 1 . 1 1 . 3 1 . 4 1 . 5 9 2

 1111314159 Context and Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Definition of Anomaly Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Classical approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 1.4.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.4.2 Solution and contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Abnormality centric approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.5.1 Limitations and Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.5.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.6 Manuscript Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Related work 11 2.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.1 Convolutional neural network . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.1.2 Action understanding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.1.3 Object detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.1.4 Object segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.1.5 Optical flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.1.6 Supervised context Summary . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.1.7 Unsupervised context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.2 Future prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2

 2 Enhanced Tracker with IOU-Tracker based Mask R-CNN and Optical flow 3.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Anomaly detection 4.1 Proposed generative backbone architecture . . . . . . . . . . . . . . . . . . . . . 4.1.1 From GAN to Conditional GAN . . . . . . . . . . . . . . . . . . . . . . . . 4.1.2 U-Net architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.3 Fundamental of Pix2pix CGAN framework . . . . . . . . . . . . . . . . . . 4.1.4 Multi-channel pix2pix-CGAN framework . . . . . . . . . . . . . . . . . . 4.1.5 Feature extraction with PSNR . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.6 CGANs backbone loss evaluation . . . . . . . . . . . . . . . . . . . . . . . 4.2 Abnormality detection by Support Vector Machine . . . . . . . . . . . . . . . . . 4.2.1 SVM based frame-level anomaly detection . . . . . . . . . . . . . . . . . . 4.2.2 Abnormality object localisation . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 Evaluation results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Conclusions and discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Conclusion and Future works 5.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Improving segmentation and tracking by classical hand-crafted methods . 5.1.2 Anomaly detection by multi-channel deep generative neural networks . . 5.2 Future works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Nous classons alors ces caractéristiques dans un contexte supervisé en construisant un petit ensemble d'entrainement à partir de quelques échantillons anormaux de l'ensemble de test original. C'est un Séparateur à Vaste Marge (SVM) qui est appliquée pour la détection des anomalies au niveau trame. Enfin, nous utilisons Mask R-CNN comme détecteur pour effectuer la localisation d'anomalies centrées objet. Notre solution est largement évaluée sur les bases de données Avenue, Ped1, Ped2 et ShanghaiTech. Nos résultats démontrent que les caractéristiques de PSNR combinées avec le SVM supervisé sont meilleures que les cartes d'erreurs calculées par les méthodes précédentes. En particulier, pour la base de données la plus difficile qu'est Shanghaitech, notre modèle surpasse jusqu'à 9% l'état-de-l'art des methodes non-supervisées. En perspective, nous prévoyons de construire une base de données pour la détection d'anomalies dans un cadre semi-supervisé, et d'intégrer un classifieur one-class SVM pour proposer un système "de bout en bout". Performance of Mask R-CNN on CityScapes dataset . . . . . . . . . . . . . . . . . 2.1 Supervised context summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Results of generating new segmentation on DAVIS 2016 by LDOF with "car" and "bus" classes on missing frames i.e. discarded frames . . . . . . . . . . . . . . . . 3.2 Results of generating new segmentation on DAVIS 2016 by Full Flow with "car" and "bus" classes on missing frames i.e. discarded frames. . . . . . . . . . . . . . 3.3 Average performance of generating new segmentation on DAVIS 2016 with "car" and "bus" classes on missing frames i.e. discarded frames. . . . . . . . . . . . . .

	xx xxii	List of abbreviations Sommaire
	List of abbreviations Sommaire Liste des tableaux
	CNN : Convolutional Neural Network 1.1
	FCN : Fully Convolutional Network Résumé	xiii
	AUC : Area Under Curve EER : Equal Error Rate Acknowledgment	xv
	SVM : Support Vector Machines GAN : Generative Adversarial Network Publication	xvii
	CGAN : Conditional Generative Adversarial Network IOU : Intersection Over Union List of abbreviations	xix
	mIOU : mean Intersection Over Union Sommaire	xxi
	LDOF : Large Displacement Optical Flow
	ROC : Receiver Operating Characteristic Liste des tableaux	xxiii
	SURF : Speed Up Robust Feature
	PSNR : Peak Signal to Noise Ratio Table des figures	xxv
	HOG : Histogram Oriented Gradients HOF : Histogram Optical Flow 1 Introduction	1
	SHT : ShanghaiTech CVAE : ConVolutional Auto Encode 2 Related work	11
	BB : Bounding boxes TP : True Positive 3 Improving detection and tracking	41
	TPR : True Positive Rate 4 Anomaly detection	67
	TN : True Negative
	TNR : True Negative Rate 5 Conclusion and Future works	121
	FP : False Positive Rate
	FPR : False Positive Rate A BONUS EXPERIMENTAL RESULTS ILLUSTRATIONS	127
	FN : False Negative FNR : False Negative Rate Table des matières	139
	ACC : Accuracy AP : Average Precision Bibliographie	145
	mAP : mean Average Precision
	MSE : Mean Square Error
	xix Version intermédiaire en date du 26 novembre 2022 xxi Version intermédiaire en date du 26 novembre 2022

détection. Inspirés par cette approche, d'une part, nous proposons un cadre multicanal flexible pour générer des caractéristiques multitypes au niveau image. D'autre part, nous étudions la possibilité d'améliorer les performances de détection par un apprentissage supervisé. Notre système est ainsi basé sur quatre GAN conditionnels (CGAN) prenant en entrée différents types d'informations d'apparence et de mouvement et produisant des informations de prédiction. Ces CGAN représentent la distinction entre événements normaux et anormaux. Ensuite, la différence entre les informations générées et les vérité-terrains est encodée par le pic du rapport signal / bruit (PSNR). 4.1 Configuration of input I, output (i.e. predicted images I g

Table des figures 1

 des1 .1 Common missing detection of Mask R-CNN in consecutive frames . . . . . . . . 2.1 AlexNet architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Visualization of features in fully trained AlexNet model [114]. . . . . . . . . . . 2.3 Two types of features in action recognitions : Hand-crafted and Deep features . 2.4 R-CNN pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Faster R-CNN architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 The Mask R-CNN framework for instance segmentation [29]. . . . . . . . . . . . 2.7 Flow-Net architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 GAN principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9 Network architecture of Mask R-CNN based on very deep ResNet [29]. . . . . . . 2.10 IOU Tracker principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.11 Learning MDTs for temporal abnormality detection . . . . . . . . . . . . . . . . . 2.12 Discriminative framework combining HOG, HOF, MBH for anomaly detection . 2.13 Environment-dependent anomaly detectors proposed by Hinami et al. [33] . . . 2.14 Stack RCNN framework for anomaly detection proposed by Luo et al. [63] . . . . 2.15 Generative framework combining motion features with autoencoder to reconstruct the scene . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.16 Anomaly detection based on future frame prediction proposed by Liu et al.

  Table des figures xxvii 4.33 Illustration of AUC performance achieved with optimized SVM model on Shan-ghaiTech dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.34 Illustration of confusion matrix achieved with optimized SVM model on Shan-ghaiTech dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.35 Illustration of samples distribution achieved with optimized SVM model on ShanghaiTech dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.36 Single abnormal object localization on Avenue dataset . . . . . . . . . . . . . . .

Table 1 .

 1

1). These false negative results or missing detections from Mask R-CNN can lead to Version intermédiaire en date du 26 novembre 2022

  Version intermédiaire en date du 26 novembre 2022

	Method			Bus				Car
		S	W	W-M	R	S	W	W-M	R
	Backward	79.31 75.59 81.76 89.46 65.13 59.21 70.92 92.10
	Forward	79.05 75.84 82.07 89.46 65.04 60.33 69.15 92.10
	Combine-B 79.35 71.47 77.35 89.46 66.87 46.91 57.86 92.10
	Combine-F	79.03 71.76 77.35 89.46 65.36 48.44 56.37 92.10
	Tableau 3.1 -Results of generating new segmentation on DAVIS 2016 by LDOF with "car" and
	"bus" classes. S : Shift ; W : Warp without morphological post-processing ; W-M : Warp with
	morphological post-processing ; R : True masks of Mask R-CNN which are eliminated to make
	the missing detection context ; Combine-B : Extension case in backward direction ; Combine-F :
	Extension case in forward direction

  .1 and Table 3.2 ; with Full Flow optical flow in Table 3.3 and Table 3.3 .

	Method			Bus			Car
		S	W	W-M	R	S	W	W-M	R
	Backward	79.24 76.72 82.36 89.46 65.66 60.16 70.55 92.10
	Forward	79.05 76.63 81.96 89.46 65.34 61.00 68.94 92.10
	Combine-B 79.31 72.42 77.57 89.46 66.41 48.03 57.33 92.10
	Combine-F	79.06 72.64 77.48 89.46 65.63 49.19 56.66 92.10
	Tableau 3.2 -Results of generating new segmentation on DAVIS 2016 by Full Flow with "car"
	and "bus" classes on missing frames i.e. discarded frames.
	Method		LDOF			Full Flow		Mask R-CNN
		Shift Warp Warp-M Shift	Warp Warp-M
	Backward	72.22 67.40	76.34	72.45 68.44	76.46	90.78
	Forward	72.05 68.08	75.61	72.19 68.81	75.50	90.78
	Combine-B 73.11 59.19	67.60	72.86 60.23	67.45	90.78
	Combine-F	72.20 60.10	66.86	72.35 60.192	67.07	90.78
	Quantitative evaluation : We observe that the morphological operators are important with
	the Warp method. After adding this post-processing, the performances are significantly increa-
			Version intermédiaire en date du 26 novembre 2022

Tableau 3.3 -Average performance of generating new segmentation on DAVIS 2016 with "car" and "bus" classes on missing frames i.e. discarded frames.

  4.1 Proposed generative backbone architectureOur general pipeline is illustrated in Figure4.1. It is based on 4 conditional GANs (CGAN) which inputs are defined in the caption section of the figure. Conditional GAN will be presented later in the chapter. A GAN is a machine learning framework that produces a generative model. It is an unsupervised method that does not need labeled training data. Its goal is to model how the training data look like to be able to generate new examples of what it has learned. In our anomaly detection application, by presenting input from normal image sequences, the GAN should be able to construct (or predict) what should be the next image content for normal context.

  Version intermédiaire en date du 26 novembre 2022 level. We find that the bounding box condition above is not good enough for evaluating the object-centric systems because it tends to report us how many frames are true positive w.r.t. the size of the intersection, so it is still frame-level. It cannot directly illustrate the bounding box performance. Hence, we define three other metrics as following :

	IOU 50 Rate =	N umber of B 50 N umber of detected boxes	(4.22)
	IOU 75 Rate =	N umber of B 75 N umber of detected boxes	(4.23)
	mIOU =	IOU N umber of detected boxes	(4.24)

Table 4

 4 .5. We split those methods in two groups. The first group contains the fully unsupervised methods without adding extra abnormal samples from original test set to training set. We notice that while the method of Ionescu et al.[START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF] is unsupervised, they calculated abnormals scores by supervised SVM strategy. The second group contains the semi-supervised and supervised methods that insert abnormal samples into training set. Generally, it is difficult to compare the methods in different scenarios, especially methods from the second group, because the number of testing samples is not the same. Our solution gets promising results on all of the 4 datasets. We achieve a moderate 2% improvement on Ped1 dataset while producing a competitive result on Ped2 and Avenue datasets. Particularly, it outperforms by 9% compared with state-of-the-art method[START_REF] Tudor Ionescu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF] on the challenging ShanghaiTech dataset. Considering the effect of the supervised scenario, we show that from 50% of the original test set size, we surpass state-of-the-art performance on Avenue, Ped1 and ShanghaiTech.

  ,4.27,4.30 and 4.33 for all of the four datasets. The shape of the curve expresses the good performances we get. All of the ROC curves are rapidly asymptotic to 1 while False Positive Rate is still very small.Then we present the confusion matrix in Figure4.25,4.28,4.31 and 4.34. We find that most of the wrong decisions are False Positive while False Negative Rate is much lower. It can explain the high performance on AUC metric of our models. In real word transportation scenario, if we consider the abnormal activities are the dangerous activities then False positive is acceptable and we must reduce the False Negative Rate. It means that our models are very adaptive to real-world

	applications.
	Besides, we draw the samples distributions for each class in Figure 4.26,4.29,4.32 and 4.35.
	Version intermédiaire en date du 26 novembre 2022

  Tableau 4.5 -Comparison of Frame level AUC on the 4 datasets between ours solution and recent state-of-the-art methods that reported their performance on at least 2 similar datasets. We split those methods in two groups. The upper group contains the fully unsupervised methods without adding extra abnormal samples from original test set to training set. The second group contains the semi-supervised and supervised methods that insert abnormal samples into training set. Generally, it is difficult to compare the methods in different scenarios, especially in the second group, because the number of testing samples is not the same. In our case, all the output channels of our model are used. 80% samples of test set are used for training SVM. SHT = ShanghaiTech Version intermédiaire en date du 26 novembre 2022

			CGAN Stream	MSE PSNR
			CGAN-1		0.72	0.82
			CGAN-2		0.69	0.81
			CGAN-3		0.73	0.79
			CGAN-4		0.62	0.67
		Methods	CGAN-(1+2+3+4) 0.81 Avenue Ped1 Ped2 SHT 0.90
		Luo et al. [62]			0.77	0.88
	Nguyen et al. [72] Tableau 4.3 -Frame-level AUC performance comparison between two methods of prediction 0.87 0.96 Unsupervised Hinami et al. [33] 0.89 0.92 error encoding : PSNR and MSE. Results are reported on Avenue dataset. 80% samples of test set methods Vu et al. [101] 0.72 0.82 0.99 are used for training SVM. Ouyang et al. [75] 0.89 0.97 0.81
		Ionescu et al. [36]		0.90	0.98 0.85
		IVC with OS [52]		0.83	0.56
		IVC with OS & FL [52]		0.83	0.50
		IVC with OS & FL &2streams [52]	0.81	0.50
	Semi and supervised methods	TripleLoss + OCSVM [52] Hasan et al. [27] Luo et al. [63] Ionescu et al. [38]		0.80 0.80 0.82 0.81	0.75 0.68	0.50 0.85 0.61 0.92 0.68 0.82
		Liu et al. [53]			0.85	0.83	0.95 0.73
		Liu et al. [52]			0.93	0.77
		Ours			0.92	0.85	0.96 0.94
		CGAN Stream	Avenue Ped1 Ped2 SHT
		CGAN-1	0.82	0.75	0.89 0.68
		CGAN-2	0.81	0.78	0.92 0.78
		CGAN-3	0.79	0.64	0.75 0.73
		CGAN-4	0.67	0.65	0.77 0.62
		CGAN-(1+2)	0.83	0.80	0.93 0.80
		CGAN-(3+4)	0.86	0.70	0.77 0.81
		CGAN-(1+4)	0.86	0.79	0.93 0.76
		CGAN-(1+2+3)	0.86	0.83	0.95 0.87
		CGAN-(1+2+4)	0.83	0.83	0.93 0.83
		CGAN-(1+2+3+4)	0.92	0.85	0.96 0.94
	Tableau 4.4 -Frame-level AUC performance on all 4 benchmarks using PSNR encoding. 80%
	samples of test set are used for training SVM. SHT = ShanghaiTech
		Version intermédiaire en date du 26 novembre 2022 Version intermédiaire en date du 26 novembre 2022

  50 Rate, IOU 75 Rate and mIOU in Table 4.7. A half of true positive boxes are acceptable boxes with IOU greater than 0.5. The similar rate of IOU 50 and IOU 75 shows that most of the acceptable boxes (i.e. B 50 ) are detected with high accuracy (i.e. B 75 ). For mIOU , there is a large Version intermédiaire en date du 26 novembre 2022

  Tableau 4.7 -Pixel-level performance of abnormal object localization on Avenue dataset. We report mIOU in both cases : i, mIOU denotes the performance without true negative samples and ii, mIOU + T N denotes the performance with true negative samples i.e. for each true negative box, we set IOU = 1.

		Metric	Performance
	Methods OC-SVM [102] GMM [102]	IOU 50 Rate IOU 75 Rate mIOU	AUC 0.49 33.16 47.55 EER 0.41 43.06 43.13 0.42
	Multilevel Representations [101] 52.82 38.83 mIOU + T N 0.86
	Ours		74.43 30.21
	Tableau 4.6 -Pixel-level AUC and EER performance of abnormal object localization on Avenue
	dataset.

Version intermédiaire en date du 26 novembre 2022 Figure 4.31 -Illustration of confusion matrix achieved with optimized SVM model on ped2 dataset. We find that almost all of the wrong decisions are False Positive while False Negative Rate is much lower. It can explain the high performance on AUC metric of our models. In real word transportation scenario, if we consider the abnormal activities are the dangerous activities then the False positive is acceptable and we must reduce the False Negative Rate. It means that our models is very adaptive to real-world applications. Version intermédiaire en date du 26 novembre 2022

Figure 4.32 -Illustration of sample distribution achieved with optimized SVM model on ped2 dataset. Almost all of the normal samples situate in higher values than abnormal ones. This distribution corresponds to the fact that PSNR techniques produce high scores if predicted images tend to similar source images. Version intermédiaire en date du 26 novembre 2022

Figure 4.33 -Illustration of optimization performance achieved with optimized SVM model on ShanghaiTech dataset. The ROC is rapidly asymptotic to 1 while False positive rate is still very small. It means that our model achieves good performance. Version intermédiaire en date du 26 novembre 2022

Figure 4.34 -Illustration of confusion matrix achieved with optimized SVM model on Shan-ghaiTech dataset. We find that almost all of the wrong decisions are False Positive while False Negative Rate is much lower. It can explain the high performance on AUC metric of our models.

In real word transportation scenario, if we consider the abnormal activities are the dangerous activities then the False positive is acceptable and we must reduce the False Negative Rate. It means that our models is very adaptive to real-world applications.

Version intermédiaire en date du 26 novembre 2022

Table des matières

 des

	Résumé	xiii
	Acknowledgment	xv
	Publication	xvii
	List of abbreviations	xix
	Sommaire	xxi
	Liste des tableaux	xxiii
	Table des figures	xxv

  . . . . 27 2.2.1 State of the art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 2.2.2 Generative Adversarial Network . . . . . . . . . . . . . . . . . . . . . . . . 28 139 2.3 Improving Detection and Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Mask R-CNN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 IOU object tracker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Anomaly Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.1 Evaluation metrics for Anomaly Detection . . . . . . . . . . . . . . . . . . 2.4.2 Early works with hand-crafted features . . . . . . . . . . . . . . . . . . . . 2.4.3 Recent successful models with Discriminative Deep learning model . . . . 2.4.4 State-of-the-art models with Generative Deep learning . . . . . . . . . . . 2.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Improving detection and tracking 3.1 Proposed methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Instance segmentation by Mask RCNN . . . . . . . . . . . . . . . . . . . . 3.1.2 Optical flow estimation by LDOF . . . . . . . . . . . . . . . . . . . . . . . 3.1.3 Optical flow estimation by Full Flow . . . . . . . . . . . . . . . . . . . . . 3.1.4 Generating object segmentation by Mask R-CNN and Optical flow . . . . . 3.1.5 Extracting SURF feature . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.6 Improving IOU Tracker with generated informations and SURF features . 3.2 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Improving detection using future generated object segmentation based on optical flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.

Table des matières

 des BibliographieVersion intermédiaire en date du[START_REF] Hamdi | Hybrid deep learning and hof for anomaly detection[END_REF] novembre 2022 ANOMALY DETECTION AND OBJECT TRACKING BY FUTURE PREDICTION USING GENERA-TIVE METHODS FOR TRANSPORTATION Résumé Actuellement, le traitement automatiquement des problèmes de transport devient un sujet actif. Dans le cadre de ce travail, nous visons à relever un défi spécifique dans ce domaine : la détection et le suivi des anomalies. Notre objectif est de construire un système flexible et efficace produisant des performances élevées sur diverses bases de données publiques. Le contexte de notre recherche est l'amélioration des approches précédentes pour obtenir de meilleurs résultats. Nous traitons deux scénarios conduisant à deux méthodes mentionnées dans les parties suivantes : (1) la segmentation et le suivi des véhicules et des piétons par des prédictions utilisant des méthodes génératives classiques basées sur des descripteurs a priori (hand crafted) et sur l'estimation des flux optiques ; (2) la détection des anomalies par des prédictions utilisant des systèmes génératifs multicanaux profonds et l'apprentissage supervisé. Notre première recherche vise à l'évaluation des performances de l'approche générative classique pour les prévisions et la détermination des ses capacités à améliorer la segmentation et le suivi d'objets. Récemment, divers détecteurs d'apprentissage profond ont été proposés e.g. Mask R-CNN qui permettent une approche efficace du problème de suivi : le suivi par détection. A l'exception de tout autre information visuelle, ce type de tracker rapide ne prend en compte que l'intersection-sur-union (IOU) entre les boîtes de délimitation pour apparier les objets. Ainsi, l'absence d'informations visuelles du tracker IOU combinée avec les possibles défaillances des détecteurs créent des trajectoires fragmentées. Nous proposons alors un tracker amélioré basé sur la détection par suivi et sur l'estimation du flux optique. Notre solution génère de nouvelles détections ou segmentations basées sur une translation temporelle en avant et en arrière des résultats des détecteurs CNNs en utilisant les vecteurs de flot optique. Cette étape permet de combler une première partie des lacunes des trajectoires. Les résultats qualitatifs montrent alors que notre solution a obtenu des performances stables avec différentes méthodes d'estimation du flot optique. Les lacunes résiduelles au sein des trajectoires sont traitées en utilisant des caractéristiques SURF. La base de données DAVIS est utilisée pour évaluer la meilleure façon de générer de nouvelles détections. Enfin, le tracker résultant est testé sur la base de données DETRAC. Les résultats qualitatifs montrent que notre approche diminue très significativement la fragmentation des trajectoires. Pour les travaux futurs associés à ce tracker, nous prévoyons d'appliquer les réseaux CGAN développés dans le cadre de la seconde partie de notre travail afin de proposer un système compétitif de suivi d'objet basé prévision. Malgré les résultats tangibles de cette première approche, les méthodes classiques présentent des limitations importantes concernant la détection d'anomalies qui est l'un de nos objectifs principaux. La fréquence plus faible des événements anormaux donne un scénario déséquilibré et leurs caractéristiques ne suivent généralement aucune relation spatiale ou temporelle. Face à ces défis, la plupart des méthodes de l'état-del'art se basent sur des réseaux prédictifs et utilisent les erreurs entre informations générées et réelles comme caractéristiques de détection. Inspirés par cette approche, d'une part, nous proposons un cadre multicanal flexible pour générer des caractéristiques multitypes au niveau image. D'autre part, nous étudions la possibilité d'améliorer les performances de détection par un apprentissage supervisé. Notre système est ainsi basé sur quatre GAN conditionnels (CGAN) prenant en entrée différents types d'informations d'apparence et de mouvement et produisant des informations de prédiction. Ces CGAN représentent la distinction entre événements normaux et anormaux. Ensuite, la différence entre les informations générées et les vérité-terrains est encodée par le pic du rapport signal / bruit (PSNR). Nous classons alors ces caractéristiques dans un contexte supervisé en construisant un petit ensemble d'entrainement à partir de quelques échantillons anormaux de l'ensemble de test original. C'est un Séparateur à Vaste Marge (SVM) qui est appliquée pour la détection des anomalies au niveau trame. Enfin, nous utilisons Mask R-CNN comme détecteur pour effectuer la localisation d'anomalies centrées objet. Notre solution est largement évaluée sur les bases de données Avenue, Ped1, Ped2 et ShanghaiTech. Nos résultats démontrent que les caractéristiques de PSNR combinées avec le SVM supervisé sont meilleures que les cartes d'erreurs calculées par les méthodes précédentes. En particulier, pour la base de données la plus difficile qu'est Shanghaitech, notre modèle surpasse jusqu'à 9% l'état-de-l'art des methodes non-supervisées. En perspective, nous prévoyons de construire une base de données pour la détection d'anomalies dans un cadre semi-supervisé, et d'intégrer un classifieur one-class SVM pour proposer un système "de bout en bout". Mots clés : détection d'anomalies, apprentissage profond, modèle génératif, application au transport

CERI SN -IMT Lille Douai

-----

Acknowledgment

Firstly, I would like to express my sincere gratitude to my director Prof. Abdelmalik Taleb-Ahmed and my supervisors Dr. Jacques Boonaert and Dr. Sebastien Ambellouis for their enthusiastic and responsible guidance during my PhD research as well as my integration into life in France. I can not successfully finish my thesis without their continuous support.