Keywords: deep learning, computer vision, unsupervised learning, self-supervised learning, video surveillance, video anomaly detection, perimeter intrusion detection vi apprentissage profond, vision par ordinateur, apprentissage non supervisé, apprentissage auto-supervisé, vidéosurveillance, détection d'anomalie vidéo, détection d'intrusion périmétrique viii

In the last two decades, we have witnessed a massive increase of surveillance cameras in our surroundings. One of the most important uses of these cameras is to detect suspectful or abnormal behaviors, e.g., a moving truck in a pedestrian zone or an intruder entering a prohibited site. These abnormal events occur very rarely and thus it is an extremely tedious and difficult task for professionals to attentively monitor the video constantly for finding these events. Therefore, an automatic video analysis system is essential. Traditional systems suffer to generalize across different types of anomalies, often rely on handcrafted rules and cannot adapt to abnormal events that they have never seen before. In the past few years, we have seen a tremendous progress in deep learning based video surveillance systems. These systems learn representative features from the data itself, generalize across different scenes and anomalies. That is why, in this thesis, we explore deep learning based methods. Majority of these methods in automatic video analysis are supervised, i.e., they require a large volume of labeled data. But since abnormal events depend on context and are rare, it is very difficult to have labeled anomalous data beforehand, and even if there is some annotated data for abnormal events, it will always be a small portion compared to normal data. Furthermore, one cannot annotate every possible event that might occur in future. So, we require approaches that can work without labeled data. Since these events occur in videos, they can have both spatial and temporal dimensions. Therefore, the approach must be able to learn pertinent spatio-temporal representations to differentiate abnormal and normal events.

Thus, in this PhD, prepared at LIRIS laboratory and in collaboration with the Foxstream enterprise, we aim to learn spatio-temporal representations from unlabeled videos to detect abnormal events. Precisely, we address the task of video anomaly detection and its sub-task, perimeter intrusion detection. We provided mathematical definitions to these tasks because they were not clearly defined in the literature. The definitions have a direct impact on the evaluation and therefore, we proposed new suitable evaluation schemes. Concerning spatio-temporal representation learning without annotations, we proposed two approaches. In the first approach, we designed a strided 3D convolutional autoencoder network and it was used for the perimeter intrusion detection task. The main idea is to learn normal representation from training data without intrusions (or anomalies) and detect intrusions (or anomalies) as they deviate from learned normality. It worked well in small-length videos but suffered in long videos, which have changes in scene dynamics like weather, lighting, etc. To address this problem, we introduced an adaptive thresholding approach using moving z-score. Our extensive experiments showed the viability of our approach in comparison with other existing methods. To further improve the spatio-temporal comprehension of normality, we introduced our second approach. It consisted of a framework that leverages unsupervised and self-supervised learning in an autoencoder. To be precise, we proposed multiple, carefully designed tasks (unsupervised and self-supervised) that are performed in a single autoencoder. This method is also v trained in an end-to-end and joint manner, where training data is without anomalies or intrusions. For detecting anomalies (or intrusions), each of the task provide an anomaly score and the combined score is used for final detection. This approach is generic and was applied to the two tasks. We obtained state-of-the-art results in all major public datasets for both video anomaly detection and perimeter intrusion detection task.

First of all, I would like to thank my thesis supervisors: Laure TOUGNE RODET and Carlos CRISPIM-JUNIOR. They not only guided me with their scientific expertise, but also supported me morally during difficult times. I am specially grateful to them for their valuable feedbacks on our papers and this manuscript. I would also like to thank my tutor of the enterprise Lionel ROBINAULT and members of the R&I team: Quentin BARTHÉLEMY and Sarah BERTRAND. They followed my work from the start of the thesis and we had very insightful scientific discussions, which helped me immensely for my research work. Without the above mentioned people, I cannot imagine where my thesis work would be. I am sincerely very grateful to all five of you.

I would also like to thank all the jury members for accepting to consider the scientific contributions presented in this thesis. My gratitude also goes for all the members of LIRIS and FOXSTREAM, with whom I had great discussions during coffee breaks. A big thanks to all my family and friends, who always supported me and helped me stayed sane and motivated.

Finally, I would like to thank ANRT, LIRIS and FOXSTREAM, for financing the work of this CIFRE thesis.

iii

Introduction

General context

Over the years, the video surveillance systems have evolved from simple video acquisition and display systems to intelligent autonomous systems [START_REF] Warsono | A comprehensive review on intelligent surveillance systems[END_REF]. The systems of today use some of the most sophisticated video analysis and decision-making algorithms to function. The massive installations of cameras in different sites, from banks to supermarkets and in prominent streets, have further helped in developing and testing these systems.

Visually surveying a site can include various tasks, such as object detection, object tracking, object segmentation, abnormal event detection, etc [START_REF] Valera | Intelligent distributed surveillance systems: a review[END_REF]. Out of these tasks, the abnormal event (anomaly) detection task has gained a major attention of the computer vision community [START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF]. Anomalies are patterns in data that do not follow a well-defined notion of normal behaviour (Chandola et al., 2009). Based on context and nature of input data, anomalies can refer to different patterns, such as abnormal sections in a time-series data, abnormal patches in an image, abnormal spatio-temporal volumes in a video, etc., as pictured in Figure 1. For data in video form, video anomaly detection (VAD) refers to the detection of unusual appearance or motion attributes in the video [START_REF] Feng | MIST: Multiple instance self-training framework for video anomaly detection[END_REF][START_REF] Li | Self-training multi-sequence learning with transformer for weakly supervised video anomaly detection[END_REF]. For example, for a video surveilled site where only pedestrians are authorized, all the vehicles are anomalies. Depending on the context, video anomaly detection can be further classified into specific tasks, such as detecting abandoned object [START_REF] Luna | Abandoned object detection in video-surveillance: Survey and comparison[END_REF], loitering [START_REF] Ramachandra | Street scene: A new dataset and evaluation protocol for video anomaly detection[END_REF], illegally parked vehicle (Xie et al., 2017b), etc. The task of perimeter intrusion detection (PID) also falls into this category [START_REF] Saligrama | Video anomaly identification[END_REF][START_REF] A A Sodemann | A review of anomaly detection in automated surveillance[END_REF][START_REF] Nayak | A comprehensive review on deep learning-based methods for video anomaly detection[END_REF]. It aims at detecting the presence of an intrusion in a secured perimeter. Intrusions are a particular type of anomalies, classified INTRODUCTION as point and contextual anomalies by (Chandola et al. (2009), Section 3.5). Moreover, the notions of perimeter, intruder movement and site protection time are crucial for the PID task, i.e., anomalous/unauthorized objects present in the video are intruders only if they are in movement inside the designated perimeter when the site is being surveyed.

In this thesis, we tackle the tasks of video anomaly detection and perimeter intrusion detection. Furthermore, both the tasks are carried out in the outdoor environment. The fact that it is an outdoor environment is very important here as it induces challenges such as changing weather and light conditions, obtrusion due to insects, animals, vegetation, etc., contrary to an indoor environment [START_REF] Matern | Automated intrusion detection for video surveillance using conditional random fields[END_REF][START_REF] Villamizar | WatchNet: Efficient and depth-based network for people detection in video surveillance systems[END_REF]. Since the input to both tasks are videos, i.e., data with spatial and temporal dimensions. Anomalies or intrusion in videos too, can have the spatio-temporal nature. Therefore, it is essential to understand the spatio-temporal representation in videos correctly, in order to detect anomalies or intrusions. Some works address spatial (frames) and temporal dimensions of video separately and then process them sequentially. But these approaches do not learn the spatio-temporal nature of videos aptly [START_REF] Hasan | Learning temporal regularity in video sequences[END_REF]Chong et Tay, 2017;[START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF]. Thus, we seek to develop approaches that learn the spatio-temporal nature of videos in a joint and cohesive manner.

Traditionally, most of the works used trajectory-based anomaly detection (Popoola et [START_REF] Oluwatoyin | Video-based abnormal human behavior recognition-a review[END_REF]. The main idea is if the objects of interest are not following the learned normal trajectories, then they are considered anomalous. This approach suffers from occlusions as it needs to monitor continuously the objects of interest. Some other approaches used low-level appearance, motion and texture features to enhance the comprehension of normality. Other approaches can be found in the works of [START_REF] Li | Anomaly detection techniques in surveillance videos[END_REF][START_REF] Kaur | An overview of anomaly detection in video surveillance[END_REF]. All these traditional approaches suffer to generalize across datasets and anomaly types, often rely on handcrafted features, and are unable to adapt to anomalies that they have never seen before [START_REF] Hu | Video anomaly detection using deep incremental slow feature analysis network[END_REF][START_REF] Ryan | Anomaly detection in video using predictive convolutional long short-term memory networks[END_REF]. For a sterile zone with very few distractions, these approaches can function to some extent for the PID task but in the case of VAD, these approaches perform poorly. Over the past few years, deep learning has surpassed traditional approaches in many major computer vision tasks. Concerning video anomaly detection too, deep learning based approaches have outperformed their traditional counterpart. This is because, given enough data, deep learning based approaches can learn the features themselves, instead of using handcrafted or chosen features. These approaches generalize with different scenes and types of anomalies. Therefore, in this thesis, we consider developing deep learning based approaches. Most of the existing works for spatio-temporal representation learning in video rely on huge volume of labeled data to obtain relevant results. But since in VAD and PID task, the examples of anomalies or intrusions are rare, it is extremely difficult to obtain annotated data. Thus, the goal of this thesis is to learn pertinent spatio-temporal representation from unlabeled videos in order to detect abnormal behaviors like anomaly or intrusion. Furthermore, the proposed methods should be suitable to run in videos of long length (hours, even days), in order to test if they are robust to real-world challenges like change in weather, luminosity, etc.

Industrial context

This thesis is carried out within the framework of a CIFRE (Convention Industrielle de Formation par la REcherche) partnership between the laboratory LIRIS (Imagine team) and the company Foxstream (CIFRE n°2019/1709). The Imagine team of laboratory LIRIS (Laboratoire d'InfoRmatique en Image et Systèmes d'information) is specialized in computer vision, machine learning and pattern recognition. On the other hand, the company Foxstream is specialized in the analysis and automatic processing of real time video content. The video analysis algorithms developed by the company (perimeter intrusion detection, object segmentation, object tracking, etc.) use "classic" signal processing or image analysis algorithms such as modeling of the static components of the scene or using low level descriptors such as SIFT, Haar wavelets, HOG, etc. With this thesis, they want to explore deep learning based solutions for perimeter intrusion detection and video anomaly detection tasks. Furthermore, since intrusions and anomalies are rare events, the company cannot provide labeled data as one cannot anticipate all possible anomalies beforehand. Therefore the proposed methods should be able to work directly in unlabeled data. Overall, the work from this thesis would help the company to move towards deep learning approaches in many of its major applications and that too, without additional cost of manual annotation.

Plan of manuscript

This manuscript is composed of 5 chapters:

• Chapter 1 details various data acquisition schemes and associated challenges in video surveillance. It also describes various datasets used in this work along with their challenges and shortcomings.

• Chapter 2 corresponds to a literature review of various topics: spatio-temporal representation learning in videos with and without annotations, video anomaly detection and perimeter intrusion detection. The basic concepts needed to understand this manuscript are also defined there.

• Chapter 3 is concerned with formally defining the tasks of video anomaly definition and perimeter intrusion definition, and revisiting their evaluation frameworks. We mathematically define the tasks and discuss existing evaluation protocols. We also propose a new, more suitable evaluation protocol.

• Chapter 4 focuses on the task of perimeter intrusion detection. For this, we propose an unsupervised autoencoder with adaptive thresholding mechanism. We provide extensive experimental results of our method in comparison with existing methods, using different evaluation protocols.

• Chapter 5 introduces a new approach for spatio-temporal representation learning from unlabeled videos. It leverages unsupervised and self-supervised learning and is applicable to both VAD and PID tasks. We perform exhaustive experiments on major public VAD and PID datasets and provide comparative analysis of our method with existing methods.

INTRODUCTION

Our first article was dedicated to the task of fall detection and perimeter intrusion detection. It was published in the Reproducible Research in Pattern Recognition (RRPR) workshop of ICPR 2021 conference [START_REF] Lohani | Spatio-temporal convolutional autoencoders for perimeter intrusion detection[END_REF]. Both these tasks are subsets of VAD task and do not contain training labels. In this work, we extended the original work to perimeter intrusion detection task. We also proposed to use precision-recall curve and it provided insightful results on the two tasks. Our next article was published in the special issue "Unusual Behavior Detection Based on Machine Learning" of the Sensors 2022 journal (Lohani et al., 2022b). In this article, we provided mathematical definition of the PID task, along with a review of existing methods and proposed a new suitable evaluation scheme. The work of adaptive PIDS was published in the IEEE International Conference on Image Processing (ICIP) 2022 (Lohani et al., 2022a). In our final contribution, we proposed multiple tasks (unsupervised and self-supervised) for video anomaly detection. We proposed these tasks on a single autoencoder and the main idea is to enrich the spatio-temporal understanding of normality. Experiments were performed in major public VAD datasets and we obtained state-of-the-art results in each of them. This work has been accepted in the International Conference on Computer Vision Theory and Applications (VISAPP), 2023.

Chapter 1

Video surveillance: Data and challenges

In this chapter, we first explain how using different acquisition schemes, the data can be acquired in a surveillance system. Then, we explain various forms of challenges associated either with scene capturing, objects of interest or complexity of the scene. Finally, we detail the datasets that we use along with their difficulties and shortcomings.

Introduction

Main steps of a modern surveillance system are data acquisition, data pre-processing, data analysis and returning an output signal (like an alarm). In this chapter, we focus on the data acquisition step. This step is very important since the input of analysis part is obtained from here. It is the basis of any surveillance system. The data acquired depends on the type of camera used like visual camera, infrared camera, thermal camera, etc. Each of these cameras have their strengths and weaknesses, and therefore they should be used carefully. Thus, in this chapter, we first explain various data acquisition schemes. Since we are interested in outdoor video surveillance, the cameras are placed in the outdoor environment. In outdoor environment, they face various challenges, e.g., challenges due to change in environmental conditions like rain, dust, or bad weather [START_REF] Hu | Research and application of intelligent intrusion detection system with accuracy analysis methodology[END_REF]. Therefore, there are various challenges associated with data in video surveillance and they will be also detailed in this chapter. Finally, we also explore different existing VAD and PID datasets, along with their associated challenges.

Data Acquisition in surveillance systems

In this section, we describe the various data acquisition systems with their associated advantages and drawbacks. The area to be surveyed is observed with the help of cameras.

All cameras can acquire the video stream, but the nature of data depends on the type of camera used. Broadly, the following categories of video capture devices are used.

Visual Camera

It is also called as the 'classic' or 'color' camera as it captures the visible light in form of grey-scale or RGB images and was the first type of cameras to be used for the video surveillance. The advantage is that it renders an image visually closer to the naked eye. However, for proper functioning, it needs a certain level of brightness in the scene from external light source like sun and furthermore it is sensitive to illumination changes [START_REF] Van De Sande | Evaluating color descriptors for object and scene recognition[END_REF][START_REF] Warsono | A comprehensive review on intelligent surveillance systems[END_REF]. During night, it needs additional lighting for the sensor to generate a sufficiently contrasted image [START_REF] Gade | Thermal cameras and applications: a survey[END_REF]. Adverse weather conditions such as fog, rain, snow, etc., further limit the observation of objects to a short distance from the camera [START_REF] Robinault | Détection et suivi d'objets mobiles par caméras fixes[END_REF], and thus make detection difficult. Even after all these drawbacks, these cameras have been used extensively in video surveillance systems [START_REF] Valera | Intelligent distributed surveillance systems: a review[END_REF][START_REF] Sun | Fast and robust algorithm of tracking multiple moving objects for intelligent video surveillance systems[END_REF].

Infrared Camera

To address the issue of scene brightness, the infrared cameras were proposed as they capture near-infrared emissions from objects [START_REF] Warsono | A comprehensive review on intelligent surveillance systems[END_REF]. To be precise, the term infrared camera is a bit of misnomer because the classic visual camera can also capture the near-infrared emissions, but it is restricted to visible bandwidth using a filter [START_REF] Robinault | Détection et suivi d'objets mobiles par caméras fixes[END_REF]. Therefore, the infrared camera is nothing, but a visual camera coupled with an infrared (IR) lighting. The IR lighting illuminates the scene with near infrared radiation (0.7-1.4 µm) and the camera captures the reflected radiation from objects in the scene in both the visible and near infrared spectrum [START_REF] Gade | Thermal cameras and applications: a survey[END_REF]. Thus, it can work during night too and can provide better contrast when an object moves past the camera [START_REF] Haritaoglu | Davis : W4: real-time surveillance of people and their activities[END_REF]. However, it is difficult to detect objects during rain in this camera as rain drops appear as thick stripes in front of the camera [START_REF] Liang | Real time intrusion detection system for outdoor environment[END_REF]. Furthermore, due to the heat from IR lighting, this camera attracts flying insects and spiders that can raise false alarms and, consequently, impact detection. Figure 1.1 demonstrates these issues with infrared camera.

Thermal Camera

Unlike the visual and infrared camera, the thermal camera does not require an external energy source for illuminating the scene. In fact, it relies on the infrared radiation emitted by all objects with a temperature above absolute zero. The thermal camera captures this radiation which is in the mid and long infrared wavelength spectrum (3-14 µm) [START_REF] Gade | Thermal cameras and applications: a survey[END_REF]. Since it does not depend on illuminating the scene, it can work even in complete darkness [START_REF] Warsono | A comprehensive review on intelligent surveillance systems[END_REF]. The thermal image is rendered grayscale where the high intensity pixels relate to high temperature. In other words, the higher the temperature of an object in the scene, the brighter it will appear in the thermal image.

Advantages of this camera include lower sensibility to weather conditions and object shadows [START_REF] Liang | Real time intrusion detection system for outdoor environment[END_REF], and long-range detection [START_REF] Xu | Background modeling methods in video analysis: A review and comparative evaluation[END_REF][START_REF] Hu | Research and application of intelligent intrusion detection system with accuracy analysis methodology[END_REF]. However, the main drawback of the thermal camera is that it is difficult to distinguish an object from its background when both have almost the same temperature [START_REF] Robinault | Détection et suivi d'objets mobiles par caméras fixes[END_REF].

Figure 1.2 shows examples of data acquisition by color and thermal camera. During the day, we can observe that three persons and vehicles are visible in both the cameras. While thermal camera provide better contrast for first two persons and is less affected by shadows, the third person is better visible in color camera since in thermal camera, the object is camouflaged with the car behind it. During night, we cannot see much from the color camera, just the light emitted by an electric scooter. In complete contrast to this, the thermal camera shows the complete scene with details and the person with its scooter is clearly visible.

Other Types of Acquisitions

The depth information of an object can be a useful information for its detection [START_REF] Villamizar | WatchNet: Efficient and depth-based network for people detection in video surveillance systems[END_REF][START_REF] Aravamuthan | Physical intrusion detection system using stereo video analytics[END_REF]. This depth information can be added to any camera type using an additional depth sensor. It determines the perspective size of an object in the scene. However, using a depth sensor also has several problems, such as mixed, lost, and noisy pixels in the depth image [START_REF] Kim | Depth image filter for mixed and noisy pixel removal in RGB-D camera systems[END_REF][START_REF] Shao | Computer vision and machine learning with RGB-D sensors[END_REF].

Another important information in the scene is the motion information. The camera type specializing in this are known as the event camera [START_REF] Lichtsteiner | A 128x128 120 db 15 µs latency asynchronous temporal contrast vision sensor[END_REF]. It generates events at the microsecond resolution. An event is triggered each time a single pixel detects a change in intensity value. It finds its application in motion detection, object segmentation, pose estimation, motion tracking, etc. [START_REF] Gallego | Event-based vision: A survey[END_REF]. Advantages include low latency (no motion blur), high temporal resolution, high dynamic range, and ultra-low power consumption. However, it cannot capture static motion and absolute intensity, and therefore is often used together with other camera types. Event cameras are not used for tasks such as video anomaly detection and perimeter intrusion detection yet, where both spatial and temporal information are essential. The main reasons for this are its extremely high cost and inability to capture visual features, such as texture and color.

Finally, we can also have acquisitions from multi-camera systems where camera types can be homogeneous or heterogeneous. Intuitively, combining multiple cameras can provide more accurate information about the targeted object and may help to overcome occlusions. Surveillance systems with multiple cameras has been studied extensively (Cai et Aggarwal, 1999;[START_REF] Hu | A survey on visual surveillance of object motion and behaviors[END_REF][START_REF] Wang | Distributed visual-target-surveillance system in wireless sensor networks[END_REF]. Recently, an intelligent perimeter intrusion detection system was proposed using an integrated image acquisition device that combines visual and thermal camera (Kim et al., 2018b). However, a multi-camera system also brings new challenges, such as camera installation, camera calibration, object matching, automated camera switching and data fusion [START_REF] Hu | A survey on visual surveillance of object motion and behaviors[END_REF].

To conclude, no acquisition scheme is perfect. Visual camera is still largely used but it does not work well when the scene illumination is low. The infrared camera can work in low illumination but it suffers in bad weather conditions, particularly during rain, and furthermore infrared lighting attracts insects, which can occlude the view. Even though scene illumination does not affect thermal cameras, but it has other problems like an object is indistinguishable from the background if their temperature is similar. Additional sensors, like the depth sensor, can also be added to any of the camera types to improve scene comprehension. Also, a multi-camera system can be made using various camera types. But even these configurations have their drawbacks. Therefore, the video acquisition scheme should be chosen according to the requirements of the site and concerned task.

Open Challenges

In the context of video surveillance, the cameras are generally fixed in a static position to monitor a site [START_REF] Valera | Intelligent distributed surveillance systems: a review[END_REF][START_REF] Bouwmans | Recent advanced statistical background modeling for foreground detection-a systematic survey[END_REF]. This site can be either indoors like a bank or agency, or outdoors like an industrial site, private property, or public places such as a city square. We focus on outdoor sites and they are more challenging due to environmental conditions like rain, dust, or bad weather [START_REF] Hu | Research and application of intelligent intrusion detection system with accuracy analysis methodology[END_REF]. The system must be operational continuously for many days and encounter changing light and weather (i-LIDS Team, 2006). Several authors have identified different challenges in the outdoor video surveillance [START_REF] Bouwmans | Recent advanced statistical background modeling for foreground detection-a systematic survey[END_REF]Brutzer et al., 2011). We can group all these challenges into three main categories. The first category is specific to the data acquisition. The second category corresponds to the captured scene as a whole and the third is linked to the objects of interest.

Challenges in capturing the scene

These challenges are related to the acquisition and transmission of the video signal. a) Noise: It is characterized by a random variation in pixel intensities or color components [START_REF] Xu | Background modeling methods in video analysis: A review and comparative evaluation[END_REF]. In video surveillance, there are two main sources of noise (Brutzer et al., 2011). Noise related to the quality of the sensor and noise related to compression. In the case of the sensor, noise is often present when the brightness of the scene is low. This noise is more frequent on thermal cameras [START_REF] Gade | Thermal cameras and applications: a survey[END_REF]. Compression noise is related to encoding and depends on the bandwidth available on the network. b) Automatic adjustments: Cameras try, as far as possible, to maximize the dynamics of their sensor to present a correctly contrasted image. They accomplish this with functionalities like auto focus, automatic gain control, automatic white balance, and auto brightness control [START_REF] Bouwmans | Traditional and recent approaches in background modeling for foreground detection: An overview[END_REF]. These adjustments results in an overall change in brightness and color levels among different frames in the video. This phenomenon is more important on thermal cameras, where the sensors also auto-adjust depending on the scene temperature [START_REF] Gade | Thermal cameras and applications: a survey[END_REF]. c) Vibrations or jitters: It refers to all the untimely movements of the camera which cause a displacement of the optical center. We are in the case of a fixed camera where the optical center and all the shooting conditions must not be modified during the acquisition of the video. However, it is not uncommon that the camera experiences some vibrations due to wind or other factors [START_REF] Xu | Background modeling methods in video analysis: A review and comparative evaluation[END_REF][START_REF] Sehairi | Comparative study of motion detection methods for video surveillance systems[END_REF]. These vibrations cause false motions in the scene [START_REF] Bouwmans | Traditional and recent approaches in background modeling for foreground detection: An overview[END_REF]. Mechanical or software solutions can be used to correct this defect [START_REF] Bouwmans | Recent advanced statistical background modeling for foreground detection-a systematic survey[END_REF]. If these solutions prove to be insufficient, the same pixel coordinates on two successive images no longer represents the same structure in the scene. d) Dirty or Misadjusted lens: When the cameras are outdoors, projections of rain, dust or bad weather can gradually degrade the quality of the image [START_REF] Hu | Research and application of intelligent intrusion detection system with accuracy analysis methodology[END_REF][START_REF] Zou | Object detection in 20 years: A survey[END_REF]. Likewise, the camera lens adjusted during installation may become out of focus over time. As a result, the video acquired by the camera gradually lose sharpness and becomes blurry [START_REF] Liang | Real time intrusion detection system for outdoor environment[END_REF]. This problem should not be taken lightly because it corresponds to many real cases. A blurred image causes a significant loss of the information (both texture and color) contained in the scene [START_REF] Tsakanikas | Video surveillance systems-current status and future trends[END_REF]. This list is not exhaustive. We can add the case of so-called day / night cameras which transmit color frames when the scene brightness is sufficient and switch to grayscale frames when the brightness is low. We can also add the case of cameras with integrated infrared lighting that we have briefly presented previously or the problems of dazzling by vehicle headlights.

Challenges related to the complexity of the scene

This second category of challenges is linked to the complexity of the scene, regardless of the presence or absence of objects of interest. In the context of outdoor video surveillance, we can be confronted with a multitude of situations which can range from a parking lot without trees, to a field bordered by tall trees and even a riverbank. In addition, the algorithms must also consider different climatic conditions and different lighting conditions. These different situations and conditions can be summarized by the following set of cases. a) Progressive illumination changes: These variations are generally caused by the change in light conditions which progressively evolve over time and by the course of the sunlight [START_REF] Bouwmans | Background modeling using mixture of gaussians for foreground detection-a survey[END_REF][START_REF] Xu | Background modeling methods in video analysis: A review and comparative evaluation[END_REF]. For example, it can be caused by the passage of cloud shadows in the scene. b) Sudden illumination changes: It corresponds to a sudden change in illumination between two successive images (Brutzer et al., 2011;[START_REF] Xu | Background modeling methods in video analysis: A review and comparative evaluation[END_REF]. This change can be either in part of the image or in the full image (Cermeño et al., 2018). It is generally caused by the activation or deactivation of exterior lighting, but it can also be linked to the sudden introduction of a massive object into the scene [START_REF] Bouwmans | Recent advanced statistical background modeling for foreground detection-a systematic survey[END_REF]. c) Dynamic background: Sometimes, some parts of the scene are dynamic and contain movements of irrelevant objects. Examples include tree branches or flag driven by the wind, the reflection of moving clouds on the surface of river, snowflakes, rain, etc. This can confuse the system to make false detections [START_REF] Liang | Real time intrusion detection system for outdoor environment[END_REF][START_REF] Bouwmans | Traditional and recent approaches in background modeling for foreground detection: An overview[END_REF]Cermeño et al., 2018). d) Shadows: Managing shadows is another big challenge (i-LIDS Team, 2006;[START_REF] Al-Najdawi | A survey of cast shadow detection algorithms[END_REF]. Shadows casted by moving objects of interest or static objects of scene can lead to false detections (specially in grayscale mode) [START_REF] Liang | Real time intrusion detection system for outdoor environment[END_REF]. Unlike color cameras, the thermal cameras do not have this problem of moving object shadows [START_REF] Gade | Thermal cameras and applications: a survey[END_REF]. e) Perspective: This challenge is linked to the fact that we want to detect objects of interest both close and far from the camera. However, the same object does not have the same size depending on its distance from the camera (see Figure 1.3) [START_REF] Hu | A survey on visual surveillance of object motion and behaviors[END_REF]. Besides, weather conditions can attenuate the contrast between the object of interest and the background of the scene. This, along with object's perspective, makes detection more difficult (Buch et Velastin, 2014). Furthermore, as the apparent object size decreases with the distance, the apparent object speed also decreases.

Challenges related to the objects of interest

This last category of challenges concerns more specifically the objects that one seeks to detect in the scene. These challenges are therefore much more linked to the problem we are trying to solve. Most video surveillance tasks face the following challenges. a) Camouflage: Camouflage, whether intentional or not, occurs when the difference in appearance between the object of interest and the background is small [START_REF] Maddalena | A self-organizing approach to background subtraction for visual surveillance applications[END_REF]Brutzer et al., 2011). It is a particularly difficult challenge when the scene is full of different sized objects as shown in Figure 1.4. It is often possible to tune the parameters of an algorithm to detect objects on a short camouflaged video sequence (Cermeño et al., 2018;Kim et al., 2018b). We can therefore manage to detect camouflaged objects by increasing the sensitivity of the algorithm. However, when these very sensitive parameters are used for monitoring a site 24 hours per day, the number of false alarms can become impossible to manage. b) Hole in the object of interest (Foreground aperture): This problem occurs when objects having insignificant texture difference with the scene background, move slowly in the scene or in the axis of the camera. Due to this, the object pixels are difficult to detect because they can be embedded in the scene background [START_REF] Maddalena | A self-organizing approach to background subtraction for visual surveillance applications[END_REF][START_REF] Bouwmans | Traditional and recent approaches in background modeling for foreground detection: An overview[END_REF]. Thus, the same object may appear as several small objects, as if there are holes in the object.

Among other challenges, we have the speed of objects and their manner of movement (i-LIDS Team, 2006). Very slow or very fast moving objects can be difficult to detect. Furthermore, the manner in which the object moves can create ambiguity, e.g., person crawling or imitating an animal.

Datasets

Since we investigate the problem of abnormal event detection in videos, we explore the concerning datasets. Concretely, we describe the datasets for video anomaly detection (VAD) and perimeter intrusion detection (PID). As mentioned previously (refer Section 1.1), we are interested in systems that operate in the outdoor environment and consequently we choose datasets that follow this criterion.

VAD datasets

Anomalies in video can be caused by anything like riding a bicyle, fighting, dancing, playing, running, etc. In other words, they depend on context and can differ from one dataset to another. A formal definition for the VAD task will be presented in Chapter 3.

VAD methods rely on a large number of datasets in the literature such as: subway entrance [START_REF] Adam | Robust real-time unusual event detection using multiple fixed-location monitors[END_REF], subway exit [START_REF] Adam | Robust real-time unusual event detection using multiple fixed-location monitors[END_REF], UMN [START_REF] Mehran | Abnormal crowd behavior detection using social force model[END_REF], UCF-Crime [START_REF] Sultani | Real-world anomaly detection in surveillance videos[END_REF], UBnormal [START_REF] Acsintoae | Ubnormal: New benchmark for supervised open-set video anomaly detection[END_REF], UCSD Pedestrian [START_REF] Li | Anomaly detection and localization in crowded scenes[END_REF], CUHK Avenue [START_REF] Lu | Abnormal event detection at 150 fps in Matlab[END_REF], ShanghaiTech (Luo et al., 2017a), etc. However, most of recent works only use the last three datasets of the list for unsupervised video anomaly detection [START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF][START_REF] Lee | Stan: Spatio-temporal adversarial networks for abnormal event detection[END_REF][START_REF] Liu | Future frame prediction for anomaly detection-a new baseline[END_REF][START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF][START_REF] Radu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF][START_REF] Park | Learning memory-guided normality for anomaly detection[END_REF]Astrid et al., 2021b;Liu et al., 2021b;[START_REF] Lv | Learning normal dynamics in videos with meta prototype network[END_REF][START_REF] Park | FastAno: Fast anomaly detection via spatio-temporal patch transformation[END_REF]. The reasons for not using the other datasets are the following: [START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF] and the dataset is already saturated (performance above 99% by many methods) [START_REF] Acsintoae | Ubnormal: New benchmark for supervised open-set video anomaly detection[END_REF]; (c) UCF-Crime: used for weakly or semi supervised anomaly detection since it contains both annotated anomalies and normal events in the training set [START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF][START_REF] Acsintoae | Ubnormal: New benchmark for supervised open-set video anomaly detection[END_REF]; (d) UBnormal: a recent dataset but it consists of synthetic data instead of real anomalies.

We detail below the three commonly used VAD datasets, i.e., UCSD Pedestrian, CUHK Avenue, and ShanghaiTech, respectively.

UCSD Pedestrian Anomaly Dataset

It is one of the most widely used datasets for video anomaly detection [START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF]. This dataset consists of video clips recorded with a stationary color camera with grayscale frames, overlooking pedestrian walkways on the UCSD campus [START_REF] Li | Anomaly detection and localization in crowded scenes[END_REF]. It has two distinct scenes, leading to two subsets of the dataset, called as Ped1 and Ped2. The difference between the two subsets is the walking direction (towards and away from the camera in Ped1, parallel to the camera plane in Ped2) and frame resolution (158 × 238 in Ped1 and 240 × 360 in Ped2). Most works use only the Ped2 dataset [START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF][START_REF] Radu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF][START_REF] Feng | MIST: Multiple instance self-training framework for video anomaly detection[END_REF]Georgescu et al., 2021a;Liu et al., 2021b;[START_REF] Park | FastAno: Fast anomaly detection via spatio-temporal patch transformation[END_REF][START_REF] Park | FastAno: Fast anomaly detection via spatio-temporal patch transformation[END_REF] because Ped1 has a considerably low video resolution [START_REF] Hinami | Joint detection and recounting of abnormal events by learning deep generic knowledge[END_REF][START_REF] Doshi | Online anomaly detection in surveillance videos with asymptotic bound on false alarm rate[END_REF][START_REF] Ouyang | Video anomaly detection by estimating likelihood of representations[END_REF][START_REF] Viet-Tuan Le | Attention-based residual autoencoder for video anomaly detection[END_REF], is suitable for pixel-level and not frame-level anomaly detection (Xu et al., 2015a;[START_REF] Luo | Video anomaly detection with sparse coding inspired deep neural networks[END_REF] and has ambiguity in anomaly annotation (some events labeled as normal in training set are considered as anomalies in testing set) [START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF]. For all these reasons, we also only focus on the UCSD Ped2 dataset.

There is a constant movement of people in this dataset, and it is considered normal. The abnormal events are due to the circulation of non-pedestrian entities, i.e., bikers, skaters, carts, and wheelchairs. All these events occur naturally, i.e., they were not staged for dataset collection. Figure 1.5 shows illustrative normal and anomalous frames of this dataset. This dataset contains 28 videos in total, with 120 to 200 frames per video. The training set has 16 videos (2,550 frames) without anomalies and the testing set comprises of 12 videos (2,010 frames) with anomalous events. Both pixel-level and frame-level annotations are provided with this dataset. The following challenges are present in this Despite being widely used, this dataset has various shortcomings. First, it is one of the smallest datasets in terms of videos, number of frames, total anomalies, and number of anomaly types. Therefore, most methods have almost perfect results on this dataset. Second, all the anomalies can be detected by only analyzing a single frame at a time. In other words, the anomalies have a dominant appearance / spatial difference from normality and it is easy to detect them based on spatial features, without requiring the temporal characteristics.

CUHK Avenue Dataset

It consists of one scene captured in the Chinese University of Hong Kong (CUHK) campus avenue with a fixed color camera [START_REF] Lu | Abnormal event detection at 150 fps in Matlab[END_REF]. People walking towards various predefined directions is considered as a normal event. The abnormal events are due to either non-pedestrian objects like bicycle or unexpected pedestrian behavior like running, loitering, jumping, throwing bag, walking in prohibited direction, etc.. We must highlight that this was the first dataset to introduce loitering (static) as an anomaly, which is important for surveillance applications. Unlike UCSD Ped2, anomalies are mostly staged in this dataset. Some illustrative normal and anomalous frames of this dataset are shown in Figure 1.6 This dataset contains 16 video clips (15,328 frames) for training and 21 for testing (15,324 frames), with a frame resolution of 360 × 640. The training videos consists of normal events with the exception of few videos with small occurrences of anomalies. The testing videos have both normal and abnormal events. Like UCSD Ped2, we have both the pixel-level and frame-level annotations. This dataset presents the following challenges: camera shakes, perspective, dynamic background, and progressive illumination changes.

Avenue dataset also has still some deficiencies. Like UCSD Ped2, we have a very small amount of data for deep neural networks and some authors argue that the amount of normal events is not sufficient for training [START_REF] Li | Video anomaly detection and localization via multivariate gaussian fully convolution adversarial autoencoder[END_REF]. Furthermore, the annotations are ambiguous for some test videos, leading to their exclusion by some works [START_REF] Hinami | Joint detection and recounting of abnormal events by learning deep generic knowledge[END_REF][START_REF] Radu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF][START_REF] Ouyang | Video anomaly detection by estimating likelihood of representations[END_REF].

ShanghaiTech Campus Dataset

All previous datasets had a single camera view (or scene) in training and testing set. Having only a single view might lead some models to overfit by memorizing the fixed background. That is why it is essential to have different views in the dataset, so that the proposed detection method is aware of different scenes and generalizes to them. The ShanghaiTech Campus dataset is one of the most challenging datasets for video anomaly detection and includes 13 different views with various lighting conditions and camera angles (Luo et al., 2017a). The videos are filmed at the ShanghaiTech university campus using color cameras with the frame resolution of 480 × 856. Pedestrians walking in the campus with regular pace is considered as a normal event. Abnormal events are caused either by the movement of unauthorized objects like bicycle, skates, strollers, etc. or by irregular pedestrian behaviors like brawling, chasing, jumping, throwing objects, robbery, etc. Figure 1 the original dataset setting as we focus on developing methods without annotations. This dataset presents the following challenges: noise, camera shakes, shadows, perspective, dynamic background, progressive and sudden illumination changes. We still have the following deficiencies in this dataset. Despite being a very large dataset, we have only a small number of anomalies (158) belonging to only 11 anomaly types. Therefore, it lacks diversity and number of anomalies. Similarly, since there are overall 13 scenes, we have a small training set for each scene. This small per-scene training set is not fully representative of all the normal activities [START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF]. Therefore, many works tend to use some form of data augmentation techniques to have enough representative data (Astrid et al., 2021a;[START_REF] Feng | MIST: Multiple instance self-training framework for video anomaly detection[END_REF].

Table 1.1 summarizes the three datasets with frame-level details. It must be noted that none of these datasets have a validation set, which makes it difficult to choose hyperparameters while training any model.

PID datasets

A perimeter intrusion detection system (PIDS) concerns with the detection of intrusions in a secured perimeter. Like VAD, the intrusions depend on the context and vary from one dataset to another. A mathematical definition for PID task will be presented in Chapter 3. PID can include various subtasks such as detection and tracking, therefore historically algorithms were tested on the datasets of these subtasks. For example, CAVIAR [START_REF] James L Crowley | Caviar context aware vision using image-based active recognition[END_REF], PETS2006 [START_REF] Thirde | Overview of the PETS2006 challenge[END_REF] and AVSS2007 (AVSS dataset, 2007) have been used to test the tracking module of the PIDS [START_REF] Julien | Two novel motion-based algorithms for surveillance video analysis on embedded platforms[END_REF][START_REF] Julien | Towards real-time and low-latency video object tracking by linking tracklets of incomplete detections[END_REF][START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF]. Concerning the PID task, most works use their private datasets. One recent work introduces a new dataset, called SIC (Cermeño et al., 2018), but it is available under strict conditions and, without annotations, thus omissions cannot be evaluated. Finally, there is only one public dataset, dedicated for the PID task: the i-LIDS sterile zone dataset (i-LIDS Team, 2006) and it has been extensively used in the literature (Buch et Velastin, 2008;[START_REF] Julien | Towards real-time and low-latency video object tracking by linking tracklets of incomplete detections[END_REF][START_REF] Julien | Perimeter-intrusion event classification for on-line detection using multiple instance learning solving temporal ambiguities[END_REF]Buch et Velastin, 2014;Cermeño et al., 2018).

i-LIDS sterile zone dataset

The imagery library for the intelligent detection systems (i-LIDS) consists a library of closed-circuit television (CCTV) video footage datasets for benchmarking video analytics systems. Concerning the PID task, they propose a sterile zone dataset, known as the i-LIDS sterile zone dataset (i-LIDS Team, 2006). This dataset comes with a proper annotation and evaluation procedure to ensure its relevance for the industrial application. The PID objective in this dataset is to detect the presence of people in a sterile zone. It consists of two sites monitored by two different cameras (view 1 in color/monochrome and view 2 in monochrome) as shown in Figure 1.8. There is a security fence in each site (view) and the aim is to detect intrusion as soon as it enters the site. Intruders try to breach the fence in various ways. For example, people may walk, run, crawl, or roll towards the fence and, on occasion, may carry climbing aids, such as a ladder. The intruders can be at three different distances from the camera: close, middle, and far. The cameras recordings are over many days, capturing different times of the day, such as dawn, day, dusk, and night. They also include various weather conditions, such as cloudy, rainy, snowy, and foggy. Furthermore, there are various distractions that might trigger false alerts, like plastic bag or paper moving due to wind, bats, birds, foxes, insects, rabbits, squirrels, shadows through the fence, etc. Figure 1.8 illustrates various distractions and intrusion examples. All these factors like different weather conditions, different times of the day, numerous distractions, and various ways in which the intruder approaches the fence make it a very challenging and realistic dataset. Finally, the challenges present in this dataset are noise, automatic adjustments (switch from color to monochrome images and viceversa), vibrations, progressive / sudden illumination changes, shadows, perspective, and camouflage.

It has two separate disks for training and testing. The PID system is developed using the training disk data and its performance is evaluated with the testing disk data. Each disk contains the two camera views, with over 20 hours of video recorded in the various previously cited situations. All videos are taken at 25 FPS with 720 × 576 frame size resolution. Each disk contains annotations, distractions, and other information for each video. The annotation provided is the time interval of each intrusion event in the video i.e., the entry and exit time of people in the respective scene. the color camera and monochrome camera respectively. Each view has videos with and without intrusions. In both training and testing disk, we have 10 videos from each view without any intrusions. In training disk, we have 216 short videos (≈ 3 minutes), with single intrusion per video. These videos can be used for training and/or validation for example. The testing disk have 7 and 6 very long videos in view 1 and view 2. These long videos range from 36 minutes to 92 minutes and contain between 10 and 31 intrusions per video. Even though the total number of intrusions are also 216 in testing disk, the long-length videos makes it more challenging. All i-LIDS videos contain various distractions and/or intruders but these long videos also have a drastic change in weather and luminosity. This makes this testing dataset even more relevant from practical perspective. Furthermore, the distractions, intrusions and other information is very well distributed in both the views and thus we do not have bias for the choice of a camera view. It must be noted that like VAD datasets, it does not contain an official validation set. However, in unsupervised context, the intrusion videos from the training set can be used as a form of validation set.

Although we have extensive real-life elements in i-LIDS dataset, it has some major drawbacks. First, we only have people as intrusion. This is a limitation because in real-life, intrusion can very well be some other object like a car, a bike, a truck, etc.. Consequently, any good human detector system might work very well in this dataset but that does not necessarily mean that it is a good PIDS. The second important limitation is that we have only two views with very similar settings. This makes it easy for the algorithms (particularly supervised learning based) to learn the scene. A multi-view dataset would have added an additional difficulty in the PID task.

Conclusion

In this chapter, we explored the data acquisition schemes in the surveillance systems, followed by the common challenges and finally the datasets used for the VAD and PID tasks.

Visual camera or color camera is still one of the most used cameras for video surveillance, even though it does not perform well when the scene has low illumination. The infrared camera eliminates this problem of visual camera with infrared lighting, but this lighting has other problems, like it attracts insects and performs poor in rainy condition. Thermal camera relies on the infrared radiation emitted by the objects and hence it does not require an external lighting, but it also has problems like we cannot distinguish an object from its background if their temperature is similar. Since none of the video acquisition device is perfect, often an additional sensor like the depth sensor is used, or multiple camera types are used as a multi-camera system, but even these configurations have their drawbacks. To conclude, one should use the video acquisition scheme depending on the requirements of the site and concerned task.

We highlighted various challenges for surveillance systems into three categories: concerning capturing the scene, concerning complexity of the scene, and concerning objects of interest. The first category comprises challenges like noisy frames, camera vibrations and gain adjustments. The second category includes challenges like illumination changes, dynamic background, scene perspective and shadows. Finally, the last category includes camouflage and foreground aperture as challenges.

We explored various datasets for the VAD and PID tasks with their difficulties and shortcomings. Concerning VAD task, three datasets are mainly used. Out of these three datasets, UCSD Ped2 and CUHK Avenue are single scene dataset, i.e., they have only one scene for training and testing. Ped2 is the smallest and easiest dataset, having anomalies with distinguishable appearance like bike, skate, etc. Avenue dataset is slightly more difficult as it also have anomalies where the normal object has temporal irregularity, like person throwing a bag, running, etc. Finally, the ShanghaiTech dataset is the biggest and most challenging dataset among the three, with 13 different scenes captured by color cameras. There is a good variety of anomalies in this dataset and contains both appearance and temporally irregular anomalies. The fact that the same model needs to understand 13 different scenes and detect anomalies is challenging. However, none of these datasets are perfect: they do not have a validation set, they lack proper anomaly description and correct test annotations (some normal events like ice-cream eating are present in test set but not the training set, causing ambiguity), they lack diversity and have small amount of anomalies, and they contain only short video clips. In real-life, the VAD system needs to protect a site for a long time (atleast few days or weeks) and must face changing scene dynamics like weather, lighting, etc., therefore longer videos are essential. Ideally, we need a new multi-view dataset of long videos, having appropriate anomaly description and annotations, containing diverse anomalies, and accompanied with a validation set.

For the PID task, we have only one public dataset, i.e., the i-LIDS sterile zone dataset and it has just two camera scenes. It is challenging as intruders enter the scene in various ways (like crawling), and there are distractions caused by animals/insects, along with changing scene dynamics like rain, snow, etc.. There is no official validation set in this dataset. It also lacks intrusion types and has only person as intruder. Furthermore, the two scenes are very similar to each other, they are like mirror images of each other. Therefore, we require more datasets for the PID task, each with a validation set. They should have multiple scenes and diverse intruder types like cars, bikes, etc.

Chapter 2

Literature review

This chapter presents a review of existing works for spatio-temporal feature learning in videos, with main focus on video anomaly detection and perimeter intrusion detection task. We first present all necessary background concepts needed for understanding the chapter. We next explore the existing spatio-temporal feature learning approaches, followed by unsupervised approaches. In the next two sections, we review existing methods concerning video anomaly detection and perimeter intrusion detection. Finally, we conclude this chapter with takeaway points along with our chosen direction to tackle these problems.

Contents

Introduction

In this chapter, we present the literature review of existing works for spatio-temporal feature learning on videos, with particular focus on two unsupervised tasks, i.e., video anomaly detection and perimeter intrusion detection. A video consists of spatial and temporal dimensions. We need to understand these space-time characteristics of the video, in order to have a meaningful interpretation of it, which helps in performing various tasks and applications like video object recognition, segmentation, anomalous behaviour detection, etc. Many works address this, but they use labeled data. However for some tasks, there is no available annotated data, such as for video anomaly detection and perimeter intrusion detection. Therefore, we need methods that can understand the spatio-temporal characteristics of videos without labeled data. In this chapter, we will first present some background concepts. Then, we will explore various ways for spacetime feature learning from video, followed by reviewing unsupervised approaches. We will then consider methods of literature concerning video anomaly detection and perimeter intrusion detection respectively.

Background

This section aims to present in a concise manner all necessary background concepts required for understanding the subsequent sections. These concepts are on different types of Artificial Neural Networks (ANNs), which aims at learning patterns within data [START_REF] Christopher | Pattern recognition and machine learning[END_REF]. ANNs originated from various attempts to represent the human brain from a mathematical perspective [START_REF] Warren | A logical calculus of the ideas immanent in nervous activity[END_REF][START_REF] Rosenblatt | The perceptron, a perceiving and recognizing automaton Project Para[END_REF].

The underlying hope was that such representation would provide with pertinent learning capabilities.

The main idea of the neural networks is to construct a model parameterized by weights (from a few hundreds up to a trillion [START_REF] Fedus | Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity[END_REF]) in order to fit a given target function. The learning of models is done with gradient descent and more specifically by using the backpropagation algorithm [START_REF] David E Rumelhart | Learning representations by back-propagating errors[END_REF]. These models are constructed by stacking layers and various types of layers were proposed to handle different types of inputs. For example, convolutional layers [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition[END_REF] were introduced to process images. They have a translational invariance property, which is highly useful for tasks such as object detection in images. For sequential data, recurrent layers were introduced (Jordan, 1990). These layers keep in an internal state, the memory of previous inputs, to help future outcomes.

In the following subsections, we begin reviewing the core concepts of neural networks via the standard feedforward network. Then, we explain the convolutional neural network, followed by recurrent neural network, autoencoder and finally the generative adversarial network.

Feedforward neural network

ANNs use artificial neurons that were designed to loosely mimic the biological human neuron. A human neuron receives inputs through axons and dendrites and, depending on the intensity of the electrical impulse received, in turn generates an impulse to subsequent neurons. For ANN, such functioning is formalized [START_REF] Warren | A logical calculus of the ideas immanent in nervous activity[END_REF] using the following equation:

f (x, w) = σ i w i x i + b (2.1)
Each input signal x i is multiplied by a dedicated weight w i , the resulting signals are summed along with a bias term and fed to an activation function σ that decides if the neuron should activate or not, and propagate the signal to subsequent neurons. Any function can serve as activation function, as long as it is continuous and non-linear. Typical activation functions are described in next subsection. The neurons are stacked to make a layer and stacked layers form a neural network model. The first layer is the raw input data (for example, pixel values of an image or video) and subsequent layers, called hidden layers, extract more abstract information of the input until one reaches the final layer, called the output layer. Figure 2.1 depicts a model with four layers. It is a feedforward neural network, where the term feedforward means that the information flows strictly in a forward direction from the input to the output. It is a fully connected network (or dense network) since each neuron on a layer is connected to all the neurons of the preceding layer. In other words, the activation value of a neuron depends exclusively on the activation values of the neurons of the preceding layer. Concretely, the activations of the neurons in the l th layer, i.e., x (l) , are a linear combination of the values of the preceding layer x (l-1) , passed through a nonlinear function σ:

x (l) = σ W (l) x (l-1) + b l (2.2)
where W (l) and b (l) are the weights matrix and the bias vector respectively. These are the parameters that are tuned during the training stage of the neural network. Usually, x (0) is referred to as input, x (N ) (N being the number of layers in a model) as prediction or output and the remaining x (i) (i = 1, . . . , N -1) as hidden layers.

Activation functions

An activation function defines how the weighted sum of the input is transformed into an output from a node or nodes in a layer of the neural network [START_REF] Haykin | A comprehensive foundation[END_REF][START_REF] Goodfellow | Deep learning mit press[END_REF]. The choice of activation function has a crucial impact on the capability and performance of the neural network, and different activation functions may be used in different parts of the network. Typically, all hidden layers use the same activation function, while the activation function for the output layer depends upon the type of prediction required. Since neural networks are typically trained using the backpropagation algorithm (refer next subsection) that requires the derivative of prediction error in order to update the weights, therefore activation functions are also typically differentiable.

σ(v) = 1 1 + e -v (2.3)
As shown in Figure 2.2, the large values of input incline towards the value 1 in output, while the small values tends closer to 0. This property of sigmoid output can be seen as a probability distribution, and thus it is used in binary classification (with single output neuron) to predict whether a certain example belongs to a fixed pre-defined class or not. However, there are two main issues with this activation function. The first issue is of the vanishing gradient. Since the output of sigmoid saturates (i.e. the curve becomes parallel to x-axis) for a large positive or large negative number, the gradients at these regions is almost zero. When n hidden layers use an activation like the sigmoid function, n small derivatives are multiplied together. Thus, the gradient decreases exponentially as we propagate down to the initial layers. A small gradient means that the weights and biases of the initial layers will not be updated effectively with each training iteration, thus making the learning inefficient. That is why, sigmoid function is not used in hidden layers. The second issue is that the sigmoid outputs are not zero-centered and it is undesirable because it can indirectly introduce zig-zagging dynamics in the gradient updates for the weights.

• Hyperbolic tangent. It is also known as the tanh activation and it squashes the input to [-1, +1] range.

σ(v) = e v -e -v e v + e -v (2.4)
It is very similar to the sigmoid activation and even has the same S-shape as shown in Figure 2.2. Since the output is centered at zero, we do no have the problem of zig-zagging dynamics in gradient updates like sigmoid. We still have the same issue of vanishing gradient but the range of inputs that do not saturate are larger here.

Like sigmoid, it is not suitable for hidden layers but can be used in output layer for binary classification. Since the output of tanh is symmetric around zero, it leads to faster convergence of the neural network.

• ReLU. The rectified linear activation function (ReLU) is a piecewise linear function that outputs the input if it is positive, otherwise, it outputs zero. It is defined as:

σ(v) = max(0, v) (2.5)
It is less susceptible to the vanishing gradient problem since the gradient for all inputs greater than 0 is 1. This is why it is one of the most used activation functions and is mostly used in hidden layers. The main issue with this activation function is called the dying ReLU. It occurs when the neuron gets stuck in the negative side and constantly outputs zero. Because gradient of 0 is also 0, it's unlikely for the neuron to ever recover. This happens when the learning rate is too high or negative bias is quite large. To address this issue, Leaky ReLU is proposed, which allows small negative values when the input is less than zero. ReLU or its variants are usually not in the output layer.

• Softmax. It is not a classic activation function that can be applied on a single neuron. The softmax function turns a vector of k real values into a vector of k real values that sum to 1. The input values can be positive, negative, zero, but the softmax transforms them into values between 0 and 1, so that they can be interpreted as probabilities. It is used as the activation function in the output layer of neural network models that predict a multinomial probability distribution [START_REF] Christopher | Pattern recognition and machine learning[END_REF]. It can be seen as a generalization of the sigmoid function which was used to represent a probability distribution over a binary variable [START_REF] Goodfellow | Deep learning mit press[END_REF]. Since softmax converts the input to a normalized probability distribution (between 0 and 1) that sum to 1, it is used in output layer of networks that do multi-class classification. It is not used for hidden layer activations. Furthermore, it should not be used for multi-label classification, i.e., when an example have more that one labels like a dog and a bone. This is because softmax simply cannot produce more than one label with values close to 1. In this case, sigmoid function is used.

We presented above some of the most common activation functions, with particular focus to functions that are later used in this thesis. There are also other activation functions like maxout, gaussian linear unit (GLU), exponential linear unit (ELU), etc. For hidden layers, ReLU activation function is the most used. However, the application and architecture of the neural network often dictates which functions to use, for example in LSTM (a type of recurrent neural network), sigmoid and tanh functions are used as a ReLU would make learning unstable and maybe even impossible. Finally, for binary class classification with single output value, sigmoid function is used in output layer. Similarly, for multi-class classification, softmax function is the preferred activation function in output layer.

Training

So far we detailed the general architecture of the neural networks along with the various activation functions, we now review the training procedure used to estimate the target function. The aim is to train a model M : X → Y to approximate a unknown target function, where X is called the input space and Y the output space. The training is done to extract the statistical properties of the inputs in order to maximize the posterior probability p(y|x) of getting the correct y ∈ Y given x ∈ X . This maximization is done through the minimization of an objective function:

min W E x,y [ L(M(x, W), y) ] , (2.6)
where E is the expectation, M is the deep network, W represents the weights of M to be set and L is the loss function measuring how close M(x, W) is to the true output. To do this minimization, gradient descent is usually used. The standard way to do such an optimization is Stochastic Gradient Descent (SGD) [START_REF] Bottou | Stochastic gradient learning in neural networks[END_REF]. It computes the average gradient

∆W k-1 = 1 |B| i∈B ∆ W k-1 L (y i , M (x i , W k-1 )) , (2.7)
where (x i , y i ) is the i-th element of the current batch B sampled from the training data, and updates the weights using the backpropagation algorithm [START_REF] David E Rumelhart | Learning representations by back-propagating errors[END_REF] as:

W k = W k-1 -η∆W k-1 , (2.8)
where ∆W k-1 is the gradient at iteration k and η is called the learning rate. For such an optimization method, the speed and convergence towards a minima is highly dependent on the shape of the surface of the objective function with respect to the weights. As shown in Figure 2.3a, the optimization process may bounce back and forth, as the slope is very steep (therefore creating a strong gradient). Multiple update rules have been further developed to prevent such problem: momentum (Qian, 1999) (Figure 2.3b), Adagrad [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF], RMSprop [START_REF] Tieleman | Rmsprop: Divide the gradient by a running average of its recent magnitude. coursera: Neural networks for machine learning[END_REF], Adam (Kingma et Ba, 2015), Adadelta [START_REF] Matthew | Adadelta: an adaptive learning rate method[END_REF], to name the most common.

Convolutional neural networks

In the last subsection, we detailed the basic architecture of a neural network as well as the training procedure used to optimize it. However, so far, we have only considered one type of layer, i.e., the dense layer, where each output neuron is connected to each input through a weight. While this layer is functional, it is not best suited for all types of input. For instance, for processing images, each output neuron would be connected to each pixel, therefore requiring a tremendous amount of weights. These weights would not only be hard to optimize but can also lead to overfitting. Furthermore, the dense layer is not invariant to translations. Translation invariance is a very desirable property for image processing and it refers to the ability to ignore positional shifts, or translations, of the target in the image. For example, a cat is still a cat regardless of whether it appears in the left half or the right half of the image.

To address these problems, layers that operate in a sliding window fashion can be used. As the window slides over the input image, weights are re-used at every target position. Therefore, translated inputs result in feature maps with translated activated neurons. Moreover, as the computations of each neuron within a feature map are independent, one can leverage the parallelism to speed up processing. Layers with such behavior are called convolutional layers. a) Convolutional layer A convolution operation can be mathematically defined as:

o(t) = (a * b)(t) = a(z)b(t -z)dz (2.9)
where a is input, b is kernel and o is the resultant output obtained by convolving input with kernel. This can be interpreted as a weighted operation applied around the neighborhood of the point t. We can easily extend this convolution operation to multiple dimensions.

For image data, we have discrete data as input in the form of pixels. Equation (2.9) can be extended for image data as:

O(i, j) = (I * K)(i, j) = (K * I)(i, j) = m n I(i + m, j + n)K(m, n) (2.10)
where I is input image, K is kernel and O is the convolved output at pixel (i, j). This output is also known as the feature map. Since convolution is commutative, we can observe that in the equation (I * K) = (K * I). The region in the input space (here image) that the kernel is looking at the current point is known as the receptive field. Finally, the displacement in the image that the kernel performs after each convolution operation is termed as the stride. It is to be noted that various modifications to this classic convolution can be obtained by increasing kernel stride, or padding the input image, or using dilation. The reader can refer to the work of [START_REF] Dumoulin | A guide to convolution arithmetic for deep learning[END_REF]. b) Pooling layer This layer is also known as the downsampling layer. It is often placed after obtaining the feature map and it summarizes the most pertinent values of it. In other words, convolutional layers detect features, while pooling layers aggregate them. Pooling layers reduce the amount of weights or parameters in the network, thus speeding up the computation and reducing the chance of overfitting the training data. Like convolution, this layer also have a kernel or filter, which performs the pooling operation over the feature map and then this filter is slided with a certain stride. Two most common pooling operations are max and average, i.e., taking maximum and average of the feature map for a given filter size. Convolutional neural network (CNN) is a type of neural network that uses convolution in at least one of its layers. However, given a model with a single convolutional layer, the context (receptive field) a given neuron can gather is limited to the size of the kernel. Increasing context information requires to increase the size of the kernel and therefore the computation needed as well as the number of weights. A simple solution to this problem is to stack multiple convolutional layers. By doing so, the receptive field will naturally increase, while keeping the desired translational invariance property. Therefore, typical CNNs are composed of many convolutional layers stacked on top of each other, where each layer is followed by an activation function like a standard ANN. As we advance in the layers, the size of feature map should generally decrease. For example, for image classification task, we need a single output from the image. To do so, generally pooling layers are used after each convolutional layer. Alternatively, strided convolutional layers can also be used as they decrease the feature map size. Finally, at the end of last layer in CNN, often a fully-connected (dense) layer is used and it reduces the final ouput to desired dimensions (like dimension 1 for image binary classification). The basic mechanism of training a CNN remains the same as for regular neural networks. Like weights of the feed-forward network, in CNN, the kernel parameters are meant to be learned. The initial weights of an untrained CNN are randomly chosen. Then given the training data and task, a loss function is used to calculate the error. This error is backpropagated to update the kernel parameters. Like the regular ANN, the training is continued until the model converges.

To conclude, CNNs are extremely versatile tools. Thanks to their limited number of weights, they help mitigate over-parameterization problems inherent to dense layers. Furthermore, they are also translation invariant and allow for easy parallelisation by design. As a result, convolutional layers are used nowadays throughout the scientific literature. For instance, 1D-CNNs have been used for automatic speech recognition [START_REF] Kiranyaz | 1d convolutional neural networks and applications: A survey[END_REF] and 2D-CNNs are the standard for image processing [START_REF] Karpathy | Large-scale video classification with convolutional neural networks[END_REF][START_REF] Goodfellow | Deep learning mit press[END_REF][START_REF] Liu | Ssd: Single shot multibox detector[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF].

Recurrent Neural Networks

For the moment we described networks that takes as input spatial signals such as images. However in this manuscript, we are also interested in working with sequential data since we want to extract information from videos. In this case the input data x = (x 1 , . . . , x t , . . . , x T ) is a sequence composed of T elements.

By design, the feedforward and convolutional neural networks are not suited to model long sequential data like a time series, speech, or text. Besides, the CNN can only gather information up to the receptive field size. Although such behavior might be sufficient for entries of fixed shape, for entries of varying length, it would be better to manage to implement some form of memory that can gather a potentially unlimited amount of information. For this purpose, there is another neural network variant, called recurrent neural network (RNN) [START_REF] Michael | Attractor dynamics and parallelism in a connectionist sequential machine[END_REF]. RNNs employ hidden vector denoted h which is recursively updated at each timestep using the element from the input, and from which the output y is predicted as:

h t = σ h (W h x t + U h h t-1 + b h )
(2.11)

y t = σ y (W y h t + b y ) (2.12)
where σ h and σ y are activations function and W h , W y , b h , b y , U h are parameters matrices and biases. As shown in Figure 2.6a, in an RNN, the information feeds cells through a loop. When it makes a decision, it takes into consideration the current input and also what it has learned from the inputs it received previously. This is also demonstrated from Equations (2.11 and 2.12). The RNN has two inputs, the present and the recent past. This is important because the sequence of data contains crucial information about future information. It allows the previous inputs to affect the current output. Due to this internal memory, they have a particular advantage to form a much deeper understanding of a sequence and its context. Classical RNN suffer from two major problems: exploding and vanish gradient [START_REF] Le Cun | Reading checks with multilayer graph transformer networks[END_REF]. The first problem refers when algorithm assigns high importance to the weights, without much reason. The second issue occurs when the values of a gradient are too small and the model stops learning.

These problems were solved with the introduction of the Long Short-Term Memory (LSTM) by [START_REF] Hochreiter | Long short-term memory[END_REF]. LSTM is special kind of RNN, capable of learning long-term dependencies. In standard RNNs, the main unit is composed by a simple structure, such as a single tanh layer. However, LSTM introduces a structure based on four gates, which are used to remember important information, to forget irrelevant information and to select which type of information is used in each iteration during the learning. This is illustrated in Figure 2.6b. This mechanism allows the gradient to backpropagate more easily essentially by smoothing out the update of the hidden vector h at each timestep by using activation functions. Finally, another way to solve the problems of a standard RNN is to use a Gated Recurrent Unit (GRU) [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF]. The GRU is a simplified version of LSTM with only two gates: update gate and reset gate. The update gate is responsible for determining the amount of previous information that needs to pass along the next state. The reset gate is used from the model to decide how much of the past information is needed to neglect; in short, it decides whether the previous cell state is important or not. Finally, these two gates decide what information should be passed to the output. In comparison to LSTM, GRU have lesser training parameters and therefore uses smaller memory and executes faster. However, LSTM can provide better accuracy in larger datasets. Therefore, the choice of RNN should depend on the task and memory constraints.

While extremely effective, the LSTM or GRU networks are not suited for sequence of images or videos as they only process vectors. Therefore, to handle such data, one must first generate a meaningful vectorial representation and then apply the classical LSTM architecture. This representation has the major drawback of not using the spatial information for temporal processing. In order to correct this issue, Convolutional Long Short-Term Memory (ConvLSTM) networks were introduced by [START_REF] Shi | Convolutional lstm network: A machine learning approach for precipitation nowcasting[END_REF]. This architecture uses the convolutions in both the input-to-state and state-to-state transitions and adds peephole connections that allow the network to look at the cell state to make decisions. Due to both convolutions and recurrent neural network in its design, it models well the spatiotemporal relationships in a video. Similarly, ConvGRU [START_REF] Ballas | Delving deeper into convolutional networks for learning video representations[END_REF] also have a similar functioning.

To conclude, RNNs facilitate the modeling of long-term dependencies through their potentially infinite memory. Comparatively, CNNs are limited to a fixed receptive field, which is cumbersome in case of sequences. However, the main drawback of the RNNs formulation is its limitation for parallel computations. As the previous state needs to be computed to output the next one, it becomes mandatory to run the calculations sequentially. Yet, despite such limitation, RNNs have been used with great success for natural language processing [START_REF] Zachary C Lipton | A critical review of recurrent neural networks for sequence learning[END_REF][START_REF] Yonghui | Bridging the gap between human and machine translation[END_REF], speech recognition [START_REF] Graves | Hybrid speech recognition with deep bidirectional lstm[END_REF][START_REF] Sak | Fast and accurate recurrent neural network acoustic models for speech recognition[END_REF], video understanding tasks [START_REF] Ballas | Delving deeper into convolutional networks for learning video representations[END_REF][START_REF] Dwibedi | Temporal reasoning in videos using convolutional gated recurrent units[END_REF]Kim et al., 2018a;[START_REF] Li | Videolstm convolves, attends and flows for action recognition[END_REF], etc.

Autoencoder

The Autoencoder (AE) is a type of feed-forward networks which possesses the autoassociation property [START_REF] Hinton | Autoencoders, minimum description length and helmholtz free energy[END_REF]Wang et al., 2016b). It is an unsupervised learning algorithm and provides an alternative to dimensionality reduction techniques like principal component analysis. It projects the data from a higher dimension to a lower dimension using linear transformations and tries to preserve the important features of the data while removing the non-essential parts. The main goal of this type of networks is to learn how to reconstruct the data from a lower dimensional space representation. Figure 2.7 depicts the classical architecture for AE. It tries to learn an approximation of the identity function, to output x that is similar to x. The identity function seems a trivial solution for this, but it can be avoided by placing constraints on the network, such as by limiting the number of hidden units. AE can be divided into three parts: encoder, code and decoder. The encoder compresses or downsamples the input into a lower dimensional representation. The space represented by this new dimensionality is often called the latent-space or bottleneck and contains the semantic representation or the code of the input. The decoder intends to reconstruct the input using only the encoding of the input. AEs have generalized the idea of encoder and decoder beyond deterministic functions to stochastic mappings f encoder (h|x) and g decoder (x|h) . The goal is to minimize arg min f,g x -(f o g)(x) 2 . The traditional AE and PCA are not suitable for images or video frames as they ignore the spatial structure and location of pixels, which is termed as being permutation invariant. Furthermore, it is important to note that when working with images, even with a small size like 100 × 100, these methods introduce large redundancy in network parameters due to dense connections. Therefore, for spatial data of images, we need a new form of autoencoder, which is presented below.

Convolutional autoencoder

The CNN successfully understands spatial characteristics of images, with many times less parameters than the fully connected networks due to convolution and pooling operations. Inspired from this, these operations are used in the autoencoder to obtain a convolutional autoencoder (CAE) [START_REF] Masci | Stacked convolutional auto-encoders for hierarchical feature extraction[END_REF]. CAEs have fewer parameters on account of their kernels being shared across many input locations/pixels. Like CNN, they use convolutional and pooling layers, or strided convolutional layers, for encoding the input image. Similarly, for decoding, the deconvolutional layer or unpooling layer, or both are used. Finally, the mean squared error loss between input and reconstructed images are back-propagated for learning. We present below the deconvolutional and unpooling layers.

Deconvolutional layer and unpooling

A deconvolutional layer in a neural network is a layer which is able to obtain a dense map from downsampled and course input [START_REF] Dumoulin | A guide to convolution arithmetic for deep learning[END_REF]. A more appropriate name is the transposed convolutional layer, as the term deconvolution is misleading since deconvolutional layers also perform convolutions. A transposed convolution has a transformation going in the opposite direction of a normal convolution, i.e., from something that has the shape of the output of some convolution to something that has the shape of its input while maintaining a connectivity pattern that is compatible with said convolution. Pooling layers in convolutional networks are required in order to decrease the number of network parameters. Unpooling layers perform the reverse of pooling layers (see Figure 2.8).

Figure 2.8: Schematic representation of pooling, unpooling, convolution and deconvolution. Source: [START_REF] Noh | Learning deconvolution network for semantic segmentation[END_REF].

The location of the maximum activation in the pooling layer is recorded in switch variables [START_REF] Noh | Learning deconvolution network for semantic segmentation[END_REF], in the unpooling layer it is placed back. The output of such an unpooling layer is sparse, as it is an enlarged version of the input map. The deconvolutional layer can produce a dense output map from the unpooling layer output. In summary, convolutional layers map multiple activations in a receptive field to a single activation, deconvolutional layers map one single activation to a field or window of multiple activations. Convolutional layers learn a filter which map a receptive field to one value, consequently deconvolutional layers learn filters that perform the opposite.

Concerning video data, CAE can be used in each image of the sequence but it will not take into account the temporal dimension. Therefore, Convolutional LSTM based autoencoder (CLSTM-AE) was proposed (Chong et Tay, 2017). It uses first few convolutional layers for encoding, then later in deeper layers, it uses the ConvLSTM layers.

Similarly for deconding, it first uses the ConvLSTM layers, followed by deconvolutional layers to get the final video reconstruction. In conclusion, autoencoder is an easy to use, unsupervised learning algorithm, and can be applied to vector, images, videos, etc.

Generative adversarial network

A Generative adversarial network (GAN) is a category of generative models, first proposed by [START_REF] Goodfellow | Generative adversarial nets[END_REF], that consists of two sub-networks in competition, a generator and a discriminator network, as shown in Figure 2.9. During the learning phase, the generator try to generate convincing data to fool the discriminator who in turn tries to detect whether the data is real or fake. In this way we obtain two trained networks, one to generate realistic data and the other to distinguish between real data and generated data. The generator network is often an autoencoder which generates some input for the discriminator, given a noisy input. The discriminator is usually a fully connected network that takes the real and generated input and predicts whether the generated input is fake or not. Like last subsection, for image data, convolutional autoencoders are use as generators and CNNs are used as discriminators. Similarly, for sequences of images, ConvLSTM or ConvGRU can be a part of the GAN architecture. Finally, the ability of GANs to generate meaningful representations has gained a major interest in many applications like object detection [START_REF] Li | Perceptual generative adversarial networks for small object detection[END_REF], data generation [START_REF] Ehsani | Segan: Segmenting and generating the invisible[END_REF], super resolution [START_REF] Ledig | Photo-realistic single image super-resolution using a generative adversarial network[END_REF], etc.

Spatio-temporal feature learning from videos

In this section, we present different methods for learning spatio-temporal features from a video stream. Since videos have both spatial and temporal dimensions, we explore in this section, various ways to extract this spatio-temporal information. The methods presented below are supervised as they were the first to be developed for spatio-temporal feature learning and furthermore, they act as a baseline for unsupervised approaches. As a reminder, in supervised learning we have as input the data with its label for training the neural network. We present below various approaches, based on how they treat the spatial and temporal dimensions of video.

Modeling spatial and temporal dimensions independently, processing them sequentially

One of the first works to obtain features from video was to use image-based 2D CNN extractors on each frame of the video and then pooling their predictions across the whole video [START_REF] Karpathy | Large-scale video classification with convolutional neural networks[END_REF]. It is very convenient to use, but it ignores the temporal structure of the video, e.g., these models potentially cannot distinguish between opening and closing of a gate. Therefore, it is essential to model temporal data in the video stream. One way to continue using image-based extractors without compromising temporal information is to first extract spatial feature vector by using a 2D CNN shared over all frames and then modeling the temporal representation of this sequence of spatial features using a recurrent neural network [START_REF] Donahue | Long-term recurrent convolutional networks for visual recognition and description[END_REF][START_REF] Ng | Beyond short snippets: Deep networks for video classification[END_REF]. Since traditional vanilla RNN suffers from the vanishing gradient problem, most of the existing methods adopt gated RNN architectures like LSTM, to model the long-term temporal dynamics in video sequences [START_REF] Du | Rpan: An end-to-end recurrent pose-attention network for action recognition in videos[END_REF][START_REF] Sun | Lattice long short-term memory for human action recognition[END_REF][START_REF] Perrett | Ddlstm: dual-domain lstm for cross-dataset action recognition[END_REF]. Figure 2.10 demonstrates this approach, where first the spatial information is extracted from each video frame using a 2D CNN, then this sequence of spatial information is fed into an LSTM network which temporally relates it, and finally the overall information is used for the action recognition task. Some other notable works are described as follows. Wu et al. (2019a) leveraged two LSTMs operating on coarse-scale and fine-scale CNN features cooperatively. [START_REF] Majd | Correlational convolutional lstm for human action recognition[END_REF] proposed the C 2 LSTM model which incorporates convolution and cross-correlation operators to learn motion and spatial features while modeling temporal dependencies. To learn the temporal information from both the past and future, some works adopt the Bi-directional LSTM consisting of two independent LSTMs, called the forward and backward LSTM respectively [START_REF] Ullah | Action recognition in video sequences using deep bi-directional lstm with cnn features[END_REF][START_REF] Zhao | Human action recognition based on improved fusion attention cnn and rnn[END_REF]. The forward LSTM is used with past frame sequence like a single LSTM network, while the backward LSTM takes a future frame sequence as input. Finally, the information from both LSTMs are fused together to perform the target task (like action recognition).

This approach of modeling spatio-temporal data also benefited from the introduction of attention mechanisms, whether it is spatial attention [START_REF] Sharma | Action recognition using visual attention[END_REF]Ge et al., 2019;[START_REF] Sudhakaran | Lsta: Long short-term attention for egocentric action recognition[END_REF], temporal attention [START_REF] Meng | Interpretable spatio-temporal attention for video action recognition[END_REF]Wu et al., 2019b), or spatio-temporal attention [START_REF] Li | Videolstm convolves, attends and flows for action recognition[END_REF][START_REF] Liu | Spatiotemporal saliencybased multi-stream networks with attention-aware lstm for action recognition[END_REF]. Sharma et al.

(2016) designed a multi-layer LSTM model, which recursively outputs attention maps weighting the input features of the next frame to focus on the important spatial features. [START_REF] Sudhakaran | Lsta: Long short-term attention for egocentric action recognition[END_REF] introduced a recurrent unit with built-in spatial attention to spatially localize the discriminative information across a video sequence. [START_REF] Li | Videolstm convolves, attends and flows for action recognition[END_REF] proposed a Video-LSTM incorporating convolutions and motion-based attention into the soft-attention LSTM (Xu et al., 2015b), to better capture both spatial and motion information. Compared to LSTM, Gated Recurrent Unit (GRU) has fewer gates, leading to fewer model parameters. Therefore, many works use GRU instead, for modeling spatiotemporal data in videos [START_REF] Ballas | Delving deeper into convolutional networks for learning video representations[END_REF][START_REF] Dwibedi | Temporal reasoning in videos using convolutional gated recurrent units[END_REF]Kim et al., 2018a).

Some drawbacks of the presented approach to model spatio-temporal are as follows. Since this approach is sequential, if the 2D CNN fails or performs poorly, the complete approach will fail. Networks like LSTM are often costly in terms of memory and time. Finally, this approach has difficulties to model fine-grained action since local motion is generally hard to model with it. This is because the recurrent network models sequence of spatial information extracted for the whole frames, thus it captures the propagation of global frame level spatial information over time, thus not paying special attention to local motion.

Two-stream modeling

Different from the last approach, in this approach two streams, i.e., appearance (spatial) and motion (temporal) are used independent of each other and later fused to obtain the overall spatio-temporal features. It usually works better than the last approach since it does not process space and time dimensions sequentially. [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF] first introduced the two-stream network consisting of two parallel networks, i.e., spatial and temporal network. The spatial network accepts raw video frames while the temporal network gets multi-frame-based optical flows as input. The final score is obtained by fusing scores from both streams. Optical flow provides an effective motion representation in the scene and can effectively remove the non-moving background information [START_REF] Berthold | Determining optical flow[END_REF]. The proposed network had very high performance on existing benchmarks, while being very efficient to train and test. Figure 2.11 shows the two-stream concept of [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF]. In recent years, the two-stream modeling has seen big improvements and is being used massively in many video tasks. Regarding neural networks, deeper networks have been introduced for both streams and they improve the performance (Wang et al., 2015b(Wang et al., , 2016a)). But deeper networks tend to overfit on the video dataset (especially smaller datasets) (Wang et al., 2015a) and therefore Wang et al. (2015b) introduced a series of good practices, including cross-modality initialization, synchronized batch normalization, corner cropping, data augmentation, large dropout ratio, etc. to prevent it. Since there are two streams, there is a stage called spatio-temporal fusion, where results from both the networks are merged to obtain the final score. The standard way is late fusion, which is a weighted average of final scores from both streams [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF]Wang et al., 2015b). Many works claim that late fusion is not the optimal solution [START_REF] Feichtenhofer | Convolutional twostream network fusion for video action recognition[END_REF][START_REF] Feichtenhofer | Spatiotemporal multiplier networks for video action recognition[END_REF]. For example, [START_REF] Feichtenhofer | Convolutional twostream network fusion for video action recognition[END_REF] shows that a fusion at an intermediate layer not only improves the performance but also reduces the number of parameters significantly as compared to original work of [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF]. As video is a temporal sequence of frames, researchers have also explored various recurrent neural networks with two-stream networks like LSTM [START_REF] Sun | Lattice long short-term memory for human action recognition[END_REF][START_REF] Li | Videolstm convolves, attends and flows for action recognition[END_REF], bi-directional LSTM [START_REF] Ullah | Action recognition in video sequences using deep bi-directional lstm with cnn features[END_REF], hierarchical multi-granularity LSTM (Li et al., 2016), etc. [START_REF] Li | Videolstm convolves, attends and flows for action recognition[END_REF] proposed VideoLSTM, a two-stream network with an LSTMbased spatial attention mechanism and a lightweight motion-based attention mechanism. It improves results on action recognition task and furthermore, the learned attention can be used for action localization as well. Other relevant works are Lattice-LSTM [START_REF] Sun | Lattice long short-term memory for human action recognition[END_REF], ShuttleNet [START_REF] Shi | Learning longterm dependencies for action recognition with a biologically-inspired deep network[END_REF], FASTER [START_REF] Zhu | Faster recurrent networks for efficient video classification[END_REF], etc. Two-stream networks still cannot capture long-range temporal information. To address this issue, Wang et al. (2016a) proposed a temporal segment network which first divides a video into uniformly distributed segments, then randomly selects one video frame within each segment and feeds them to the network which shares weights for input frames from all the segments. In the end, a segmental consensus is performed with one of the operators like average pooling, max pooling, weighted average etc., to aggregate information from the sampled video frames. Since this network uses the whole video as input and provides video-level prediction, it models long-range temporal structure. Other relevant works are DVOF [START_REF] Lan | Deep local video feature for action recognition[END_REF], TLE (Diba et al., 2017b), TRN [START_REF] Zhou | Temporal relational reasoning in videos[END_REF], TSM [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF], etc.

To conclude, the two-stream approach is better than the last approach since it takes care of both spatial and temporal dimensions of the video stream. This approach does not treat the spatial and temporal dimensions jointly but instead relies on an external component to extract the motion information. If the motion information is correctly extracted, the spatial and temporal networks can learn the features of video, which are later fused together to make the final decision. This approach has seen a big evolution since its beginning and is still being largely used in many video tasks. If the joint spatiotemporal information is not very important for an application and one can rely on external detectors for motion information, then this approach provides a good alternative.

Jointly modeling spatio-temporal dimensions

A conceptually simple way to understand a video is to consider it as a 3D tensor with two spatial and one temporal dimension. One of the simplest ways to jointly model spatial and temporal dimensions is to extend 2D convolutions into three dimensions. It is called a 3D convolution and it is an extension of 2D convolution which consists of learning space-time kernel filters instead of space kernel filters only. Figure 2.12 demonstrates the difference between 2D and 3D convolutions. [START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF].

A 3D CNN is basically a network like 2D CNN where 2D convolution and pooling operations are replaced by 3D convolution and pooling operations. The seminal work for using 3D CNNs is of [START_REF] Baccouche | Sequential deep learning for human action recognition[END_REF]. While inspiring, the network was not deep enough to show its potential. [START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF] extended it to a deeper 3D network, named C3D. C3D follows the modular design and learns the spatio-temporal features from raw videos in an end-to-end learning framework. It showed strong generalization capability and encouraging performance on video action recognition task but was computationally more expensive than its 2D counterpart. Figure 2.13 demonstrates the schema for video action recognition task using 3D CNN. We can observe that the 3D CNN takes a video volume as input and jointly understand the spatio-temporal features (like 2D CNN do for images) to predict the output. Important characteristics of 3D CNNs are that they directly create hierarchical representations of spatio-temporal data and are very powerful in modeling discriminative features. However, the main issue is that they have many more parameters than 2D CNNs because of the additional kernel dimension, and this makes them harder and longer to train. Since 3D CNN needs high memory space and running time [START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF][START_REF] Qiu | Learning spatio-temporal representation with pseudo-3d residual networks[END_REF], some works propose to factorize 3D convolutions (Xie et al., 2017a;[START_REF] Tran | A closer look at spatiotemporal convolutions for action recognition[END_REF]. Specifically, they decompose 3D convolution kernels (e.g., 3 × 3 × 3) into 2D spatial kernels (e.g., 1 × 3 × 3) followed by a 1D temporal kernels (e.g., 3 × 1 × 1), called as (2+1)D kernel. This allows to reduce the number of trainable parameters significantly while still modeling space-time features. Another alternative to reduce the complexity of 3D CNN is to use 2D convolutions in place of some 3D convolutions in the same network. A temporal shift module is proposed by [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF], which shifts a part of the channels along the temporal dimension to perform temporal interaction between the features from adjacent frames. Different from these parameter-free temporal shift operations, [START_REF] Sudhakaran | Gate-shift networks for video action recognition[END_REF] introduced a lightweight Gate-Shift Module (GSM), which uses learnable spatial gating blocks for spatial-temporal decomposition of 3D convolutions. S3D [START_REF] Xie | Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification[END_REF] combines the merits of approaches mentioned above. It replaces the 3D convolutions at the bottom of the network with 2D kernels and factorizes the remaining 3D kernels into (2+1)D kernels. It demonstrates that this kind of top-heavy network has higher recognition accuracy, along with reduction in model size and training complexity.

The 3D CNN based methods generally perform spatio-temporal processing over fixed time intervals via the window-based 3D convolutional operations, where each convolution operation only attends to relatively short-term context in videos [START_REF] Baccouche | Sequential deep learning for human action recognition[END_REF][START_REF] Ji | 3d convolutional neural networks for human action recognition[END_REF][START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF]. Thus, by design and due to high computational requirements, they are not suitable for long-range spatio-temporal relations in videos. Hence, several approaches focused on modeling these long-term dependencies. Diba et al. (2017a) proposed temporal 3D convolutional networks, where the temporal transition layer models variable temporal convolution kernel depths. It can efficiently capture the appearance and temporal information at short, middle, and long terms. They reinforced this in their subsequent work [START_REF] Diba | Spatio-temporal channel correlation networks for action classification[END_REF], with the introduction of a new block with residual connections, which can model the inter-channel correlations of a 3D CNN with respect to the temporal and spatial features. A long-term temporal convolution framework was proposed by [START_REF] Varol | Long-term temporal convolutions for action recognition[END_REF] to model the long-term temporal information in videos. They increase the temporal extents of 3D convolutional layers at the cost of reducing the spatial resolution. Finally, [START_REF] Hussein | Timeception for complex action recognition[END_REF] proposed Timeception, a multi-scale temporal-only convolutional network to account for large variations and tolerate a variety of temporal extents in complex and long actions.

This approach is a good alternative to the last two approaches. The most important quality of this approach is that it learns jointly the spatio-temporal features of the video. This is an essential point since it depends only the video in hand to learn its feature and does not require external supervision. Even though this approach has seen significant improvement throughout the years, it still remains computationally expensive and cannot treat a large video sequence at once. As a concluding remark, this approach should be used when we want to model the spatio-temporal dimension correctly in the video and when the long range temporal dependencies are not important.

Hybrid approaches

To further enhance the spatio-temporal comprehension of videos, several works have investigated using 3D CNN based models with two-stream or multi-stream designs (Carreira et Zisserman, 2017;[START_REF] Wang | Two-stream 3-d convnet fusion for action recognition in videos with arbitrary size and length[END_REF][START_REF] Feichtenhofer | Slowfast networks for video recognition[END_REF][START_REF] Li | Spatio-temporal deformable 3d convnets with attention for action recognition[END_REF]. Carreira et Zisserman (2017) introduced the two-stream inflated 3D CNN (I3D), made by inflating the convolutional and pooling kernels of a 2D CNN with an additional temporal dimension. Similarly, [START_REF] Wang | Two-stream 3-d convnet fusion for action recognition in videos with arbitrary size and length[END_REF] integrated a two-stream 3D CNN with an LSTM model to capture the long-range temporal dependencies. To better recognize actions with low inter-class variability, [START_REF] Martin | Fine grained sport action recognition with twin spatio-temporal convolutional neural networks: Application to table tennis[END_REF] introduced a new siamese network with two streams using 3D convolutions. SlowFast network [START_REF] Feichtenhofer | Slowfast networks for video recognition[END_REF] extend the idea of two-stream approaches but without the need of optical flow as input. This 3D CNN based two-stream network have a slow pathway and a fast pathway that operate on frames at low and high frame rates to capture semantic and motion, respectively. At each layer of this network, the features of the fast and slow pathways are fused (by summation or concatenation) to share the motion and semantic information among sub networks. Finally, [START_REF] Li | Spatio-temporal deformable 3d convnets with attention for action recognition[END_REF] introduced a two-stream spatio-temporal deformable 3D CNN with attention mechanisms to capture the long-range temporal and long-distance spatial dependencies. Recently, a new neural network, called the Transformer was introduced [START_REF] Vaswani | Attention is all you need[END_REF]. It mainly utilizes the self-attention mechanism [START_REF] Bahdanau | Neural machine translation by jointly learning to align and translate[END_REF] to extract intrinsic features and shows great potential for extensive use in AI applications. It was first applied to NLP for machine translation tasks [START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Devlin | Bert: Pretraining of deep bidirectional transformers for language understanding[END_REF]. Only recently, it has been applied in vision tasks [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF]Chen et al., 2020), but it has shown a very strong potential with performance equal or better than CNN based approaches for various tasks [START_REF] Zeng | Learning joint spatial-temporal transformations for video inpainting[END_REF][START_REF] Khan | Transformers in vision: A survey[END_REF]. Transformers also have various drawbacks like they require very massive datasets for large-scale training, extremely heavy computational requirements, and furthermore generalization and robustness of transformers is still an open issue [START_REF] Han | A survey on vision transformer[END_REF].

In this section, we studied different types of approaches to learn spatio-temporal features from video. We observed that two-stream, 3D CNN and hybrid approaches all rest viable for effective learning in video. While two-stream approaches require an external detector for motion and does not treat the spatial and temporal dimensions jointly, the 3D CNN based approach requires high computational power. Finally, the hybrid approaches takes best of the both worlds but their drawbacks does accompany these approaches, like they remain computationally expensive. Finally, the choice of approach depends on the task at hand and requirements. In this thesis, we want to learn spatio-temporal features jointly in video and furthemore we want to use minimum supervision possible. For the applications, i.e., video anomaly detection and perimeter intrusion detection, we do not require long spatio-temporal information. For example, an intrusion that was detected 10 minutes ago, does not have any direct relation in detecting the current intrusion. Therefore, we choose the 3D CNN based approach in this thesis. The next section will present how can we use these approaches in an unsupervised context.

Unsupervised spatio-temporal video understanding

This section presents various approaches for unsupervised learning of spatio-temporal features from the video stream. To recall, in unsupervised learning, we do not have labels with data during training the neural network. In the last section, we explored various approaches to obtain spatio-temporal features. All the approaches, whether 2D CNN with a recurrent network, two-stream modeling or 3D CNN, can be used in the context of unsupervised learning. We just need to employ them either in reconstructive, predictive, or generative models, that are presented below.

Reconstruction models

The main idea here is to reconstruct a given input, in order to learn pertinent features of the input data. The input data can be a vector, image or a video sequence. In our case, it is the video sequence. These models include methods such as auto encoders (refer Section 2.2.4) or sparse coding, that are used to extract different linear and non-linear representations of appearance (image) or motion (stream) or both, to model meaningful patterns of unlabeled videos. We can simply extend the idea of CAE for images to videos and use many of the video feature extraction approaches described in last section. [START_REF] Hasan | Learning temporal regularity in video sequences[END_REF] proposed a spatio-temporal stacked autoencoder with a video clip of ten frames as input. It consists of a stack of 2D convolutions and 2D deconvolutions for encoding and decoding, respectively. Similarly, Chong et Tay (2017) introduced a convolutional LSTM based autoencoder, with the same input of ten frames. They used 2D convolutions with LSTM for encoding and LSTM with 2D deconvolutions for decoding.

By using LSTM, they were able to better understand the temporal characteristics of video stream. Similarly, 3D convolutional autoencoder [START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF] encodes and decodes using 3D convolutions and 3D deconvolutions respectively. In theory, any approach of learning spatio-temporal features from videos described in Section 2.3 can be used with an autoencoder. More works on unsupervised deep learning on videos will be detailed in upcoming sections for video anomaly detection and perimeter intrusion detection tasks.

Predictive models

Another approach based on unsupervised deep learning tends to use predictive modeling. It is well-known in time series analysis under auto-regressive models. Different from reconstructive models where the objective is to train a model to reconstruct the input data, predictive models try to predict a current sequence of input using the previous sequences. Concerning video data, the objective is to model output of current frame (or future frame) as a function of past t frames. Since it concerns temporal sequence, traditionally recurrent neural networks like LSTM were used [START_REF] Srivastava | Unsupervised learning of video representations using lstms[END_REF]Luo et al., 2017b). Recently, there have also been attempts to perform efficient video prediction using convolutional autoencoder networks. The function of an AE can be determined by considering its output values. When the output values are only the reconstruction of the inputs, the AE is a reconstructive model. When the output values are the values after the input values in the time axis, the model is said to be predictive. Medel et Savakis ( 2016) introduced a ConvLSTM-based AE where the encoder extracts representations from an input sequence, a first decoder that uses these representations to reconstruct the input sequence, and a second decoder that uses the same representations to predict the next frame. Similarly, [START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF] proposed a network made up of an encoder and two decoders, the first for reconstruction and the second for prediction. In this network, 3D convolution layers are used instead of ConvLSTMs, for the learning of spatiotemporal representations. More works will be discussed with VAD as the application in later sections.

Generative models

Generative models like Generative Adversarial Networks (GAN), Adversarially trained AutoEncoders (AAE) and Variational Autoencoders (VAE) are used for the purpose of modeling the likelihood of video samples in unlabeled data. These models are also used massively in unsupervised video learning tasks like video anomaly detection [START_REF] Ravanbakhsh | Abnormal event detection in videos using generative adversarial nets[END_REF][START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF]. For example, [START_REF] Lee | Stan: Spatio-temporal adversarial networks for abnormal event detection[END_REF] proposed a generative adversarial network, called STAN (spatio-temporal adversarial networks) for this VAD task. They use a video clip as input. The middle frame of the video clip is removed and the resultant clip is fed to a spatio-temporal generator. The generator is a convolutional autoencoder with convolutions, ConvLSTMs and deconvolutions. It generates the missing middle frame, which is added to its position in the video clip. The spatio-temporal discriminator, made using 3D convolution layers, takes the generated video clip and original video clip as input and tries to distinguish the clips. Once the two networks are trained, the detection of abnormal events is done using the losses from both generator and discriminator. More unsupervised generative works are detailed in later sections.

In this section, we presented various ways to unsupervisely learn spatio-temporal features of video. We can observe that all of these approaches are different and none of them is more suitable than others. Infact, they can also be combined together if required, and can potentially lead to better generalization. All of these approaches have been extensively developed in the last few years. Choosing the correct approach, depends on the video task being tackled, its requirements, and the spatio-temporal feature learner (like a 2D CNN, 3DCNN, etc.) being used. Our chosen 3D CNN based approach translates to 3D CAE in unsupervised learning. This 3DCAE can be used as a reconstructive model, predictive model and even as the generator of a generative model. In this thesis, we explore the first two ways and even combine them. The generative model requires extra computation on top of the generator via discriminator, therefore we avoided that approach to be computationally light.

Video anomaly detection (VAD) methods

Given a video stream, this task deals with the detection of anomalies. The anomalies are context dependent and vary from one dataset to another as described in Section 1.4.1. Since anomaly is a rare event and the datasets are usually not annotated. Therefore, most methods use unsupervised deep learning, with or without external supervision from pre-trained detectors. We focus only on these methods. It must be noted that there are other methods which rearrange the current VAD datasets to have weak labels and thus use weakly supervised learning. Some methods use semi-supervised learning by using a portion of labeled data for training. Finally, there are also a few methods that reorganize VAD datasets to use supervised learning. We do not review these kind of methods as they are well out of the scope of this study.

To systematically review the VAD methods, we first explore several types of input that they can take, followed by different proxy tasks that they can perform and finally some auxiliary components that they use to enhance their performance.

Types of input

To understand different approaches for video anomaly detection, we must first consider the different types of input they use. Even though the task is detecting anomalies in video, the methods can use frames of video, frame patches, objects in frame, objects in video snippet, etc. as input. We explain below different input types used for VAD.

1. Full Frame (FF): This input means using a single frame as input. This means that the video is read frame by frame and each frame is an input to the VAD method. It can be used with any VAD task but has been predominantly used in the reconstruction task [START_REF] Ravanbakhsh | Abnormal event detection in videos using generative adversarial nets[END_REF][START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Ye | AnoPCN: Video anomaly detection via deep predictive coding network[END_REF]. One major issue with using FF as input is that we miss to capture the temporal information of video sequence.

Frame patch (FP):

This input is made by dividing the video frame into smaller patches. The video is read as frames which are converted into smaller patches and each patch is considered as a separate input. If an anomaly is detected in one of the frame patches, then the frame is labeled as anomalous [START_REF] Sabokrou | Adversarially learned one-class classifier for novelty detection[END_REF]. We have similar problem here as we lose the temporal dimension and due to patches, we loose part of the spatial information.

3. Frame object (FO): It refers to the objects in the video frame. A pre-trained object detector is first used to detect all possible objects of interest and then these objects are used as the input to the method [START_REF] Radu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF]Georgescu et al., 2021b). Since the input is object, rather than the whole frame, we do not have background noise. The problem is that it is highly biased to the external dataset where the object detector was trained, it assumes that objects of interest causing anomalies are known in advance and finally if the object detector fails, the VAD will fail too.

4. Video clip (VC): It refers to a video snippet with a fix number of consecutive frames.

It is one of the most widely used input for VAD task [START_REF] Hasan | Learning temporal regularity in video sequences[END_REF][START_REF] Lee | Stan: Spatio-temporal adversarial networks for abnormal event detection[END_REF][START_REF] Dong | Dual discriminator generative adversarial network for video anomaly detection[END_REF][START_REF] Luo | Video cloze procedure for self-supervised spatio-temporal learning[END_REF][START_REF] Park | FastAno: Fast anomaly detection via spatio-temporal patch transformation[END_REF]. Since it is a video, we have both spatial and temporal dimensions to exploit. It has been used with all the proxy tasks for VAD. However, it takes more time and resources to process video clips and furthermore, it is comparatively difficult to extract pertinent information since often the anomalies occupy a small spatio-temporal volume in the video clip. et al., 2021b;Georgescu et al., 2021a). The problem with this input is same as that of frame object due to its dependence on pre-trained object detector.

Types of tasks

Since anomaly is a rare event, there is lack of anomaly examples for training a supervised two-class classifier. Due to this, most works address the VAD task using a proxy task like frame reconstruction, prediction, etc. These tasks are called proxy tasks since they address the main task (VAD is binary classification task) indirectly. We describe below the classification of approaches based on the proxy task that they use.

Reconstruction task

This task concerns the reconstruction of an input entity like frame, video clip, object, etc. The aim is to learn normality by trying to reconstruct entities in normal videos (without any anomaly). Since the representations were learned from normal videos, it is assumed that anomalies will be harder to reconstruct and thus they can be separated from correctly reconstructed normal entities. This task is usually performed using a convolutional autoencoder [START_REF] Masci | Stacked convolutional auto-encoders for hierarchical feature extraction[END_REF] or an adversarial network like a GAN [START_REF] Goodfellow | Generative adversarial nets[END_REF]. Autoencoder (AE) encodes the input data into the latent space through an encoder and then reconstructs it using a decoder. The anomaly measure is (2016), using reconstruction task for video anomaly detection. They use a video clip of 10 frames as input and reconstructs it. They use 2D convolutions and pooling layers for encoding and 2D deconvolutions and unpooling layers, for decoding. The mean squared error (MSE) between input clip and its reconstruction is used as the reconstruction error (anomaly measure). By using only 2D convolutions, the temporal features of the video clip cannot be taken into account. To address this, Chong et Tay (2017) proposed to add convolutional long-short term memory (LSTM) layers in both encoder and decoder. Taking it one step further, [START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF] proposed a 3D convolutional autoencoder with 3D convolutions and 3D deconvolutions for VAD. Unlike 2d convolutions plus LSTM, the 3D convolutional autoencoder jointly captures the spatio-temporal features from normal videos. Recently, many works used some AE variant for VAD, using FF, VC or VCO as input [START_REF] Ye | AnoPCN: Video anomaly detection via deep predictive coding network[END_REF][START_REF] Ouyang | Video anomaly detection by estimating likelihood of representations[END_REF]Astrid et al., 2021b;[START_REF] Park | FastAno: Fast anomaly detection via spatio-temporal patch transformation[END_REF].

One of the earliest works for VAD using GANs, with reconstruction as proxy task, was proposed by [START_REF] Ravanbakhsh | Abnormal event detection in videos using generative adversarial nets[END_REF]. Two conditional GANs are trained, with input as pairs of frames and noise vectors, which generate corresponding frames of a different modality (raw frames to optical flows and vice versa in the two GANs). The discriminators are asked to classify pairs of input and generated representations of frames as real or fake. Assuming that anomalies are not reconstructed well, they fuse reconstruction errors from both modalities as an anomaly score. Some works use both AEs and GANs for the reconstruction task [START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Ye | AnoPCN: Video anomaly detection via deep predictive coding network[END_REF]. One such work is proposed by [START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF], where they learn a correspondence between common object appearances and their associated motions in a two-stream model. Using an FF as input, they use a single encoder coupled with two decoders, one that predicts motion and another that reconstructs the input frame. This entire network is considered as a generator in a conditional GAN setup, where the discriminator is another network that distinguishes between pairs of input frames and corresponding real/estimated flow fields. For testing, they calculate loss scores at a patch-level. For other VAD works with reconstruction tasks, the reader can refer to [START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF][START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF]. The main drawback of using this task is that sometimes the neural network generalize too well and even reconstruct the anomalies very well. In that case, the difference between normal and abnormal sample is neglible and thus anomaly cannot be detected.

Prediction task

This task deals with the prediction of an unknown entity, given a known entity. For example, predicting a future frame, given a video clip with present and previous frames. Concerning VAD, the future frame prediction task is one of the most used tasks [START_REF] Liu | Future frame prediction for anomaly detection-a new baseline[END_REF][START_REF] Lv | Learning normal dynamics in videos with meta prototype network[END_REF][START_REF] Park | Learning memory-guided normality for anomaly detection[END_REF]. Other prediction tasks for VAD include future object prediction (Liu et al., 2021b) or missing bounding box prediction (Georgescu et al., 2021a). The input entities used are VC, VCO and VCP, with VC being the most common [START_REF] Lee | Stan: Spatio-temporal adversarial networks for abnormal event detection[END_REF][START_REF] Dong | Dual discriminator generative adversarial network for video anomaly detection[END_REF]. Models performing prediction task are also trained only on normal videos, with the assumption that they can precisely predict the entities in normal test sequences but will fail to correctly predict in anomalous test sequences. This requires comprehension of how normal spatio-temporal patterns propagate along the video clip. The anomaly score for this task can be calculated by measuring the difference between real and predicted entities or by calculating the conditional probability of a new observation based on the previous samples [START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF]. Autoencoders, GANs and their combination are typically used for this task. [START_REF] Liu | Future frame prediction for anomaly detection-a new baseline[END_REF]. They propose a GAN, where the encoder is a U-Net style network [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] that takes training video clips of length t as input and predicts the future frame t + 1. The discriminator network checks if the future frame is real (ground truth) or predicted, by minimizing the intensity and gradient loss. Furthermore, they add motion (temporal) constraint by using FlowNet [START_REF] Dosovitskiy | Flownet: Learning optical flow with convolutional networks[END_REF] to estimate pairs of optical flow maps between the frame at t and real or predicted frame at t + 1. The anomaly score is composed of L1 score between flow maps and intensity plus gradient scores from frame prediction. Some other prominent VAD works that use GANs for prediction task are [START_REF] Lee | Stan: Spatio-temporal adversarial networks for abnormal event detection[END_REF][START_REF] Dong | Dual discriminator generative adversarial network for video anomaly detection[END_REF], while works using AE for prediction tasks are [START_REF] Park | Learning memory-guided normality for anomaly detection[END_REF][START_REF] Lv | Learning normal dynamics in videos with meta prototype network[END_REF][START_REF] Viet-Tuan Le | Attention-based residual autoencoder for video anomaly detection[END_REF][START_REF] Park | FastAno: Fast anomaly detection via spatio-temporal patch transformation[END_REF]. Since prediction task depends on the given input to learn the features in order to predict the unknown entity, it can fail when the input contains only stationary anomalies. For example, for the future frame prediction task, when the input clip contains an stationary car (anomaly), the model is likely to predict this car in future frame too. Thus, the prediction error would be too small to differentiate it from the normality.

Self-supervised task

This type of task uses self-supervisory signals from the data itself and does not require external annotations. In other words, identify any hidden part of the data from any unhidden part of the data, e.g., predict the missing patch of an image, given the image without patch. This learning paradigm is known as self-supervised learning. In recent years, the self-supervised learning is being used massively for different applications in various domains (Liu et al., 2021a) like natural language processing (NLP), robotics, computer vision: applied to image and video analysis, etc. It is typically used as a pretraining step (pretext task) to enrich a learning module, which is later used for supervised downstream tasks like video classification, detection, etc. [START_REF] Jing | Self-supervised visual feature learning with deep neural networks: A survey[END_REF]. Some well-known self-supervised tasks for video representation learning are video playback rate perception [START_REF] Yao | Video playback rate perception for selfsupervised spatio-temporal representation learning[END_REF] [START_REF] Yu | Cloze test helps: Effective video anomaly detection via learning to complete video events[END_REF]. DNN refers to a deep neural network and video event is the VCO input.

Concerning VAD, there are only two works that use self-supervised tasks. The first work is by [START_REF] Yu | Cloze test helps: Effective video anomaly detection via learning to complete video events[END_REF], where they propose a new self-supervised VAD task, called video event completion. This task is an adaptation of video cloze procedure task [START_REF] Luo | Video cloze procedure for self-supervised spatio-temporal learning[END_REF]. In this task, first VCO is extracted from a video clip using a pre-trained object detector and a series of appearance and motion-based operations. The VCO is referred as video event here and contains image patches with objects of interest like human. From each video event, one patch is erased randomly and a DNN (U-Net style autoencoder) predicts it using the rest of incomplete video event, known as appearance completion. The MSE between predicted and actual patch is used as an anomaly score. Similarly, motion completion is also done on the same video events: optical flow for each video patch is extracted using a pre-trained network and then similar pipeline like appearance completion is followed using the same U-Net style autoencoder. Again, MSE is used as an anomaly score here. The final anomaly score is a weighted sum of appearance and motion anomaly scores. Figure 2.16 demonstrates this self-supervised video event completion task along with a comparison with reconstruction and prediction tasks. The second work is proposed by Georgescu et al. (2021a), where they perform multiple self-supervised tasks using the same 3D CNN in a multi-task learning paradigm. They use VCO as input, using YOLOv3 object detector [START_REF] Redmon | Yolov3: An incremental improvement[END_REF]. The proposed tasks are: (i) arrow of time prediction (discriminating forward and backward moving objects), (ii) motion irregularity detection (distinguishing objects captured in consecutive frames versus objects captured in intermittent frames), (iii) middle bounding box prediction (given objects in preceding and succeeding frames), (iv) model distillation: estimating normality-specific class probabilities by distilling pre-trained classification (ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]) and detection (MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF]) teachers. Both these works show promising results but they rely heavily on external supervision using object detectors, which inhibits their generalizing capability as discussed in Section 2.5.1.

Classification task

Despite the fact that video anomaly datasets are imbalanced and that only normal classes are present during training, there are still some methods that use a classifier to detect anomalies. Given only normal videos, they either perform one-class classification [START_REF] Hanh | Anomaly detection using a convolutional winnertake-all autoencoder[END_REF][START_REF] Sabokrou | Adversarially learned one-class classifier for novelty detection[END_REF] (2019) proposed one of the first approaches. Their input is FO, which is obtained by using a pre-trained single-shot detector (SSD) [START_REF] Sun | Lattice long short-term memory for human action recognition[END_REF] on each frame of the video. They train convolutional auto-encoders on appearance and gradient features of these objects to learn latent representations and then perform k-means clustering followed by training of k one-class SVMs to make binary one-versus-rest classifications. Each cluster represents a different type of normality. At test time, they simply use the inverse of the maximum of k classification scores as an anomaly score, meaning if a sample does not belong to any cluster, it is considered as anomalous. The classification task fails when the anomaly resemble too much with normality in the latent space, thus making it difficult to separate anomalies from normality.

Multiple task learning

Since no task is perfect for VAD, many approaches use multiple tasks to benefit from advantages of different tasks [START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF][START_REF] Tang | Integrating prediction and reconstruction for anomaly detection[END_REF]Liu et al., 2021b). et al., 2010). They train on normal data to reconstruct the current video clip and predict the future frame video clip. During testing, they use only the reconstruction error of the test video clip as the anomaly measure. They demonstrated that using these tasks together boosts the detection of video anomalies. Even though multiple task learning benefits from advantages of different tasks, it is often difficult to combine various tasks and furthemore this makes the model computationally very expensive. So, the tasks must be chosen wisely with taking into consideration the gain in performance (if there is) in relation to the expense of memory and computation time.

Auxiliary components for enhancing VAD

To enhance the detection of anomalies, often many auxiliary components are used along with the VAD tasks. We describe below some of the most used components.

Optical flow

Since moving objects can cause anomalies, it is important to reinforce motion analysis for VAD systems. Many motion analysis tasks employ optical flow as a fundamental basis upon which more semantic interpretation is built [START_REF] Fortun | Optical flow modeling and computation: A survey[END_REF]. Optical flow is defined as the estimation of a dense motion field, corresponding to the displacement of each pixel [START_REF] Steven | The computation of optical flow[END_REF]. Concerning VAD, approaches use optical flow to enrich the input with motion information before feeding it to the deep neural network [START_REF] Hanh | Anomaly detection using a convolutional winnertake-all autoencoder[END_REF][START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF][START_REF] Ravanbakhsh | Abnormal event detection in videos using generative adversarial nets[END_REF][START_REF] Liu | Future frame prediction for anomaly detection-a new baseline[END_REF][START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Dong | Dual discriminator generative adversarial network for video anomaly detection[END_REF][START_REF] Yu | Cloze test helps: Effective video anomaly detection via learning to complete video events[END_REF]Liu et al., 2021b;Georgescu et al., 2021b).

Pre-trained feature extractor

Instead of learning features from anomaly datasets directly, many methods use a pretrained feature extractor trained on an external dataset. Pre-trained extractors can be used to extract features (of frames / video clips), to extract objects in the video [START_REF] Radu | Object-centric auto-encoders and dummy anomalies for abnormal event detection in video[END_REF][START_REF] Ouyang | Video anomaly detection by estimating likelihood of representations[END_REF]Georgescu et al., 2021a;[START_REF] Viet-Tuan Le | Attention-based residual autoencoder for video anomaly detection[END_REF], or to extract optical flow [START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF][START_REF] Ravanbakhsh | Abnormal event detection in videos using generative adversarial nets[END_REF][START_REF] Liu | Future frame prediction for anomaly detection-a new baseline[END_REF][START_REF] Nguyen | Anomaly detection in video sequence with appearance-motion correspondence[END_REF][START_REF] Dong | Dual discriminator generative adversarial network for video anomaly detection[END_REF][START_REF] Yu | Cloze test helps: Effective video anomaly detection via learning to complete video events[END_REF]Liu et al., 2021b;Georgescu et al., 2021b). The advantage of features extractors for VAD is that they save computational capacity, so bigger architectures can be made, and they can add additional information like optical flow or object information. The main disadvantage is that they are biased towards the external dataset and an incorrect feature/object/optical flow estimation can lead to failure of the whole system.

Data transformation

Data transformation refers to the modification of input data in various ways so that it helps in better anomaly detection. The transformed data is used in two ways: as an augmentation of data to the original input of the model, or as an auxiliary input along with the original input of the model. Some major works using data transformation in the first way are [START_REF] Hasan | Learning temporal regularity in video sequences[END_REF]Chong et Tay, 2017;[START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF] an anomaly. The motivation is to train the AE in order to reduce the reconstruction loss for normal inputs while increasing it for pseudo anomaly inputs, thus making model sensitive to anomalies with temporally incoherent behavior.

Clustering

It is the task of grouping or segmenting a collection of entities into subsets or "clusters", such that the entities within each cluster are more closely related to one another than to those assigned to other clusters [START_REF] Hastie | The elements of statistical learning: data mining, inference, and prediction[END_REF]. Concerning VAD, Chang et al. (2020) clusters the encoder output of an AE with reconstruction as proxy task. They propose a deep k-means clustering in encoder to obtain a more compressed data representation and extract the common factors of variation within the normal dataset. By minimizing the distance between the data representation and cluster centers, normal examples are closely mapped to the cluster center while anomalous examples are mapped away from the cluster center. The cluster centers can be deemed as a certain kind of normality within the training dataset. Since the model is only trained on normal events, the distance between cluster and abnormal representations is much higher than between the normal patterns. Reconstruction error and the cluster distance are together used as an anomaly score. With a similar strategy, [START_REF] Ouyang | Video anomaly detection by estimating likelihood of representations[END_REF] proposes to cluster the latent manifolds of the autoencoder using expectation maximization. Different from these approaches, Ionescu et al. (2019) first clusters the latent space of an autoencoder into different types of normality, and then train a binary classifier following the one-versus-rest scheme to separate normality clusters from one another. In the inference phase, a test sample is labeled as abnormal if the highest classification score is negative, i.e., the sample is not attributed to any normality cluster.

Memory, attention and other units

One of the main challenges with CNN based networks like autoencoder or GAN is that they have very powerful representational capacity which can even hinder to distinguish between normality and abnormality, if used naively. For example, reconstruction-based VAD methods using AE may even reconstruct the abnormal frames well [START_REF] Zong | Deep autoencoding gaussian mixture model for unsupervised anomaly detection[END_REF][START_REF] Liu | Future frame prediction for anomaly detection-a new baseline[END_REF]. Hence, the assumption that reconstruction error is comparatively high for abnormal test frames might be violated.

One popular approach to handle this issue is to use memory units with the network. The prototypical patterns of normal data are recorded into the memory during training, whereas while testing on a test input, the most relevant elements from the memory are retrieved to perform the given task (reconstruction, prediction, etc.). Figure 2.20 shows one such approach, namely MemAE [START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF]. MemAE or memory-augmented autoencoder is a reconstruction task-based VAD method, which learns and updates the memory contents during training, to store the prototypical elements of the normal data. In the test phase, the memory is fixed, and reconstruction is performed using items selected from the memory. Taking this work forward, [START_REF] Park | Learning memory-guided normality for anomaly detection[END_REF] propose a similar strategy to enhance AE for VAD by an improved memory module. They separate stored memory items explicitly using feature compactness and separateness losses, which enables using a small number of memory items compared to MemAE (10 vs 2,000 for MemAE). Furthermore, they also update the memory at test time, while discriminating anomalies simultaneously, suggesting their model memorizes normal patterns of test data. Overall, their model better records diverse and discriminative normal patterns for VAD. Finally, a recent work by Liu et al. (2021b) uses multiple memory units attached in different parts of the AE and they propose an ideal arrangement of these memory units along with skip connections for the VAD task.

Figure 2.20: Proposed schema of MemAE [START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF]. The encoder takes the input x to produce encoding z, which is taken as a query by the memory addressing unit to obtain the soft addressing weights w. The memory slots can be used to model the whole encoding or the features on one pixel (as shown in the figure). The updated encoding z passes via decoder to produce the reconstruction x.

Another important approach to enhance neural network is to use attention modules [START_REF] Zagoruyko | Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer[END_REF][START_REF] Vaswani | Attention is all you need[END_REF][START_REF] Wang | Learning unsupervised video object segmentation through visual attention[END_REF]. In simple words, attention is a technique that imitates human cognitive attention, enhancing a part of input, such as an object and neglecting the remaining parts [START_REF] Zhou | Attention-driven loss for anomaly detection in video surveillance[END_REF]. Concerning VAD, Le et [START_REF] Viet-Tuan Le | Attention-based residual autoencoder for video anomaly detection[END_REF] proposed an attention-based residual autoencoder with future frame prediction as the proxy task. To exploit channel dependency of features, they propose a channel attention module made of two convolutional layers and it is applied in each layer of the decoder. Similarly, Chang et al. (2020) proposed a cluster driven autoencoder with future frame prediction task, where a variance-based attention module is designed to assign an importance to moving part of video clips, thus improving detection of anomalies with large temporal movements like person running, jumping, etc. Different from these approaches, [START_REF] Lv | Learning normal dynamics in videos with meta prototype network[END_REF] proposed an AE-based VAD method with a dynamic prototype unit. They learn diverse patterns of the normal data in the form of prototypes. Prototype is a compact representation of pertinent normal data. A novel attention operation on the AE encoding map assigns a normalcy weight to each pixel location to form a normalcy map. Then, prototypes are obtained as an ensemble of the local encoding vectors under the guidance of normalcy weights. Multiple parallel attention operations are applied to generate a pool of prototypes, which represent diverse and compact dynamics of the normal patterns. Finally, the AE encoding map is aggregated with the normalcy encoding reconstructed by prototypes for latter frame prediction.

Recently, some other types of units have been proposed to tackle with the issue of powerful representational capacity of CNN based VAD networks. Szymanowicz et al.

(2022) proposed a vector quantized autoencoder for detecting video anomalies using the future frame prediction task. They store features of the encoder in a learnable codebook using a vector quantization module. The vector quantization leads to discretization of high-level features into embeddings, which are used by decoder to predict the future frame. Since decoder has no direct access to input features but only the discrete embeddings of codebook, it is unable to correctly predict the anomalous frames, thus improving the video anomaly detection. This signifies that rather than using just the spatial data through a single frame, spatio-temporal data via video clips is essential for VAD. Regarding proxy tasks, reconstruction task is the most preferred, followed by the prediction task, and they are often combined together. The classification task has been used only four times and it uses patch or object type inputs, requiring pre-processing through an external detector or data transformation. This signifies that it is not easy to perform classification task directly on the raw frame or video clip. The SST task has been recently used for VAD and there are only two methods that use them. Both of them use VCO as input, i.e., they need an external object detector to first detect objects and then perform the task. They obtain excellent performances as they do not suffer from scene background and associated noise. However, one must note that in real life, we cannot always detect all the possible anomalous object and furthermore we might not even know which objects will cause anomaly (unlike offline datasets where we know which objects are potentially anomalous). The most used learning component is the autoencoder (2D CAE, 3D CAE, etc.), but other components like GAN, CLSTM, etc., are also used. The choice of learning component depends on the chosen task, input and architecture design.

For auxiliary components, pre-trained detector, optical flow and data transformations are mostly used. These components ensure the input data needed for the methods concern.

Rest of the auxiliary components like memory, attention, clustering, codebook, etc., are used to reinforce the learning component. This signifies that most learning components do require this auxiliary help to better detect anomalies. To conclude, reconstruction, prediction, SST, or their combinations should be used for VAD, with minimum or no auxiliary component. Ideally, video clip should be used as input, since it is the spatiotemporal data and does not require any processing or external supervision of any sort, thus it is an unbiased input.

Perimeter intrusion detection (PID) methods

In this section, we review the existing perimeter intrusion detection methods. It must be noted that only a few methods tackle the complete PID task. Unlike VAD, the PID task has not gained enough scientific attention in recent years, even though private enterprises kept on improving their intrusion detection systems, but they did not share their works publicly. One simple evidence to support this claim is that there is only one public dataset available for this task, unlike the VAD with many datasets. Nevertheless, we review the few existing public methods for this task. PID is closely related to other computer vision tasks like motion detection, object detection, tracking, etc. [START_REF] Valera | Intelligent distributed surveillance systems: a review[END_REF]. Indeed, these tasks are part of the typical PIDS pipeline as shown in Fig. 2.21. In order to examine PID methods, we must review methods that tackled one or several of these tasks with the aim to improve the PIDS pipeline. We describe below various parts of the pipeline, along with some major methods. It should be noted that here we do not describe data acquisition schemes since it has been already explored in Section 1.2.

Pre-Processing

In the case of protecting a perimeter, the cameras are usually CCTV and therefore they are not necessarily of the highest quality. It is because there are several cameras for each site and it is economically not viable for most people to invest too much in expensive cameras. Also, the task is just to detect the presence of an intrusion and not necessarily do profound analysis like face detection of an intruder (which would require high quality cameras). This low-quality camera sensor and adverse environmental conditions such as snow, fog, rain, extreme sunshine, etc., may produce highly noisy video streams. We cannot directly feed this noisy data into the detection algorithm. Therefore, video enhancement is needed to remove noise and improve the visual appearance of the video. We can classify the existing video enhancement methods into two broad categories [START_REF] Sos S Agaian | Transform coefficient histogrambased image enhancement algorithms using contrast entropy[END_REF][START_REF] Rao | A survey of video enhancement techniques[END_REF]): spatial domain enhancement and transform domain enhancement.

Spatial domain video enhancement deals directly with pixels, i.e., it makes a direct manipulation of pixels in video frames. It is conceptually simple and has a low time complexity, which favors real-time implementation but lacks robustness. Some surveys on this method can be found in [START_REF] Bennett | Video enhancement using per-pixel virtual exposures[END_REF][START_REF] Mittal | An efficient video enhancement method using LA*B* analysis[END_REF][START_REF] Rao | An effective night video enhancement algorithm[END_REF]. In most PIDS, some standard spatial enhancement is carried out on raw frames (Buch et Velastin, 2008[START_REF] Bouwmans | Traditional and recent approaches in background modeling for foreground detection: An overview[END_REF]Cermeño et al., 2018), such as image resizing, image normalization, mean centering, color space conversion (RGB to grayscale or vice versa), histogram equalization, etc.

Contrary to spatial domain, the transform domain video enhancement operates on the transform coefficients of the video frame, such as Fourier transform, discrete wavelet transform and discrete cosine transform [START_REF] Walid | Real time video sharpness enhancement by wavelet-based luminance transient improvement[END_REF][START_REF] Rao | An effective night video enhancement algorithm[END_REF]. The video quality is enhanced by manipulating these coefficients. This category of methods has a low computational complexity with the ease of manipulating the frequency composition of the video frame. Some major examples of PIDS using these techniques are (Buch et Velastin, 2008[START_REF] Bouwmans | Traditional and recent approaches in background modeling for foreground detection: An overview[END_REF]. They first use fast Fourier transform (FFT) on video frame patches and then decrease noise by removing very low or high frequencies from the FFT.

Apart from video enhancement, some other pre-processing can be conducted depending on the PIDS. Buch et Velastin ( 2014) first define patches of 16-pixel squares in each video frame. Then, they designate two regions per frame: grass and fence area for segregating the scene into an authorized and unauthorized zone. Their pre-processing is demonstrated in Figure 2.22. Similarly, another common pre-processing is to have a fixed spatial perimeter in each frame of the video (Cermeño et al., 2018;Kim et al., 2018b;[START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF]. This helps the PIDS to focus only on this region of the scene and to ignore activities outside this perimeter.

Detection

This is one of the most important steps of the pipeline since the goal of a PIDS is to detect certain categories of objects that might cause an intrusion. Detection in video can belong to one of the two families: (i) detection of blobs, analyzing the pixel motion, and (ii) detection of objects, analyzing the image appearance with the localization and classification of objects. We describe below various detection strategies and associated PID methods.

Motion Detection

Intrusion is caused by a moving object in a protected perimeter during an unauthorized time. Therefore, motion detection is essential in a PIDS. The principal approaches can be classified into three categories. a) Optical Flow It literally refers to the displacements of intensity patterns. It is an approximation of image motion defined as the projection of velocities of 3D surface points onto the imaging plane of a visual sensor (Beauchemin et [START_REF] Steven | The computation of optical flow[END_REF]. Optical-flowbased methods use partial derivatives with respect to the spatial and temporal coordinates to calculate the motion between video frames. One key benefit of using optical flow is that it is robust to multiple and simultaneous cameras and camera shakes. However, most optical flow methods are computationally complex, very sensitive to noise and tough to implement in real-time settings. Some surveys on optical flow approaches are Beauchemin et [START_REF] Steven | The computation of optical flow[END_REF]; [START_REF] Stiller | Estimating motion in image sequences[END_REF]. Concerning PIDS, Kim et al. (2018b) implemented an optical flow based detection to compare it with their proposed detection method.

b) Temporal Differencing It uses pixel-wise differences among consecutive video frames to extract moving regions. This technique is adaptive to dynamic environments and has a low computational complexity. However, it can fail to extract all of the relevant pixels and can leave holes in regions. Some important studies can be found in [START_REF] Sehairi | Comparative study of motion detection methods for video surveillance systems[END_REF][START_REF] Dimitriou | Dimitrios Tzovaras : An integrated framework for the timely detection of petty crimes[END_REF][START_REF] Tsakanikas | Video surveillance systems-current status and future trends[END_REF]Cermeño et al., 2018). The works of Buch et Velastin (2008[START_REF] Bouwmans | Traditional and recent approaches in background modeling for foreground detection: An overview[END_REF] use simple inter-frame difference followed by some morphological operations for motion detection in their PIDS. c) Background Subtraction This is one of the most popular and key techniques for detecting moving objects in video. Background subtraction detects moving regions by taking the difference between the current frame and the reference frame, often referred to as the 'background model'. The detection ability depends on the adaptiveness of the background model. Some popular background subtraction methods are: running Gaussian average (RGA) [START_REF] Richard Wren | Pfinder: Real-time tracking of the human body[END_REF], Gaussian mixture model (GMM) [START_REF] Stauffer | Adaptive background mixture models for real-time tracking[END_REF], kernel density estimator (KDE) [START_REF] Elgammal | Non-parametric model for background subtraction[END_REF] and visual background extractor (ViBe) [START_REF] Barnich | ViBe: a powerful random technique to estimate the background in video sequences[END_REF]. Background subtraction mainly suffers from illumination changes, dynamic background, shadows, camouflage, video noise, etc. [START_REF] Bouwmans | Recent advanced statistical background modeling for foreground detection-a systematic survey[END_REF]. These effects can cause a background object to appear as a false foreground object and vice-versa. There are dedicated datasets, like the BMC dataset [START_REF] Vacavant | A benchmark dataset for outdoor foreground/background extraction[END_REF], for testing these methods. Most comprehensive surveys on background subtraction-based methods are [START_REF] Hu | A survey on visual surveillance of object motion and behaviors[END_REF][START_REF] Bouwmans | Recent advanced statistical background modeling for foreground detection-a systematic survey[END_REF][START_REF] Xu | Background modeling methods in video analysis: A review and comparative evaluation[END_REF][START_REF] Garcia-Garcia | Background subtraction in real applications: Challenges, current models and future directions[END_REF].

Concerning PIDS, Buch et Velastin (2008[START_REF] Bouwmans | Traditional and recent approaches in background modeling for foreground detection: An overview[END_REF] As already stated, background subtraction also has some drawbacks. In Figure 2.24, we demonstrate this using two examples proposed by [START_REF] Julien | Towards real-time and low-latency video object tracking by linking tracklets of incomplete detections[END_REF]. The first example, depicted in Figure 2.24a, shows false detections due to snowfall. Since the background subtraction models scene background and detects any moving object, it produces false alarms for these moving snow flakes. These are termed as false alarms due to the dynamic background. In Figure 2.24b, the second example shows split-object detections due to camouflage effect as object appearance matches with background (both have similar white texture).

Object Detection

In the last decade, object detection, i.e., object localization and classification, has been a field of intensive research [START_REF] Zou | Object detection in 20 years: A survey[END_REF]. Intrusion detection is closely related to it as intruders belong to certain categories of objects, such as people, car, bike, etc., to be detected in a protected area. Even though intrusion can be caused by vehicles, animals, etc., the state of art mostly focuses on detecting people as intruders. Object detection methods can be classified into traditional and deep-learning-based detectors [START_REF] Zou | Object detection in 20 years: A survey[END_REF]. Some traditional object detectors are the Viola-Jones detector [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF], histograms of oriented gradients (HOG) detector [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF] and deformable part-based model (DPM) [START_REF] Felzenszwalb | A discriminatively trained, multiscale, deformable part model[END_REF]. Deep learning based methods such as Faster R-CNN [START_REF] Shaoqing Ren | Faster R-CNN: towards real-time object detection with region proposal networks[END_REF] and YOLO [START_REF] Redmon | YOLO9000: better, faster, stronger[END_REF], have achieved remarkable performances in object detection domain. Still human detection can be challenging, especially in scenes with an atypical human pose, such as crawling or creeping, occluded scenes [START_REF] Patino | PETS2021: Through-foliage detection and tracking challenge and evaluation[END_REF] and scenes with low luminosity, such as during night. In regard to PIDS, [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] uses a pre-trained YOLO network for intruder object detection. Kim et al. (2018b) first detects moving objects with background modeling and then classifies them into one of the ten classes, containing wild animals and human (intruder). For this object classification, they propose a 2D CNN-based supervised classifier as shown in Figure 2.25.

Tracking

It is essential for a PIDS to keep track of a detected intruder object, otherwise it will be redetected (or not) and may cause unnecessary alarms. Furthermore, perimeter protection solutions may use this information to impose detection constraints. For example, leaving an area can be allowed, but not entering it. We can also think of raising an intrusion alarm only if an object is inside the area for a specified amount of time.

Buch et Velastin (2008) proposed to use the Kalman filter on the texture of objects with a motion mask to build object tracks. Particle filters have also been used to track intruders (Cermeño et al., 2018;Kim et al., 2018b). In (Buch et Velastin, 2014)

Joint Detection and Tracking

As video stream has two components, spatial and temporal, it is usually analyzed in two steps. The first step captures spatial patterns by using detection on each frame (see Section 2.6.2), whereas the second step uses tracking to apprehend temporal coherence (see Section 2.6.3). This approach creates the hypothesis that spatial and temporal dimensions are independent and can be processed sequentially.

Recent approaches jointly model spatial and temporal dimensions using 3D convolutions and improve results in video analysis [START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF][START_REF] Jiao | New generation deep learning for video object detection: A survey[END_REF]. Like anomaly detection, an autoencoder or GAN based on 3D convolutions/deconvolutions can lead to an implicit joint detection and tracking [START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF][START_REF] Sun | Adversarial 3d convolutional autoencoder for abnormal event detection in videos[END_REF].

Post-Processing

A major failure for the PIDS is when it misses to detect intrusions. In fact, ideally a system should always try to detect as much as possible, even at the cost of some false alarms (Cermeño et al., 2018). These false alarms need to be filtered, which is why we need some form of post-processing. Even though this step is crucial in a PIDS, there are few publications on this topic because companies prefer to keep their post-processing confidential. However, despite this, we can list several post-processing techniques.

One of the most common post-processing is to filter objects of interest outside the chosen perimeter (Cermeño et al., 2018;Kim et al., 2018b;[START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF]. Sometimes, blobs are inconsistent across time, such as rain drops, and a filter can check the coherence of the blob trajectory. Detected objects can also be filtered with a minimum threshold on the blob size. For example, Cermeño et al. (2018) filters all the objects with a size less than four pixels. Since foreground objects are bigger than background ones, perspective calibration learns the dimension of object of interest as a function of its position in the scene. This allows to filter objects with a size smaller than the expected size of the object of interest at the same position in the scene (AXIS Communications, 2020; FOXSTREAM Smart Video Analytics, 2021).

Alarm

In a perimeter intrusion detection system, it is essential to send an alarm signal to the security personnel to indicate the potential occurrence of an intrusion. This alarm signal alerts the client to verify if there is an intrusion in the site and then to take appropriate action against it. This alarm may also indicate a false alert in case there is no intrusion in the site. It is also very important to have this alarm as soon as the intrusion enters the surveilled place. The reason for this is that if the intrusion has already been inside the perimeter for long time, it would have already done the damage to the protected site. For all these reasons, in a PIDS, it is essential to send alarms at right time intervals (at the beginning of intrusions). The detected and tracked objects are transformed into alarms by using some sensibility thresholds to set the omissions-false alarms trade-off (Buch et Velastin, 2008;Kim et al., 2018b;[START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF][START_REF] Julien | Perimeter-intrusion event classification for on-line detection using multiple instance learning solving temporal ambiguities[END_REF]. Generally, these thresholds are manually tuned during the actual deployment of the PIDS. Furthermore, some high-level rules can be applied to trigger the alarm. In the work of Buch et Velastin ( 2014), the alarm is triggered if the intruder moves towards the target for a minimum time of 2 seconds. In (Kim et al., 2018b;[START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF], the alarm is generated as long as the intruder is inside the protected perimeter. Figure 2.26 demonstrates this for the work of [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF]. Taking this forward, Cermeño et al.

(2018) adds an extra constraint that the object must be tracked for at least three frames to give an alarm.

System Deployment

The deployment of a perimeter intrusion detection system is realized in three stages: an optional offline model training, then an online initialization of the system, and, finally, online execution.

Model Training

There are some PIDS which require an offline training on part of the dataset for their detection or tracking steps [START_REF] Julien | Perimeter-intrusion event classification for on-line detection using multiple instance learning solving temporal ambiguities[END_REF]Kim et al., 2018b;[START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF]. This training can be supervised, requiring labeled videos (tagging intrusion frames or events); or, it can be unsupervised, under the assumption that there is no annotated data. The supervised training can also be done on an external dataset and then the trained detectors are used for detection or tracking in PIDS. [START_REF] Julien | Perimeter-intrusion event classification for on-line detection using multiple instance learning solving temporal ambiguities[END_REF] proposed a classifier model trained as a multiple instance learning problem by employing image-based features to distinguish intruder objects from moving vegetation and other distractions. The works of (Kim et al., 2018b;[START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] use supervised object detectors for detection step. So far, we do not have any work which use the unsupervised learning for perimeter intrusion detection. The first column presents video frames from two different sites with green zones depicting the areas to protect. In the second column, potential intruder objects detected by YOLO detector are in yellow bounding boxes and since some of them are inside the protected zone, the alarm is raised (shown with red zone).

System Initialization

When a PIDS is installed in a new site, it may need several seconds to set the internal state of the system in alignment with the new scene. For example, to initialize the mean and standard deviation of a GMM (Buch et Velastin, 2014;Cermeño et al., 2018). This is called the system initialization. This online initialization must be well distinguished from the offline model training. Furthermore, a PIDS can have sensibility thresholds, which are manually tuned by the installer during the deployment.

System Execution

The final stage is the online execution of the PIDS. It includes all the steps of the pipeline as illustrated in the right part of Figure 2.21. To provide a reliable protection, most of the systems work between 5 and 25 frames per second (Buch et Velastin, 2014;Cermeño et al., 2018).

Conclusion

We showed that concerning spatio-temporal feature learning, there are broadly four approaches. The first type of approach uses 2D CNN for learning spatial features, followed by an RNN-type network for learning temporal features. The second approach has two network streams to learn different and independent features like appearance and motion, which are finally fused together to obtain the overall space-time features. The third approach uses 3D convolution based networks and jointly learns spatio-temporal features. Finally, the fourth approach is a hybrid approach, which include a combination of above approaches. In this thesis, we want to learn spatio-temporal features jointly in video, with minimum supervision possible. Therefore, the 3D convolution-based approaches are ideal for us, with new variants being also fast and low on computational complexity. We do not want to rely on external supervision for optical flow, etc., thus two-stream or hybrid approaches are not suitable.

To move from these supervised spatio-temporal approaches towards unsupervised approaches, three principal modeling approaches can be used, i.e., reconstructive, predictive and generative. Reconstruction models principally use auto-encoder with the aim to reconstruct the input video stream. Predictive models try to predict the future frames or sequences, given observed sequence. They use LSTM-based models and autoencoder variants. Finally, the generative models principally use generative adversarial networks to model the likelihood of video samples in an unlabeled dataset. We found all three approaches are extensively used in the literature and none of them is more suitable than others. The choice of approach depends on the video task being tackled, its requirements, and the spatio-temporal feature learner (like a 2D CNN, 3DCNN, etc.) being used. Our chosen 3D convolution-based approach can be used as 3D CAE in unsupervised learning. It can be used as a reconstructive model, predictive model and even as the generator of a generative model. In this thesis, we explore the first two approaches and also combine them. The generative model requires extra computation on top of the generator, via the discriminator, therefore we avoided that approach to be computationally light.

Concerning video anomaly detection, we found that most methods use video clip based inputs like VC, VCP and VCO as input, thus signifying the importance of spatio-temporal input data. Majority of methods use the reconstruction or prediction as proxy task and they accept any type of input. The classification and SST tasks are used less and they require the help of external object detector or data transformation for obtaining the input to their model. This external supervision is not only biased to the external dataset but also in real life, we cannot detect all the possible anomalous objects beforehand (unlike offline datasets where we know which objects are potentially anomalous). The most used learning component is the autoencoder but the choice of learning component depends on the chosen task, input and architecture design. To better detect anomalies, the learning components often require reinforcement from auxiliary components like memory, attention, clustering, codebook, etc. To conclude, reconstruction, prediction, SST, or their combinations should be used for VAD, with minimum or no auxiliary component. Ideally, video clip should be used as input, since it is the spatio-temporal data and does not require any processing or external supervision of any sort.

Like VAD, the task of perimeter intrusion detection concerns with detection of rare and unknown event, i.e., intrusion. In real life PIDS, we do not have labeled data for these intrusions and thus we need to use an unsupervised approach. Irrespective of this, none of the methods in state of the art uses unsupervised learning. The simple answer to this is the lack of public dataset, i.e., there is only one dataset called i-LIDS (i-LIDS Team, 2006). This dataset was released in 2006, so most traditional methods used nondeep learning based approaches. Furthermore, the i-LIDS dataset provide annotation in training set for developing supervised PIDS. This is potentially because during its launch time (2006), deep learning was not explored much, even less unsupervised deep learning.

In our PIDS review, we found that majority of works use their private datasets and mostly the visual camera is used for data acquisition and only one system is multi-camera based. Most traditional systems use background modeling for detection, whereas recent deep-learning-based models use 2D CNN or YOLO detector. Concerning tracking, the Kalman filter, particle filter, tracklet-based tracking and SORT algorithm are used. For alarms, most systems have their own method-specific rules. Three systems use supervised training and the rest do not have a deep learning based training step. Since no two systems use the same dataset or same part of the dataset and many use different evaluation strategies, it is impossible to quantitatively compare the existing methods. Overall, we observe that even though there are only human intruders in the i-LIDS dataset, it still has the potential because of various distractions, weather conditions and intruder approach manners. Therefore, still an unsupervised deep learning based approach can be evaluated on this public dataset.

Since in this thesis, we explore for unsupervised deep learning approaches for both VAD and PID tasks. We will use 3D CAE as our initial point, as stated above. In Chapter 4, we propose a 3D CAE for PID with reconstruction as the proxy task. It has an adaptive thresholding mechanism to adapt with changing scene dynamics like weather, light conditions, etc., and it will be discussed later in that chapter. In Chapter 5, we enrich our 3D CAE with different unsupervised and self-supervised tasks. This new model shows promising results for both VAD and PID tasks, without need of external supervision or memory modules.

Chapter 3

Defining tasks and their evaluations

In this chapter, we are concerned with formally defining the two tasks that we tackle in this thesis, i.e., video anomaly definition and perimeter intrusion definition. We mathematically describe these two tasks in the first two sections. Then, we explain various evaluation protocols for these tasks. Since the existing protocols are not fully suitable for the PID task, we propose a new edge-level evaluation protocol for it. Next, we describe various evaluation metrics used for these tasks, with a proper explanation concerning which metrics should be used for each of the tasks. Part of this work was published in the special issue "Unusual Behavior Detection Based on Machine Learning" of the Sensors 2022 journal (Lohani et al., 2022b).

Introduction

In this thesis, we are focused principally on video anomaly detection (VAD) and perimeter intrusion detection (PID) task. Since past few years, both these tasks have attained a major attention in the computer vision research (Chandola et al., 2009;[START_REF] Saligrama | Video anomaly identification[END_REF][START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF][START_REF] Aravamuthan | Physical intrusion detection system using stereo video analytics[END_REF][START_REF] Nayak | A comprehensive review on deep learning-based methods for video anomaly detection[END_REF][START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF]. Even after all this attention, their definitions still lack clarity, since they are not defined formally [START_REF] Julien | Perimeter-intrusion event classification for on-line detection using multiple instance learning solving temporal ambiguities[END_REF]Kim et al., 2018b;[START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF][START_REF] Aravamuthan | Physical intrusion detection system using stereo video analytics[END_REF][START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF]. It is essential to mathematically define these tasks as it has a direct impact on the evaluation. For example, when a PID system detects an intrusion, it sends a short video to the security post, where a human operator validates the alarm as a true intrusion or otherwise. This short video, composed of several frames before and after the suspected intrusion, must contain the intruder so that the operator can decide to act on it. The end of the intrusion event is generally not relevant for this application. In practice, we would like to detect intrusion as soon as it occurs; thus, we must obey time constraints. This requires a suitable evaluation protocol that takes these constraints into account. Thus, in this chapter, we first mathematically define the two tasks. Then we describe the existing evaluation protocols and metrics, with a correct explanation of which protocols / metrics to be used for the two tasks. We also propose a new evaluation protocol for the PID task that conforms correctly with its definition and requirements.

Defining video anomaly detection

Video anomaly detection is the task of localizing anomalies in space and/or time in a video.

In literature, anomalies are described simply as activities that are out of the ordinary (Chandola et al., 2009;[START_REF] Saligrama | Video anomaly identification[END_REF][START_REF] A A Sodemann | A review of anomaly detection in automated surveillance[END_REF]. Anomalies are also known as abnormalities, novelties, and outliers among other similar terms. Examples of studied anomalies range from fighting and falling detection, to unattended baggage at airports, to a person loitering outside a building. According to [START_REF] Saligrama | Video anomaly identification[END_REF], video anomalies can be thought of as the occurrence of unusual appearance or motion attributes or the occurrence of usual appearance or motion attributes in unusual locations or times.

One simple implication of this definition is that video anomalies are context dependent. This signifies that an activity considered anomalous in one context or scene may be normal in another and vice-versa. For example, in one scene, skating is considered a normal activity, while in another, it is anomalous. Therefore, real-world anomalous events are complicated and diverse, and it is difficult to list all the possible anomalous events. It is indeed desirable that the anomaly detection algorithm does not rely on a big prior information about the events, as it can mislead the algorithm to detect/ignore certain types of normal/abnormal events.

Overall, the most fundamental question here is to determine what are the events which can be designated as anomalous. Anomaly definition has always been debated in the literature due to its subjective nature and the complexity of human behaviors [START_REF] Jiang | Detecting contextual anomalies of crowd motion in surveillance video[END_REF][START_REF] Saligrama | Video anomaly identification[END_REF][START_REF] A A Sodemann | A review of anomaly detection in automated surveillance[END_REF][START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF]. In general, an event is considered abnormal if it deviates from observed or learned ordinary events (i.e., the event with low occurrence or statistical representation in the learned model) or the event is not known, or it is outstanding. As aforementioned, another important aspect about the definition of anomalies is the context or environment of the scene. Besides, the difficulty of defining anomalous events increases as the semantics of events grow. In other words, the greater the semantics of an event, more complex situations must be described to determine the anomalous events out of the normal.

To determine whether an event is anomalous, there are two main scenarios: (i) the expected anomalies are known, (ii) the expected anomalies are unknown. In the first scenario, we know some information of anomalies in the data, i.e., we have some sort of labels for anomalies available during training. It is exceptional to have this since anomalies are rarely labeled beforehand. In the second scenario, we just have the data without any anomaly information. It is assumed that this data is only normal, i.e., without anomalies. This is a more plausible setting since anomalies are rare and it is not possible to have all potential normal and abnormal samples available during training. The main challenge here is to understand pertinent features of normal events from a diversity of scenes and contexts, which would lead to distinguish anomalous events from the normal events [START_REF] A A Sodemann | A review of anomaly detection in automated surveillance[END_REF][START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF].

Please note that throughout the manuscript anomaly and abnormality denotes the same thing, i.e., they are synonyms for each other. Similarly, their adjective forms anomalous and abnormal are used interchangeably.

We define below the two cases concerning video anomaly detection.

VAD with anomaly information

It refers to video anomaly detection when normal examples and at-least some abnormal examples are known. We can also say that in this case we have at-least some supervision via anomalous and normal examples during training.

Fully Supervised VAD

Conceptually, the most direct case is fully supervised VAD having well-defined explanation of normality and abnormality, i.e., training data have labels for both normal and abnormal class. This case is particularly common for some VAD sub-tasks like fall detection [START_REF] Vishwakarma | Automatic detection of human fall in video[END_REF][START_REF] Kong | Learning spatiotemporal representations for human fall detection in surveillance video[END_REF], fighting detection [START_REF] Esen | Fight detection in surveillance videos[END_REF] and traffic violation detection [START_REF] Fu | Similarity based vehicle trajectory clustering and anomaly detection[END_REF][START_REF] Zen | Earth mover's prototypes: A convex learning approach for discovering activity patterns in dynamic scenes[END_REF] [START_REF] Huo | Abnormal event detection via multiinstance dictionary learning[END_REF][START_REF] He | An anomaly-introduced learning method for abnormal event detection[END_REF][START_REF] Liu | Exploring background-bias for anomaly detection in surveillance videos[END_REF][START_REF] Landi | Anomaly locality in video surveillance[END_REF].

Even though fully supervised VAD can produce good results for a dataset, its outcome is not generalizable to other datasets. Furthermore, the big assumption of having labeled data is rarely feasible in real life, especially for mid to large length datasets, which would require extensive manual or semi-automated annotations. Since the portion of abnormal samples is extremely small in comparison to the normal samples (as anomaly is a rare event), there is a huge imbalance of classes in VAD, and this makes it even more difficult for a supervised classifier to perform well [START_REF] Kim | Gan-based anomaly detection in imbalance problems[END_REF]. Moreover, the big diversity of anomalies disturbs the proper training procedure and practically makes it infeasible to consider all anomaly types. Thus, supervised VAD has a very limited applicability in real-life applications.

Semi-Supervised VAD

Since anomalies are very diverse and rare, it is implausible and computationally expensive to access the complete range of anomalous and normal events for training. However, using both normal and abnormal samples during training provides a better anomaly understanding to the model. Therefore, some works propose semi-supervised VAD, where they use a very small portion of abnormal samples along with the usual training dataset (with no anomalies) [START_REF] Akcay | Ganomaly: Semisupervised anomaly detection via adversarial training[END_REF][START_REF] Ruff | Deep semi-supervised anomaly detection[END_REF]Liu et al., 2019).

Weakly Supervised VAD

Another, more realistic form of supervision in VAD is weak supervision. It comprises of video-level labels (as opposed to video clip or frame-level labels), i.e., a label indicating whether a video is normal or contains anomaly somewhere, but we do not know where [START_REF] Sultani | Real-world anomaly detection in surveillance videos[END_REF]. UCF-Crime dataset contains these kinds of labels by default [START_REF] Sultani | Real-world anomaly detection in surveillance videos[END_REF]. Researchers also re-arranged the well-known ShanghaiTech (Luo et al., 2017a) dataset to have weakly labeled anomalous videos during training. It is known as re-organized ShanghaiTech [START_REF] Zhong | Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection[END_REF] dataset. Some notable works on weakly supervised video anomaly detection are [START_REF] Sultani | Real-world anomaly detection in surveillance videos[END_REF][START_REF] Zhong | Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection[END_REF][START_REF] Majhi | Dam: Dissimilarity attention module for weakly-supervised video anomaly detection[END_REF][START_REF] Feng | MIST: Multiple instance self-training framework for video anomaly detection[END_REF][START_REF] Tian | Weakly-supervised video anomaly detection with robust temporal feature magnitude learning[END_REF][START_REF] Li | Self-training multi-sequence learning with transformer for weakly supervised video anomaly detection[END_REF].

Formalism: VAD with anomaly information

We assume x ∈ X and y ∈ Y respectively as the input and the output of a VAD system. X and Y represent respectively the input and output space. In the case of video anomaly detection, x can be either a video clip, a video frame or something else like object in frames, depending on the method (refer Section 2.5.1 for more details), and y ∈ {0, 1} is the associated label where 1 indicates anomalous class and 0 otherwise. We assume a training set D = {(x i , y i )} N i=1 ⊂ X × Y composed of N training samples. The goal is to learn a mapping function that can correctly predict the label y given the input x. In VAD with supervision, the anomalies are simply defined by labels given in the labeled dataset.

In general, a discriminative model, represented by function f and composed of parameters θ is supervisely trained as:

θ * = arg min θ E x∼X [ L dis (f θ (x), y) ] , (3.1)
where L dis is a discriminative loss function. This can be easily used for the VAD task with a dedicated loss function.

Like the usual supervised binary classification setup, during testing phase, an input sample x can produce a two-dimensional output (or a single dimensional output) ŷ = f θ (x), where each dimension corresponds to the probability of normal and abnormal classes in [0, 1]. The class with higher probability is predicted.

VAD without anomaly information

In this case, we have training data with no labels. The main assumption is that the training data has none to very negligible amount of anomalies, which is true since anomalies are rare. In other words, the training data principally have normal events, i.e., only one class. Based on this, a popular paradigm for anomaly detection is one-class classification, i.e., to encode the usual pattern with only normal training samples [START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF][START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF]. Then, the distinctive encoded patterns are detected as anomalies. In this technique, anomalies are detected solely based on intrinsic properties of normal data instances. The main idea is: "Something that has not being seen before is considered anomalous". Since in unsupervised VAD, we do not have strict assumptions about the training data, it leads to generalizability, especially for applications in which there is not a precise definition for anomalous events [START_REF] Mohammadi | Image/video deep anomaly detection: A survey[END_REF].

Formalism

The formalism proposed here is inspired by probably approximately correct (PAC) learning framework [START_REF] Leslie | A theory of the learnable[END_REF]. In unsupervised VAD, the training set D does not contain labels, i.e., D = {x i } N i=1 ⊂ X with N training samples. Usually, it is equivalent to assume that all videos of D are normal videos, i.e., their labels y = 0. Hence, video clips follow a distribution N , usually called normality. A reconstructive model [START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF], like an autoencoder, is trained as:

θ * = arg min θ E x∼N [ L rec (f θ (x), x) ] , (3.2)
where f is a reconstruction function composed of parameters θ and L rec is a reconstructive loss function. During testing, a reconstruction error ŷ is calculated for an input sample x as:

ŷ = R ( L rec (f θ (x), x) ) , (3.3)
where R is a common re-scaling function to transform the error into a probability in [0, 1].

Then, when the reconstruction error is low, i.e., it is inferior to a threshold << 1, the input sample is classified as normal with a high confidence, denoted 1 -δ where δ << 1:

p ( x ∼ N | ŷ ≤ ) ≥ 1 -δ . (3.4)
Conversely, when the reconstruction error is high, i.e., it is superior to the threshold 1 -, the input sample is classified as abnormal with a high confidence:

p ( x ∼ A | ŷ ≥ 1 -) ≥ 1 -δ , (3.5)
where A is the distribution followed by abnormal videos. Finally, we can define abnormality as the complement of the distribution of training set: A = X \N . It must be well noted that this representation takes care of complicated or debated cases for VAD using the uncertainty parameter δ << 1. It denotes in Equations (3.4 and 3.5) that we can never be completely sure that if the reconstruction error is small, it is normal and if it is large, it is an anomaly. It other words, we cannot be sure that all representations far from distribution followed by training data should be anomaly. For example, in ShanghaiTech dataset, the normal activities like picking up stones and eating ice-creams are present only in the testing set and not the training set. Therefore, even if the reconstruction error is high for these activities, we cannot be fully certain that they are anomaly due to the uncertainty parameter δ. Like reconstructive models, this definition of anomalies can be easily extended to generative models also.

Defining perimeter intrusion detection

In simple words, the task of perimeter intrusion detection (PID) aims to detect the presence of an unauthorized moving object in a protected site during a certain time. As described before, PID is a specific type of VAD task. Furthermore, intrusions are a particular type of anomalies, classified as point and contextual anomalies (Chandola et al., 2009). But contrary to VAD, we can more concretely define this task as it has certain unique attributes like perimeter, intruder object type, movement constraints and site protection time.

The PID task has been defined in various ways in the state-of-the-art. Vijverberg et al. (2014) defined a perimeter intrusion detection system (PIDS) as a monitoring system that identifies the presence of humans or devices in a pre-defined field of view. In (Kim et al., 2018b), it is defined as a system that detects physical intrusions on a site having a protective barrier to isolate it from outside. [START_REF] Aravamuthan | Physical intrusion detection system using stereo video analytics[END_REF] described it as a system that detects the movements of intruders attempting to breach a security wall or region and alert security. However, all these definitions lack clarity and formalization; for example, the following questions need to be addressed: "what are intruders?", "does moving intruder cause intrusion?" and "is a perimeter necessary?". To answer all these questions concretely, we must mathematically define a PIDS. Before defining a PIDS, we need a definition of intrusion. Since objects cause intrusion, we first define an object in the video.

Object in the Video

We define a video V acquired for n frames during the interval T = [1, n] as:

V = I t ∈ R H×W ×D t∈T , (3.6)
where I t denotes the frame at the time instant t, with height H, width W and number of channels D. To define an object in the video, we must first specify the object definition at frame-level. An object in a frame or image is defined with a spatial specification and a class that distinguishes one family of objects from another (such as humans, animals, or cars). The spatial specification can be either on pixel-level by allocating each pixel to an object or background, or on area-level by encapsulating the object in a bounding box. We choose the bounding box specification as it has been used in the literature extensively [START_REF] Zou | Object detection in 20 years: A survey[END_REF]. It should be noted that the choice of spatial specification (bounding box or otherwise) cannot have an impact on the intrusion definition. Thus, we define an object at frame-level with a class and a bounding box. To define an object in the video, we consider all of the frames where it is present. Therefore, an object o i in the video is defined as:

o i = {b i,t } t∈T , c i ∈ C , (3.7)
where c i is the class of the object from the set of object classes C, and b i,t is its bounding box at time instant t, which is defined as:

b i,t = {g i,t , w i,t , h i,t } , (3.8)
where g i,t , w i,t and h i,t are the center, width and height of the bounding box, respectively. The center is defined by its coordinates as g i,t = (x i,t , y i,t ) ∈ I t . Note that, instead of the bounding box center, it is also possible to choose other points, such as the bounding box bottom, as reference. We illustrate these definitions in Figure 3.1.

Intrusion event

For the protection of a site, some parameters must be defined to qualify objects as nonauthorized (na), i.e., intruders.

• S na ⊆ R H×W : the subset of the frame/image, defining the surface to protect.

• T na ⊆ T : the time interval during which the surface must be protected (e.g., protection only during night).

• C na ⊆ C : the set of non-authorized classes, such as person, car, truck, etc. These classes of objects are considered as possible intruders and can be different from one site to another.

Since C na is a non-finite set (it is impossible to make the exhaustive list of nonauthorized objects, exposing the system to omissions), it is easier to explicitly define a short list of authorized objects C a (such as small animals), which leads to C na = C\C a .

An object causes an intrusion event if it belongs to a non-authorized class and is moving in a protected area during a prohibited time interval. Our hypothesis is that the object should be moving and not stationary to cause an intrusion event. It is to tackle the case where the unauthorized object enters and stays for a long period of time in the site. For example, if a car enters the site, it causes an intrusion event (assuming car belongs to non-authorized class) but later if the car stays stationary in the site for many days, it is not necessarily causing an intrusion event. We are interested in knowing if the intruder has entered the site, thus our hypothesis of moving object. We define the intrusion event caused by an object o i as:

IE(o i ) = { I t s.t. c i ∈ C na
and t ∈ T na and grad g i,t > 0 and g i,t ∈ S na } t∈T , (3.9) where grad g i,t is the gradient of object o i at instant t and it being non-zero signifies that the object is in motion. Thus, the intrusion event caused by object o i is a collection of all the frames I t such that t ∈ T na , object class c i ∈ C na , the gradient is non-zero and the bounding box center lies in the protected area.

Figure 3.2 illustrates the intrusion event caused by an object. The surface to protect S na is depicted with a yellow trapezoid in each frame, and we assume that we want to protect it during the entire video. One object is present in the video, and it is shown with a rectangular bounding box plus a center. The object is in motion from the second frame until the eighth frame. While in motion, the center of object lies in S na from fourth to seventh frame, causing an intrusion event. Thus, this object triggers an intrusion event for four frames. Since a video can have more than one object causing intrusion events, we define the intrusion events of the whole video containing j objects as:

IE(V) = j i=1 IE(o i ) .
(3.10)

Intrusion interval

Figure 3.3 shows three intrusion events caused by three objects in the video. We can observe that intrusion events of objects 1 and 2 overlap in time for two frames. It means that for those two frames, there were two objects causing intrusion events simultaneously.

The intrusion events of this video is a collection of all the intrusion frames, marked by 1 (see Figure 3.3). In the context of video surveillance, we are concerned with whether there is an intrusion event, regardless of whether one object or many objects are causing it. Therefore, we are interested in an interval of a contiguous sequence of intrusion frames.

We term this as an intrusion interval, and the task of intrusion detection is focused on detecting them. Formally, an intrusion interval II ⊆ IE(V) is defined on a closed interval as:

II = {I t ∈ IE(V) with t ∈ [t start , t end ] s.t. I tstart-1 / ∈ IE(V) and I t end +1 / ∈ IE(V)} , (3.11)
where t start and t end denote the first and last frames of an intrusion interval. In other words, an intrusion interval is a contiguous sequence of frames of maximal size derived from IE(V).

PIDS

Given a precise definition of intrusion, we can now define a perimeter intrusion detection system (PIDS). Given a video V and intrusion parameters (S na , T na , C na ), the prediction of a PIDS can be defined as:

P(V, S na , T na , C na ) = {p t ∈ {0, 1}} t∈Tna , (3.12)
where pt is a binary prediction for each frame t of video V for time T na , with 1 denoting a frame predicted as an intrusion, and 0 otherwise. Therefore, a PIDS classifies each frame into an intrusion frame or otherwise. This type of output is useful when we want to evaluate a PIDS at frame-level (refer Section 3.4.1). In a real-life surveillance system, the system sends an alarm signal to surveillance personnel as soon as there is a transition from a normal to intrusion state (i-LIDS Team, 2006). The output of the PIDS O(P) can be derived from P as follows:

O(P) = { pt ∈ P s.t. pt-1 ∈ P with pt-1 = 0 and pt = 1 } t∈Tna .

(3.13) This is a set of intrusion alarms created by the system, marked by the rising edge, i.e., the transition of the system state from non-alarm to alarm. These alarms alert the surveillance personnel about a suspicious activity. For each alarm, a mini clip is sent containing some frames before the alarm and some frames after the alarm. The surveillance personnel visually analyze this mini-clip and decide whether it is an actual intrusion activity or a false alarm. Therefore, we need an evaluation scheme that takes into account this real-life constraint.

Evaluation protocols

Given a video or set of videos and the requested task, the system can detect intrusions or anomalies. To evaluate the performance, we need to compare the predicted output with ground truth annotations. The way this evaluation is carried out is essential as each evaluation scheme quantifies different aspects of the system and depending on the task, one should wisely choose the correct evaluation methodology. The following subsections present different evaluation protocols. It must be noted that only frame-level evaluation is used in both the tasks and other evaluation protocols are only used for the PID task.

Frame-level evaluation

As the name suggests, in this type of evaluation, we are interested in checking whether each frame of the video is correctly classified or not.

VAD task

Irrespective of the VAD types and formalisms defined in Section 3.2, every system outputs whether a frame is normal or abnormal. Since per-frame anomaly ground truth is normally available in VAD datasets (refer Section 1.4.1), the predictions can be easily compared with it. It is simply the binary classification evaluation of each frame of the video [START_REF] Sokolova | Beyond accuracy, Fscore and ROC: a family of discriminant measures for performance evaluation[END_REF]. First, the elements related to the confusion matrix, i.e., true positive (TP), false negative (FN), false positive (FP) and true negative (TN), are computed with anomaly as the objective class. Then, the performance of VAD is evaluated using the metrics, such as the precision, recall, AUROC, AUPR, etc., as defined in Section 3. 5.

In frame level evaluation, the anomalies are evaluated at the lowest level, i.e., frames. This means that here we loose the notion of anomalous events. For example, if we have a good frame level score, it does not necessarily mean that all anomalous events are well detected and maybe a few long anomalous events are contributing a large portion of the overall score while many small anomalous events are missed or incorrectly detected. In short, a good frame level anomaly score just gives an overall idea for a VAD performance and does not guarantee if most anomalous events are correctly detected. Even after these drawbacks, the VAD community still mostly use the frame-level evaluation protocol (Popoola et [START_REF] Oluwatoyin | Video-based abnormal human behavior recognition-a review[END_REF][START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF][START_REF] Nayak | A comprehensive review on deep learning-based methods for video anomaly detection[END_REF][START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF]. In this work, we continue with this evaluation so that we can fairly compare with other VAD works.

PID task

For a video V and given intrusion parameters (S na , T na , C na ), the frame-level ground truth is defined as: (3.14) where p t is the ground truth label for each frame t of the video V at time T na ; value 1 denotes an intrusion class, and 0 otherwise. Given ground truth G and prediction P (see Equation (3.12)), the frame-level intrusion evaluation is computed like frame-level evaluation of VAD task by computing confusion matrix elements and using metrics like precision, recall, F 1 score, etc.

G(V, S na , T na , C na ) = { p t ∈ {0, 1} s.t. p t = 1 if I t ∈ IE(V) } t∈Tna ,
In this type of evaluation, each frame contributes equally to the overall score. Thus, it can provide the same overall score for an algorithm that fails to detect multiple intrusion events altogether, i.e., detecting no frames in those events, versus an algorithm that detects all intrusion events but does not detect some intrusion frames in multiple intrusion events. This is an undesirable evaluation in the case of intrusion detection because we cannot afford to have omissions of intrusion events. In reality, we are more interested in knowing if the system is able to detect the intrusion events correctly as a whole. This demands an event-level evaluation. In other words, we want to detect intrusion intervals (IIs) from the video. More specifically, we are interested in evaluating whether the beginnings of these intrusion intervals are detected correctly. This is because, if an intrusion event is detected too late, then that detection is not very useful (i- LIDS Team, 2006;Buch et Velastin, 2014). The idea is to detect each intrusion interval as soon as it occurs and, thus, we need an evaluation scheme that takes this into account.

i-LIDS evaluation

To correctly evaluate PID systems, the i-LIDS dataset is provided with an evaluation protocol (i-LIDS Team, 2006). It focuses on evaluating intrusion at the event-level rather than frame-level. To be precise, an intrusion is considered correctly detected if there is at least one system alarm within 10 seconds from the start of the intrusion event.

For an II of the video and alarms O(P), the rules of the i-LIDS evaluation protocol are as follows:

1. TP: if there is at least one alarm within 10 seconds from the beginning of the II. If there are multiple alarm candidates, the first one is taken, and the rest are ignored.

2. FN: if there is no alarm within 10 seconds from the beginning of the II.

3. FP: if there is an alarm but not within 10 seconds from the beginning of the II. If there are consecutive FPs within a 5-seconds gap among them, only the first one is considered and the rest are ignored.

Apart from these, one rule is specific to the i-LIDS-dataset: all IIs and alarms that start within 5 minutes from the beginning of the video are ignored. This means that they wanted to give a preparation time to the system. This evaluation scheme is not generic and has several drawbacks, as illustrated in Figure 3.4. It penalizes an alarm as an FP after 10 seconds from the beginning of an II without considering the duration of intrusion. If the II has a long duration (such as an hour) and we have an alarm at the 11 th second, it is not ideal to mark it as an FP. From a practical point of view, the surveillance personnel will receive a mini clip as soon as the alarm is triggered and, if the intrusion is present, then it is not sensible to mark this as an FP. Instead, this alarm should be ignored as it is not detected within 10 seconds. Similarly, each alarm after 10 seconds but within II is considered as an FP, and this strongly penalizes the system precision. Instead, these extra alarms should be counted without assigning them as an FP.

Figure 3.4: Illustration of i-LIDS evaluation protocol, highlighting its drawback on an intrusion example starting at 9 th second. Since no alarm has been raised in the first 10 seconds of the intrusion, a false negative (FN) is counted. Following alarms, at 22 nd and 37 th second, are marked as false positive (FP) because they do not occur within 10 seconds from the beginning of the intrusion.

Edge-level evaluation

To appropriately evaluate a PIDS while considering the real-world aspects, we propose a new evaluation protocol. An intrusion event begins with a transition from a nonintrusion to intrusion state, i.e., we have a rising edge as shown in Figure 3.5. Similarly, an intrusion event stops by a reverse transition, i.e., a falling edge. We are interested in detecting intrusion within a few frames from the rising edge. Since we focus on this rising edge, we call this the edge-level evaluation. In other words, we emphasize detecting the beginning of intrusion intervals. We first define the following terms from an intrusion interval of the video (see Figure 3.5).

The intrusion interval neighborhood IN is an expanded interval defined by m pre frames before and m post frames after the II:

IN(II, m pre , m post ) = [t pre , t post ] s.t. t pre = t start (II) -m pre and t post = t end (II) + m post .
These m pre and m post frames are in the range of one to five (less than 1/5 seconds for a video at 25 FPS) and are added in order to take into account the error of annotation. This error is because it is difficult to mark the exact frame at which the intrusion starts or ends. This tolerance further permits not strictly penalizing the system when an intrusion event is detected a few frames before the actual event or when the system detects a few more intrusion frames after the actual event is finished. These cases arise often when the intrusion object is in the scene but not inside the surface to protect. Therefore, IN is an interval where the actual intrusion activity takes place, and an alarm given by a PIDS in this interval can be counted as either TP or ignored. An alarm given outside IN must be a false alarm and should be counted as an FP. The intrusion beginning neighborhood IBN is an interval comprising m pre frames before and m frames from the beginning of II:

IBN(II, m pre , m) = [t pre , t m ] s.t. t pre = t start (II) -m pre and t m = t start (II) + m .
This interval highlights the importance of the initial frames of an II, where an intruder has just entered the protected area, and it is in this interval where we ideally want the PIDS to raise an alarm. An alarm raised in IBN must be a TP.

For an II and alarms O(P), the possible outcomes at edge-level are defined as:

1. TP: if there is at least one alarm in IBN. For multiple alarms in IBN, only the first one is considered, and the rest are ignored.

2. FN: if there is no alarm in IBN.

3. FP: if an alarm is outside IN. Each alarm outside of IN is counted as an FP.

In this evaluation scheme, alarms lying outside IBN but inside IN are ignored. This means that we neither adversely penalize these alarms as an FP nor count them as a TP. In event-level evaluation, whether i-LIDS or this scheme, we do not define a true negative (TN). A TN is when a normal (non-intrusion) event is detected as such; in other words, how well we are classifying a normal event as normal. However, this is not the aim of intrusion detection; indeed, it is the opposite. Furthermore, the calculation of TN is ambiguous. We cannot generalize what length of the non-intrusion video should be considered as a TN. For example, a non-intrusion video clip of 5 minutes cannot be considered as similar to a non-intrusion video clip of 5 days. These rules are for individual IIs, but how we deal with scenarios where the intrusion neighborhoods are so close that they intersect one another is another matter. If INs of two or more II intersect one another, then we merge them into a single IN. The new IN consists of m pre frames of the first II and m post frames of the last II, and all the frames in between are merged as an II. Algorithm 1 summarizes the protocol to evaluate a video at edge-level.

Evaluation metrics

Since both PID and VAD tasks are finally a binary classification task, i.e., to classify a frame or event as an intrusion / anomaly (positive) or otherwise (negative), we can naturally apply common metrics to obtain a quantitative performance measure. We describe below some widely used metrics in the literature. They are classified into two categories based on the threshold dependency.

Threshold dependent metrics

Most of the binary classification methods irrespective of application to VAD or PID task, produce a continuous output value within a pre-defined range. As an example, reconstruction-based VAD methods produce an output in [0, 1] range, indicating the probability for an input to be anomalous (refer Section (2.5.2.1 and 3.2). To convert this continuous output value into discrete predictions {0, 1}, we need to choose a threshold. This threshold simply creates a boundary and values above and below it are converted to 1 and 0 respectively.

Given ground truth values, with these binary predictions, we can compute various metrics as defined below. They also work for methods where the output is not continuous but discrete values. When any system (PID or VAD) is deployed in a site, we need to set the threshold and therefore, these metrics are essential in a real-life online scenario. Concerning their use, the PID community always use these metrics for evaluation (Buch et Velastin, 2014;[START_REF] Julien | Perimeter-intrusion event classification for on-line detection using multiple instance learning solving temporal ambiguities[END_REF]Kim et al., 2018b), while in video anomaly detection, these are rarely used and instead the offline threshold independent metrics are used.

Precision

The precision is the percentage of correctly predicted output out of the total predictions, as defined in the Equation (3.15). It is particularly useful when we want to measure how false alarms are affecting a VAD or PID system. A high value of precision denotes that we have very low false alarms. Precision = TP TP + FP .

(3.15)

Recall

It is also known as sensitivity or true positive rate (TPR). It is the percentage of correct predictions out of the total predictions, as defined in the Equation (3.16). It is useful when the cost of the false negatives is high, i.e., when we cannot afford to have omissions of intrusions or anomalies. Usually, in any system, the preference is to have the minimum omissions possible. This means that we need to ideally maximize recall.

Recall = TP TP + FN .

(3.16)

False positive rate (FPR)

It is also known as the false alarm rate. As shown in the Equation (3.17), it is the ratio between false alarms or false positives and total ground truth negatives (non-intrusion or non-anomaly in our case). True negatives (TN) have a major impact in FPR.

FPR = FP FP + TN .

(3.17)

Concerning VAD and PID, true negatives refer to the case where normal frames or events (without anomaly or intrusion) are correctly detected. Both these tasks usually have abundance of normal frames since anomaly or intrusions are rare events. Therefore, this metric is not very useful while evaluating systems for these tasks. However, popular VAD datasets like UCSD Pedestrian, CUHK Avenue, etc. have almost balanced testing set (see Section 1.4.1) . Thus, FPR is computed for them and it is used to calculate AUROC score (more details in Section 3.5.2.1).

F β Score

To take into account both omissions (FN) and false alarms (FP), we need a way to combine precision and recall. The F β score combines them with a bias parameter β, which provides more or less importance to recall or precision.

F β = (1 + β 2 ) × Precision × Recall (β 2 × Precision) + Recall . (3.18)
The most common values used for β are 0.5, 1 and 2. With β = 1, we obtain the F 1 score. The F 1 score is the harmonic mean of precision and recall and it is widely used [START_REF] Julien | Towards real-time and low-latency video object tracking by linking tracklets of incomplete detections[END_REF][START_REF] Julien | Perimeter-intrusion event classification for on-line detection using multiple instance learning solving temporal ambiguities[END_REF][START_REF] Sehairi | Comparative study of motion detection methods for video surveillance systems[END_REF]. The choice of β depends on the precision versus recall demand. For the i-LIDS dataset, there are two system roles with different bias values (i-LIDS Team, 2006). The roles are called 'Operational Alert' and 'Event Recording', with β as 0.81 and 0.87, respectively (in (i-LIDS Team, 2006), α is used instead of β, where α = β 2 ). The former role is designed for real-time intrusion detection and, therefore, has a lower β value to give more importance to precision, as false alarms are essential here. The latter role is for non-real-time systems, where videos are recorded and analyzed on an offline basis. It has a higher β value, as we cannot afford omissions in this case. Most PID systems (Buch et Velastin, 2008[START_REF] Bouwmans | Traditional and recent approaches in background modeling for foreground detection: An overview[END_REF]Cermeño et al., 2018) use this F β metric for evaluating on the i-LIDS dataset.

Threshold independent metrics

Often in threshold dependent metrics, it is not clear how the threshold is chosen [START_REF] Saito | The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets[END_REF]. Therefore, systems tend to choose a threshold to maximize the final score, e.g., Buch et Velastin ( 2014) chose a high detection threshold to eliminate false alarms, as the metric used is F β with β = 0.81, which favors precision. Alternatively, a system can be evaluated over a range of all possible thresholds, thus avoiding bias towards the choice of a fixed single threshold. We present below these types of threshold independent metrics. The main requirement of these metrics is that the system output should be continuous, and they are not adapted for discrete output systems. Contrary to threshold dependent metrics, these are not used in online system deployment but in offline system evaluation instead. These metrics are rarely used in PID community but are very prevalent for the video anomaly detection task [START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF][START_REF] Nayak | A comprehensive review on deep learning-based methods for video anomaly detection[END_REF][START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF].

AUROC

It stands for area under the receiver operating characteristics (ROC) curve. The ROC curve shows the trade-off between sensitivity and specificity [START_REF] Fawcett | An introduction to ROC analysis[END_REF]. It is created by plotting the TPR against the FPR, where each point corresponds to a threshold from the list of possible thresholds. This is depicted in Figure 3.6. In ROC curve, a classifier with random performance is shown via a straight diagonal line from (0, 0) to (1,1), and this line is the baseline of ROC, with AUROC value of 0.5. The best possible value of AUROC is 1, which signifies perfect classification. Since this score contains FPR, it is influenced by true negatives (refer Section 3.5.1.3). Therefore, AUROC is not suitable for imbalanced datasets [START_REF] Saito | The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets[END_REF][START_REF] Sokolova | Beyond accuracy, Fscore and ROC: a family of discriminant measures for performance evaluation[END_REF]. Since anomalies or intrusions are rare events, there are often very few anomalies or intrusions compared to the normal events. Therefore, theoretically AUROC should not be used for these tasks. However, recent VAD datasets have balanced dataset for testing phase and therefore AUROC is primarily used there [START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF][START_REF] Nayak | A comprehensive review on deep learning-based methods for video anomaly detection[END_REF]. Given the VAD test set, the ROC curve is obtained by varying the thresholds on the frame level predictions across the whole test set. It is called as "AUROC on all videos" or microaveraged AUROC. However, some VAD works compute a "AUROC per video" and report the average, also called macro-averaged AUC (Georgescu et al., 2021a,b;[START_REF] Ristea | Self-supervised predictive convolutional attentive block for anomaly detection[END_REF]. In this metric, the succession of thresholds to estimate the ROC curve is not common to all test videos. Since thresholds are adapted to each video, ROC curve is in risk to be overfitted, providing overly optimistic performances. Consequently, AUROC should always be measured as "AUROC on all videos", computed on the whole test set with thresholds common to all test videos concatenated together [START_REF] Fawcett | An introduction to ROC analysis[END_REF].

AUPR

It refers to area under the precision-recall (PR) curve. The precision-recall plot shows precision values for corresponding sensitivity (recall) values. Like the ROC curve, it is model-wide evaluation with complete set of possible thresholds, as shown in Figure 3.6. The AUPR values range from 0 to 1, where 0 and 1 are for the worst and perfect classification, respectively. It is particularly useful when the datasets are highly imbalanced, such as in PID or VAD [START_REF] Saito | The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets[END_REF]. This is because it uses precision and recall which are focused on predictions of correct class (intrusion or anomaly), rather than that of incorrect class (normal class), i.e., no dependency on true negatives unlike ROC. Concerning VAD, it has been used in some works [START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF] and it must be used if future datasets are imbalanced (which theoretically they should be).

Conclusion

In this chapter, we defined the two tasks that we tackle in this thesis, i.e., the video anomaly detection and the perimeter intrusion detection. We provided proper formalism to both these tasks. This clear mathematical formulation is essential for properly evaluating works in these tasks. We explained the usual evaluation protocols and since they were not suitable according to the definition and task requirements, we provided a new evaluation protocol. We also studied various evaluation metrics and explained which ones are suitable for which tasks.

Concerning video anomaly detection, we are interested in detecting the occurrence of unusual appearance or motion attributes in unusual locations in the video. There are two principal ways to define an anomaly: either with some labeled anomaly examples during model training, or with only normal (non-anomaly) examples. The anomalous examples can provide weak, semi or full supervision. This first case of VAD with supervision is not practical for real-life use. The two main problems with it are: (1) since anomaly occur so rarely, it is practically impossible to have annotations for all possible future anomalous activities in any natural scene, and ( 2) even if all possible anomalous activities were available for supervision, the task itself would reduce to binary video classification where the anomalous class is "known". This vanquish the spirit of video anomaly detection where the ultimate goal in practice is to detect any deviation from normality. In contrary to this, in the second case, the normality is learned from the data. We have data with only normal scenes in the case. Hence, the main idea is to understand normality from this data and then the distribution outside this normality is defined as the anomaly. We propose a formalism for this using the PAC learning framework, which takes into account the complicated cases where normality and abnormality are conceptually very close to each other. For VAD evaluation, frame-level evaluation is followed, therefore the concept of anomalous events is neglected. Furthermore, the threshold-independent metrics are used instead of online metrics. In short, frame-level AUROC metric is used as an evaluation measure in most VAD systems.

Perimeter intrusion detection is a special sub-task of VAD with several specificities like perimeter, intruder object type, movement constraints, etc. It has never been systematically defined before. We first mathematically define how an object can cause an intrusion event, followed by the definition of intrusion intervals in the video. We proposed that the task of perimeter intrusion detection is basically to detect the beginning of these intrusion intervals. Since these intrusion intervals are nothing but events, so we first explained the existing i-LIDS evaluation protocol. We found several drawbacks in this evaluation scheme. Therefore, we proposed a new evaluation protocol for PIDS, called edge level evaluation. It is completely coherent with the PID definition and basically focuses on evaluating a PIDS based on the rising edge from non-intrusion to intrusion state. In other words, this protocol helps evaluating if an intrusion was detected within the first few seconds of its beginning. Finally, similar to i-LIDS evaluation protocol, the thresholddependent metrics like precision, recall and F 1 score can be used with the proposed edge level evaluation.

Introduction

In this chapter, we introduce an unsupervised deep learning method for perimeter intruder detection. As already stated in previous chapters, intrusions are rare by nature and therefore it is very difficult to have large amounts of annotated examples for training a supervised model. We saw in the review of existing PID methods (refer to Section 2.6) that most methods do not use deep learning. The ones that use deep learning rely on externally trained supervised object detectors and thus their functioning is biased on the external dataset, and furthermore they assume that intruder object class is known beforehand. Therefore, there is a need of methods which learn from the data itself, without requiring need of annotation or external dataset. We propose such an approach in this chapter.

Inspired from the methods of video anomaly detection, this work is based on video clip reconstruction proxy task for PID. The same principle as VAD is followed here: learning normality while training and detecting intrusions as deviations from normality. One major challenge for any PIDS is to adapt to changing weather, light, and environmental conditions. Standard reconstruction based methods are sensitive to these dynamic conditions (Chalapathy et Chawla, 2019;[START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF]. To address this limitation, we propose an adaptive strategy so that the presented unsupervised PIDS works well irrespective of scene dynamics.

In the rest of the chapter, we present our proposed method and provide an extensive comparative analysis of it with other methods.

Unsupervised and Adaptive Perimeter Intrusion Detector

In this section, we present our proposed AE-Adapt (AutoEncoder with Adaptive thresholding) method. It is the first unsupervised deep learning approach for perimeter intrusion detection. It has two main steps: learning spatio-temporal normality using an autoencoder and detecting intrusions using this learned representation via an adaptive mechanism. These two steps are described in following subsections. Before presenting them, we must define the input to our method. We use video clip as the input unit. We mathematically define it as follows. Given a video V with n frames {I 1 , I 2 , . . . , I n }, a video clip V l,s of length l and temporal gap s between frames is defined as:

V l,s = I 1 , I 1+s , . . . , I 1+(l-1)s = {I 1+xs } 0≤x<l , (4.1)
where for simplicity, it is assumed that the clip starts from the 1 st frame.

Learning spatio-temporal normality

We propose a strided 3D convolutional autoencoder (S3DCAE) as autoencoder for our PIDS as shown in Figure 4.1. Given a video, we use a sliding window to extract video clips. Each video clip is fed to the S3DCAE, which reconstructs it. The idea is to train the autoencoder with only normal videos (without intrusions) and minimize the error between input and reconstructed video clips. The intuition is that the trained autoencoder should correctly reconstruct normal video clips and badly reconstruct the video clips with intrusions. In other words, it will learn the normality and reconstruction error between the input and reconstructed clip will help to distinguish between non-intrusion and intrusion.

S3DCAE architecture

The main components of this architecture are 3D convolutions and 3D deconvolutions. They were used since they jointly model spatio-temporal information and are better suited for feature learning from videos than 2D convolutions, 2D convolutions with LSTM, etc. (Section 2.3).

Let us denote the input as X, where X is a video clip of length T and temporal gap of 1, i.e., X = V l=T,s=1 = {I 1 , I 2 , . . . , I T }. The proposed autoencoder takes X and outputs a reconstructed video clip X of same dimensions. The error between these clips is the training loss function (MSE), defined as:

L = 1 T × D × H × W X -X 2 F , (4.2)
where D, H and W denotes channels (depth), height and width of each frame and • F denotes the Frobenius norm. Figure 4.2 shows the architecture of S3DCAE. Each encoder layer consists of a strided 3D convolution while each decoder layer uses a strided 3D deconvolution [START_REF] Matthew D Zeiler | Deconvolutional networks[END_REF], with kernel size of 5 × 3 × 3 (shape : temporal length, height, width). A stride of 2 × 2 × 2 is used in each operation except the last 3D deconvolution where 1 × 1 × 1 stride is used. This signifies that each encoder layer reduces the input volume into half, while each decoder layer (except the last) does the opposite. Encoder is composed of two layers containing (16, 8) filters, while decoder also have two layers with (8, 16) filters respectively. The ReLU activation function is used between each layer, except the final layer, which uses the hyperbolic tangent (tanh) activation. A dropout layer with dropout probability of 0.25 is applied after the first layer. Overall, we have a light architecture with only 15, 889 parameters (1011 MB size). Finally, this model is trained using the Adadelta optimizer (Zeiler, 2012).

Detecting Intrusions

Once we have a trained model, we use it during testing phase with the hypothesis that the intrusion frames will be badly reconstructed, i.e., with a high reconstruction error. Figure 4.3 illustrates how intrusions are detected in our system. A video clip is obtained using a sliding window, the learned S3DCAE takes it as input and reconstructs it. Then, the frame level reconstruction error is computed using the input and reconstructed clip. This reconstruction error is used by an adaptive strategy with moving z-score to produce an intrusion alarm. These steps are explained in detail below. Figure 4.3: Detecting intrusion using AE-Adapt: an input video clip is reconstructed after going through the S3DCAE. The reconstruction error is calculated between these clips and fed to a moving z-score module, the output of which is thresholded to raise an intrusion alarm.

Frame level reconstruction error

From a test video, we obtain video clips using the sliding window like training. A test video clip of length T frames and temporal gap s = 1, i.e., {I 1 , I 2 , . . . , I T } goes through S3DCAE to produce a reconstructed clip { Î1 , Î2 , . . . , ÎT }. The reconstruction error can then be calculated as:

r = 1 T × D × H × W T k=1 Îk -I k 2 F . (4.3)
is close to 0, the system relies on evolution of error historically and the current value of error plays a smaller role. This process with Equation (4.4) and Equation (4.5) is repeated for each new frame of the video.

Experiments and results

In this section, we first describe our experimental setup with data, methods and evaluation protocols. Next, we present various results comparing our approach different existing PID systems.

Implementation details

Here, we present the data used in experiments, the implementation details of our approach and other methods, and finally the evaluation protocols used.

Data

We used the two cameras views of i-LIDS sterile zone dataset as described in Table 1.2.

The frame resolution of i-LIDS dataset is 720x576. The training set contains 10 nonintrusion videos per view. The testing set contains 17 and 16 videos of view 1 and view 2, with 7 and 6 videos containing intrusions respectively (from 36 to 92 minutes in length).

For more details on i-LIDS dataset, please refer to Section 1.4.2.

Methods

To compare our method with others using different evaluation protocols, we need their source code. From the list of PID methods (refer Table 2.2), only one method, i.e., [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF], provides an implementation of their method. Therefore, we select it for our PIDS comparison. Since no classical PIDS (non-deep learning based) provides source code, we use GOFPID [START_REF] Barthélemy | PyGOFPID is a Python package for good old fashioned perimeter intrusion detection systems for video protection[END_REF], which allows to build the traditional PIDS pipeline. With GOFPID, we can have a representative method for the classical PIDS, and thus we include it in our experiments. The method of [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] follows the typical PIDS pipeline (see Figure 2.21), along with some deep-learning-based components. Given a video, frames are extracted and the user is asked to draw a perimeter in the frame for protection. Then, a potential intruder object should be chosen from a list of pre-defined classes, such as person, car, etc. The system detects the intruder object with bounding box using the pre-trained YOLO v2 network [START_REF] Redmon | YOLO9000: better, faster, stronger[END_REF], having a detection threshold of 0.25 for class human. If the detected intruder object is inside the pre-defined perimeter, then it is considered as an intrusion. Finally, the object is tracked using simple online and real-time tracking (SORT) algorithm [START_REF] Bewley | Simple online and realtime tracking[END_REF]. For testing it in i-LIDS, we first drew a protection perimeter following the fences for each view. We then chose person as an intruder class. Then, frames of each video are fed into the system, and we obtained a binary intrusion/non-intrusion prediction for each frame.

GOFPID [START_REF] Barthélemy | PyGOFPID is a Python package for good old fashioned perimeter intrusion detection systems for video protection[END_REF] helps to define the traditional PIDS pipeline. It also provides suitable configuration for the i-LIDS dataset. We use this default i-LIDS configuration as the classical PIDS method, and it can be described as follows. First, the user is asked to draw a perimeter in the first frame of the video. Along with perimeter, the perspective size of the intruder object (here person) at different distances from the camera is also requested. We use the same perimeter like last method and use the perspective settings provided for i-LIDS with GOFPID. Given these settings, the input video frame is first denoised by spatial blurring using a Gaussian filter. Then, the foreground detection is performed using an improved adaptive Gaussian mixture model [START_REF] Zivkovic | Improved adaptive gaussian mixture model for background subtraction[END_REF]. The obtained foreground mask is then denoised by mathematical morphology. The created foreground blob is then checked for two conditions: whether it is inside the perimeter and if it obeys the minimum object size according to the perspective setting. If these conditions are met, then the blob is tracked using blob tracking. If the blob is tracked for a minimum of 3 frames and it displaces for a minimum distance (set beforehand), then an intrusion alarm is raised. Testing this method on i-LIDS dataset was straightforward and a binary intrusion/non-intrusion output is obtained for each frame.

Concerning the proposed AE-Adapt method, we trained and tested our model in each view (view 1 and view 2) of i-LIDS dataset. Test uses the same protection perimeter as defined for the last two methods, signifying that all activities (intrusion or otherwise) outside this perimeter are ignored. Non-intrusion videos from training set of each view are used for training, using one Nvidia RTX 3090 GPU, with a batch size of 32. Each frame is converted to grayscale, pre-processed using histogram equalization and pixels were rescaled to [-1, 1]. For each video, input video clip is constructed using 8 frames with temporal stride of one frame, leading to an input of shape 8 × 720 × 576 × 1. Adaptive thresholding is initialized with reconstruction errors of first 10 frames, moving z-score is used with z th = 4.5 and α = 0.01, chosen from the validation set.

Evaluation Protocols

We used the three evaluation protocols studied in Section 3.4: frame-level (FL), i-LIDS and edge-level (EL) evaluations. For edge-level evaluation, we set the tolerance variables m pre and m post to 3, i.e., less than 1 second of tolerance. For variable m, the following values were chosen: 5, 10 and 50. This signifies that we tested the methods to detect within 1, 2 and 10 seconds from the beginning of intrusion (refer Section 3.4.3). The parameters for 1 and 2 seconds were selected to satisfy the demands of a real-time PIDS, whereas the parameter for 10 second was chosen in order to be comparable with i-LIDS evaluation protocol. For each evaluation protocol, we present the results in terms of precision, recall and F 1 score as defined in Section 3.5.

Comparison of Evaluation Protocols for PIDS

Here, we present overall results of different methods on view 1 and view 2 of i-LIDS dataset, using different evaluation protocols.

Results concerning View 1

Figure 4.5 shows the results of the three methods on view 1 of the i-LIDS test dataset through different evaluation protocols. To better understand results, we must first comprehend how these methods predict intrusions. Figure 4.4 shows predictions using these methods on portions of the video taken from the i-LIDS test dataset.

Observe how each method differs in intrusion detection behavior. It can be observed that method of [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] has the highest number of correct frame predictions, followed by GOFPID and our method has the least frame predictions. Similarly, the same trend follows for omissions, since [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] leaves least number of frames undetected and our method detects only few frames per intrusion. Both these observations explain the frame level recall scores of the three methods as seen in Figure 4.5. All the three methods have few false detections as seen in Figure 4.4, but the ratio of correctly predicted frames to false detections is highest in [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF], followed by GOFPID and finally our proposed method. This explain the frame level precision scores of the three methods, with [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] having the best score. Overall, this leads to the frame level F 1 scores, as shown in the left subfigure of The i-LIDS evaluation protocol functions on event level and concerns with intrusion alarms rather than frame-level binary prediction. To recall, an intrusion alarm is raised as soon as a system changes its state from 0 to 1. For the i-LIDS evaluation protocol, if there is at least one alarm in the first 10 seconds of intrusion (ground truth), it is counted as a true positive (TP). However, an alarm after 10 seconds is counted as a false positive (FP), as already explained in Section 3.4.2. It can be observed that [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] has a high number of intrusion alarms from the beginning to end of each intrusion interval. These alarms at the end are considered as false detections, even though they happen during the intrusion. This explains their poor precision score in the i-LIDS evaluation protocol as shown in Figure 4.5. Since this method rarely misses producing alarms within the first 10 seconds, it has a very low number of false negatives (FNs), and this is depicted with a high recall score. For GOFPID, we can observe in Figure 4.4 that it produces alarms in beginning of almost all intrusion intervals, this leads to its excellent recall score in i-LIDS evaluation protocol (see Figure 4.5). But this method also have alarms at the end of intrusion intervals and even outside them (see Figure 4.4), and these are counted as FP, which is aligned with their poor precision score. Our method has a lower number of alarms, and most of them occur at the beginning of intrusions as shown in Figure 4.4. This leads to a smaller FP due to fewer late predictions, and we have a good precision value as seen in Figure 4.5. We also have few omissions within the first 10 seconds; therefore, we observe a good recall value in terms of i-LIDS evaluation. In the edge-level evaluation protocol (Figure 4.5), if an alarm is not raised within the first m seconds of an intrusion, then it is considered as an omission or false negative. But alarms raised within the intrusion, even though after m seconds from beginning are not counted as FP like i-LIDS evaluation protocol. Since it is a difficult task to raise the alarm in 1 st second of an intrusion in comparison to 2 nd or 10 th second, we observe an increase in recall for all methods from EL (1 s) to EL (10 s). [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] and GOFPID have a lower recall than our method in EL (1 s) and EL (2 s) because they often raise alarms later than 2 seconds. Regarding precision, here, the excessive late alarms of these two methods are not penalized as FPs. Therefore, in contrast to i-LIDS evaluation, we observe a better precision value for [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] and GOFPID in edge-level evaluation. Our method raises fewer false alarms than both of them, thus, has a better precision in edge-level evaluation.

Overall, we observe that predictions of [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] fluctuates frequently between non-intrusion and intrusion prediction, thus producing a high number of alarms. This is because it is focused on human detection and it often loses track of the human from one frame to another, and then re-detects it, causing more intrusion alarms. Furthermore, it has very few predictions outside intrusions. That is why it has a good precision and recall value at frame-level. However, for i-LIDS and edge-level, we observe that their method has a low precision value. This is because, from this high number of alarms, only few are relevant, i.e., the ones at the beginning of intrusions. Other alarms are either treated as false alarms or discarded depending on the protocol. The aim of intrusion detection is to detect the intrusion as soon as possible and not just to classify each frame; therefore, it is important to use the right evaluation protocol that emphasizes these characteristics.

Unlike the method of [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF], GOFPID does not have depend on person detection for PIDS. It first detects blobs. The blobs are then filtered out with the perspective size of intruder objects. The resultant blobs are tracked irrespective of the blob information, i.e., whether it is an intruder blob (human for example), or a non-intruder blob (camera noise or insects). An intrusion alarm is raised as soon as tracked blobs have minimum displacement. When an actual intruder enters the scene in i-LIDS dataset, it is moving and thus the alarm is raised by this system. After some time, either the intrusion stops (as there is a fence) or blob is too small, thus leading to no alarm state. Finally, often the intrusion starts moving again (like for going on other side of fence), which causes another alarm as seen in Figure 4.4. There are a few alarms which are raised not for intrusion, for example, when there is sudden change in light in the scene, or due to climatic conditions like snow. This explains the high precision value and low recall value in frame level. But in reality, the intrusion alarms were raised in beginning of intrusions and both i-LIDS and edge-level protocols captured it with high recall values. The i-LIDS evaluation highly penalized alarms end of intrusion alarms (leading to poor precision score), while edge-level evaluation addressed this issue. Thus, edge-level protocol helped in correctly evaluating this system

For our method, it can be observed that we have a lower number of alarms. It focuses more on the beginning of intrusion events and, due to the z-score, it quickly adapts itself with the scene dynamics. Since only beginning of intrusion are detected, we have a very poor frame-level recall. Our method is sensitive to flickering light or moving light conditions, therefore it has some false alarms. The relation between few correctly predicted frames and some false alarm frames leads to poor frame level precision. Therefore, we have a poorer frame level score. However, in both i-LIDS and at edge-level, we obtain a good score because the proposed method raises the alarm as soon as the intrusion occurs, as it is expected from a PIDS.

Results concerning View 2

The results of view 2 are shown in Figure 4.6. The GOFPID method follows has similar results like in View 1. The recall values are slightly decreased in all protocols because few difficult intrusions were not detected like person rolling far away from camera with same clothes as background. The precision values have slightly increases in event level metrics because there is no camera switch from color to monochrome (since only monochrome videos) and there are less distractions like animals etc. Therefore, the GOFPID method detects lesser number of irrelevant blobs as intrusions here. Thus, it obtains overall better results than View 1. We can observe that in all evaluation protocols, our method has a similar trend of results as in view 1, but with a slight decrease in metric values. This slight decrease in values is due to an increase in FPs and FNs, which is linked to more instances of light fluctuation and intrusions at a far distance. [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] obtain similar results to view 1 in frame-level and i-LIDS-level evaluation. In edge-level evaluation, we observe a perfect precision score of 1. This is because this method did not predict any false detection outside the intrusion interval in view 2. This perfect precision score augments their overall score of in edge-level evaluation. This important detail of no false detections was not captured by the i-LIDS-level protocol, and we observe a poor value of precision for them. Overall from this subsection, we can conclude that the edge-level protocol is the most suited for evaluating any PIDS. It can be seen as an evolved version of i-LIDS evaluation protocol and thus adheres to real-life constraints, like detecting in beginning of intrusion event. Furthermore, using its parameters, we can evaluate at different lengths from the beginning of intrusions, like 1 second, 2 seconds, etc. Concerning compared methods, we found that their performances were very close to one another in edge-level evaluation. The GOFPID method is representative of the classical PIDS. After extracting blob, it filters out objects with irregular perspective size and tracks the resultant blobs for atleast three frames and checks whether the blob moves a minimum distance. These well formed post-processing operations allows it to filter many irrelevant blobs of objects. Since this dataset has only person as intruder without too much distractions, this method performs very well to detect intruders with a high precision. It only fails in complicated cases where the object is too far and too similar with the background. Some false alarms are caused due to fast moving light in the scene or due to snow. The method of Nayak et al.

(2019) uses pre-trained person detector. Since this dataset has only person as intruder, this method works very well. It rarely detects noise due to snow or lighting conditions as intrusions, thus it has a very good precision value in edge-level evaluation. When intruder enters the scene, it detects it but then looses it in tracking and have re-detections. Also, it has problem in detecting when the person in crawling or log-rolling in the far away distance. It detects most easily when the person is standing. Due to this, it often have intrusion alarms late, around the end of intrusion, and this leads to a lower recall value for this method. Finally, our unsupervised and adaptive PID system performed equally well in comparison to last two methods. The false alarms are mainly caused due to changing light conditions and insects on camera. But these are very few in number, thus we have an overall good precision score in edge-level evaluation. Our method detects intrusions and adapts itself, we rarely miss an intrusion event, therefore we have a very good recall value. The missed intrusions include cases where intruders are very far from the camera, having clothes similar to scene background. Our method has still a room for improvement and we will present a new method in the next chapter. Since i-LIDS dataset is not very difficult, having only person as intruder, we also test methods on a private dataset which is detailed in annex Section A.

Effect of adaptive thresholding

In this subsection, we explore the impact of adaptive thresholding in the proposed method. Figure 4.7 shows the impact of various thresholding strategies. We can observe that the reconstruction error varies a lot from one intrusion to another. Therefore, trying to select an optimal threshold on it can lead to unsatisfactory results, e.g., false alarms before and after 1 st intrusion. After min-max rescaling, we obtain a similar conclusion as only the range of values have changed to [0, 1], without any significant difference in threshold choosing strategy. We can see that the moving z-score adjusts itself with the reconstruction error. It detects the beginning of each intrusion and then updates itself.

In Table 4.1, we quantitatively compare the thresholding strategies on proposed S3DCAE. We observe that the direct outcome from the autoencoder leads to a poor performance. As expected, it is difficult to choose a threshold when the error varies due to scene dynamics. Rescaled reconstruction error using min-max (MM) scheme improves the overall result by only 5%. Since i-LIDS test-set videos are very long, the rescaling scheme failed to handle changes in reconstruction error caused by changing scene dynamics. We can clearly observe that adding an adaptive thresholding with moving z-score (MZ) almost doubles the overall performance from F 1 of 0.49 to 0.92. These results strongly support our proposition that an adaptive component is necessary for the deployment of a PIDS in real-life scenes.

Working illustration of AE-Adapt

In this subsection, we demonstrate the working of our proposed method, shown in Figure 4.8. For ease of comprehension, we continue with the example of Figure 4.7.

Since it is continuously snowing (distraction) in the scene, there are small zigzag effects in reconstruction error. We can observe in images that the intruders are at different distances from camera and the magnitude of reconstruction error depends on it. For instance, the intruders far from camera occupy a smaller area in the image and thus have smaller reconstruction error. We also observe a high local RE in the beginning of intrusions (see images in top row). This is because when an intruder enters the scene, there is a high spatio-temporal change, which for autoencoder is abnormal (normal training videos in i-LIDS dataset have extremely low number of examples for where objects move) and thus it is badly reconstructed. Two images in bottom row show that even though the intruder is present in the scene, the reconstruction error is relatively lower. The reason behind is the same, i.e., spatio-temporal change. The spatio-temporal change is comparatively lower in these cases, since the intruders are hardly moving (trying to break in). Finally, we can observe via the green curve that the moving z-score just detects the beginning of these intrusions and then adapts itself with the reconstruction error. This is what we expect ideally from a PIDS.

Conclusion

In this chapter, we introduced a new unsupervised method called AE-Adapt, for detecting intrusions in a video stream. We proposed a strided 3D convolutional autoencoder that learns spatio-temporal normality while training only on normal videos. To detect intrusions during testing, the error between input and reconstructed video clips is used. This error is thresholded to detect intrusions. But this error depends on the changing scene dynamics like weather, temperature, etc. We show that naively thresholding this error, especially on long videos, performs poorly. Therefore, we proposed an adaptive thresholding strategy using moving z-score on this error. We found that this adaptive We also compared our PIDS with two existing methods, using different evaluation protocols. We showed that intrusion detection should be evaluated at a higher level than frame level, because frame level evaluation cannot tell if the beginning of intrusion events is detected or not, which is essential for a PIDS. That is why almost all methods in the state-of-the-art use i-LIDS evaluation. However, the major problem with this evaluation is that it penalizes all alarms after 10 seconds, even if they are well within the intrusion interval. It is indeed too strict in counting FPs, and this hits the overall score negatively, although the system has detected intrusion well within first 10 seconds. We proposed edgelevel evaluation to address this issue, where we can parameterize the time after intrusion beginning for the evaluation. This helps in testing the system for different practical time settings and testing its robustness. More importantly, the alarms after a specified time are not counted as false positives if they are within the intrusion. We found that irrespective of method, edge level evaluation should be used for PIDS evaluation.

Concerning compared methods, we found that all the three methods have equivalent performance on i-LIDS dataset, with edge-level evaluation. The method of [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] depends on a pre-trained person detector and i-LIDS have only people as intruders, it was an easy task for this method. GOFPID uses traditional PIDS pipeline and also manually drawn perspective to filter object blobs of irregular size. It performs well in this dataset, since there are very few non-intruder moving objects. Our proposed method first learned normality from training set and then detected intrusions as deviations from it. Theoretically, our method can detect intrusions caused by other objects since it does not use beforehand information about intruder class. In fact, for edge-level evalaution for 1 and 2 seconds, our method outperformed other methods. This means it is suitable for rapid real time detection as well. Nevertheless, our proposed method still has drawbacks.

That is why, in the next chapter, we try to enhance the spatio-temporal comprehension of normality, to better detect intrusions. Also, since i-LIDS dataset was comparatively easy, we also test these methods on a private dataset, available in annex.

Chapter 5

Leveraging Unsupervised and Self-Supervised Learning

In this chapter, we introduce a new approach for spatio-temporal video representation learning. It leverages unsupervised and self-supervised learning and is applicable to both VAD and PID tasks. In the coming sections, we first explain our generic approach. Next, we reveal how it is applied to VAD and PID tasks. Then, the following two sections show experiments and results for the two tasks. Major part of this work was accepted in International Conference on Computer Vision Theory and Applications (VISAPP), 2023.

Contents

Introduction

In this chapter, we propose a new approach to address the task of video anomaly detection (VAD) and perimeter intrusion detection (PID). As described in Section 2.5, one of the most highly successful approaches to tackle VAD is to use a deep autoencoder (AE) [START_REF] Hasan | Learning temporal regularity in video sequences[END_REF]Luo et al., 2017a;[START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF] and it also has the least assumptions on data. We also showed in Chapter 4 that the same principle of AE for VAD can be also used for the task of PID. Therefore in this chapter, we proceed with the AE based approaches, proposing a method that leverages unsupervised and selfsupervised learning on a single AE. To this end, we devise multiple tasks to enhance the normal spatio-temporal understanding of the AE by training it only on the normal data. Each task has its specific objective: (i) video clip reconstruction (VCR) to learn spatio-temporal characteristics of the normal videos; (ii) future frame prediction (FFP) to learn how normal spatio-temporal patterns propagate along the videos; (iii) playback rate prediction (PRP) to strengthen the playback speed perception of the encoder. Our method is end-to-end trainable and is jointly trained on the three tasks. While testing, the anomaly or intrusion is detected if the combined score of the three tasks is high. Most of the current methods use error measures like mean squared error (MSE) or peak signal to noise ratio (PSNR) for comparing input and reconstructed frames (Astrid et al., 2021a,b;[START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF][START_REF] Park | Learning memory-guided normality for anomaly detection[END_REF][START_REF] Lv | Learning normal dynamics in videos with meta prototype network[END_REF]. But these measures are prone to noise [START_REF] Gudi | Proximally sensitive error for anomaly detection and feature learning[END_REF][START_REF] Sinha | A perceptually based comparison of image similarity metrics[END_REF]) and therefore we introduce a new measure, called the blur pooled error (BPE). It is locally sensitive and keeps only relevant pixels for error calculation. Most works apply a min-max rescaling to anomaly scores per video [START_REF] Liu | Future frame prediction for anomaly detection-a new baseline[END_REF][START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF][START_REF] Park | Learning memory-guided normality for anomaly detection[END_REF]Astrid et al., 2021a,b). It is sensitive to extreme values and to address this issue, we propose a robust rescaling of scores for VAD.

Overall, in this chapter, we propose a new approach for spatio-temporal learning in videos. It is generic and we applied it to VAD and PID tasks. In the following sections, we will present our method in general, then its application to VAD and PID tasks, followed by respective experiments and results.

Proposed approach

In this section, we present our proposed LUSS-AE (Leveraging Unsupervised and Self-Supervised AutoEncoder) method. The main idea is to learn normal spatio-temporal features in order to better discriminate anomalies or intrusions from normal patterns. To this end, we propose to train a 3D convolutional autoencoder (3D CAE) on normal videos using carefully designed tasks in a unsupervised and self-supervised manner. The video clip reconstruction task learns spatio-temporal characteristics of normal videos. The future frame prediction task is designed to learn the propagation of spatio-temporal patterns in the normal videos. Finally, the playback rate prediction task strengthens the speed understanding of the encoder. The autoencoder is jointly trained on all these tasks. During testing, each of these branches provides a score to distinguish anomalous or intrusion frames from the normal ones. Figure 5.1 illustrates overall schema of our approach. A video clip of T + 1 consecutive frames is extracted from the video. The first T frames of this clip goes through the 3D CAE, which reconstructs it. This reconstruction, after passing through an activation function, is compared with original clip to perform the video clip reconstruction task. The same reconstruction (before activation) is fed to the H FFP head, which predicts a future frame. This frame is compared with the (T + 1) th frame of the input clip, thus performing the future frame prediction task. Another video clip of T frames is extracted from the video. It is either accelerated or kept at the original playback rate. Then, this video clip is fed to the encoder of the 3D CAE. This output is fed to the H PRP head, which produces a prediction for playback rate of the input clip. This prediction is compared with the actual playback rate of the video clip, to perform the playback rate prediction task. All these tasks are performed jointly and the weighted sum of errors is backpropagated together in the network.

In the following subsections, we first detail the role of each task and how they are trained together, followed by how these tasks act during the testing phase.

Learning normal spatio-temporal patterns using multiple tasks

In this subsection, we explain how the proposed tasks help in learning normal characteristics during training. We describe each task with its role, followed by details on how all these tasks are trained in a joint manner.

Video Clip Reconstruction (VCR)

Reconstructing a video clip is one of the most popular tasks for unsupervised VAD (Astrid et al., 2021a,b;[START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF]Liu et al., 2021b;[START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF]. We explored its principle and some major VAD methods in Section 2.5.2.1. In the last chapter, we proposed a video clip reconstruction based PID method and we demonstrated its usefulness for the task. Concretely, the VCR task aims to reconstruct an input video clip, mostly using an autoencoder (AE) type network. The AE is trained only on normal video clips with the learning objective of minimizing the mean-squared error (MSE) between the input and reconstructed clips. The main hypothesis is that the abnormal or intrusion clips will be badly reconstructed during testing. Using Eq. ( 4.1), a non-strided video clip of length T + 1 frames can be defined as:

V T +1,1 = {I 1 , I 2 , . . . , I T , I T +1 } .

(5.1)

The first T frames of this clip is used for the VCR task and we denote it as X VCR , i.e., X VCR = {I 1 , I 2 , . . . , I T }. This video clip goes through the autoencoder followed by an activation function to produce a reconstructed clip XVCR as:

XVCR = tanh(Dec(Enc(X VCR ))) , (5.2)
where Enc and Dec stand for encoder and decoder networks, respectively, and tanh is the activation function.

The loss function can then be defined as:

L VCR = 1 T × D × H × W XVCR -X VCR 2 F , (5.3)
where D, H and W denotes channels (depth), height and width of each frame and • F denotes the Frobenius norm.

Future Frame Prediction (FFP)

Predicting a future frame is also a well-spread task for unsupervised VAD [START_REF] Liu | Future frame prediction for anomaly detection-a new baseline[END_REF](Liu et al., , 2021b;;[START_REF] Lv | Learning normal dynamics in videos with meta prototype network[END_REF][START_REF] Park | Learning memory-guided normality for anomaly detection[END_REF] and it was discussed thoroughly in Section 2.5.2.2.

To our knowledge, it has not yet been applied for the PID task. FFP aims to predict an unseen future frame, given an input video clip. This requires comprehension of how normal spatio-temporal patterns propagate along the video clip. Like VCR task, the objective is to minimize the MSE, but between the predicted and actual future frame. Since the AE is trained only on normal videos, it should predict the anomalous or intrusion frames poorly. This task uses the same input of VCR task, i.e., X VCR . After passing through the AE, the video clip X VCR goes through the prediction head H FFP to predict the future frame as: XFFP = H FFP (Dec(Enc(X VCR ))) .

( 5.4) This frame is compared with the actual future frame, i.e., frame T + 1 of V T +1,1 (see Eq. ( 5.1)) denoted as X FFP , where X FFP = I T +1 . The loss function is then defined as:

L FFP = 1 D × H × W XFFP -X FFP 2 F .
(5.5)

Self-supervised Playback Rate Prediction (PRP)

Section 2.5.2.3 exhibited that self-supervised learning is very recent for the VAD task.

Like FFP task, it has never been studied for the PID. We propose here a self-supervised task, i.e., playback rate prediction (PRP), applicable to both VAD and PID tasks. The PRP task does exist in self-supervised representation learning and it is used as a pretext task to learn transferable semantic spatio-temporal features for downstream tasks like action recognition (Liu et al., 2021a). In other words, first PRP task is performed and later the learned model is adapted to downstream tasks. Contrary to them, we adapt the PRP task, and train it via a single AE with two other tasks, all done simultaneously in a joint and end to end manner. The original PRP task generates speed labels for video clip sampled at different rates and aims at predicting them [START_REF] Benaim | SpeedNet: Learning the speediness in videos[END_REF][START_REF] Wang | Self-supervised video representation learning by pace prediction[END_REF][START_REF] Yao | Video playback rate perception for selfsupervised spatio-temporal representation learning[END_REF]. Since we know that the training videos are normal in our case, we adapt this task to generate two speed-rate labels: original playback rate (implying normal behaviour) and accelerated playback rate with 2x to 5x speed (implying abnormal / intrusion behaviour). The motive is to enforce the encoder with motion comprehension of normal videos. During testing, we hypothesize that the encoder would detect anomalies or intrusions caused by irregular and abrupt motion.

Concretely, given a video clip, this task aims to predict its playback rate. The clip with default playback rate of the video is termed as the original playback rate clip and the clip formed by skipping 1 (2x), 2 (3x), 3 (4x) or 4 (5x) frames is designated as an accelerated playback rate clip. The input X PRP is a video clip of length T , chosen between an original playback rate (class c = 1) and an accelerated playback rate (class c = 2) with equal chance (50% probability each): .6) where the accelerated playback rate clip, when c = 2, is a temporally strided video clip with temporal gap s randomly chosen between 2 and 5. This input X PRP goes through the encoder, playback rate prediction head H PRP and softmax activation to produce prediction as: ŷ = softmax(H PRP (Enc(X PRP ))) ∈ R 2 (5.7)

X PRP = V T,1 when c = 1 V T,s∈{2,3,4,5} when c = 2 , ( 5
The loss function for this classification task is the binary cross-entropy (BCE), defined as: 5.8) where y is the one-hot encoding of the ground truth classes for X PRP .

L PRP = - c=1,2 y[c] log (ŷ[c]) , (

Training objective

A single autoencoder is trained with the above-mentioned tasks and the overall training loss is defined as the weighted sum of individual loss functions:

L = λ 1 L VCR + λ 2 L FFP + λ 3 L PRP , (5.9)
where λ 1 , λ 2 and λ 3 are the weights in (0, 1] that regulate the importance of each task.

Testing phase

In this subsection, we describe how the LUSS-AE is used during testing. Given a test video clip, each of the three tasks provides a score and the overall score is a weighted sum of these scores. Depending on the aimed task (VAD or PID), this score is post-processed accordingly, e.g., for VAD, it is normalized. After post-processing, the high values of the score represents anomaly or intrusion. We first define below some image or video error measures and then describe how these measures help to calculate the overall score in inference phase.

Image error measures

To quantitatively assess how well a future frame is predicted or how well a video clip is reconstructed, we need to compare them with the appropriate ground truth using an error measure. For this, the most widely used measure is MSE [START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF][START_REF] Liu | Future frame prediction for anomaly detection-a new baseline[END_REF][START_REF] Lv | Learning normal dynamics in videos with meta prototype network[END_REF][START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF]. Given two images J, Ĵ ∈ R H×W ×D (in our case, original and reconstructed frame), the MSE is calculated as:

MSE(J, Ĵ) = 1 D × H × W Ĵ -J 2 F .
(5.10)

Since last few years, many works use the peak signal to noise ratio (PSNR) measure (Astrid et al., 2021a,b;[START_REF] Park | FastAno: Fast anomaly detection via spatio-temporal patch transformation[END_REF][START_REF] Ye | AnoPCN: Video anomaly detection via deep predictive coding network[END_REF]. But PSNR also depends on MSE as can be seen in its mathematical formulation. Both MSE and PSNR integrate errors on the whole image and therefore are prone to random and incoherent noise [START_REF] Gudi | Proximally sensitive error for anomaly detection and feature learning[END_REF][START_REF] Sinha | A perceptually based comparison of image similarity metrics[END_REF]. To overcome this, a new measure called the proximally sensitive error (PSE) is proposed by [START_REF] Gudi | Proximally sensitive error for anomaly detection and feature learning[END_REF]. It is defined as: 5.11) where * is the convolution operator and G (σ,k) is a 2D Gaussian kernel with size k and standard deviation σ, given by:

PSE(J, Ĵ) = 1 D × H × W ( Ĵ -J) * G (σ,k) 2 F , (
G (σ,k) [i, j] = 1 2πσ 2 e -i 2 +j 2 2σ 2 , (5.12)
where i and j are the pixel coordinates centered in the kernel. Thanks to the Gaussian convolution, PSE smooths incoherent noise and is locally sensitive. However, an anomaly or intrusion generating an important error in some pixels can disappear in the noise of all other pixels of the high-dimensional image.

To address this, we take this idea one step further and introduce the blur pooled error (BPE), defined as:

BPE(J, Ĵ) = 1 D × H × W S b (( Ĵ -J) * B k ) 2 F , (5.13)
where B k is a generic 2D low-pass filter with kernel size k, and S b signifies a subsampling with stride b [START_REF] Zhang | Making convolutional networks shift-invariant again[END_REF]. Using a low-pass filter smooths incoherent noise, like in PSE.

Then, the subsampling keeps only the most pertinent values from it. Thus, this error has an increased sensitivity to anomalies / intrusions than PSE. All these measures can easily be extended to video clip by applying them to its constituent frames.

Test score

Here, we will focus on how different tasks of our approach act during testing phase and provide scores. Concretely, we will see how these scores are calculated and how the final score is obtained. Figure 5.2 shows the schema of our method during testing phase. Given a test video, we first generate non-strided clip of length T + 1. The first T frames of this clip, i.e., I 1:T is the input to our method. This input first goes through the encoder and the resultant latent space is input to two branches, i.e., decoder and H PRP head. The one that goes through H PRP head produces predictions ŷ which are used to compute the A PRP

Application to tasks

So far, we introduced the main idea of LUSS-AE. It is generic and can be theoretically applied to any unsupervised video learning task. In this section, we explain concretely how it is applied to video anomaly detection and perimeter intrusion detection task, and what is the final measure that leads to detection of anomalies and intrusions.

VAD task

We recall that in this manuscript, we tackle the problem of video anomaly detection without anomaly information (refer Section 3.2.2). In other words, we have training data with no labels, and it is considered safe to assume that it contains only normal events. The idea is to understand these normal events during model training to detect anomalies as a deviation to normality during testing. We use our proposed LUSS-AE with video clip (VC) as the input unit. As explained in Section 2.5.1, video clip is an ideal input for VAD since it contains both spatial and temporal information and does not have bias towards external detectors. Our motivation is to propose a generic VAD method without extra supervision or bias. We present below the network architecture we use in LUSS-AE for VAD and the final anomaly score to detect anomalies.

Network architecture

In this subsection, we detail the network architecture of different components of our method. Precisely, the components are autoencoder, the future frame prediction head and the playback rate prediction head.

a) Autoencoder

We do not propose a new autoencoder architecture and instead use a widely popular architecture for VAD. Like this, we can study the viability of our method and understand how exactly it is better or worse in comparison to the existing works. We use the strided 3D convolutional autoencoder proposed by [START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF], and it is also used by other works (Astrid et al., 2021a,b). The reason for choosing this architecture is because there are 3D convolutions and 3D deconvolutions, which help in learning spatiotemporal features jointly from a video stream (refer to Sections 2.3.3 and 4.2.1.1 for more information). In Figure 5.3, we demonstrate this architecture. It takes a video clip of 16 frames as input, where each frame is 256 × 256 in size. The encoder uses a series of 3D convolutions with kernel size of 3 × 3 × 3 (shape : temporal length, height, width). The stride in first convolution is of shape 1 × 2 × 2, signifying it reduces only the spatial dimension by half. Rest of the convolutions diminish all the three dimensions by half using 2 × 2 × 2 stride. The decoder uses 3D deconvolutions or transposed convolutions with the same kernel size of 3 × 3 × 3. The stride is 2 × 2 × 2, except for the last deconvolution which has 1 × 2 × 2 as stride. Each layer is followed by 3D batch normalization and LeakyReLU activation, except for the last layer. Finally, the output is of the same shape as the input.

b) Future frame prediction head Figure 5.4 shows the proposed architecture of H FFP head for VAD task. It takes the output of decoder as its input. This input is passed through a 3D convolution with kernel size of 3 × 3 × 3 and stride 16 × 1 × 1, followed by tanh activation function to produce the future frame. c) Playback rate prediction head The proposed architecture of H PRP head for VAD is shown in Figure 5.5. It takes the encoder output as its input, i.e., an input of size (256,2,16,16). The input is reduced to a smaller dimension using a series of maxpooling and 2D convolution operations. This reduced input is then flattened to a vector of 512 dimensions. Then, three consecutive fully connected (FC) layers reduce the vector dimension to 2, where the first two FC layers are followed by ReLU activation function. Finally, this vector is converted into probabilities using the softmax activation function.

Anomaly score

Most VAD works apply a min-max rescaling to scores per video [START_REF] Liu | Future frame prediction for anomaly detection-a new baseline[END_REF][START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF][START_REF] Park | Learning memory-guided normality for anomaly detection[END_REF]Astrid et al., 2021a,b). This scaling bounds values to interval [0, 1] where the minimum and maximum values are forced to be 0 and 1 respectively. Due to this, it is prone to outliers with extreme values. To address this issue, we propose a robust rescaling per video. For a test video with n frames, the rescaled anomaly score for frame t is defined as: 5.18) where med(•) and iqr 1-99 (•) are respectively the median and the interquantile range (between 1 st and 99 th percentiles) of scores {A i } n i=1 . Finally, like previous methods, the higher scores correspond to anomalies.

Ãt = A t -med ({A i }) iqr 1-99 ({A i }) , (

PID task

For PID task, we follow the same intuition as the last chapter. In other words, we train LUSS-AE on training data with no labels to understand normality, so that intrusion can be detected during testing as they would follow a deviation from normality. Like VAD and previous PID method (see Section 4.2.1.1), we use video clip as input. We present below the network architecture we propose in LUSS-AE for PID and the final intrusion score to detect intrusions.

Network architecture

In this subsection, we detail the network architecture of different components of our method for PID. a) Autoencoder For autoencoder, we use the same 3DCAE architecture that we proposed in last chapter. In this way, we can investigate how well or worse the LUSS-AE performs in comparison to PID method proposed in the last chapter. As shown in Figure 4.1, this autoencoder takes an input video clip of 8 frames, where each frame is 720 × 576 in size. The encoder uses a series of strided 3D convolutions to produce a latent space of (8,2,180,144). The decoder takes this latent space as input and uses a series 3D deconvolutions to produce an output with same size as that of the input video clip. Refer Section 4.2.1.1 for more details. Even though autoencoder architecture for both VAD and PID have strided 3D convolutions and deconvolutions, the architecture of PID takes only 8 frames as input instead of 16, has a much higher frame resolution than VAD, and smaller amount of filters and layers, making it computationally fast. b) Future frame prediction head The proposed architecture H FFP for PID is shown in Figure 5.6. It takes the output of decoder as its input. A 3D convolution with kernel size of 3×3×3 and stride 8×1×1 is first applied to this input, and then a tanh activation function is used to produce the future frame. c) Playback rate prediction head The proposed architecture of H PRP head for PID is shown in Figure 5.7. The encoder output of size (8, 2, 180, 144) is its input. This input is reduced to a smaller dimension using a series of max-pooling and 2D convolution operations. It is then flattened to a vector of 828 dimensions. Then, three consecutive fully connected (FC) layers reduce the vector dimension to 2, where the first two FC layers are followed by ReLU activation function. Finally, this vector is converted to probabilities using a softmax activation function.

Adaptive Intrusion score

To finally detect intrusion, we need a score that describes it. The overall testing score A of Equation ( 5.17) can be used in three ways as an intrusion score: (i) directly using A as intrusion score; (ii) using normalized A (like in VAD task) as intrusion score; (iii) using adaptive mechanism of moving z-score (MZ) on A (refer to Section 4.2.2.2). We concluded in the last chapter that adaptive thresholding is essential for a PID system, therefore we choose the third way for our experiments with LUSS-AE. We follow exactly the same method of adaptive thresholding as described in last chapter. We just replace r values with A values in Equations (4.4 and 4.5), respectively. Since we use LUSS-AE with adaptive thresholding, we name this method as LUSS-AE-Adapt, in correspondence to the nomenclature of the last chapter.

VAD: Experiments and results

We perform experiments on three publicly available benchmark datasets: UCSD Ped2 [START_REF] Li | Anomaly detection and localization in crowded scenes[END_REF], CUHK Avenue [START_REF] Lu | Abnormal event detection at 150 fps in Matlab[END_REF], and ShanghaiTech (Luo et al., 2017a). Each dataset has a standard training / test division, where the training set consists of only normal videos while testing set has videos with one or more anomalous events (refer to Section 1.4.1). We evaluate with the frame-level area under the receiver operating characteristic (AUROC) metric. Precisely, we use "AUROC on all videos" (micro-averaged AUROC) computed on the whole test set with thresholds common to all test videos. Refer to Section 3.5.2.1 for more details.

Implementation details

For both training and testing in each dataset, the frames are resized to 256 × 256 and pixels are rescaled to the range [-1, 1]. The autoencoder takes a video clip of 16 frames in grayscale as input, i.e., T =16, D=1, H=256 and W =256 respectively. While training, the input clip for PRP task is chosen between original playback rate and accelerated playback rate with equal chance (50% probability for each). For each batch of accelerated playback rate, the value of s is chosen randomly from (2,3,4,5) with equal chance for the four values. The balance weights in the training objective function are set to λ 1 =0.6, λ 2 =0.4 and λ 3 =1, respectively. The whole model is trained end-to-end using the Adam optimizer (Kingma et Ba, 2015) with a learning rate of 10 -4 and a batch size of 16. While testing, a single video clip is used for the three tasks (see Figure 5.2). We use different measures like MSE, PSE and BPE for the anomaly score. For PSE and BPE, we use σ=1, b=2 while keeping the same kernel size of k=5. For blur kernel, we use a Gaussian filter. After grid search, we set the optimal weights for anomaly score (α 1 , α 2 , α 3 ) as (0.1, 0.8, 1), (0.1, 1, 0.1) and (0.2, 0.2, 0.9) for Ped2, Avenue and ShanghaiTech dataset, respectively.

Quantitative results

In this subsection, we present a quantitative comparison of our method with existing state of the art methods. Table 5.1 shows this quantitative results. The listed methods belong to the category of VAD without training labels, with and without external object detectors. As explained in Section 2.5, methods with external supervision via pre-trained object detectors should not be directly compared with fully unsupervised VAD methods and most work respect this notion (Astrid et al., 2021a;[START_REF] Park | FastAno: Fast anomaly detection via spatio-temporal patch transformation[END_REF]. Therefore, even though we list object-centric methods for comparison purpose, our discussion below will solely focus on fully unsupervised methods.

In the fully unsupervised section of the table, the methods marked with * denote nonreproducible results, i.e., the originally claimed results were found to be incorrect. These works have been re-implemented by [START_REF] Menon | Re learning memory guided normality for anomaly detection[END_REF] We can observe that our method outperforms all the other methods across all the datasets. The performance gain in Ped2 and Avenue datasets is less significant as in the Shanghai dataset. In fact, it has been suggested to not use the Ped2 dataset as the performance in it is almost saturated [START_REF] Acsintoae | Ubnormal: New benchmark for supervised open-set video anomaly detection[END_REF]. The Shanghai dataset is considered one of the most difficult dataset and our high performance gain of 5.24% in comparison to the best method in state of the art, demonstrates the viability of our method. Furthermore, the fact that our method works on all the datasets, irrespective of the type of anomalies, shows its generalizing ability. Even though, we use the same architecture for autoencoder like many other methods (Astrid et al., 2021a,b;[START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF], still our proposed method outperforms them, without using any sort of external memory or supervision. Even though our method focus on end to end detection without external supervision, still we obtain competitive performance, equivalent to object-centric methods. All these points demonstrate the strength and effectiveness of our LUSS-AE method.

Qualitative results

In this part, we discuss the strengths and weaknesses of our method via visual examples. 8 demonstrates an example of our method tested on a video with two anomalous events, both containing movement of bikes. Here, the people move with relatively normal speed while bikes move with fast speed. Also, the number of people are relatively less in this example and bike does occupy a big area, which means its displacement causes a big spatio-temporal change. We can observe that as soon as the bike enters the scene, we have a high jump in A PRP and it remains high until the bike exits the scene. It jumps up again in next scene and have highest value when two bikes move in the scene. Regarding A VCR and A FFP , the scores remain high when bikes are in the scene. Overall, our method detects both anomalous events in this example. person turning in wrong direction and person jumping, brawling/chasing action, and stone picking. We can observe that the problem is difficult in this example as not only the anomalies are spatio-temporal but they also depend on context. In the first anomalous event, A VCR and A FFP have higher values than A PRP in the beginning. The anomaly here consists of person turning in wrong direction which is well captured by the VCR and FFP task. Later, when the person jumped, A PRP suddenly increases, indicating its sensitivity to abrupt motion. During the second anomalous event, we observe that A PRP has higher values than A VCR and A FFP , thus contributing primarily to detect the anomaly. The A PRP starts to increase just before the start of this event because the person in red starts to suddenly approach the other person. We then observe a first peak of A PRP as one person pushes other to the ground. We later observe a big second peak of A PRP and this relates to fast movement chasing. However, the third event is very rare (picking up stones) and does not contain large spatio-temporal movement in the scene and thus our method fails to detect it. In fact, the score in later frames is slightly higher than the third event because there are three people in close proximity, trying to change directions, which is considered anomalous for Shanghai dataset. Overall, the VCR and FFP tasks work better in anomalies without abrupt motions and PRP task addresses the anomalies with sudden motions. There is still a room to improve the spatio-temporal comprehension of AE for VAD, possibly with a data augmentation technique as the datasets lack more examples of targetted scenarios.

Ablation studies

In this subsection, we study the importance of different parts of our proposed method.

Are all tasks useful ?

In this ablation study, we analyze the impact of different tasks on the autoencoder for detecting anomalies. Table 5.2 shows combinations of different tasks and their respective AUROC performances on Avenue and Shanghai datasets. Concretely, for each of these combinations, we train and test AE using the chosen tasks, and use the introduced BPE for computing anomaly score during inference. We observe that when AE is trained only with the VCR task, the VAD performance is the least, i.e., 82.72% and 73.11%. We consider this as the baseline since this a standard task in VAD [START_REF] Ba | An overview of deep learning based methods for unsupervised and semi-supervised anomaly detection in videos[END_REF] and we combine it with other tasks to assess their impact. Using VCR and FFP tasks together, boost the performance with a gain of 5.23% and 3.24% over the baseline, indicating the importance of learning propagation of normal spatio-temporal patterns via the FFP task. Similarly, when PRP task is trained together with the VCR task, we observe an increase of 4.71% and 6.09% in comparison to baseline, validating that indeed playback speed perception enriches the comprehension of normality for anomaly detection. Finally, when all the tasked are used together, we observe a substantial yield in performance with 6.32% and 8.10% over the baseline, demonstrating the effectiveness of our proposed approach to train AE by leveraging unsupervised and self-supervised tasks.

Is the autoencoder enriched by FFP and PRP?

In this work, we employ a 3D CAE used in many previous works (Astrid et al., 2021a,b;[START_REF] Gong | Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection[END_REF]. All these works used it with the VCR task. In this study, we first reproduced their work by training and testing 3D CAE with the VCR task. Then, we trained it with the proposed tasks, i.e., VCR, FFP and PRP, and tested using only the VCR task. This way, we can assess the impact of these tasks on AE's comprehension of normality during training to detect anomalies during testing. Table 5.3 exhibit this ablation study. It can be observed by training the same autoencoder with proposed tasks, a gain of 3.76% and 2.26% is obtained on Avenue and Shanghai datasets, respectively. Since the three tasks are used only during training and only VCR is used while testing, this gain in performance reveals that these tasks enriched the normality comprehension of the 3D CAE.

Are error measures equivalent?

The goal of this ablation study is to see the effect of different error measures, i.e., MSE, PSE and BPE, on the VAD task. Since these measures apply to VCR and FFP tasks, PRP task will not be considered here. Figure 5.10 shows an visual example using the FFP task to better understand these measures. We can observe that AE did not correctly predict this frame, where dropping bag is an anomaly. The error frame for MSE highlights this anomalous region of the image but it also captures the background noise of the frame. The PSE error frame has a more visible region of anomaly, and it smooths some background noise. Finally, the BPE error frame has a principal focus on anomalous region and has theleast amount of background noise. Furthermore, the BPE frame is smaller than other maps as we remove irrelevant pixels via subsampling. Table 5.4 shows the quantitative impact of these error measures. To be precise, we train our method with the three tasks and test it with the respective tasks and measures shown in the table. We can observe that the PSE improves the performance in both tasks with 1.01% and 1.23% respectively, signifying the importance of proximity error and noise reduction. BPE provides a significant boost in results with 1.53% and 1.62% performance improvement over MSE in the two tasks. This validates the importance of our proposed BPE for the VAD task.

Ground truth

PID: Experiments and results

We perform experiments on the i-LIDS sterile zone dataset with the two cameras views as described in Table 1.2. Using only non-intrusion videos, model was trained on view 1 of i-LIDS training set and tested on view 1 of i-LIDS test set (same procedure for view 2). Test set uses the same protection perimeter as defined in the last chapter. Since, in the last chapter, we concluded the importance of edge-level (EL) protocol for PID evaluation, we use only this protocol in this chapter. Finally, we do not demonstrate the ablation studies like in VAD (refer Section 5.4.4) because we had similar observations and thus explanations would be repetitive. In the next subsections, we first describe the implementation details of LUSS-AE-Adapt for PID, followed by overall results and how it differs from PIDS proposed in the last chapter. Finally, in the last subsection we provide a summary of how our method performs in a real-world PID case.

Implementation details

Each frame is converted to grayscale, pre-processed using histogram equalization and pixels were rescaled to [-1, 1]. The autoencoder we use takes a video clip of 8 frames as input, i.e., T =8, D=1, H=576 and W =720 respectively. Like VAD, the input clip for PRP task is chosen between original playback rate and accelerated playback rate with equal chance. For accelerated playback rate, s is chosen randomly from (2,3,4,5) with equal chance for the four values. Concerning training objective function, the balance weights are experimentally set to λ 1 =0.6, λ 2 =0.4 and λ 3 =1 respectively. The whole model is trained end-to-end using the Adam optimizer (Kingma et Ba, 2015) with a learning rate of 10 -4 and a batch size of 24.

While testing, we used the BPE measure for the overall test score. For blur kernel, we use a Gaussian filter with σ=1, b=2 and kernel size k of 5. After grid search, we set the optimal weights for final score (α 1 , α 2 , α 3 ) as (1, 1, 0.1). Finally, the adaptive thresholding is done on moving z-score of A. We use values of first 10 frames for initialization, moving z-score is used with z th = 4.5 and α = 0.01, like work in previous chapter. Following the last chapter, for edge-level evaluation, we set the tolerance variables m pre and m post to 3, i.e., less than 1 second of tolerance. For variable m, the following values were chosen: 5, 10 and 50. This signifies that we tested the methods to detect within 1, 2 and 10 s from the beginning of intrusion.

Results

Tables 5.5 and 5.6 show the comparative results for PID in View 1 and View 2 of i-LIDS datasets respectively. The first three methods mentioned here are well defined in Section 4.3.1.2 and were used in evaluation in the last chapter. Their results shown in these tables are taken from the last chapter. We can observe that the proposed LUSS-AE-Adapt outperforms AE-Adapt in both views. In View 1, we obtain the state-of-the-art results with a good margin in terms of F 1 scores in every edge-level metric. Concerning View 2, we still outperform other methods in 1 and 2 seconds of edge-level evaluation and we are just slightly behind [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] in 10 seconds. This means that we detect better than other methods in online scenario, i.e., 1 or 2 seconds. It can also be observed that LUSS-AE-Adapt have constantly better precision and recall scores than AE-Adapt, regardless of edge-level evalaution parameters or camera view. This signifies that indeed using our proposed LUSS-AE-Adapt for PID increases the spatio-temporal comprehension of normality, in order to better detect intrusions.

Method

Importance of adaptiveness in proposed PID methods

Even though the same 3DCAE is used in both AE-Adapt and LUSS-AE-Adapt, we would like to explore where exactly these two methods differ in terms of PID performance. Table 5.7 shows the comparative performances of these two methods, with and without moving z-score, on View 1 of i-LIDS dataset. Please note that without moving z-score, AE-Adapt or LUSS-AE-Adapt looses the adaptiveness (Adapt) and boils down to just autoencoders.

Method

Without We can observe that LUSS-AE-Adapt is 7% better than AE-Adapt when no moving z-score is used. This means that using carefully chosen tasks increases the intrusion detection ability of the autoencoder. When moving z-score is used, both the methods get massive gains in performances. Moreover, the gap in their performances lowers down drastically. This indicates that adaptive thresholding is essential for a PIDS when long time-length videos are present. Even though autoencoder can detect the intrusions locally (for example, if small videos are taken like in VAD datasets), thresholding it's reconstruction error globally for long-length videos leads to poor performance. This is because even though reconstruction error is high locally for intrusions in different scene dynamics like day and night, but the gap between these errors is very large and thus one cannot find a fixed optimal threshold to detect all the intrusions. Thus, the adaptive component is a must and it increases the ability of detecting intrusion in both PID systems.

Testing proposed PIDS in a more challenging real-world scenario

Until now, all experiments concerning intrusion detection were done in i-LIDS dataset.

As already explained in previous chapters, even though this dataset pose some interesting challenges, it still lacks in many ways, like it only has people as intruders, the scenes are simple and most of the time if something is moving, it is an intrusion (thus lack of distractions). To address this, we performed experiments on a private industrial dataset. This dataset has sheeps roaming in a perimeter continuously and the aim is to detect human intruder which can walk, run, crawl, etc. Moreover, the intruder is present at various distances from the camera and at a very far distance, it appears so small that it is very difficult to recognize. We tested our best proposed method, i.e., LUSS-AE-Adapt, an industrial PIDS (an evoloved form of GOFPID) and method of [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] on this. We found that our method outperforms the two methods with a huge margin of 26% and 31% respectively. This proves the viability of our method which learns from the data itself in an end-to-end and unsupervised manner. The performance in this dataset validates that our approach is very suitable for real-world perimeter intrusion detection problem. Please refer to Annex A for detailed dataset description, methods analysis and results.

Conclusion

In this chapter, we tackled the problem of detecting video anomalies and intrusions without annotations. To address this, we proposed a novel scheme that leverages unsupervised and self-supervised learning on a single autoencoder. Our method is end-to-end trained on the normal data and jointly learns to discriminate anomalies / intrusions from normality using three chosen tasks: (i) unsupervised video clip reconstruction; (ii) unsupervised future frame prediction; (iii) self-supervised playback rate prediction. The video clip reconstruction task is meant to learn spatio-temporal characteristics of normal videos. The future frame prediction task is designed to learn the propagation of spatio-temporal patterns in the normal videos. Finally, the playback rate prediction task is for strengthening the speed understanding of the encoder. It was the first time when the PRP was adapted for VAD or PID tasks. To focus on anomalous / intrusional regions in the video, we also proposed a new error measure, called the blur pooled error (BPE) and a robust rescaling of anomaly scores.

When the proposed LUSS-AE is applied to VAD task, we outperformed the state-ofthe-art methods in three major datasets. The high performance gain in comparison to previous methods on the Shanghai dataset proved the significance of our method. This is because the Shanghai dataset is considered one of most difficult VAD dataset with thirteen different scenes and various anomaly types. Our experiments demonstrate that the chosen tasks enriched the spatio-temporal comprehension of the autoencoder to better understand the normality for detecting anomalies. Our ablation study demonstrated the importance of each task and we found that the PRP task plays a major role in LUSS-AE. Furthermore, a significant boost in performance with respect to MSE, showed the significance of the BPE as it removes the background noise by keeping only the pertinent pixels.

Concerning application of LUSS-AE to PID, we found that we outperform all other methods in the i-LIDS dataset. We found through an ablation study that indeed LUSS-AE leads to a significant improvement in autoencoder performance for PID, but this effect is less visible when we use the adaptive thresholding using moving z-score. This indicates that adaptive thresholding is essential for PIDS when there are long-length videos. We further tested our proposed approach on a real-world industrial dataset having sheeps as distractors and we found that our approach outperforms existing approaches with a huge margin. All this validated the viability of our approach for detecting intrusions.

Overall, we can conclude that our proposed LUSS-AE is generic and can be successfully applied to both VAD and PID tasks and leads to superior performances in both tasks. However, there is still a room for improvement. Concerning VAD, there are still complicated cases (like anomaly at far distance or anomaly with low spatio-temporal movement), where our method fails. We need to further improve our approach by proposing new proxy tasks, or by using modules like attention. Training network using BPE also needs to be investigated. Concerning the PID task, we have similar shortcomings. Therefore, we need to further improve our approach for obtaining an even better VAD / PID system. real-life constraint must be considered by the evaluation protocol. The i-LIDS evaluation protocol adhered to this constraint and counted a detection as valid if it is within the first 10 seconds of actual intrusion. The main problem with this evaluation protocol is that it strongly penalizes the system. It treats detections as false alarms if they occur after 10 seconds from intrusion beginning, even if the alarms are within the intrusion. And while doing so, it does not consider the duration of intrusion. For example, if an intruder enters a site and stay there for 5 minutes and an alarm is triggered at 12 th second after the beginning of intrusion. Then, the i-LIDS evaluation protocol will count it as a false alarm, even though the alarm is triggered during the intrusion which would stay in site for almost 5 minutes. We addressed drawbacks of this protocol and introduced an edge level evaluation protocol. Our evaluation protocol adheres with real-life constraints of detecting the beginning of intrusions. Moreover, it has tolerance limits of few frames before and after the intrusion, which permits not to adversely penalize a system if an intrusion is detected a little bit in advance (like a system might detect a person as soon as it steps on the scene, while annotation is when the whole person is in the scene) or when the alarm takes some time to stop after the intrusion.

Defining video anomaly detection is not trivial since anomalies depend on context and they differ from one dataset to another. In this thesis, we defined VAD on the basis of available annotations. If annotations of both normal and abnormal events are present during training, then the VAD problem boils down to binary classification, i.e., understanding the two classes (normal and anomalies) with training and then detecting anomalies during testing. However, realistically, it is very difficult to have annotations during training since anomaly is a rare event and it is practically impossible to have annotations for all possible future anomalous activities. Therefore, the second case is more realistic, where it is considered that training data primarily contains normal (non-anomalous) videos. We mathematically describe this case using the probably approximately correct (PAC) learning framework and consequently anomalies are defined as the complement of normality distribution present in the training data. This definition takes into account the anomaly context, since anomalies are defined according to the normal distribution provided with the dataset. Furthermore, our definition takes care of debated cases for VAD with the uncertainty parameter of PAC framework. For example, in ShanghaiTech dataset, normal activities like eating ice-creams are present only in the testing set and not the training set. Therefore, even though there is high probability for them to be abnormal, we cannot be fully certain that they are anomaly due to the uncertainty parameter. For evaluating VAD methods, we used the frame level binary class evaluation. This is done to be comparable with the state-of-the-art methods, all of which use this evaluation protocol.

Even though our PAC formulation addressed some complicated cases in VAD, there is still a need for a new dataset with more precise anomaly annotation (during testing).

The training data should be representative of normalities and any deviation from it should define some degree of anomaliness. Concerning evaluation, anomalies are also events and like PID, we want to finally know if these events are detected or not. Frame-level evaluation does not provide much information for anomalous event detection. For example, with frame level evaluation, we cannot differentiate between a method that detected some frames of all anomalous events, with the one that detected only few events having large number of frames. Therefore, an event level evaluation should also be used in VAD. For this, datasets like Shanghai and Avenue should be manually annotated at event level. Concerning event level evaluation, we focused on detecting the beginning of event, but our proposed protocol can be extended for detecting the complete event if required. In some cases, researchers might want to detect if the alarm is raised during the whole event.

In that case, our protocol cannot work since it is dedicated to detect rising edges and not the frames of an event. Therefore, a suitable evaluation protocol should be developed in that case. Similarly, we did not penalize re-detections in our evaluation as long as it is during the event. But one can penalize them to some extent, depending on the system output requirements.

Unsupervised spatio-temporal video understanding

In this thesis, we explored spatio-temporal learning in videos for unsupervised tasks. We first proposed a strided 3D convolutional autoencoder (S3DCAE) for this, in chapter 4. To learn spatio-temporal features in a joint manner from videos, we used only 3D convolutions and 3D deconvolutions. The encoder layers generally use pooling operation for compressing information, while decoder layers use the unpooling operation. We instead proposed to use strided convolutions and deconvolutions, therefore our architecture was light in memory and fast in execution. The proposed autoencoder learns normality during training as it is trained only on normal videos and detects intrusions as they are badly reconstructed (having high reconstruction error) during testing. Therefore, it is capable to understand spatio-temporal normality and detect intrusions (or even anomalies) in an unsupervised way.

The second method that we proposed, called LUSS-AE, leverages unsupervised and self-supervised learning using a single autoencoder for spatio-temporal feature learning. It was proposed in Chapter 5. Like the previous approach, the aim is to learn normality during training, for detecting anomalies/intrusions during testing. The main idea is to enrich the spatio-temporal comprehension of an autoencoder using different tasks. We proposed three tasks: video clip reconstruction (VCR), future frame prediction (FFP) and playback rate perception (PRP). VCR is meant to learn spatio-temporal characteristics, FFP is designed to learn the propagation of spatio-temporal patterns, and PRP is intended to strengthen the speed understanding of the encoder. We demonstrated that these tasks enhanced the spatio-temporal comprehension of the autoencoder. With successful application on both VAD and PID tasks, we showed its effectiveness and generic behavior.

Notice that one issue with LUSS-AE is that the PRP task has only two classes: original and accelerated. We group different speeds like 2x, 3x and 4x into a single class (accelerated). Semantically, these different speeds have different spatio-temporal information and therefore they should be in different classes. We can further add cases like slow speed (0.5x) and even negative speed, implying reverse playback. This will make LUSS-AE even more speed aware. Another idea could be to use speed regression instead of speed classification. In real life, often the speed cannot be discretely categorized as 2x or 3x, etc., but can be represented by a continuous value. In speed classification, each speed like 2x or 3x, were treated independently but there is a semantic relation among speeds, e.g., speed of 2x and 3x are more closer to each other than speed of 5x. The speed regression will permit the network to understand the semantics among different speeds and it should enrich its spatio-temporal comprehension by making it even more speed aware. Finally, more tasks can be added to our AE like backward / forward video clip prediction, masking input and predicting it, input flip or rotation prediction, etc. Attention modules can further enhance the spatio-temporal comprehension, but they do require extra computational / memory capacity. There is also a new family of methods that use a transformer network [START_REF] Bertasius | Is space-time attention all you need for video understanding[END_REF][START_REF] Neimark | Asselmann : Video transformer network[END_REF]. These methods are now used extensively in spatio-temporal video learning, especially in self-supervised learning, as a pre-training step. However, they need an very large amount of GPU / CPU computational capacity and long time to train. For instance, video transformer network [START_REF] Neimark | Asselmann : Video transformer network[END_REF] has 114 million parameters with 4,218 GFLOPs, while our proposed LUSS-AE for VAD has 6.12 million parameters with 16.30 GFLOPs only. Even though smaller and quicker variants of this new family of methods are coming, we are still far to have a model that runs for real-time tasks like video anomaly detection and perimeter intrusion detection.

Adaptiveness for online outdoor event detection

Since we focused on detecting intrusion or anomalies in the outdoor environment, the obvious challenge is to face changing scene dynamics like weather, temperature, lighting conditions, etc. Any detection system must adapt itself to these changes, otherwise it will underperform. This challenge is particularly dominant for long length videos, like we have in the i-LIDS dataset. To address this challenge, we proposed an adaptive mechanism using a moving z-score on the reconstruction error of the autoencoder. It detected intrusions as soon as they occur and then adapted itself with the scene dynamics. It is generic and brought massive gains in performance to both the network architectures we proposed in this thesis. The LUSS-AE with adaptive thresholding (LUSS-AE-Adapt) was also tested in a private dataset. This dataset is very challenging since majority of time, there are sheeps moving in the scene, which is considered a normal behavior and intrusions are caused by human intruders entering the scene in various ways like crawling, rolling, etc. It was filmed with thermal camera, which makes it even more difficult to distinguish intrusions with sheeps. Our adaptive LUSS-AE method outperformed an object detector based method and a private industrial method by a large margin. This validated the importance of adaptiveness.

Notice that when a VAD or PID system is deployed in real-life, it does need to face these changing scene dynamics. To develop these adaptive approaches, we need appropriate datasets. Concerning PID, the i-LIDS dataset does conform to this need, but it lacks different types of intrusions like cars, trucks, skateboard, etc.. Furthermore, the two views of this dataset are very similar to each other, they are like mirror images of each other. Also, the amount of distractions is very low in this dataset and thus any person detector system can work well. Therefore, we need more public PID datasets which address the above mentioned weaknesses. Even though we also tested our adaptive approach on a challenging dataset, unfortunately it cannot be made public. In regard to VAD, it is unfortunate that we do not have long length test videos in any current VAD dataset. This would have allowed the development of approaches that adapts themselves with time. Thus, for development of more adaptive approaches, we would require more realistic VAD and PID datasets. Furthermore, different types of camera sensors like thermal, depth, etc. and multiple viewpoints could further help both VAD and PID approaches.

Towards more generic and deployable models

Our proposed approaches learn in offline mode from training data to detect events during testing. When these approaches are installed in a site, they need to update themselves with the newly observed data, i.e., learning new normalities on the go. Naively updating model weights with newly observed data can lead to catastrophic forgetting [START_REF] Kirkpatrick | Overcoming catastrophic forgetting in neural networks[END_REF]. Another solution is to gather data and re-train the models but this would quickly become infeasible due to computational and storage needs. Thus, we need to preserve the initial knowledge of models without reusing the initial data, and also acclimate the model to newly observed data. To accomplish this, continual learning can be used. Recently, it has been successfully used for VAD (Doshi et Yilmaz, 2020b[START_REF] Barthélemy | PyGOFPID is a Python package for good old fashioned perimeter intrusion detection systems for video protection[END_REF]. The approaches we proposed are trained on each dataset separately. But how can we use a model trained on one dataset to work on another? Ideally, we would like to just use few frames of the new dataset to adapt the current model. This is precisely what few shot learning does [START_REF] Brenden M Lake | Human-level concept learning through probabilistic program induction[END_REF][START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF]. Recently, some works have used it in VAD too [START_REF] Lu | Few-shot sceneadaptive anomaly detection[END_REF][START_REF] Lv | Learning normal dynamics in videos with meta prototype network[END_REF]. This property is extremely desirable since then, we can use a model trained on an offline dataset to a number of sites and adapt to each one of them. As already mentioned, we can improve our model with more suitable tasks. However, the new trend in computer vision is to use transformer like model due to their immense generalization capacity. In future, smaller and faster transformer like models can be used in video surveillance.

So far, we have evaluated different methods on the i-LIDS dataset. As already mentioned before, this dataset is interesting but it still has some major drawbacks, which makes it an easy task for any simple person detection. The intrusion in real-life are not so easy to detect and furthermore they can be very well non-human entity like wild animals or cars. To address this, we evaluate methods on a private dataset that depicts more challenging conditions like very dynamic background, different distances from camera, change in light and scene dynamics and hence it is more inline with recent challenges in video surveillance.

Sheeps dataset

This dataset contains videos of a site surveilled by a thermal camera that principally contains sheep. The videos are recorded for many days and therefore present varying light conditions in the scene. Some illustrative frames are shown in The intrusions in this dataset are very rare and caused by people. In other words, when some person enters the scene, it needs to be detected. The human intruders enter in various ways like running, walking, crawling, etc. Furthermore, they are situated at different distances from the camera like near, middle and far. Overall, the task is hard since the PIDS method need to ignore sheeps but at the same time detect human intruders who often act like sheep. Some illustrative examples of intrusions in this dataset are shown in The videos in this dataset are captured at 5 FPS with frame resolution of 384x288. Since intrusions are very rare in this dataset, the training set does not contain intrusions. The training set contains 50 videos (approx. 20,000 frames) with sheeps at different configurations and having different lighting conditions. In test set, each video contain atleast one intrusion. The intrusion can be at different distance from the camera and depicts human crawling, running, etc., accompanied with varied lighting conditions and sheeps. We have in total 36 videos (11,189 frames) of intrusions (7,077 frames). Since each video has different level of difficulty for detecting intrusion, the dataset also comes with video level label from 1 to 5, where 1 is the easiest and 5 is the most difficult case.

Methods and experimental setup

We compare three methods on this dataset: LUSS-AE-Adapt, [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] and an industrial PIDS. We chose LUSS-AE-Adapt because it has enhanced spatio-temporal comprehension and performs better than AE-Adapt. Instead of using GOFPID, we use a private industrial PIDS. This is because their functioning is similar (follows similar traditional pipeline) but industrial PIDS performs better than GOFPID and thus is closer to the state of the art practiced by industry. For LUSS-AE-Adapt, we use the same architecture as proposed in Chapter 5. Just the input frame size is updated to 384x288. Other settings remain the same as mentioned in Section 5.5.1. For [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] also, the same setting like before (see Section 4.3.1) as used with human as the intrusion class. The industrial PIDS was also used with appropriate settings. For evaluation, we use the proposed edge-level protocol (EL (10 s)), as we concluded in Chapter 4 that it should be used for evaluating PID methods. For better comprehension, we also show results for some levels of difficulty.

Results

Table 5.8 presents results of the three methods on the whole Sheeps dataset. We can observe that our proposed method has outperformed the other two methods with a large margin. The F 1 score is almost double in comparison to [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] and has a huge margin 0.26 in comparison to industrial PIDS. We can also observe that our method also has the best precision and recall values. This is important because [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] also has a good recall value but it is at the cost of high number of false alarms, depicted by the poor precision value.

Methods

Precision Recall F 1 To better understand the results, we also provide results at different difficulty levels. For sake of simplicity, we provide the results of only three difficulties, i.e., 1, 2 and 5. The other difficulty have similar results, meaning when difficulty increases performance often decreases. The difficulty 1 and 2 correspond to easier cases, while the difficulty of level 5 corresponds to the most difficult case (shown in rightmost image of Figure A.2).

Method

Difficulty Table 5.9 shows the methods comparison based on different difficulty. We observe that the method of [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF] performs similar for the three distances. In fact, the person object detector seems to fail for distinguishing between person and sheep. That is why, there is a recall of 1 and very low precision. In other words, their system is always raising intrusion alarms (thus perfect recall), irrespective of intruder or sheep, and thus have a very high number of false alarms, depicted with very low precision value. The industrial PIDS performs better than [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF]. For difficulty 1 and 2, their precision values are better than the last method, meaning they have fewer false alarms. For difficulty 5, their method performs poorly and is not able to different between intruders and sheeps. For our proposed LUSS-AE-Adapt method, we observe that it performs better than the two methods irrespective of the difficulty. We obtain an excellent F 1 score of 0.92 for difficulty 1, which is very interesting for this difficult PID dataset. We also observe that the F 1 value decreases with increasing difficulty. The main component of this decrease in scores is due to the recall values. This means that with increase in difficulty, we detect lesser intrusions but we do not raise a lot of false alarms.

Conclusion

In this annex, we tested our proposed PIDS with different existing methods on a private dataset. This dataset is very interesting because it contains constant movement of sheeps (distractors) in a scene, where human intruder tries to enter in various ways. The change in lighting conditions, arbitrary movement of sheeps, different distances of intruder from the camera, and various ways in which the intruder enters the scene, makes this dataset very challenging. The method of [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF], which has excellent performance in i-LIDS dataset, performs poorly here. This is precisely because their object detector failed to detect human intruder among sheeps. To be precise, their detector mostly detected some sheep as human, thus causing massive false alarms. Therefore, here we observed the risk of using pre-trained detectors for PIDS. Concerning the private industrial PIDS, it performed a bit better than [START_REF] Nayak | Video-based real-time intrusion detection system using deep-learning for smart city applications[END_REF]. However, it still fails to detect intrusions at difficult cases and often confuse sheeps with intruders. We found that our proposed LUSS-AE-Adapt method outperformed the two other methods by a good margin. Since our method learned normality, it was able to better differentiate between sheeps and humans. Our adaptive method has an excellent result for easier cases and even with increase in difficulty, our method does not trigger a large number of false alarms. One of the most important aspect is that our method is completely without supervision and learns to detect intrusions just from the unlabeled data. Overall, we clearly see the viablity of our method in real-world perimeter intrusion detection task, provided we have a dataset to train the method beforehand.

Figure 1 :

 1 Figure 1: Venn diagram to illustrate the taxonomy of tasks in anomaly detection.

Contents 1 . 1

 11 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2 Data Acquisition in surveillance systems . . . . . . . . . . . . 6 1.2.1 Visual Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.2 Infrared Camera . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.2.3 Thermal Camera . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2.4 Other Types of Acquisitions . . . . . . . . . . . . . . . . . . . . 8 1.3 Open Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.1 Challenges in capturing the scene . . . . . . . . . . . . . . . . . 9 1.3.2 Challenges related to the complexity of the scene . . . . . . . . 10 1.3.3 Challenges related to the objects of interest . . . . . . . . . . . 11 1.4 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.4.1 VAD datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.4.2 PID datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Figure 1 . 1 :

 11 Figure 1.1: Issues with the infrared camera: low visibility in rain (left) and insect on camera (right).

Figure 1 . 2 :

 12 Figure 1.2: Data acquisition on a site by color (left) and thermal camera (right). Each camera captures two scenes at the same instant, one during day (top row) and the other during night (bottom row).

Figure 1 . 3 :

 13 Figure 1.3: Effect of perspective: three people (with blue bounding boxes) in the same image with different apparent size. Image source: ShanghaiTech dataset (Luo et al., 2017a).

Figure 1 . 4 :

 14 Figure 1.4: Camouflaged image with three persons to detect. Source: (Robinault, 2021).

  (a) Subway entrance and subway exit dataset: the scenes are in indoor environment rather than outdoors. No clear training sets, for instance, Cheng et al. (2015) use the first 15 minute videos for training, while Lu et al. (2013) use more than half of the videos for training; (b) UMN: some scenes are in indoor environment, no clear training and testing split

Figure 1 . 5 :

 15 Figure 1.5: Example frames from the UCSD Ped2 dataset. First row contains frames without any anomaly, with increasing crowd volume (left to right order). Second row shows anomalous frames with anomalies highlighted in red bounding boxes.

Figure 1

 1 Figure 1.6: Example frames from the CUHK Avenue dataset. First row contains frames without any anomaly but with different scene dynamics. Second row shows the following anomalies in red bounding boxes (left to right order): bicycle, throwing bag and running person, loitering and person walking in wrong direction.

Figure 1 .

 1 Figure 1.7: Some frames from the ShanghaiTech dataset, each from different scene. First row contains frames without any anomaly but with different scene dynamics. Second row depicts the following anomalies in red bounding boxes (left to right order): chasing, small truck, brawling and two people jumping.

Figure 1

 1 Figure 1.8: Example frames drawn from i-LIDS sterile zone dataset with various intrusion (intruders in red boxes) and non-intrusion frames. The color and black and white frames belong to view 1 and view 2 respectively. The time of the day in four columns are dawn, dusk, night, and day. The distractions (in blue boxes) in non-intrusion row (left to right order) are a rabbit, birds with rain, fox and an insect on camera respectively. In intrusion row (left to right order), we have two intruders, intruder with ladder during snow, log rolling intruder and crawling intruder respectively.

Figure 2 . 1 :

 21 Figure 2.1: Pictorial representation of a feed-forward neural network. Source:[START_REF] Nielsen | [END_REF]

Figure 2 . 2 :

 22 Figure 2.2: Common activation functions.

Figure 2 .

 2 Figure 2.3: SGD optimization with and without momentum. On the left, without momentum the optimization process bounces between the ravine's slopes whereas, on the right, with momentum, optimization is smoother. Source:[START_REF] Orr | [END_REF].

Figure 2 .

 2 4 shows an example of convolution operation applied on an 7 × 7 image using a kernel of size 3 × 3 and stride 1 × 1. The orange region in the image shows the receptive field of the kernel and green unit in feature map shows the corresponding convolved output.

Figure 2 . 4 :

 24 Figure 2.4: Convolution of an image I with kernel K using stride 1 × 1.

Figure 2 .

 2 5 demonstrates these two operations. The feature map and filter size are 4 × 4 and 2 × 2 respectively. The filter is first placed on the top left of feature map (shown with yellow color) to perform the pooling operation (max or average), resulting in value (yellow color) shown in right side of the figure. Then, the filter displaces with stride two to reach the top right (shown in blue color) and performs the pooling operation again. This pooling process continues until the full feature map is covered. Finally, we can observe the overall results of max-pooling and average pooling layers in the figure.

Figure 2 . 5 :

 25 Figure 2.5: Examples of max pooling and average pooling operations with filter size and stride as two.

Figure 2 .

 2 Figure 2.6: Pictorial representation of a simple RNN and an LSTM. Source: (MathWorks, 2022).

Figure 2 .

 2 Figure 2.7: Classic autoencoder architecture. Source: (Wikipedia contributors, 2022)

Figure 2 .

 2 Figure2.9: Illustrating Generative adversarial network (GAN). Source:[START_REF] Hayes | Logan: Evaluating privacy leakage of generative models using generative adversarial networks[END_REF]

Figure 2 .

 2 Figure 2.10: Illustrative example of 2D CNN for spatial feature extraction and an LSTM to model this spatial information over time, applied to action recognition task.

Figure 2 .

 2 Figure 2.11: Illustration of Two-stream model for video action recognition proposed by Simonyan et Zisserman (2014).

Figure 2 .

 2 Figure 2.12: Comparison of 2D convolution (left image) and 3D convolution (right image) operations. Figure reproduced from[START_REF] Tran | Learning spatiotemporal features with 3d convolutional networks[END_REF].

Figure 2 .

 2 Figure 2.13: Illustration of 3D CNN for video action recognition.

Figure 2 .

 2 Figure 2.14: The architecture of the convolutional autoencoder proposed in Hasan et al. (2016) for video anomaly detection.

Figure 2 .

 2 Figure 2.14 presents an example of convolutional autoencoder proposed by[START_REF] Hasan | Learning temporal regularity in video sequences[END_REF], using reconstruction task for video anomaly detection. They use a video clip of 10 frames as input and reconstructs it. They use 2D convolutions and pooling layers for encoding and 2D deconvolutions and unpooling layers, for decoding. The mean squared error (MSE) between input clip and its reconstruction is used as the reconstruction error (anomaly measure). By using only 2D convolutions, the temporal features of the video clip cannot be taken into account. To address this, Chong et Tay (2017) proposed to add convolutional long-short term memory (LSTM) layers in both encoder and decoder. Taking it one step further,[START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF] proposed a 3D convolutional autoencoder with 3D convolutions and 3D deconvolutions for VAD. Unlike 2d convolutions plus LSTM, the 3D convolutional autoencoder jointly captures the spatio-temporal features from normal videos. Recently, many works used some AE variant for VAD, using FF, VC or VCO as input[START_REF] Ye | AnoPCN: Video anomaly detection via deep predictive coding network[END_REF][START_REF] Ouyang | Video anomaly detection by estimating likelihood of representations[END_REF] Astrid et al., 2021b;[START_REF] Park | FastAno: Fast anomaly detection via spatio-temporal patch transformation[END_REF].

Figure 2 .

 2 Figure 2.15: The future frame prediction architecture proposed by Liu et al. (2018) for video anomaly detection.

Figure 2 .

 2 Figure 2.15 presents an example of future frame prediction architecture proposed by[START_REF] Liu | Future frame prediction for anomaly detection-a new baseline[END_REF]. They propose a GAN, where the encoder is a U-Net style network[START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] that takes training video clips of length t as input and predicts the future frame t + 1. The discriminator network checks if the future frame is real (ground truth) or predicted, by minimizing the intensity and gradient loss. Furthermore, they add motion (temporal) constraint by using FlowNet[START_REF] Dosovitskiy | Flownet: Learning optical flow with convolutional networks[END_REF] to estimate pairs of optical flow maps between the frame at t and real or predicted frame at t + 1. The anomaly score is composed of L1 score between flow maps and intensity plus gradient scores from frame prediction. Some other prominent VAD works that use GANs for prediction task are[START_REF] Lee | Stan: Spatio-temporal adversarial networks for abnormal event detection[END_REF][START_REF] Dong | Dual discriminator generative adversarial network for video anomaly detection[END_REF], while works using AE for prediction tasks are[START_REF] Park | Learning memory-guided normality for anomaly detection[END_REF][START_REF] Lv | Learning normal dynamics in videos with meta prototype network[END_REF][START_REF] Viet-Tuan Le | Attention-based residual autoencoder for video anomaly detection[END_REF][START_REF] Park | FastAno: Fast anomaly detection via spatio-temporal patch transformation[END_REF]. Since prediction task depends on the given input to learn the features in order to predict the unknown entity, it can fail when the input contains only stationary anomalies. For example, for the future frame prediction task, when the input clip contains an stationary car (anomaly), the model is likely to predict this car in future frame too. Thus, the prediction error would be too small to differentiate it from the normality.

  , video pace or speed prediction[START_REF] Benaim | SpeedNet: Learning the speediness in videos[END_REF][START_REF] Wang | Self-supervised video representation learning by pace prediction[END_REF], relative speed perception (Chen et al., 2021), video cloze procedure[START_REF] Luo | Video cloze procedure for self-supervised spatio-temporal learning[END_REF], etc.

Figure 2 .

 2 Figure 2.16: Comparison of VAD tasks: reconstruction (left), frame prediction (middle) and self-supervised: video event completion (right)[START_REF] Yu | Cloze test helps: Effective video anomaly detection via learning to complete video events[END_REF]. DNN refers to a deep neural network and video event is the VCO input.

  or do multi-class classification by adding dummy anomalies (Ionescu et al., 2019; Georgescu et al., 2021b).

Figure 2 .

 2 Figure 2.17: One-class classification VAD proposed by Sabokrou et al. (2018). R and D are generator and discriminator modules of the adversarially learned GAN. X, X, Z and X refers to input, input with noise, latent space and reconstruction respectively, where target class 1 represents normal (non-anomaly) class.

Figure 2 .

 2 Figure 2.17 presents an example of one-class classification approach for VAD. The input of this approach are patches from video frames, i.e., FP rather than the whole frames. It adopts a GAN where the discriminator (D) is tasked with distinguishing original image patches from reconstructions of noisy patches obtained from a denoising auto-encoder network (R) which plays the role of generator. Since R is trained only on image patches from training data, it deteriorates reconstruction of outliers and thus enables D to distinguish an anomalous image patch from its reconstruction easily.Concerning VAD with multi-class classification, Ionescu et al. (2019) proposed one of the first approaches. Their input is FO, which is obtained by using a pre-trained single-shot detector (SSD)[START_REF] Sun | Lattice long short-term memory for human action recognition[END_REF] on each frame of the video. They train convolutional auto-encoders on appearance and gradient features of these objects to learn latent representations and then perform k-means clustering followed by training of k one-class SVMs to make binary one-versus-rest classifications. Each cluster represents a different type of normality. At test time, they simply use the inverse of the maximum of k classification scores as an anomaly score, meaning if a sample does not belong to any cluster, it is considered as anomalous. The classification task fails when the anomaly resemble too much with normality in the latent space, thus making it difficult to separate anomalies from normality.

Figure 2 . 18 :

 218 Figure 2.18: Network architecture proposed by Zhao et al. (2017) for reconstructing present frames and predicting future frames.

Figure 2 .

 2 Figure 2.18 presents the work of[START_REF] Zhao | Spatiotemporal autoencoder for video anomaly detection[END_REF] where they combine reconstruction and prediction tasks. The propose an encoder consisting of 3D convolutions[START_REF] Baccouche | Sequential deep learning for human action recognition[END_REF] and two decoders (one for each task) consisting of 3D deconvolutions[START_REF] Matthew D Zeiler | Deconvolutional networks[END_REF]. They train on normal data to reconstruct the current video clip and predict the future frame video clip. During testing, they use only the reconstruction error of the test video clip as the anomaly measure. They demonstrated that using these tasks together boosts the detection of video anomalies. Even though multiple task learning benefits from advantages of different tasks, it is often difficult to combine various tasks and furthemore this makes the model computationally very expensive. So, the tasks must be chosen wisely with taking into consideration the gain in performance (if there is) in relation to the expense of memory and computation time.

  . Hasan et al. (2016) and Chong et Tay (2017) increase the size of their input data by generating video clips with different strides (stride-2 and stride-3) between frames. Zhao et al. (2017) instead augments the data by creating video clips with various transformations like random cropping, changing brightness and Gaussian blurring. The main motivation behind this type of data transformation is to train the model with a sufficiently large dataset with different input variations so that the model becomes adaptive and retain the most pertinent features of normal input. Few important VAD works using data transformation in the second way are (Georgescu et al., 2021a,b; Astrid et al., 2021a,b; Park et al., 2020). Georgescu et al. (2021a) transforms data into various ways for different tasks like reading frames backwards or not for arrow of time detection, or skipping frames in video clip for irregular/regular motion task, etc. Georgescu et al. (2021b) and Astrid et al. (2021a,b) transform data to form pseudo anomalies in order to use normal and pseudo-abnormal data while training.

Figure 2 .

 2 Figure 2.19: Autoencoder based VAD work of Astrid et al. (2021a) with normal input and pseudo anomaly input, where probability p regulates quantity of pseudo anomalous input.

Figure 2 .

 2 Figure 2.19 shows an example of VAD work that use pseudo anomalies synthesized from normal data as an auxiliary input to the AE. The pseudo anomalies are formed by temporally striding the input to create temporal incoherence in video clip, which simulates

Figure 2 .

 2 Figure 2.21: Typical pipeline of a PIDS, composed of an optional offline training (left part) and an online detection (middle part). An illustrative example of different steps in online detection is shown in right part of the figure (best viewed in color).

Figure 2 .

 2 Figure 2.22: Pre-processing proposed by Buch et Velastin (2014), applied on the i-LIDS dataset. After normalization of video frame, patches of 16 × 16 pixels with 20% overlap are constructed on the region of interest, i.e., ground and fence. These patches are used as input for their PIDS instead of the whole video frame.

  use Gaussian background modeling to discriminate people (intruder) from background. The study of Vijverberg et al. (2014) uses background subtraction to extract object blobs from the video frame, which are later used for tracking. The work of Kim et al. (2018b) detects moving objects by comparing a background model with an input video frame in real-time.Figure 2.23 illustrates the

Figure 2 .

 2 Figure 2.23: Moving object detection using background subtraction proposed by Kim et al. (2018b). The first column represents frames from a thermal and color camera respectively, with objects of interest in white bounding boxes. The second column demonstrates the output after background subtraction and image binarization, where white regions represent detected moving objects.

  (a) False detections due to snowfall in a private dataset. Yellow blobs show detections along with their index number in blue. (b) Object split into extra detections in i-LIDS dataset. Blue blobs show splitobject detections along with their index number.

Figure 2 .

 2 Figure 2.24: Examples of incorrect detections using background subtraction method proposed by Vijverberg et al. (2013).

Figure 2 .

 2 Figure 2.25: Moving object classification architecture proposed by Kim et al. (2018b).Input frame of size 224 × 224 × 3 goes through a series of 2D convolutions, local response normalization and max-pooling units, followed by fully connected layers to finally give predictions of length ten, where each unit depicts probability of a particular class from a list of human (intruder) and wild animal classes.

  , an intruder is tracked by logging positions of foreground objects over time. The work of (Vijverberg et al., 2013) proposes a tracking algorithm based on tracklet clustering. Finally, Nayak et al. (2019) uses the simple on-line and real-time tracking (SORT) algorithm (Bewley et al., 2016) for tracking intruders.

Figure 2 .

 2 Figure 2.26: Working illustration of the PIDS proposed by Nayak et al. (2019) on CAVIAR (Crowley et al., 2005) (first row) and a private dataset (Nayak et al., 2019) (second row).The first column presents video frames from two different sites with green zones depicting the areas to protect. In the second column, potential intruder objects detected by YOLO detector are in yellow bounding boxes and since some of them are inside the protected zone, the alarm is raised (shown with red zone).

Contents 3 . 1

 31 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 3.2 Defining video anomaly detection . . . . . . . . . . . . . . . . . 66 3.2.1 VAD with anomaly information . . . . . . . . . . . . . . . . . . 67 3.2.2 VAD without anomaly information . . . . . . . . . . . . . . . . 69 3.3 Defining perimeter intrusion detection . . . . . . . . . . . . . . 70 3.3.1 Object in the Video . . . . . . . . . . . . . . . . . . . . . . . . 70 3.3.2 Intrusion event . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 3.3.3 Intrusion interval . . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.3.4 PIDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.4 Evaluation protocols . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.4.1 Frame-level evaluation . . . . . . . . . . . . . . . . . . . . . . . 74 3.4.2 i-LIDS evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.4.3 Edge-level evaluation . . . . . . . . . . . . . . . . . . . . . . . . 76 3.5 Evaluation metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 77 3.5.1 Threshold dependent metrics . . . . . . . . . . . . . . . . . . . 78 3.5.2 Threshold independent metrics . . . . . . . . . . . . . . . . . .

Figure 3 . 1 :

 31 Figure 3.1: Video with n frames of height H, width W and channels D = 1. Two objects, shown in orange and green bounding boxes, are defined as o 1 = ({b 1,1 , b 1,2 , b 1,3 }, c 1 ) and o 2 = ({b 2,3 }, c 2 ), where c 1 , c 2 ∈ C are the object classes. Here, {b 1,1 , b 1,2 , b 1,3 } are bounding boxes of object 1 on first three frames and {b 2,3 } represents bounding box of object 2 at frame 3.

Figure 3 . 2 :

 32 Figure 3.2: Illustration of an intrusion event caused by a single object. Video with n frames, where S na is shown with yellow surface and object with a green bounding box plus center. The object causes an intrusion event for four frames from frame I 4 to I 7 , colored in red.

  Figure 3.3 depicts two intrusion intervals of the video.

Figure 3 . 3 :

 33 Figure 3.3: Illustration of intrusion events of the video and intrusion intervals. Objects o 1 , o 2 and o 3 cause intrusion events IE(o 1 ), IE(o 2 ) and IE(o 3 ), marked with value 1. IE(V) is collection of all the frames with value 1. Two intrusion intervals II 1 and II 2 are shown in red intervals.

Figure 3 . 5 :

 35 Figure 3.5: The top subfigure is an illustration of the definitions of edge-level evaluation terms on ground truth, with time in abscissa and non-intrusion (0) and intrusion (1) class for each frame of the video in ordinate. The next two subfigures represent examples of alarms and possible outcomes (TP, FP, FN) for two different PIDSs evaluated by the edge-level protocol.

Figure 3

 3 Figure 3.6: Illustration of ROC (left sub figure) and PR curve (right sub figure). Each point in dotted lines represent values ( (TPR, FPR) for ROC and (Precision, Recall) for PR) corresponding to a threshold. The area under the ROC or PR curve, i.e., AUROC or AUPR is depicted with colored areas.

Figure 4 . 1 :

 41 Figure 4.1: Learning normality: A sliding window (in yellow) extracts video clips of fixed length. Each video clip is fed to S3DCAE, which reconstructs it. The mean squared error (MSE) between input and reconstructed clip is backpropagated to update learnable parameters of the autoencoder.

Figure 4 . 2 :

 42 Figure 4.2: S3DCAE architecture for PID with encoder and decoder. Conv3D and De-conv3D refer to 3D convolution and 3D deconvolution respectively, where s is stride, and the number that followsdenotes number of filters. The rightmost column refers to the data shape (channels, temporal length, height, width).

Figure 4

 4

  (a) GOFPID (Barthélemy, 2022) (b) Nayak et al. (2019) (c) Proposed method: AE-Adapt

Figure 4 . 4 :

 44 Figure 4.4: Example of per-frame predictions of the three methods on portions of video taken from i-LIDS test dataset. The intrusion intervals (ground truth) are shown in light red strips, the abscissa represents frames and ordinate shows prediction, where 1 signifies intrusion and 0 otherwise.

Figure 4 . 5 :

 45 Figure 4.5: The results on view 1 of i-LIDS test set for three methods: GOFPID (Barthélemy, 2022) (in green), Nayak et al. (2019) (in blue) and our method (in orange). The abscissa represents three evaluation protocols: frame-level (FL) (on left subfigure), i-LIDS and edge-level (EL) (on right subfigure). i-LIDS is evaluated by default for 10 s whereas, for EL, we show results for 1, 2 and 10 s. The ordinate represents values of three metrics: precision, recall and F 1 score.

Figure 4 .

 4 Figure 4.6: The results on view 2 of i-LIDS test set for three methods: GOFPID (Barthélemy, 2022) (in green), Nayak et al. (2019) (in blue) and our method (in orange). The abscissa represents three evaluation protocols: frame-level (FL) (on left subfigure), i-LIDS and edge-level (EL) (on right subfigure). i-LIDS is evaluated by default for 10 s whereas, for EL, we show results for 1, 2 and 10 s. The ordinate represents values of three metrics: precision, recall and F 1 score.

Figure 4 .

 4 Figure 4.7: Qualitative comparison of thresholding strategies on a video, with ground truth (GT) in blue, thresholds in green and alarms in red. Fixed threshold on reconstruction error of S3DCAE (top); fixed threshold on min-max (MM) rescaled reconstruction error (middle); adaptive threshold by moving z-score (MZ) of reconstruction error (bottom).

Figure 4 .

 4 Figure 4.8: AE-Adapt working illustration on a portion of video of i-LIDS test set. In the curve, reconstruction error (RE) is shown in black, threshold in green and alarm in red. GT stands for ground truth per frame, where blue region indicates intrusion. Some illustrative frames are shown above and below the plot, where the blue bounding box shows intruder.

Figure 5 . 1 :

 51 Figure 5.1: Overall schema of the proposed LUSS-AE method.

Figure 5 .

 5 Figure 5.3: Autoencoder architecture for VAD with encoder and decoder. Conv3D and Deconv3D refers to 3D convolution and 3D deconvolution respectively, and the number that follows them, denotes amount of filters. The rightmost column refers to the data shape (channels or filters, temporal length, frame height, frame width).

Figure 5 . 4 :

 54 Figure 5.4: Architecture of FFP head i.e., H FFP , for VAD task. Conv3D refers to 3D convolution, where k and s are kernel size and stride respectively. The rightmost column refers to the data shape.

Figure 5 . 5 :

 55 Figure 5.5: Architecture of PRP head i.e., H PRP , for VAD task. The different layers are depicted in middle column with violet color and the number that follows them, denotes amount of filters or units. Kernel size and stride are denoted by k and s, wherever applicable. The rightmost column refers to the data shape (channels, temporal length, frame height, frame width).

Figure 5 .

 5 Figure 5.6: Architecture of FFP head i.e., H FFP , for perimeter intrusion detection. Conv3D refers to 3D convolution, where k and s are kernel size and stride respectively.The rightmost column refers to the data shape (channels, temporal length, frame height, frame width).

Figure 5 . 7 :

 57 Figure 5.7: Architecture of PRP head i.e., H PRP , for perimeter intrusion detection. The different layers are depicted in middle column with violet color. Kernel size and stride are denoted by k and s, wherever applicable. The rightmost column refers to the data shape.

Figure 5 .

 5 Figure 5.8: LUSS-AE working illustration on video 11_0176 of Shanghai test set. Anomaly scores (A VCR , A FFP , A PRP , A) are plotted per video frame; red regions depict anomalous events and some illustrative frames are shown above the plot, where the yellow and red bounding boxes exhibit objects of interest and anomalies respectively.

Figure 5 .

 5 Figure 5.8 demonstrates an example of our method tested on a video with two anomalous events, both containing movement of bikes. Here, the people move with relatively normal speed while bikes move with fast speed. Also, the number of people are relatively less in this example and bike does occupy a big area, which means its displacement causes a big spatio-temporal change. We can observe that as soon as the bike enters the scene, we have a high jump in A PRP and it remains high until the bike exits the scene. It jumps up again in next scene and have highest value when two bikes move in the scene. Regarding A VCR and A FFP , the scores remain high when bikes are in the scene. Overall, our method detects both anomalous events in this example.

Figure 5 .

 5 Figure 5.9: LUSS-AE working illustration on video 05_0023 of Shanghai test set. Anomaly scores (A VCR , A FFP , A PRP , A) are plotted per video frame; red regions depict ground truth anomalous events, and some illustrative frames are shown above the plot, where the yellow and red bounding boxes exhibit objects of interest and anomalies respectively.

Figure 5 .

 5 Figure 5.9 demonstrates another example containing three different anomalous events:

Figure 5 .

 5 Figure 5.10: Illustration of different error measures on a test frame of the Avenue dataset. From left to right: actual frame (ground truth), predicted frame, error frame for MSE, for PSE and for BPE. (Best viewed in color)

  Figure A.1.

Figure A. 1 :

 1 Figure A.1: Example frames from the Sheeps dataset without intrusions. The images show video frames from different time of the day. The sheeps present are moving in random manner. Also, observe the sheeps at the top-right end of the third image.

  Figure A.2.

Figure A. 2 :

 2 Figure A.2: Example frames from the Sheeps dataset with intrusions shown in redbounding boxes. The images are shown in increasing level of intrusion detection difficulty.

2. 21

 21 Typical pipeline of a PIDS, composed of an optional offline training (left part) and an online detection (middle part). An illustrative example of different steps in online detection is shown in right part of the figure (best viewed in color). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 2.22 Pre-processing proposed by Buch et Velastin (2014), applied on the i-LIDS dataset. After normalization of video frame, patches of 16 × 16 pixels with 20% overlap are constructed on the region of interest, i.e., ground and fence. These patches are used as input for their PIDS instead of the whole video frame. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 2.23 Moving object detection using background subtraction proposed by Kim et al. (2018b). The first column represents frames from a thermal and color camera respectively, with objects of interest in white bounding boxes. The second column demonstrates the output after background subtraction and image binarization, where white regions represent detected moving objects. 57 2.24 Examples of incorrect detections using background subtraction method proposed by Vijverberg et al. (2013). . . . . . . . . . . . . . . . . . . . . . . 58 2.25 Moving object classification architecture proposed by Kim et al. (2018b). Input frame of size 224 × 224 × 3 goes through a series of 2D convolutions, local response normalization and max-pooling units, followed by fully connected layers to finally give predictions of length ten, where each unit depicts probability of a particular class from a list of human (intruder) and wild animal classes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 2.26 Working illustration of the PIDS proposed by Nayak et al. (2019) on CAVIAR (Crowley et al., 2005) (first row) and a private dataset (Nayak et al., 2019) (second row). The first column presents video frames from two different sites with green zones depicting the areas to protect. In the second column, potential intruder objects detected by YOLO detector are in yellow bounding boxes and since some of them are inside the protected zone, the alarm is raised (shown with red zone). . . . . . . . . . . . . . . . 61 3.1 Video with n frames of height H, width W and channels D = 1. Two objects, shown in orange and green bounding boxes, are defined as o 1 = ({b 1,1 , b 1,2 , b 1,3 }, c 1 ) and o 2 = ({b 2,3 }, c 2 ), where c 1 , c 2 ∈ C are the object classes. Here, {b 1,1 , b 1,2 , b 1,3 } are bounding boxes of object 1 on first three frames and {b 2,3 } represents bounding box of object 2 at frame 3. . . . . . 71 3.2 Illustration of an intrusion event caused by a single object. Video with n frames, where S na is shown with yellow surface and object with a green bounding box plus center. The object causes an intrusion event for four frames from frame I 4 to I 7 , colored in red. . . . . . . . . . . . . . . . . . . 72 3.3 Illustration of intrusion events of the video and intrusion intervals. Objects o 1 , o 2 and o 3 cause intrusion events IE(o 1 ), IE(o 2 ) and IE(o 3 ), marked with value 1. IE(V) is collection of all the frames with value 1. Two intrusion intervals II 1 and II 2 are shown in red intervals. . . . . . . . . . . . . . . 73 3.4 Illustration of i-LIDS evaluation protocol, highlighting its drawback on an intrusion example starting at 9 th second. Since no alarm has been raised in the first 10 seconds of the intrusion, a false negative (FN) is counted. Following alarms, at 22 nd and 37 th second, are marked as false positive (FP) because they do not occur within 10 seconds from the beginning of the intrusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 The top subfigure is an illustration of the definitions of edge-level evaluation terms on ground truth, with time in abscissa and non-intrusion (0) and intrusion (1) class for each frame of the video in ordinate. The next two subfigures represent examples of alarms and possible outcomes (TP, FP, FN) for two different PIDSs evaluated by the edge-level protocol. . . . . . 3.6 Illustration of ROC (left sub figure) and PR curve (right sub figure). Each point in dotted lines represent values ( (TPR, FPR) for ROC and (Precision, Recall) for PR) corresponding to a threshold. The area under the ROC or PR curve, i.e., AUROC or AUPR is depicted with colored areas. . 4.1 Learning normality: A sliding window (in yellow) extracts video clips of fixed length. Each video clip is fed to S3DCAE, which reconstructs it. The mean squared error (MSE) between input and reconstructed clip is backpropagated to update learnable parameters of the autoencoder. . . . . 4.2 S3DCAE architecture for PID with encoder and decoder. Conv3D and De-conv3D refer to 3D convolution and 3D deconvolution respectively, where s is stride, and the number that followsdenotes number of filters. The rightmost column refers to the data shape (channels, temporal length, height, width). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Detecting intrusion using AE-Adapt: an input video clip is reconstructed after going through the S3DCAE. The reconstruction error is calculated between these clips and fed to a moving z-score module, the output of which is thresholded to raise an intrusion alarm. . . . . . . . . . . . . . . . 4.4 Example of per-frame predictions of the three methods on portions of video taken from i-LIDS test dataset. The intrusion intervals (ground truth) are shown in light red strips, the abscissa represents frames and ordinate shows prediction, where 1 signifies intrusion and 0 otherwise. . . . . . . . . . . . 4.5 The results on view 1 of i-LIDS test set for three methods: GOFPID (Barthélemy, 2022) (in green), Nayak et al. (2019) (in blue) and our method (in orange). The abscissa represents three evaluation protocols: frame-level (FL) (on left subfigure), i-LIDS and edge-level (EL) (on right subfigure). i-LIDS is evaluated by default for 10 s whereas, for EL, we show results for 1, 2 and 10 s. The ordinate represents values of three metrics: precision, recall and F 1 score. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 The results on view 2 of i-LIDS test set for three methods: GOFPID (Barthélemy, 2022) (in green), Nayak et al. (2019) (in blue) and our method (in orange). The abscissa represents three evaluation protocols: frame-level (FL) (on left subfigure), i-LIDS and edge-level (EL) (on right subfigure). i-LIDS is evaluated by default for 10 s whereas, for EL, we show results for 1, 2 and 10 s. The ordinate represents values of three metrics: precision, recall and F 1 score. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table 1 .

 1 .7 illustrates various normal and abnormal events in this dataset. It is one of the biggest anomaly detection datasets with 330 training (274,515 frames) and 107 test videos (42,883 frames). Like other datasets, training videos only contain normal events and testing videos have both normal and abnormal events. For each anomalous event both pixel-level and frame-level ground-truth annotations are available. It is worth noting that recently[START_REF] Zhong | Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection[END_REF] proposed a reorganization of this dataset to also have annotated anomalies during training for weakly supervised learning. We focus on

	Dataset	Total	Number of frames Train Test Normal Abnormal	#Anomalies	#Anomaly types
	UCSD Ped2 4,560	2,550 2,010 2,924	1,636	23	5
	Avenue	30,652 15,328 15,324 26,832	3,820	47	5
	ShanghaiTech 317,398 274,515 42,883 300,308 17,090	158	11

1: Description of video anomaly detection datasets.

Table 1 .

 1

	View	Set	#Videos	#Intrusions	Number of videos per intrusion count 0 1 10 13 15 17 31
	View 1	train test	123 17	113 113	10 10	113 0	0 2	0 1	0 1	0 2	0 1
	View 2	train test	113 16	103 103	10 10	103 0	0 1	0 1	0 1	0 2	0 1

2 summarizes the i-LIDS dataset where view 1 and view 2 corresponds to

Table 1 . 2

 12

: i-LIDS Sterile Zone dataset description.

Table 2 .

 2 Similarly, Cho et al. (2022) proposed normal density estimation (NDE) unit which estimate the density of normality in their AE based VAD method.

	Method	Input	Task	Principal compo. Auxiliary compo.
	Hasan et al. (2016)	VC	Recon	AE	DT
	Chong et Tay (2017)	VC	Recon	AE, CLSTM	DT
	Zhao et al. (2017)	VC Pred, Recon	AE	OF, pre., DT
	Ravanbakhsh et al. (2017) FF	Recon	Adv	OF, pre.
	Tran et Hogg (2017)	VCP Classification	AE, SVM	OF
	Lee et al. (2018)	VC	Pred	AE, CLSTM, Adv
	Sabokrou et al. (2018)	FP Classification	Adv
	Liu et al. (2018)	VC	Pred	AE, Adv	OF, pre.
	Nguyen et Meunier (2019) FF	Recon	AE, Adv	OF, pre.
	Ye et al. (2019)	FF	Recon	AE, Adv
	Gong et al. (2019)	VC	Recon	AE	mem.
	Ionescu et al. (2019)	FO Classification	AE, SVM	clust., pre.
	Dong et al. (2020)	VC	Pred	AE, Adv	OF, pre.
	Tang et al. (2020)	VC Pred, Recon	AE, Adv
	Yu et al. (2020)	VCO	SST	AE	OF, pre.
	Chang et al. (2020)	VC	Recon	AE	clust., att.
	Park et al. (2020)	VC	Pred	AE	mem.
	Ouyang et Sanchez (2021) VCO	Recon	AE, GMM	pre., clust.
	Liu et al. (2021b)	VCO Pred, Recon	AE	OF, pre., mem.
	Georgescu et al. (2021a) VCO	SST	AE, FCN	pre., DT
	Georgescu et al. (2021b)	FO Classification	AE, Adv	OF, pre., DT
	Astrid et al. (2021a)	VC	Recon	AE	DT
	Astrid et al. (2021b)	VC	Recon	AE	DT
	Lv et al. (2021)	VC	Pred	AE	att.
	Le et Kim (2022)	VC	Pred	AE	att., pre.
	Cho et al. (2022)	VC	Recon	AE	NDE
	Szymanowicz et al. (2022) VC	Pred	AE	codebook
	Park et al. (2022)	VCP	Pred	AE	DT

1: Review of major video anomaly detection methods.

Table 2 .

 2 1 presents a review summary of various VAD methods. The first column refers to the publication associated with the method and the second column shows various input data (refer Section 2.5.1). In third column, the proxy tasks are mentioned where Pred, Recon and SST refer to prediction, reconstruction, and self-supervised task, respectively. The fourth column refers to main learning component of the method, where the undeclared terms Adv, CLSTM, GMM and FCN refer to adversial unit, convolutional LSTM, gaussian mixture modeling and fully connected network. Final column is of the auxiliary component, where the abbreviated terms are: DT -data transformation, OF -optical flow, pre. -pre-trained feature extractor, clust. -clustering, mem. -memory unit, att.attention unit and NDE -normal density estimation. We can observe the following from Table2.1. Most of the methods use video clip based inputs like VC, VCP and VCO and only a very few methods use the frame based input.

Table 2 .

 2 2 summarizes major PIDS works with various methods used in different steps of the pipeline. It can be observed that most systems use the visual camera for data acquisition and only one system is multi-camera based. Most traditional systems use background modeling for detection, whereas recent deep-learning-based models use 2D CNN or YOLO detector. Concerning tracking, the Kalman filter, particle filter, trackletbased tracking and SORT algorithm are used. For alarms, most systems have their own method-specific rules. Three systems use supervised training and the rest do not have a training step.

	Publication	Data Acquisition	Pre-processing	Detection Tracking	Post-processing	Alarm	Model training
	Buch et Velastin (2008)	Visual camera	frame patches, FFT	inter-frame differencing	Kalman filter		rule-based	none
	Vijverberg et al. (2013)	Visual camera		background subtraction	tracklet tracking			none
	Buch et Velastin (2014)	Visual camera	frame patches, FFT	Gaussian background modeling	Kalman filter		rule-based	none
	Vijverberg et al. (2014)	Visual camera		background subtraction	tracklet tracking		rule-based	supervised
	Kim et al. (2018b)	Multi-camera: visual and thermal	resize, calibration, perimeter	2D CNN	particle filter	outside perimeter	rule-based	supervised
	Cermeño et al. (2018)	Visual camera	perimeter	RGA	particle filter	object size rule	rule-based	none
	Nayak et al. (2019)	Visual camera	perimeter	YOLO v2	SORT		rule-based	supervised

Table 2 .

 2 2: PIDS reviewed in chronological order, where columns represent steps of the pipeline and model training needs. denotes unavailability of the step, whereas denotes that the step is available but not detailed.

  . Concerning VAD, since most datasets do not have a training set with anomalies, some works try to rearrange the datasets to add anomalies in the training set and provide their annotations. For example, He et al. (2018) re-organized UCSD Ped datasets (Li et al., 2013) and provided annotations. Similarly, Landi et al. (2019) and Liu et Ma (2019) provided bounding box annotations in UCF-Crime dataset. Given both normal and abnormal classes during training, a supervised binary classifier can efficiently make a distinction between normal and abnormal videos during testing. Some works on supervised video anomaly detection are

Algorithm 1 :

 1 Edge-Level Evaluation of a PIDS 1 Initialize variables m, m pre and m post . 2 Calculate IN for all IIs of the video. 3 If two or more INs intersect, merge them into a single expanded IN. 4 Calculate intrusion beginning neighbourhood IBN for each II. 5 Obtain alarms O(P) from the PIDS. 6 Calculate TP, FN and FP. 7 Calculate precision, recall and other metrics.

Table 4 .

 4 1: Quantitative comparison of thresholding strategies, on View 1 of i-LIDS dataset. S3DCAE stands for proposed autoencoder with frame reconstruction error, MM for min-max scaling, and MZ for moving z-score. EL (10 s) is used as evaluation protocol.

	Methods	Precision Recall F 1
	S3DCAE	0.54	0.44 0.49
	S3DCAE + MM	0.49	0.59 0.54
	S3DCAE + MZ	0.92	0.91 0.92

Table 5 .

 5 [START_REF] Lu | Learnable locality-sensitive hashing for video anomaly detection[END_REF], and the new results have been updated in the table. 1: Quantitative comparison with the existing state of the art methods: AUROC (%) for VAD is computed on Ped2, Avenue and Shanghai datasets. Numbers in bold indicate the best performance, and

		Methods	Ped2	Avenue	Shanghai
	Object-centric	OCAE (Ionescu et al., 2019) Any-Shot (Doshi et Yilmaz, 2020a) VEC (Yu et al., 2020) SSMTL (Georgescu et al., 2021a) Back-Agnostic (Georgescu et al., 2021b)	94.30 97.80 97.30 92.4 98.70	87.40 86.40 89.60 91.50 92.30	78.70 71.62 74.80 82.40 82.70
		AnoPCN (Ye et al., 2019)	96.80	86.20	73.60
		MemAE (Gong et al., 2019)	94.10	83.30	71.20
		UNet-inte (Tang et al., 2020)	96.30	85.10	73.00
		Cluster AE (Chang et al., 2020)	96.50	86.00	73.30
	Fully unsupervised	MNAD (Park et al., 2020) * MNAD (Menon et Stephen, 2021) STEAL Net (Astrid et al., 2021b) LNTRA (Astrid et al., 2021a) MPN (Lv et al., 2021) * MPN (Lu et al., 2022)	97.00 * 96.33 98.40 96.50 96.90 * 96.13	88.50 * 87.91 87.10 84.91 89.50 * 83.90	70.50 * 67.81 73.70 75.97 73.80 * 73.00
		ITAE (Cho et al., 2022)	97.30	85.80	74.70
		VQU-Net (Szymanowicz et al., 2022)	89.20	88.30	-
		FastAno (Park et al., 2022)	96.30	85.30	72.20
		LUSS-AE [ours]	98.52	89.04	81.21

* indicate non-reproducible results.

Table 5 .

 5 2: Influence of different tasks (VCR, FFP and FRP) used during training and testing of AE on Avenue and Shanghai datasets, in terms of AUROC (%).

	Tasks	AUROC (%)
	VCR FFP PRP Avenue Shanghai
		82.72	73.11
		87.95	76.35
		87.43	79.20
		89.04	81.21

Table 5 .

 5 3: Influence of tasks (VCR, FFP and PRP) used during training of autoencoder on Avenue and Shanghai datasets, when tested only with VCR, in terms of AUROC (%).

	Training tasks	Testing only with VCR Avenue Shanghai
	VCR	82.72	73.11
	VCR + FFP + PRP 86.48	75.37

Table 5 . 4

 54

	Error measure	Testing tasks VCR FFP
	MSE	84.95 85.38
	PSE	85.96 86.61
	BPE	86.48 87.00

: Influence of error measure (MSE, PSE and BPE) on each task (VCR and FFP) during testing of AE on Avenue dataset, in terms of AUROC (%).

Table 5 .

 5

		EL (1 s)	EL (2 s)	EL (10 s)
		Precision Recall F 1 Precision Recall F 1 Precision Recall F 1
	GOFPID	0.69	0.61 0.65	0.76	0.87 0.81	0.78	0.98 0.87
	Nayak et al. (2019) 0.62	0.52 0.57	0.70	0.74 0.72	0.74	0.93 0.83
	AE-Adapt	0.90	0.74 0.81	0.92	0.90 0.91	0.92	0.91 0.92
	LUSS-AE-Adapt	0.91	0.78 0.84 0.93	0.91 0.92 0.95	0.93 0.94

5: PID methods comparison based on edge-level (EL) evaluation at 1, 2 and 10 seconds for View 1 of i-LIDS dataset.

Table 5 .

 5 8: PID methods comparison based on edge-level (EL) evaluation at 10 seconds in the Sheeps dataset.

	Nayak et al. (2019)	0.23	0.78 0.36
	Industrial PIDS	0.32	0.57 0.41
	LUSS-AE-Adapt	0.57	0.81 0.67

Table 5 .

 5 Precision Recall F 1 Precision Recall F 1 Precision Recall F 1 9: PID methods comparison at different difficulty levels in the Sheeps dataset, where edge-level (EL) evaluation at 10 seconds is used.

			-1		Difficulty -2		Difficulty -5
	Nayak et al. (2019) 0.21	1	0.35	0.25	1	0.40	0.20	1	0.33
	Industrial PIDS	0.45	0.83 0.58	0.50	0.77 0.60	0.10	0.22 0.14
	LUSS-AE-Adapt	0.86	1 0.92 0.73	0.89 0.80 0.71	0.56 0.63

  4.7 Qualitative comparison of thresholding strategies on a video, with ground truth (GT) in blue, thresholds in green and alarms in red. Fixed threshold on reconstruction error of S3DCAE (top); fixed threshold on min-max (MM) rescaled reconstruction error (middle); adaptive threshold by moving z-score (MZ) of reconstruction error (bottom). . . . . . . . . . . . . . . . . 94 4.8 AE-Adapt working illustration on a portion of video of i-LIDS test set. In the curve, reconstruction error (RE) is shown in black, threshold in green and alarm in red. GT stands for ground truth per frame, where blue region indicates intrusion. Some illustrative frames are shown above and below the plot, where the blue bounding box shows intruder. . . . . . . . . . . . 96 5.1 Overall schema of the proposed LUSS-AE method. . . . . . . . . . . . . . 101 5.2 Overall schema of the proposed LUSS-AE method during testing. A window of T + 1 consecutive frames is drawn sequentially from the test video. The first T frames from this window are the input of the system and the output for each task is computed on it. The anomaly score is determined for each task, i.e., A VCR , A FFP and A PRP , and the final anomaly score is their weighted sum. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 5.3 Autoencoder architecture for VAD with encoder and decoder. Conv3D and Deconv3D refers to 3D convolution and 3D deconvolution respectively, and the number that follows them, denotes amount of filters. The rightmost column refers to the data shape (channels or filters, temporal length, frame height, frame width). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.4 Architecture of FFP head i.e., H FFP , for VAD task. Conv3D refers to 3D convolution, where k and s are kernel size and stride respectively. The rightmost column refers to the data shape. . . . . . . . . . . . . . . . . . . 107 5.5 Architecture of PRP head i.e., H PRP , for VAD task. The different layers are depicted in middle column with violet color and the number that follows them, denotes amount of filters or units. Kernel size and stride are denoted by k and s, wherever applicable. The rightmost column refers to the data shape (channels, temporal length, frame height, frame width). . . . . . . . 108 5.6 Architecture of FFP head i.e., H FFP , for perimeter intrusion detection. Conv3D refers to 3D convolution, where k and s are kernel size and stride respectively. The rightmost column refers to the data shape (channels, temporal length, frame height, frame width). . . . . . . . . . . . . . . . . . 109 5.7 Architecture of PRP head i.e., H PRP , for perimeter intrusion detection. The different layers are depicted in middle column with violet color. Kernel size and stride are denoted by k and s, wherever applicable. The rightmost column refers to the data shape. . . . . . . . . . . . . . . . . . . . . . . . . 109 5.8 LUSS-AE working illustration on video 11_0176 of Shanghai test set. Anomaly scores (A VCR , A FFP , A PRP , A) are plotted per video frame; red regions depict anomalous events and some illustrative frames are shown above the plot, where the yellow and red bounding boxes exhibit objects of interest and anomalies respectively. . . . . . . . . . . . . . . . . . . . . . 112

2.2. BACKGROUND

2.3. SPATIO-TEMPORAL FEATURE LEARNING FROM VIDEOS

3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Acknowledgments

Chapter 4

Unsupervised autoencoder and adaptive detection

In this chapter, we introduce an unsupervised perimeter intrusion detector. We first introduce how we learn spatio-temporal normality from videos with an end-to-end trainable neural network, followed by how intrusions are detected using an adaptive mechanism that handles long-term variations in scene dynamics. We then provide a thorough comparative analysis with other methods on i-LIDS dataset, using different evaluation protocols. Part of this work was published in IEEE International Conference on Image Processing (ICIP) 2022 (Lohani et al., 2022a).

Contents

For real-time applications, we need a reconstruction error for each frame that the system encounters. Our method is designed for video clip level and not frame level. The video clips of length T are formed with temporal gap of one between each video clip. This means that if we have a video with n frames {I 1 , I 2 , . . . , I n } and T = 8, the first video clip is {I 1 , I 2 , . . . , I 8 }, the second video clip is {I 2 , I 3 , . . . , I 9 }, etc. Since each video clip provides a single scalar value of reconstruction error (see Equation 4.3). To convert this video clip level reconstruction error to frame level error, we assume that the value of each video clip reconstruction error corresponds to the last frame of the video clip. This implies that for first T -1 frames of a video, there are no frame level reconstruction errors, i.e., r x with x ≥ T .

Adaptive thresholding using moving z-score

A PIDS installed in a site records videos for many days. This long video sequence has very high scene dynamics with changes in light, weather, temperature, scene background, etc. Since reconstruction error changes with scene dynamics, naively thresholding it can provide undesirable outcomes [START_REF] Ribeiro | A study of deep convolutional auto-encoders for anomaly detection in videos[END_REF]. To resolve this issue, most work using unsupervised autoencoder try to standardize the per-frame reconstruction error for each video [START_REF] Ribeiro | A study of deep convolutional auto-encoders for anomaly detection in videos[END_REF][START_REF] Ramachandra | A survey of single-scene video anomaly detection[END_REF]. They usually use the min-max (MM) rescaling which force the error values to be in the [0,1] range, where 0 and 1 represents normal and intrusion case, respectively. Thresholding on this rescaled reconstruction error is dangerous because this standardization will force the lower values of reconstruction errors (e.g., values of night) to be considered as normal (since forced to be near 0), even if there is an intrusion.

To address this issue of adjusting to scene dynamics, we propose an adaptive mechanism which follows the reconstruction error along time and trigger an alarm as soon as it deviates from the normal behavior and then continue adapting to the new values. It is also well aligned with perimeter intrusion detection definition, where the goal is to detect just the beginning of intrusions and not every intrusion frame in the video. We compute the moving z-score (MZ) of reconstruction errors to provide a temporal standardization of values [START_REF] Barthélemy | The Riemannian potato field: a tool for online signal quality index of EEG[END_REF]. Once standardized, values can be easily compared to a fixed threshold z th . Using frame level reconstruction errors for first t frames, i.e., {r 1 , r 2 , . . . , r t }, the system is initialized by computing the mean and standard deviation of them, denoted as µ t and σ t , respectively. For the (t + 1) th frame, z-score of reconstruction error r t+1 is computed as:

If this value is greater than a fixed chosen threshold, i.e., z t+1 ≥ z th , the system raises an intrusion alarm. Regardless of whether an alarm is raised or not, the moving mean and standard deviation are updated as:

where α ∈ [0, 1] defines the speed of the exponential update. If α is close to 1, then it implies that the system adapts itself quickly with the current value of reconstruction error, without taking much into account the evolution of error. Contrary to this, when α score. While the one that goes through decoder bifurcates again into two branches, i.e., reconstruction and H FFP head. With reconstruction branch, the input clip is reconstructed as Î1:T and the A VCR score is computed. The branch of H FFP head produces the future frame ÎT+1 , which is compared with frame I T +1 of the input to calculate the A FFP . The anomaly scores for the three tasks are described below: (i) the VCR anomaly score is defined as:

where I t , Ît are the t th frame and its reconstruction respectively, and E can be one of three error measures, i.e., MSE, PSE or BPE respectively.

(ii) the FFP anomaly score is computed as:

(5.15) (iii) the PRP anomaly score is defined as the probability of accelerated class (c = 2): 5.16) where ŷ is the output of PRP branch as defined in Eq. (5.8).

The final score is defined as:

where α 1 , α 2 , α 3 are the weights in [0, 1] for the three scores.

Conclusion and perspectives

In this thesis, we explored unsupervised deep learning based approaches for learning spatio-temporal representations of videos. These approaches were developed for application in video surveillance domain and in particular for video anomaly detection task and perimeter intrusion detection task. We first formally defined these tasks in chapter 3, along with suitable evaluation protocols. Then, in chapter 4, we proposed an unsupervised autoencoder based method for perimeter intrusion detection. It was coupled with an adaptive thresholding strategy to handle changing scene dynamics. Finally, we also proposed a new generic method in chapter 5. With carefully chosen tasks, it leverages unsupervised and self-supervised learning in an single autoencoder. This approach is applicable to both video anomaly detection and perimeter intrusion detection task. All the proposed approaches were compared with state of the art methods on standard datasets and obtained superior performances.

Importance of tasks definition and suitable evaluation

Before solving any problem, it is essential to correctly understand it first and this requires defining it as precisely as possible. That is why, in chapter 3, we proposed mathematical definitions for the two main tasks that we address in this thesis along with corresponding evaluation protocols.

Concerning perimeter intrusion definition (PID), we first defined how an intrusion event is triggered by an object. An object causes an intrusion event if it belongs to a non-authorized class (like person, car, etc.) and is moving in a protected area during a prohibited time interval. The classes considered as non-authorized, the protected area and the prohibited time interval depend on the end user or client and can vary from one site to another. We can have multiple objects causing intrusion events and these events can also overlap. For example, if a car causes an intrusion event and then after some time, a person also triggers an intrusion event (when car is still there), then these two events overlap. In the case of PID, at any instant, we are concerned if there is an intrusion event, regardless of whether one object or many objects are causing it. Therefore, it is safe to merge intrusion events when they overlap. Consequently, the intrusion event of the whole video is just the union of its intrusion events. The constituent frames of the intrusion event of video are called the intrusion frames. The main concern in PID are the time intervals in video where intrusions occur. We define these time intervals using intrusion intervals, where an intrusion interval corresponds to a contiguous sequence of intrusion frames. The goal of PID task is to detect the beginning of intrusions intervals of the video. The beginning is essential, because if an intrusion is detected too late, then that detection is not very useful in terms of PID requirements (for client/user, the intruder is too close or inside the protected site) (i- LIDS Team, 2006;Buch et Velastin, 2014)

List of Tables